The attacker stabbed the border police officer near the Damascus Gate in the historic walls of the Old City.
“The stabbed officer was lightly wounded and was evacuated for medical treatment,” the police said in a statement.
“Border Police officers engaged with the terrorist, neutralized him with gunfire, and concluded the attack swiftly,” the police said.
A police spokesman later told AFP the attacker had been shot dead. He did not provide details about the attacker’s identity.
Tensions between Palestinians and Israeli Jews are frequent in the Old City and have only heightened since the start of the Gaza war more than 11 months ago.
In a separate statement, Israel’s Magen David Adom emergency service said the 20-year-old wounded officer had an injury to his upper body.
Police and border forces were on the scene and investigating the incident, it added.
Jerusalem, and in particular the Old City, is a holy city for the three Abrahamic religions and remains a key issue at the heart of the Israeli-Palestinian conflict.
Israel claims Jerusalem as its indivisible capital, but the United Nations and the international community consider Israel’s annexation of East Jerusalem to be illegal.
Palestinians aspire to make occupied east Jerusalem, which includes the walled Old City and its holy sites, the capital of a future independent state.