Castro had other matters on his mind. He asked endless questions about a Boston-based project to archive thousands of documents left in Ernest Hemingway’s former Havana home. And he was indignant at the imprisonment of three Cuban spies convicted in 2001. This was typical: no American official leaves Cuba, says one former Obama official, without hearing “a litany of historical grievances.”

Even so, McGovern sensed a distinctly hopeful tone underneath. “We have to talk about the present and the future,” Castro told him. “Because if we talk about the past we will never resolve it.”

( Also on POLITICO: Obama’s December surprise)

“I thought, ‘Maybe this is a guy we can do business with,’ ” McGovern says.

That hope became reality on Wednesday, when President Obama announced that he would normalize relations between America and Cuba, more than 50 years after they became a casualty of Fidel Castro’s communist revolution and Cold War geopolitics. Coming with little warning and at a time when Washington’s attention has been fixed on the Middle East and Russia, the announcement was as sudden as it was stunning.

But it was a work long in progress, the product of 18 months of furtive diplomacy in which President Obama designated an unlikely emissary — his foreign policy speechwriter — to secret meetings with Cuban officials in Canada and Rome.

It was also the fulfillment of Obama’s promise in 2008 that he would, under the right conditions, engage in direct diplomacy with the Cuban leadership. He might have acted sooner but for a handful of prisoners — including USAID contractor Alan Gross, who was abducted less than a year into Obama’s presidency — and at least one bungled attempt at freelance diplomacy.

( Also on POLITICO: GOP livid over Cuba)

“The changes are really consistent with the road that we’ve been on since the beginning,” says Dan Restrepo, Obama’s top aide for Latin American affairs until mid-2012. “You can connect a pretty direct line from here from a debate in South Carolina [in 2007] where the president first talked about engaging in direct diplomacy with our adversaries, including Castro.”

Obama’s rivals ridiculed him as naïve after that debate, in which he said he would talk directly with the leaders of Iran, North Korea and Cuba. Obama would later hedge that position. But he never backed down from his view that America’s diplomacy with its enemies had to be revamped.

Once in office Obama took modest steps to liberalize U.S. policy towards Cuba, including by relaxing a travel ban for family members of Cubans living in the U.S. But the December 2009 arrest of Gross, a USAID contractor sentenced to a 15-year prison sentence for bringing computer equipment to Havana’s Jewish community — he was accused of being a spy — froze grander plans. Some senior officials speculated that hard-liners in Havana had ordered Gross’s arrest for just that reason.

But Obama was determined to press on. He had cited Cuba policy as an area where politicians “make the same empty promises year after year,” saying that “they come down to Miami, they talk tough, they go back to Washington and nothing changes in Cuba.”

( Also on POLITICO: The price of U.S.-Cuba deal)

The White House saw Cuba’s release of 52 political prisoners in 2010 as an important sign of hope, even as Gross’s imprisonment remained a sticking point.

The problem was only exacerbated by the involvement of former UN Ambassador and New Mexico Governor Bill Richardson, who had brokered some high-profile foreign prisoner releases in the past and volunteered himself for a bid to free Gross.

“Richardson decided he was going to go and save the day,” says a former administration official. “He was freelancing.”

But the New Mexican’s visit was a fiasco. After Richardson arrived in Havana in September 2011, traveling as a private citizen, he portrayed Gross to the media as a political prisoner, which angered Cuban officials, who denied him a meeting with the American. Richardson left Cuba empty-handed, and recently conceded to NewsmaxTV that his grandstanding had been a mistake.

“I screwed that one up,” Richardson said. Obama officials heartily agreed.

By May of this year, advocates of a change in Cuba policy were growing impatient. Gross was in weekly telephone contact with Washington via Tim Rieser, an aide to Senator Patrick Leahy (D-Vt.), who also took a keen interest in US-Cuba relations. His health was failing and he had thoughts of suicide. Secretary of State John Kerry would later warn his Cuban counterpart that if something happened to Gross, “there would never, ever be a better relationship with the US,” according to an administration official.

Leahy and several other members of Congress visited Obama in the Oval Office, where they were joined by Joe Biden for what McGovern describes as “a pretty candid meeting” in which he and his colleagues pressed Obama to move faster.

“You said you were going to do this!” McGovern told the president. “Let’s just do it!”

Obama said he was working on it. In fact, he had begun his final and decisive push a year earlier. Obama had discussed the importance of a new Cuba policy when he first raised the Secretary of State job with Kerry. In the spring of 2013, Obama authorized direct talks between key White House staffers and Cuban officials.

One of those staffers was Ben Rhodes, a deputy national security advisor in his mid-30s who has worked with Obama since 2007. Rhodes’s background is in speechwriting and press management, but he has moved steadily deeper into Obama’s inner advisory circle. The former creative writing student has no formal diplomatic experience, but one former White House colleague says he made sense as an emissary to a country in which the U.S. has no embassy or ambassador: “All it takes is one Google search for these guys to know that Ben speaks to the President, and has daily access, and can be a trusted back channel.”

Joined by the NSC’s top staffer for Latin America, Ricardo Zuniga, Rhodes traveled to Canada in June 2013 to meet with Cuban officials for the first of several meetings to discuss a deal that would free Gross, and pave the way to a grander diplomatic thaw.

As a condition for Gross’s release, the Cubans had initially demanded an end to U.S. pro-democracy programs in their country, according to a former official. But their interest shifted to the three remaining members of the “Cuban Five,” who had been arrested in Miami for spying in 2001. One of them, Gerardo Hernández, was convicted in 2001 of conspiracy to commit murder in connection with the 1996 shootdown of two private airplanes dropping anti-regime leaflets.

McGovern says Castro was particularly exercised by Hernández’s sentence. “I gave the order. I’m the one responsible,” he said.

U.S. officials refused to swap the imprisoned Cuban spies for Gross, whom they insisted had not been a spy like them. But a solution appeared in the form of an American spy who had been held in Cuba since the mid-1990s. A plan emerged to trade the Cubans for the unnamed American. Gross’s release would technically be an unrelated gesture of goodwill.

The talks were given a mighty push by Pope Francis, an Argentinian and a towering figure in overwhelmingly Catholic Cuba. Francis sent letters to both Obama and Castro in early summer urging a prisoner swap and better diplomatic relations. One senior administration official called the pope’s intervention “very rare,” adding: “We haven’t received [other] communications like this from the pope that I’m aware of.”

Francis also hosted a decisive meeting at the Vatican this fall — attended by Rhodes and Zuniga — in which the two sides discussed final arrangements.

But the prisoner swap wouldn’t be enough. Obama had promised voters that he would not shift his Cuba policy unless the Castro regime undertook broader reforms.

“If you take significant steps toward democracy, beginning with the freeing of all political prisoners, we will take steps to begin normalizing relations,” Obama said in a May 2008 campaign trip to Florida.

As part of the final agreement, Castro’s regime agreed to release another 53 political prisoners. It also has promised to increase internet connectivity and allow U.S. telecom providers to operate in the country.

But Castro has agreed to no specific democratic reforms, and Obama’s critics — including some Democrats — trashed the announcement as a capitulation to a dictator.

For Obama, however, it was the fulfillment of a mission he first declared more than seven years ago. And it was an opportunity to shape a foreign policy legacy that has mostly been defined by crises around the world.

But it was a secret until the very end. Even McGovern, long known in Washington as one of Cuba’s closest friends, had no inkling a major breakthrough was imminent — not until his phone rang Tuesday night.

It was Vice President Joe Biden. “I’d like you to come down to Andrews [Air Force Base] and be there when Alan lands,” he said.