President Museveni has promised to "work closely" with Maj. Gen. (Rtd) Evariste Ndayishimye, who was sworn-in on Thursday as the new president of Burundi.
Mr Museveni said that he's hopeful that the new leader will have a successful tenure in office to ensure continued peace, progress and prosperity of Burundi.
The new leader was sworn following a massive victory of the CNDD-FDD party in the May 20 elections.
"The resounding victory was a clear manifestation of the level of trust and confidence the people of Burundi have in the new President and his party. I also thank the people of Burundi for being peaceful during and after the election," Mr Museveni tweeted. "On my own behalf and that of Ugandans, I wish His Excellency Ndayishimye. We also look to working closely with you to strengthen bilateral ties of our two countries."
Mr Ndayishimiye was sworn in at a colourful ceremony on Thursday, taking the helm of a troubled nation after the sudden death of his predecessor.
At a stadium in the capital Gitega where the crowd was dressed in identical outfits, Mr Ndayishimiye pledged to "devote all my force to defending the superior interests of the nation and ensure the national unity and cohesion of the Burundian people, peace and social justice."
A 21-gun salute rang out after he signed the pledge before the seven members of the constitutional court and took a tour of the stadium before the cheering crowd.
Before taking the oath, Mr Ndayishimiye knelt surrounded by the leaders of the Catholic, Anglican, evangelical and Muslim faiths, who prayed for him.
"Understand that you are a son of God and as such must bring peace among Burundians, you know how much we need it," said the Catholic archbishop of Gitega, Simon Ntamwana.
"Bring back to our country the refugees in the camps, bring back the intellectuals in exile so that they can take part in the development of our country, renew ties with the international community so they can help us develop," he said.
Nkurunziza, who ruled the East Africa nation for 15 often tumultuous years, was said by the government to have died of a heart attack last week.
His 2015 run for a third term in office sparked protests and a failed coup, with violence leaving at least 1,200 dead while some 400,000 fled the country.
UN rights investigators have said the period since 2015 has been marked by likely crimes against humanity committed by state forces, citing extrajudicial executions, arbitrary arrests, disappearances, torture and sexual violence.