The 70-year-old Hayatou had ruled African soccer since 1988 and was FIFA's most senior vice president because of his status as the longest-serving executive at the world soccer body. He also lost that position Thursday, and his place on FIFA's ruling council.
Ahmad won 34-20 in the vote of Africa's 54 member federations. He will become a FIFA vice president and join the council.
Hayatou left the congress room in Addis Ababa when the result was announced without commenting.
"This is sweet victory," said Ahmad, who was held aloft by officials when his victory was announced and then hugged people around him. "When you work hard for years and months and you succeed, that is great."
Ahmad takes over on an initial four-year term, and has promised to modernize CAF and make it more transparent.
His first job, he said earlier Thursday, would be to introduce a new code of ethics for the African soccer body. He has also pledged to extend ethics checks on African soccer officials.
Hayatou's departure is the end of an era in African soccer. The Cameroonian, who took power when Ronald Reagan was still president of the United States, was seeking an eighth term in office.
-DailyMail