The sultan deposed his father in a bloodless coup with British support in 1970 and set Oman on a path to development, using its oil wealth.
Widely regarded as popular, he was also an absolute monarch and any dissenting voices were silenced.
No cause of death has been confirmed. His cousin Haitham bin Tariq Al Said has sworn in as his successor.
The former culture and heritage minister took the oath of office on Saturday after a meeting of the Royal Family Council, the government said.
The sultan is the paramount decision-maker in Oman. He also holds the positions of prime minister, supreme commander of the armed forces, minister of defence, minister of finance and minister of foreign affairs.
Last month Sultan Qaboos – who had no heir or designated successor – spent a week in Belgium for medical treatment, and there were reports he was suffering from cancer.
“With great sorrow and deep sadness… the royal court mourns His Majesty Sultan Qaboos bin Said, who passed away on Friday,” a court statement said earlier, announcing three days of national mourning.
Images showed a crowed of men gathered outside the Sultan Qaboos Grand Mosque in the capital, Muscat, where the casket had been taken and prayers were being held.
For almost five decades, Sultan Qaboos completely dominated the political life of Oman, which is home to 4.6 million people, of whom about 43% are expatriates.
At the age of 29 he overthrew his father, Said bin Taimur, a reclusive and ultra-conservative ruler who banned a range of things, including listening to the radio or wearing sunglasses, and decided who could get married, be educated or leave the country.
Sultan Qaboos immediately declared that he intended to establish a modern government and use oil money to develop a country where at the time there were only 10km (six miles) of paved roads and three schools.
In the first few years of his rule, he suppressed with the help of British special forces an insurgency in the southern province of Dhofar by tribesmen backed by the Marxist People’s Democratic Republic of Yemen.
Described as charismatic and visionary, he pursued a neutral path in foreign affairs and was able to facilitate secret talks between the United States and Iran in 2013 that led to a landmark nuclear deal two years later.
A degree of discontent surfaced in 2011 during the so-called Arab Spring. There was no major upheaval in Oman, but thousands of people took to the streets across the country to demand better wages, more jobs and an end to corruption.
Security forces initially tolerated the protests, but later used tear gas, rubber bullets and live ammunition to disperse them. Two people were killed and dozens of people were injured. Hundreds were prosecuted under laws criminalising “illegal gatherings” and “insulting the sultan”.
The protests failed to produce anything in the way of major change. But Sultan Qaboos did remove several long-serving ministers perceived as corrupt, widened the powers of the Consultative Council, and promised to create more public sector jobs.
Since then, the authorities have continued to block local independent newspapers and magazines critical of the government, confiscate books, and harass activists, according to Human Rights Watch.