Mr Javid rejected an order to fire his team of aides, saying “no self-respecting minister” could accept such a condition.
He has been replaced as chancellor by Chief Secretary to the Treasury Rishi Sunak – who just seven months ago was a junior housing minister.
Mr Javid had been due to deliver his first Budget in four weeks’ time.
The former home secretary was appointed chancellor by Mr Johnson when he became prime minister in July, making him the shortest.
His resignation follows rumours of tensions between Mr Javid and the prime minister’s senior adviser Dominic Cummings.
“He has turned down the job of chancellor of the exchequer,” a source close to Mr Javid, who had been expected to remain in place, said.
“The prime minister said he had to fire all his special advisers and replace them with Number 10 special advisers to make it one team. The chancellor said no self-respecting minister would accept those terms.”
In other reshuffle moves:
Paymaster General Oliver Dowden is the new culture secretary, replacing Baroness Morgan;
Northern Ireland Secretary Julian Smith and Business Secretary Andrea Leadsom have been sacked;
Attorney General Geoffrey Cox, who attended cabinet, was asked to resign by the PM;
Priti Patel remains as Home Secretary;
Dominic Raab remains as Foreign Secretary;
Michael Gove remains in his role as minister for the Cabinet Office;
Health Secretary Matt Hancock is staying in his post, and Liz Truss will carry on as international trade secretary and minister for women and equalities.
International Development Secretary Alok Sharma been appointed business secretary and minister for the upcoming climate conference COP26, in Glasgow.
He is being replaced at the international development department by junior defence minister Anne-Marie Trevelyan.
Boris Johnson is expected to appoint a new minister to oversee the building of the HS2 rail line, final approval for which was given this week.
There will now be a new joint team of No 10 and 11 special advisers, the BBC understands.
Mr Sunak, 39, was educated at Winchester College and Oxford University, after which he went on to found an investment firm.
In 2015, he was elected MP for Richmond, North Yorkshire, replacing former Conservative leader William Hague.
Mr Sunak became a housing minister in 2018, before being promoted to chief secretary to the Treasury last July.
He stood in for Mr Johnson during the BBC’s seven-way debate ahead of December’s general election.
Arriving at the Treasury, Mr Sunak said he was “delighted to be appointed” chancellor and had “a lot to get on with”.
Commenting on Mr Javid’s resignation, Labour’s shadow chancellor John McDonnell said: “This must be a historical record with the government in crisis after just over two months in power.
“Dominic Cummings has clearly won the battle to take absolute control of the Treasury and install his stooge as chancellor.”
Julian Smith’s sacking – weeks after he brokered the deal which restored the power-sharing administration in Stormont – was greeted with shock in Northern Ireland.
The former minister said on Twitter that doing the job had been “the biggest privilege” and he was “extremely grateful” to have been given the chance to serve “this amazing part of our country”.
Ireland’s Prime Minister Leo Varadkar called Mr Smith “one of Britain’s finest politicians of our time”.