David Boswell was jailed for 18 years for the sex attack on the young girl in the 1990s.
But Pembroke Town Council have voted at a meeting to keep his picture on display in the town hall.
Instead of taking the photograph down, councillors voted to turn it to face the wall.
Current mayor Gareth Jones said it would be turned to face the wall “to account for his crimes”.
“It is a part of history and this shows the character of the person and their shame. We’re not trying to whitewash history,” he said.
The vote was held at a meeting of the town council, which is independent from the county council, on Thursday night.
One councillor resigned following the vote, saying he did not “want to remain a part of a town council that shows no understanding of the importance of this matter to the public”.
In a statement seen by BBC Wales, councillor Jon Harvey said he thought councillors would realise a “grave error of judgement had been made” by leaving the photograph up.
He said he believed the photograph would be taken down out of “empathy for and as a mark of respect to the survivors of his most heinous of crimes”.
But said he believed that many of the councillors present were “more concerned about the former mayors place in history”.
Boswell, from Pembroke Dock, was jailed in July 2018 after being found guilty of four counts of indecent assault and one count of rape after a trial at Swansea Crown Court.
He raped and indecently assaulted one girl and indecently assaulted another during the 1990s.
The former soldier and lorry driver was only town mayor for three months before stepping down after being charged by police.