Mr Turnbull says it is an attack on Australian values.
But others have applauded the City of Yarra's decision as a mark of respect for Indigenous people, who say it is not a day for celebration.
Melbourne's inner-city Yarra Council has decided to drop references to Australia Day and lobby the Federal Government to change the day from January the 26th.
The council will also stop conducting Australia Day citizenship ceremonies.
The vote in the suburban council chamber has reverberated across the country and sparked heated debate in the nation's capital, led by Prime Minister Malcolm Turnbull.
"The decision recently of the Yarra Council is utterly out of step with Australian values. They are seeking to take a day which unites Australia and turn it into one which divides us. To change the date of Australia Day would be to turn our back on Australian values, on the great achievement of 24 million Australians here in the greatest, most successful multicultural society in the world."
But the local mayor, Amanda Stone, has hit back.
"That's a really disappointing comment from a prime minister who understands what reconciliation means and would surely know what the 26th of January represents to Aboriginal people."
Federal Opposition Leader Bill Shorten argues reconciliation is more about changing hearts and minds than it is about moving public holidays.
"I do not support changing the date of Australia Day. It is a day of ... it is a day of ... it is a day of citizenship ceremonies, of looking to the future, of celebrating all our cultures and faiths and traditions."
But Greens leader Richard Di Natale has backed the council.
"I think it's a brave decision and the right decision, and we, as a nation, need to have a conversation about whether Australia Day should be celebrated on a day that brings so much pain and hardship to our First Australians. And good on the council for showing some leadership."
The council's move to stop conducting citizenship ceremonies on that day has sharply divided views on the streets of suburban Richmond.
"I do feel it's really good for the Indigenous people. Then again, you want to celebrate Australia Day as a whole as well."
"Australia Day's Australia Day. It's always been on the 26th since I've known. So, that's it."
"Unfortunately, the date chosen is an inappropriate date now, given the knowledge we have via the Indigenous people."
One local Aboriginal woman, Ros Sultan, tried to express the pain many feel over a day that marks the British colonisation of the country.
"It's about taking out the party element, because there is no reason, for us, to celebrate that time of arrival."
The council says it tested the mood of the municipality through a series of interviews and street surveys with hundreds of residents.
But some in the streets suggest the consultation was not good enough.
"I would say sack the council and ... the people didn't get a say at all, did they. They never got a say at all."
"Most people are not happy."
