With Prime Minister Anthony Albanese on Thursday releasing details of the referendum and the question Australians will vote on, Mr Dreyfus said a 'no' vote would lead to long periods of inaction on reform.
"It would be a very long time before we returned to any question of recognition. I think it would be a tremendous setback for relations with our First Peoples," he told ABC Radio on Friday.
"I'm focused on success because the consequences of failure would be dire."
A new section would be placed in the constitution, recognising Aboriginal and Torres Strait Islanders as the "First Peoples of Australia".
The proposed question being put to voters is:Â "A Proposed Law: to alter the constitution to recognise the First Peoples of Australia by establishing an Aboriginal and Torres Strait Islander voice. Do you approve this proposed alteration?"
Laws setting out the referendum will be introduced to parliament next week, with the vote to take place between October and December.
The Liberals have yet to outline their position on whether to support the voice but Opposition Leader Peter Dutton has urged the government to release legal advice presented by the solicitor-general.
Indigenous Australians Minister Linda Burney said there was nothing to fear from the referendum and urged support from the coalition.
"The Liberals are looking for excuses, and I think they've almost run out of excuses," she told ABC Radio.
"My real hope is that (Mr Dutton) provides bipartisan support in the way that happened during the week on the machinery bill ... if there is not bipartisan support, that would be unfortunate, but it will not stop the referendum going ahead."
Deputy Liberal leader Sussan Ley said the announcement of the referendum question was an "emotional day" but more detail on the proposal was needed.
"The prime minister said this would be a modest change, but then later he said it would change the country, so we'll keep asking for that detail and ask him questions on behalf of all Australians," she told Seven's Sunrise program.
"We'll do it in a respectful way because it is an incredibly important issue, you don't get a blank cheque to change the constitution."
Deputy Prime Minister Richard Marles said the voice would be a practical way to improve outcomes for Indigenous people.
"This isn't radical in the sense that when people who are affected by our policies are listened to, often you get the policies more right," he told Nine's Today program.
"It is about recognising our First Nations people in a practical way to hear their voice."
Prominent 'no' campaigner Warren Mundine said the prime minister's announcement of the question had not allayed his concerns on the proposal.
"It is like being asked 'would you like cake with your coffee?'We like to know what is in the cake before we say yes to it," he told ABC TV.
"It is not going to change one iota anything on the ground of Aboriginal people."
A key concern has been that decisions might be delayed or taken to court because representations by the voice must be considered by government decision-makers before they can validly make a decision.
However, constitutional expert Anne Twomey said the addition of words saying parliament can make laws with respect to "matters relating to the Aboriginal and Torres Strait Islander Voice, including its composition, functions, powers and procedures" was important.
"They are intended to permit parliament to legislate about the effect of the voice's representations, so it is a matter for parliament to decide whether the representations of the voice must be considered by decision-makers when making administrative decisions," she said.
14°C