The Liberal Party has not yet revealed whether it would support a Yes campaign in the referendum, due to be held in the next financial year.
Nationals leader David Littleproud confirmed his party would not back the proposal.
Prime Minister Anthony Albanese has maintained the referendum would be about establishing Indigenous recognition in the constitution.
A model for a consultative body is expected to be decided by the federal parliament following the referendum.
But Mr Dutton said the prime minister should stop treating the Australian public "like mugs" and provide details on the model before a referendum is held.
"There (are) lots of people in the debate at the moment who don't support the voice but possibly could support it when they get the detail and vice versa," he told Sydney radio 2GB on Thursday.
"Everybody is in favour of reconciliation ... but to be told this is a model you must accept and you're unreasonable if you ask for the detail, that's not how you conduct a public debate."
A meeting of Indigenous affairs ministers this week confirmed the principles of the voice, including that it would provide independent advice to the parliament and government, and be chosen by First Nations people "based on the wishes of local communities".
However, it would not have a program delivery function or have a veto power.
Mr Dutton said changing Australia's constitution was not something to be taken lightly.
He warned the Mr Albanese would have a lot to answer for if he presided over a referendum process that resulted in failure.
"If the prime minister wants to rush this through in the dead of night, I don't think that's going to happen," he said.
"The prime minister should have more trust in the Australian public to decide whether they think this (voice) is the answer and explain whether this is going to make a bad situation better or worse."
