Opposition Leader Peter Dutton, who opposes the voice being enshrined in the constitution, has been calling on Mr Albanese to release the advice provided by the solicitor-general to cabinet.
Mr Albanese said while the full cabinet documentation would not be released, the solicitor-general's view would be made known through Attorney-General Mark Dreyfus.
"The solicitor-general's view will be made very clear by himself through the attorney-general, which is the appropriate forum for it to take," Mr Albanese told the ABC's 7.30 program on Monday night.
He said it was not appropriate to release cabinet documents, in line with the process followed by the previous coalition government and past Labor governments.
Mr Albanese said Mr Dutton was raising questions and doubts "by not having any substance to his opposition" to the voice.
The Liberal Party has called for a legislated, but not constitutionally enshrined, voice process involving representative local and regional bodies.
Former Nationals leader Barnaby Joyce said he had "no problems" with constitutional recognition, if he was able to see the solicitor-general's advice first.
"We agree with the local and regional bodies ... I just don't believe we should be inserting a racial clause in our constitution in 2023," he told ABC radio.
Mr Joyce said the voice to parliament would be a "massive change to how democracy works".
"We're now dealing with a consultative power by a selected group, not an elected group … and that's a massive move away from the democratic process," he said.
Opposition frontbencher Simon Birmingham said the prime minister should consider the offer put forward by Mr Dutton to secure bipartisan support for constitutional recognition of Indigenous people.
"That would be a national unifying moment and (the prime minister) should seize that opportunity and consider that," he told Sky News.
Senator Birmingham said Mr Albanese had been "quick to laud" Julian Leeser for quitting his frontbench position to campaign in support of the voice, but said the prime minister should consider his suggestions in relation to the wording of the proposed change.
Mr Albanese said he had made it clear there would be local and regional bodies, which may differ across states and territories.
He pointed to South Australia's decision to set up a state voice.
"It's quite clear that the national voice would work with state voices, were they to be established as well," Mr Albanese said.
"So clearly we want to hear from local communities, working the way up. But you need a national voice as well."
He said the voice would also ensure rural and remote representatives and gender balance, as well as ensuring regions such as the Torres Strait were properly represented.
The final model of the voice would be legislated once and if the referendum on the voice passed later in the year.
In Alice Springs last week, Mr Dutton said he had brought to the prime minister's attention in October 2022 the issue of children who had been sexually abused being placed back in the hands of perpetrators.
Asked about this, Mr Albanese told the ABC he was not aware of the allegations.
"It's possible that there may well have been a letter somewhere ... but he has not raised any specific issue about any claim, about any individual circumstance with me," he said.
"If he did, I would say to him that he should report that to the police."