The hastily convened polls were taking place in the eastern regions of Luhansk and Donetsk as well as in Zaporizhzhya and Kherson in the south, with authorities the Luhansk city of Alchevsk even saying voting could take place in bomb shelters.
According to the pro-Russian authorities, two people died in a hotel near Kherson in a missile attack.
Meanwhile, in the city of Enerhodar in the Zaporizhzhya region, a polling station had to be moved to another location due to massive shelling from the Ukrainian side, Russian state news agency TASS reported.
Neither Ukraine nor the international community recognise the vote under Russia as the occupying power.
They are not considered to be legitimate referendums because they are being held without the permission of Ukrainian authorities, under martial law and not according to democratic principles.
The voting is scheduled to run until Tuesday, with the population asked to vote "yes" or "no" on whether the territories should join the Russian Federation.
A positive result for the Kremlin, seen as a foregone conclusion, could lead to Russian President Vladimir Putin annexing the regions as early as Friday.
Putin has previously said that Russia would then consider Ukrainian attacks on the regions as an aggression on Russian territory, and would use all means to defend it.
Ukrainian President Volodymyr Zelenskiy and the G7 have described the referendums as a "sham".
Ukraine's allies are preparing a new package of sanctions in response to the possible annexation.
In 2014, Russia already annexed the Black Sea peninsula of Crimea.
Serbia's government, despite its good relations with Russia, has meanwhile joined other European countries in saying they will not recognise the results of the referendums in the Russian-occupied territories in Ukraine.
Recognition of these referendums "would completely violate our national and state interests, the preservation of sovereignty and territorial integrity and the inviolability of borders," Serbia's Foreign Minister Nikola Selakovic said at a press conference in Belgrade, Serbian media reported.
Selakovic was referring to what Serbia sees as a parallel between Russia's attack on Ukraine and Serbia's own situation with the issue of Kosovan independence.
The former Serbian province of Kosovo declared its independence in 2008, a move still not recognised by Serbia, but by most other countries.
Zelenskiy said on Sunday his country would regain all the territory Russia had taken.
"We will definitely liberate our entire country - from Kherson to the Luhansk region, from Crimea to the Donetsk region," he said on the Telegram messaging app.
Putin on Wednesday ordered Russia's first military mobilisation since World War II.
The move triggered protests across Russia and sent many men of military age fleeing.
Two of Russia's most senior MPs on Sunday addressed a string of complaints about the mobilisation, ordering regional officials to get a handle on the situation and swiftly solve the "excesses" that have stoked public anger.
More than 2000 people have been detained across Russia for protesting against the draft, according to independent monitoring group OVD-Info.
In Russia, where all criticism of the conflict is banned, the demonstrations are among the first signs of discontent since the war began.
In the southern Russian region of Dagestan, police clashed on Sunday with people opposed to the mobilisation.
Public anger has appeared to be particularly strong in poor ethnic minority regions like Dagestan, a Muslim-majority region located on the shores of the Caspian Sea in the mountainous north Caucasus.
with Reuters
