Juan Fernando Quintero opened the scoring for Colombia in the 31st minute with a direct free kick from the left edge of the penalty box.
The hosts equalised in the 57th minute with an own goal by Davinson Sánchez.
Uruguay then took the lead for the first time through Rodrigo Aguirre in the 60th minute, before the clash erupted in injury time.
André Gomez equalised for Colombia in the 96th minute from close range to seemingly secure his team a draw, until Manuel Ugarte gave the hosts the victory four minutes later with a shot from the edge of the box.
It was Ugarte's first international goal and it put Uruguay second in the standings on 19 points.
Bielsa has been under fire since October, when just-retired striker Luis Suárez criticised him for the way he treated players during the Copa America, saying the "friction" hurts.
Other Uruguay players did not defend their coach from the criticism, and Bielsa himself admitted his grasp over the team was shaken.
"I believe that wins like tonight's are healing," Bielsa said at a media conference.
"It was a thrilling match. It was evident that the players' willpower was needed for such a victory, though it looked impossible after an equalizer so late in the match. We needed this."
Colombia boss Nestor Lorenzo was naturally disappointed.
"It was an even, hard-fought match. We played well and we were able to dominate, but in some cases we had a few turnovers that cost us a lot," Lorenzo said.
"The way the goals came, you feel frustration and guilt, but then the team recovered and we got the equaliser and what happened at the end was frustrating, to be beaten like that."
Elsewhere, Peru and Chile played out a goalless draw in Lima, as both sides slipped further away from qualification spots.
Peru remain ninth with seven points, just one ahead of Chile.
Argentina, who lost 2-1 to Paraguay on Thursday, remain on top of the 10-team South American round-robin competition with 22 points from 11 matches.
Uruguay and Colombia are three points behind. Brazil are in fourth place in the standings with 17 points after their 1-1 draw at Venezuela.
Also on Thursday, Ecuador beat Bolivia 4-0. The Ecuadorians remain in fifth position with 16 points, while Bolivia are in eighth place with 12 points.
The top six teams will qualify automatically for the 2026 World Cup hosted by the United States, Mexico and Canada.