Leverkusen were the fourth top-tier team eliminated by Bielefeld on their way to the decider, having previously ousted Bundesliga sides Union Berlin, Freiburg and Werder Bremen.
Bielefeld are only the fourth team from the third tier to reach the Cup final, following Hertha Berlin's reserves in 2003, Energie Cottbus in 1997 and Union Berlin in 2001. All three then lost the final.
"I'm just proud of this team. Today nobody will sleep in the city," Bielefeld coach Mitch Kniat told broadcasters ARD.
Leverkusen took the lead through Jonathan Tah in the 17th minute, but the hosts were level just three minutes later through Marius Woerl.
Leverkusen goalkeeper Lukas Hradecky made a big save to deny Noah Sarenren Bazee one-on-one in the 26th but the underdogs found their winner shortly before the break as Maximilian Grosser finished off Louis Oppie's free kick.
The title holders couldn't react in the second half, with Bielefeld controlling the game. Their best chance was with Patrick Schick in the 81st, but his header hit the post - which was kissed by Bielefeld goalkeeper Jonas Kersken.
"This was by far our worst game of the season. We did a lot of things wrong," Leverkusen midfielder Robert Andrich said.Â
He committed the foul which resulted in the free kick for Bielefeld and, eventually, the winner.
Bielefeld will now face the winner of Wednesday's duel between VfB Stuttgart and RB Leipzig.
"We didn't need luck at all because we pressed the entire time," Kniat added.
Leverkusen were without playmaker Florian Wirtz, who is recovering from injury. They will now fully focus on their Bundesliga title defence. In the league standings, they are six points behind leaders Bayern Munich with seven games left.
19°C