Jan-Lennard Struff rallied to defeat Denis Shapovalov 4-6, 7-5, 7-6 (7-5) and clinch the victory in the second singles match of the tie in Malaga.
Earlier, Daniel Altmaier had given Germany the first point of the day with a 7-6 (7-5), 6-4 win over Gabriel Diallo.
Germany will next face the Netherlands on Friday for a spot in the final. The Dutch defeated Spain 2-1 on Tuesday in a tie that marked Rafael Nadal's retirement from professional tennis.
In the last two quarter-finals on Thursday, Australia will face the United States with defending champions Italy taking on Argentina.
"They have a strong team. It will be a tough task," Struff said of the Dutch.
Germany last made it to the semis in 2021 and lifted the last of their three Davis Cup trophies in 1993.
Canada's lone title came in 2022, when Struff also defeated Shapovalov in three sets in the quarter-final round.
In Wednesday's first singles rubber on Wednesday, Altmaier served 10 aces against nine from Diallo. The German had only one double fault against six of his opponent.
"I haven't played a Davis Cup match throughout the whole year, and then right away playing in such an important match which is to lead the tie, so I'm really happy (with) the way I could handle it mentally," Altmaier said.
Germany's Daniel Altmaier celebrates his singles victory over Canada's Gabriel Diallo in Malaga. (AP PHOTO)
Diallo said he wasn't "as loose" as he wanted.
"It was quite patchy," Diallo said.
"Some bits and pieces, moments where I was playing well, moments where I was a little bit more scrappy.
"But sometimes that's how it goes. Sometimes the moment gets a little bit to you. Today in some moments it got to me, and I didn't manage to find my rhythm and find my strides to get a win."
Both nations were without their top-ranked players, world No. 2 Alexander Zverev for Germany and Canada missing world No. 29 Felix Auger-Aliassime.
31°C