Defender Antonio Rudiger scored the decisive spot-kick in a 4-2 shootout win after two Atletico players missed.
Marco Llorente's shot struck the bar after Julian Alvarez's score after he slipped was disallowed because the ball struck his right foot twice.
Madrid also beat Atletico in a shootout to win the 2016 final - part of a streak of eliminating their derby rivals in four straight years.
Madrid move on to the quarter-finals to face Arsenal, who also advanced on Wednesday night along with Aston Villa and Borussia Dortmund.
The quarter-finals lineup was completed with Madrid's win in a tense derby that ended 2-2 on aggregate score after extra time.
Atletico led 1-0 after 90 minutes and extra time in its Metropolitano Stadium to cancel out Madrid's 2-1 advantage from the first leg last week.
Two key incidents defined regulation time. Atletico Madrid scored within 30 seconds and Real Madrid missed a penalty in the 70th minute.
Atletico took the lead withtheir first attack when England midfielder Conor Gallagher pounced on the ball from close range when a cross by Rodrigo De Paul was deflected into the goalmouth.
Madrid star Vinícius Júnior blazed a penalty kick high over the Atletico goal when he could have equalised. He was substituted in extra time for his teenage fellow Brazilian, Endrick.
Arsenal rested some regulars in a 2-2 draw with PSV Eindhoven to run up a 9-3 aggregate score.
Aston Villa also had a stress-free evening at home to ensure England has two teams in the quarter-finals, one night after Premier League leaders Liverpool were beaten at Anfield by Paris Saint-Germain.
Villa won 3-0 against Club Brugge, who played with 10 men from the 17th, after a 3-1 win in Belgium last week. Brugge defender Kyriani Sabbe was sent off for pulling back Marcus Rashford when running clear on goal.
Substitute Marco Asensio, on loan at Villa from PSG, scored twice in the second half to ensure his temporary club will meet his parent club next.
Borussia Dortmund rallied with two second-half goals to win 2-1 at Lille and advance 3-2 on aggregate. The beaten finalists last season now face Barcelona.
The quarter-finals pairings are: Arsenal v Real Madrid, Paris Saint-Germain v Aston Villa, Barcelona v Borussia Dortmund and Bayern Munich v Inter Milan.
First-leg games are on April 8-9 and return games are one week later.
28°C