They advanced to the quarter-finals with a 4-2 penalty kicks shootout win over the Democratic Republic of Congo.
This was after a 2-2 extra-time stalemate in their last 16 tie.
The tournament debutants had upset Nigeria to finish top of their group.
They added another major scalp by edging the two-time champions to set up a meeting with either Tunisia or Ghana in the last eight.
Ibrahim Amada had opened the scoring for Madagascar in the ninth minute but Cedric Bakambu equalised in the 21st minute.
Madagascar restored their lead in the 77th minute through veteran captain Faneva Ima Andriatsima but Chancel Mbemba made it 2-2 in the final seconds to take the game into extra time.