The defeat means the 38-year-old American’s quest for a 24th Grand Slam singles title goes on.
Wang served for the match at 5-4 in the second set but Williams broke her for the first time and then won a tie-break to level at one set all.
The 27th seed recovered and sealed victory on her third match point.
“After the second set I was a little bit confused, but my mind always said I had to focus on the court, on every point and trust myself,” said Wang, who now faces Tunisian Ons Jabeur in the fourth round.
It is Williams’ earliest exit at the Australian Open since 2006, when she lost at this stage to Daniela Hantuchova.
Opportunities to equal Margaret Court’s record of Grand Slam singles titles are running out for Williams, who is in her 23rd year as a professional.
She has not won a Slam since the 2017 Australian Open, when she was eight weeks pregnant, but did secure the Auckland International earlier this month, which was her first tour title since Melbourne three years ago.
Williams had beaten 28-year-old Wang 6-1 6-0 in just 44 minutes at the US Open in September when the Chinese player won just 15 points.
However, the first few games of this second meeting suggested it would be much more competitive on Rod Laver Arena.
Wang grew in confidence after she saved three break points in the sixth game of the opening set. Her firm forehands began to trouble Williams, whose movement could not match her opponent’s.
Wang – who defeated current world number one Ashleigh Barty at the US Open before losing to Williams – broke for a 5-4 lead and then served out to seal the first set.
Williams was struggling to gain the upper hand as the unforced errors totted up – they would eventually reach 56 – and she was broken again when Wang converted the second of her three break points with a deep forehand winner.
The Chinese player’s form deserted her as she served for the match at 5-4, allowing Williams to break back with a superb forehand winner at the end of a 24-shot rally. The American saved two more break points at 5-5 before eventually levelling the match in a one-sided tie-break.
The final set went with serve until the 12th and final game. Wang had wasted two chances to seal victory on Williams’ serve before she was presented with another, which was taken when the American netted a backhand.