They were set 312 to win but a miserable batting display saw them bowled out for 177 in 40.5 overs to lose by 134 runs.
It means they have lost their opening two games and four consecutive World Cup matches for the first time.
South Africa posted 311-7, with Quinton de Kock hitting 109 for his second century of the tournament.
He joins compatriot AB de Villiers as the only batter to start a Men’s World Cup with consecutive hundreds.
There was a controversial decision to give Marcus Stoinis out after South Africa called for a review, which left Australia 70-6 in their chase.
Third umpire Richard Kettleborough deemed the right-hander had gloved Kagiso Rabada down the leg side, but replays suggested Stoinis’ hand was off the bat when the ball grazed that glove.
Kettleborough’s explanation was both Stoinis’ gloves were touching, but that did not appear to be the case on the pictures shown.
Australia’s next game is against Sri Lanka on Monday, while South Africa play the Netherlands on Tuesday.
Friday’s World Cup action sees New Zealand, who have won their first two games, face Bangladesh in Chennai.
Australia went into their innings with some momentum after Mitchell Starc finished with a double-wicket maiden, but it was soon lost.
It was a turgid start from Mitchell Marsh and David Warner, with both struggling for any fluency.
Marsh was the first to go, miscuing Marco Jansen to mid-off, before Warner flashed Lungi Ngidi, who bowled with excellent control up front, to point.
There was confusion when Steve Smith was given out lbw on review despite the front-on replay suggesting the ball may have gone on to miss leg stump.
The technology initially just showed the end result of the HawkEye process, leading to bewilderment from Smith and his partner Marnus Labuschagne, before it was later replayed in full to show its accuracy.
Josh Inglis and Glenn Maxwell soon followed Smith, before Stoinis’ controversial dismissal.
When asked about the decision after the game, Australia batter Marnus Labuschagne said: “It looked like there was clear daylight between the two gloves and the handle.
“But I have been into the third umpire’s room and the screens are pretty big and much clearer than me looking at a pixelated screen.
“I’m sure we’ll get clarity or will seek clarity because it’s a World Cup and we don’t want small decisions that can be avoided to affect the outcome of the games. With the position we were in, it’s hard to say it affected the outcome, but for the future.”
Labuschagne made 46, and showed some resistance in a stand of 69 with Starc, but the margin of defeat overtakes their 118-run loss against India in 1983.
It leaves Australia’s campaign in trouble. This is the second World Cup to follow this format and the final qualification spot in 2019 was taken by New Zealand on 11 points.
Bad weather played a part in that campaign, but is unlikely to this time, meaning sides may need 12 points to qualify for the semi-finals. Australia would need to win six of their remaining seven games to reach that mark.
This was another clinical display from South Africa, five days on from blistering their way to a World Cup record total in their opening-game win over Sri Lanka.
Temba Bavuma’s side came into the tournament under the radar but have been one of the standout sides in the opening two rounds of games.
Their innings was superbly led by De Kock, who cut and drove at will, as he raced to his 19th one-day international ton off 90 balls with a towering pull shot for six.
He was ably supported by Aiden Markram, who capitalised on being dropped on one by Pat Cummins in his follow-through to make 56.
That Cummins drop was one of five by Australia, including one by wicketkeeper Inglis, who came into the side for Alex Carey, as they were surprisingly subdued and flat.
Maxwell was the pick of the bowlers, taking 2-34 from 10 overs, and although Australia improved with the ball as the innings progressed – South Africa were 171-2 after 30 overs and added just 140 in the final 20 – two of those drops came in the penultimate over.
It is a sign of South Africa’s batting ability and power that it felt like they left plenty of runs out there, which is an ominous sign for the other sides in the tournament.
Australia captain Pat Cummins: “Quinton de Kock batted really well, we couldn’t get a breakthrough but from where they were, we were quite happy and thought 312 was chaseable.
“It looked like hard work to bat out there at night and it zipped around more than it did in the day, their bowlers bowled really well, but we were well off the mark.
South Africa captain Temba Bavuma: “If I was to be really greedy, we could have been stronger towards the end with the bat but we bowled really well.”
Player of the match, South Africa’s Quinton de Kock: “It was a great win for the boys.
“We are pretty pleased with ourselves but we’re only two games in and anything can happen. World Cups are strange and things can change quickly.”
