Goals by Lars Stindl and Leon Goretzka, either side of a penalty by Julian Draxler, saw Germany home to leave them second in Group B behind Chile, who they face in Kazan on Thursday.
Celtic midfielder Tommy Rogic had made it 1-1 in the first half and Tomi Juric added Australia's second after the break from a goal that was allowed to stand despite a suggestion of handball leading to a review from the video assistant referee.
The Socceroos then had a string of late chances to equalise, but Germany held out.
Australia next face Cameroon, who lost 2-0 to Chile in their opening game, in Saint Petersburg on Thursday with both teams needing a win in their bid to reach the semi-finals.
Much had been made before kick-off of Germany bringing a young squad and their starting line-up had just 138 international appearances between them.
The arrival of the world champions in Sochi failed to excite local fans and only 28 605 of the Fisht Stadium's available 47 700 seats were filled.
Those who did turn out saw Germany waste no time in taking the lead with just five minutes gone.
After Goretzka went close, winger Julian Brandt cut the ball back for Stindl, who slotted home his first goal for his country on only his third international appearance.
With Bayer Leverkusen's Brandt causing mayhem down the right flank, striker Sandro Wagner had two clear chances, a diving header and then a shot which both went wide.
Brandt's fiercely-hit shot from outside the area produced a diving save from Socceroos goalkeeper Maty Ryan, and Australia barely escaped their own half in the opening 30 minutes, but they eventually started to build some momentum.
Defender Trent Sainsbury wasted a golden chance when he got in behind the defence, but he headed Aaron Mooy's free-kick wide with the goal at his mercy.
Their breakthrough came when a Rogic shot was blocked, but the Celtic midfielder slammed home the rebound to pull the Socceroos level on 41 minutes.
It was not to last.
When Queens Park Rangers midfielder Massimo Luongo scythed down Goretzka in the area three minutes later, Germany captain Draxler drilled home the resulting penalty to make it 2-1 at the break.
And Germany extended their lead with a copybook move on 48 minutes.
Brandt combined with Joshua Kimmich, who floated a cross over the defence for Schalke's Goretzka, who took it in his stride and rifled home.
Australia made it 3-2 when Juric tapped home from close range after Germany goalkeeper Bernd Leno failed to hold a deflected Rogic shot on 56 minutes.
With the Germans screaming for a handball, the decision to give the goal was referred to the video assistant referee.
Replays showed the ball striking Juric's arm before Leno saved the initial shot, but the goal was allowed to stand.
Australia kept up the pressure after that as substitutes James Troisi and Robbie Kruse, as well as Juric, all tested Leno in the final quarter.
At the other end, German substitute Timo Werner hit the post after holding off two defenders and beating Ryan.
Veteran midfielder Tim Cahill, now 37, came on in the dying stages for his 98th international appearance, but the Socceroos' all-time top-scorer failed to add to his tally of 48 goals.