The Gunners, fielding a much-changed side from the one that drew with Chelsea in midweek and with manager Arsene Wenger watching from the stands because of a touchline ban, trailed 2-1 at halftime as Eric Lichaj scored twice.
Penalties by Ben Brereton and Kieran Dowell sealed victories
Per Mertesacker had equalised for Arsenal in the first half and Danny Welbeck pulled Arsenal back to 3-2 down with 12 minutes left but the Premier League side could have few complaints at a bitter exit.
It did little to detract from the party atmosphere alongside
the River Trent though as
There were boos from some of the Arsenal fans at the end after watching their side concede four goals to lower league opposition in the FA Cup for the first time since 1908.
It was also Wenger's first taste of a third round exit.
His decision to play such a weakened line-up – only Rob Holding and Ainsley Maitland-Niles kept their places from the starting side in the impressive 2-2 draw with Chelsea in midweek – quickly began to look like a poor decision.
Lichaj headed in from close range after 20 minutes, but the lead lasted only three minutes as Mertesacker finished well when the ball fell to him in the box.
American Lichaj produced a moment of magic to put
Teenager Brereton sent Ospina the wrong way from the penalty spot after Rob Holding had fouled Matty Cash in the 64th minute.
Arsenal, missing a host of match-winners including Alexis Sanchez, Mesut Ozil and Aaron Ramsey, were thrown a lifeline with 12 minutes to go when England striker Welbeck punished a mistake by Forest keeper Jordan Smith.
Former Arsenal player Armand Traore was then brought down by Mathieu Debuchy as he surged into the box in the 84th minute as Forest were awarded another penalty.
Dowell converted although Arsenal were furious as it appeared as though he kicked the ball twice after slipping when he was about to strike the spot-kick.
Nottingham Forest 4-2 Arsenal
Tottenham Hotspur 3-0 AFC Wimbledon
Shrewsbury Town 0-0 West Ham United
Newport County 2-1 Leeds United