Pompey lost skipper David Connolly to a straight red card in the 38th minute following an off-the-ball incident with Sam Hird.
But it was not until the 87th minute that Armand Gnaunduillet broke the deadlock for Chesterfield, followed three minutes later by fellow substitute Tendayi Darikwa to maintain the Spireites' unbeaten start to the season.
In being held to a 1-1 draw at home to Rochdale, Oxford slipped from top spot as Matthew Lund's 51st-minute header cancelled out James Constable's first-half strike.
Burton, AFC Wimbledon and Exeter moved up to third, fourth and fifth on the back of wins over bottom-of-the-table Accrington, Fleetwood and York respectively.
The Brewers, however, had to wait until the eighth minute of stoppage time - as a result of a head injury to the visitors' Jimmy Phillips - for Adam Reed to poach a close-range strike at Livingstone Road.
Wimbledon's 2-0 home win was more comfortable as a penalty from Harry Pell - his third goal of the campaign - and a tap-in from Michael Smith came in the closing minutes of the first half.
At St James' Park, Exeter had to come from a goal down after York had taken the lead through Ryan Jarvis, with Alan Gow coming off the bench to score both goals, his first just 15 seconds after his 23rd-minute arrival.
As with Exeter, Morecambe and Wycombe were both 2-1 winners on home soil, beating Plymouth Argyle and Southend respectively.
Morecambe also had to come from behind after Reuben Reid had opened the scoring from the spot in the 27th minute for the Pilgrims.
Kevin Ellison swiftly equalised before Padraig Amond's maiden strike for the club, after recently signing on a free from Accrington, settled matters in the 84th minute.
Steven Craig, after a 13-year career in Scotland, scored his first goal in English football to help Wycombe to victory over Southend.
Dean Morgan headed a seventh-minute opener, but the visitors replied just seven minutes later as Freddy Eastwood scored his fourth goal in four games, only for Craig - a free transfer from Partick - to glance home a 32nd-minute winner.
Bristol Rovers, Bury and Mansfield also enjoyed home wins, the former through 19-year-old defender Tom Lockyer in the second minute to beat Northampton 1-0.
At Gigg Lane, Andy Procter turned into his net to give Cheltenham a fourth-minute lead, only for Bury to reply with three goals in nine minutes soon after.
Danny Mayor equalised with a stunning 30-yard strike before setting up Procter and Nathan Cameron, with substitute Jessy Reindorf completing a 4-1 win in the 76th minute.
Mansfield were convincing 3-0 winners at home to Dagenham & Redbridge via goals from Sam Clucas, Jamie McGuire and Matt Rhead.
Newport's Andy Hughes scored his first goal for three years to help his side to a 1-1 draw at Scunthorpe who had taken a second-minute lead through Sam Winnall.
In the only other game, Torquay were held to a goalless draw at home to second-bottom Hartlepool who are still to score a goal from their five league games this term.