Charlton manager Chris Powell admitted the sending-off of goalkeeper Ben Hamer had a crucial effect on the 1-0 defeat to Leyton Orient.
Matthew Spring struck the winner with a deflected strike halfway through the first-half to inflict only the second defeat of the year on the Addicks.
Powell said: "I suppose that referees are under orders that if a keeper does handle outside the box, then it's a red card. But I think the circumstances of it - corner of the box and going away from goal and people covering back - I think its harsh and it changes the whole game.
"But I don't think they would have scored anyway. But you can't blame the referee because from his point of view, he's made the correct call.
"It was always going to be difficult to play for 85 minutes with 10 men against a very good side.
"In the end, we had a couple of chances to nick a point and that was the plan - to soak it up for as long as we could and then maybe go for it in the last 15 minutes which I think we did."