Colchester boss Joe Dunne pointed to the dismissal of Magnus Okuonghae as the key moment in the 4-1 JPT defeat at Dagenham.
Brian Wilson's first goal since November 2010 put the League One U's on course for their first win in the competition for nearly five years.
Former Colchester winger Medy Elito levelled before influential centre-back Okuonghae - nominated for August's League One player of the month award - was given his marching orders for a second bookable offence.
Brian Saah quickly put the Daggers ahead within the space of a minute before Abu Ogogo and Louis Dennis made sure League Two Dagenham triumphed.
Dunne said: "We were excellent in the first half and deserved the lead, it was great to see Brian (Wilson) get on the scoresheet.
"But the sending-off has changed the game and, from then on, it's always going to be difficult.
"I'm disappointed with the sending-off, it gave Dagenham the upper hand and they took full advantage.
"We were playing well up until that point, but when you go down to 10 men you're always leaving yourself with a lot to do.
"They scored a couple of quick goals and we didn't settle very well. I was disappointed to concede more goals but we have to bounce back from this defeat."