Marcel Garvey believes Castres' slow start cost them dearly as they lost 41-17 to a strong Ulster side.