Lord Ouseley claims football has come a long way since he helped to instigate Kick It Out, the organisation set up in the early 1990s with the purposes of eradicating racism and discrimination from all levels of football.
Lord Ouseley, who is currently chairman of Kick It Out, was made a life peer 13 years ago for his services to local government and community relations, having previously been knighted in 1997.
Speaking on Sky Sports News on the organisation's 20th anniversary, Ousley said going to football matches nowadays is a much more pleasant pastime than it used to be.
Asked why he decided to set up Kick It Out, he said: "It was prompted by the experience of going to matches, particularly the Premier League which had just started.
"The violence, abuse and harassment that went on was horrifying and I decided I wouldn't watch football any more.
"Then the opportunity came in 1993 when, at the time, I was the chair of the Commission for Racial Equality and had the authority and some degree of power to do something about those experiences.
"With a colleague I wrote to all 92 professional clubs, approached the FA, the new Premier League, the Football League and many others to see if they were interested in joining in a campaign to try to do something about this.
"Fortunately, we got some support and started from there and that's the journey we have been on for the last 20 years."
Lord Ouseley also claimed the experience of watching football in this country had improved greatly over the past two decades with Britain now considered a leading example of how to deal with discrimination in the game.
He added: "The reality is that, from 1993 to now, we have a totally different environment we operate in. The football world has moved on, it recognised the problems which existed back then and the government has passed legislation.
"We even get accolades for England fans when they go abroad, when before they weren't even welcome in countries. We have moved a long way and we now have an environment where it is a pleasure to go and watch football."