Williams was born August 2, 1923 in Brunswick, Georgia. He began boxing professionally in 1940 and won the World Lightweight Boxing Championship in 1945. He successfully defended the title five times before losing it in 1951.
He was named Ring Magazine Fighter of the Year in 1948. Williams retired from boxing in 1956 with a record of 125 wins, 24 losses, and 5 draws. He was inducted into the International Boxing Hall of Fame in 1990.