Brown was born May 18, 1925 in Baton Rouge, Louisiana. He began his professional boxing career at the age of 17 but it was interrupted by World War II. During his almost two years in the United States Navy, Brown took part in seven Pacific invasions and won the All-Service Lightweight Boxing Championship.
After his discharge from the service in 1945, he resumed his boxing career and in 1956 won the World Lightweight Boxing Championship. Brown held the title for six years, making eleven successful defenses before losing in 1962. In 1961, Ring Magazine named him Fighter of the Year. Brown continued to fight until retiring in 1970 with a record of 105 wins, 46 losses, and 13 draws. Brown was inducted into the International Boxing Hall of Fame in 1996.