Nelson was born June 4, 1932 in St. Louis, Missouri. He learned to play the piano when he was six and started to play the saxophone when he was eleven. After military service in the Marines, Nelson studied music composition and theory at Lincoln University and graduated in 1958.
Nelson began leading his own groups in 1959 and after recording six albums his big breakthrough came with the 1961 album “The Blues and the Abstract Truth” which included “Stolen Moments,” now considered a jazz standard. Between 1966 and his death, Nelson led several all-star bands.
He also composed music for television, films, and produced and arranged for stars such as Nancy Wilson, James Brown, The Temptations, and Diana Ross.