Elder dropped out of high school and worked as a caddy where he developed his game by watching his clients. In 1959, he was drafted into the United States Army where he served until 1961. After his discharge, he joined the United Golf Association Tour for black players where he won 18 of 22 tournaments.
In 1968, Elder gained his Professional Golf Association tour card and in 1974 he won his first PGA tournament. That came at the Monsanto Open where Elder and other black players had to change their clothes in the parking lot because the club members would not allow non-whites into the clubhouse. The win gained him entry into the 1975 Masters Tournament. Leading up to that tournament, Elder received a substantial amount of hate mail and threats.
In 1979, Elder became the first African American to qualify to play in the Ryder Cup and in 1984 he joined the Senior PGA Tour. Over his career, Elder has won four PGA tournaments and eight senior tournaments. In 1974, Elder established the Lee Elder Scholarship Fund to offer financial aid to low-income men and women seeking college assistance. He has actively promoted Summer Youth Golf Development programs and raised money for UNCF.