Murphy was born January 17, 1889 in Baltimore, Maryland. He earned his Bachelor of Arts degree from Howard University in 1911 and his Master of Arts degree from Harvard University in 1913.
Murphy served as chairman of the German Department at Howard from 1913 to 1918. In 1922, he assumed control of the Baltimore Afro-American newspaper and over the next four decades solidified it as a major African American newspaper, increasing circulation from 14,000 to more than 200,000 with over 200 employees. Under his leadership, the newspaper was deeply involved in the organization of the 1965 March on Washington for Jobs and Freedom.
Murphy was awarded the NAACP Spingarn Medal in 1955. The Fine Arts Center and the Carl J. Murphy Scholarship Fund at Morgan State University are named in his honor.