In 1858, he graduated from the University of Glasgow in Scotland and in 1864 was ordained a Presbyterian minister. In 1865, Cardozo became superintendant of an American Missionary Association school which he turned into Avery Normal Institute.
Cardozo was a delegate to the 1868 South Carolina constitutional convention and as chair of the education committee advocated for integrated public schools. Also that year, he was elected South Carolina Secretary of State, making him the first African American to hold a statewide office in the U.S. In that position, he reformed the South Carolina Land Commission which distributed land to formerly enslaved residents.
In 1872, he was elected state treasurer, a position he held until 1877. From 1884 to 1896, Cardozo served as principal of the Colored Preparatory High School in Washington D.C. which he made one of the leading schools for African Americans. Cardozo died July 22, 1903 and Cardozo Senior High School in D.C. is named in his honor.