Patterson was born enslaved September 12, 1840 in Raleigh, North Carolina. Her family gained their freedom in 1852 and moved to Oberlin, Ohio in 1856. Patterson enrolled in Oberlin College in 1857 and graduated with a Bachelor of Arts and highest honors in 1861.
After graduation, Patterson taught at various schools, including the Institute for Colored Youth in Philadelphia and the Preparatory High School for Colored Youth (later renamed Dunbar High School) in Washington, D. C. Patterson served as the latter school’s first Black principal from 1871 to 1872 and was reappointed from 1873 to 1884. During her administration, the school grew from 50 to 172 students. Patterson continued to teach at the school until her death.