Born in Marshall, Texas, Farmer was an educator, administrator, and one of the founders of the Congress of Racial Equality-CORE.

Raised in an environment that valued education and religious faith, James Farmer was an outstanding student. After skipping several grades in elementary school, he entered Wiley College in Marshall, Texas (where his father, one of the few African American Ph.D.s in the South, had taught), at the age of 14. Graduating in 1938, Farmer went on to Howard University's School of Religion. He graduated from Howard in 1941. Farmer opposed war in general, and more specifically objected to serving in the segregated armed forces. When the U.S. entered World War II later that year, he applied for conscientious objector status but found he was deferred from the draft because he had a divinity degree.

Rather than become an ordained Methodist minister, Farmer, who told his father he would rather fight that church's policy of segregated congregations, chose instead to go to work for the Fellowship of Reconciliation (FOR). Farmer was FOR's secretary for race relations, helping the Quaker, pacifist organization craft its responses to such social ills as war, violence, bigotry, and poverty. It was a job that left Farmer, who was then living in Chicago, Illinois, enough time to begin forming his own approach to these issues — one based less on FOR's religious pacifism than on the principle of nonviolent resistance.

By the late 1960s, Farmer, seeing CORE drift away from its Gandhian roots, left the organization he had helped found and had led for more than 20 years. Always an active writer and speaker, he continued to lecture publicly on civil rights and eventually took a teaching position at Lincoln University in Pennsylvania. In 1968 Farmer ran for U.S. Congress on the Republican Party ticket and was defeated by Shirley Chisholm, an African American running as a Democrat. Shortly thereafter, he went to work for President Richard Nixon's administration as Assistant Secretary of Health, Education, and Welfare.

In the years since retiring from politics (1971), Farmer has served on many boards, including the Coalition of American Public Employees. He has also continued to teach and lecture widely. In 1985 he published his autobiography, titled Lay Bare the Heart, and in 1998 President Bill Clinton awarded him the Congressional Medal of Freedom.