Bayard Rustin was born into a Quaker family, and the pacifism he learned from the Society of Friends remained with him his entire life. After a comfortable childhood in West Chester, Pennsylvania, he studied at West Chester State College. Before graduating, he moved to Harlem during the 1930s and began studying at City College, while singing in local clubs with African American folk artists Josh White and Huddie Ledbetter. Attracted to the Young Communist League's stance on race issues, Rustin joined the group in 1936 and worked as an organizer until 1941 when he quit the party.

However, his resistance to the government continued throughout 1941 when Rustin was asked by A. Philip Randolph to help plan a 1941 march on Washington, D.C., to protest discrimination in defense industries. The march was called off when President Roosevelt made concessions. During World War II, Rustin traveled to California to help interned Japanese Americans protect their property. As a pacifist, Rustin spent two and a half years in prison for refusing to serve in the military.

Rustin's involvement in the Fellowship of Reconciliation (FOR), a radical pacifist movement, connected him to the establishment of the New York branch of the Congress of Racial Equality (CORE).

Throughout the 1940s and 1950s, he led weekend seminars on nonviolent action for both groups. Rustin helped organize the Montgomery Bus Boycott in 1955, and he was also involved in the formation of the Southern Christian Leadership Conference. In August of 1963 he served as the coordinator of the March on Washington, an event attended by 200,000 people. Rustin was arrested 23 times, he continued to believe that racial equality should be pursued through nonviolent means.