Kelly didn't last. He opened the diocese's first Catholic school and worked to build out missions, but he was soon entangled in jurisdictional fights with Baltimore. To end the feuding, Rome reassigned him to the Diocese of Waterford and Lismore in Ireland in early 1822, less than two years after his arrival. The chair sat empty for the next eighteen. By the time Pope Gregory XVI named Richard Whelan as the new bishop in 1840, Richmond had become a stronghold of the Know-Nothing Party - the nativist movement whose anti-Catholic bigotry sometimes spilled into open violence. Whelan inherited a diocese with six priests. He recruited more from the Societies for the Propagation of the Faith in Paris, Lyon, and Vienna, established a seminary, and called the first diocesan synod in 1855. Yellow fever and cholera swept through Virginia during his tenure. The work of building a Catholic infrastructure under these conditions was slow, dangerous, and largely done by immigrants - especially the French refugees who had founded the first Catholic parish in Norfolk, St. Patrick's, in 1791 after fleeing the French Revolution.

By the late nineteenth century the diocese had stabilized enough to build. Bishop John Keane founded schools and churches for Black Catholics in the 1880s despite institutional opposition. His successor, Augustine Van de Vyver, oversaw a transformative 1901 gift from philanthropist Thomas Fortune Ryan and his wife of nearly $500,000 - enough to buy land and construct a new Sacred Heart Cathedral, which was consecrated in 1906. Van de Vyver also opened an industrial college for African-American boys in Rock Castle, Virginia, while Saint Katharine Drexel and the Sisters of the Blessed Sacrament opened a parallel school for girls. But the racial reckoning was uneven. In 1902, Belgian Josephite priest Joseph Anciaux wrote to Rome condemning the U.S. Catholic hierarchy's acceptance of segregation and naming Van de Vyver personally for what Anciaux called timidity in the face of 'negro haters.' By October, Van de Vyver had forced Anciaux out of the diocese. The pattern - genuine institutional reform alongside genuine institutional cowardice - would recur.

Peter Ireton, who became bishop in 1945, ran one of the great expansions in American Catholic history: 42 new parishes, 24 new schools, and a Catholic population that grew from 37,000 to 147,000 during his thirteen years in the chair. His successor John Russell, who took over in 1958, was a civil rights champion who interviewed parents of prospective Catholic school students for signs of racism. Walter Sullivan, bishop from 1974 to 2003, established a joint Catholic-Episcopalian parish in Virginia Beach with separate altars and created a diocesan Commission on Sexual Minorities to reach out to LGBTQ+ Catholics. In 1974 Pope Paul VI redrew the boundaries, creating the Diocese of Arlington to serve Northern Virginia and returning the two Eastern Shore counties to Richmond. The current diocese spans central and southern Virginia, Hampton Roads, and the Eastern Shore - 226,674 Catholics in 138 parishes as of 2022, served by 135 priests.

The diocese has also lived through the same reckoning that has marked Catholic dioceses across the United States. In February 2019, Bishop Barry Knestout - appointed by Pope Francis in 2017 - released a list of 42 priests with credible and substantiated accusations of sexual abuse, covering allegations from the 1950s through 1993. Four more names were added in 2021. In October 2020, the diocese disclosed it had paid $6.3 million to settle 51 of 68 abuse claims. The list of names is public; the cases include men whose victims first reported them decades before the diocese acknowledged what they had done. Behind each name is a young person whose trust was used against them - children whose ability to believe in the institution charged with their spiritual care was shattered by someone wearing its collar. The diocese has also continued its mission: nine hospitals, seven facilities for the elderly, nine Catholic high schools and 24 elementary schools serving over 8,300 students. Both things are true at once.