Columbia was barely a generation old when its first resident Catholic priest arrived in 1820. South Carolina was overwhelmingly Protestant - Baptist, Methodist, Presbyterian, Lutheran - and Catholics in the capital were few enough to count individually. Yet within four years they had built a church. The first St. Peter's, completed in 1824, was designed by Robert Mills, the South Carolina-born architect later famous for the Washington Monument and the U.S. Treasury Building. Mills was a Protestant who had trained under Jefferson; he designed Catholic churches anyway, because the city needed buildings and Mills built whatever the city needed. That first St. Peter's survived the Civil War, including Sherman's burning of Columbia in February 1865 that consumed much of the surrounding city. It stood until the early 1900s, when the growing parish demolished it to make way for something larger.

The current church was designed by Frank Pierce Milburn, a prolific Southern architect responsible for courthouses, train stations, and university buildings across the post-Reconstruction South. Father Thomas J. Hegarty led the fundraising. Construction began in 1906, finished in 1908, and the building was dedicated in January 1909. Milburn worked in the Gothic Revival vocabulary that high-church Catholics and Episcopalians both favored in the period: dark red brick, Bedford limestone trim, dull-glazed terracotta details, Buckingham slate roof. The spire rises 163 feet above grade and is topped with a cross. Inside, the cruciform plan stretches 131 feet long and 79 feet wide; the high nave is 51 feet from floor to ceiling, with windows tall enough to fill the space with colored light on a clear afternoon. In 1989 the structure was listed on the National Register of Historic Places.

The bell was cast in 1911 in Baltimore by the McShane Bell Foundry, weighs 3,500 pounds, and measures 55 inches across. McShane had been making church bells since 1856; their bronze rings out from Catholic parishes across the Eastern Seaboard, and St. Peter's is one of dozens of Southern churches whose Sunday morning carries a McShane voice. The church graveyard behind the building holds Irish names disproportionate to their share of nineteenth-century Columbia, including some of the indentured laborers who dug the nearby Columbia Canal in the 1820s and were buried at the parish after deaths from cholera, exhaustion, and accident. The Ancient Order of Hibernians dedicated a granite memorial to those workers in 2008. They lie within sight of the basilica's spire.