The house was built in 1818 by John Brockenbrough, president of the Bank of Virginia. Designed by Robert Mills -- who would later design the Washington Monument -- it rose on K Street (later renamed Clay Street) in Richmond's affluent Shockoe Hill neighborhood, two blocks north of the Virginia State Capitol. Brockenbrough's neighbors read like a roster of early American power: Chief Justice John Marshall lived nearby, along with Aaron Burr's defense attorney John Wickham and future U.S. Senator Benjamin Watkins Leigh. Sold by the Brockenbrough family in 1844, the house passed through a succession of wealthy owners during the antebellum period, including U.S. Congressman James Seddon, who would later serve as Confederate Secretary of War. Just before the Civil War, Lewis Dabney Crenshaw purchased the house, added a third floor, and sold it to the City of Richmond. The city leased it to the Confederate government, and in the summer of 1861 it became the Executive Mansion of a new nation.

The family that moved in was young. Jefferson Davis arrived with his wife Varina, six-year-old Margaret, four-year-old Jefferson Jr., and two-year-old Joseph. Two more children -- William and Varina Anne, known as "Winnie" -- were born in the house in 1861 and 1864. Among the neighborhood playmates of the Davis children was a boy named George Smith Patton, whose father commanded the 22nd Virginia Infantry and whose grandson would command the U.S. Third Army in World War Two. Tragedy struck in the spring of 1864 when young Joseph Davis died after falling fifteen feet from the railing of the east portico. Meanwhile, the president himself was deteriorating. Davis suffered recurring malaria, facial neuralgia, cataracts in his left eye, unhealed wounds from the Mexican War including bone spurs in his heel, and chronic insomnia. He kept an office on the second floor, where his personal secretary Colonel Burton Harrison also lived -- not unlike the arrangement at the White House in Washington, which would not gain its West Wing until the Theodore Roosevelt administration.

Richmond was evacuated on April 2, 1865. The Davis family fled, and within twelve hours soldiers from Major General Godfrey Weitzel's XVIII Corps had seized the house intact. Lincoln, who was at nearby City Point -- now Hopewell, Virginia -- traveled up the James River to tour the captured city. Admiral David Dixon Porter accompanied him to the mansion, where they held meetings with local officials including Confederate Brigadier General Joseph Reid Anderson, owner of the Tredegar Iron Works. The house then became headquarters for Military District Number One during Reconstruction, serving as an occasional residence for the commanding officers of the Department of Virginia. Major Generals Edward O.C. Ord, Alfred Terry, Henry Halleck, and Edward R.S. Canby all served there. When Reconstruction ended in Virginia in October 1870, the city reclaimed the property and converted it into Richmond Central School, one of the first public schools in postwar Richmond.

In 1890, the city announced plans to demolish the old mansion to build a modern school. The Confederate Memorial Literary Society formed with one purpose: saving the house. The CMLS raised funds, acquired the deed, and opened the building as the Confederate Museum in 1896. For eight decades it housed collections and exhibits, earning the popular name "White House of the Confederacy" -- though some called it the "Gray House" after its actual exterior color. It was added to the National Register of Historic Places in 1960, designated a National Historic Landmark in 1966, and listed on the Virginia Landmark Register in 1969. When a purpose-built museum was completed next door in 1976, the collections moved out and a full-scale restoration began. Over twelve years, the exterior and first and second floor interiors were returned to their wartime appearance, complete with period furnishings and original pieces from the Davis era. The house reopened for public tours in June 1988.

Today the mansion sits on the campus of Virginia Commonwealth University, part of the American Civil War Museum that formed in 2014 when the Museum of the Confederacy merged with the American Civil War Center. The merger itself reflected a broader shift in how Richmond tells the story of the war -- not just the Confederate perspective that dominated for a century, but the fuller picture that includes the enslaved people who worked within these walls. Ellen Barnes McGinnis, an enslaved woman who served as personal maid to Varina Davis, is among those whose stories the museum now works to tell. The house remains open for public tours, its rooms restored to the years when a family governed a rebellion from a banker's mansion on a Richmond hilltop, while the nation they tried to leave fought its way back to reclaim them.