The White House sat on the Pamunkey River in New Kent County, on land first patented by Colonel John Lightfoot III, who built the original mansion just before 1700 while serving as Counselor of State of colonial Virginia. The Lightfoots eventually sold the property and other parcels to John Custis, father of Daniel Parke Custis. Three successive houses would rise on the same pre-1700 foundation. Daniel Parke Custis owned it when he married Martha Dandridge. After Daniel died young, leaving Martha a wealthy widow at twenty-six, she met a tall Virginia colonel on his way to attend the House of Burgesses in Williamsburg. He stopped at her house when he fell suddenly ill. He stopped again on his return. The next January they were married in one of the rooms of the White House, and Martha moved with her two children to George's home at Mount Vernon.

George Washington had no biological children. Martha had four by her first marriage; only two survived to adulthood, and only one, John Parke Custis, survived his stepfather. John's son, George Washington Parke Custis, was raised in part by the Washingtons at Mount Vernon and went on to build Arlington House in the District of Columbia in 1802 as a memorial to the man he had called grandfather. George Washington Parke Custis's daughter Mary Anna Randolph Custis was born at Arlington in 1807. In 1831, she married a young West Point graduate named Robert E. Lee. They had seven children. By the inheritance terms of George Washington Parke Custis's will, the White House plantation passed to one of those children: William Henry Fitzhugh "Rooney" Lee, who had studied at Harvard and entered the U.S. Army. Robert and Mary Lee themselves never owned the White House, and never owned Arlington either, though both would shape their lives.

Virginia seceded in April 1861. Robert E. Lee, then a colonel recently returned from command of the Department of Texas, was offered command of the entire Union Army by Abraham Lincoln. He spent a night thinking it over at Arlington House, then resigned his commission and went home to fight for Virginia. All three of his sons followed him into Confederate service. His wife Mary Anna Randolph Custis Lee had been crippled for years by rheumatoid arthritis; by 1861 she was using a wheelchair. She and her daughters left Arlington as Union forces closed in and made their way down to the White House on the Pamunkey, the property their son Rooney now owned. In the spring of 1862, George McClellan's Union army landed on the peninsula and pushed inland. White House Landing, with its crossing on the Richmond and York River Railroad, was an ideal supply base. The Union army took it. The historic mansion where Washington had married Martha was destroyed by Union troops in 1862, leaving only ruined brick walls in a clearing along the river.

The story of the White House gathers up several of the most consequential threads in American history: the founding generation, the slow building of the Lee family inheritance from Washington's marriage, and the Civil War that ended the world both families had been born into. The mansion has not been rebuilt. The location can still be found, by GPS or by Google Maps, along the Pamunkey north of where it meets the Mattaponi to form the York River. The Richmond and York River Railroad, completed in 1861, crossed here on its way between Richmond and West Point. From the air, the foundations and the cleared site of the old mansion are visible as a slight clearing along the riverbank. The country around it is still rural, still hauntingly quiet, the kind of place where the absence of a building tells you what was lost more clearly than any monument could.