Westover's story does not begin with the Byrds. In 1665, Sir John Pawlett demised most of the property to Theodorick Bland for £170. Bland lived here until his death in 1671 and was buried in the chancel of the Westover Church he had helped build. His sons Theodorick and Richard inherited. The Byrd family, who would make Westover synonymous with Virginia gentry, arrived later. William Byrd II became famous for his witty diaries and for founding the city of Richmond. For decades, the elegant Georgian mansion was assumed to be his work, dating to around 1730. The National Historic Landmark designation in 1960 carried that date. Then dendrochronology, the dating of tree rings in surviving timbers, told a different story: the wood was felled in the 1750s. The house was likely built and first occupied not by William Byrd II but by his son, William Byrd III, who gambled the family fortune away and shot himself in 1777. The National Park Service has accepted the revised date.

Like every plantation along this stretch of the James, Westover ran on tobacco. Tobacco is a labor-intensive crop, requiring constant attention from seedbed to barn, and the Byrds depended on hundreds of enslaved Africans to do that work. The original grounds included slave quarters whose footprints have now mostly vanished. Beyond the fields, enslaved people worked as cooks, laundresses, coopers, carpenters, blacksmiths, and house servants. When tobacco wore out the soil and 19th-century markets shifted toward less labor-intensive crops, planters in the Upper South did not free the people they no longer needed in the fields. They sold them, breaking families apart in the domestic slave trade that fed the cotton economies of the Deep South. That trade ran without pause until the Civil War.

William Byrd III's widow died in 1814, and Westover passed out of the Byrd family. Three later owners shaped what remains today. Lawyer John Armistead Selden, connected by marriage to both the Selden and Lee families, kept meticulous diaries that are still mined by historians of 19th-century Virginia agriculture. Augustus Harrison Drewry, who as a Confederate officer commanded the gun batteries at Fort Darling during the Battle of Drewry's Bluff in May 1862, lived at Westover from after the war until his death in 1899. He rebuilt a wing that had burned when Union General Fitz John Porter used the house as Fifth Corps headquarters, connected the dependencies to the main house, and pushed the dining room out to the full depth of the building. Diplomat Richard Teller Crane II bought the property in the early 20th century, restored it, and the Crane family still owns it. Hurricane Isabel in 2003 took ten feet of riverbank, a 250-year-old road, and a 150-year-old poplar from the south lawn.

What makes Westover famous in the architecture textbooks is its balance: a perfectly proportioned Georgian central block with matching dependencies, brick walls laid in Flemish bond, a steeply pitched roof, twin chimneys at each end. The front door alone, set within an elaborate stone surround, is widely considered one of the finest of the colonial period. The style draws on the Governor's Palace in Williamsburg about thirty-five miles east. The grounds and the garden, with its tulip poplars and boxwood, open to the public daily from 9 a.m. to 6 p.m.; the house interior can be toured by appointment. It is still very much a private home, the Crane family in residence, a working piece of history rather than a museum frozen in amber.