Jefferson did not buy this land. His wife Martha Wayles Skelton inherited a half-interest in her father John Wayles's estate when Wayles died in 1773. The estate included 135 enslaved men, women, and children spread across tracts in Amherst, Cumberland, Charles City, Goochland, and Powhatan counties — and the property that would become Poplar Forest. The inheritance also included substantial debts that haunted Jefferson for the rest of his life. After Martha's death in 1782, Jefferson held Poplar Forest in sole ownership until 1790, when he gave 1,000 acres and six enslaved families to his daughter Martha and her husband Thomas Mann Randolph Jr. The remaining property stayed with Jefferson, who began designing the retreat villa in earnest around 1806, while he was still President, and pushed construction forward after his retirement in 1809.

Jefferson supervised construction from Washington beginning in 1806, while he was still President. Brick by brick, the octagonal house rose under the supervision of his enslaved master craftsmen and hired white tradesmen. The plan is precise: a central square dining room twenty feet on each side and twenty feet high — a perfect cube — surrounded on three sides by elongated octagonal rooms, with an entry hall on the fourth side dividing two smaller rooms. The skylight over the dining room is a Jeffersonian signature; he had used the same device at Monticello. The exterior shows the Palladian symmetry he loved, and the long flanking wings of brick service buildings, planted with paper mulberries to soften the geometry, complete a composition that is unmistakably his. The villa was substantially complete by 1826, the year Jefferson died.

Enslaved people lived and worked at Poplar Forest from 1766 until the abolition of slavery in 1865. Archaeology and the surviving documentary record have allowed scholars to name some of them. James Hubbard — known as Jame — was purchased by Jefferson when he was thirty and put in charge of the field laborers. He fathered six children with another enslaved woman named Cate, fostered several other children, and worked as a hogkeeper in his older years. His extended family at Poplar Forest included Nace, Hannah, Nancy, Joan, James, and Phill. After Jefferson's death in 1826, his son-in-law Thomas Mann Randolph Jr. divided and sold most of the property and sold many of the enslaved people on it to pay down Jefferson's debts — families split apart to satisfy ledgers. Recent excavations behind the house, in an area planted with paper mulberries, uncovered the outlines of a likely slave cabin occupied between 1840 and 1860. A pierced Spanish half-real recovered from the site suggests someone wore it as an adornment, perhaps an amulet or a charm — a small, precious thing kept on the body.

Poplar Forest changed hands several times in the nineteenth century. The Eppes family sold it to William Cobbs in 1828, and through Cobbs's son-in-law Edward Hutter it remained in the family for several generations. In 1984 the Corporation for Thomas Jefferson's Poplar Forest, a nonprofit, bought fifty acres and the surviving buildings to preserve them. The corporation has since reacquired roughly 617 acres of the original plantation. The villa first opened to visitors in 1986. Restoration of the main house — using nineteenth-century materials and techniques, including hand-burnt limestone for traditional lime mortar and plaster — was completed in April 2023. The property has been a National Historic Landmark since 1971. Today guided tours work along two parallel tracks: the architecture of Jefferson's retreat house, and the history of the enslaved community whose labor made it possible. Both stories are told.