Between 1747 and 1759, William Randolph III assembled the Wilton estate out of more than a dozen contiguous tracts of land on the north bank of the James River. He inherited his father's Fighting Creek property in 1742, then bought over a thousand acres from William Finney Jr. in 1747, plus smaller tracts from Richard Randolph, William Bayley, and Arthur Giles. By 1753 he had completed his Georgian manor house and named it Wilton. The two-story brick structure became one of the most significant James River plantation mansions — designed in the formal Georgian style favored by Tidewater Virginia's wealthiest planters. The house faced the river. Tobacco was the cash crop, with wheat increasingly added as soil exhaustion set in. The Randolph family belonged to Virginia's small interconnected aristocracy — William III's grandson on another line was Thomas Jefferson, and Randolph names recur throughout the founding generation of the United States. Wilton's wealth — the land, the bricks, the imported furnishings — was extracted from human beings the family claimed as property.

The Randolphs enslaved more than a hundred African American men, women, and children at Wilton across multiple generations. Enslaved laborers fired the kilns that produced the bricks. They cut the timber. They built the house. They cleared the fields, planted and tended tobacco, harvested it, packed it into hogsheads, and rolled the hogsheads down to the riverside wharf where it was shipped to Britain. They cooked the meals in the detached kitchen. They cared for Randolph children, hauled water, washed clothes, tended livestock. By the early 1800s, Wilton's enslaved community was the largest in Henrico County — meaning the Randolphs profited from the unpaid labor of a larger group of people than any of their neighbors. Their names are largely lost. The museum has begun, in the twenty-first century, to research and recover what can be known of these individuals. The mansion was their forced workplace; its preserved elegance is built directly on their suffering. Visitors today are asked to hold both facts at once.

Like many Tidewater tobacco plantations, Wilton ran out of soil and ran out of money. William Randolph III died in 1762 at age 52, leaving the estate to his 23-year-old son Peyton. Peyton died at 46 in 1784, leaving Wilton to his five-year-old son William Randolph IV. William IV died at 26 in 1815, leaving the house to his five-year-old son Robert. Robert died at 29 in 1839, leaving a heavily indebted Wilton to his four-year-old daughter Catherine. Each generation inherited less than the one before. In 1833 the writer Catharine Sedgwick visited and described "broken down fences, a falling piazza, defaced paint, banisters tied up with ropes" — a forlorn ruin. The total assessed value of the estate dropped from $74,664 in 1832 to $45,066 in 1850. Catherine Randolph filed to sell Wilton in 1859. The suit was not settled until 1875. The auction was held on July 27, 1859: William C. Knight paid $49,517 for 1,237.93 acres including the house. Catherine was the last Randolph owner. The plantation changed hands four more times after the Civil War.

By 1933, Wilton was in foreclosure with the Bank of Commerce and Trust. Across the river, commercial development was creeping toward the property. The Pocahontas Parkway would eventually be built just to its west. The National Society of the Colonial Dames of America in the Commonwealth of Virginia stepped in. They bought the mansion, then made the radical decision to move it: dismantling the brick walls, the interior paneling, the wooden floors, numbering each piece, hauling it fifteen miles upriver, and rebuilding it on a new site overlooking the same James River near Richmond's western edge. The reconstruction was complete by 1935. Historic American Buildings Survey photographers documented both the original site and the relocation. The house has been open to the public as a museum since 1952. The original site east of the Pocahontas Parkway remains, almost ironically, devoted to agriculture in the twenty-first century — the World's End farm still farming.

The collection at Wilton today numbers about 1,400 artifacts — 17th-, 18th-, and 19th-century silver, ceramics, textiles, paintings, documents, and furniture reflecting Tidewater planter life. The house was listed on the National Register of Historic Places in 1976. Wilton remains the only public 18th-century plantation house within the city of Richmond. The museum's current programming includes the documented history of enslavement at the site — not as footnote but as central content, since the labor of those individuals built and sustained the entire estate. The site offers tours, educational programs, and ongoing research into the enslaved families whose names appear in scattered Randolph documents. From the air, the relocated house presents a small, austere Georgian brick rectangle in a wooded enclosure — a deliberate piece of preserved colonial geometry sitting peacefully above a river that once carried the tobacco that paid for it.