James Cathcart Johnston, who completed Hayes a year after his father's death in 1816, was called a bachelor by Edenton society. Research published in 2013 by historian Mary Maillard established what the documents had always implied: Johnston never married, but he fathered four daughters with Edith "Edy" Wood, a woman he had manumitted from slavery, who lived in nearby Hertford. Two of those daughters died as children in 1836, aged eight and nine. The eldest, Mary Virginia Wood, moved north and married into the Forten family of Philadelphia - one of the wealthiest free Black families in the antebellum United States. Mary Virginia Wood Forten died of tuberculosis in Philadelphia in 1840, leaving behind a three-year-old daughter named Charlotte. That child would grow up to become Charlotte Forten Grimke - diarist, teacher of formerly enslaved children on the Sea Islands during the Civil War, poet, and one of the most important Black women writers of the nineteenth century. The line ran from an enslaved mother in Hertford County to a national voice for equal rights in two generations. Hayes was a piece of architecture. The family that made it matter was the one Edenton would not name.

Construction began in 1814 and finished in 1817. The labor that quarried the stone, fired the brick, dressed the cypress timber, raised the central block, and finished the curved hyphens connecting the dependencies was almost entirely enslaved. The plantation account books list expenditures for tools and overseer wages; the people who did the work appear, when they appear, by first name only, or as numbers in an inventory. To call Hayes William Nichols's accomplishment is to credit the architect for a drawing. The villa is what people who could not refuse made it. The Hayes Gatehouse, where James Johnston himself lived during construction, predates the main house and was built by the same hands. The Wood family - Edy Wood's family - had ties to the property that long outlasted Johnston's life, in ways the polite history of the house has never fully reckoned with.

James C. Johnston died at the end of the Civil War. He left Hayes to Edward Wood of nearby Greenfield Plantation, described in the will as his "friend and advisor." Edward Wood moved his family in. Wood, like Johnston before him, had children with his wife Caroline Gilliam Wood - and also fathered a biracial daughter, Mariah, with a woman he held in slavery. Mariah was raised inside the Wood household by her aunt Sarah Wood-Walker, a house servant. The pattern was not unique to Hayes; it was the pattern of plantation life across the antebellum and Reconstruction South, the private fact that the public architecture concealed. Wood descendants live at Hayes today, and the property still functions as a working farm growing cotton, tobacco, peanuts, and grains. North Carolina governors and state legislators have been entertained there. The cemetery just outside the house holds Penelope Barker, leader of the 1774 Edenton Tea Party; James Iredell, an Associate Justice of the U.S. Supreme Court; Samuel Johnston, governor and senator; and, until 1906, James Wilson, signer of the Declaration of Independence.

Climb to the belvedere on the roof of the central block and you see what the families who built and held Hayes saw: Edenton Bay shining south, Queen Anne's Creek curling north, fields running flat to a cypress horizon. The grounds are private; the National Historic Landmark designation does not open them to the public. From the air the geometry reads cleanly - a long central block, two pavilions linked by curving wings, the cemetery a small white rectangle just east of the house. The official story is in the architecture. The other story, the longer one, runs through Edy Wood and her daughters and Mariah and a Philadelphia tuberculosis ward and a teaching post on the Sea Islands and a long correspondence with Charlotte Grimke's husband Francis. None of it is visible from the air. All of it is the house.