Samuel Woodruff, a merchant and onetime mayor of Elizabethtown, built the house around 1750 on the north side of what is now East Jersey Street. It was a substantial property: a two-story wood-frame structure with a shingled exterior, brick end chimneys, and a symmetrical five-bay facade crowned by a tripartite Palladian window above the center entrance. In its original configuration, two wings brought the number of rooms to eighteen. The main block followed the classic four-over-four center hall plan that marked a gentleman's residence in colonial New Jersey. When Woodruff died in 1768, the property passed to his son, but financial difficulties eventually forced a public auction. In 1772, Elias Boudinot, a lawyer with rising political ambitions, purchased the estate.

Boudinot was a man of considerable influence. A lawyer, politician, and diplomat, he would go on to serve as president of the Continental Congress from 1782 to 1783 - making him, in a technical sense, the head of the American government before the Constitution created the presidency. But before all that, in 1772, he and his wife Hannah Stockton Boudinot opened their home to a teenage immigrant from the Caribbean named Alexander Hamilton, who was attending the nearby Elizabethtown Academy. Hamilton's stay was brief, but the connection between the ambitious young scholar and his powerful patron would endure. When the Revolution erupted, Hannah moved to the family property in Basking Ridge for safety, and Elias took up his duties as Commissary General for the Continental armies. How the house was used during the war remains unknown.

The Boudinots returned after the war and lived at Boxwood Hall until 1795, when Elias sold it to Jonathan Dayton. Dayton had signed the Constitution at the age of 26 - the youngest delegate to do so - and was then serving as Speaker of the U.S. House of Representatives. He moved in with his wife Susan and their children and promptly set about redecorating, installing an Adamesque mantelpiece in the east parlor and a Federal mantelpiece in the west. The house reflected his status and ambition. But Dayton's association with Aaron Burr, and by extension the Burr Conspiracy of 1807 - a murky scheme that may have involved separating western territories from the United States - brought financial and political ruin. Dayton was forced to sell the mansion to his son-in-law, Dr. Oliver Hetfield Spencer, while retaining the right to live there for the rest of his life. He died at Boxwood Hall in 1824.

The house passed through many hands over the next century, serving various purposes as Elizabeth grew from a colonial town into an industrial city. By the 1930s, the building faced demolition. A nonprofit organization formed to save it, and the property was eventually turned over to the state of New Jersey. Restoration work in the 1940s returned the interior to something approaching its eighteenth-century appearance. Today Boxwood Hall State Historic Site operates as a museum at 1073 East Jersey Street, designated a National Historic Landmark for its association with Boudinot. The Palladian window still catches the light above the front door. The rooms where a future treasury secretary studied as a teenager, where a president of Congress entertained, and where a disgraced Speaker lived out his final years stand quiet and preserved - a single address that touched three very different American lives.