The house had been built in 1798 and 1799 by Samuel Jackson, a Philadelphia merchant who held it briefly before the mortgage passed to the federal government. The two-story brick house had a central passage, four rooms per floor, semicircular back rooms, and two brick wings connected to the main block by covered walkways. In 1804 it was bought at auction by Joseph Nourse, who served as the first Register of the United States Treasury under Washington, Adams, Jefferson, Madison, Monroe, Adams, and Jackson, a tenure of forty years that remains one of the longest in the history of the federal civil service. Nourse and his family lived in the house from 1804 to 1813. In 1813 they sold it to Charles Carroll of Bellevue, a cousin of the Declaration of Independence signer Charles Carroll of Carrollton. The following August, the British came.

Between 1785 and 1840, census records show that ten to thirteen enslaved or indentured people worked at Dumbarton House at various times. The names of some are known. An enslaved Black woman named Dinah cooked. Another named Fran helped her in the kitchen. An enslaved Black man named Bacchus worked the grounds. A cobbler named Juba made and repaired shoes. A man known as Frank served as a manservant. A woman named Jane, whose race is not recorded, also worked there. Betsy, Polly, Will, and Black Peter appear in passing in the household papers. The Federal-period elegance of the rooms upstairs, the silver, the tea services, the polished mahogany, was built and maintained by their work. The museum that operates here today has begun the work of telling their part of the story, alongside the Nourses and the Carrolls.

In 1915, the District of Columbia decided to build a new bridge across Rock Creek to connect the two halves of Q Street, which had been separated by the gorge for a century. The new alignment ran directly through Dumbarton House. The owners faced a choice: tear it down or move it out of the way. They chose to move it. The center of the brick house had a full cellar that gave the wreckers something solid to jack against. Workers hoisted the central block onto rollers, attached an animal (the records do not specify whether horse or mule) to a windlass, and slowly hauled the entire building roughly fifty feet north onto a new foundation. The two brick wings, which had no cellars, were dismantled and rebuilt in place. The house has stood at its current location ever since.

The National Society of the Colonial Dames of America bought the house in 1928 for its national headquarters and renamed it Dumbarton House, drawing the name from a Scottish landmark called the Rock of Dumbarton, which had also given its name to the surrounding eighteenth-century estate and to nearby Dumbarton Oaks. The Society opened the two principal floors of the house as a museum on the two-hundredth anniversary of George Washington's birth in 1932. The rooms were filled with donated furniture, silver, ceramics, and paintings from the Federal period. A 2006 study, the Historic Furnishings Plan, used the Nourse family's correspondence and Joseph Nourse's surviving account books to reconstruct what the interior actually looked like between 1804 and 1813. The museum is still implementing that plan, slowly swapping in pieces that more accurately reflect what was here when Dolley Madison arrived from the burning city.

The acre of grounds includes the East Garden, designed by Virginia landscape architect M. Meade Palmer in the 1950s on land donated by the Belin family of neighboring Evermay. In 2010 Georgetown landscape architect Guy Williams installed an herb garden inside the East Garden, replanted from a list of plants Nourse himself recorded growing on the property. Both guided and self-guided tours are offered. The ballroom opens onto a walled terrace and is rented out for weddings. The house, listed on the National Register of Historic Places, sits at the corner of Q Street and 27th Street NW, a few blocks east of Wisconsin Avenue and just south of Dumbarton Oaks. Most visitors to Georgetown walk right past it on the way to somewhere else. They miss one of the few houses in Washington that remembers what August 24, 1814 looked like through a friend's window.