The land was part of an early grant made by Queen Anne in 1702 to Colonel Ninian Beall, a Scottish soldier who named it the Rock of Dumbarton after the volcanic plug on the River Clyde. The first house on the property was built around 1801 by William Hammond Dorsey, and the central block of that house still anchors the present building. Edward Magruder Linthicum enlarged it in the 1840s and 1850s and called the place The Oaks. Vice President John C. Calhoun lived here from 1822 to 1829. Henry F. Blount bought it in 1891. Robert Woods Bliss, a career diplomat with significant inherited wealth, and his wife Mildred Barnes Bliss, an heiress and serious patron of the arts, acquired the property in 1920 and renamed it Dumbarton Oaks in 1933, combining the two historic names. They eventually assembled fifty-four acres of grounds and turned the estate into one of the most carefully designed pieces of cultural infrastructure in twentieth-century America.

The terraced gardens that climb down the slope behind the house are the masterpiece of Beatrix Farrand, the only woman among the founding members of the American Society of Landscape Architects and one of the most accomplished American garden designers of the early twentieth century. Farrand and Mildred Bliss worked together on the gardens for thirty years, from 1921 to 1951. The result is a sequence of formal terraces near the house that gradually loosen into wilder, more naturalistic spaces as you descend. The Pebble Garden, the Rose Garden, Lover's Lane Pool, the Star, the North Vista, the Ellipse, the Orangery, the Pool Garden, and the Kitchen Garden each occupy their own piece of the slope with their own logic. The naturalistic lower section, twenty-seven acres, was deeded to the National Park Service in 1940 and became Dumbarton Oaks Park, a wooded valley still open to the public free of charge.

In late August through early October 1944, delegations from China, the Soviet Union, the United Kingdom, and the United States met in the house's reception rooms and music room to draft the structure of what would become the United Nations. The official name was the Washington Conversations on International Peace and Security Organization. Among historians and diplomats, the meetings are remembered as the Dumbarton Oaks Conference. The proposals worked out here, including the structure of the Security Council with its permanent members and the General Assembly, the relationships between the bodies, and the basic mission of the new organization, became the working drafts taken to San Francisco in April 1945 to be finalized as the United Nations Charter. The Bliss music room, the same room where Stravinsky's concerto had premiered six years earlier, was used for plenary sessions when the principal conference rooms grew too small.

In 1940 the Blisses gave Dumbarton Oaks to Harvard University, Robert's alma mater. The gift included sixteen acres of garden, the house and its furnishings, the rapidly growing collections of Byzantine and Pre-Columbian art, and an endowment for research fellowships. The arrangement made Dumbarton Oaks a Harvard institution but kept it physically and intellectually rooted in Washington. The Blisses retained a residence on the property until their deaths (Robert in 1962, Mildred in 1969), and Mildred's will contained a remarkable preamble insisting that Dumbarton Oaks was conceived as a new pattern where quality and not number shall determine the choice of scholars, that the house and gardens have their educational importance, and that trees are noble elements to be protected by successive generations. The institute has supported scholarship in Byzantine studies, Pre-Columbian studies, and garden and landscape architecture ever since.

In 1959 the Blisses commissioned Philip Johnson to design a separate pavilion to house the Robert Woods Bliss Collection of Pre-Columbian Art. The result, completed in 1963, is one of Johnson's most successful small buildings: eight domed circular galleries set inside a perfect square, with curved glass walls between them so that the surrounding garden seems to walk right up to the displays. Johnson cited the sixteenth-century Ottoman architect Sinan as inspiration. Inside, jade carvings by Olmec sculptors, Aztec stone deities, Maya relief panels with the likenesses of kings, and Andean gold and textiles sit on teak floors edged with green Vermont marble. The Byzantine Collection occupies its own pavilion designed by Thomas Tileston Waterman in 1938. A separate library by Robert Venturi, completed in 2005, holds more than two hundred thousand volumes, including Mildred Bliss's extraordinary garden rare book collection: original Redoute watercolors, sixteenth-century herbals, Renaissance treatises on hydraulic fountains, every culture's attempt to keep gardens on the page. The collection, the place, and the gardens are all open to the public most days of the week. Admission is modest. The music room is sometimes still used for concerts.