John Tayloe III was, by most accounts, the wealthiest planter in Virginia. Educated at Eton and Cambridge, heir to the 60,000-acre Mount Airy plantation empire along the Rappahannock River, he had every reason to build his city house in established Philadelphia. But George Washington had other plans. Washington, who was connected to Tayloe through family marriage, persuaded the colonel to invest in the fledgling capital instead. On April 19, 1797, Tayloe paid $1,000 for a lot in open country west of the half-built President's House, about a mile from Georgetown. He chose architect William Thornton, the same man who designed the United States Capitol, to create something bold. Thornton delivered: a three-story brick house whose plan combined a circle, two rectangles, and a triangle to fit an irregular lot. The result was a zenith of Federal architecture, with Coade stone ornaments imported from England and local Aquia Creek sandstone anchoring the walls. No one has ever satisfactorily explained why a six-sided building was christened the Octagon.

In August 1814, British forces marched into Washington and set the White House ablaze. The Octagon survived. Ann Ogle Tayloe, thinking quickly, had offered the house to the French consul, and when the redcoats arrived they found a French flag flying from the entrance. Whether it was diplomacy or the standing British orders not to damage private property, the house was spared. When First Lady Dolley Madison fled the city ahead of the flames, she sent her pet parrot to the French consulate at the Octagon for safekeeping. By September 8, 1814, President James Madison and Dolley had moved in, paying the Tayloes $500 for six months' rent. The Octagon became one of only five houses ever to serve as the presidential residence in American history, and one of just three that still stand, alongside the White House and Blair House. Dolley threw her famous Wednesday night parties, known as 'squeezes,' in the drawing rooms. And on February 17, 1815, in the upstairs study, President Madison ratified the Treaty of Ghent, ending the War of 1812.

John Tayloe III died at the Octagon in 1828. His wife Ann lived on there until her death in 1855, and then the slow decline began. The Tayloe children rented the house to a girls' school in the 1860s and to the federal government in the 1870s, when the U.S. Navy's Hydrographic Office occupied the rooms where treaties had been signed. By the 1880s, ten families crowded the house tenement-style, probably factory workers from Foggy Bottom's industrial quarter, one family per room. Rescue came from an unexpected direction: in 1898, the American Institute of Architects chose the Octagon as their national headquarters. They rented it for four years, then purchased it outright in 1902. The AIA operated from the building until the 1960s, when their new headquarters rose next door. Declared a National Historic Landmark in 1960, the Octagon opened as a museum in 1970 and was restored to its 1817-1818 appearance in the early 1990s.

The Octagon is reputed to be one of the most haunted buildings in Washington, D.C., and its ghost stories have been circulating for nearly two centuries. The oldest legend involves the servants' call bells. Virginia Tayloe Lewis, a granddaughter of Colonel Tayloe, recorded in an unpublished manuscript: 'The bells rang for a long time after my Grandfather Tayloe's death, and every one said that the house was haunted; the wires were cut and still they rang.' By 1874, writer Mary Clemmer Ames reported that a powerful dinner guest tried to stop the ringing by grabbing the bell wires and was 'lifted bodily from the floor.' Then there is the legend of Dolley Madison's ghost, said to host spectral receptions in the front hall, heralded by the scent of lilacs. And the tale of Colonel Tayloe's daughters falling to their deaths on the spiral staircase, a dramatic story that first appeared in a 1908 newspaper article but has no historical basis whatsoever. In 1888, twelve men spent the night to debunk the legends and reported hearing 'three feminine shrieks' at midnight, followed by 'the clanking of sabers and tramping of footfalls' until dawn.