On August 24, 1814, during the War of 1812, British troops set the White House on fire in retaliation for American actions in Canada. Only the exterior walls survived, and most of those had to be torn down because the fire and weather had weakened them. Of the objects looted that night, only three have ever been recovered: a copy of the Lansdowne portrait of Washington, rescued during the chaos; a jewelry box returned by a Canadian man in 1939; and a medicine chest that belonged to James Madison, returned the same year by descendants of a Royal Navy officer. President James Monroe moved back into the partially rebuilt residence in October 1817. The South Portico was added in 1824; the North Portico in 1829. The fire that nearly destroyed the building also gave it the form it has today, because almost everything visible above the basement was rebuilt in the years that followed.

By 1948, after the fourth-floor attic of the Coolidge years and the Truman Balcony of 1948 had piled extra weight onto a building still held up by 150-year-old timber beams, the White House was declared in imminent danger of collapse. President Harry Truman moved across the street to Blair House and stayed there from 1949 to 1952. The Philadelphia contractor John McShain gutted the residence. Workers numbered every interior fixture, dismantled the rooms, and built a new steel frame inside the original sandstone walls. Two sub-basements were dug, a bomb shelter installed, and central air conditioning added. Truman had the original timber framing sawed into paneling, which now lines the Vermeil Room, the Library, the China Room, and the Map Room on the ground floor. The total cost was $5.7 million in 1952 dollars. What stands today is a 1952 building inside the shell of an 1800 one.

Jacqueline Kennedy thought the Truman-era interiors were beneath the building's history. In 1961 she enlisted Henry Francis du Pont of Winterthur and the Parisian decorator Stephane Boudin to redo the State Rooms with a coherent historical sensibility - Federal style for the Green Room, French Empire for the Blue, American Empire for the Red, Louis XVI for the Yellow Oval. She tracked down original White House furniture that had been sold off over the decades and persuaded Congress to declare the building a museum in September 1961, preventing further dispersal. The Diplomatic Reception Room got an 1834 Zuber wallpaper called Vue de l'Amerique Nord, salvaged from a Maryland house just before it was demolished for a grocery store. On Valentine's Day 1962, Kennedy gave a televised tour of the restoration to 80 million Americans. The first official White House guidebook, edited under her supervision, sold well enough to help pay for the work.

Theodore Roosevelt moved all the executive offices out of the residence in 1902, building a temporary West Wing to give his large family room to breathe. William Howard Taft enlarged it seven years later and added the first Oval Office, which his architect Nathan C. Wyeth designed as a literal oval - echoing, perhaps, the bow-fronted Blue Room directly across the South Lawn. The West Wing burned on Christmas Eve 1929 and Herbert Hoover moved his staff back in by April 1930. Franklin Roosevelt relocated the Oval Office to its current spot, next to the Rose Garden, in the 1930s. Today the West Wing houses about fifty staff at any given time, the Cabinet Room, the Situation Room (built over Roosevelt's old swimming pool), and the James S. Brady Press Briefing Room, also built over that pool. Aaron Sorkin's television show partially replicated the Oval Office on a Burbank sound stage, which is how a lot of Americans first imagined what the West Wing actually looked like inside.

The White House has been under near-constant renovation since the first cornerstone was laid. Andrew Jackson's 1829 inauguration drew about 20,000 people inside; aides had to lure the crowd back outside with washtubs of orange juice and whiskey. Public open houses continued in some form until the 1930s. In 2025 the East Wing - built in 1902, enlarged by FDR in 1942 to conceal the construction of the Presidential Emergency Operations Center beneath it - was torn down to make room for a privately funded 89,000-square-foot ballroom addition, the largest structural change to the complex since Truman. Two commemorative magnolia trees, planted for Warren Harding and Franklin Roosevelt, came down with the wing. The building keeps changing because the people in it do. What Adams asked heaven for - that only honest and wise men ever rule under its roof - is still the open question of the place. The walls outlast everyone who walks through them.