Before this building, the State, War, and Navy Departments operated out of three small federal buildings flanking the White House, built between 1799 and 1820 on land that had been the Washington Jockey Club. After the Civil War the federal government's responsibilities had multiplied beyond the capacity of those small offices. In 1869 Congress appointed a commission to find a site and produce plans for a new combined building. President Ulysses S. Grant commissioned the project. Construction began in 1871 and finished in 1888, seventeen years later, at a cost of $10,038,482, an extraordinary sum for the period. The architect was Alfred B. Mullett, the Supervising Architect of the Treasury Department, who chose the French Second Empire style: mansard roofs, dormer windows, granite columns, ornamental ironwork at every visible joint. The new building stood directly across West Executive Avenue from the White House, a flamboyant contrast to the neoclassicism of the buildings around it.

Every block of the exterior granite was quarried, cut, and polished on Vinalhaven Island in Penobscot Bay, Maine, under contract with the Bodwell Granite Company. Schooners carried it down the New England coast and up the Potomac to Washington. The interior was designed by the Italian-born architect Richard von Ezdorf, who used cast iron almost everywhere structural and decorative work was needed. The skylights over the major stairwells are cast iron and glass. The hardware was bespoke: each doorknob carried a cast pattern indicating which of the three original departments occupied the room behind the door, so a clerk hurrying through the corridors could know at a glance whether he was entering State, War, or Navy. Most of those doorknobs are still in place. The corridors run a total of nearly two miles.

Theodore Roosevelt worked in the building as Assistant Secretary of the Navy. So did Franklin Roosevelt thirty years later. William Howard Taft worked here as Secretary of War. Eisenhower had an office here while assigned to the Army General Staff. Truman moved his presidential staff into the building during the complete reconstruction of the White House from 1948 to 1952 and worked out of these rooms for nearly four years. Herbert Hoover used the Secretary of the Navy's suite for several months in 1929 after a Christmas Eve fire damaged the Oval Office. Eisenhower held the first televised presidential news conference here in the Indian Treaty Room (Room 474) on January 19, 1955. Richard Nixon kept a private hideaway office in Room 180 throughout his presidency and preferred to do most of his actual work there, using the Oval Office for ceremonial occasions only. Japanese emissaries met Secretary of State Cordell Hull in this building on the afternoon of December 7, 1941, after Pearl Harbor.

The original three departments outgrew the building and finally moved out completely in the late 1930s. The Department of State moved to Foggy Bottom; the War Department's offices became the nucleus of the Pentagon. The Old Executive Office Building, as it became known, was repurposed for the growing Executive Office of the President. By the 1950s it was so disliked, and so inefficient compared to modern office construction, that the Eisenhower administration nearly demolished it in 1957. Eisenhower himself reportedly considered the building ugly but valued the proximity to the White House. The plan to tear it down was eventually shelved. In 1969 the building was designated a National Historic Landmark, which made future demolition essentially impossible. A restoration of the original Secretary of the Navy suite was completed in 1987, and it now serves as the ceremonial office of the Vice President. The full building was modernized between 2008 and 2014 while preserving its historic character.

In 1999, the Old Executive Office Building was renamed for Dwight D. Eisenhower, the thirty-fourth president and the five-star Army general who had commanded the Allied forces in Europe during the Second World War. Eisenhower had worked in the building several times in his career, both during his Army service and during his presidency. The building today houses the Office of the Vice President, the Office of Management and Budget, the National Security Council, and the Council of Economic Advisers. It is connected to the West Wing of the White House by a covered passage beneath West Executive Avenue. After the September 11 attacks, the 17th Street side was vacated, hardened, and modernized. A small fire in December 2007 damaged some of the offices of Vice President Cheney's staff. The Vinalhaven granite still holds the building up. The doorknobs still tell anyone who looks closely whether the room behind them once belonged to State, War, or Navy.