In 1756 John Spencer, the first Earl Spencer, decided he needed a London house worthy of his position and his ancestral country seat at Althorp. He chose a triangular site at 27 St James's Place, just off Green Park in the heart of Westminster's clubland. His first architect was John Vardy, who had trained under William Kent and is responsible for the exterior facades that survive today. In 1758, the project changed hands. Vardy was replaced by James "Athenian" Stuart, a former classical scholar who had spent years in Greece making the first accurate drawings of the Parthenon and the monuments of Athens. Stuart designed the interiors using authentic Greek details, the painted Painted Room, the Great Room with its giltwood and palms. Spencer House became one of the very first examples in London of what would soon be called the neoclassical style. The Spencer family lived in it, more or less continuously, until 1895.

In the late 19th and early 20th centuries the Spencers began to let the house out for periods, often to wealthy Americans. In 1897 the 9th Duke of Marlborough and his bride Consuelo Vanderbilt leased it; their first child John, the future 10th Duke, was born here on 18 September 1897. Consuelo felt that giving birth at the Spencer family's London house was appropriate, given that the Spencer-Churchills descended from the same family. In 1901 the lease passed to Mrs Ogden Goelet of New York and her daughter Mary, who would later marry the 8th Duke of Roxburghe. The 5th Earl's wife, Charlotte Spencer, died here in 1903; King Edward VII, Queen Alexandra, and their two daughters had called on her three weeks earlier to inquire after her health. From 1919 the house was leased to the wealthy American widow Mrs William Bateman Leeds, who would soon marry Prince Christopher of Greece and become Princess Anastasia of Greece, opening the door to that extraordinary 1922 winter of Greek royal exile.

After the death of the 6th Earl in 1922, the Spencers gave up living at the house. Heavy estate duties of around £359,000 against an estate valued at just under £1.2 million ate badly into the family's resources. The 7th Earl, Albert Spencer, kept the freehold and refused to sell, even when offers came from the Bath Club and the Royal Society in the 1940s. His grandson Charles, the 9th Earl Spencer (and brother of Diana, Princess of Wales) credits Albert with the decision to hold on through the lean years. Many other London townhouses of equivalent grandeur, owned by other peers, were sold and demolished in the same period; today only a handful survive, including Spencer House, Lancaster House, Bridgewater House, and Apsley House. The house had its own war service, lending its rooms to the Royal Commissions on Defence of the Realm Losses from 1917 to 1919, and being requisitioned again during the Second World War. The Blitz left some damage.

By 1985 the house had been let to the Economist Intelligence Unit for over twenty years and was showing its age. That year Jacob Rothschild, 4th Baron Rothschild, secured a 96-year lease through RIT Capital Partners at an annual rent of £85,000. He spent £16 million restoring the house to its 18th-century appearance over the next five years. The restoration, which took approximately five years to complete, culminated in a 500-person reception held on 19 November 1990 to celebrate the project. Diana, Princess of Wales, attended; the freeholder, her brother Charles Spencer, was her host. The Painted Room was returned to its James Stuart designs. The Great Room's gilding was restored. Furniture and paintings of the period were brought back from Althorp on loan. The result is unusual in London: a complete 18th-century aristocratic interior, in its original setting, restored well enough to function as a working venue.

The terms of the Rothschild lease require some quirks. RIT must redecorate the external stonework every three years and the internal rooms every seven. They are limited in how often they can hold events. The Spencer Trustees retain reserved use of certain "fine rooms." Most importantly, the lease requires the house to be open to the public on Sundays except during August, meaning that for most of the year anyone can pay a ticket and walk through state rooms designed by an architect who measured the Parthenon by hand. Spencer House today is also a private events venue, with banquet hire reportedly running at £10,000 plus catering. It is the last great Georgian London townhouse open to visitors in something close to its original form, and its survival, against odds that swept away the others, owed itself first to one earl's stubbornness and then to one financier's generosity.