The original palace was given to Edward II in 1305 by the Bishop of Durham, Anthony Bek. For two centuries it was a working royal residence. According to one account, the incident that inspired Edward III's foundation of the Order of the Garter took place here. As Henry IV's favourite palace, Eltham hosted Manuel II Palaiologos - the only Byzantine emperor ever to visit England - from December 1400 to mid-February 1401. A joust was held in his honour, and the jousting tilt yard still survives. The 1401 tournament was commemorated in literary form as thirteen old French letters addressed to Henry's daughter Blanche of England, each purportedly written by a legendary patron praising one of the combatants. The letters were probably read aloud during the event itself. Edward IV built the Great Hall in the 1470s. Its hammerbeam roof is the third-largest of its type in England. The future Henry VIII grew up here. It was in this hall in 1499 that the eight-year-old prince met and impressed the scholar Erasmus, introduced by Thomas More.

Tudor courts often used Eltham for Christmas, but the rise of Greenwich Palace - more easily reached by river - left Eltham increasingly for hunting only. The deer remained plentiful in three deer parks totalling over 1,200 acres: the Great Park (596 acres), the Little or Middle Park (333 acres), and the Home or Lee Park (336 acres). The courtier Roger Aston, keeper of the little park in 1610, built four bridges for the convenience of James VI and I. In the 1630s, by which time the palace was no longer used by the royal family, Sir Anthony van Dyck was given the use of a suite of rooms as a country retreat - the great Flemish portraitist of the Stuart court taking his ease where kings had once jousted. The English Civil War broke the place. The parks were stripped of trees and deer. John Evelyn visited on 22 April 1656 and recorded: Went to see his Majesty's house at Eltham; both the palace and chapel in miserable ruins, the noble wood and park destroyed by Rich the rebel. The palace never recovered. Charles II bestowed Eltham on John Shaw after the Restoration, but by then it had been reduced to Edward IV's Great Hall, the former buttery (called Court House), a bridge across the moat, and some walling. The Shaw descendants held it as late as 1893.

In 1933, Stephen Courtauld - of the textile family that had made vast wealth from rayon and silk - and his wife Virginia (Ginie, née Peirano) acquired a 99-year lease on the site. They commissioned the architects Seely & Paget to restore the great hall and build a modern home attached to it. Seely and Paget added a minstrel's gallery and timber screen to the hall, and designed the external profile of the new house in a neo-Wren style inspired by Hampton Court and Trinity College, Cambridge. Inside, however, the decoration went somewhere entirely different. The entrance hall, by the Swedish designer Rolf Engstromer, was crowned by a glazed dome that flooded blackbean veneer and figurative marquetry with light. The dining room, drawing room, and Virginia's circular bedroom with its adjoining bathroom were the work of the Italian designer Piero Malacrida de Saint-August. Bedrooms were by Seely and Paget. Throughout, the materials were the most luxurious available: rare veneers, onyx, marble, gold leaf, fitted technology that was years ahead of British standards. The Courtaulds had a pet ring-tailed lemur named Mah-Jongg who had his own room on the upper floor with a hatch leading to the downstairs flower room. Mah-Jongg had the run of the house. Guests, by all accounts, learned to keep their valuables put away.

In September 1940, Stephen Courtauld was on duty on the Great Hall roof as a fire watcher in the local Civil Defence Service when the hall was badly damaged by German incendiary bombs. The Courtaulds remained at Eltham until 1944, then moved first to Scotland and then to Southern Rhodesia (now Zimbabwe). In March 1945, the palace went to the Royal Army Educational Corps, which moved all its administration here in 1948 and stayed until the Educational and Training Services Branch reorganisation in 1992. In 1995, English Heritage assumed management, completing major repairs and restorations of interiors and gardens by 1999. The medieval great hall and the Art Deco mansion now welcome visitors as a single attraction - one that demonstrates how completely a building can keep being itself across radically different eras. The hammerbeam roof Edward IV's masons raised in the 1470s now sits adjacent to a circular bedroom that would not have looked out of place on the Cunard liner Queen Mary. Both are equally Eltham.