Sir Alan de Mortimer acquired the barony of Aberdour in 1126 by marrying Anicea, daughter of Sir John de Vipont. Around 1140 he built St Fillan's Church, which still stands next to the castle. His descendants probably put up the first hall house around 1200 - the rectangular two-storey block whose lowest courses survive today. In 1386 Aberdour and Dalkeith were combined into a single barony under the Douglases, with Dalkeith as the main seat and Aberdour as the second home. The Douglases kept adding. The hall house became a tower house in the 15th century. A new stair tower and south block went up around 1500. Two more 16th-century extensions followed. Around 1635 came the final addition: an elegant east range with renaissance details, a long gallery for entertaining, and a painted ceiling decorated in fruit, foliage, and heraldic emblems.

The most notorious owner of Aberdour was James Douglas, 4th Earl of Morton, who held the castle from 1553. In 1566 Morton helped plan the rebellion against Mary, Queen of Scots, that ended in the murder of her Italian secretary David Riccio at Holyrood. The plot failed politically and Morton fled to England. Mary was forced to abdicate in July 1567. Morton was later Regent of Scotland for the young James VI. He was implicated in the 1567 murder of Mary's second husband, Lord Darnley - the explosion at Kirk o' Field - and was executed in 1581 on the young king's orders. His Aberdour mother-in-law, the dowager Countess Katherine Stewart, had held a garrison at the castle during the Rough Wooing of 1549. Family quarrels at Aberdour came with armies attached.

The 4th Earl of Morton also created the four broad L-shaped terraces that step down the south slope of the castle. They are the oldest documented garden of their kind in Scotland, predating the Renaissance gardens of England by a generation. At the bottom an orchard was laid out in 1690 and replanted in recent decades. The high retaining walls were rebuilt in 1981. To the west, a 17th-century walled garden covers about 5,000 square metres within walls four metres high. Carved pediments mark the gates: strapwork and the Douglas heart over the west door that led to the terraces; the date 1632 and the initials of the Earl and Countess on the east door, which opened to St Fillan's Church. In the centre of the garden stands a polygonal sundial of the early 17th century, mounted now on a 19th-century base.

The Douglas line ran into trouble. William Douglas, 7th Earl of Morton, was Treasurer of Scotland from 1630 to 1636 and a fierce supporter of Charles I during the Wars of the Three Kingdoms. He spent his fortune on the royal cause, and in 1642 was forced to sell Dalkeith to the Earl of Buccleuch. Aberdour became the main residence. A fire in the late 17th century damaged the central range, and in 1725 the family bought nearby Aberdour House and effectively abandoned the medieval castle. By the early 19th century only the east range was still roofed. The tower collapsed in stages. Historic Environment Scotland took the property into care and opened it to the public. Today the east range and its painted ceiling survive intact; the terraced gardens still march down the slope; and on a clear day you can see Edinburgh across the Forth, just as the Douglases did.