The Gordons did not start here. They took their name from lands near Kelso in the Scottish Borders, and only worked their way north when Robert the Bruce, in 1296, rewarded Sir Adam Gordon with great swaths of Aberdeenshire, Banffshire, and Moray. By 1479, George Gordon, 2nd Earl of Huntly, broke ground on a tower house in the Forest of Enzie that locals called Bog o' Gight - Windy Bog. Six storeys of dressed stone rose where peat had been. The Earls were styled Cock o' the North, which sounds like a boast and was. Through the Reformation they stayed stubbornly Catholic. Through the Jacobite risings they backed the Stuarts. The visiting Englishman Richard Franck stared up at the towers in 1672 and wrote that they seemed to "storm the air and seemingly make dents in the very clouds." One contemporary inventory lists a parrot cage in the long gallery.

Alexander Gordon, 4th Duke of Gordon, held the dukedom for seventy-five years, a span long enough to remake a landscape. In 1769 he hired the Edinburgh architect John Baxter, who absorbed the medieval tower into a four-storey central block flanked by long wings - 173 metres of pale Neoclassical facade, longer than two football fields end-to-end. The Duke also moved the entire village of Fochabers southwards to open up his park, and laid out the great Walled Garden of 1803-1804 with stone brought from Burghead. His wife Jane Maxwell did the rest. Brilliant, mercurial, eventually estranged, she turned Gordon Castle into a Highland salon. Robert Burns came in 1787 and wrote the poem "Castle Gordon" out of gratitude. Their son George raised the regiment that became the Gordon Highlanders, and helped legalise whisky production for a tenant named George Smith - the same Smith whose Glenlivet still pours.

The First World War turned the castle into a hospital. Wounded Gordon Highlanders convalesced in rooms that had once held Edwardian shooting parties. Lord Bernard Gordon-Lennox died at the First Battle of Ypres in November 1914, aged 36. His son, Lord Settrington, was killed five years later during the British intervention in North Russia. The pre-war world that filled this place did not come back. By 1938, double death duties forced the 9th Duke to hand the castle and grounds to the Crown Estate, and most of the contents were sold. The Second World War brought soldiers, then harsh winters, then dry rot. By 1950 the central block - the part Baxter had built around the old tower - was beyond saving.

In the early 1950s Sir George Gordon-Lennox, known as Geordie, the son of the Lord Bernard who died at Ypres, bought the property back from the Crown. He and his wife Nan walked through the ruined central block and made a choice that still divides Scottish heritage opinion. They demolished it. The four-storey core and one of the long wings came down in 1954. The east wing and the medieval Bog o' Gight tower were converted into a family house, the proportions suddenly intimate where they had been imperial. The Walled Garden, by then half-forgotten, has since 2008 been restored under Angus and Zara Gordon Lennox with the designer Arne Maynard. Espaliered fruit, vegetable beds, a cafe, Gordon Castle Gin distilled from estate botanicals - the heart of the estate is now the garden, not the house.

From a few thousand feet, the cropped footprint of the demolished central block is still legible in the lawn. The surviving east wing reads as a discreet country house. The Walled Garden, one of the largest in Britain, is the green square that catches the light. The Spey winds north past Fochabers to the sea at Spey Bay, where dolphins feed at the river mouth. To the south, the wooded skirts of the Cairngorms begin to rise. It is a quietly humbling view: a family that once held more land than some German principalities, distilled down to the wing they could still afford to heat.