Spynie began as something stranger than a palace - it was, briefly, the cathedral of the entire Diocese of Moray. The early bishops had no fixed seat at all, drifting between houses at Birnie, Kinneddar and Spynie. In 1206 Pope Innocent III gave Bishop Bricius de Douglas formal permission to settle the cathedral here, beside a tidal loch that opened to the Moray Firth. The arrangement lasted less than two decades. Bricius himself attended the Fourth Lateran Council in 1215 and seems to have come home convinced that Spynie was the wrong place - too exposed to seaborne raiders, too far from the burgh markets at Elgin. He died in 1222 before the move; his successor Andrew of Moray finished it. On 19 July 1224 the cathedral was transferred to Elgin. The bishops kept the palace anyway. They liked it there.

Alexander Stewart, Earl of Buchan - history calls him the Wolf of Badenoch - burned Elgin Cathedral in June 1390 and seems to have seized Spynie as well. His brother King Robert III had to issue an order in August forbidding him to meddle further. The penance was theatrical: barefoot in the church at Perth before the king, the earl was forgiven. The strangest twist came after Bishop Alexander Bur's death in 1397, when the crown - claiming a vacant see - made the reformed Wolf the warden of the very castle he had recently sacked. Stone began replacing the original wooden ringwork through the 14th century, and the surviving gatehouse, with its gatekeeper's room and small fireplace, shows the careful hand of masons working under the Bishop of Moray's John Innes, who was also the king's Master of Works. The same craftsmen were repairing Elgin Cathedral after the fire.

By the mid-16th century Spynie had become a place where Scotland kept people it could not quite execute or trust. Bishop Patrick Hepburn, the last Catholic bishop here, installed wide-mouthed gun loops and enlarged the windows and went on holding prisoners through the Reformation. Edward Sutton, 4th Baron Dudley, captured at Hume Castle in 1548, spent time at Spynie during the Rough Wooing. Christopher Rokeby, an English agent who had tried to draw Mary, Queen of Scots into a conspiracy with English Catholics, was locked here for eighteen months from 1566. The following year Hepburn sheltered his kinsman the Earl of Bothwell - Mary's third husband - after his flight from Carberry Hill. Bothwell rested at Spynie before sailing on to Orkney, Shetland, and ultimately Denmark, where he died in a Danish prison still chained, by some accounts, to a pillar.

The 17th century closed the palace through war rather than neglect. Bishop John Guthrie refused to subscribe to the Covenant and prepared Spynie for siege; in July 1640, the Covenanter Colonel Robert Monro arrived with 800 men and Guthrie surrendered the same day. He was kept under house arrest in his own home, forced to pay the garrison's upkeep. Five years later, after Montrose's victory at Auldearn, royalists burned Spynie's farm buildings but didn't attempt the walls. The Marquis of Huntly besieged the castle later that year and the defences held until John Middleton relieved it. Bishop William Hay was finally expelled in 1689 for refusing the oath to William and Mary. After that, locals quarried the walls for building stone until the 19th century. The tower's east wall developed a pronounced bulge - the legacy of Covenanter attempts to blow it up - and a concrete plinth, hidden behind a reconstructed curtain wall, now holds the great mass upright. Historic Scotland opened the site in 1994.