The name comes from Scots Gaelic: fionn for white, leitir for cliff or steep slope. The cliffs at Findlater contain quartz, and in the right light they catch the sun and read as pale stone against grey North Sea water. The castle sits on a fifty foot high outcrop overlooking the Moray Firth, fifteen kilometres west of Banff and a few miles from the village of Sandend. The first historical reference dates to 1246. King Alexander III repaired the castle in the 1260s, anticipating an invasion by Haakon IV of Norway. The Norwegians came, took the castle, and held it for a time. The remains visible today are largely from a fourteenth-century rebuilding, modelled on Roslin Castle south of Edinburgh.

Walter Ogilvy obtained a licence from James II to repair the castle in February 1445, beginning the long Ogilvy connection to the site. James V of Scotland visited in November 1535, on his way home from a pilgrimage to Tain. James Ogilvy of Findlater, Master of Household to both Mary of Guise and her daughter Mary, Queen of Scots, was at the very centre of Scottish court life, which is what made his domestic plan so unusually grim. The attempt to break his father's mind by sleep deprivation is documented in the historical record, however briefly. The plan failed. James lost his inheritance to John Gordon, a son of the Earl of Huntly, in some legal manoeuvre the records do not fully explain. When the elder Ogilvy died, his widow married John Gordon, and Gordon imprisoned her.

John Gordon was a fighter as well as a schemer. In July 1562 he and James Ogilvy met in Edinburgh and Gordon injured Lord Ogilvy's arm in the brawl. By the custom of the time, Gordon was imprisoned until his victim healed. He used the time poorly. By September 1562 Mary, Queen of Scots was on a progress through the north, and she sent an army from Dunbar Castle with artillery to take Findlater and remove Gordon. When she came near herself on the twentieth, she sent a trumpeter to demand the castle's surrender. He was refused. The Earl of Huntly, Gordon's father, tried in October to send the keys of Findlater and Auchindoun to the queen, but the messenger was of such low rank that Mary refused to believe the gesture was real. The matter resolved itself the way such things often did. Huntly was defeated at the Battle of Corrichie outside Aberdeen, and John Gordon was executed there shortly after.

James Ogilvy was made 1st Earl of Findlater in 1638, restoring the family fortune in title if not in domestic comfort. He built Cullen House inland, where the family could live with central fireplaces and dry floors and views of trees rather than salt spray. By the time he died in 1653 the cliff castle was little used and beginning its decline. Three hundred and seventy years later the decline has reached the predictable end. What remains is a substantial stack of broken walls on a fifty foot quartz cliff, accessible by a narrow neck of land, free to enter and not particularly safe to explore. The Earls of Findlater and Seafield, the senior line of the Ogilvy family, kept the title for centuries afterwards. The castle that gave them the name keeps the cliff.