In 1130, a Pictish-descended lord called Oengus, Mormaer of Moray, led a rebellion against David I. He lost, and he died in the fighting. King David then handed Moray to outsiders - men he could trust to hold a rebellious province for the crown. Freskin was one of them, and almost certainly Flemish, though his origins are murky enough that historians have argued for an Anglo-Saxon background or even a local Gael who fought on the king's side. Whatever his birth, his family stuck. By the thirteenth century his descendants were calling themselves de Moravia - 'of Moray' - and had become one of the most powerful families in northern Scotland. The castle he raised on the boggy ground east of the Loch of Spynie was certainly standing by 1151, when King David himself stopped to visit.

The first Duffus was a motte-and-bailey, the standard Norman frontier fortress. A man-made mound with steeply sloping sides, a wide ditch around the base, a flat top crowned by a wooden tower behind a palisade. Steps led up from the bailey - the lower walled enclosure where the bake-house, brew-house, workshops and stables stood. Up on the motte lived the lord and his immediate household. In a country where stone fortresses were still rare, an earthwork like Duffus was state of the art. The Scottish chronicles suggest the place may have been wrecked in 1297 during the First War of Scottish Independence, and possibly damaged again in 1306 when Robert the Bruce campaigned through Moray. By then the family who owned it was no longer Freskin's direct line.

In 1270 the castle passed by marriage to Sir Reginald le Chen, who married Mary de Moravia - one of Freskin's descendants. His grandson and namesake died in 1345, and Duffus went to a daughter, Mariot, who had married Nicholas Sutherland, second son of the 4th Earl of Sutherland. The Sutherlands were themselves descended from Freskin, and they would hold Duffus for the next 360 years. Sometime in the early 1300s the family decided to replace the timber tower with stone. The trouble was simple: they were building a heavy stone keep on top of a man-made mound that sat on a bog. The motte was never engineered to carry stone. As the centuries passed, the great square tower began to lean. A whole corner eventually broke away and slid down the slope of the motte - it is still there, embedded in the hillside, leaning at a drunken angle.

By 1705, when the 2nd Lord Duffus died, the castle had become unsuitable for living. The family moved out. The Laich of Moray reclaimed nothing - the ruin still stands, a square stone keep on a slipping mound, surrounded by the wide flat fields that were once marsh. Drainage works in the 1800s lowered the water table and turned the Laich into some of the best farmland in Scotland. The bog beneath Duffus dried, but by then it had already done its damage. Today the castle is in the care of Historic Environment Scotland, open to anyone who wants to walk the bailey, climb the slope, and stand inside walls that have been falling, very slowly, for seven hundred years.