After the Norman Conquest and the brutal Harrying of the North in 1069 to 1070, William the Conqueror needed loyal men holding the frontier against Danish counter-invasion. He gave Holderness to Drogo de la Beuvriere, a Flemish mercenary, and made him the first Lord of Holderness. Drogo's estate stretched from the Humber to Bridlington. Skipsea Castle was its administrative centre and his principal residence, his caput, perched on a strategic inland harbour. The name Skipsea has Scandinavian roots and meant a lake navigable by ships. A channel ran from the castle to the North Sea two kilometres east, and Drogo's complex included a private harbour, probably a boat yard, and a fresh-water fishery. The whole low-lying region around the castle was sometimes called an island.

Drogo did not enjoy his lordship for long. His wife, who was reportedly a kinswoman of William the Conqueror, died in suspicious circumstances. Drogo fled England rather than face questions. The Conqueror reassigned Skipsea to Odo, the Count of Aumale, and the castle passed through that family for the next two centuries. Around 1160, the counts laid out a planned town called Skipsea Brough alongside the bailey, hoping that trade through the harbour would bring revenue and defenders to the most vulnerable side of the castle. The plan did not succeed. By 1260 only three burgesses were paying rent in the town. The harbour silted, the mere drained, and Skipsea Brough remained one of the great failed medieval new towns of England, today a handful of buildings standing where streets had been mapped.

In January 1220 William de Forz, the Count of Aumale by marriage, rebelled against the young King Henry III over the seizure of the manor of Driffield and other grievances. Henry excommunicated him and ordered the northern barons to besiege his castles. Royalist forces took Skipsea, William surrendered and was eventually pardoned, and the King issued an order to destroy the castle. Whether the order was fully carried out remains unclear from the archaeology. The earthworks survive substantially intact, with a deliberate gap called Scotch Gap cut through the bank that may date from this slighting episode. By 1397 the castle was officially valued as worthless, the 20 acres around it used as pasture, and the local lords moved their working residence to the nearby manor at Cleeton.

The site you can walk today is one of the largest motte-and-bailey castles in England. The motte itself is 100 metres in diameter and 11 metres high, built deliberately onto the existing Iron Age mound with sand and gravel to make it appear even larger. A bank and a ten-metre-wide ditch ring the base. On top, only a quarter of an acre of flat space carried the timber keep and possibly a stone gatehouse. The bailey curves around it across 8.25 acres, with ramparts up to four metres tall and another ditch of equal depth, the main entrance through the Bail Gate on the south side. Skipsea Mere, the artificial lake that once divided motte from bailey, has been drained for centuries. English Heritage took the site into guardianship in 1911 and manages it today, free to visit.