Eye Castle is a motte and bailey structure: an earthen mound topped by a fortification, with a walled courtyard — the bailey — alongside. The motte at Eye is 160 feet in diameter and 40 feet high; the bailey originally extended some 400 feet. The castle is unusual in being one of only two castles mentioned in the Domesday Book of 1086 as a source of income for their owners. The income came from a market held within the castle bailey — and that market competed directly with the Bishop of Norwich's market at Hoxne, a few miles away. Such commercial arrangements were common enough in Norman England; what is less common is finding them recorded so explicitly in the Domesday survey. The castle's early owners controlled not just the fortification but the economic life of the surrounding region.

Eye's history in the twelfth century tracks the larger story of English baronial conflict. William Malet's son Robert was exiled and killed at the Battle of Tinchebray in 1106, after which Henry I confiscated the castle. Stephen of Blois received it as a gift in 1113 and held it when he became king in 1135. Henry II, coming to power in 1154, was equally systematic in consolidating castles he considered threats: he confiscated Eye alongside the four major Bigod strongholds — Framlingham, Bungay, Walton, and Thetford — as he worked to re-establish royal control across East Anglia. When Hugh Bigod joined the revolt of Henry's sons in 1173, Eye was among the castles attacked. The attack failed, and the castle was rebuilt, with two square towers added to the north side of the inner bailey. Then, in 1265, during the Second Barons' War, it was sacked and largely abandoned. By the 14th century it was mostly a ruin, though parts were maintained as a prison.

A windmill was built on top of the motte in 1561 or 1562 — which says something about how thoroughly military ambition had given way to practical necessity. The park that had surrounded the castle was broken up in the early 17th century and turned into fields. A workhouse and a school appeared inside the castle bailey in the 1830s. Then, in 1844, Sir Edward Kerrison built a stone house on the motte itself — a Gothic Revival structure meant to command the same views that Norman fortifications once claimed. It did not last. The house decayed into ruin, and became known to subsequent generations as Kerrison's Folly. The mound and some stone fragments of the original castle still remain. The site is a scheduled monument and a Grade I listed building. Historically the Lord of the Manor of Eye Sokemere holds the title of Constable of the Castle — a ceremonial continuity of Norman administrative language that the Palmer family of Haughley still carries today.