The first castle on this site was a humble motte-and-bailey, thrown up in the Forest of Galtres by Bertram de Bulmer, Sheriff of York during the chaos of Stephen's reign. Its grass-covered mound still rises south of the parish churchyard. The stone castle that gives the village its dramatic skyline came two centuries later. John, Lord Neville obtained a licence to crenellate from Richard II in 1382, though work had probably already begun. The architect is credited as John Lewyn, the same master mason responsible for Bolton Castle in Wensleydale. Both castles share the distinctive late-fourteenth-century northern style: a quadrangular plan with massive rectangular corner towers, parapets stepping up to form turrets, the whole arrangement designed equally for comfort and defence. The fabric was rubble mudstone dressed with sandstone quarried at Terrington, a few miles east.

In 1471, Sheriff Hutton passed by marriage to Richard, Duke of Gloucester, the future Richard III. For the next dozen years, this was his northern headquarters. Its proximity to York made it convenient when he needed to consult the city's magistrates or muster troops against Scottish raiders. In October 1480, the Earl of Northumberland's warning that the Scots might retaliate reached Richard here, and he commanded the response. When he became king in 1483, he established the Council of the North at Sheriff Hutton, an administrative body that would govern the region for the next century and a half. In 1484 he made the castle the household of Edward, Earl of Warwick, his ten-year-old nephew, and John de la Pole, Earl of Lincoln, the man Richard had named his probable heir.

In August 1485, as Henry Tudor's invasion force advanced through Wales and the Midlands, Richard gathered the surviving Plantagenet children at Sheriff Hutton for safekeeping. His niece Elizabeth of York came from London with her sisters. Warwick and Lincoln were here. So was John of Gloucester, Richard's illegitimate son. If Richard won at Bosworth, these children would form the core of his dynastic future. If he lost, Henry would need to neutralise them. He lost. Lambert Simnel, a pretender claiming to be Warwick, was held briefly here before being transferred to the Tower. Elizabeth went south to marry Henry. Within a year the Council of the North continued to function from these walls under Henry's authority, but the castle's golden age as a royal residence was already over.

By 1572 Henry, Earl of Huntingdon described Sheriff Hutton as 'an olde Castell almost ruinated.' He had hoped to persuade the President of the Council of the North to use it as a residence, but the council had moved to York and the towers were already crumbling. The Ingram family bought the wreck in 1622 and quarried it for building stone, which can still be spotted in cottages throughout the village. By the early twentieth century the ruins served as a farmyard, with cattle wandering between the broken bases of the corner towers. The site became a scheduled ancient monument in the 1950s. The four tower stumps still stand, taller than seems possible for a building so thoroughly stripped, their rubble cores exposed where the dressed stone was carted away. Privately owned and rarely opened to the public, the castle keeps its silence.