The first castle at Tutbury was raised by Henry de Ferrers, one of William the Conqueror's most powerful Norman followers, sometime after 1066. Henry made it his headquarters in the wapentake of Appletree, the medieval administrative district that included the hunting forests of Duffield Frith. Around 1080 he and his wife Bertha endowed nearby Tutbury Priory, and for a time the new English priory was a dependency of the Norman abbey of St Pierre-sur-Dives, an act of long-distance feudal piety that linked this corner of Staffordshire to the Calvados hills back home. The de Ferrers family held the castle for a century, until William de Ferrers picked the wrong side in the revolt of 1173 against Henry II and saw Tutbury slighted, that is, deliberately damaged so it could not be used militarily, along with twenty other rebel strongholds.

The de Ferrers kept rebelling. In 1264, after the family backed Simon de Montfort against the crown, Prince Edward, the future Edward I, nearly destroyed the castle. Five years later Henry III gave the confiscated estates to his second son, Edmund Crouchback, founding the Duchy of Lancaster, the royal estate to which Tutbury still belongs after 756 years. By the early 1300s the castle had been rebuilt as a residence rather than a fortress. Thomas, second Earl of Lancaster, made Tutbury his principal home and lived there in what one contemporary called "more than princely style." In 1313 his household accounts ran to 22,000 pounds, a stupefying sum at a time when food was cheap. Local farmers and craftsmen found a guaranteed market for everything they could produce. For a few decades, Tutbury was less a castle than a small inland court.

Mary Stuart arrived at Tutbury on 4 February 1569, after a long winter journey south from Bolton Castle in Yorkshire. The English Privy Council wanted her far from the Scottish border, and Tutbury, with its high enclosure and its remote position, suited Elizabeth's anxieties. Tapestries and bedding were rushed from the Tower of London, but bad weather delayed the wagons, and Bess of Hardwick had to strip her own house at Sheffield to furnish the queen's rooms. Mary noted, with a Frenchwoman's eye, that the castle looked like the hunting lodge at the Bois de Vincennes outside Paris. She complained of the damp, the wet plaster, the rotting carpentry. She passed time embroidering with Lady Livingston and Mary Seton in Bess of Hardwick's chamber. She was moved out in April, brought back in September with a tighter guard, rushed to Coventry that November when northern Catholics rose in her name, and ordered back to Tutbury on Christmas Eve. By May 1570 she was gone, sent on to Chatsworth and a long road that would end on the scaffold at Fotheringhay.

In January 1585 Mary came back, and this time her treatment hardened. Her keepers, Ralph Sadler and John Somers, struggled to find acceptable hangings for her bedchamber, which sat under the eaves of a timber-built lodging with windows that opened only into the castle courtyard. To keep warm she rigged a tent of tapestry over her bed. The French ambassador Michel de Castelnau wrote to her son James VI describing her early optimism about "Teutbery," but the letters that followed catalogued small humiliations. Sadler took her hawking on the River Dove with a guard of 40 or 50 horsemen, an indulgence that scandalized Elizabeth and got him replaced in April by the puritanical Sir Amyas Paulet. Paulet took down Mary's cloth of estate, the canopy that signaled her queenship, from the room where he ate his meals. He stopped her servants walking on the wall walk. He curtailed her almsgiving in the town and disarmed her Scottish household, many of whom had carried pistols. By Christmas Eve 1585 she was moved on to Chartley, and from Chartley to the Babington plot, and from the Babington plot to her death.

The castle's last act as a real military post came in the English Civil War. Held for the king by Sir Andrew Kniveton, it withstood one Parliamentary assault in 1643 and held out until April 1646, when its garrison surrendered after a three-week siege. Parliament, having decided that the place was untenable, ordered the fortifications demolished between 1647 and 1648. The walls came down, the towers were left as picturesque stumps, and in 1780 a romantic-minded landowner crowned the motte with a sham medieval tower, Julius' Tower, named for nothing in particular. Today the castle is a scheduled monument and Grade I listed, open to visitors who climb the hill knowing that a French-speaking queen once shivered in a room here with a tapestry tent over her bed.