Long before Hastings, a more modest house had stood at Ashby. The Domesday Book records a manor here in 1086, and through the 12th and 13th centuries the La Zouche family held it under the Earls of Leicester, giving the village the second half of its peculiar name. By the mid-1300s the building included a hall, a chamber, service wings, a dovecote, an orchard, a rabbit warren, and a 60-acre deer park. The historian Norman Pounds described it as a "rather modest manor house," comfortable but unimportant. The Le Zouche line died out in 1399, and the manor's inheritance drifted through decades of Wars of the Roses confusion before the Yorkist William Hastings acquired it. In 1474 Edward IV granted him licence to crenellate four of his manors and build a 3,000-acre deer park around them. Ashby was the project Hastings poured himself into.

The castle eventually passed to George Hastings, made Earl of Huntingdon by Henry VIII in 1529. George rebuilt parts in brick and redesigned the gardens. His grandson Henry inherited in 1560 and maintained a household of 77 servants, on the scale of a small county aristocracy. In 1569 Henry Hastings took on a heavier responsibility: he was made one of the keepers of Mary Queen of Scots, who had been accused of plotting against her cousin Elizabeth I. Mary spent time at Ashby that year, a prisoner of state in a castle that doubled as a country residence. Henry himself spent most of his time at York, presiding over the Council of the North. The castle's rooms held a queen whose claim to the English throne was used by every Catholic conspirator of the era and whose execution at Fotheringhay in 1587 ended Elizabeth's long reluctance to kill another monarch. Visitors to Ashby walked through chambers where she had been watched, fed, and waited.

The Hastings family had a talent for hosting kings. James I came three times between 1612 and 1617. Charles I visited in 1634. The Countess of Derby was welcomed in August 1607 with a specially-commissioned Masque at Ashby Castle, the kind of expensive entertainment that involved music, costumes, and elaborate scenery. The gardens were redeveloped repeatedly for these visits, becoming what the historian John Goodall has called "one of the best-preserved and most important" examples of early Tudor garden design. The cost ruined the family's finances. By the start of the Civil War in 1642 the estate had been diminished by generations of expenditure, and the family struggled to maintain its former regional prominence. The eldest son Ferdinando inherited the earldom and stayed neutral. His younger brother Henry chose the king.

Henry Hastings, later Lord Loughborough, made Ashby Castle his base of operations for Royalist forces across the Midlands. Buildings in the town were pulled down for materiel, tunnels were dug, an "Irish fort" was built. The siege came eventually, the castle surrendered after long resistance, and a fresh rebellion in 1648 convinced Parliament to slight the fortifications permanently. The two great towers were destroyed with gunpowder and undermining, leaving the dramatic split that still defines the ruin today, with one half of the Hastings Tower standing and the other half collapsed. Parts of the remaining buildings were turned into a new house, and the Hastings family moved their main residence to Donington Hall. In 1819 Sir Walter Scott's novel Ivanhoe made the castle suddenly famous; its owner Francis Rawdon opened the ruins to visitors. The Rawdon family maintained them until 1932, when they could no longer afford to, and the castle passed to the Ministry of Works. Today English Heritage runs it; 15,164 people visited in 2015.