The Astley family held the manor from the twelfth century. In 1266 the Crown granted them a licence to crenellate, the medieval royal permission to fortify a private house, but the building was always more manor than castle, a fortified house with a moat and a gatehouse rather than a true defensive structure. The line ran out with Sir William Astley, who died in 1420 leaving everything to his daughter Joan, who had married into the Greys of Ruthyn five years earlier. The Greys were marcher lords, a dynasty of border barons who had held the Welsh frontier for the Crown for generations. With the Astley inheritance they acquired a comfortable manor deep in safe English country. Their descendants rebuilt it in 1555 as a two-storey rectangular house with attics hidden behind embattled parapets. Most of what now stands ruined comes from that Tudor rebuild.

Henry Grey, Duke of Suffolk, grew up at Astley. He married Frances Brandon, the granddaughter of Henry VII through his sister Mary. Their daughter Jane was raised partly here. When Edward VI lay dying in 1553 a faction at court, terrified of his Catholic half-sister Mary, prevailed on him to disinherit her in favour of Jane. Jane was sixteen. She did not want to be queen and said so. They put a crown on her anyway. Nine days later Mary entered London, the people of England behind her, and the coup collapsed. Jane and her young husband Guildford Dudley were sent to the Tower. Suffolk himself fled to Astley and hid in a hollow tree on his own grounds, where, the story goes, he was betrayed by a park keeper. Jane was beheaded on Tower Green on 12 February 1554. Her father followed eleven days later. The Crown seized Astley, slighted the house, and sold it to a man named Edward Chamberlain who patched it up.

During the English Civil War, Astley became a Parliamentary garrison, one of a network of small troublesome strongholds in the West Midlands that the Royalists never quite got around to subduing. The muster lists submitted to the committee at Warwick name the local commander as Captain Hunt and his lieutenant, Goodere Hunt. The historian Ann Hughes has noted that Hunt was, in the words of Royalist propaganda broadsheets, an illiterate shoemaker, prosecuted in 1647 for requisitioning a gentleman's horse. The Royalists pamphleteered against these rebel towns, where shopkeepers and cobblers were said to be running garrisons. The muster at Astley on 9 July 1645 listed 79 officers and 462 cavalry under Major Hawkswell. Most of the troopers were quartered in the village. After the war the property settled into the long quiet of a secondary house. In 1674 it was bought by the Newdigates of Arbury Hall, the family that the novelist George Eliot would later write about in her Scenes of Clerical Life.

The Newdigates leased Astley out during the twentieth century. In the early 1960s it became a country hotel. On a night in 1978 a fire broke out and the building was almost entirely destroyed. For more than twenty years it stood as a roofless ruin on English Heritage's Buildings at Risk register, while owners and trusts argued over what to do. In the early 2000s the Landmark Trust took on the project together with English Heritage and Witherford Watson Mann Architects, a small London practice. Their solution was unprecedented. Rather than restore the ruin or replace it, they inserted a sharply modern brick-and-timber house directly into the old walls, leaving the most damaged sections as open courtyards roofed only by the sky. Visitors stay there now as a Landmark Trust holiday let. In 2013 the building won the Royal Institute of British Architects Stirling Prize, Britain's highest architectural award. The judges called it an exceptional example of the blending of an ancient monument with modern architecture. The three queens get a paragraph in the guest information. The shoemaker captain does not.