In 1275 a member of the Caverswall family asked Edward I for permission to crenellate his manor house, and Edward, in the careful fashion of medieval kings managing private fortifications, said yes. The resulting castle was approximately rectangular, with four angle towers, a keep, and curtain walls enclosing the whole. It was a proper medieval moated fortress, the kind built when minor lords wanted protection from local rivals as much as from foreign armies. The family who built it gave their name to the village and held the castle through the 14th century. By the 15th century the property had passed to the Montgomerys, three of whom served as High Sheriff of Staffordshire. By the end of the 16th century, the medieval castle was "much decayed and neglected," and what stood was barely defensible against weather, let alone enemies.

In 1615 Matthew Cradock, a Stafford wool merchant who had served as the first mayor of Stafford the year before and would become Member of Parliament for the town in 1621, bought the ruined castle. He rebuilt within the medieval walls in a thoroughly Jacobean style, the resulting mansion attributed by tradition to Robert Smythson or his son John, the great country-house architects of the period. The Cradock male line failed, and in 1655 the estate was sold to William Joliffe, who served as High Sheriff in 1663 but was eventually forced to sell as well. From there Caverswall passed through a long sequence of owners. In 1811 a Benedictine community of nuns took possession and used the castle as a convent until 1853, when they sold to Sir Perceval Radcliffe and relocated to Oulton Abbey. In the 1880s the Wedgwood family, whose pottery business defined nearby Stoke-on-Trent, rented it for a time.

After the Wedgwoods came W.E. Bowers, who purchased it in 1891 and spent decades on improvements. His successor W.A. Bowers sold it in 1933 to the Sisters of the Holy Ghost. In 1965 the sisters sold to another order, the Daughters of the House of Mary. When they left in 1977 the castle was broken up and sold in lots, the start of the modern era of attempted commercial use. From the late 1970s onward, application after application has been refused. A small hotel: no. Photography sessions: no. School and historic-interest groups: no. The reasons cited usually involve traffic on the narrow lanes around the village and concerns about the building's setting. The result has been a strange standstill: a Grade I listed castle of 2,030 square yards, in private hands, technically a working home, but legally prohibited from doing the things castles in England usually do.

What remains at Caverswall is unusual: a complete early-Jacobean mansion sitting like an architectural fossil inside genuinely medieval walls. The 13th-century curtain wall still encloses the site. The angle towers still stand where the Caverswall family raised them. The 1615 mansion within is itself now four centuries old, longer-lived than the medieval castle it replaced. For visitors driving past on the lanes that twist out of Caverswall village, the view is largely unchanged from what wedding guests, nuns, Wedgwoods, and frustrated tycoons have all seen. The dungeon, the eighteen bedrooms, the long history of religious orders quietly maintaining the place: all of it sits behind walls that no school group is permitted to enter. The 2021 sale changed the ownership. It did not change the planning file.