The castle's first major test came during the civil war known as the Anarchy, the conflict between King Stephen and Empress Matilda that tore England apart in the 1140s. In 1143, Geoffrey de Mandeville — a supporter of Matilda — attacked Cambridge, raiding the town and temporarily seizing the castle before Stephen could restore control. Stephen then ordered a supporting fort built at Burwell to shore up Cambridge's defences. Geoffrey died attacking Burwell Castle the following year, which resolved the immediate threat. Cambridge Castle settled into its administrative role: holding the sheriff's court, storing records, watching the northern road. King John expanded it ahead of the First Barons' War; Edward I rebuilt it comprehensively, adding circular towers at each corner, a gatehouse, a barbican, and a circular stone keep on the motte.

Edward I's castle was formidable. It did not stay that way. The late medieval period saw the castle's stonework gradually given away — or sold, or granted — for other purposes. Mary I handed over stone for a mansion at Sawston in the sixteenth century. Emmanuel College and Magdalene College received grants of building material. By 1604, a contemporary account described the surrounding walls as "rased and utterly ruinated". Only the gatehouse, pressed into service as a jail, and the keep itself remained intact. The Cambridge colleges had cannibalized their own castle. What had been built to dominate the city was quietly absorbed into it.

When the English Civil War broke out, Parliament needed Cambridge Castle defensible again. Oliver Cromwell ordered emergency work: two new earthwork bastions were added, a brick barracks was constructed in the old bailey. In 1643 the governor of Cambridge reported with satisfaction that "our town and castle are now very strongly fortified... with breastworks and bulwarks." The castle saw no fighting during the war. In 1647, Parliament ordered the remaining fortifications slighted — deliberately damaged beyond further military use. The castle's martial life was over. It became a jail again, then a more elaborate county prison built by G. Byfield between 1807 and 1811 with an innovative octagonal structure influenced by prison reformer John Howard. That prison was demolished in 1932.

The Shire Hall, headquarters of Cambridgeshire County Council, was built in 1932 using the bricks from the demolished prison — which had itself been built partly from the castle's own stones. Cambridge has a long tradition of turning one era's structures into the next era's materials. What survives of the original castle is the motte: a 10-metre grassy mound that stands on the highest point in the city. The site is open to the public daily, free of charge. From the top, you can see the spire of All Saints' Church, the tower of St John's College chapel, and the long roofline of King's College Chapel. It is the best view in Cambridge, and it is all that is left of a castle that was built to make this city afraid.