The Temple of Claudius rose between AD 49 and AD 60, a marble-fronted shrine raised on a podium thirty-two metres long, the imperial cult given its grandest expression in the British province. Boudica's army burned the city of Camulodunum to the ground in AD 60 or 61, but the temple's massive stone base survived. A thousand years passed. The site was holy ground, then ruined ground, then Saxon ground with a small chapel beside it. When Norman builders began work in the 1070s, they did the medieval equivalent of finding a foundation slab and pouring a building on top: they cleared what remained of Claudius's temple and laid their keep walls directly along its edges. The result is a keep whose footprint - one hundred and fifty-two by one hundred and twelve feet - is not really a Norman decision at all. It is a Roman decision, made under Nero, reused under William.

Look at the walls and the doubled history is visible in the masonry. Essex has no natural stone, so the Normans quarried the ruins around them: courses of grey septaria from the local clay, Roman brick laid in herringbone patterns, ashlar dressings of Barnack limestone hauled in from Northamptonshire. Every brick in the lower walls was made by a Roman hand. The design has been compared to the White Tower in London and to the Norman keep at Ivry-la-Bataille in Normandy - both attributed to the bishop-architect Gundulf of Rochester, though that connection at Colchester rests more on resemblance than record. A royal charter of 1101 confirms the castle had been built by then, granted by Henry I to Eudo Dapifer, the king's steward. The Colchester Chronicle, written two centuries later, dates the work to 1076. The medieval mind liked symmetry. William's barons compared themselves to the Roman Senate. Building a Norman fortress on a temple of Claudius was a statement: we are the heirs.

The castle's military life was brief. In 1216, during the war between King John and the rebel barons, French troops held the keep against a three-month siege before being starved into surrender. After that the building drifted from fortress into the slower trouble of disuse. By the seventeenth century it was a ruin with tenants. In 1683 an ironmonger named John Wheely was licensed to take it down for the value of the stone. He attacked the upper walls with screws and gunpowder, blowing off the top storeys, until the demolition stopped paying. He gave up. What he left behind is the squat, immensely thick lower keep that stands today - shorter than the original, but still extraordinary. The exact height of the castle as the Normans finished it remains debated. Wheely took the answer with him.

Once it stopped being a castle, it became a prison, and the prison stories accumulated. In 1645 Matthew Hopkins, the self-appointed Witchfinder General, used cells here to interrogate women accused of witchcraft during the panic he stoked across East Anglia. In 1648, after the Royalist garrison of Colchester surrendered ending an eleven-week siege, Sir Charles Lucas and Sir George Lisle were taken behind the castle and shot by parliamentary order. A small obelisk marks where they fell. Local tradition still insists that grass refuses to grow on the spot. In 1656 the Quaker preacher James Parnell - imprisoned for blasphemy at the age of nineteen - died in the castle's dungeons, weak from cold and a starvation fast. He became one of the early Quaker martyrs.

In 1727 the keep was bought as a gift for Sarah Gray, wife of the MP for Colchester. The Grays restored what remained, leasing portions as a corn merchant's store and a county jail, and laying out a garden around the building. By 1860 the castle had become a museum, and in 1922 the Borough Council bought it outright. Today the keep houses one of the most important Roman collections in Britain. Visitors who pay to descend below the museum floor can walk on the Roman vaults of the temple of Claudius itself - the foundation that decided everything. You stand on a Roman emperor's shrine, beneath a Norman tower, inside an English town that calls itself Britain's oldest recorded settlement. Few places in the country wear their layers so plainly.