The castle began as a statement. Hubert de Burgh, justiciar of England and one of the most powerful men in the kingdom under King John, started building here around 1230 on a ridge overlooking the broad mouth of the Thames. The position was clever: high enough to see ships approaching from the North Sea, close enough to London to matter, and set above marshland that no army could easily cross. Edward III rebuilt it in the 1360s as part of his coastal defences against the French. Royal masons gave it three new towers and walls thick enough to withstand siege engines. For a brief moment in the late fourteenth century, this was a working military stronghold guarding the eastern approach to the capital.

Then came Richard Rich. The man is one of the more notorious figures of Tudor England, a lawyer whose perjured testimony helped send Thomas More and John Fisher to the block. Edward VI granted him the castle, and Rich saw what it was worth: not as a fortress, but as quarried stone. Between roughly 1551 and 1575, his workmen pulled the castle apart piece by piece, selling the masonry to local builders. What had taken three centuries to construct came down in a single generation. By the time Rich's descendants finished with it, the castle was already what visitors see today - two stumps of tower, fragments of curtain wall, the ghost of a great hall.

When John Constable arrived in 1814, the ruin had passed into something else. The castle was no longer a fortress or a quarry but a meditation - the kind of subject Romantic painters could not leave alone. He made the drawing, set it aside, and only returned to it after his wife Maria died in 1828. The oil painting, Hadleigh Castle, the Mouth of the Thames - Morning after a Stormy Night, was exhibited at the Royal Academy in 1829. The towers stand against scudding clouds. Light breaks over the estuary. It is a painting about grief and weather and the way landscape outlasts the people who love it. The full-scale sketch hangs at the Tate. The finished painting lives in New Haven, at the Yale Center for British Art.

In 1891, William Booth bought the castle and the surrounding land for the Salvation Army. He had an idea: teach the urban poor of London to farm, then ship them to the colonies as agricultural workers. The Hadleigh Farm Colony lasted in various forms into the twentieth century, and the working farm still surrounds the ruin today. The Salvation Army handed the castle to the Ministry of Works in 1948, and English Heritage now manages it as a scheduled monument and Grade I listed building. The slow problem has always been the cliff. The whole ridge is built on London Clay, which slumps unpredictably toward the marshes below. The north-east tower largely collapsed in the 1950s. More serious slippages followed in 1969, 1970, and 2002. The castle is, quite literally, sliding into the view that made it famous.

In 2008, the farm just below the castle was announced as the venue for the mountain biking competition of the 2012 London Olympics. For two summer weeks, the slopes Constable painted carried fluorescent-helmeted cyclists down purpose-built dirt tracks while medieval stone watched from above. The Salvation Army farm still operates as a training centre. The country park around it is a Site of Special Scientific Interest, prized for its invertebrates - shrill carder bees and brown-banded bumble bees that thrive in the rough grassland the farm has preserved. From the surviving tower you can still see what Hubert de Burgh saw: the Thames widening into estuary, container ships now where French galleys once threatened, the same long horizon under the same restless sky.