Severndroog Castle is sixty-three feet tall, triangular in plan, with a hexagonal turret at each corner and battlements crowning the whole composition. It looks like a small medieval keep, which is exactly what Lady James and Richard Jupp wanted. But it has never functioned as a castle. It is a folly, a building intended to add picturesque drama to the landscape, and it gave Lady James a monument to her husband that could be seen for miles. Designated a Grade II* listed building in 1954, the castle stands in Castle Wood, on Shooter's Hill in the Royal Borough of Greenwich, surrounded by ancient sweet chestnuts and oaks. From the rooftop viewing platform, on a clear day, you can see features in seven different counties: Kent and Essex obviously, but on the right sort of London afternoon, also Surrey and Sussex and Hertfordshire and Buckinghamshire and Berkshire.

The folly turned out to be more than a memorial. In the 1780s, General William Roy was conducting his great trigonometric survey to connect the Royal Greenwich Observatory with the Paris Observatory, the project that would lead directly to the founding of the Ordnance Survey in 1791. Roy needed sightlines, and Severndroog stood high enough above the Thames basin to be a critical station. A massive 36-inch theodolite (now in London's Science Museum) was temporarily hoisted onto the roof. The angles Roy measured from this platform, between 1784 and 1790, helped fix the geometry of southern England and France in relation to each other. In 1848 the Royal Engineers used the castle again for their survey of London. The folly built as a tribute to one man's victory across the sea ended up being one of the instruments by which Britain measured itself.

After Lady James died in 1798, Severndroog passed through various owners: John Blades, a former Sheriff of London; a Mr Barlow, a ship owner who built nearby Castle Wood House; Thomas Jackson, a railway and docks contractor of Eltham Park. In 1922, the London County Council bought the tower, and it became a local visitor attraction with a ground-floor tearoom serving Sunday-afternoon cake to walkers from across south-east London. When the GLC was abolished in 1986, responsibility passed to Greenwich Council. By 1988, however, the council could no longer afford the upkeep. The doors were locked. The windows were boarded up. The folly sat empty for sixteen years while suburban London thickened around the woods at its feet.

In 2002 a community group called the Severndroog Castle Building Preservation Trust was formed. In 2004 the building featured in the BBC television series Restoration, presented by Griff Rhys Jones, Ptolemy Dean, and Marianne Suhr. The programme followed buildings that were at risk and rallied public support for their rescue. Severndroog won enough attention and funding for restoration work to begin. The roof was repaired, the interior renovated, the tearoom rebuilt. In July 2014, after twenty-five years closed, Severndroog Castle reopened to the public. Volunteers staff the entrance and the ground-floor cafe. Visitors climb the spiral staircase to the rooftop platform, the same one where Roy's theodolite once turned slowly through its measurements. The castle can be hired for weddings, which Lady James would probably have appreciated.

Shooter's Hill is on the way to almost nowhere now, a quiet wooded ridge between Blackheath and Bexleyheath that the A2 used to pass over before the motorway replaced it. The Green Chain Walk and the Capital Ring long-distance paths both pass the castle, threading through Eltham Common and Castle Wood from Eltham to the Thames-side at Thamesmead. Walking up through the chestnuts you smell wood smoke and damp leaves before you see the tower. Then it rises through the trees, pink-grey stone, with its turrets and battlements, looking implausible. A memorial built by a widow for a sailor who had razed a fort in India ended up helping to map London, then nearly fell down, then was saved by a TV show. Sometimes follies have the longest lives.