The story handed down is that Vanbrugh modelled the building on the Bastille, where he had been imprisoned for over four years in his youth on suspicion of espionage. The house was reportedly called Bastille House before it settled into the name Vanbrugh Castle. Whether or not the resemblance is exact, the design choice is striking. Here is a man at the height of English architectural fashion, a Surveyor to the Royal Naval Hospital just down the hill, building his own home to look like the structure that once held him. Most architects display their wealth. Vanbrugh seems to have displayed his memory. The arched corbel table running below the parapet is a detail that appears nowhere else at this date except his own Kings Weston House. He was making his keep in a vocabulary nobody else was using.

Aged 55, Vanbrugh married the 26-year-old Henrietta Maria Yarborough in January 1719. He added a wing to the east side soon after in the same battlemented style, making the house lopsided. It is said to be the first asymmetric house built in Europe since the Renaissance. Whether or not that claim survives careful checking, the point is that buildings of his era simply did not look like this. Houses were meant to be balanced, classical, predictable. The Vanbrugh family arrangement was different. He built the Nunnery for his brother Philip a little way off, Mince-pie House for his brother Charles, and later added two white towers of patented white brick, perhaps for his two sons. Visitors approaching from Greenwich passed all of it through crenelated gateways, a small private medieval village invented in 1719.

Vanbrugh died in 1726. The house passed through a succession of owners: a Holford couple, then in 1838 the engineer Dr Laurence Holker Potts who set up a laboratory inside to design equipment for treating spinal injuries. The novelist Mary Anna Needell was born in the house in 1830. In 1907 the oil merchant Alexander Duckham bought it as his London home and added a weathervane shaped like a duck in flight, a small joke at his own surname that still gestures over the roofline. Duckham donated the castle to the RAF Benevolent Fund in 1920 to be used as a school for the children of RAF personnel killed in service. The school choir, by an arrangement that almost nobody alive remembers, sang services in the Royal Naval College chapel on the far side of Greenwich Park.

The boys were taught in the only schoolroom by Captain Slimming, then moved up to the John Roan School nearby. The Wakefield Wing was added in 1938. When war broke out in 1939, they were evacuated to Rye and Bexhill, but once it became clear that German invasion would most likely come ashore in exactly that part of the coast, the school moved again to Wales. The choral conductor James William Webb-Jones served as headmaster from 1951 to 1955. Peter Stanley Lyons, simultaneously Director of Music at the Royal Naval College in Greenwich, ran music at the castle from 1950 to 1954. In 1951 the building was given Grade I listing. The school finally moved out in 1976, relocating to the Duke of Kent School in Surrey, and four people bought the castle for £100,000 and turned it into four private flats.

Stand on the lead roof and the view stretches west past the columns of the Old Royal Naval College down to the bend in the Thames, with the towers of Westminster visible on a clear day. This was a view Vanbrugh had professional reasons to care about. As Surveyor to the Royal Naval Hospital from 1716 he was responsible for what is now the Old Royal Naval College, and his roof terrace was effectively his quality-control deck. In the 1980s, scenes from the Bob Hoskins film Mona Lisa were shot on the circular driveway out front. The duck weathervane is still there. So is the conical copper roof, oxidised green now, holding the line of a small private fortress invented by one of the most famous Baroque architects in England to remind himself he had once been locked up.