The Downs anchorage off this stretch of coast was one of the most important in England. The Goodwin Sands sheltered ships from the open Channel; Deal Beach beyond gave any invader an easy landing point. So Walmer, Deal, and Sandown went up side by side, connected by a 2.5-mile defensive ditch and a chain of earthwork forts called the Great Turf, the Little Turf Bulwark, the Great White Bulwark of Clay, and the Walmer Bulwark. By 1597, Walmer's artillery comprised a cannon, a culverin, five demi-culverins, a saker, a minion, and a falcon - a complete arsenal of the late Tudor age. The first garrison was a captain, two lieutenants, two porters, ten gunners, and three soldiers, costing the Crown £174 a year.

Walmer's serious moment of action came during the Second English Civil War of 1648. Parliament held the castle at the start, but when Vice-Admiral William Batten encouraged the fleet to switch sides and join the Royalists, Walmer and Deal Castle declared for King Charles I, shortly after Sandown did the same. With both fortresses and the navy now Royalist, Parliament feared a foreign landing. Colonel Nathaniel Rich marched on the Downs after Parliament's victory at Maidstone, and Walmer was the first to fall - besieged through June and into July, surrendering on 12 July 1648. The castle was badly damaged. Repairs were estimated at £500, a substantial sum for the period. By the end of the century, Walmer's military usefulness was already fading. When the 1st Duke of Dorset became Lord Warden of the Cinque Ports in 1708 and found Dover Castle uninhabitable, he simply moved into Walmer instead - and the long transformation from fortress to home began.

William Pitt the Younger took the post in 1792, in debt and grateful for the £3,000 salary George III meant as a quiet supplement to his income. Pitt made it his main residence by 1803. His remarkable niece Lady Hester Stanhope joined him from 1803 to 1806 and, together with Pitt, transformed the castle gardens from a simple kitchen garden into a landscaped sweep of ornamental enclosures - she famously enlisted the Dover militia to help with the planting. Pitt formed a volunteer cavalry unit and a fleet of 35 armed fishing luggers, reviewing them from the castle as French invasion fears swelled. The Duke of Wellington took over as Lord Warden in 1829 and called Walmer the most charming marine residence. Queen Victoria visited him twice, once as princess and once as queen. Wellington died here on 14 September 1852, in his small bedroom with its original camp bed and chair - which are still in the room today. His embalmed body lay in state until 10 November. When the room opened to the public for the final two days, around 9,000 people came to see him.

Lord Granville took the castle in 1865 and spent years quietly reassembling the furniture and objects Pitt and Wellington had once used. The diplomat Baron de Malortie praised the homely atmosphere - the family and guests gathering in the drawing room after breakfast, Granville answering government correspondence amid the daily life of the household. The businessman and politician William Smith arrived as Lord Warden in 1891 and died at Walmer during his first visit that October. Smith had proposed protecting the historic contents legally so successive Lord Wardens could not strip the place; an Indenture of Heirlooms Bill duly followed. Lord Curzon's wife Mary died in 1906, and Curzon blamed Walmer's cold and uncomfortable condition for her death. By 1904, the War Office had concluded the castle had no military value and offered it to the Office of Works on condition of receiving £2,400 for repairs.

Twentieth-century Lord Wardens have been a varied list: Winston Churchill, Robert Menzies (the Australian prime minister), and Queen Elizabeth The Queen Mother. They made only intermittent use of the place. In 1997, for the Queen Mother's 95th birthday, garden designer Penelope Hobhouse laid out a new garden in her honour - a deliberately old-fashioned, scented English garden tucked among the older Pitt and Granville plantings. English Heritage runs the castle today, displaying objects from across its successive lives: Pitt's chairs, Wellington's bed, Queen Mother memorabilia, Tudor cannon. You can stand on the parapet where Tudor gunners would have watched for French sails, look out across the Downs to the Goodwin Sands, and find it hard to remember that this small, comfortable house was ever designed to kill people.