James Louis West - born Lucadou in 1761, the son of a London banker - had served as an officer in the Bengal Army and lived in India before resigning his commission in 1791 to join his father's bank. When his father died in 1802 he inherited a substantial fortune, and around 1818 he began building Wadhurst Castle on the site of the Maplehurst farmhouse, in an elevated position west of the town of Wadhurst overlooking the High Weald countryside. He never married and died in 1819, leaving the estate in trust. In 1826 the trustees swapped it with Captain Aylmer Haly, who had served in the 4th Kings Own Regiment, married Amelia Bannister in 1803 and fathered five children, one of whom - William O'Grady Haly - became a famous general. Haly bought neighbouring farms to enlarge the estate, and renamed it Wadhurst Castle.

Benjamin Harding, a Sussex landowner and justice of the peace, bought the place in 1840. His wife Louisa Le Neve - formerly Thacker - was a personal friend of the novelist Maria Louise Rame, who wrote under the name Ouida and visited Wadhurst Castle to call on her friend. Harding commissioned Edward Buckton Lamb, one of the more inventive Victorian architects, to remodel the building into the castellated mansion that stands today. Lamb's drawings were exhibited at the Royal Academy. He added the Winter Garden, embellished the exterior with turrets, spires and window tracery, and developed the terraced gardens and the entrance lodge that still stands on the B2099. Harding died in 1849 and the house went on the market, where it was bought by Edward Watson-Smyth in 1844 - though the dates suggest the chronology in this account is muddled, with Watson-Smyth most likely arriving shortly after Harding's death.

Edward Watson-Smyth enlarged the castle to the north-east and extended the parkland after his marriage in 1846 to Mary Elizabeth Georgiana Watson Hay. The couple had no children, so on Edward's death in 1869 the estate passed to his brother Robert, a wealthy landowner who had married Louisa Maxfield in 1845 and had six children. The 1881 Census records Robert and Louisa at Wadhurst with three daughters, a butler, a cook, a ladies' maid and three domestic servants. Robert died in 1884 and the castle passed to his son William Douglas Watson-Smyth, an Eton and Cambridge man, big game hunter and twice married. The third generation, George Robert Watson-Smyth, inherited in 1918 - having spent his early twenties as a captain in the 13th Hussars on the Western Front, where in 1915 he was badly wounded in France and lost a leg.

After the First World War the estate was no longer affordable as a private home. Parts were sold off; the northern portion became housing called Castle Walk. In 1927 George Robert Watson-Smyth converted part of the castle into a country club for the adjoining golf course. He sold the property in 1931 to Paul Geoffrey Bankart, who continued the country club operation. Then in 1933 a serious fire damaged a substantial part of the building. The next year the castle was sold to Alfred Charles Matthews, an Australian-born architect, who repaired it and lived there with his wife Alice. He advertised it for sale in 1939, did not sell, and was overtaken by events: the building was requisitioned for war purposes. Matthews returned to Australia, where he died in Tasmania in 1945.

Wadhurst Castle came back on the market in 1947. In 1955 the Fitzgerald family bought it; the estate is now jointly owned by the Fitzgerald and Clough families. The castle and its terraced gardens are both Grade II listed, with the castle running as a licensed wedding venue. The wider story is typical of a particular English type: a name changed, a farmhouse called a castle, a Gothic refacing by a Victorian architect for an ambitious landowner, three generations of family occupation, a war that ended the model, conversion to a clubhouse, near-loss to fire, sale to an immigrant who tried and failed to make it work, then quiet survival into the late 20th and early 21st centuries as a venue. The pile on the ridge above Wadhurst is, in a way, more thoroughly Victorian for never having been a real castle in the first place.