Walter May - father of the tower's builder - commissioned the original house in the late 1780s, replacing the medieval manor house of Hadlow Court Lodge. The architect, J. Dugdale, designed it in the fashionable Strawberry Hill Gothic style pioneered by Horace Walpole - the playful, picturesque version of medieval architecture that English gentlemen had taken up since the 1750s. The house was extended over the following decades. Walter Barton May inherited it in 1823 along with the estate, and a second inheritance in 1832 from his wife's family gave him the means to build big. The 40-foot octagonal lantern that crowns the tower was added in 1840; a second smaller tower was added in 1852 and dismantled in 1905. Walter Barton May died in 1858 and the estate was sold.

Robert Rodger, a JP and High Sheriff of Kent, bought the estate in 1865. He died in 1882, his son William Wallace Rodger-Cunliffe in 1888. In 1900 the castle was sold to Benjamin Scott Foster MacGeagh, a retired trader who also became a JP and High Sheriff of Kent and died in 1907. His son Dr Thomas Edwin Foster MacGeagh was a Harley Street physician who commuted to London by driving his carriage to Tonbridge and catching the train - making him an unusually early example of the long-distance medical commuter. He sold the estate in 1919 to Henry Thomas Pearson, whose farming family occupied it until 1946. The castle itself was demolished in 1951, but the painter Bernard Hailstone saved the servants' quarters, several stables and the Coach House. The entrance gateway and lodges still stand on the High Street - a heavy Gothic presence on a Kentish village street.

Hadlow Tower was already a listed building when the house came down, having been listed on 17 April 1951. Without the rest of the building to anchor it, the tower became all the more conspicuous. It is built of stuccoed brick rising in eight tall storeys to the octagonal lantern; the whole thing is decorated with the pointed-arch windows, crocketed pinnacles and ornamental stringcourses that the Gothic Revival demanded. The interior was always more impressive than functional - more a viewing platform than a residence, more a statement than a building. Throughout the 20th century the tower sat above the village, slowly deteriorating. The Great Storm of 1987 caused serious damage, and the lantern at the very top was removed in 1996 because it was no longer safe. The condition then worsened rapidly. By the mid-2000s repair was estimated at 4 million pounds.

In July 2006 Tonbridge and Malling Borough Council announced its intent to issue a compulsory purchase order to save the tower. The CPO was confirmed by the Secretary of State for Culture, Media and Sport in March 2008. The council planned to take possession of the tower and transfer it to the Vivat Trust, a heritage charity specialising in restoring buildings at risk and operating them as holiday lets to fund maintenance. In January 2011 the compulsory purchase was completed; the council sold the tower to the Vivat Trust for 1 pound. Restoration began in February 2011, funded with 2 million pounds from the Heritage Lottery Fund as part of a 4-million-pound total. The lantern was rebuilt. The interior was rebuilt as holiday accommodation with public exhibition space on the ground floor. The work was completed in February 2013, making Hadlow Tower the tallest folly in Britain.

The Vivat Trust subsequently collapsed financially, and in May 2016 the tower went on the market. In 2017 it was sold to a private individual for 425,000 pounds. It was relisted in 2020 with Sotheby's International Realty at 1.475 million pounds. The exhibition centre on the ground floor was open Thursdays from May to October when the Save Hadlow Tower Action Group, known as SHTAG, organised access. The legend that May built the tower to spy on his wife remains the most repeated story about the place; the more verifiable version - of a wealthy Gothic enthusiast in 1830s Kent making a single extravagant gesture - is more accurate but less satisfying. Either way, the tower stands. Twenty-six years after its lantern was removed for safety, it carries one again.