Lullingstone is in the Domesday Book of 1086, which is to say it was already old when the Normans counted it. In 1279 the manor was bought by Gregory de Rokesley, who served eight terms as Lord Mayor of London and was reckoned the richest man of his generation in the City. The Rokesleys held it for generations, then it passed to the Peche family. In 1497, Sir John Peche, High Sheriff of Kent and later joint Lord Deputy of Calais, began building the present manor house. The outer gatehouse he raised, possibly on earlier foundations, is one of the first buildings in England constructed entirely of brick: a startling material choice in a country that still mostly built in stone and oak. The gatehouse still stands, Grade I listed, at the start of the long drive.

Henry VIII visited Lullingstone with Anne Boleyn. They came to hunt: the 120-acre park around the castle was a fenced deer park, and the house functioned partly as a hunting lodge. In 1543 the estate passed by marriage to Sir Percyvall Hart, chief steward and knight harbinger to Henry VIII and then to all three of his successors: Edward VI, Mary I, and Elizabeth I. Sir Percyvall built the main house between 1543 and 1580, originally calling it Lullingstone House. His grandson Sir Percival sent fish and poultry up to King James at Theobalds in May 1603. A later Percival Hart, High Sheriff of Kent in 1706, remodelled the building in the Queen Anne style and renamed it Lullingstone Castle, which was a deliberate piece of self-mythologising on a house that had never functioned as a fortress.

In the early twentieth century Zoe Dyke created the Lullingstone Silk Farm on the estate, raising silkworms whose threads were spun into silk of exceptional quality. Queen Mary visited and was sufficiently impressed that the farm was commissioned to produce silk for the coronation robe of King George VI in 1937. The silkworms went on working through royal occasions: silk from Lullingstone was used in the wedding dress of Princess Elizabeth, the future Elizabeth II, in 1947, and again in the wedding dress of Lady Diana Spencer in 1981. The farm has since moved away from the estate, but the silkworms put Lullingstone into the lining of three reigns. The castle itself was briefly out of the family during the twentieth century: sold in 1934 to the Kemp Town Brewery, resold to Kent County Council in 1938, and occupied by the Army during the Second World War. The house, however, remained with the Hart Dykes.

Tom Hart Dyke is the twentieth Hart Dyke to own Lullingstone. In 2000, on a plant-hunting expedition in the Darien Gap looking for rare orchids, he was kidnapped along with a friend by guerrillas and held captive for nine months. He spent that time, by his own account, sketching out the design of a garden in the shape of a world map: each continent planted with native species from that part of the world. When he was finally released and made it home, he converted the castle's walled herb garden, designed in an earlier era by the garden writer Eleanour Sinclair Rohde, into the World Garden of Plants. The conversion was filmed for the BBC2 series Save Lullingstone Castle and a follow-up, Return to Lullingstone Castle, in spring 2007. The garden now contains plants from across the world, planted geographically: visitors literally walk through continents. A wire baobab tree stands sentinel where Africa would be.

The drive in passes through the gates of the Tudor gatehouse and reveals the house in red brick across a lawn. To the side stands St Botolph's Church, of Norman foundation or older, with much later restoration, its small churchyard punctuated by ancient yews. The 120-acre estate now forms Lullingstone Country Park, with public footpaths winding through some of the oldest oak trees in Britain. Queen Anne's bathhouse hides somewhere in the grounds, along with an 18th-century icehouse. The Roman villa, England's most famous, is just down the lane. The castle and the World Garden are open from April through September, when you can buy a ticket and walk straight into 500 years of family history, and then into a contemporary plant collector's living atlas. In 2011 the meerkats Sergei and Aleksandr filmed a Compare the Market advert here, which is how a lot of younger British viewers know the place.