The Norman keep is the visible part of a much longer history. Archaeological excavations carried out in the 1920s suggested that the polygonal Norman tower stands on the foundations of a much older Anglo-Saxon fortification, possibly dating from the seventh century. Henry II is traditionally credited with having ordered the new stone keep built around 1174, replacing whatever earlier earthwork had stood here above the River Stour valley. The location was chosen for the same reasons that have always favoured Chilham - the high ground above the river crossing between Canterbury and Ashford, the commanding view down the valley, the position on the pilgrim and trade route to Dover. When Edward II's entourage stopped here in June 1320 hosted by Baron Badlesmere - the same Badlesmere who would be hanged at Canterbury two years later after rebelling against the king - they were already arriving at a place with five centuries of fortified history behind it.

Sir Dudley Digges completed the Jacobean manor house in 1616. Its plan is unusual - hexagonal, with five angled ranges and the sixth side left open onto the lawns and terraces. It has battlemented parapets, clustered faceted brick chimneys, and corner towers with squared ogee caps. A Victorian tradition attributed the design to Inigo Jones; modern architectural historians don't believe it. Jones's collaborator Nicholas Stone, the master mason who worked with Jones on the Banqueting House in Whitehall, did carve a funerary monument to Lady Digges for the chapel of Chilham Church in the early 1630s, and if any Jones-style influence is visible at the castle it might travel through Stone rather than from Jones directly. Either way, Chilham's hexagonal manor is one of the finer 17th-century country houses in south-east England and commands exceptional views across the Stour valley. The gardens were once thought to have been laid out by the elder John Tradescant, the great Stuart plant collector, but they were extensively redesigned in the 18th century - first under the London banker James Colebrooke and then under Thomas Heron, who in turn brought in Capability Brown to recommend further changes, some of which were actually implemented.

Chilham was bought by James Wildman in 1794 and inherited in 1816 by his son James Beckford Wildman, whose name carried the legacy of his uncle the slave-owning Jamaica planter William Beckford. The Wildman family's wealth came from sugar estates in the West Indies worked by enslaved people. In 1861 James Beckford Wildman was forced to sell Chilham because of falling income from the family's West Indian estates following the abolition of slavery in 1833 and the subsequent compensation system - compensation that went to the former slave-owners, not to the people they had enslaved. The financial unwinding of a British country house's slave-economy foundations is not a tidy story, and Chilham is one of dozens of English estates whose 19th-century crises trace directly to the end of slavery. The estate then passed to Charles Hardy, who commissioned David Brandon to make substantial changes in 1862; plans of these alterations survive in the Victoria and Albert Museum. Hardy's son Charles Stewart Hardy added the fishing lake in the 1860s.

In the early 1920s the South African-born diamond and mining magnate Sir Edmund Davis and his wife Mary - herself a painter - commissioned Herbert Baker, then at the height of his career, to remodel Chilham for them. Their plans, like Brandon's, are preserved at the V&A. The Davises were serious art collectors; in 1938 five of their old-master paintings were stolen from the castle. The two gatehouses to the grounds were added in the early 1920s, replacing very different Victorian ones. During the same decade the Keep itself was the country residence of the British painters and art collectors Charles Ricketts and Charles Haslewood Shannon, the great life-partner artists and stage designers, who lived there until 1925. The 13th Viscount Massereene owned the estate from 1949 until his death in 1992. Then the financier and UKIP political activist Stuart Wheeler bought it and lived there with his three daughters - Sarah, Jacquetta, and Charlotte - until his death in 2020. His wife had died in 2016.

Chilham has put in a long second career as a film location. In 1965 the castle appeared in The Amorous Adventures of Moll Flanders, with Kim Novak in the lead and supporting performances from Leo McKern and Angela Lansbury. In 1985 it played Makepeace's family home in the episode of Dempsey & Makepeace called Cry God For Harry, mostly filmed here. In 1989 the first episode of the ITV game show Interceptor staged a medieval joust on the grounds. In 1994 the castle appeared as Simeon Lee's fictional Gorston Hall in Agatha Christie's Poirot, and again in 2006 as the home of Cardew Pye in Agatha Christie's Marple: The Moving Finger. It has also appeared in Hercule Poirot's Christmas, a season of Married... with Children, and the British drama series Moon and Son. The site now hosts the Chilham Park Equestrian Centre. On 13 April 2021 the castle was put up for sale at £15 million. Across nearly 900 years it has played royal lodge, Jacobean showpiece, slave-economy victim, painters' retreat, magnate's set-piece, family home, and dressing-up box for British television. Few buildings in England have worn so many roles, or worn them so well.