Eynsford was a strategic point long before the Normans arrived. The manor lay on a crossing of the River Darent, and in 970 it was acquired by Christ Church, Canterbury. In the early 11th century a stone building was raised on an artificial terrace - it's possible there were earlier stone buildings on the same site. A ditch and rampart surrounded it, each about 5 metres wide and up to 3 metres deep and high. This was a burh - an Anglo-Saxon fortified manor - and like some other burhs may have had an entrance tower called a burh-geat, symbolising the status of its owner. After the Norman invasion of 1066, Odo, the Bishop of Bayeux and the new Earl of Kent, probably acquired the manor; Canterbury's former hold on it had been lost. The Saxon burh appears to have continued in use under the new regime. But Odo's lands and the Archbishop's bordered each other, and Lanfranc remained concerned about the threat his powerful neighbour posed. So Lanfranc authorised William de Eynsford to upgrade the defences. A proper Norman castle rose on the Saxon foundations.

William de Eynsford's family held the castle on behalf of the archbishop until 1261 - through six descendants, all named William and conveniently distinguished by historians as William de Eynsford II through to VII. Around 1130 a new hall was built, replacing the older Anglo-Saxon buildings, and the defences were strengthened with a new gatehouse and a raised curtain wall. William I retired to become a monk in the 1130s. His son William II had passed control to William III by the late 1140s. William III became embroiled in the great dispute of his age - between Henry II and Archbishop Thomas Becket. He sided with the rebels against King John during the later civil war, was captured at Rochester Castle, and lost his estates to the Crown. Both William VI and his son William VII died young, ending the family line in 1261. An inquiry divided the de Eynsford lands between William Heringaud, a powerful east Kent landowner, and Nicholas de Criol, a smaller Kentish landowner. Both were descended from William V. The castle, unoccupied, was seized by a royal official, then passed to the royal judge Ralph de Sandwich. Around 1300 someone moved back in - probably a widow of one of the Williams, or a castle bailiff.

In 1307, Ralph de Sandwich sold his half of the castle to another royal judge, William Inge. Inge then tried to exercise his rights, which brought him into conflict with Nicholas's grandson and heir - also named Nicholas de Criol. The dispute escalated, and in 1312 Inge brought a legal case alleging that Nicholas and two of his brothers had attacked several of his properties in the area. The de Criols, Inge claimed, had broken down the doors and windows at Eynsford, ransacked it, and released his livestock. The case was probably accurate. After 1312 the castle was never reoccupied. The defensive technology was already obsolete; the legal owners couldn't agree on anything; the surrounding country no longer needed a strongpoint on the Darent. Eynsford fell into ruin. By the 18th century it had been put to ignominious use - the castle held hunting kennels and stables. The Eynsford huntsmen kept their dogs and horses within the curtain walls that had once protected the Archbishop's interests.

In 1835 the castle ceased to be used as kennels and the architect Edward Cresy was employed to remove the more recent modifications and survey the medieval structure. Cresy removed up to 9 feet of accumulated debris - centuries of kennel dirt, stable straw, and falling stone. Then the castle was neglected again, and parts of the north-western wall collapsed in 1872. In 1897, a local landowner named E. D. Till leased the castle and undertook restoration work. She agreed a preservation plan, bought the freehold, and in 1937 transferred ownership to the Society for the Protection of Ancient Buildings. The leasehold ran out, and the castle passed into the guardianship of the Ministry of Works in 1948. Excavations in the early 1980s by Valerie Horsman led to a major reassessment of the fortification's earlier history - revealing the Anglo-Saxon burh beneath the Norman walls. In the 21st century, English Heritage runs the site, open to visitors, protected as an ancient monument. The current timber bridge across the moat dates from the 1960s, but the inner bailey survives as a low earth terrace forming an irregular polygon up to 200 feet across, protected by the original curtain wall. In 2018 a tabloid story claimed a ghostly black monk had been photographed at the castle. Kenny Biddle, writing in Skeptical Inquirer, pointed out that instead of using Occam's Razor and looking for the most obvious solution - a tourist - the original reporters had reached straight for the ghost. The castle has stood there long enough that it can probably afford the occasional embellishment.