England in the 1140s had no functioning king. The civil war known as the Anarchy - the contest between King Stephen and Empress Matilda for the throne - allowed magnates everywhere to build castles without royal permission. These were called adulterine castles, illegitimate fortifications thrown up to enforce local power. William de Warenne, 2nd Earl of Surrey, built one at Allington on a bend in the Medway - a moated mound, probably a motte and bailey, controlling river traffic between Maidstone and the Thames. When Henry II restored royal authority, he ordered the demolition of unauthorised castles. Allington was slighted in 1174, one of at least twenty-one castles destroyed on Henry's orders. The site reverted to an unfortified manor house. The river kept flowing. The Medway forgot what had stood there.

The current castle dates from between 1279 and 1299. Stephen de Pencester, Lord Warden of the Cinque Ports under Edward I, received a licence to crenellate the existing manor house - the royal permission required to build a serious castle. He built six round towers along the curtain wall, integrated the old manor into the new structure, and made extensive use of brick. The brickwork was unusually early for England - Stephen had interests in an Essex brickyard, and Allington shows the technology in one of its first English applications. The castle passed by marriage to the Cobham family in the fourteenth century. By 1398, documentary sources describe it as in a very bad condition. The Cobhams held it without much affection for nearly a century. It deteriorated further.

Sir Henry Wyatt bought Allington in 1492. He was a Lancastrian who had backed Henry Tudor before Bosworth and ridden out the Yorkist regime in part by being imprisoned, tortured, and possibly fed by his cat - a story the family told and others embellished. Wyatt undertook major alterations to Allington, dividing the courtyard with a new two-storey building that contained one of the first long galleries in England. Henry VII visited him here. So did Henry VIII, three times - in 1527, 1530, and 1536. Cardinal Wolsey came in 1527. Katherine Parr came in 1544. The Wyatts hosted them all. Sir Thomas Wyatt the Elder, the poet who introduced the sonnet to English literature and conducted his famously dangerous flirtation with Anne Boleyn, was Sir Henry's son. He had grown up here. His own son, Sir Thomas Wyatt the Younger, lost everything for the family in 1554.

The Crown granted Allington in 1568 to John Astley, Master of the Jewel House to Queen Elizabeth, but Astley did not live there. Around 1600 two farm houses were built within the castle grounds while the main structure decayed. A serious fire in the second half of the sixteenth century destroyed most of the Great Hall and the north-east wing. An early seventeenth-century lessee named John Best made matters worse, pulling down the battlements and adding a half-timbered gabled second storey to replace the fire-damaged sections. Sir Robert Marsham, the 2nd Baron Romney, bought the castle in 1720 but did not live there. The Long Gallery burned in another fire in the early nineteenth century. Charles Marsham, 5th Earl of Romney, nearly demolished what was left a few decades later - he was dissuaded only by the Rector of the nearby Church of St Lawrence. By 1900, the castle was a ruin.

Sir Martin Conway - mountaineer, Alpine explorer, polar adventurer, art historian, and eventually Member of Parliament - bought Allington in 1905. His wife was Katrina, an American heiress whose fortune funded what became a decades-long restoration. Conway brought in the architect Philip Tilden and they worked through the 1910s and 1920s rebuilding what had been lost. The towers were repaired. The roofs went back on. The interiors were furnished with Conway's collection of medieval art and early printed books. He was raised to the peerage as Baron Conway of Allington in 1931. After Conway's death, the castle was occupied for nearly fifty years by a community of Carmelite friars and nuns, who used it as a religious retreat. In 1999 it returned to private hands. Sir Robert Worcester, the founder of the MORI polling company, made it his home after 1999. There is no public access except via the Kentish Lady river boat from Maidstone.