Sutton Valence Castle was built in the second half of the 12th century, most likely by Baldwin of Bethune, the Count of Aumale - or possibly by Baldwin's father-in-law, William le Gros, 1st Earl of Albemarle. Both were powerful Anglo-Norman magnates. The location was then known as Sudtone or Town Sutton, and the local fortification was meant to control the strategic route between Maidstone, the inland market town on the Medway, and the south-coast ports of Rye and Old Winchelsea. The castle keep itself - the central residential and defensive tower - was constructed around 1200, probably as an upgrade to the earlier 12th-century works. The castle proper probably had an inner bailey, an outer bailey, and a protective barbican on the eastern approach. Today, only earthworks mark the baileys; the keep alone survives in any architectural form, and only as a stump of its original three storeys.

In 1203, Baldwin gave the castle to his daughter Alicia on her marriage to William Marshal - whose father, William Marshal the Elder, was one of the most celebrated knights of medieval Europe and had served as regent of England for the young Henry III. William Marshal the Younger later remarried, and after his death the castle passed to his second wife Eleanor. Eleanor then married Simon de Montfort, 6th Earl of Leicester, the firebrand baron whose rebellion against Henry III briefly remade English government. De Montfort summoned the parliament of 1265, the first in English history to include representatives of the commons. Months later, in August 1265, he was killed at the Battle of Evesham by royal forces under the future Edward I. Eleanor lost ownership of Sutton Valence Castle in the political collapse that followed. Henry III gave the castle to his own half-brother, William de Valence, in 1265 - and from him the castle has taken its modern name. The Valences were of the king's blood through Henry III's mother, Isabella of Angouleme, and her second husband Hugh X of Lusignan.

Aymer de Valence, William's son, inherited the castle in 1307. Aymer was one of the major political figures of Edward II's troubled reign - a moderate baron who tried to broker peace between the king and his opponents, and who is now buried in Westminster Abbey under one of its most beautiful early 14th-century canopied tombs. The Valences travelled around their many estates, focusing increasingly on a handful of preferred residences. They stayed at Sutton Valence only occasionally. After Aymer's death in 1324, the castle passed by marriage to Lawrence, Lord Hastings. The Hastings family held it until 1390, when it transferred to Reginald, Lord Grey de Ruthin. By this point there are very few historical records of the castle, suggesting it had ceased to function as an active aristocratic residence. By the 15th century it stood in ruin.

Edward Hasted, the Kent historian whose four-volume "History and Topographical Survey of the County of Kent" was published between 1778 and 1799, came to Sutton Valence in the late 18th century. He described the castle as "now almost covered with ivy, and the branches of the trees which sprout out from the walls of it." The 1778 sketch of the keep that accompanies surveys from this period shows a ragged, ivy-throttled tower rising above scrub. The walls were of Kentish ragstone and flint rubble - the standard building materials of the area, pulled from local quarries and from the surface stones of the surrounding fields. Archaeological excavations in the mid-1950s, carried out with the help of Maidstone Museum and Sutton Valence School, concentrated on the keep area, mapping what survived of its foundations. In 1976 the castle was placed in the guardianship of the British state, and it is now owned by English Heritage.

Conservation work in the 1980s stabilised what remained of the keep, removing the ivy that had been pulling the stones apart for centuries. Today the surviving fragment is a Grade II listed building and a Scheduled Monument, open to the public free of charge through English Heritage. From the path up the spur, you see the broken corner of the keep wall, the courses of ragstone clear in the sunlight, and beyond it the long open view south across the Weald that gave the site its strategic value eight hundred years ago. The barbican, the outer bailey, the inner bailey - all now just earthworks under turf. From the air, the site is barely visible: a small notch in the contour of the hill, with a stub of stone at the centre. From the ground, with the wind coming up off the Weald and the high ground falling away to the south, it is easier to imagine William de Valence riding up the lane in 1265, taking possession of his half-brother's gift.