Less than a year after Hastings, William the Conqueror was already pushing west into Wales. He made one of his closest allies, William fitz Osbern, the Earl of Hereford and let him loose along the border. FitzOsbern captured Monmouth and Chepstow. Somewhere between 1067 and 1071, probably under his orders, three new earth-and-timber fortifications were thrown up along the Monnow valley to block the route to Hereford from any Welsh resurgence. Skenfrith was one of them. In 1135, when a major Welsh revolt followed the death of Henry I, King Stephen pulled the three castles together administratively. They became the lordship known as the Three Castles, an awkward name that nonetheless stuck for centuries, and continued to serve as a defensive bloc against Welsh attack into the late medieval period. By 1186 the Crown was spending real money on Skenfrith: forty-three pounds on defences, more in 1190, probably establishing the first stone keep and curtain wall.

In 1201 King John handed the Three Castles to Hubert de Burgh, then his household chamberlain, soon his most trusted official. Hubert started rebuilding at Grosmont, then was captured fighting in France and disappeared from the picture for several years. Once released, he picked up exactly where he had left off, becoming royal justiciar, then Earl of Kent, then the regent of England in all but name during the boy-king Henry III's minority. By the time Hubert finally got Skenfrith back in 1219, his ambitions had grown. He flattened the older earthworks, spread them out twelve feet deep across the site, and demolished the twelfth-century stone fortifications. On top he built something new: a polygon of four walls, the longest about eighty metres, ringed by round corner towers. A two-storey hall range ran along the south-west wall. At the centre rose a circular three-storey keep, twelve metres tall and ten metres across, the lord's chamber lit by big windows and warmed by a large fireplace on the second floor. Earth was piled around the base, mimicking a motte. Skenfrith was now state-of-the-art.

Through the thirteenth century, the Welsh princes had not given up. In 1262, when Prince Llywelyn ap Gruffudd attacked Abergavenny, Skenfrith Castle was urgently readied; its constable, Gilbert Talbot, received orders to garrison it "by every man, and at whatever cost." That phrase, from the medieval Crown records, gives a sense of how genuine the panic was. The danger passed without an assault on Skenfrith itself, but the Welsh threat dictated the castle's existence. Then, twenty years later, Edward I conquered Wales. By 1282 the principality was broken, and Skenfrith's military reason for being went with it. The castle continued as an administrative centre under Henry VI and successive kings, but the great gates were rarely barred in earnest again. Repairs slowed. By 1538 the buildings were drifting toward ruin, and a 1613 description called the place "ruynous and decayed."

For nearly four centuries Skenfrith Castle was an antiquarian curiosity, a stone shell in a fold of the Monnow valley used as a sheep pen and occasionally drawn by visiting watercolourists. It belonged to the Earl of Lancaster's earldom and then the duchy of Lancaster, which kept hold of it until 1825. By the early twentieth century it had passed into the hands of a lawyer named Harold Sands, who carried out conservation work at his own expense and then gave the castle to the National Trust. In 1936 it was placed into the care of the state. Extensive repairs followed. Today it is managed by Cadw, the Welsh heritage agency, listed at Grade II*, and walkable in any season, no ticket office and no fee. The circular keep and Hubert's polygon of walls remain the clearest survival of his ambition. Stand inside the keep, look up through the open top at the sky, and you can hear the River Monnow on the other side of the wall, moving south toward the Wye.