In 1201, King John gave the three border castles of Grosmont, Skenfrith, and White Castle to Hubert de Burgh, a minor landowner who had risen as John's chamberlain when John was still a prince and who would become one of the most powerful men in England. Hubert began his upgrades at Grosmont. He rebuilt the hall block in stone, a substantial two-storey building with pilaster buttresses, 96 feet across, the floors linked by a spiral staircase. The first floor held the great hall and the solar, the private chamber for the lord, separated by a wooden screen. The hall had a fireplace in the middle of its outer wall, flanked by two large windows looking out over the valley. The ground floor held service rooms lit by narrow loops. The closest analogue in England is Hubert's own hall at Christchurch Castle in Dorset, which suggests the same hand at work. The result, the historian Paul Remfry has argued, was secure but high-status accommodation, a fortress where one could still hold a feast.

Hubert's career did not stay on its upward trajectory. He was captured fighting in France, lost his castles for a time, recovered them by 1219, became Earl of Kent and royal justiciar at the height of his influence, then fell from power in 1232. The Three Castles were stripped from him and given to a royal servant named Walerund Teutonicus. The next year, Henry III led an army into Wales against Richard Marshal, the 3rd Earl of Pembroke, who had thrown in with the Welsh against the Crown. In November 1233 the royal army camped outside Grosmont Castle. Richard's night attack on the encampment did not get inside the walls, but it scattered the rest of the king's force in the darkness. Henry's army fled. The castle, occupied by Walerund's men, stayed put. It was not the kind of episode anyone wanted in the official chronicles, but it happened, and it tells you something about the place. Grosmont was a refuge that night, not a battlefield.

After Edward I's conquest of Wales in 1282, Grosmont's military purpose largely evaporated. The castles continued to function as administrative centres, but the urgency had gone out of them. In the first half of the 14th century, the interior was modernised either by Henry of Lancaster, 3rd Earl of Lancaster, or by his son Henry of Grosmont, who took his name from the place. The southwest tower was converted into a three-storey suite of rooms. The west tower was altered. Most strikingly, a whole new north block was built over the remains of one of the old circular towers and the postern gate, comprising three distinct buildings including a three-storey residential tower. The block carries a distinctive octagonal chimney with a carved top, the kind of fine detail you do not put on a building you mean to abandon. By 1404, when Richard Beauchamp, 13th Earl of Warwick, fought a battle near Grosmont against the Welsh, the castle was still active. It served as the local refuge during Owain Glyndwr's revolt the following year. After that, the strategic clock ran out.

The Three Castles estate stayed with the duchy of Lancaster until 1825, when it was sold to Henry Somerset, the 6th Duke of Beaufort. The Beauforts kept it until 1902, when the 9th Duke sold Grosmont Castle to Sir Joseph Bradney, a soldier and local historian who wrote one of the standard works on Monmouthshire's history. In 1922 Frances Lucas-Scudamore placed the castle in the care of the state. Conservation began promptly, including clearing the basement of the north block of debris that had accumulated over four centuries. Today the castle is managed by Cadw, the Welsh heritage agency, and is protected under UK law as a Grade I listed building.

Grosmont Castle in its current form is mostly Hubert de Burgh's 13th-century work with the 14th-century additions sitting on top. The gatehouse was originally a two-storey rectangular tower with later additions including a buttressed drawbridge pit; only limited parts now survive. The castle had an inner and outer ward, but the outer has been encroached upon by the gardens of the village that grew up around the castle, the houses of Grosmont pushing right up to where the medieval defences once stood. The hall block is the showpiece, still recognisably the room where Hubert held his court. The 14th-century octagonal chimney rises from the north block like a stone signature, finer than it strictly needed to be. Walk up from the village, pay no entry fee (Cadw maintains the site as open access), and you can stand in what is left of the great hall, look out the windows toward the Black Mountains, and imagine the night the king's army fled.