Roxburgh was a fortress that changed hands too often to be safe in any of them. King David I founded it in the twelfth century, and it appears in the records by around 1128. In 1174 it was surrendered to England after William I of Scotland was captured at Alnwick, and remained largely in English hands for much of the next century. When the Wars of Scottish Independence began, Edward I of England held the castle and used it as a prison. Mary Bruce, sister of Robert the Bruce, was kept here from 1306 to 1310 in a cage hung on the outside of the castle walls. The same fate was inflicted on the Countess of Buchan at Berwick. These were not symbolic punishments. They were public displays of cruelty intended to humiliate the Bruce family and warn the Scottish nobility against further rebellion. Mary Bruce was eventually moved to less savage confinement, and her brother went on to win at Bannockburn in 1314, six months after Douglas took Roxburgh.

After Bruce demolished it, Roxburgh did not stay flat for long. Edward III of England retook the site in 1334 and rebuilt the castle. Alexander Ramsay and his men recaptured it for the Scots on 30 March 1342 with a daring night escalade reminiscent of Douglas's earlier feat. The English took it again shortly after the Battle of Neville's Cross in October 1346. A Scottish siege in 1417 forced major repairs. In August 1436 King James I of Scotland prepared a massive siege with fine large guns, German gun crews under Johannes Paule, Master of the King's engines, three thousand Highlanders and Islesmen brought by Alexander of Islay, Earl of Ross, and men-at-arms from across the Lowlands. He was on the verge of attacking when his queen arrived with news of a plot against his life. James abandoned the siege and his army dispersed, leaving the expensive German equipment behind. The campaign collapsed before it began.

In August 1460, the Scots came back again. King James II personally led the siege of Roxburgh, then in English hands. He had developed a particular enthusiasm for artillery and had assembled an impressive train of guns for the attack. On 3 August, while standing close to one of his own great bombards, the gun exploded. Metal fragments tore through the king's body and killed him at age twenty-nine. His army, perhaps energised by the disaster, stormed the castle anyway, took it, and held it. James's queen, Mary of Guelders, ordered the castle demolished. The royal burgh of Roxburgh, which had once been a wealthy administrative centre of southern Scotland and seat of a royal mint, never recovered. Within a few generations both castle and town had effectively disappeared from the map. Mary of Guelders had her nine-year-old son crowned as James III a few days later, in haste, at Kelso Abbey across the river.

There was one more attempt to revive the site. In 1545, during the Rough Wooing, an English garrison under Ralph Bulmer built a rectangular fort on the old site at the order of the Earl of Hertford. The work was troubled from the start. Bulmer complained that the surveyor William Ridgeway visited only rarely and the work was incomplete. He wanted to build a blacksmith's forge and a bulwark to the south to secure access to drinking water. The fort was destroyed in 1550 by the terms of the Treaty of Boulogne, which ended the Anglo-French war and required the English to abandon their Scottish strongholds. After that, Roxburgh fell silent. The ruins today are little more than a grassy mound with some fragments of stone walls. James Hogg's 1822 novel The Three Perils of Man drew its inspiration from the 1314 night attack, fictionalising what Douglas's men did on the slope in the dark. Stand on the site now and you can see Floors Castle across the river, the dukes' Georgian mansion glittering on the far bank, while the castle that gave Roxburghshire its name has been quietly returning to the field for over five centuries.