King David I commissioned the castle in the 1120s, when Berwick was Scotland's wealthiest port and the county town of a Berwickshire that included land on both sides of the Tweed. The castle stood west of the town, separated from it by a broad moat - a stronghold that could fall without taking the town with it, or hold out after the walls fell. It changed hands repeatedly. The English took it under the Treaty of Falaise in 1175. Edward I of England stormed and sacked it in March 1296, beginning the First War of Scottish Independence with the massacre of Berwick. Robert the Bruce's forces recovered it in April 1318. By the time Edward III's brother Richard, Duke of Gloucester - the future Richard III - captured it in August 1482, the castle had been bought, betrayed, traded, and conquered more than a dozen times. After Richard, it never changed hands again.

The roll of governors reads like a syllabus of the Anglo-Scottish wars. Sir William Douglas surrendered the castle to Edward I after the 1296 massacre. Edmond de Caillou, a Gascon installed by the English, was killed at the Battle of Skaithmuir in 1316. Patrick de Dunbar, fifth Earl of March, was Keeper of the Castle during the desperate summer of 1333, when Edward III was hanging the governor's hostage son in sight of the walls below at Halidon Hill. Sir Patrick Hepburn, the last Scottish governor, surrendered to Gloucester in 1482. Then came the English: Sir William Drury, Francis Russell of Bedford, Sir George Bowes who in 1568 escorted Mary, Queen of Scots from Carlisle to Bolton Castle and whose sister married John Knox. Each name a story, each tenure a few years of bricks against the inevitable next siege.

Elizabeth I's engineers rendered the medieval castle obsolete in the late 16th century by building the magnificent trace italienne ramparts around the town itself - artillery-proof bastions in the latest Italian style, designed to deflect cannonballs the old stone walls could not. The castle began its long decline. In August 1590 John Selby reported that the round tower used as the castle's only gun emplacement had simply collapsed in wet weather. Repairs were estimated at £200. Peregrine Bertie, 13th Baron Willoughby de Eresby and Governor of Berwick, wrote that the place was "scarcely defended with so good a wall as an ancient monastery's orchard," patrolled only by aged veterans. The surveyor of fortifications recommended demolition. But Bertie had rebuilt some structures "for his own use and pleasure" by October 1600, treating the old fortress as a country retreat.

After the Union of the Crowns in 1603, the garrison was reduced to a hundred men and pensioners, and the Scottish courtier George Home, first Earl of Dunbar, began building a private house on the castle site. James Murray of Kilbaberton, the master of royal building works in Scotland, was involved in the construction. The mansion was never finished, but rumours of its splendour reached London. George Chaworth wrote to the Earl of Shrewsbury in 1607 that its proportions were magnificent and its long gallery would make the one at Worksop Manor look like a garret. When William Brereton visited Berwick in 1633 he found a half-built "stately platform" and a long gallery with a five-yard mantelpiece, abandoned at the Earl's death in 1611. People had walked on its flat roof to look at the landscape. The view from above the Tweed would have been extraordinary.

The castle's stones live in other Berwick buildings now. Large parts of the structure were quarried in the 17th century to build Holy Trinity Parish Church during the Commonwealth - Oliver Cromwell granted special licence for its construction, and it became the most northerly Anglican parish church in England, built from the ruins of a Scottish royal fortress. The pulpit inside is reputed to have been built for John Knox, who served as the resident Protestant minister at Berwick in 1549-1550. The castle's Great Hall lasted until 1847, when the Royal Border Bridge railway viaduct needed approaches and Berwick-upon-Tweed station needed a platform. What remains today is maintained by English Heritage and free to visit at any hour. Substantial walls and the White Wall step down toward the river, still standing where they have stood for nine centuries.