Ranulf Flambard, Bishop of Durham from 1099 to 1128, gave the order. The Scots had been pressing into the bishopric's northern lands too easily, and a strong castle on the Tweed would give the church a forward base. Construction began in 1121. The castle did not stay safe for long. In 1136 David I of Scotland invaded Northumberland and took the castle. It was handed back, then taken again in 1138, this time badly damaged. It sat derelict until Hugh de Puiset, who served as Bishop of Durham from 1153 to 1195, rebuilt it, probably under the direction of his architect Richard of Wolviston. The keep, eighty-eight feet high and a massive eighty-four by sixty feet in plan, is attributed to Hugh de Puiset's reconstruction. Its surviving walls, twenty-eight feet thick in places along the south-west wall, give some sense of why successive Scottish armies threw themselves at it without success.

Norham hosted both kings of Britain in 1209, when William the Lion came south to do homage for his English lands to King John, both of them lodged in the castle's chambers. Edward I, who would earn his nickname Hammer of the Scots, visited more than once. He was here in 1292 when John Balliol did homage to him as King of Scotland, the agreement that would soon collapse into the Wars of Scottish Independence. In 1296 Edward invaded Scotland from this border, and his queen, Marguerite of France, remained at Norham during his campaign. In 1497 James IV of Scotland tried to take the castle with artillery, including a massive twenty-inch bombard called Mons Meg, which now sits at Edinburgh Castle. He failed. The garrison was relieved by an English army, and James went home.

August 1513 changed everything. James IV came back, this time with a properly modern artillery train. His guns pounded the outer defences of Norham for several days until the outer ward fell, and the rest of the castle surrendered. By then the outer walls had largely been destroyed. Norham, which had repelled siege after siege for nearly four centuries, had been broken in days by the new technology. Weeks later, James IV was killed at the Battle of Flodden a few miles to the south, and Norham fell back into English hands. Cardinal Wolsey's chaplain William Frankelayn inspected the castle on 29 August 1515 and found it well fortified with contremurs and murderers, the period terms for protective walls and small breech-loading guns. The bishop of Durham, Thomas Ruthall, ordered restoration that continued until 1521. For another century the castle held a strong garrison through the periodic border crises.

In 1603 James VI of Scotland inherited the English throne, and the border ceased to matter in the way it had for five centuries. Norham, like every Border fortress, slipped into irrelevance. Queen Elizabeth I had already refused to spend money on it. Lord Hunsdon, captain in 1574, had been reduced to propping up the floors of the hall, parlour, and kitchen with ship's masts. By the end of the sixteenth century the castle was decaying. In 1797 a young watercolourist named J. M. W. Turner came north and painted Norham Castle. The painting, which was exhibited at the Royal Academy the following year, made his reputation. He returned again and again, and once, in 1831, he reportedly tipped his hat to the castle in passing. His final treatment of the subject, Norham Castle, Sunrise, painted around 1845, hangs in Tate Britain and is regarded as one of the great forerunners of modern abstract painting, a wash of luminous yellow in which the castle is barely a ghost. The ruins are now in the care of English Heritage and open to visitors.