The castle stood on a high, detached perpendicular rock above the harbour, accessible only from a single side, connected to the main fortress by a 69-foot passage of masonry. The 18th-century antiquary Francis Grose called this rock the citadel and supposed it had been the original keep. Its octagonal interior, 54 feet across, still preserved five gun-ports the locals called arrow-holes, four feet wide at the mouth and tapering down to sixteen inches at the inside, designed to give the gunner the widest possible sweep while keeping the embrasure small enough to deny return fire. The main body of buildings stretched 165 feet east to west and 210 feet north to south. In its centre was a gateway carved with the arms of George, 10th Earl of Dunbar, who succeeded in 1369, who held not just the earldom but the lordship of Annandale and the Isle of Man, inherited from his heroic aunt, Black Agnes of Dunbar.

Black Agnes is the name that still carries here. Agnes Randolph, Countess of Dunbar, held this castle through a five-month English siege in 1338 while her husband was away, mocking the Earl of Salisbury from the walls and famously dusting off the parapet with a handkerchief after each English bombardment to show how little she thought of his siege engines. Two centuries later the castle was again at the centre of Scottish high politics. During the minority of James V, the Duke of Albany held it with a garrison of 100 French soldiers paid for by Francis I of France. New artillery bulwarks were built. An Italian fortification drawing by Antonio da Sangallo the Younger, the great Renaissance military architect, may show what was planned. By the 1540s and the Rough Wooing, the French garrison was being supplied with money and engineers from Paris. Lorenzo Pomarelli, an Italian military engineer, worked on the defences. Mary of Guise herself fled here in June 1559 in fear of the Protestant Lords of the Congregation.

After her return to Scotland in 1561, Mary Queen of Scots made Dunbar Castle one of her most important fortresses. She visited in September of that year, then again in February and November 1564. In 1565 during the Chaseabout Raid, the rebellion against her marriage to Lord Darnley, she ordered the gun emplacements repaired and the garrison provisioned. But the castle's most dramatic appearance in Mary's story came on 24 April 1567. The Earl of Bothwell brought her here, by force or by collusion no one has ever entirely settled, just weeks after the murder of her husband Lord Darnley at Kirk o' Field in Edinburgh. From Dunbar she went on to face her enemies at the Battle of Carberry Hill on 15 June 1567. She lost. The reckoning that followed swept Bothwell into exile, Mary into imprisonment, and Dunbar Castle into the destruction the Scottish Parliament voted on six months later.

The act of Parliament was explicit. Dunbar Castle and the fortress on Inchkeith were both to be cast down utterly to the ground. William Drury reported back to England that Regent Moray was unable to finish the work in early 1568 because the master carpenter, probably Andrew Mansioun, said the demolition would cost 2,000 Scottish pounds. The same sum, the report continued, spent on rebuilding would make Dunbar one of the strongest places in Scotland. The demolition went ahead anyway, just not completely. In September 1568 stone was being selected for reuse at the Shore of Leith. For the next 275 years the half-demolished castle stood as a slowly weathering ruin above the harbour, picturesque enough to attract J. M. W. Turner who sketched it in the early 19th century.

Then in 1842 the engineers came back. Dunbar needed a new harbour. The castle ruins were quarried for stone, and the entire north end of the fortress was removed by what one contemporary account called a specially-invented electrical detonation system. Whether the description was technically accurate or just journalistic enthusiasm for the new technology of electric blasting, the result was the same. The castle that had survived sieges, slightings, French garrisons, and Italian engineers was finally reduced by Victorian harbour works. What remains today is the southern portion, perched above the modern Victoria Harbour, still showing the carved arms of the Earl of Dunbar on a gateway, still recognisable as one of the great fortresses of medieval Scotland, still red sandstone bleaching slowly in the salt air.