The castle was built either by Alan fitz Walter, Steward of Scotland, who died in 1204, or by his son Walter, who died in 1246. During Alan's tenure the family added the Isle of Bute to their lordship. A wooden castle came first, but a stone circular curtain wall - 9 ft thick, 26 ft high, around 138 ft across, sitting on a low mound - was in place by the 1230s. The wall was built of coursed ashlar and had only two openings: a main gate with a simple timber door, and a small postern in the west wall. A broad moat connected to the sea. Battlements ran around the top, accessed by open stairs. Holes in the upper masonry would have supported a timber bretasche, a projecting wooden gallery that extended the battlement outward. It was a strong castle. But not strong enough.

Rothesay's fame begins with an attack. In the 1230s, the Norwegian-backed fleet of Óspakr-Hákon, King of the Isles, came to Bute and assaulted the castle. Hákonar saga Hákonarsonar, the Old Norse king-saga that preserves the story, records that Óspakr's forces fought for three days to take the place, breaking down part of the eastern wall by hewing the stone with their axes. This is the earliest recorded account of an assault on a Scottish castle. In 1263, Rothesay was taken again, this time by the Norwegian forces of Hákon Hákonarson, King of Norway, on his way south to fight the Battle of Largs. Although the battle was tactically indecisive, Hákon's campaign was a strategic failure, and Norse power in western Scotland effectively ended with it. In the later 13th century, the castle was strengthened by the addition of four round towers built into the curtain wall. Only the northwest tower survives intact.

During the Wars of Scottish Independence, Rothesay was held by the English; Robert the Bruce took it in 1311. It returned to English hands in 1334, then to the Scots again. After the Stewart accession to the throne in 1371, the castle became a favourite residence of kings Robert II and Robert III - and Robert III died here in 1406. In 1398, Robert III made his eldest son David the Duke of Rothesay, beginning the tradition by which the heir to the Scottish (and now British) throne carries the title. In 1462 the castle survived a siege by the forces of John of Islay, Earl of Ross - the last Lord of the Isles - whose long campaign against royal authority eventually ended in his attainder. Construction of a forework gatehouse keep began in the early 16th century to provide more modern accommodation for James IV, and the curtain wall was raised to ten metres in height under James V.

In 1527 the castle withstood a siege by the Master of Ruthven, which destroyed much of the burgh of Rothesay. In 1544, the castle fell to the Earl of Lennox during the so-called Rough Wooing - the English crown's brutal attempt to force a marriage between Mary, Queen of Scots, and Edward VI. Oliver Cromwell's New Model Army garrisoned Rothesay during the Commonwealth's occupation of Scotland in the early 1650s. On their departure in 1660, the troops partially dismantled the structure. What remained was burned in 1685 by the supporters of Archibald Campbell, 9th Earl of Argyll, during his rising in support of the Monmouth Rebellion against James VII. After that, the castle was effectively abandoned. Two centuries of slow decay followed, with rubbish piling up in the courtyard and the moat silting over.

In 1816-17 the 2nd Marquess of Bute employed seventy men to excavate the ruins, hauling out the accumulated rubbish. Real conservation work began in 1871, when the 3rd Marquess - a serious restorer of historic buildings - hired his regular architect, William Burges, to advise. The Burges restorations continued until 1900 and included clearing the moat and adding red sandstone to the forework, reinstating the hall roof. The result altered the character of the building in ways that purists still question, but it preserved structures that would otherwise have collapsed. In 1961 Rothesay Castle was given to the state. It is now a Scheduled Ancient Monument in the care of Historic Environment Scotland. Visitors can climb the walls and look out over the town, the bay, and back across the Firth toward the mainland - the same horizon the Norse fleets and the Stewart kings once watched from the same stones.