Irvine's earliest record dates to 1163, when the harbour lay at the sea-gait, the route running down from the town to the water's edge. The first castle here, almost certainly a wooden tower on a motte, was built to defend it. Benedict of Peterborough mentions a castle of Hirun in 1184. The harbour, though, had a problem: it kept silting up. Early authors record sand drifting into the channels so badly that ships could be stranded for months. By 1606 the original harbour was useless. Robert II made Irvine a Royal Burgh in 1372, and Seagate Castle is now the last of the town's old civil and ecclesiastical buildings to survive. The Carmelite friary, the tollbooth, the chapel, the bridge, and the silted-up port have all vanished without trace. What remains is a fortified townhouse that the Montgomerie family treated as both stronghold and stage.

A plaque at the entrance records that Mary, Queen of Scots stayed here in 1563. The 3rd Earl was a great supporter, and tradition holds he entertained Mary along with her four ladies-in-waiting, the famous Four Marys: Mary Seton, Mary Beaton, Mary Fleming, and Mary Livingston. Some say Irvine's Marymass festival takes its name from that visit. Others trace it to Lady Mary of Eglinton, the figure on the Montgomerie family crest, said to have decapitated a Viking who had tried to kidnap her. The Carters of Irvine continue the Marymass tradition to this day. The castle's other major moment came in 1297, at the earlier building on this site. Edward I sent Sir Henry Percy north with a punitive force to crush a Scottish uprising against his dethronement of John Balliol. Robert Bruce, then Earl of Carrick, Bishop Wishart and others led the Scottish army. After encamping in sight of the English at Tarryholme, with Trindlemoss Loch between the armies, the Scots argued among themselves and submitted without a fight. Rodolph de Eglinton is said to have hosted the negotiations here. Bruce never accepted the surrender terms, and Edward I never confirmed them. The document survives in London as the Capitulation of Irvine.

Pass through the vaulted pend and you enter a courtyard with three towers projecting east: two round, one triangular, the asymmetry probably forced by the lie of the land. There was once a guardroom and possibly a prison pit; the kitchen retains its arch, an enormous window, aumbries, and a serving hatch into the passageway. Romanesque mouldings here resemble work of around 1573 found at the old castle of Blairquhan and at sites near Maybole, suggesting a West of Scotland school of stonemasons. Some of the dressed stone almost certainly came from Kilwinning Abbey, which was being progressively demolished after the Reformation. The cesspool sat just outside the kitchen's back door, with garderobe pipes from upstairs feeding into it. The arrangement was convenient and, by any modern reckoning, alarming. Over the centuries, Seagate became one of several Montgomerie dower houses, joined by Easter Chambers in Kilwinning, Kilmaurs Place, Auchans, and Redburn. It was a place for the widows of earls to live with dignity after their husbands' deaths.

The castle was inhabited until around 1746. Then Alexander Montgomerie, 10th Earl of Eglinton, took the roof off to use in a church he was building in Ardrossan parish. Roofless buildings do not last well in Scottish rain, and Seagate began its slow ruination. In 1859 the crowds at the Robert Burns Centenary celebrations sang Auld Lang Syne outside its walls. In 1945 Mrs Walker of Castlepark gave the castle to the Burgh of Irvine. Trial excavations in October 1992 by Irvine Development Corporation found medieval deposits 2.8 metres deep and a short length of wall, but no firm dating evidence. In 1971 the town council carried out repairs; Enterprise Ayrshire returned in the 1990s for more restoration. In 2010 the building was in danger of partial collapse, and North Ayrshire Council consolidated the structure in 2011. There is even a folktale that ley tunnels run from Seagate to Stanecastle and to Dundonald Castle. Roofless or not, the castle still does what it was built to do: it dominates Seagate, and it carries the town's longest continuous memory.