The site sits on a promontory of a raised beach, jutting out at a height of about 60 feet above coastal flats, with steep slopes or cliffs falling away on each side of the point. It is a natural fortress. A wooden watchtower probably stood here long before the stone tower house, raised against Viking raiders working their way up the Clyde from the open sea. The castle that survives today is a 15th-century tower house of three storeys, built on the site of an earlier fortification. It stands within the private gardens of Ardgowan House and is not accessible to the public, but it can be seen from the public path that runs directly below. It is protected as a Category B listed building.

Inverkip Castle features in what came to be called the First War of Scottish Independence. In 1301, King Edward I sent two armies into Scotland with the plan that they would meet at Inverkip. His own force struck through the east. His son Edward, Prince of Wales, led a western army that captured Turnberry and Bothwell castles. The armies finally met at Linlithgow to overwinter, never quite reaching the planned rendezvous. By 1306 the castle was held for Edward by a Lothian Scot, Sir Adam Gordon. In mid-March that year it was besieged by Robert Bruce's supporters under Robert Boyd of Cunningham. After Bruce's defeat at the Battle of Methven on 19 June 1306, Edward ordered Sir Thomas Randolph - later 1st Earl of Moray - held prisoner at Inverkip, under Gordon's supervision, with no option of bail. He had apparently been freed by September 1307.

John Barbour's The Brus, the great 14th-century epic of Bruce's wars, records that when Sir James Douglas defeated Sir Philip Mowbray in 1307, Sir Philip abandoned his troops and rode alone to Inverkip Castle. The poem says: 'Sync throu the Largis, him allane / Till Ennerkip the way he tane, / Rigcht to the castell that wes then / Stuffyt all with Ingless men, / That him resaiffyt in daynte.' The English garrison received him gladly, then later escaped from the castle by sea - a route the promontory and its raised beach made possible. It is the kind of detail that reminds you the medieval Firth of Clyde was a busy and contested maritime corridor, not the quiet coastal strip it can feel like today.

In 1403, King Robert III - Bruce's great-grandson - granted the lands of Ardgowan to his natural son, Sir John Stewart. The Stewart family built the late-15th-century tower house that still stands, perched on the south edge of the promontory above the cliff. In 1667, Archibald Stewart was created a Baronet. The title is still held by the Shaw Stewart family today. The 3rd Baronet married Helen Houston, heiress of the Shaws of Greenock, in 1730. Their son Sir John Shaw Stewart, 4th Baronet, decided the medieval tower was no longer fit to live in and commissioned a new house from the architect Hugh Cairncross. Construction of Ardgowan House began in 1798 and finished around 1801. The old castle was abandoned. In 1936 its ruin was consolidated and repaired so it would not be lost to the weather.

Today the tower house stands as a controlled ruin, its three storeys mostly roofless, walls weathered to lichen-grey, perched on the cliff above the coastal flats. Ardgowan House and its gardens host weddings and events; the castle is part of the scenery but not part of the tour. Walk the public path below the promontory and you can look up at the same walls Robert the Bruce's siege engines were used against. Beyond, on a clear day, the Firth of Clyde opens out toward Cowal and Bute. The English garrison who escaped by sea in 1307 would have left from a shore now lined with the modern village of Inverkip and the marina at Inverkip Bay.