The castle dates from 1502, built on the eastern shore of the Firth of Clyde at the north-western corner of Ayrshire. Its early ownership runs through a tangle of Cunninghame and Montgomery family arrangements typical of medieval Scotland. The Cunninghames of Kilmaurs held the lands during the reign of Robert III, 1390 to 1406. Around 1460, the northern portion passed to the Montgomeries, becoming Skelmorlie-Montgomerie. The remainder stayed Cunninghame. Anne, sister of Alexander de Montgomerie, had married a Cunninghame of Kilmaurs, and the division of the lands may have been part of that marriage. On 6 June 1461, Sir Alexander de Montgomerie of Ardrossan formally granted Skelmorlie to his second son George, founding the cadet branch of the Montgomeries of Skelmorlie. The name may come from Norse roots meaning "shelter leeside of the great rock," Skel- equivalent to Skerries.

Sir Robert Montgomerie, the seventh laird, was knighted by James VI and made a baronet by Charles I in 1628. His path to the title was bloody. Patrick Maxwell of Newark Castle in Port Glasgow had killed Sir Robert's father and brother, leaving the young Robert as the new heir of Skelmorlie. The story records Patrick discovering Sir Robert one day in Newark Castle and saying, with calculated cruelty: "Come down Robin, out o' that corner, come down, man, to me, who did you so good a turn as to make you young laird and auld laird o' Skelmorlie in one day." To make a man laird of his own family seat by murdering his father is the kind of remark that explains why Scottish family feuds ran for generations. The third baronet, also called Robert, married Antonia Scott of Rossie. She was an ardent Covenanter, and her husband was repeatedly fined for her attendance at illegal field worship meetings known as conventicles. Their religious dissent cost the family money. It did not stop her.

Sir Robert Montgomerie of Skelmorlie commissioned the Skelmorlie Aisle in the kirk of Largs in 1636 as a family mausoleum. The kirk itself is long gone, demolished centuries ago. The Aisle survives, in the care of Historic Environment Scotland and open in summer, with its remarkable painted ceiling. A stylised view of Skelmorlie Castle is among the panels. The third coffin in the vault is said to be that of Sir Hugh Montgomerie of Eaglesham, a hero of the Battle of Otterburn in 1388, the engagement immortalised in border ballad. A story passed down at Skelmorlie tells of a local warlock bringing the Devil to do mischief to Sir Robert, only to find the laird so absorbed in prayer that the Devil gave up. The tale says more about Sir Robert's habits, and his neighbours' perception of them, than about any actual visitation.

In 1731, Hugh Montgomerie of Busbie, Lord Provost of Glasgow, purchased the castle from the fourth baronet and inherited the title. He had served as a commissioner negotiating the Union with England, sat in Scotland's last parliament, and was appointed to the first parliament of the United Kingdom representing Glasgow. He died in 1735 without children. The direct line ended. Timothy Pont, surveying Cunninghame around 1600, had called Skelmorlie "a fair well built house. It is certainly, in point of situation, a very pleasant and most delightful place, with its old-fashioned gardens, terrace and shrubbery. The view from it over the Firth of Clyde, to the opposite islands of Bute, Arran and Cumbraes, is not to be surpassed in picturesque scenery by any prospect in Britain." By the 1850s the castle was a run-down tower. The Glasgow Port wine merchant John Graham restored it from 1852, rebuilding the old tower in 1856 with the architect William Railton of Kilmarnock. The 16th Earl of Eglinton moved here from the abandoned Eglinton Castle in the mid-1920s and died at Skelmorlie in 1945. After a 1959 fire and a sale by the 18th Earl, the castle passed to the Wilson family and then in 2009 to new private owners. Five centuries on, the view remains as Pont described it.