The Olivers came to this corner of Limerick in 1658 with the Cromwellian land settlements. Captain Robert Oliver, one of Cromwell's soldiers, took possession of the lands the Otway family had held before him - the original Irish name Cloch an Otbhaidhigh, 'the stone structure of Otway', getting anglicized over the centuries into Clonodfoy, which is what the present castle was sometimes called. The first Castle Oliver stood a thousand yards to the southwest. It is famous for one of its residents above all others: Eliza Oliver, born in that earlier house, who became the mother of Lola Montez, the Irish-born adventuress and dancer who, in the late 1840s, was the favourite and lover of Ludwig I of Bavaria. Her influence over the king contributed to his abdication in 1848. Lola Montez had a career that ran through Paris, London, New York, Australia, and the California gold fields. Her grandmother's house was a few hundred yards from where the present castle now stands.

Captain Oliver's descendant Richard Oliver married a Yorkshire heiress and inherited Parlington Hall near Leeds. He moved away, taking the Gascoigne name, and left Castle Oliver to deteriorate in the hands of a bailiff. His two daughters, Mary Isabella and Elizabeth, inherited eventually. Both married Trenches of Woodlawn in Galway; Elizabeth married Frederic Mason Trench, 2nd Baron Ashtown, in 1852. The sisters were unusually accomplished. They designed and executed both the stained glass and the verre eglomise - back-painted glass panels - on the ballroom fireplace at Castle Oliver, work that has largely survived to this day. Mary Isabella, the elder sister, was a skilled wood-turner who published, under a male pseudonym, a book called The Art of Wood-Turning that is still cited as authoritative. They commissioned the present castle from George Fowler Jones in 1845. He had already done substantial work for them in the north of England - almshouses, churches - and now he produced this Scottish Baronial fantasia on a Cromwellian site.

Castle Oliver was built for entertaining, not for defence. The ballroom, drawing room, library, morning room, dining room, and great hall all carry hand-painted ceilings, ornamental corbels, decorative stencil work, and the sisters' own stained glass. Gothic arched doors connect ballroom to drawing room. The cellar runs deep enough to hold its legendary 55,000 bottles. The castle was inherited by Elizabeth's step-grandson, the Honourable William Cosby Trench, and remained in the Trench family until 1978. Mrs Lynn Trench sold the property that year to the racing driver Billy Coleman. After that, the castle changed hands several times, the lands and lodges were broken up by a local bank, and the house itself fell into decay. It was featured in the Irish Georgian Society's book Vanishing Houses of Ireland. Vandals and thieves did their work.

Restoration began in 1988 when Damian Haughton bought the castle and, according to a later owner, sealed most of the worst leaks in the roof. Nicholas Browne, who acquired it in 1998, made it habitable again and later wrote a history called Castle Oliver and the Oliver Gascoignes. In 2006 the castle passed to Declan and Emma Cormack from County Antrim, who pushed the restoration to high standards - reviving the library, the St Patrick stained glass window, the wood panelling, cornices, and many of the fireplaces - and lived there with their three children Michael, Shane, and Ciara. In 2015 they sold it to the Ralph family from Australia, who use it for several months of the year. In the summer of 2014, during Limerick's year as a UNESCO City of Culture, Castle Oliver was open to the public for house tours. The fifty-five thousand bottles, if they still exist, sleep in the cellar.