The Boyle family came up from Caen in Normandy after the Conquest, and by around 1140 they had settled at Kelburn. Their first structure on the site was a wooden tower. In the 1200s that was replaced by a Norman stone keep, sections of which are still in use today, embedded in the core of the modern castle. A tower house went up in the late 16th century, incorporating fragments of even older masonry on its eastern side. In 1700, the first Earl added the extensions that give the castle its current French chateau-like silhouette, complete with pepper-pot turrets. Few families anywhere have stayed put this long. Kelburn is one of the oldest castles in Scotland and has been continuously inhabited by the same family for longer than any other, a fact that the painted exterior cheerfully refuses to advertise.

The Brazilian artists, working in 2007, treated the harled walls as canvas. Historic Scotland agreed to the project on the understanding that the colour would come down when the castle was re-harled, a deal that suited both sides until people started loving the result. The work appeared on the BBC television programme The Culture Show, and Kelburn featured the same year in another BBC series, Crisis at the Castle, documenting the family's financial struggles to keep the place running. In September 2010 it was reported that Historic Scotland was pressuring Lord Glasgow to remove the paintings, a claim both parties denied. In August 2011 the Earl wrote formally requesting permission to keep the graffiti permanently. The walls remained brightly painted into the 2010s. A tourist attraction had been born from a maintenance problem.

Behind the castle the Kel Burn does the geological work that gave the estate its name. The stream rises on the moors more than 800 feet above the castle and falls, in the space of just over half a mile, through a chain of waterfalls and gorges to the Firth of Clyde. Over millennia the burn has carved a glen that fills with ferns, wildflowers, shrubs, and trees. Walks up through the woodland reveal views across Largs, the open Firth of Clyde, and the long backbone of Arran rising in the west. The country park around the castle opened to the public in 1977, a decision made by Patrick Boyle — later the 10th Earl of Glasgow — who still lives in the castle with his family. The estate is now part working country park, part graffiti pilgrimage, part family home that visitors are politely asked not to wander into unannounced.

On 16 February 2009 the castle suffered minor fire damage from an electrical fault. The blaze took hold around 1:45 in the morning, engulfed a top-floor room and spread to the roof. Twenty-five firefighters worked for more than five hours before extinguishing it at 7:20 am. The castle survived. Each July, the grounds host the Kelburn Garden Party, a music and arts festival spread across multiple stages on the estate. Established in 2009, the festival has become a regular sell-out, drawing visitors who arrive for the music and stay to study the painted walls. The estate also hosts weddings and other private events. Few country houses have arrived at this particular intersection of medieval continuity, contemporary street art, festival culture, and family life. Most never tried.