The castle was not, originally, a castle at all. A modest Georgian house stood on the site in the early nineteenth century, owned by Colonel John Campbell of Possil. In the 1850s the Colonel's son, also named John, inherited the estate and decided the existing building was insufficient to his ambitions. He had it demolished. In its place he commissioned David Bryce, the Edinburgh architect who more than any other figure popularised the Scottish Baronial style of the Victorian era - all crowstepped gables, conical turrets, and corbelled bartizans. The new house, completed in 1858, was called Duart House. It was later renamed Torosay to avoid confusion with the genuinely medieval Duart Castle, the MacLean seat that stands a few miles south on the Sound of Mull. Historic Scotland formally listed the buildings and gardens in 1987.

The grounds at Torosay are arguably more famous than the house. Walter Murray Guthrie, the estate's owner around the turn of the twentieth century, learned that an abandoned garden near Milan was being broken up. He bought nineteen marble statues in the style of the Italian sculptor Antonio Bonazza and arranged to have them shipped to Scotland - not as freight, which would have been prohibitively expensive, but as ballast in the holds of returning cargo ships. The statues now line the Statue Walk, a formal terrace bordered by clipped hedging that descends through the gardens toward the Sound of Mull. Olive Guthrie, who managed the estate in the early twentieth century, was Winston Churchill's aunt by marriage. Churchill visited in the 1930s. So did King George of Greece. The thriller writer Angela Thirkell dedicated her 1940 novel Weep No More to Olive Guthrie of Torosay.

For decades the castle was open to the public, linked to the Craignure ferry terminal by the narrow-gauge Isle of Mull Railway. That arrangement ended in 2012, when Christopher Guthrie-James - the fifth laird and last of his line at Torosay - sold the property. He spoke of relief rather than regret. The buyer, the McLean Fund, closed the house for renovation. Kenneth Donald McLean, who became the sixth laird, has reportedly spent more than a million pounds on the work. The castle is now permanently closed to visitors. The gardens, however, still open on the first Sunday of each month from April to October - so the statues that Walter Guthrie shipped from Milan as ballast still stand under Mull's western light, a small piece of Italy at the edge of the Scottish Hebrides.