Shapinsay is small - about ten square miles - and it has been inhabited for nearly as long as humans have lived in Orkney. Northeast of the castle stands the Broch of Burroughston, an Iron Age stone tower close enough to count as a near neighbour at the geological scale. East of the castle the Mor Stein, a Neolithic standing stone, has been catching shadows for five thousand years. The patch of ground that became Balfour Castle was originally part of the Estate of Sound, which included a stately home destroyed by British Hanoverian soldiers during the Jacobite rising of 1745. In 1782 the area was sold to Major Thomas Balfour, who built a more modest house on the burnt foundations. By 1846 his grandson David Balfour, recently inheriting and recently married, had bought up nearly all the land on the island around the castle, including the ruins of the medieval Linton Chapel, and was ready to build something more ambitious.

David Bryce was an Edinburgh architect in his late thirties when Colonel David Balfour commissioned him to build the new castle. Bryce had been working in partnership with William Burn and was just stepping out on his own. Balfour Castle was the first major independent commission to bear his name, and Bryce poured into it the vocabulary that would later define Scots baronial revival: corner turrets, crow-stepped gables, conical caps, big mullioned windows, a deliberately picturesque silhouette that combined Renaissance French ideas with imagined medieval Scotland. He laid out an L-shaped plan around the older 1782 structure, incorporating the earlier house rather than demolishing it - a piece of practical economy that also gave the building a kind of layered authenticity. The gardens were installed by Craigie Inglis Halkett. Within a few decades Bryce was the most prolific country-house architect in Scotland.

The Balfours of Shapinsay had been on the island since 1782, when Thomas Balfour bought the land. They were a landed family of modest scale - not the great Lowland aristocracy, not poor by any means, the kind of Scottish rural gentry that ran their estates as small kingdoms with their own farms, tenants, schools, and sometimes their own piers. The castle was the family seat for over a century. By the late twentieth century the costs of maintaining a Bryce castle in the Orkneys had become difficult, and the family eventually let the property as a country house hotel, which it remained as of 2021. By 2025, however, it had reverted to its original use as a private residence. The Balfour name still attaches to the village immediately north of the castle - Balfour - and to the small ferry pier where the daily Shapinsay ferry from Kirkwall ties up.

Balfour Castle is a Category A listed building - the highest of three Scottish heritage designations, applied to buildings of national or international importance. The grounds and formal gardens are separately listed in the Inventory of Gardens and Designed Landscapes in Scotland. The shelter belts of trees that surround the castle are themselves remarkable: Orkney is so windswept that most of the islands have almost no woodland, and the planted woods of Balfour stand out from the air like green islands within the green island. The walled garden, laid out in the mid-nineteenth century, still produces vegetables for the house. From the south lawn the view runs across the small village of Elwick, past the ferry pier, and over the Wide Firth to Mainland Orkney and Kirkwall. The Bryce silhouette - turrets, gables, conical caps - sits at the centre of it all, doing exactly what its architect intended.