Like many country estates in Ulster, Crom began as a Plantation property. In 1611, as part of the Plantation of Ulster, a Scottish planter named Michael Balfour, the Laird of Mountwhinney, constructed a house on the lough shore opposite Inishfendra Island. Balfour built in the typical Plantation pattern: lime and stone walls enclosed within a defensive bawn. The site sat at the southwestern end of Upper Lough Erne, a strategic position controlling water traffic between the upper and lower lakes. The bawn walls offered protection against the displaced Gaelic Irish, who had reason to resent their new neighbours. The Old Castle of Crom occupied this position for nearly a century before the wars of the 1680s arrived to test it.

In 1689, during the Williamite War in Ireland, the Old Crom Castle survived two Jacobite sieges. Colonel Abraham Creighton, whose family by then owned the castle, held out against the besieging forces of King James II until reinforcements from Enniskillen arrived. Enniskillen had become one of the great Williamite strongholds in Ireland, defended by a force of local Protestants who fought with disproportionate effectiveness against the much larger Jacobite armies. The relief column that came to Crom set the stage for the Battle of Newtownbutler in July, when the Williamites met James's army at the townland of Kilgarrett, one mile south of the village then called Achadh Gé. The Williamites captured or killed up to 3,000 Jacobite soldiers. The village was renamed Newtownbutler after the battle, and the sieges of Crom passed into the folk memory of Northern Ireland's Protestant community.

In 1764, a domestic fire destroyed the Old Castle. Domestic fires were the bane of country houses in the eighteenth century, when wooden floors, candles, and open fireplaces existed in fragile equilibrium with the limestone walls around them. The blaze gutted the building. Two towers, sections of wall, and a ha-ha survive today as ruins, sitting picturesquely on the lough shore. The ha-ha, an eighteenth-century landscape feature, is a sunken ditch designed to keep livestock out of the formal gardens without interrupting the view from the house. Its presence indicates that significant landscaping had already been done around the Old Castle before its destruction. The Creighton family did not rebuild on the same site. They lived in temporary accommodation and slowly developed plans for something larger.

In 1840, the English architect Edward Blore designed the present Crom Castle. Blore was one of the most prolific country house architects of the early Victorian period, with commissions at Buckingham Palace, Lambeth Palace, and dozens of estates across Britain and Ireland. His Crom Castle is built of stone, with a central battlemented tower containing the main entrance and smaller towers to one side. It sits within a 1,900-acre estate, with formal gardens and panoramic views across Upper Lough Erne. The boathouse on the shore, in Tudor style with arched doors and windows at ground-floor level, was rebuilt in 1841 by George Sudden. Before the twentieth century, much travel in County Fermanagh was easier by water than by road, and arriving guests would frequently disembark at this boathouse, climb the path up to the castle, and be welcomed into the new world the Creightons had built atop the ashes of the old.

In 1987, Harry Erne, the 6th Earl of Erne, gave the wider estate to the National Trust to manage. The castle itself remained the private property of the family. The current owner, John Erne, the 7th Earl, lives at the castle but allows it to be hired for weddings, private events, and television productions. The estate's 1,900 acres include forest walks, the Old Castle ruins, the boathouse, and the famous yew trees, which are believed to be at least 800 years old. They were already ancient when Michael Balfour built his planter's house in 1611. They are still standing now, somehow, against all odds, having outlived sieges, fires, and four centuries of dynastic upheaval.

Crom Castle has had two unusual moments of recent fame. In August 2010, the estate hosted a great classic yacht and steamboat regatta, recreating the 1890s races in Trial Bay. The boats used were historic types: Norfolk Broads One-Designs, Lough Erne Fairies, Fife One Designs from Anglesey, and a pair of Colleens. The races took place on Upper Lough Erne within sight of the castle, and the boats moored each evening at the boathouse in Crom Bay. Then, in 2013, the BBC television adaptation of P. G. Wodehouse's Blandings was filmed on location at Crom Castle. The series portrays the fictional Blandings Castle in Shropshire, but the real-life Crom proved a perfect stand-in for Wodehouse's comedic country house. The estate, having survived siege, fire, and four centuries of property change, found itself improbably cast as the home of Lord Emsworth and his beloved prize-winning pig.