The island's recorded European history begins violently. John Hanson and his son-in-law Ephraim Young arrived from Gouldsboro, Maine, in 1777, the first American Loyalists in the area. They settled on Consquamcook Island with their families but held no legal title. Six years later, when a larger group of loyalists arrived, Captain Samuel Osborn was granted ownership. When Hanson and Young refused to leave, Osborn turned the 38 guns of the warship Arethusa on the island for target practice -- a persuasion technique that proved effective. The dispossessed families petitioned Governor General Carleton in Halifax, but lost. In 1791, Osborn sold the island to Reverend Samuel Andrews, the town minister, whose small stone cottage still stands today. Andrews never found a buyer when he tried to sell in 1798, and the property passed through four generations of his family until Edwin Andrews sold it to Van Horne in 1891.

William Van Horne first visited St. Andrews in 1889 while inspecting the New Brunswick Railway, which the CPR had just purchased. He fell for the place immediately. Two years later, he bought 150 acres and began building Covenhoven, a summer home he designed himself from red sandstone quarried on the island. Between 1892 and 1901, the house grew to 50 rooms and 10,000 square feet, with architect Edward Maxwell -- the man behind the Chateau Frontenac -- assisting on later additions. The varying stages of construction gave Covenhoven three different roof pitches, numerous staircases, and oddly connected rooms that made the house feel like a place that had grown organically rather than been planned. Van Horne hung approximately eighty works of art on the walls, many painted by himself, with birch trees as a recurring subject. Maxwell also designed a grand chateau-style barn in 1898, one of the largest and most admired in the Maritimes. It housed prizewinning Clydesdale horses and Dutch Belted cattle -- one of the only such herds in North America. Greenhouses produced grapes, peaches, nectarines, and cherries, while the farm's produce was shipped by night train to Montreal in winter. At the island's southern tip, Van Horne built a perfectly round bathhouse with panoramic bay views upstairs and a tidal swimming pool below.

Van Horne died in 1915, and his daughter Adaline maintained the estate until her own death in 1941. After that, the island drifted. Cost-cutting measures through the 1940s and 1950s scaled back the gardens and greenhouses. When Van Horne's granddaughter Beverley Ann took possession in 1953, she showed little interest in restoration, and the property sat largely empty. An American company bought it in 1961 and formed the Van Horne Island Club, envisioning a private resort with a golf course, yacht club, and airstrip. But the island's remoteness -- the same quality that had attracted Van Horne -- defeated them. After twelve years and considerable investment, they had hosted only friends and family. A Maine developer named Norman Langdon tried next and met the same fate. By 1977, having spent roughly $300,000, Langdon auctioned off the house's contents in an event that drew national media attention. The Van Horne home on Sherbrooke Street in Montreal had been demolished in 1973, making Covenhoven the last surviving Van Horne residence. When businessmen took an option on the island itself, the Province of New Brunswick declared it a protected site and eventually purchased the property.

Ministers Island carries two distinct National Historic Site designations. The first, recognized in 1978, honors the pre-Columbian shell middens left by the Passamaquoddy people -- layers of discarded shells that archaeologists read like geological strata, each layer recording centuries of seasonal habitation. A cairn marks the site today. The second designation, granted in 1996, commemorates the Van Horne estate and its connection to the man who helped unify Canada through rail. Since 2006, a nonprofit has managed tours and restoration work. The stone bathhouse and windmill were restored in 2008, and the great barn was repaired after storm damage in 2016. Twenty-one of Van Horne's original paintings have been returned to the house. Between 2012 and 2017, visitor numbers grew by 330 percent. The island that once defeated resort developers now thrives as a public heritage site -- still governed, as it always has been, by the rhythm of the tides.