In 1720, French military personnel from Fortress Louisbourg founded Port-la-Joye on the southwestern shore of the harbour, directly across the water from what is now Charlottetown. Michel Hache-Gallant led the effort, using his sloop to ferry Acadian settlers from Louisbourg on Ile Royal. The settlement that grew around the outpost was modest and often miserable. Troops garrisoned here were rarely relieved, morale was low, and the wooden barracks offered poor shelter -- wind, rain, and snow found their way between picket walls and through rotten planked roofs. Yet Acadian farmers established themselves in the surrounding area, and for nearly four decades under French control, Port-la-Joye served as the island's seat of government and its port of entry.

The wars that shaped northeastern North America reached this quiet harbour twice in quick succession. In 1745, following the first Siege of Louisbourg during King George's War, New England troops landed at Port-la-Joye and burned the community to the ground. The French garrison of just 20 soldiers under Joseph de Pont Duvivier fled up the Northeast River, now called the Hillsborough. Reinforced by Acadian settlers and Mi'kmaq fighters, they drove the New Englanders back to their ships. The following year, a larger French and Mi'kmaq force under Montesson surprised and defeated 200 British soldiers on the river's banks, killing or capturing 34 New England irregulars. France reclaimed the island, and between 1748 and 1749, built a star-shaped fortification at the site in the style of the great military engineer Vauban.

The final Siege of Louisbourg in 1758 brought the island under British control for good. Lord Rollo arrived at Port-la-Joye with 500 soldiers expecting a few hundred Acadians. He found between 3,000 and 5,000. His men rounded up roughly 3,000 for deportation to France. The toll was devastating: 700 Acadian deportees drowned when their ships sank crossing the Atlantic, and an estimated 900 more died of disease. Some 1,600 Acadians evaded capture by hiding in forests on the island's western reaches. The British replaced the French fortification with a stockade fort completed on October 10, 1758, naming it Fort Amherst. It held 190 soldiers of the 28th Regiment of Foot, 18 cannon, and buildings ranging from officer's quarters to a bakehouse and prison. A mutiny in 1762 ended in courts-martial, demotions, hundreds of lashes, and one execution.

In 1764, Samuel Holland arrived at Fort Amherst as newly appointed Surveyor-General of North America, tasked with mapping British possessions north of the Potomac River. Finding the fort unsuitable for his needs, he built a dwelling at a spot he called Observation Cove, about 1.5 kilometres to the south. Through a harsh winter, Holland, his deputy Thomas Wright, engineers, and soldiers surveyed the entire island, dividing it into three counties, 15 parishes, and 67 townships. His survey selected the site across the harbour for a new colonial capital. Charlottetown was named by Holland after Queen Charlotte, consort of King George III. Two years later, the garrison moved across the water, and Fort Amherst was abandoned. By 1779, nothing remained but a ditch and earthworks.

The site became farmland, passed through various owners -- including the island's first governor, Walter Patterson, who named his property Warren Farm -- and quietly decayed until the federal government purchased it in 1959. A visitor centre opened in 1973. For decades, the site bore only two of its three names. In 2008, Mi'kmaq spiritual leader John Joe Sark called the inclusion of Amherst's name "a terrible blotch on Canada," citing the general's support for distributing smallpox-infected blankets to First Nations people. Mi'kmaq historian Daniel N. Paul added his voice to the call for change. In February 2018, the site was officially renamed Skmaqn--Port-la-Joye--Fort Amherst, restoring the Mi'kmaq name for the place where their leaders had gathered with French allies for decades. The waiting, it turned out, was for recognition.