The Portuguese named it Nossa Senhora de Anunciada when they laid the first stones in 1575, dedicating the fortification to the Virgin Mary. Governor Gaspar de Mello completed the structure in 1580, and for twenty-five years it served as Portugal's anchor in the central Maluku Islands. But the Portuguese grip on the spice trade was slipping. They had been expelled from Ternate in 1575 and were fighting constant skirmishes with Muslim communities on Ambon's northern coast. The fort's harbor faced the Banda Sea, and through it flowed the cloves that made these islands worth fighting over. When the Dutch arrived in 1605, commander Steven van der Hagen took the fort without firing a single shot. The Portuguese garrison surrendered, and the conquerors renamed the fortress Victoria — victory.

Under Dutch control, Fort Victoria became far more than a military outpost. Before the founding of Batavia in 1619, the VOC's governors-general alternated their residence between Fort Victoria and Fort Oranje on Ternate. Thirty-three officials worked within its walls managing the trade in cloves, mace, and nutmeg — the only objective the Dutch East India Company pursued in Ambon. The VOC signed innumerable agreements with local rulers, all designed to ensure that every harvested clove ended up in Dutch hands. Resistance was met with overwhelming force. When the leader of Hitu, Kakiali, organized opposition from his stronghold at Wawani, he was murdered by a traitor in 1643. The last holdout, Telukibesi, fought with 300 men from the elevated fortress of Kapahaha until VOC troops found a steep track to scale the rock in 1646. Tens of thousands of Moluccans became victims of VOC brutality during these centuries of enforced monopoly.

Around 1754, a major earthquake shook Ambon and left Fort Victoria severely damaged. The Dutch rebuilt it and rechristened the structure Nieuw Victoria — New Victory — as though sheer will could make the stones forget what had broken them. The renovated fort continued to serve as the seat of colonial government, military command, and the machinery of the spice trade. In front of its walls, a harbor connected the island to the wider archipelago, and the road along Honipopu Beach became known as Victoria Boulevard. Inside, the rooms held strategic planning chambers and food stores. Giant cannons guarded the approaches. Carved wooden statues, maps charting Ambon City's expansion from the 17th through the 19th centuries, and portraits of Dutch administrators filled other chambers — a gallery of colonial power preserved in oil paint and stone.

When the British returned Maluku to the Dutch in 1817 after the Napoleonic Wars, two centuries of exploitation had left the Moluccan people with little patience for another round of colonial rule. Thomas Matulessy, a former sergeant in the British army, took the title Kapitan Pattimura and rallied the people of the Ambon Islands. On May 15, 1817, his forces attacked Fort Duurstede on Saparua Island, killing the Dutch Resident Johannes Rudolph van den Berg and his family. The victory sent shockwaves across Maluku and inspired uprisings throughout the region. But the Dutch counterattack was decisive. Pattimura was captured, brought to Fort Victoria, and hanged on December 16, 1817. Today Indonesia honors him as a national hero, and the fort where he died now serves as the headquarters of the Pattimura XVI Military Command — bearing the name of the man its walls once condemned.

Fort Victoria witnessed one more convulsion in 1950, when the Indonesian army stormed it during the suppression of the separatist Republic of South Maluku. Since 2017, the Ambon provincial government has listed the fort as a national cultural heritage site, acknowledging its layered significance. Walking its grounds today, visitors encounter Portuguese foundations beneath Dutch renovations, cannons that once defended a spice monopoly, and the memory of a hero who died resisting it. The harbor in front of the fort still connects Ambon to the surrounding islands, though the ships now carry passengers rather than cloves bound for Amsterdam. Fort Victoria endures as Ambon's oldest building, a stone chronicle of the forces that shaped the Spice Islands — commerce, conquest, resistance, and the long arc toward independence.