The Portuguese laid the first stone on the feast day of Saint John the Baptist in June 1522 and named the fort accordingly: Sao Joao Baptista de Ternate. By February 1523 it was complete, positioned on Ternate's southwest coast near the sultan's court but seven kilometers from the island's main reef-free harbor at Talangame. The site was chosen for defense, not convenience. Three narrow passages through the encircling coral reefs allowed small craft to arrive for loading cloves but prevented larger warships from closing to bombardment range. Visitors consistently described the fort as incomplete, and it was not until Antonio Galvao's governorship beginning in 1536 that the defenses were substantially improved.

The relationship between the Portuguese and the Sultan of Ternate deteriorated over decades of broken promises and forced spice monopolies. In 1570, the Portuguese governor Lopez de Mesquita had Sultan Hairun murdered, likely to remove a leader who was proving increasingly resistant to Portuguese demands. The assassination backfired catastrophically. Hairun's son, Sultan Baab, known to history as Babullah, declared war. For five years his forces besieged the Portuguese inside their fort. In 1575, the garrison surrendered and retreated to Ambon, ending a half-century of Portuguese presence on Ternate. Babullah occupied the fort, renamed it Gammalamma, and converted it into his royal palace, extensively modifying the European defenses and constructing an additional fort, Fort Kota Janji, five kilometers to the east.

In 1606, a Spanish expedition from the Philippines recaptured Kastella, taking Sultan Saidi Berkat hostage and exiling him to Manila. The Spanish transformed Gammalamma into a powerful fortress they called Ciudad del Rosario. Dutch Admiral Paulus van Caerden, captured by the Spanish in 1610 and imprisoned within its walls, reportedly deemed it "invulnerable." For 57 years, from 1606 to 1663, Ciudad del Rosario served as the Spanish capital of the Moluccas. Within its walls grew a colonial city of churches, a Franciscan monastery, and a hospital. When the Spanish finally departed the Spice Islands in 1663, they partially demolished the defenses rather than leave them intact for the Dutch who would inevitably follow.

Today, Fort Kastela's remains spread across a large area on Ternate's southwest coast, bisected by the main island road and interspersed with local houses. Sections of the thick Spanish-era walls and bastions still stand. A monument commemorates the 1575 Ternatan victory over the Portuguese, a point of local pride marking the moment when an indigenous sultanate expelled a European colonial power through five years of sustained siege warfare. The long list of Portuguese captains who commanded the fort, from Antonio de Brito in 1522 to Nuno Pereira de Lacerda in 1575, reads like a roster of empire's servants stationed at the furthest edge of their world, each inheriting a fort that was never quite finished and a relationship with the local sultan that was never quite stable.