The castle exists because of fear. In 1664, tensions between Great Britain and the Netherlands were rising, and Commander Zacharias Wagenaer - successor to Jan van Riebeeck at the Cape - received orders to build a stone fortress. The first stone was laid on 2 January 1666, but the Dutch East India Company was famously reluctant to spend money. Construction dragged on for years, interrupted by budget disputes and labor shortages. The VOC would not enslave the local Khoikhoi and San peoples, fearing revolt from a population whose size they could not gauge, so they imported enslaved workers from across the Indian Ocean world instead. On 26 April 1679, the five bastions were finally named after the titles of William III of Orange-Nassau: Leerdam, Buuren, Katzenellenbogen, Nassau, and Oranje.

When the castle was completed, its walls rose directly from the shoreline of Table Bay. Ships anchored within sight of its guns, and the pentagonal fortress commanded the approaches to the tiny Dutch settlement. Today, the castle sits hundreds of meters inland, stranded by two centuries of land reclamation that pushed Cape Town's waterfront steadily northward. The Foreshore district - office blocks, highways, the convention center - now occupies what was once open water. It is a disorienting transformation. The castle was designed as a maritime fortress, its cannons trained on the sea, and now it faces a traffic circle. Yet the yellow-painted walls still carry their original purpose in their bones: the color was chosen to reduce heat and glare, the courtyard's dividing wall was built to protect civilians during attack, and the De Kat Balcony, designed by Louis Michel Thibault and sculpted by Anton Anreith between 1786 and 1790, served as the platform from which announcements were made to soldiers, enslaved people, and burghers alike.

During the Second Boer War (1899-1902), part of the castle served as a prison. Among its inmates was Fritz Joubert Duquesne, a Boer soldier who would go on to become one of the most remarkable spies of the twentieth century - later known as the man who claimed to have killed Lord Kitchener and as the leader of the Duquesne Spy Ring, the largest espionage case in American history at the time of its exposure in 1941. But at the Castle of Good Hope, Duquesne was just a prisoner with an iron spoon. Night after night he scraped cement from between the stones of his cell wall, slowly excavating a tunnel through walls built to withstand cannon fire. He nearly made it out, but a large stone slipped and pinned him in the passage. Guards found him the next morning, unconscious but alive.

The castle courtyard displays six flags that have flown over the Cape in chronological order: the Dutch Prince's Flag, the flag of Great Britain, the Batavian flag, the flag of the United Kingdom, the old South African flag, and the current South African flag. Each represents a transfer of power, a shift in who controlled this strategic point between the Atlantic and Indian Oceans. The castle was declared a historical monument in 1936 - the first site in South Africa to receive that protection - and extensive restorations in the 1980s made it the best-preserved Dutch East India Company fort in existence. Today it houses the Castle Military Museum, the Cape Heritage Museum, and the William Fehr Collection of paintings and antique furniture. Every weekday at 10am, the Castle Guard performs the Key Ceremony, a ritual based on drills from three centuries ago, formally opening the gates to visitors who come to walk through the oldest doorway in South Africa.