The castle began around 1158 as a fortified tower raised by Valdemar the Great on an islet in the Als Sound, part of a chain of defences against Wendish raiding from the southern Baltic. The islet was later joined to the island of Als, and the place took its name from its position at the south end of that island — Sønderborg, the southern fortress, built on the same logic as English Sudbury. Almost immediately it became something to fight over. The Danish king and the duke of Schleswig contested the island and the town for generations, and the castle changed hands repeatedly. Around 1350 it was enlarged with huge outer walls and the Blue Tower, Blåtårn. In 1490 it passed to the Danish crown, and King Hans and his son Christian II built it up into one of the strongest fortresses in the kingdom — which is one reason it later made such a convenient prison for the second of them.

Between 1549 and 1557, Christian III had the fortress rebuilt as a four-winged Renaissance castle by the architect Hercules von Oberberg, keeping King Hans's west wing and adding three more. After the king's death in 1559, von Oberberg built something rarer for his widow. Queen Mother Dorothea's chapel, completed between 1568 and 1570, is almost untouched, and is reckoned among the oldest and best-preserved Lutheran royal chapels in Europe — a room built for a confession still working out what its churches should look like. Many of its furnishings came from the Antwerp workshop of the painter Frans Floris. The organ dates from about 1570 and is attributed to Hermann Raphaëlis, a Dutch-descended builder summoned to Denmark around 1550 who also made instruments for Roskilde Cathedral and Copenhagen Castle. Its case was largely repainted in 1626, when local carver Niels Tagsen supplied the pulpit and three figures for it. When the organ was reconstructed in 1996, the museum kept some original parts on display, including a rat's nest recovered from one of the bellows.

After Dorothea died in 1571 the castle went to her son, Hans the Younger, and became the seat of a miniature state: the duchy of Schleswig-Holstein-Sonderburg. It did not outlast him. On his death in 1622 the duchy was partitioned, and by 1667 the ruined line was absorbed back into the Danish throne. A royal prefect moved in, and for the next half-century the place mostly sat. Then Frederik IV had it rebuilt again, in Baroque style, by contractor general Wilhelm von Platen between 1718 and 1726 — the shape the building largely keeps today. Its last working career was military: from 1867, in the years when this was German territory, it served as barracks, and it stayed barracks until 1920.

The Riddersal runs thirty-four metres, long enough that a ducal court could hold a ball in it without anyone feeling crowded. It was where guests were received, where the music played, where a small dynasty performed being a dynasty. It has not entirely retired. In 1998 it hosted the wedding reception of Princess Alexandra of Sayn-Wittgenstein-Berleburg, a niece of Queen Margrethe II, to Count Jefferson-Friedrich von Pfeil und Klein-Ellguth — one more entry in a guest list that stretches back to Valdemar Atterdag's marriage to Helvig of Schleswig in the fourteenth century.

When the barracks emptied at the end of the First World War, the last duke of Augustenborg let the Sønderborg County Museum move into part of the building, and since 1921 the castle has housed the Sønderborg Castle Museum — the principal museum of the former Duchy of Schleswig. Its collections run from the Middle Ages to the present, but they lean hard on the things this borderland cannot stop thinking about: the Schleswig wars of 1848–50 and 1864, the Second World War, and the reunification of 1920. There are also exhibitions on navigation, textiles and handicrafts, and a small collection of South Jutland painters. The museum now forms part of Museum Sønderjylland. Outside, the original ramparts were made visible again in the gardens during the 1970s. It is a short drive from here to the Dybbøl redoubts, and the museum's 1864 rooms are the place most visitors go before or after standing on that hill.