The story of the castle hill above Arnsberg begins around 1060, when Count Bernhard II of Werl built the so-called old castle, the Rudenburg, on a hill at the confluence of the Walpke and the Ruhr. Sometime between 1070 and 1080 his successor Konrad II began the transfer of the Counts of Werl's seat to Arnsberg. The Grafenburg, on the hill opposite the Rudenburg, is the spur where today's ruins stand. Tradition once attributed its construction to Konrad in 1077; modern scholarship now credits the move to Count Friedrich the Belligerent around 1100. In 1102 a castle here was destroyed by Archbishop Frederick I of Cologne after Friedrich the Belligerent sided with Emperor Henry IV during the Investiture Controversy, the great medieval brawl between empire and church over who got to appoint bishops. The castle did not stay destroyed. The Counts of Arnsberg, and after them the Cologne archbishops who absorbed the county, kept building back, kept holding the hill.

During the Soest Feud of 1444 to 1449, Arnsberg Castle was the main base for the troops of Archbishop Dietrich II von Moers of Cologne, a conflict that pitted Cologne's territorial ambitions against the city of Soest. The castle outlasted the feud and slid into a quieter century of disrepair before being redesigned in 1575 under Elector Salentin of Isenburg. By the early 18th century, when the great Westphalian architect Johann Conrad Schlaun was asked to draw plans for it in the years between 1730 and 1735, the building was being reimagined as a baroque palace and hunting lodge for Clemens August of Bavaria, the prince-elector and serial palace-builder whose other commissions include Augustusburg at Bruhl. Schlaun's drawings, preserved at the LWL Landesmuseum in Munster, show an outside staircase, fireplaces, plasterwork ceilings, and a chapel altar that never quite achieved their final form.

In 1762, in the late chaotic phase of the Seven Years War, Arnsberg Castle came under sustained bombardment. The numbers were extraordinary for a building of essentially residential character: roughly 2,000 cannon shots, 300 fireballs, 1,200 cannonballs. Whatever walls survived the artillery were then blown up by mines several days later. The verdict of later historians has been blunt. The decision to reduce the castle was strategically unimportant. Cologne had already lost effective control of the territory. The destruction served no clear military purpose. What was lost was a cultural inheritance, the seat of the Counts of Arnsberg and a baroque residential palace in mid-rebuild. The smoke cleared and there was only a ruin.

The 19th century did with Arnsberg what the 19th century liked to do with ruins. The Dusseldorf garden architect Maximilian Friedrich Weyhe transformed the Schlossberg into a landscape park in the romantic style between 1818 and 1821. Some of the original gothic arches were reconstructed soon after, partly out of antiquarian curiosity and partly because a properly evocative ruin needs the right silhouette. In 1897 the city of Arnsberg acquired the ruins. The architect Engelbert Seibertz drew up plans for a Kaiser Wilhelm Tower with a restaurant and museum at the site. The First World War killed the project. More recently the city has cleared overgrown walls, relocated a war memorial, opened a circular path, and planted a vineyard at the foot of the ruin in imitation of historical models. There is local belief that the 19th century pastor and poet August Disselhoff composed the song Nun ade, du mein lieb Heimatland, Now farewell, my beloved homeland, in these ruins. Today a regular ruins festival is held on the site to support preservation and draw visitors. The castle that lost its purpose in 1762 is now, again, the heart of the town's identity. Useless as a fortress and beloved as a view.