The castle was founded before 1260 as a comital seat of the Counts of Berg, built on a small island at the mouth of the Dussel where the river joined the Rhine. The preserved round tower dates from the thirteenth century. In 1288, the Battle of Worringen ended the Archbishop of Cologne's regional power - Adolf VIII of Berg fought on the winning side, alongside John I of Brabant and the counts of Julich and Mark. The victory propelled Dusseldorf into a town with city rights and the castle into something more ambitious. By 1384 a three-winged complex occupied roughly the area of today's Burgplatz. By 1399, two chapels stood within it. In the smaller chapel, Duke Wilhelm took an oath of fealty to England's King Richard II on April 23, 1399 - a diplomatic moment captured in records that survive while the room itself does not.

In 1585, the castle hosted the wedding of Johann Wilhelm of Julich-Cleves-Berg to Jakobea of Baden. Frans Hogenberg produced copperplate engravings of the festivities, recording the throne room, the chapel with its blind arcades and Corinthian pilasters, the inner courtyards with their columns. It was the cultural peak of the castle's life. What followed was not. Jakobea's marriage soured, the last duke Johann Wilhelm suffered severe mental illness and the dynasty died out by 1609, and Jakobea herself died in 1597 under circumstances that contemporaries debated and that Heine, two centuries later, transmuted into the legend of the headless woman in rustling black silk. In 1613, the Palatine-Neuburg prince Wolfgang Wilhelm and Brandenburg's elector Johann Sigismund met inside these walls to negotiate the War of the Julich Succession. The negotiations failed memorably: in the heat of argument, Johann Sigismund slapped Wolfgang Wilhelm across the face.

When Elector Johann Wilhelm - known as Jan Wellem - made Dusseldorf his court residence at the end of the seventeenth century, the castle bloomed. He had it modernized in the latest taste, added a bakehouse, brewery, stables, riding school, theater, ballroom, and a banqueting hall with windows on the Rhine. Between 1709 and 1712, according to plans by the Italian architect Matteo Alberti, an extraordinary new wing was added: the Gemaldegalerie Dusseldorf. It was the world's first independent gallery building - architecture purpose-built to display paintings. Jan Wellem and his wife Anna Maria Luisa de' Medici filled it with Renaissance and Baroque masterworks. After Jan Wellem's death the court moved to Heidelberg and then Mannheim, and in 1805 the collection was carried off to Munich, where it became the foundation of what is now the Alte Pinakothek.

The end came in pieces. French revolutionary artillery shelled Dusseldorf on October 6, 1794, igniting a fire that consumed parts of the castle along with much of the surrounding old town. Napoleon, visiting in 1811, decreed the building should be restored and made into a university. Restoration came eventually under Prussian rule - architects Rudolf Wiegmann and Friedrich August Stuler rebuilt sections in the neo-Renaissance style, completing twenty-four halls by 1852, and the surviving tower received its current lantern roof from a design by King Frederick William IV himself. Then, in the night of March 19 to 20, 1872, a fire broke out for unknown reasons on the upper floor of the Rhine-facing wing. The entire palace was lost. Only the round tower could be saved, thanks largely to the curator Andreas Muller, who rescued valuable holdings as the building burned around him. The southern wing held on until 1896, then was demolished. What remains is the Schlossturm - now home to the Schifffahrtmuseum, the museum of Rhine shipping - and an outline in the pavement of Burgplatz where a six-hundred-year palace once stood.