Castle Hill is an outcrop of Wenlock Group limestone that rises above the town of Dudley like a geological declaration. The same limestone that made the hill defensible also made it valuable: during the Industrial Revolution, quarries carved deep into the rock, extracting the raw material for buildings and lime kilns across the Black Country. The neighboring Wren's Nest Hill is now a scheduled monument preserving the best-surviving remains of Dudley's limestone industry. Beneath Castle Hill, the Dudley Canal Tunnel runs through the rock -- though not directly under the castle itself. The motte, the oldest surviving structure, stands roughly nine meters high, its core of limestone rubble encased in clay. The four drum towers of the keep, each nearly ten meters in diameter, date from the rebuilding begun by Roger de Somery II in 1262.

The castle's revolving door of owners reads like a medieval soap opera. The Paganel family built the first stone castle, strong enough to withstand a siege by King Stephen's forces in 1138. But when Gervase Paganel joined a failed rebellion against Henry II in 1173, the king ordered the castle demolished. The Somery dynasty inherited the ruins and began rebuilding in 1262, a project that continued into the 14th century. John de Somery, the last male heir, died in 1321, and the castle passed through marriages to the Sutton family. John Dudley, Duke of Northumberland, added a range of residential buildings within the fortifications before his execution in 1553 for attempting to place Lady Jane Grey on the throne. The castle was even considered as a prison for Mary, Queen of Scots -- though that plan never materialized.

The English Civil War brought Dudley Castle its final active chapter as a military fortification. Parliamentary forces besieged it in 1644, and the garrison held out until May 13, 1646, when it surrendered to Sir William Brereton's forces. Parliament then ordered the castle slighted -- deliberately damaged to prevent it from being used again. The keep was partially demolished, giving the ruins their current broken profile visible from across the town. Some habitable buildings survived, and the Earls of Dudley used them occasionally, though by this point they preferred Himley Hall, four miles away. A stable block was added before 1700, the last structure ever built within the castle walls. Then in 1750, fire consumed the remaining residential buildings. After that, the castle simply waited, open to the sky, hosting occasional fetes and pageants through the 19th century.

The ruins acquired a reputation as one of the most haunted sites in England. The most persistent ghost is the Grey Lady, believed to be Dorothy Beaumont, who died in the castle along with her baby shortly after childbirth. She had requested burial beside her daughter and asked that her husband attend the funeral; neither wish was granted. Visitors and staff report her presence near the keep and in the Grey Lady Tavern on the castle grounds. The underground chapel, which contains an ancient stone coffin attributed to John de Somery, is said to be the most active site for unexplained phenomena. In 1937, Dudley Zoo opened on the castle grounds, filling the medieval landscape with modernist concrete enclosures designed by the Tecton architectural group. Today the castle is a Grade I listed building, though structural problems led to its placement on Historic England's Heritage at Risk register in 2020 -- yet another chapter in a story defined by construction, destruction, and precarious survival.