The building was conceived as a Free Library and School of Art - a thoroughly Victorian compound institution combining literacy, technical education, and cultural improvement under one roof. Dudley Borough Council laid the foundation stone on 3 July 1883. The library and art school opened first. The art gallery did not formally open until 1 August 1888, when Mayor Benjamin Hingley cut the ribbon. In 1906 the Dudley Geographical Society handed over its collection of local fossils and minerals; in 1911 those specimens were installed in what had been the lending library, opening as the museum proper in 1912. The arrangement was very Black Country: rocks, books, paintings, and trade-skill training all sharing a single Victorian envelope. Few buildings in Britain combined so many municipal cultural functions for so long with so little fuss.

The geology collection was the museum's most important holding. About 15,000 specimens, drawn principally from nearby Wren's Nest Hill, where Silurian limestone has yielded some of the most spectacular fossils in the British Isles. The Wren's Nest became the country's first urban National Nature Reserve in 1956. The trilobite Calymene blumenbachii - the famous Dudley Bug, the Dudley Locust - was identified there in 1749 by Charles Lyttleton, and remains the symbol of the borough. The Scottish geologist Roderick Murchison drew nearly two-thirds of the palaeontological evidence for his 1839 book The Silurian System from Dudley specimens. The museum's exhibits, 'Dudley Unearthed' and 'Fantastic Fossils,' walked visitors through the area's deep time: Silurian seas teeming with corals, crinoids, gastropods, trilobites, cephalopods; the Carboniferous forests that became the coal seams underneath the town; the eruption of the Dudley Volcano around 315 million years ago, whose volcanic rocks survive at Barrow Hill.

Brooke Robinson was the local Conservative MP for Dudley from 1886 to 1906 and the town's coroner for many years. When he died in 1911 he left his collection to the people of Dudley. The bequest filled a single room: 17th-, 18th-, and 19th-century British and European paintings, furniture, ceramics. Oriental ceramics, Japanese netsuke and inro, Bilston enamels, commemorative medals, Greek and Roman and Egyptian pottery. Personal memorabilia from Robinson and his two wives. Originally the collection was kept in a purpose-built room at the town hall. In 1979 it moved into the museum and joined the geology and the fine art. Bilston enamel was a Black Country speciality - small painted boxes made of metal coated with enamel, a craft that flourished briefly in the eighteenth century in the towns just east of Dudley before being undercut by Staffordshire ceramics.

One section of the gallery held memorabilia from a young man born in Dudley in 1936. Duncan Edwards joined Manchester United at fifteen, won two Football League titles before he turned twenty-two, and earned eighteen caps for England in a career barely begun. On 6 February 1958 his plane crashed at Munich while returning the Manchester United team from a European Cup tie. Eight of his teammates and twenty-three of the people on board died on impact or shortly after. Edwards survived the initial crash. He fought through fifteen days in a Munich hospital, his injuries appearing to stabilise, before his kidneys gave way and he died on 21 February. He was twenty-one years old. Statues, streets, and stained-glass windows around Dudley remember him. The museum's small room of caps and shirts and photographs let visitors see what one young footballer's life had looked like before it was cut so short.

The collections did not vanish. Many of them moved to the Dudley Archives Centre on Tipton Road, where Dudley Museum at the Archives now displays a smaller selection. The new building also houses the headquarters of the aspiring Black Country Geopark - a project organised by local councils to win UNESCO Geopark status for the area's extraordinary geological heritage. The fight to keep the original museum open included a public petition, demonstrations, and considerable local press coverage. The council pointed at its budget. The building on St James's Road no longer admits the public; the fossils and the netsuke and Duncan Edwards's caps are mostly safe in storage or on display elsewhere. The closure was the kind of choice austerity made common across post-2010 England: not dramatic, not televised, just the small permanent diminishment of one local cultural infrastructure at a time, in places like Dudley where it took longest to build.