The museum's first curator was a painter named Louis Joseph Watteau, who was not the famous Watteau but his great-nephew. He made the first inventory of the works confiscated during the Revolution in 1795 — walking through requisitioned churches and convents, listing what had been ripped from altars and what had been left on walls. His son François-Louis-Joseph Watteau succeeded him as deputy curator from 1808 to 1823. Locally they were always called "the Watteau of Lille," a nickname that did them justice and slightly underplayed them at the same time. When the museum opened in 1809, it was housed in a church that had itself been confiscated from the Récollet friars. The collection then moved to the city hall. In 1866 it absorbed the personal collection of Jean-Baptiste Wicar — drawings, including a celebrated Raphael study for the Alba Madonna — and the holdings shifted from large to enormous.

Lille decided in the 1880s that its art deserved a building of its own. Construction began in 1885 under mayor Géry Legrand and was completed in 1892. The architects, Edouard Bérard and Fernand Etienne-Charles Delmas, came from Paris and delivered a Baroque-revival palace facing the Préfecture across Place de la République — pale stone, deep cornices, paired columns. It was monumental in a deliberately old-fashioned way. By the late 1980s the building was failing, and the city had also decided to bring Vauban's eighteenth-century relief models of fortified Flemish towns to Lille. Renovation by Jean-Marc Ibos and Myrto Vitart began in 1991 and finished in 1997. They added a 700 square metre basement gallery for temporary exhibitions and dedicated rooms for the relief models and nineteenth-century sculpture. The renovated building covers 22,000 square metres.

In the basement, in dim conservation light, sit fourteen scale models of fortified towns from the old French and Flemish border. Most are at 1/600 scale. They were built between 1691 and the 1820s, mostly for Louis XIV and Louis XV, as strategic planning tools — three-dimensional intelligence on every bastion, gate, and field of fire in places like Calais, Ypres, Tournai, Namur, and Maastricht. The model of Lille itself was built between 1740 and 1743 by the engineer Nicolas de Nézot, in seven tables, out of paper, silk and wood. It shows the city as Vauban had refortified it after the French annexation of 1667 — the bastioned pentagon citadel, the wet moats, the ravelins and hornworks that withheld two months of Allied bombardment in 1708. The collection moved from Paris to Lille in the renovation; for the first time, the city of Lille could see itself in miniature, surrounded by the towns it had spent three centuries fighting.

Upstairs, the painting collection unfolds as a kind of textbook in chronological cross-section. The fifteenth-century Flemish primitives — Dieric Bouts' paired Ascension of the Elect and Fall of the Damned, painted around 1450 — give way to a dense seventeenth-century room thick with Rubens, Van Dyck, Jordaens, and Philippe de Champaigne, much of it lifted from churches in the wars. Then Goya's two late masterpieces enter the eighteenth-century gallery: The Young People ("The Letter") and The Old People ("Time"), painted around 1808–1812 during the Peninsular War, when Spain was being occupied by the same Napoleonic state that was, at exactly that moment, shipping its plunder to Lille. The pair is one of the deepest meditations on aging and vanity in European painting. They hang less than a metre from each other, and they ask the visitor to stand between them. The museum has held major exhibitions on Goya twice — in 1998–99 and again in 2008 — and the rooms still treat him as the keystone of the collection.