Felix Nussbaum was born in Osnabrück on 11 December 1904 and began formal study of art in 1920, first in Hamburg and then in Berlin. By late 1932 he had reached the point most young German painters were aiming for: a place at the German Academy's Villa Massimo in Rome, where he was a guest from October of that year. He left early, in May 1933, after a dispute - the academy under the new government had no further use for a Jewish student. The year he arrived in Rome, 1932, arson had already destroyed a large part of everything he had painted up to then.

He did not go home. He went to Belgium. In 1940 the Belgian authorities arrested him as a hostile alien and sent him to the camp at Saint-Cyprien in southern France; he escaped from a transport being returned to Germany and made his way back to Belgium, by then occupied, where he lived without papers. His parents, Rahel and Philipp, were murdered at Auschwitz in February 1944.

The last address Nussbaum and his wife, the painter Felka Platek, lived at was 22 rue Archimède in Brussels. He worked in rooms barely large enough for a canvas, and the work he made there is the reason anyone outside Osnabrück knows his name. In Self-Portrait with Jewish Identity Card, painted in 1943, he holds his papers up toward whoever is looking, collar turned against the cold, eyes going straight through the viewer. Triumph of Death followed in 1944.

Someone denounced them. He was arrested in the summer of 1944 and put aboard the last deportation train to leave the transit camp at Mechelen. It reached Auschwitz on 2 August. Felka was almost certainly murdered in the selection that day. Nussbaum was not killed on arrival - he was registered as prisoner B-3594, and camp records show him receiving medical treatment as late as 20 September 1944. He was murdered at some point between that date and the liberation of the camp on 27 January 1945. His brother was deported that same September and died at Stutthof in December.

The recovery took a long time. A show of his major work opened at the Jewish Museum in New York in 1985; soon afterwards Osnabrück's Museum of Cultural History cleared two rooms for a permanent display. In 1991 the city decided that two rooms were not enough, and in 1996 Libeskind's proposal won the competition. The building opened in 1998.

It is made of three volumes that intersect at hostile angles. The first, clad in oak, holds the work Nussbaum made before the war, when he was simply a promising German painter. The second is concrete, and it slices violently through the first: this is the Nussbaum Gang, a corridor whose proportions echo the cramped Brussels rooms where the last canvases were made. The third volume, in metal, holds paintings that only surfaced later. Roughly 160 works hang across the galleries.

Not everyone thinks the building serves the paintings. The Times noted that the narrow tunnel and subdued lighting impose an atmosphere of oppression, and that the architecture clearly uses the idiom of displacement, loss and incomprehension. Jonathan Glancey, writing in The Guardian, called it a masterpiece in architectural dialogue with the paintings hung on its walls. The counter-argument is straightforward and hard to dismiss: Nussbaum's work is powerful enough on its own, and a neutral white room would let it speak without an architect speaking over it.

The museum also runs an exhibition space on racism and intolerance, which is the part of its brief that keeps changing. Both positions on the building are defensible and the place does not settle the question. What it does do is make it impossible to walk through the last four years of a man's life at a comfortable pace.