It began in May 1939 with a hundred men transferred from Sachsenhausen and no camp to put them in. They were housed in a tent at the foot of the hill the castle stands on, and later in huts thrown up on the Kuhkampsberg, the rise opposite, so that the prisoners looked across at the building they were being worked to death for. Only then was a permanent camp built, in the Niederhagen district of the village. For its first years the place was an appendage of somewhere else - a satellite of Sachsenhausen, then of Buchenwald - which is a bureaucratic way of saying that nobody was expected to ask about it. On 1 September 1941 it became a concentration camp in its own right, Niederhagen, with 480 prisoners inside the wire.

The camp filled with people the regime had categorised and discarded. Political prisoners. Sinti and Roma and Yeniche. Homosexual men. Jewish men. Soviet prisoners of war, and forced labourers seized in Poland, the Soviet Union, Czechoslovakia, France, the Netherlands and Belgium. And, in numbers unusual for any camp, Jehovah's Witnesses, imprisoned for refusing military service and refusing the Hitler salute, and holding to that refusal inside the wire. One of the prisoners was Gerrit Visser, a Dutch trade unionist from Hengelo, born in 1894, who wrote letters home from German captivity and died in 1942; a book has since been built around his life and his correspondence, one man's route from a factory town in Overijssel to a hillside in Westphalia. Most of the 3,900 left no such trace. They are known as a category, a nationality, a number in a ledger.

The work had a purpose that would be laughable if it had not been lethal. Himmler had leased the Wewelsburg in 1934 and meant to turn it into what he called the centre of the world after the Final Victory - an SS site of vast, half-specified ambition, requiring stone cut and hauled and set by hand. That labour was supplied by the camp. Men died of hunger, of cold, of untreated disease, and of what the record calls the consequences of ill-treatment, which is the phrase historians use when a man is beaten until he stops working and then beaten for not working. In 1942 the SS built a crematorium specifically for the camp, an administrative admission that the deaths were not going to stop. Fifty-six people were killed in formal executions carried out on Himmler's orders: forty-two hanged, and fourteen Soviet prisoners of war shot. Adolf Haas, commandant from September 1941, is recorded telling people he was the lord god of Wewelsburg. He went on to command Bergen-Belsen.

On 1 May 1943 Niederhagen was dissolved as an independent camp. Construction had outrun both the war and Himmler's budget, and most of the survivors were shipped on to Buchenwald and beyond. About fifty men were kept back to carry on working, guarded, in a camp that officially no longer existed - the smallest and last of the site's prisoner populations, still cutting stone for a monument to a victory that was by then plainly not coming. They were still there on 2 April 1945, when the U.S. 3rd Armored Division reached the village and let them out. Days earlier the SS had tried to blow the castle up, run short of explosives, set it burning and left.

The barracks came down in 1967 and the estate went up over the ground. For decades the camp was the part of the local story that nobody had built anything for, while the castle above it drew visitors interested in a mythology the SS never actually wrote. That has changed. A memorial now marks the former roll-call square, the flat open ground where prisoners were counted in all weather and where punishments were carried out in front of everyone; and the former guardhouse holds a memorial museum whose permanent exhibition treats the SS as an institution rather than an enigma. It is a small correction, made late, and it is the right one. The visitor who comes for the castle should understand that the walls were raised by people whose names are mostly gone, in a camp now under somebody's garden.