Voergaard's recorded history reaches back to 1481, when the castle belonged to Stygge Krumpen, Bishop of Borglum. During the Count's Feud, a civil war that tore through Denmark in the 1530s, Skipper Clement's army of peasants seized it. The Reformation followed, and the Crown confiscated the property in 1536. In 1578, King Frederick II traded Voergaard to Karen Krabbe in exchange for another estate. It was Krabbe's daughter, Ingeborg Skeel, who transformed the castle, completing a major expansion by 1588. Ingeborg is remembered as a formidable and ruthless figure. Legend holds that she drowned the castle's architect in the moat so he could never build anything to rival it. She is also said to have severed children's fingers to steal their grain. Whether or not the tales are true, Ingeborg was undeniably a gifted businesswoman who donated to the local poorhouse and established a hospital and school in nearby Saeby. Her ghost, the stories say, has never left.

Over two centuries, Voergaard changed hands repeatedly and much of its land was sold off. In 1872, Peder Bronnum Scavenius, a politician and landowner, purchased the estate and set about reassembling what had been dispersed. By his death in 1914, the property covered 1,944 hectares, making it one of Denmark's largest. His son Erik Scavenius inherited Voergaard and held it from 1914 to 1945. Erik served as Denmark's Prime Minister during World War II, one of the most controversial figures in Danish political history for his policy of cooperation with the German occupation. By the time Erik left Voergaard in 1945, the castle was architecturally intact but largely empty of significant furnishings.

In 1955, Ejnar Oberbech-Clausen bought Voergaard. He was a Dane who had left for France in 1906 and married Marie Henriette Chenu-Lafitte, the widow of his former employer and daughter of Jules-Emile Pean, one of the great French surgeons of the nineteenth century. Through his marriage, Oberbech-Clausen became an Imperial Count. His wife's art collection, inherited from both her father and her first husband, was extraordinary, and the couple owned several chateaux near Bordeaux. When Marie Henriette died in an air raid in 1941, Oberbech-Clausen eventually decided to return to Denmark. With the French government's approval, he loaded twelve train cars with paintings, furniture, and decorative arts and brought them home to Voergaard. He spent years on a comprehensive restoration of the castle before his death in 1963, after which a foundation opened the property to the public.

The castle itself is a two-winged, L-shaped structure built in red brick, flanked by two octagonal corner towers. Its most striking architectural feature is the sandstone portal on the east wing, a gift from King Frederick II that was originally designed for Frederiksborg Castle. The moat that surrounds the building has been there since the earliest days, and the large park, originally laid out in 1768, was redesigned in the French style when Oberbech-Clausen arrived. In the grounds stands a half-timbered building that once served as the Voer Birk, a manor court where petty criminals and local troublemakers were dealt with, a reminder that for centuries the castle's owners were not just residents but rulers of the surrounding community.

The art collection now includes works attributed to Francisco Goya, Peter Paul Rubens, Raphael, El Greco, Antoine Watteau, and Frans Hals. The furniture includes pieces that belonged to Louis XIV and Louis XVI. Visitors walking through rooms hung with Old Masters in a moated castle on the Jutland flatlands experience a jarring and wonderful dislocation, as if a wing of the Louvre had been transplanted to a Danish farm. That effect is the legacy of one man's improbable journey from North Jutland to Bordeaux and back, carrying half a century of accumulated beauty with him in twelve railway cars.