Anthony Patras was born in Grenoble in 1718, sent to Geneva by an uncle to study, then to Batavia in the Dutch East Indies, where the same uncle - Abraham Patras - happened to be Governor-General. When Abraham died in 1737, twenty-year-old Anthony inherited a fortune accumulated under the Dutch East India Company. He married a daughter of a member of the Council of India in 1738, lived through the 1740 Batavia massacre of ethnic Chinese (a violence the Dutch authorities committed under his uncle's office), and returned to the Republic. He bought his way into the council of the small Frisian town of Sloten and served decades as its representative in the States General of the Netherlands. The house de Swart designed for him on the Lange Voorhout was meant to be his showpiece in The Hague. Patras died in 1764 before he had properly used it. The fortune that built the house was, in its origins, an inheritance from a colonial regime - a fact the architecture itself never advertised.

Through the 18th and 19th centuries the house passed through ownership like a parcel. Patras's widow sold it in 1778. Three different owners cycled through in the years that followed. In 1796 Archibald Hope of the Hope banking family - a cousin of Henry Hope and Jan Hope, whose firm financed kingdoms - moved in. Fifteen years later, in 1811, Napoleon and his wife Marie-Louise stopped over in The Hague on one of the Emperor's swings through the territories he was rearranging on the map of Europe. They slept here. One night, between conquests. The inventory was sold off in December 1821 over more than two weeks of auctions. The next owner was the 7th Earl of Athlone, a Dutch nobleman of Anglo-Dutch ancestry. After him came a Foreign Minister with an art collection so large it took an estate sale to disperse it. The Lange Voorhout was a stage. The cast kept changing.

In 1848 the building was bought by Prince Henry of the Netherlands, governor of Luxembourg and the king's younger brother - and after his death in 1879, his widow Princess Marie of Prussia lived in it until about 1885. The decisive turn came in 1896, when Queen Emma - the widow of King William III, mother of the reigning Queen Wilhelmina, regent during her daughter's minority - bought the palace from her sister-in-law. Emma had buried her husband eight years earlier and arranged the country's affairs while her daughter grew up. She had the Lange Voorhout renovated and rebuilt to her taste and moved in as her winter palace in 1901; summers, she stayed at Soestdijk. She remained a presence in the building until her death in 1934. Walk through the rooms today and the wallpaper, the panelling, the staircase you climb - much of it was chosen by a woman who quietly held the Dutch monarchy together during one of its most fragile decades.

After Emma, the palace became the working office for the next three Dutch queens. Wilhelmina used it. Juliana used it - and was the first Dutch queen to start the ride to the Ridderzaal in the Golden Coach from this front gate every Prinsjesdag. Beatrix used it until 1984, when she moved her office to Paleis Noordeinde a few streets away. In 1991 Princess Juliana, by then a private citizen, sold the building to the city of The Hague - on the condition, written into the deed, that it would be used only for cultural purposes. The Kunstmuseum Den Haag held exhibitions in it: Rodin, Venetian glass, Frida Kahlo. None of them stayed permanently. The palace was waiting for the right tenant.

In November 2002 the building opened in its present role: Escher in Het Paleis, the permanent home of the work of M.C. Escher - the Dutch graphic artist whose impossible staircases and tessellated lizards have probably been printed on more dorm-room posters than the work of any other 20th-century artist alive or dead. The collision is delightful. A neoclassical palace built with East Indies money, lived in by a Queen Mother, briefly slept in by Napoleon, now houses an artist whose entire career was about gently breaking the rules that make rooms and stairs and gravity work. The Dutch artist Hans van Bentem hung fifteen chandeliers through the rooms - a shark, a skull, spiders, a sea horse, a starburst doubled by mirrors. Donald Judd designed the parquet floor in geometric coloured panels in 1991-92. The building Emma chose for its quiet dignity now hums with optical jokes. She would probably have approved more than her formal portraits suggest.