When the house was first built, it stood at the waterfront. The Grote Aa, a tributary of the Zwarte Water, ran through Zwolle then, and ships moored just outside this address to be weighed and to portage between river systems. The old weigh house stood nearby. In the early 17th century a merchant named Hendrik Wolfsen expanded and improved the facade, probably to his own specifications. His daughter, the painter Aleijda Wolfsen, was born here on 22 October 1648. After cholera outbreaks in the 19th century the city filled in the Grote Aa entirely, and Melkmarkt became a street instead of a harbour. The house that once looked out at moored cargo ships now faces shop windows.

Aleijda Wolfsen kept painting after her marriage, working from a studio added to the house during her lifetime. Her sitting room, the Grote Sael, was fitted with a carved mantelpiece, and the hallways got marble floors and carved woodwork. Then in 1686, something rarer happened. A small painting school opened in the house under the instruction of Wilhelmus Beurs, and his pupils were four women: the half-sisters Cornelia van Marle and Aleida Greve, along with Sophia Holt and Anna Cornelia Holt. Beurs published a book on painting in 1692 and dedicated it to these four students. Its frontispiece shows three women in a room contemplating three paintings the same size as the ones they had made under his teaching. Works the women produced in 1686 still hang in the regent's room today.

Aleijda Wolfsen died in 1692 giving birth to her fifteenth child. The historical sources note her age and circumstance briefly, but the line carries the weight of how often early modern women's lives ended this way. In 1706 her former pupil Aleida Greve and Greve's younger sisters bought the house from Wolfsen's heirs. Greve eventually wrote the will that locked the sitting room in place. After her death the property became the Vrouwenhuis, the Women's House, an old age home for women that gave the building its current name. Since 1987 the ground floor and the oldest sections function as a museum, while some of the former ladies' rooms upstairs are rented to younger women. The continuity is unusual: a house owned by women, lived in by women, and now still partly occupied by them.

The art on the walls is the heart of what makes the Vrouwenhuis significant. Paintings by Greve, by the Holt sisters, by van Marle, and quite possibly some left over from Aleijda Wolfsen herself, hang in roughly the arrangement Greve fixed in her will. Whether each canvas was painted in the house, collected elsewhere by the sisters, or inherited from Wolfsen is not always clear, and the historians who have studied the collection treat that ambiguity as part of its texture. The most famous artist represented apart from the women themselves is Peter van den Velde, the Antwerp marine painter, whose signed View of Gibraltar hangs in the regent's room along with two more marine views likely from his hand. The museum is small and accessible only by special request, which keeps it close to what Greve intended: a private room kept private.