Colonel John Browne built the first Westport House around 1670. He was Catholic and Jacobite, supplying cannonballs from his iron mines to King James II's army in the Williamite Wars. The Catholic side lost at Aughrim and Limerick in 1691, and Browne lost most of his estate in the confiscations that followed. His grandson, also John, did what so many Anglo-Irish Catholics did under the Penal Laws: he converted to the Church of Ireland to save what was left. He then rebuilt the family fortunes spectacularly, hiring the architect Richard Cassels to design a new house around 1730 using locally quarried limestone. He became the 1st Earl of Altamont in 1771. His son married into Jamaican plantation money. His grandson was made the 1st Marquess of Sligo in 1800. The pirate queen's bloodline had become the Irish peerage.

Howe Peter Browne, the 2nd Marquess of Sligo, inherited Westport House in 1809 at twenty-one. He was a friend of Lord Byron and the Prince Regent, a member of the most reckless generation of Regency aristocracy, and obsessed with classical antiquity. On a tour of Greece, he excavated at Mycenae and uncovered the three-thousand-year-old columns of the Treasury of Atreus. He wanted to bring them home to Mayo. To do so, he poached two British naval seamen off their ship in time of war, an act of treason, and was sentenced to four months in Newgate Prison when he got home. The columns made it to Westport, where they were eventually returned to Greece. Howe Browne also added the north and south wings, the library, and the painted drawing-room ceiling at Westport House, then turned his attention to politics.

In 1834, Howe Browne was appointed Governor of Jamaica. The Slavery Abolition Act had just passed, but the law replaced outright slavery with a transitional 'apprenticeship system' that kept the same people working the same plantations under almost the same conditions. Sligo was supposed to administer this halfway state. He met fierce opposition from plantation owners and most of the colonial establishment. He pushed back hard, and in 1838 he became the first major Caribbean landowner to free the enslaved people on his own family's plantations, ahead of full emancipation. The freed workers founded the first 'free village' in the world and named it Sligoville in his honour. The town still exists, a small community in Jamaica's St. Catherine Parish, still bearing the name of the Mayo Marquess. The story is more complicated than the statue would suggest. The Brownes had owned enslaved Africans for generations through the family's Jamaican holdings, inherited via Howe Browne's grandmother Elizabeth Kelly. The 2nd Marquess accepted substantial compensation from the British government when slavery ended. One historian has called the 'Champion of the Slaves' exhibition at Westport House 'hyperbole', noting that Sligo 'benefited from slavery from the cradle to the grave and did not free his slaves until the institution of slavery was abolished by an act of parliament'. Both things are true. The history does not resolve cleanly.

When Howe Browne died in 1845, his son George inherited Westport House and almost simultaneously, the Great Famine. The west of Ireland was hit hardest. Westport quay became the funnel through which George imported cargoes of meal for the starving tenants on his estate. He spent £50,000 of his own money on famine relief, an enormous sum, propping up the local workhouse and writing pamphlet after pamphlet to the British government demanding more action. The famine killed roughly a million Irish people and drove another million abroad. In 1854, the British government offered George the Order of St Patrick, an honour his father and grandfather had both held. He refused it. 'I have no desire for the honour,' he wrote, disillusioned with how Britain had let Ireland starve. He left Westport for Surrey when he married Isobel Peyronnet in 1878. She did not care for the country. He never went back.

The Brownes owned Westport House for almost three hundred years. The 11th Marquess, Jeremy Browne, died in 2014 and left the estate to his five daughters by means of private legislation passed by the Seanad in 1993, bypassing his cousin Sebastian who inherited the title but nothing else. By 2015 the estate was in NAMA for ten million euros of debt. On 17 January 2017, the Browne sisters sold Westport House to the local Hughes family, owners of the Portwest workwear company, ending three centuries of unbroken family residence in a single transaction. The Hughes family committed fifty million euros to refurbishment. The mahogany doors are still Jamaican. The original Joshua Reynolds and William Beechey portraits still hang in the dining rooms. The Mayo Legion Flag brought by General Humbert in 1798 still survives. The house is open to the public, and the family who lived in it for ten generations have stepped quietly out of the frame.