When Edward Akroyd bought Bankfield in 1838, on his engagement to Elizabeth Fearby of York, it was a modest eight-roomed house. Akroyd was the son of Jonathan Akroyd, the rich Halifax worsted mill owner. He and his brother Henry worked in the family business and lived at Woodside Mansion in Boothtown. When their father died in 1848, Edward's inheritance funded an ambitious expansion: he encased the original 18th-century building in fairfaced stone and added two loggias, a dining room, an Anglican chapel, and kitchens. By 1867, when he was Member of Parliament for Halifax, he needed something even grander. He commissioned John Bownas Atkinson of York to design a new wing at a cost of £20,000. The future Edward VII had come to Halifax in 1863 to open the Town Hall, and Akroyd, who had not had the space to host him for dinner, made sure he would never be embarrassed that way again.

The library at Bankfield was originally also the smoking room, which makes it all the more remarkable that the original painted ceiling has survived. It is heavily painted, gilded, and sculpted in eclectic Victorian neoclassical style. The background is cream; the dominant accent is Pompeiian red, an earth pigment that surged into fashion after Giuseppe Fiorelli's renewed excavations at Pompeii in 1860. Four medallions show the named poets Shakespeare, Milton, Chaucer, and Tennyson, all heroes of the Romantic movement. Stylised goats and fauns share space with cornucopia, grapes, birds, flowers, angels, and putti, all painted to imitate Italian maiolica. A Green Man peers from the decorative edge. Small panels carry classical motifs and portraits on deliberately distressed backgrounds, made to look like fragments of ancient fresco. The whole ceiling has been carefully restored.

When Henry Ling Roth took over as curator of the Halifax museums in 1900, Bankfield was, in one contemporary description, "a small, confused, unattractive museum." Roth was an anthropologist with a passion for textiles. Over twenty-five years he classified and rearranged the collections for educational purposes, focusing on the peoples of the world and the technology of cloth. He brought in textile machinery from Calderdale and displayed an old spinning jenny that had been in use at Dobcross until 1916. He acquired spinning wheels, looms, and textiles from across Asia, Africa, and the Americas. He published 23 numbers of the Bankfield Museum Notes. By the time he was commended in 1916, Bankfield had become "an important centre of spectacular interest and research." The textiles gallery remains the museum's anchor.

The Duke of Wellington's Regiment Museum, housed in Bankfield since the regiment's headquarters moved nearby, tells the story of a regiment that traced its beginnings to 1702, when it was the Earl of Huntingdon's Regiment of Foot. It absorbed the 33rd Regiment of Foot, of which Arthur Wellesley, the first Duke of Wellington, had been colonel; the regiment was renamed for him when he died. The 76th, with which it later combined, had served in India and carried two stands of colours, including an honorary stand from the East India Company, so the combined regiment carried four colours on parade. Battle honours range from Dettingen in 1743 to the Hook in Korea in 1953 to a Theatre Honour in the Iraq War in 2003. In 2009, the regiment was amalgamated into the Yorkshire Regiment. The museum was reopened in two phases, in October 2005 by Lady Jane Wellesley and in November 2008 after further Heritage Lottery Fund grants. Edward Akroyd himself had paid in 1860 to raise the 4th Yorkshire West Riding (Halifax) Rifle Volunteers, the local ancestor of the regiment.

The house sits in Boothtown on the hillside above Halifax, a Grade II listed building set in its own grounds. Admission is free, supported by Calderdale Council. The museum hosts rotating temporary exhibitions, including textile shows and educational programmes for Key Stage 2 and 3 pupils. The textiles, the regimental galleries, and the restored Victorian rooms are the core attractions, but the building itself, with Akroyd's ambition still visible in every cornice, is arguably the largest exhibit. The Akroydon model worker housing that Edward built down the hill still stands too, a reminder that this was a man who tried to build not just a house but a whole industrial society around it.