Charles Higgins, Cecil's father, bought the lease for Castle Lane from the Duke of Bedford in 1837. He built the Castle Brewery in 1838 and the family house, Castle Close, in 1846. The brewery prospered, then declined, then closed. The house passed from the family to Bedford Corporation in the 1920s and was used as council offices until Cecil Higgins's will redirected it. The complicated provisions Higgins left behind dictated how the museum was to be run, who would advise on acquisitions, and what kinds of purchases the trust fund could make. The first major decision after his death was made by the Art Gallery Board in 1951: to begin collecting English watercolours. The director of the Victoria and Albert Museum, Sir Leigh Ashton, was asked to nominate an authority to approve acquisitions, and he chose Graham Reynolds of the V&A. From 1955, Reynolds was succeeded by two figures who would shape the collection's identity: Edward Croft-Murray of the British Museum, who handled pre-1850 acquisitions, and Ronald Alley of the Tate Gallery, who handled everything from 1850 onwards. The years between 1952 and 1964 were the most productive period in the gallery's collecting history. Over five hundred watercolours were acquired in twelve years - works by Turner, Cotman, Cox, Girtin, the great names of the English landscape tradition, alongside obscurer figures whose reputations Croft-Murray and Alley felt were worth preserving.

In 1971, the gallery's collecting policy turned sharply towards Victorian decorative arts, on the basis of a single acquisition that came under tragic circumstances. Charles Handley-Read had spent the 1950s and 1960s building, with his wife Lavinia, what would turn out to be the most important private collection of Victorian art in postwar Britain - architectural drawings, furniture, ceramics, metalwork, all bought at a time when Victorian design was deeply unfashionable and prices were low. He had applied for a curatorial post at the Tate in 1950, been rejected, and reacted by deciding to build his own museum in his own home. The 1972 Royal Academy exhibition Victorian and Edwardian Decorative Art, The Handley-Read Collection displayed what they had assembled. By the time it opened, both Charles and Lavinia were dead. They had taken their own lives in October and December 1971, two months apart. The Cecil Higgins Art Gallery acquired more than two hundred pieces from the collection in 1971, and the Handley-Read acquisition transformed the gallery's identity. It now holds nationally significant collections of work by William Burges - the architect-designer of Cardiff Castle and Castell Coch, whose elaborate Gothic Revival furniture and ceremonial objects are some of the strangest beautiful things made in 19th-century Britain - and by Edward Bawden, the 20th-century linocut printmaker whose magnificent prints depict everything from Brighton seafront to Egyptian temples.

Among the more recent additions to the collection is a substantial group of works by Dora Carrington, the Bloomsbury Group painter who grew up in Bedford with her family, attending Bedford High School before moving on to the Slade School of Art in London. The collection includes two major oil paintings, 'Mrs Box' and 'Spanish Boy,' along with drawings of her brothers Noel and Teddy Carrington and a sketch of Bedford Market. Carrington's life was unconventional even by Bloomsbury standards - she had a long and tortured relationship with the homosexual critic Lytton Strachey, with whom she lived for many years, and after his death from stomach cancer in 1932 she shot herself with a rifle. She was thirty-eight. The Bedford collection reaches back behind the tragedy to the years when Carrington was an art student painting her family and the small world of an Edwardian Bedfordshire household. The gallery also holds prints by Picasso, Lichtenstein, Dürer, Whistler, and a 2004 donation of more than seventy prints from the Scottish artist Alan Davie. The print collection now numbers over four hundred pieces.

The 2013 redevelopment was the most ambitious physical change in the institution's history. Three previously separate venues - the Cecil Higgins Art Gallery in the family home of 1846, the Bedford Museum in the 1838 brewery, and the Bedford Gallery in an 1840-41 building originally constructed as a Whig clubhouse and later used as a Sunday school, a billiard hall, and a BBC music department evacuation site during the Second World War - were physically joined together and rebranded as The Higgins. The new name was chosen deliberately to evoke both the family and the building rather than any one of the previous identities. The project cost £5.8 million, funded by Bedford Borough Council, the Heritage Lottery Fund, Bedford Charity and other sources. The reopening in June 2013 created what is now a single accessible cultural building with the Bedford Museum's archaeology, social history and natural history collections on one side, and the Cecil Higgins decorative arts and fine arts collections on the other, all reached through a courtyard entrance in the old brewery yard. Outside, the Castle Quarter of Bedford - with the Mound, the John Bunyan Museum, the riverfront Embankment - has been knitted together as a cultural quarter that did not quite exist a generation ago. The Higgins is its centrepiece.