Architect Gordon Bunshaft of Skidmore, Owings and Merrill designed the museum in 1969 as an open cylinder of pre-cast concrete elevated on four massive piers. The drum is eighty-two feet tall and 231 feet in diameter. Its hollow center forms a circular interior courtyard with a fountain at the bottom. The galleries occupy two ring-shaped floors around the courtyard, lit by a continuous band of windows facing inward. Bunshaft wanted a building that would not compete with the artwork. Critics initially loathed it. The Washington Post called it the bunker. Architectural critic Ada Louise Huxtable called it a maimed monument. Time has been kinder. The Hirshhorn building is now generally regarded as one of the most successful pieces of mid-century modern museum architecture in the United States, and the rounded form continues to surprise visitors who approach the Mall expecting another rectangular neoclassical box.

Congress had been authorizing a federal museum of contemporary art on the Mall since the late 1930s. The National Gallery of Art, dedicated in 1941, was largely a museum of older European masters, with no remit for modern American or international work. The need persisted but no collection had been found. Then in 1962 the Guggenheim Museum in New York hosted a sculpture show drawn entirely from Joseph Hirshhorn's private collection. The art world realized for the first time how vast and how systematic the holdings were. Hirshhorn had been buying both classic moderns (Rodin, Picasso, Matisse, Brancusi, Giacometti) and contemporaries who interested him (de Kooning, Calder, David Smith, Henry Moore). Cities and countries began bidding. Israel, Italy, Canada, California, and New York all made offers. Lyndon Johnson personally lobbied Hirshhorn at the White House. Hirshhorn agreed to give the collection to the United States. The deal was signed in 1966. Most of the money for the building came from federal appropriations. Hirshhorn himself contributed a million dollars.

The recessed garden across Jefferson Drive from the main building was designed by Lester Collins and dedicated in 1981. It is sunken about ten feet below the street level of the Mall, creating a sheltered space for sculpture that is partly hidden from passing pedestrians. Major large-scale works by Henry Moore, Auguste Rodin, Aristide Maillol, Joan Miro, and David Smith stand in carefully placed groupings. The Rodin Burghers of Calais is one of the casts authorized during the artist's lifetime. The Henry Moore King and Queen has become one of the best-known photo subjects on the National Mall. The garden was redesigned by artist and architect Hiroshi Sugimoto in a major renovation—the first significant change to the layout since 1981—with construction beginning in November 2023 and the revitalized garden set to reopen in October 2026. The redesign widens the Mall entrance, doubles the number of access ramps for accessibility, and adds new seating and shaded areas.

The Hirshhorn now holds about twelve thousand objects spanning the late nineteenth century to the present. The focus is on art made since World War II, with significant collecting in the contemporary moment. Major holdings include Edward Hopper, Mark Rothko, Willem de Kooning, Helen Frankenthaler, Jackson Pollock, Mary Cassatt, Alice Neel, Lee Bontecou, Eva Hesse, and Cy Twombly. The sculpture collection has the most depth: 660 sculptures by Auguste Rodin alone, including many of the small studies that lay behind his larger commissioned monuments. The Hirshhorn has been particularly aggressive about acquiring work by women, by Black artists, and by international contemporaries since the 2000s. Melissa Chiu, named director in 2014, oversaw the expansion of the collecting agenda into Asian contemporary art, including major acquisitions of work by Cai Guo-Qiang, Ai Weiwei, and Yayoi Kusama.

In 2009 then-director Richard Koshalek announced an ambitious plan to install a 145-foot inflatable structure called the Seasonal Inflatable Structure across the museum's exterior. Designed by the firm Diller Scofidio + Renfro, the Bubble would have temporarily transformed the cylinder into an even more striking visual presence for several months each year. The project ran into funding problems and public opposition. The Smithsonian Board of Regents voted in 2011 not to proceed. Koshalek resigned in 2013. The Hirshhorn entered a quieter period under his successors. The museum today functions as one of the most-visited contemporary art museums in the United States, with about a million annual visitors moving through the round galleries. Admission is free. The Yayoi Kusama Infinity Mirror Rooms—two of which entered the permanent collection after the landmark 2017 touring exhibition—attract hours-long lines on busy weekends. Joseph Hirshhorn would probably have approved.