Charles Lang Freer made his fortune manufacturing railroad cars in Detroit, the Michigan Car Company being one of the most successful suppliers to a rapidly expanding national rail network. He retired wealthy at the age of forty-six in 1900. He had been collecting art since the 1880s, beginning with American etchings and prints and gradually expanding to Whistler oils and watercolors. Whistler, the famously prickly American expatriate, took to Freer and the two became close friends. Through Whistler, Freer was introduced to East Asian art, which redirected his collecting toward what would eventually become an 8,000-piece collection of ceramics, paintings, sculpture, manuscripts, and decorative arts from China, Japan, Korea, the Islamic world, and the ancient Near East. Whistler made it clear to Freer that any future home for his Whistlers would have to be in a city where tourists actually came.

In 1904 Charles Moore, an aide to Michigan Senator James McMillan and later the chairman of the United States Commission of Fine Arts, persuaded Freer to leave his collection permanently to the federal government in Washington. Freer offered the Smithsonian his collection, a building, and an endowment, on three unusual conditions: the building had to be designed to his specifications, no items from the permanent collection could ever be loaned out, and only items from his collection could be displayed in his gallery. The Smithsonian initially balked at the loan restriction. The intercession of President Theodore Roosevelt, who knew Freer personally and admired the collection, smoothed the deal. The Smithsonian Board of Regents accepted the gift in 1906. The Freer Gallery is one of the very few American museums that operates under a permanent no-loan policy for its founding collection. The autographed Roosevelt letter inviting Freer to the White House is in the Freer's archive.

Freer hired the architect Charles Adams Platt to design the building, which he wanted to feel Italian rather than imperial: a Florentine palazzo of granite, with a central interior courtyard open to the sky, lined with arcades, and surrounded on the ground floor by galleries arranged for contemplation rather than spectacle. Construction began in 1916 and finished in 1921, though the gallery did not open to the public until 1923. Freer himself died in 1919 and did not see the completed building. The Freer was the first Smithsonian museum dedicated entirely to art, predating the National Gallery of Art (1941) and the Hirshhorn (1974). It opened to the public on May 9, 1923. The collection has grown from Freer's original eight thousand objects to more than twenty-six thousand. The Smithsonian's later Arthur M. Sackler Gallery, opened next door in 1987 with looser exhibition rules to accommodate traveling shows, was administratively joined with the Freer from the start; the two were rebranded together in 2019 as the National Museum of Asian Art.

The Peacock Room remains the Freer's most famous single object. Whistler's original design for Frederick Leyland's London dining room transformed an existing space of carved Cordoba leather and antique shelving into a unified visual statement of blue and gold. Whistler painted four golden peacocks on the wall opposite the window, two facing off in a state of obvious quarrel, in transparent reference to Leyland and himself. Leyland refused to pay the full bill Whistler presented. The two never spoke again. After Leyland's death the room remained largely untouched. When Freer bought it in 1904 and shipped it to Detroit, he kept it intact. Today it occupies a permanent dedicated gallery in the Freer, illuminated by the same window arrangement Whistler designed, the painted peacocks still in their century-and-a-half quarrel.

The Freer collection today spans six thousand years and most of Asia. Egyptian stone sculpture and ancient Near Eastern ceramics, Chinese paintings from the Tang through the Qing dynasties, Korean celadons and ink paintings, Japanese folding screens (including some of the finest Edo-period works outside Japan), Persian illuminated manuscripts including a complete Shahnameh, Indian sculpture, Buddhist statuary from across the silk road, and American art selected to show what Freer believed to be the East-West conversation in turn-of-the-century painting. The Freer and Sackler together hold the largest Asian art research library in the United States. The museum is open every day of the year except Christmas. Admission is free. Over eleven thousand objects from the combined collections are available online in high resolution. Tours are free. The reading rooms are open to anyone who wants to look at Whistler etchings or Chinese bronzes in person.