Thomas Whaley designed the two-story Greek Revival house himself. Construction began on May 6, 1856, and the house was completed in 1857 at a cost of more than $10,000 — substantial for the period and the location. The bricks were made in Whaley's own brickyard on Conde Street. The house was furnished with mahogany and rosewood furniture, Brussels carpets, and damask drapes. It was considered the finest home in Southern California at the time of its completion. It was also the first brick house of its kind in San Diego. The Whaley family used it as a home, but it quickly became something more: San Diego's unofficial center of commercial and civic life.

The Whaley House functioned as San Diego's first commercial theater in October 1868, when Thomas Whaley rented an upstairs bedroom to the Tanner Troupe, a traveling theater company. For opening night, the small room held a stage, a few benches, and 150 people — ladies were advised not to wear their hoop skirts or petticoats to allow more room. The theater operator, Thomas Tanner, died seventeen days after opening. His troupe disbanded by the end of January 1869. The house also served as the county courthouse in 1869 and operated as a general store. A single building serving as home, theater, courthouse, and store is unusual in any era; in San Diego in the 1860s, it reflected both Whaley's entrepreneurial instincts and the city's raw, improvised quality.

The Whaley family suffered serious losses in the house. A daughter, Violet, died by suicide in 1885, leaving a note that was a passage from Thomas Hood's poem 'Bridge of Sighs' — the same poem that Edgar Allan Poe had cited as an example of painting with words. Violet's sister Corinne, engaged at the time, had her engagement broken off because of the scandal the death raised. After these events, Thomas Whaley built a separate home for the family on State Street and left the Whaley House vacant for more than two decades. It opened as a museum in 1960, managed by the San Diego Historical Shrine Foundation. It has been maintained by Save Our Heritage Organisation since 2000.

Travel Channel's America's Most Haunted once called the Whaley House the most supernatural home in the United States. Ghost Adventures, BuzzFeed Unsolved, Ghost Files, and numerous other paranormal programs have filmed investigations inside. In 1964, television personality Regis Philbin visited the museum while filming for a local morning show and claimed to have a paranormal encounter with Mrs. Whaley — 'There was something going on in that house,' he said. The ghost of Yankee Jim Robinson, the ghost of Thomas Whaley, the ghost of Violet Whaley: the house has accumulated its spirits over 170 years of San Diego history. Whether or not they exist, they have kept the building crowded with visitors.