Dooley was a Richmond lawyer who made a fortune in railroads and finance, then gave much of it away. By 1912 he and Sallie May already had a mansion in Richmond - Maymont, now a public park. Swannanoa was meant to be a summer place, but Dooley pursued it with the seriousness of a permanent monument. He set the villa on the crest of the Blue Ridge at the border of Augusta and Nelson Counties. It had its own power plant - making it the first house with electricity in Nelson County - along with central heat, a built-in elevator, and a dumbwaiter modeled on the one Thomas Jefferson designed for Monticello, twenty-seven miles to the east. Dooley died in 1922 at 82, leaving everything to Sallie. She died in 1925 at 79. The villa had been occupied only intermittently.

Dooley's sisters sold Swannanoa in 1926 to a Richmond corporation that opened a country club and 18-hole golf course on the grounds in 1927. A small stone payment booth near the course was rumored to also house the region's best moonshine still, with high-placed customers from Washington stopping by during Prohibition. In 1928, President Calvin Coolidge and First Lady Grace Coolidge spent Thanksgiving at the club. The Great Depression starved the country club of customers, and the Dooley sisters took the property back in 1932. It sat largely empty through the 1930s. In 1942, the U.S. Navy considered buying Swannanoa for $200,000 and converting it into a secret interrogation facility for prisoners of war - but decided Congress would never approve buying anything that looked so much like a palace, and chose a Civilian Conservation Corps camp at Fort Hunt instead.

In 1948, the mansion's owners leased Swannanoa for fifty years to Walter Russell, a polymath sculptor, painter, and self-styled philosopher who ran what he called the University of Science and Philosophy out of the building. Russell taught a cosmology blending mysticism, art, and amateur physics. He installed his sculptures in the rotunda and gave lectures beneath the Tiffany window. For half a century, the Renaissance villa on the ridge functioned as a kind of contemplative academy. When the lease ended in 1998, the property returned to its corporate owners, and a three-million-dollar exterior restoration ran from 1999 to 2006. Russell's followers are still active elsewhere, and his books are still in print. The villa, meanwhile, returned to being a villa.

Swannanoa sits at Rockfish Gap, where the Skyline Drive through Shenandoah National Park ends and the Blue Ridge Parkway begins. The Appalachian Trail crosses the gap a few hundred yards away. From the villa's marble terraces, the Shenandoah Valley opens to the west and the Rockfish Valley spreads east toward Charlottesville. Since 2007, Skyline Swannanoa, Inc. has operated the estate as a private venue for booked events and occasional ticketed tours. The general public cannot wander through. But from the air, the white villa is unmistakable on the ridge crest - a Florentine apparition where you would expect only forest and stone.