Winfield Ebey chose a site on the western shore of Whidbey Island, directly across Admiralty Inlet from Port Townsend, where ferry traffic was constant and travelers had nowhere else to stay. He dismantled his brother's cabins, which had stood on the opposite side of the ravine behind the new building, and used the salvaged one-by-twelve fir planks to construct the inn. The planks run from the bottom of the first floor to the top of the second, an unusual construction method that reflects the era's practical approach to available materials. The structure is a vernacular residence with Greek Revival elements: clapboard siding, a wood-shingled gable roof, and a gable-roofed dormer on the north facade. Like most pioneer buildings, it started small. Additions came as need dictated, and several interior walls still show weathering from their time as exterior surfaces.

The inn's location was its business model. Port Townsend, directly across Admiralty Inlet, was the official port of entry for all of Puget Sound, and every vessel moving through the region stopped there. Travelers heading north to Coupeville, Whidbey Island, La Conner, and points beyond would cross the inlet by ferry and arrive at Ebey's Landing with nowhere to sleep, eat, or resupply until they reached the next settlement. The Ebey Inn filled every gap: rooms for overnight guests, a tavern, a post office, and merchandise for sale. It served ferry traffic for four decades until a new dock was built near Fort Casey at the turn of the twentieth century, redirecting travelers away from the landing. The Ebey family held the property for 57 years before Isaac's grandson sold it in 1917.

The Ferry House has never been modernized. There is no indoor plumbing. There is no electricity. The building stands today almost exactly as it was constructed in 1860, making it one of the most authentic surviving pioneer structures in Washington State. A lightning strike in 1917 ripped the siding from the west wall, destroyed two windows, and knocked out one of the two original interior chimneys. The chimney was replaced with an exterior one that has since disappeared. An upper veranda once extended from a second-story door above the front entrance; whether it blew off, rotted away, or was deliberately dismantled remains unknown. The door opening survives as a window, a ghost feature hinting at a porch that no longer exists. In 1998, a film crew added wallpaper to several interior walls and the front door for the movie Snow Falling on Cedars, one of the few physical alterations the house has received in more than 160 years.

The Ferry House is now owned by the National Park Service as part of Ebey's Landing National Historical Reserve, a 17,500-acre protected landscape established in 1978 to preserve the rural working community of central Whidbey Island. The reserve contains more than 400 historic buildings, but the Ferry House is among the oldest and most unaltered. The building remains locked year-round, though the grounds are open to pedestrians. On May 2, 2010, the Park Service unlocked the doors for a public open house as part of the Seattle-Puget Sound Partners in Preservation Initiative. Visitors could tour only the first floor, but it was the first time in many years that anyone had walked through the rooms where Winfield Ebey's grief became a building, and a building became a business, and a business became a relic that outlasted the world it was built to serve.