Long before the Norse, Celtic Christianity had reached Pabbay. The island still bears a Pictish carved stone from that early period. Hermits and small monastic communities favoured isolated islands across the Hebrides, choosing solitude as a spiritual discipline. The papar were the men the Vikings found here when they arrived around the ninth century, the monks whose presence gave the island its Norse name. Iron Age remains tell of even earlier settlement. Pabbay has been a place of people for thousands of years, but never of many at one time. At only 250 hectares, the island never supported a large population. Even at its peak, the community was small.

The 1897 fishing disaster did not destroy Pabbay in a single stroke, but it removed the foundation. Five Pabbay men were lost, more than half of the island's working male population. Their deaths cast a shadow across every neighbouring island as well. The Mingulay fishermen, watching from twelve miles away, did not forget. Without enough men to crew the boats and bring in the catch, Pabbay could not sustain itself. The remaining families eventually moved away. By the early twentieth century the island had been abandoned. The National Trust for Scotland purchased it in 2000 along with Mingulay and Berneray, and by July 2007 only two sheep remained. With no permanent mammalian residents, Pabbay has become a sanctuary for ground-nesting birds.

Pabbay's principal bay is Bagh Ban, the white bay, on the southeast coast. Well sheltered from the north and west, it was the natural choice for the early settlers. Rosinish, also called Rubha Phabach, is a small headland on the east coast attached to the rest of the island only by a narrow natural arch called Steir. The cliffs are Lewisian gneiss, some of the oldest exposed rock on Earth, dating back nearly three billion years. The same rock that makes the island geologically ancient also makes it a climbers' destination. Pabbay is considered one of the finest sea-cliff climbing venues in the United Kingdom. The Great Arch route, graded E8 6c, is among the hardest sea-cliff climbs in the country.

Pabbay's name is used today for one of the three houses at Castlebay Community School on Barra. The other two are Mingulay and Sandray, three uninhabited Barra Isles given symbolic life through schoolchildren in green and red and blue. The children who carry the Pabbay name to inter-house competitions are descended in many cases from the families who left after 1897. The island is otherwise silent. The seabirds nest on the gneiss. The Atlantic still strikes the western shore. Climbers come in summer and pitch tents on the bay grass. The papar are long gone, and so are the families. The wind carries the name and the rest is the work of memory.