The name has confused people for two centuries. Rufus suggests William II - William the Red, son of the Conqueror - and the castle was reportedly built for him. The romantic version held since the late 18th century. Modern historians offer an alternative candidate: Gilbert de Clare, 7th Earl of Gloucester, called the Red Earl for the colour of his hair. Either way, the man whose name the castle carries had nothing to do with the stones standing today. What you see now is the second castle on this rock, raised between 1432 and 1460 by Richard, Duke of York. The first, recorded in records of 1142 when Robert, Earl of Gloucester wrenched it from King Stephen for the Empress Maud, has vanished into the cliff. The sea takes a foot of Portland every century or so.

The castle has two names, and they catch the awkward moment when castle design changed forever. The pentagonal tower carries five embrasures cut through walls more than two metres thick - circular gun ports for early cannon, not the narrow slits an archer would use. Late medieval gunners stood here. But generations of visitors who saw the embrasures from a distance assumed they were arrow loops, and the second name stuck: Bow and Arrow Castle. Three of the five sides are noticeably longer than the others, giving the building its distinctive irregular footprint. Stone corbels in groups of three run along the north and west walls, the surviving fittings for a parapet that has long since fallen. The walls of roughly squared rubble carry no roof and never quite did - this was a fighting platform, not a residence.

Between 1797 and 1800, the politician and writer John Penn built Pennsylvania Castle next door, a Gothic Revival mansion overlooking Church Ope Cove. Penn's estate took in the medieval ruin and the older parish church of St Andrew's that lay below, and Penn went to work making them picturesque. He threw a round-arched stone bridge across Church Ope Road - it is still there, listed Grade I as part of the castle. He punched two large new openings through the ancient walls: a rounded arch in the north elevation and a Tudor-pointed arch in the south, replacing the original doorway. What had been a working fortress became part of his garden composition. In 1989 the seaward arch fell into the sea, taking a piece of the keep with it.

J.M.W. Turner painted Rufus Castle around 1811, and in a rare hesitation drew the arch twice on the same sheet - the upper version giving the ruin more drama than the actual stone offered. His watercolour now lives in the Victoria Gallery and Museum at the University of Liverpool. Thomas Hardy used the place as the Red King's Castle in his Portland novel The Well-Beloved. The early chapters of Victor Hugo's The Man Who Laughs unfold along this same stretch of cliff. The castle is privately owned and closed to visitors, but the public footpath along the coast gives an excellent view. In 2010 English Heritage spent 150,000 pounds shoring up the north walls. The work was done quietly, by a small team of historic-building specialists, and finished by November of that year.

Stand on the path beside the castle and the geometry of this corner of Portland clicks into focus. Below you, the green of Church Ope Cove cuts a wedge into the limestone. Beyond it, the lower ruin of St Andrew's church - parishioners had to abandon it in the 18th century because the cliff kept moving. South, the ridge runs out to Portland Bill, with the lighthouse just visible. Northeast, Weymouth Bay opens up, the harbour and beach distant pale lines. Three miles east of where you are standing, the Shambles still lurks; in heavy weather you can sometimes see the swell break over it. The castle, listed Grade I since 1951 and one of only three Grade I structures on Portland, simply continues its long watch.