Three spots on the island offer great swimming. To get to Faraglione, signposted left from the port, walk 1km along the road west of town until you see a couple of rocks sticking out of the water just offshore. For something quieter and more remote, take the 4km cross-island trail to Capo Grosso on the far northern shore, where there is also a lighthouse.
Alternatively, take a right out of town and walk along the dirt road. The road forks 300m past the first bend; take the rocky path down towards the sea and keep going until you get to Cala Minnola, a small bay with crystal-clear water where, outside the month of August, you can swim in peace and tranquillity.