In colonial times this small fortified harbour island was a sorry site of suffering, used to isolate recalcitrant convicts and nicknamed ‘Pinchgut’ for its meagre rations. Fears of Russian invasion during the Crimean War led to its fortification. The NPWS offers tours of the Martello tower (prebook online), although plenty of people just pop over to have lunch at the outdoor restaurant. Without the tour, the exhibition is a little desultory and you can't access the tower.
Both Captain Cook Cruises and Manly Fast Ferry/Eco-Hopper have several services per day heading to the island from Darling Harbour and Circular Quay. The ticket includes the island access fee. Private boats aren't allowed to land. The restaurant was closed at time of research but due to reopen in 2018.