Just picking a fight for the hell of it here, but why on Earth would you want to travel with a tiny bag that can't hold anything useful?
Don't you ultra-light guys ever want to do anything other than sit around in shorts and T-shirt? Don't you ever want to put on a shirt and decent trousers, wear proper shoes? Don't you ever go anywhere nice or meet people other than grubby backpackers? Have you never wanted to pull on your sweat pants and baggy old wool sweater or hoody?
I can understand you don't want to carry a complete set of dive gear, but are you content to use a rented snorkel and mask every time? Don't you ever wish for your own gear when you sail or climb mountains or play your favourite sports? Or don't you do anything when you travel?
Wouldn't it be nice to have space in your pack for a real physical book, and bottle of your favourite hooch for emergencies? Or to cram in a few souvenirs?
Personally, I take a big bag and leave it at wherever my base will be while I'm an area. Then I have my small pack that I can easily live out of for a week or two on my expeditions. For longer-term, after many years of travel experiences, I have concluded that it's good to have more stuff in a store-room somewhere.
But that's just me. I guess it really comes down to how and why you're travelling. So ... how and why are you travelling that you don't need to carry anything?

