Well, in less than 24 hours it is. But it may involve an overnight stay in Koh Kong.
Go to the Trat bus station, there are minivans there that go to the Hat Yai/Koh Kong border. At Koh Kong at 8am every morning there is a ferry that goes to Sihanoukville, arriving at midday. Cost is $20.