The drive is fine. MaiPenRai run a shuttle service (song thaw thing - covered truck with padded benches) at 1pm to the resort - it is a 30 min drive on mainly paved roads with a couple of rutted steep sections.
Perfectly safe - you can even ask to ride in the front.
It used to be a 45 minute "exciting" drive as the road was in a terrible state, but its packed solid now and lots more is paved.
The driver is safe and responsible and yes, he slows right down on the bumpy bits.
If you miss the shuttle service (150b) then I think it is about a 500 baht taxi - hence it keeps the idiots away.