It is feasible but mind that you have to give the rental agency enough time to organise the red tape and insurance for you. It is not possible to do this within just a few hours. No way you could simply take the car and buy insurance at the border. Omani immigration is checking all documents and would easily see if you didn't have the proper documents with you. There is no way you could fix this without the car rental's prior approval.
Typically they will charge you a significant extra fee, and the insurance will also be quite expensive (haven't done it by myself recently so don't ask me how much exactly). An alternative you might wish to think about is to take a cheap flight (flydubai, Oman Air, airarabia) to Muscat and rent from there. One of the cheaper agencies at Muscat airport is Thrifty's where you can book a car through their international website without the slightest hassle.