Yemen and Oman are interesting, though Oman isn't really a budget destination.
In Yemen, the area around Sana'a is completely safe, but other bits (Marib, the North) really should be avoided at the moment. The most interesting areas are relatively safe, though: Wadi Hadramawt, the mountains between Sana'a and Mukalla, and the towns on the way down to Aden (also Socotra Island, though this is a bit more expensive). Outside of Sana'a, you'll need (easily acquired) permits. If you're interested in Arab culture, though, Yemen is far better than the rest of the Peninsula, since Yemen doesn't have the enormous guest worker population of, for instance, the UAE; old city Sana'a is by far the highlight. Tourist infrastructure is fairly well-developed. Yemen is also good in the summer, since Sana'a is in the mountains and can be 20 or so degrees cooler than Dubai or Muscat.
Realistically, you'd be flying from Oman to Yemen, though - it's a long 1000km or so from Muscat to Salalah near the Yemen border, then another (somewhat dangerous) drive to Sana'a. The cheapest route is Air Arabia via Sharjah; I flew into Dubai, then bussed into Oman, rented a car for a few days to see the more remote sights, then bussed back to Sharjah to catch my flight. If you want to visit Musandam, there are no busses that I could find patrolling the coastal UAE route through Khor Fakkan down to Oman; I was able to get a ride to the nearest Oman town from the border from a friendly local.
Oman is a bit duller - it's much more developed than my Thesiger-infused mind would have supposed. The beaches in Muscat are quite nice, and the area near Mutrah and its souks is fun to walk around. There are some interesting (though not well guided/signposted) forts up near Nizwa, and some great small towns in the Jebel Akhdar (particular Misfah al-Ibrayeen). I also had some good experiences hiking the edge of the desert out near Wahiba. English is widespread, and travel is easy, aside from the need to rent a car to visit many areas that are not well-served by public transit. If you're coming from the UAE, you'll find its easy to get around on public transit.
As for other countries, how about Brazil? South Africa (including Lesotho and, to a lesser extent, Swaziland) also might be a good fit.