Simple, cheap but high-res camera.

The Olympus SZ range uses the standard small 1/2.3" sensor crammed with way too many pixels.
The one in the Canon S95/S100 is no real biggie either but it is about twice as large. This results in much better image quality even at low sensitivity/ISO - and higher price of the camera itself.

This has nothing to do with Canon or Olympus or any other brand in fact, if you compare a small sensor size Canon it results will be similar.

Samples from the Comparometer (http://www.imaging-resource.com/IMCOMP/COMPS01.HTM):
The first camera is the Canon S95, the second the Olympus SZ-30, image at 200 ISO:
http://s181.photobucket.com/albums/x270/egoipse/?action=view&current=S95-sz30.jpg

As pizwat says, it's not about megapixels and the biggest zoom when it comes to image quality, but much more about sensor capability (and how a company extracts the info and converts it to an actual image).
The short rule is the bigger the sensor the better the image (of course the lens in front of that sensor needs to be good too, but that's not the problem on those compact cameras these days, the sensor is usually more the limiting factor on compacts).

Here is the S95 vs the D90, a dSLR from 2008:
http://s181.photobucket.com/albums/x270/egoipse/?action=view&current=S95-D90.jpg

If there is less light available, not sunny bright conditions, and you need to crank up the sensitivity the difference starts showing even more. Here the Olympus SZ-30 vs the Nikon D90 at 800 ISO:
http://s181.photobucket.com/albums/x270/egoipse/?action=view&current=SZ30-D90.jpg

So if you are looking for improved image quality go for a second hand dSLR, if not something like a Canon S95 which uses the larger 1/1.7" size sensor.
Sensor sizes:
http://upload.wikimedia.org/wikipedia/commons/f/f0/Sensor_sizes_overlaid_inside.svg
Most dSLR's use the APS-C size in this illustration, btw.

When it comes down from converting the data collected by the sensor into an image Canon uses very effective algorithms, Olympus does quite well too, but others aren't.

I don't own a compact camera btw, I prefer the vastly superior image quality of dSLR's, Nikon in my case. However if I would buy any compact I would go for one of the S-series Canons, saw a second hand S90 for 170$ recently and have to say I was tempted - didn't go for it as I know I would not have been happy with it in the long run as even if it is good for a compact it still is a big step down from the dSLR I'm used too.

As said, make up your own mind, samples of differences in image quality are all there, head over to comparometer and check out various cameras.

Dpreview has a comparing section too:
http://www.dpreview.com/reviews/q42010highendcompactgroup/12

If you want an Olympus a comparable model would be the XZ-1 - which comes once more with a "large size compact" 1/1.6" size sensor - and a price tag as well.
++

Beautiful camera the SZ - though, even though it'll be upgraded soon.
The S-90 is still plodding along nicely, 4 years on. That says a lot about it - huh?
Much nicer looking than the Canon (very 70's looking) - a packet cheaper too.
Many more features.
And of course, many photo clean up software kits, take care of lacking features nowadays.
Couldn't be without the 'Panoramic' and '3D' on the Olympus - either. They're lush.
And of course, OP is asking about a 'Cheap' camera.
Olympus wins that one - too.
Good to choose though.
Read what the experts say....

http://www.dpreview.com/news/2011/2/8/olysz10

http://www.dpreview.com/reviews/canons100/14

Good reviews for both - and sure, pro's and cons for both.
You choose.

4 years on? That's the Nikon D90.

The Canon S90 is 3yrs (2009), the S95 is 2yrs (2010) and the S100 is less than 1yrs (December 2011).
And yes, all of them beat a small sensor camera like the SZ series (even the newest one) in terms of image quality.
A second hand S95 might be better for OP because of budget limits.

No "photo clean up" software can put back detail into a picture which the sensor failed to capture.
The "many more features' will not give you a single improvement in basic image quality delivered by the sensor.
And OP is looking exactly for that one thing - image quality.

Looks? Now that's a rather personal thing; I much prefer simple (retro) designs than those spaceship looking ones.
However has nothing to do with image quality at the end.
+++

No "photo clean up" software can put back detail into a picture which the sensor failed to capture

Why is that?

Retro indeed - that, the Canon certainly is.
OP is looking also for 'Cheap', as mentioned in OP.
Gimme the extra Olympus features (including a proper optical zoom) any day.
Close call, I'll give you that....
Until prices and all the Olympus's bucket load of additional features are brought into it.

• Why is that?

Astonishes me that you have to ask that.
Because if there is no detail in an image file you can't put in back in there anymore?

You simply can't go from left to right with the help of software:
http://s181.photobucket.com/albums/x270/egoipse/?action=view&current=SZ30-D90.jpg

• Gimme the extra Olympus features (including a proper optical zoom) any day.

For you yes, but you don't care (or can't distinguish) about image quality. OP does care, he/she wants higher image quality.

I wouldn't call it a 'close call' - any 1?1.7" size sensor camera is clearly the winner looking at image quality. Of course it comes at a higher cost, that's why OP might better look at second hands. And yes, those seconds hands will still get better image quality than a new 1/2,3" inch sensor camera.
Once more, the additional features, including a 'proper zoom' (whatever that is?) does not improved image quality per se.

A second hand dSLR might be the best bet for OP actually as it delivers best image quality. By a wide margin.
++++

Where do the Canon Sx200-210is fit into this equation?

SX 210 has the same smallest size 1/2.3" sensor like the Olympus SZ range. Big zoom, IS and whatever, but lacking in image quality in comparison.
++++

Whereas the Canon S-range lacks any zoom at all - as such.
They all have their good and bad bits.

OK - ish, I suppose, bit still lagging behind lots of others - and very small in comparison to many.
And the OP is looking for a cheap camera.
Both those Canons aint cheap in comparison, to better others.

• lacks any zoom at all

Wrong, they do have zooms as well:

28-105mm on the S95
24-120mm on the S100

What they don't have is ridiculous 500mm zooms that require 400 ISO or higher on a bright day (despite IS) which already considerably degrades the image quality on those tiny sensor cameras as seen above.

Yes they all have good and bad bits.
You can't have the biggest zoom with a superb sensor in the smallest package at the lowest price.
If it sounds too good to be true...
+++