The only cheaper option is to take the city buses from Victoria to Swartz Bay (1 hour, about $3), walk on the next ferry (1.5 hours on the water, about $11), then take the city bus from Tsawwassen to Airport station (45 minutes? 1 hour? longer? shorter? $4.50 on weekdays before 6pm, $2.25 after 6pm and weekends). Then transfer on to the B-line bus from Airport Station to downtown (20 minutes to 45 minutes, depending if there's any rush hour traffic). The only issue with that is that there's potentially more waiting involved and it's much more slow going on the bus. If you have luggage with you, you pretty much have to keep it on your lap, in the aisle (hoping it's not full of passengers standing), or if you're lucky, at the very front where there's enough room for 2 suitcases. It saves you money, but probably tags an extra 2 hours on to your journey.
Of course, you could pay $120 for a seaplane ride and get from downtown Victoria to downtown Vancouver in 35 minutes.

Wimipi, let me put my thoughts to you straight.
Victoria is nice for exploring, but the older more interesting area for tourists is small and easily covered in a morning (2-3 hours). Here is a map to give you an idea of the scale. While you are there, take some pictures of the parliament buildings and try to stop in Market Square which is a nice little arts n crafts-y area.
Now, since you don't have much time on the island, I suggest renting a car, and doing a very nice drive out to Tofino. The day after you arrive in the area, head over to Pacific Rim National Park (Long Beach). Even if you don't do anything but hang out on the beach, the atmosphere is nice and you can watch surfers ride the waves or even rent a board and wetsuit and go join them, depending on you fitness level. Tofino itself is worth a walkthrough, there are a few art galleries and things going on there. If you find Tofino prohibitively expensive then Ucluelet is a more than acceptable alternative. Try to spend at least two nights in the area.
And finally, you are right, you don't need to rent a car in Vancouver, but if you are just using it as a stop-through and not planning to explore Vancouver at all (although Vancouver is a great city to explore), then I'd try to by-pass it and take one of those ferries that go straight to Victoria- Seattle is closer to Victoria than Vancouver, after all.

OP - the thing is that Vancouver Island is huge - like 10 hours driving from one end to the other or almost as long as Ireland (both parts), and covers an extent that is 3/4 the size of Switzerland. Most people don't realise how big the Island actually is... Victoria is only the very southernmost tip.
imho Victoria is charming, but if you had more time and a car then seeing other areas would be a great thing - like J_Bone's suggestion.

All,
Once again many thanks for all the feedback.
@Carmanah: I agree, that sounds more complicated then Pacific Coach Lines and I could imaging that you easy could waste more time on waiting for buses/ferries.
@J_Bone: That sounds as a good idea. Most likely I'll do that. By the way, I'll have to get to Vancouver anyway as I'll be going to the Banff and Jasper area later. However, I am not planning on sightseeing Vancouver by car as I find it more stressful to get around in cities, that I do not know, by car than by public transport/taxis.
Thanks again,
Witek