For non-touristy stuff...
...well the good thing about Taiwan is that while there are other Asian tourists and tour groups, you're not going to see places saturated with backpackers anywhere. Shi-da has studenty type foreigners but Taipei has no Khao San Road to speak of.
My favorites, some on the tourist route and some not:
1.) Bao'an and Confucius Temples (I second Kaminoge)
2.) National Palace Museum (yay!)
3.) Maokong (good views and good tea with far fewer tourists and such than Yangmingshan) with Zhinan Temple
4.) Jiufen and Pingxi - you could do both in one long day, including a short hike! But if you can only choose one, choose Jiufen as it is more immediately rewarding with amazing views and funky teahouses. The shopping street is touristy but there are other places to wander.
5.) Wandering around, drinking coffee or beer in Shi-da's back lanes (off Shi-da Road)
6.) Raohe Night Market - got a lot to offer but not as touristy as Shilin
7.) Xinbeitou - the public hotsprings or Longnaitang if you're OK with getting nekkid with old folks, or soaking your feet in the warm mineral water stream
8.) Wufenpu Fashion Market - near Raohe, got lots of locals but not touristy per se - best at night, with a lot to see even if you don't buy
Stick something you "have" to see in there like CKS Memorial Hall or Taipei 101 (neither of which are really necessary and neither is worth more than an hour or so and maybe a day trip on the HSR down to Tainan and you've got yourself a stopover.