Columbus takes no drinking water from the Olentangy. There's an intake on the Scioto along Dublin Rd., and an intake on Big Walnut (I get my water from there, even though at Kenny/Henderson I'm much closer to the Scioto). There's also a wellfield that serves the south side, and according to Columbus' own data, that water tends be free of agricultural chemicals found in the other sources.
The cause for the sewer overflows is rainwater going through the shared storm/sanitary sewers. This works fine until there's too much rainwater, then they system activates relief points. Columbus has a map of them, with their discharge histories:
http://gis.columbus.gov/ssocso/
You can canoe or kayak just about anywhere on the local streams. If you're not on a reservoir like Griggs or Oshay, the streams tend to have pools and riffles and you can expect to have to get out of the boat and carry it over the rocks. On the "downtown pool" you can put a motorboat in at the city power plant, and go as far as the dam at the water plant on Dublin Rd., lots of people fish there. You can also put in at Greenlawn and go as far as the Town St. lowhead.
There are many places along the streams where you'd never know you're in the middle of a city, all you see is wooded streambed.