Looked at Toronto weather and for almost the entire year I'd be good with just my toj1 and a scarf. http://www.yr.no/place/Canada/Ontario/Toronto/statistics.html
Of course it also depends on what you're doing outside. If it's just walking 15-20-30 minutes you don't have that much time to be cold, but if you're standing around outside you do. I wouldn't do manual labor in my toj1 anyways.