Talk about Gentoo, I started using it since the day that you have to install everything from source and config everything manually. Yeah, my machine took nearly 2 days to compile everything (8 years ago I think). Don't use Gentoo anymore, but must admit that I have learned from that experience a lot.
If you want to take that a step further, you can do Linux From Scratch. I wouldn't use an LFS system for regular use because package management would be a headache, but it's a useful exercise to go through at least once.