Human beings need to need. At base, we need food and shelter; once those nneds are met, new needs take their place. A child raised in modern-day America may have everying he could possibly need from a survival standpoint, and may live better than 98% of the world's population, but that still won't stop him from complaining to his mother that he needs to have a t-shirt from the brand all the cool kids are wearing. We will never be satisfied, it is not in our nature.
In addition, I think our need to need creates a myopia concerning the things we lack. What we lack suddenly becomes the most important thing in the world, and at the advanced stages of this myopia, the failures and disappointments of our lives- endemic only to being human, really- can seem entirely attributable to what we lack, or even what we only think we lack.
Ever seen 'Extreme Makeover'? Many times the end results were actually very minor, but almost invariably the recipients believed they had a brand new lease on life. Yeah, they're moving to LA and becoming models! Years of ingrained, cylical thinking had caused them to completely lose perspective. Everything that is wrong is because of this one thing, and once that one obstacle is removed, everything will set straight. I wouldn't mind seeing a follow-up a few months later, when reality finally sets in.
I am certainly familiar with a defeatist attitude, and am certainly familiar with disappointment. Whatever I've been given as far as penis size is concerned, a great lot of good it's done me.
I doubt my words will offer much consolation to the OP, all I can offer him is the assurance that it is certainly a bigger deal in his mind than it is in consensus reality. Perhaps the fact that it is almost never acceptable for a man to display his genitals in public, and the fact that it is not something we discuss openly, precludes the underendowed from adopting a healthy sense of self-deprecation about it, which, to some degree or another, is the attitude adopted by most people with other, more visible shortcomings.