It may, but it would probably be okay to wear.
I don't see it as 'informal," per se, although it could possibly be seen to detract from the technical formality of the tie.
If I were to take a chance and choose one side, I would be in favor of the sweater.
There may be people at your job who are used to seeing the components of the dress code, and judging it based on their presence rather than the way they are worn or presented. For them, it might seem "off" somehow, since the sweater would be covering the body of your shirt and a good portion of the tie blade.
My personal opinion is that some sort of garment should be worn over a shirt and tie, since the combination is meant to be worn under something else. A sport coat or blazer would be the most appropriate, but since those are not required, the next best thing would be a V-neck sweater of exactly the type you have mentioned. It will streamline the ensemble, making the end result far more cohesive and complete. The "V" is good, as well, because it allows your tie to show, demonstrating that you have indeed incorporated the required components.
Medium to dark grey, navy, burgundy, charcoal, and tan are all potentially acceptable colors. Charcoal is probably the safest. Patterns and holiday imagery are best avoided unless golfing or attending a holiday party. Argyle, if not very carefully chosen and worn in the appropriate situation, will look uninspired or trite. A solid, subdued V-neck sweater is generally going to be well received, though.
You should give it a try. The worst that can come of it is that you'll be looked at a little oddly, and maybe asked to take it off, but that seems unlikely. The dress code you describe seems to align most closely to "Business Casual," in which a discreet sweater is usually just fine.
-VF