All grains contain anti-nutrients. Basically it can make it harder for your body to take up the nutrients. Some people are allergic to gluten as well. Many people have also found that wheat in any form makes them hold more water.
If you feel fine eating wheat, by all means continue. I know that I can't take in too much at a time, and if it's oatmeal it has to be cooked. Otherwise my stomach goes crazy and cramps. By cutting out whole grains it's gonna get a lot harder to get a good amount of fiber, but I've found that I do fine on 10-15g of fiber or so a day.