But much more than 5% held the belief that Black were only fit for slavery and that was Blacks caste because they were less than human. I don't think it is a stretch to say slavery was one of the main things the culture was based upon.
No more than the rest of the country. My point is not that the South was somehow noble in its regard for blacks, it is that most of the country held blacks to be inferior, not just the South. Certainly the average white southerner did not regard blacks as his equal, but nor did the average American, northern or southern. This type of bigotry is not equal to support of slavery. The wealthy southern plantation owner may have supported slavery, but that doesn't mean all southerners did. 95% of southerners, and the rest of the country, was simply filled with ignorant bigots.
For example, Lincoln on blacks:
"I will say, then, that I am not, nor ever have been, in favor of bringing about in any way the social and political equality of the white and black races"”that I am not, nor ever have been, in favor of making voters or jurors of Negroes, nor of qualifying them to hold office, nor to intermarry with white people; and I will say in addition to this, that there is a physical difference between the white and black races which I believe will forever forbid the two races living together on terms of social and political equality. And inasmuch as they cannot so live, while they do remain together there must be the position of superior and inferior, and I, as much as any other man, am in favor of having the superior position assigned to the white race."