You called me "son" recently. I didn't appreciate it, but I said nothing. I've read other instances where you've called people "son," and they've told you that they don't appreciate it. Despite people expressing disapproval of being called "son," you continue to do it.
In the lexicons of American English and African American Vernacular English, the term is sometimes used (1) by older men addressing younger men, implying the speaker's seniority; and (2) as one of endearment between young Black males and others who imitate hip hop culture, mainly in urban and inner city.
The origin of the term "Son" in the vernacular context was used among American East Coast urban youths as a derogatory term that extended beyond justifying seniority. Often, it was used to claim or instigate one's sentiment toward a rival. The term's derogatory intention began to shift as rap groups like the Wu-Tang Clan used it in their lyrics of the rough ghetto life as a form of endearment. As urban/hip-hop culture has been portrayed as a glamorous subculture to the youths today, the term has been commonly used as playful greeting for those who seek an urban identity to develop their own culture from and will use the term "Son" as well other terms found in rap lyrics like "Nigga", Cuhz (Cousin). Some find differentiation in if the term is derogatory based on how it is enunciated or structured in the sentence, and if it is spoken in ebonics. Mainly, in how the term is pronounced in comparison to the sentence structure as well as the body language (i.e.- gestural, proxemics, etc.,).
As you can see, the word can have both a positive and negative connotation. Nearly each time I've read your use of it, you are bickering with someone. Therefore, it's pretty reasonable for the person you are calling "son" to interpret it in the condescending, not the endearing, way. If you want to hang your hat on you're just being endearing, then that's fine. However, don't be shocked that nobody actually believes you.