Yep you should, unless you're an IT guy. Casual Friday, should still be a button up shirt, no tie (or if you have to a knit tie) and a shirt with more texture or pattern in it such as an oxford or some kind of subtle check or stripe that doesnt look like a business shirt. In saying that though, I often wear dark denim, navy blazer, white shirt and black captoes on a Friday if I want to be a bit more casual but still might have to see some clients.
Michael Sy, which one is you? Identify yourself!