I'm very much a layman, but to me it looks like they get destroyed defensively when he's off the court. Guys can just run in towards the hoop and also if a shot misses the rebound is generally easy for Adams and Kanter.
I think the 4 fouls early was more of a reason for the loss last game than him playing particularly poorly.
With that said, it's a bit chicken and egg. He has to foul if he's giving up points to easily.