critics --- some of them anyway -- having racial undertones to their criticisms of him, so I suspect it is them he is referring to. Outside of that context, I can see where it would seem out of the blue or a non sequitur, but he did not "bring it up" so to speak. It has already been "brought up" and that's what he's responding to.