Gabriel's betrayal bothered me too, but I hope the ultimate confrontation with him and the two groups (Rick's and Deanna's) is written intelligently.
Gabriel's claims should be easily shot down. 1) Given how recently he joined the group there's no weight to his claims of everything they've done to survive. 2) Given that Rick's group is traveling with two helpless people (Gabriel and Eugene), and continued to protect them belies the Gabriel's claim that they are evil. 3) Gabriel's own past is the "unspeakable thing" that was perpetrated in the name of survival.
The wild card is the loss of Deanna's son. That combined with her fear of the new group's influence, and Gabriel's betrayal will probably be the central plot device of the last couple of episodes this season...along with Dr. Wifebeater's ultimate comeuppance.
They have been watching them for a while also. They heard/ saw what they do.