Reference keeps reappearing in wrong group

To elaborate, the reference can be added to a desired group, but not removed from the undesired group. I can remove it from the undesired group temporarily by adding it to the trash bin, however, when I add it back to the desired group, it also re-adds itself to the undesired group.