The remove reference from group is right there in your image, but I think you need to call tech support, as the short cut should not remove it from the library - IF YOU ARE IN THE GROUP that you want to remove it from, it actually asks me if that is what I want to do and tells me that it won’t delete it from the library, just remove it from the group (but you can click a box to make that notification not appear again). Are you then moving it from trash to the new group? That would likely be restoring it to the library and probably restores it with the properties, including the “member of group” property?
You need to go into the group you don’t want it to be in. You can drag it from there to the new group, and then Ctrl D on the copy in the old group or right click and remove from group option.
That option will be greyed out, if you are in “all references”.
