When looking at a reference, whether in All References or in one particular Custom Group, It should be possible to delete that reference from any/other Custom Groups by context-menu (as when adding a Reference to a Custom Group) without need to switch to display of the Custom Group from which the Reference is to be removed.
The right-click context menu to add references to custom group is almost useless because I need to scroll down all the way down if the group name is like “zoological”. This may take a minute with holding the click at the small down arrow. That’s another reason I need structured group.