All references are, well ALL the references? It wouldn’t make sense to remove references that have been moved to a group?
There is an unfiled references group for that?
You can right click and hide selected references - but then you are seeing “showing references” rather than all references, and it will not be saved that way.
You can create a group that is all references, minus specific groups to create a smaller list. If it were minus all groups, you would end up essentially with the “unfiled” group that is already there.