How to organise my references

Don’t copy and paste them.  Just drag them.  It will not move them out of another group, but can associate a record in multiple groups.  Copy and paste makes a new copy so it is a duplicate.