Got the same issue…
I strongly agree that the “Remove” command should be relabeled to “Delete File” (while “Deleted Annotation” should be relabelled to “Remove Annotation” to better match what the user thinks he/she does)
Certainly also enabling deleted pdf files to go into either the Endnote Trash bin or the computer’s Recycle bin to allow for recovery is a must.
These things don’t happen that often but when they do a few hours of work vanish and it seems a very high price when put in the context of changing 2 labels and placing in the Endnote Trash a copy of the reference including the pdf file.