"Endnote cannot open this file"

Not sure, but there are maximums on library size (4Gb I think, which is more than you report) and number of files (65,535 files).  https://support.clarivate.com/Endnote/s/article/EndNote-Upper-size-limits-on-library-files-created-through-the-Compressed-Library-command?language=en_US

Also, I have run into problems if PDF file name are too long, especially if trying to unpack in a deeper folder structure.