I have the same problem in X8.01.
It started after I had installed X8.01 on a laptop and was syncing from my on-line library. After about 1600 refs, most of them with associated pdf documents, the library window disappeared and all attempts to re-open this library failed with the above error message.
Restarting EndNote or restarting the laptop (Windows 10) did not help.
Attempts to use the Recover library function resulted in “0 references recovered” without any other error message.
Repairing EndNote from the installation files also did not help.
Copying the .data and .enl files to another folder and renaming them (a trick which has been reported to help in several cases) also did not help.
Any suggestions?
Finn-Mogens Haug
Professor emeritus
Institute of Basic Medical Sciences
University of Oslo