Also on X8 for Mac. I also had the same issue and in spite of multiple restarts, trashing the plist and also resetting the preferences; nothing worked. It rendered my library unusable. The library had so many issues that I got to the point where every launch of the software resulted in the spinning pinwheel of doom (Mac users will understand!). So I ended up going on a rush after launch to change the preferences to not automatically open the ‘problem’ library. Then with the library closed all I could do was to run the recovery option on the library (having closed it). However, the the ‘recovered’ library provided the “open it again later” message, so I disabled all account synchronisation and then I created a new library based on a compressed export. Having done the recovery, this at least allowed me to finally successfully use the compressed library option to export my main group set from the original library as the basis for the aforementioned new library. I was then able to drag and drop some additional references from the original library into the new version.
This process was pretty time-consuming and I have been working professionally with EndNote for years. I wanted to share this in case it is helpful to others. For my part I shall be backing up this particular library offline as the auto-synchronisation account check-in process seemed to preempt the issue (it just seemed to get stuck in an authentication cycle).