This solution worked to fix the endless “Formatting Bibliography” window in both Endnote X5 and X6, from Henry LJ’s post on another thread.
"First in EndNote click on Tools > Library Summary. Make a note of the location of your library, for once the preferences have been reset EndNote will not know where your library is located.
Quit EndNote and Word. In the Finder click on Go, hold down the Option key and click on Library from the drop-down menu.
In the Library folder open the Preferences folder and move the following files to the Desktop
Move the following file to the Trash
Go back one level to get out of the Preferences folder and return to the Library folder. Open the Application Support folder, open the Microsoft folder, open the Office folder and open the User Templates folder. Rename Normal.dotm to Normal.OLD and move the file to the Desktop.
Go back one level to get out of the User Templates folder and get back into the Office folder.
Open the Preferences folder and open the Office 2011 folder. Move the following files to the Desktop.
“OLE Registration Database”
“Office Registration Cache”
After that go to /Applications/Microsoft Office 2011/Office/Startup/Word. Move “EndNote CWYW Word 2011.bundle” to the Trash.
Once you have done this restart the computer. When the computer has been restarted go to /Applications/EndNote X6 and double click on the EndNote X6 application to open the EndNote program and let EndNote rebuild its preferences. In the “Get Started” window choose “Open an existing library” and browse to /Applications/EndNote X6/Examples and open the Sample library.
In EndNote click on EndNote X6 > Customizer. When the components window comes up, check the box for Cite While You Write and click on “next.” Click on “next” to go past the prompt for an alternate Cite While You Write path and click on “done” when the process is finished.
Leave EndNote running and open Word to let Word start to rebuild its preferences. Once in Word click on Word > Preferences > File Locations > Startup > Modify. Browse to /Applications/Microsoft Office 2011/Office/Startup. Highlight the Word folder and click on “Choose.” The location of Startup should now say Hard Drive/Applications/Microsoft Office 2011/Office/Startup/Word.
Click on OK to close the preferences and quit Word.
When you reopen Word you should now have the Endnote X6 commands under the Tools menu. You can first try to insert some citations from the Sample library into a new blank document. Once you have verified that the commands are working properly, close the Sample library, open your regular library and insert some citations into another new blank document. Once you have verified that everything works properly in the new document you can open one of your regular documents and test to see if anything is different."