My problem is similar. Instant formatting is not working. When I select
Tools → EndNote X5 → Update Citations and Bibliography
Endnote displays a progress bar (Matching unformatted citations) that eventually fails stating " EndNote is not responding. Error = -1712"
I performed this test operation after closing both EndNote (15.0.0.7473) and Word version 14.2.4 (120824), then restarting word and inserting a single citation.
I am running a Late 2009 MBPro 17 w/8 Gb of RAM, under OSX 10.8.2
LIke all others, this is happening during a crippling deadline. I would appreciate a speedy response, of course.