Endnote can't import

[rani]

I am very frustrated with responses from Endnote support regarding file corruption, where endnote refuses to import files or else copy files to another database - which is only an issue because your files get so bogged down and impossible to use when they’re large - on the basis that an attachment is incorrect.

Endote support diminishes people who don’t feel like checking each and every citation to identify the bad attachments but I say YOU try doing this when you have 80,000  citations in it.   One by one, figure out why your software went wrong?

I have repeatedly tried to fix it with the “file recovery” utility, but that makes no difference.  Why the hell not?  Why can’t you set up a utility for something like this, rather than expecting us to spend a week trying to clear up problems that you could have fixed?  

I hate it that I have to deal with multiple files because I don’t have time to wait 60 seconds for each transaction to process.  (I have a fast computer, Win7 64 bit and 16 GB of RAM and giant hard drives so it’s not my computer, it’s your software).\

Why on earth are you leaving us to deal with the problem when it should be pretty straight forward for you to fix this for us?

Rani

[crazygecko]

Sorry to hear about your situation.  Am curious, however, when you say “endnote refuses to import files” what type of files (e.g., program, file extension) are you trying to import into EndNote?  Also, given the 80,000+ citations did you test importing a smaller subset to see if the import works/not work?

[rani]

OK, I looked at one of my smaller Endnote databases (why do we even have to resort to multiple databases we can’t access at the same time?) and I haven’t seen a single Endnote pdf sub-folder with multiple files.

I’m guessing the problem is either with Endnote’s dealing with umlauts on top of author names, or else it has to do with recording of multiple downloads that had to be redone.

Either way, I think this is something Endnote should be responsible to fix without giving us long explanations of workarounds that take us a bunch of days to accomplish.

Rani

[rani]

Thank you so much for replying.  I am particularly frustrated because I’ve just realized that every single one of my Endnote databases - which are an artifact of Endnote not being able to deal wtih really large databases - is corrupted, and won’t import.

Now, back to your question (and thank you!)  When I say that Endnote refuses to import files, I am referring to this process:

1.  First, in my new emergency mode, I took every one of my files - even the ones with only 225 citations with pdfs in it - and went through the recovery mode.

2.  Then, I create a new Endnote file (childergo.enl) to import the different files into

3.  Then I click import → file → and point to the supposedly recovered file to import.

4.  Then I get a message from Endnote X4

Could not copy reference

A file associated with the “file attachments” field of the reference is missing from the source database."

And it asks me to click “OK”

5.  Since that time, I’ve gone through every single subfolder for one of the smaller databases - and please keep in mind the only reason I have multiple databases is because Endnote is so poor at handling large databases - and checked, one by one and none of the subfolders have more than one file in it.

But my point is that I have not changed a single file name or subfolder name.  Why is Endnote making this so difficult for us to figure out a fix?  My next step is to go one by one through the Endnote citations directly - opening the citation page one by one within Endote to look for multiple partially downloaded files where only one works in the subfolder. 

But why is it my job to do this?  It’s a huge job, especially for my biggest Endnote database, which has 115,000 citations.  Do you honestly expect us to struggle with a workaround when this would be a pretty straight forward utility for Endnote to develop to fix the problem?

I apologize profusely if you don’t actually work for Thompson.  I just got tee’d off by the rather lackadaisickal response to someone who did seem to work for Endnote who had the exact seemingly insurmountable problem that I’m struggling with .

Best, Rani

>>Sorry to hear about your situation.  Am curious, however, when you say “endnote refuses to import files” to inquire what type of files (e.g., program, file extension) are you’re trying to import into EndNote?  Also, given the 80,000+ citations did you test importing a smaller subset to see if the import works/not work?>>

[rani]

In case I didn’t answer your  question, I am referring to a variety of types of attempts.

It does not work when I try to import an Endnote file - as an Endnote file, regardless of the number of times I’ve tried to recover it through the Endnote utility.

It also does not work when I try to copy the citations from one database to another, whether it be highlighting all of them (Ctrl-A) or partial copy and then trying to paste to another database.

And it does not work when I highlight citations, right mouse-click and select copy to another database.

And it does not work when I try to export a compressed copy of even my smaller databases - the smallest of which is only 250 cites with pdfs - then recreate a new enl file and then try to import it.

If you have another suggestions, I’ll kiss your feet.  But my databases are too large to try the long-winded approaches I saw.

[crazygecko]

Thanks for the additional info and, just to clarify, I don’t work for Thomson Reuters. This is a community forum where users like us (and periodically also the Thomson Reuters folk)  post questions and solutions (hopefully). 

In reading your assorted posts, could I ask a few more questions regarding the EndNote library files (.enl) being imported?  Notably, each .enl library file has a corresponding .data folder which contains the pdf files. For example, “My EndNote Library.enl” will have “My EndNote Library.data” folder.  So here are the questions:

1. You say you haven’t change the file/folder names but did their location on the directory change? Are both the .enl file(s) and their corresponding .data folder(s) located in the same directory?  For example, “My EndNote Library.enl” and corresponding “My EndNote Library.data” are both located in My Documents - not that .enl is located in My Documents and .data is located in a folder within My Documents:  Having the .data folder located in a different part of the computer directory could confound the import process as EndNote isn’t able to locate the folder/pdfs.

You might also do a small test by copying one .enl file (or a subset) and corresponding .data folder to the desktop (or in the My Documents folder if they’ve been “buried” in subfolders) and see if you can import the file into the new library file.

2. Assuming that #1 has both the .enl file(s) and corresponding .data folder(s) in the same directory, did you re-verify that the the file attachment link to the corresponding pdf still work as expected?  For example, open a reference having a pdf attachment and click the icon to see if the pdf opens? 

3. Also it wasn’t clear to me what OS and version of EndNote you’re using but did you check whether the .enl file(s) may inadvertently be set to “Read-only”?

[leanne]

Another user here - who has libraries that are 15 years or more old and are certainly classed as large and who doesn’t have this kind of problem. 

In addition to the careful CG’s questions, please clarify from where and how are you downloading the files you are importing.

Are all the imports you are refering to, imports from existing endnote libraries (which are already corrupted) or from a online database? 

 Are you using the “find full text” feature or attaching your own PDFs. If your own PDFs,  how long are the PDF names?

[rani]

Thanks so much for getting back.   On your questions…

>>> 1. You say you haven’t change the file/folder names but did their location on the directory change? >>>>>

No, all the data folders remain in my main Endnote.fil folder, where I also keep the enl files.

>>>>You might also do a small test by copying one .enl file (or a subset) and corresponding .data folder to the desktop (or in the My Documents folder if they’ve been “buried” in subfolders) and see if you can import the file into the new library file.>>>>>

Some of the files will copy.  And some will import.  So for example, yesterday when I tried to import a file with 1,000 citations that had pdfs, it imported the first 82 cites, and notified me that it stopped and could not continue because something is wrong with the attachment. 

So, some of them copy, and when I went through the cites, one by one and tried to copy them (very time consuming), I did go through the sets that led to a rejection, and found that some of the file attachments have “shadow” / ghost attachments… I’m attaching two jpgs, one which has a print to screen image of what I see in the faulty cites.

>>>Assuming that #1 has both the .enl file(s) and corresponding .data folder(s) in the same directory, did you re-verify that the the file attachment link to the corresponding pdf still work as expected?  For example, open a reference having a pdf attachment and click the icon to see if the pdf opens?>>>

Most of them work fine, it’s just the cites with the ghost files (again, see jpg) that don’t work and muck up everything I’m trying to do.

Honestly, this is such a  no brainer in the Ednote software development end.   Why do they set the software so it cancels the entire import because of one bad link?  It should instead move that cite to an error folder similar to the duplicates folder and let the importing continue.

>>>>lso it wasn’t clear to me what OS and version of EndNote you’re using but did you check whether the .enl file(s) may inadvertently be set to “Read-only”?

Thanks again for your efforts.  That really is not the problem.  Unless Endnote is really messing up something obvious like that.  I hope the images answer your question.  Thanks again, Rani

[rani]

Hi, Leanne, thanks also for following up.

Yes, in my databases I have sometimes attached files that I scanned, using the Endnote software feature “add attachment” or something like that.  When I do that I am quite careful to have the file names be short and to have it as one file name without spaces.

But this is not the issue in the database I just described because all the articles were downloaded from Sage’s online database.  I’m a subscriber and got the cites by downloading as import citations from their website and then I did a “select all” in the database, right-clicked and asked it to download the articles - which it did.

Hope you noticed my jpgs in last email. I’m attaching the relevent one again and also a new one relevant to that one citation… after I clear the ghost files from the cite attachments, and try to open the pdf with a full name and pdf extention, I still get error messages from Endnote.  I’m attaching a new jpg about that message.

Best, Rani

[rani]

Followup.  I’m attaching two print to screen jpgs with images of what happens after I delete the pdfs and ghost images from the citation and then close endnote, open it up again and then re-download the pdfs.

Yet again, the ghost images show up!  And yet again Endnote blocks me at every step.

By the way, I did a deep disk sector check in case it related to my hard drive, there were no errors.

[crazygecko]

@rani wrote:

 

…So, some of them copy, and when I went through the cites, one by one and tried to copy them (very time consuming), I did go through the sets that led to a rejection, and found that some of the file attachments have “shadow” / ghost attachments… I’m attaching two jpgs, one which has a print to screen image of what I see in the faulty cites.

 

…Most of them work fine, it’s just the cites with the ghost files (again, see jpg) that don’t work and muck up everything I’m trying to do.

 

Honestly, this is such a  no brainer in the Ednote software development end.   Why do they set the software so it cancels the entire import because of one bad link?  It should instead move that cite to an error folder similar to the duplicates folder and let the importing continue.

Just to clarify some of your prior comments, when you said “all the data folders remain in my main Endnote.fil folder, where I also keep the enl files” does this mean your main EndNote folder is named “Endnote.fil” or was this a typo?  Also to further clarify, the .enl files are not placed within a .data folder but maintained outside of the folder but within the same directory?

Thanks for running through the tests which shifts focus to these “ghost” attachments as  the heart of the problem. Coincidentally, another user (fitz_m) reported a similar problem of “phantom” attachments from ProQuest (refer to this thread where fitz_m also posted tech support’s solution. In that case, EndNote tech support identified the problem as the filter.)  But this raises a side-question: did you notice a pattern of the “ghost” attachments?  Do they occur at random or are they affiliated with a certain database or journal?

In your case,  it appears removing the “ghost” attachments is the first priority but identifying and removing them is a big problem as you’ve already noted.  Since the ghost attachment icon is image-based and mimics a real pdf it doesn’t have any unique characteristic to easily identify and remove (either in EndNote or another software) - apart from opening each reference and deleting the icons.  Do you know whether the pdfs have DOI identifiers?  A possibility (but still painful) would be to test making a copy of the .enl file, delete all the attachments; then copy the pdfs from the .data folder into a single folder and import the pdfs - the idea is to remove the “ghosts” by creating “new” attachments.  But if the number of “DOI”-less pdfs outnumber the “ghosts” than it’s not worth the effort.

Sorry, Rani, I don’t there’s a quick and easy fix at hand. Since these situations seem to occur (based on your and fitz_m’s experiences and maybe others as well), there should be an EndNote tool/app for removing these ghost icons in a quick and easy fashion - your last comment would be a great product enhancement.

[rani]

Gecko, thank you so much for responding.

You’re right, there’s no easy fix for me but it would be an easy fix for Endnote developers - and something they should have done long ago.  All they need to do is set their software so it keeps importing the files and moves the corrupted citation to a different error folder.  But no, it just gums up the works so there’s no option for me unless I’m willing to spend countless weeks trying to clear up the problem.

I seem to have more problems now that I’m importing from Sage, so it’s possible they’re adding something to gum up the works.    But why some of them and not others?  And why do they support endnote export and download?

But again, this whole problem is really a problemw with lack of functionality of endote’s software.  I should not have to waste my time when they could just add some code to prevent these problems - and to enable the software to keep importing the files that are proper.

On my comment, all my enl files and the .data files are in the same folder.   I know what you’re asking, but I have it set up the way it’s supposed to be - and the way I’ve had it for years.

On your other question, at this point all of my databases are messed up and  corrupt.   Including my most important one, which has 150,000 cites and 45,000 pdf articles.

Best, Rani

[rani]

Interesting.  The two ghost cites I noticed have two DOIs each

[crazygecko]

rani wrote

 

 On your other question, at this point all of my databases are messed up and  corrupt.   Including my most important one, which has 150,000 cites and 45,000 pdf articles.

 

Best, Rani

Thanks for the info, Rani.  Quite agree about the inclusion of a “fix” in EndNote to address this situation.  BTW, from various postings on the forum by users and Thomson Reuter folk, a last option is to send the damaged libraries to tech support as there’s a possibility of recovering the files.

Also if you’re able export the cites/references as a tab delimited file then import the .txt file into EndNote?  Granted it won’t address the pdfs but at least the cites/references will be housed in a “clean” and new .enl library file.

[rani]

Thanks so much, Gecko, but I’m going to have to wait for them to come up with a fix.   Should be easy for them to do, but near impossible from my end.    My biggest endnote database is close to 30 GB in size - the smaller ones are less important.   I can still use them, I just can’t import or export them - something I need to do with my biggest file to avoid having to wait 2 minutes in between transactions for it to activate.

PLEASE Endnote software guys, come up with a fix for this.   Rani

[sharone]

Rani, please check your private messages. Thanks!

[rani]

After spending a lot of time with my corrupted files - and since upgrading to Endnote X5, which did not seem to help the problem - I have a better handle on what the problem is.

For those interested in seeing the file, I tried to attach a “badfiles.enlx” for download, but this server will not allow uploads of enl or enlz or zip files.  So I uploaded it to my website. It can be downloaded at www.childergo.com/badfiles.zip

This is an endnote file with 11 bad citation attachments - which I could not copy or export to a new database.  As a result, I needed to selectively delete all the other 180 citations that did not go bad om the database.   But unfortunately, in this version the attachments may not show.

I think part of the problem seems to be Sage’s tendency to put subtitles on the second line, so for articles such as “Editorial: Good is not bad”, on importing Endnote only sees the title “Editorial”, and not the full title and decides these are the same article - and correspondingly imports a second DOI.

This would not be so bad if it were note that Endnote stops in its tracks when it notices one of the attachments is off.  My problem is not the bad files - I don’t care about them directly.  It’s that Endnote’s import system seems to have corrupted many file attachments and in the process it means I can’t recover the large databases, I can’t copy or move citations over unless I leave out the bad ones and I can’t resave the database.  This is an untennable and miserable situation for me since I’ve struggled so hard to keep large databases to enable me to search all in one place.

I think some of my problem also may occur when I search for duplicates.   Since Endnote often misses articles that are identical, I started searching for duplicates without the issue or without the pages ln the duplication search.  I found a ton of duplicates this way, but I think in the process it must have corrupted the files - and there is no fix for this with Endnote’s software.

I could go through my large database citations one by one, but why should I spend weeks on this when Endnote could greatly help by providing an improved database recovery system in the Tools menu that would separate the bad cite attachments.  It would also help if they enabled me to search my dataabases for multiple files in the attachment folder, so I could try to find the bad cites.  It would also help if they enabled me to search for citations in my database that have more than one DOI in them - which seems to mean it’s likely corrupted. 

In the meantime, I wanted to mention it to others - I’d love some suggestions for my problem but also to warn you about this.    Download at www.childergo.com/badfiles.zip

Best, Rani

[leanne]

I understand that the program should either not allow you download files that will lead to a corruption in the first place, or be able to identify and remove or rectify a corrupt record after the fact and that it was good to alert the developers to this serious problem. 

One way to at least minimize your pain going forward, would be to first download the files to an empty library and from there, check out the results, before you import them into your main library.  At least that way you could recognize or investigate the problem before you corrupt the main library? 

I routinely connect from or import into a temporary library, as it (a) allows me to globally edit the keywords and thus auto group things once they are in my main library via smart groups, (b) allow me to weed out references I don’t really want or need (c) discard duplicates during the import step and (d) I guess would alert me to a problem if there is one with a specific download or import.     

[rani]

Thanks for getting back, Leanne.  Your suggestion is a last resort solution for future searching, but it doesn’t help me at all with my main database, which has 125,000 citations and 50,000 attachements, which is already corrupted. I’ve spent years developing it and many of the attachments were ones that I hand scanned and added as attachments.   The files also have my notes about the articles.  

The thing is, this is something that is near impossible for me to deal with but that Endnote can readily fix.  It’s completely unacceptable to have the database import and export just cancel out - vaporize - when it finds a corrupted attachment that resulted from its own software.

PLEASE ENDNOTE, a fix is in your hands.   Rani

[rani]

Followup, after spending a day struggling with the corrupted cites.

In every case where the cite is corrupted, it appears that Endnote also imported a second DOI record. 

Much of the problem seems to be caused by Endnote’s importing of files - it only looks at the first line of the title when comparing duplicates, but I notice now that a number of publishers list two lines of titles.   For example “Laws and rules” is the first line, and then a name for the principal is on the second line.   Endnote is inserting new DOIs, which corrupts the database.