Duplicate check in Endnote X.2

Dear all,

from my point of view the duplicate check in Endnote X.2 is not an improvement - since duplicate and original references are not listed together anymore.

When you work with up to 5000 references going back to “All references” to make sure that the marked duplicate is really identical with the original reference it takes a lot of time.

How are your experiences with this new “feature”?

1 Like
It is very easy to see the “original” and “duplicate” records listed together in EndNote X2. Follow these steps:

   1. Run Find Duplicates
   2. Go to the Duplicates group and highlight all references
   3. Go to the All References group.
   4. Scroll through the list of references to see that the duplicate will be highlighted in All References and its “original” should be right above it depending on the sort options.

The reason we changed this is that with EndNote X1 and prior as soon as one clicked on a record in the Library window the highlighting was lost. To get the highlighting back, one had to rerun the Find Duplicates function; now all one has to do to get the highlighting back is repeat the above short steps.

Jason Rollins, EndNote Product Development

The all references view doesn’t make it very easy to find 5 or 6 duplicates in a library of 5000 records Jason…

1 Like

Thanks a lot for your support Leanne. It is really not very easy to find the duplicates in the all references view - also duplicate und original references sometimes are not even listed in order (in Endnote X.2).

So I really would prefer the duplicate check like in Endnote X.1 and prior (at least to have the option)!

I vote for this!

I also have thousands of records, and not easy to find and clean up duplicates in X2.

>4. Scroll through the list of references to see that the duplicate will be highlighted in All References and its “original” should be right above it depending on the sort options.

Yes, I can do it easily if my library has a few hundreds of records. I sort the library by article title, and then going back and forth between “All references” and “Duplicated references”, I was managed to identify duplicated references next each other. Unfortunately (and probably true with most of long-time users), I have several thousands of records in my library, and it is really painful to make sure which one I can delete. I had newer references which I already edited custom fields, and I had to delete older records in that case. For these ones, I had to do the “back and forth” job so many times.

I hoped I can display all the duplicated references (“original” and duplicates) in the “Duplicated references” group, then sort the reference “within” the group, like title, author or record number. If I can search within the group, that’ll be great because I had to find duplicates that has empty Custom 1 filed.

Best regards,

Message Edited by myoshigi on 10-29-2008 10:35 AM

@myoshigi wrote:

 

The easiest way, I believe, is to display all the duplicated references (“original” and duplicates) in the “Duplicated references” group, then sort the reference “within” the group, like title, author or record number. Then I don’t need to do the “back and forth” job.

 

Best regards,

That’s a request right?  I can’t see an easy way to do that, with the current implementation. 

I just might keep a version of X around and use that for duplicate searches!  – but I am very careful about introducing duplicates in the first place. 

Yes, it is my request for the next updates or upgrades.

I agree I need to be really careful not to import duplicated records, but the way I have developed my library is to import whole bunch of records from online resources, and delete all the duplicates later. With X2, I can’t no longer do that, because I have to import almost one-by-one, which is killing me (really…it is.)

Best regards,

@myoshigi wrote:

I have to import almost one-by-one, which is killing me (really…it is.)

regards,

That is why I do this instead:  I copy things to a temporary library and  import the temporary library into my main library. --from the main library, file import, file type:endnote library, browse to temporary library location, using the discard duplicates or the put duplicates in a duplicate library.

I agree with everyone here… it is an absolute pain to work through deleting duplicates from a library of 2000 references as I had to do the other day. Its all very well to deal with this by always importing into a temporary library first, but doesn’t this defeat the purpose of being able to ‘find duplicates’? This is definitely NOT an enhancement in X2! Please bring back the old system where the duplicate pairs are showing in the one window.

For those with widescreen monitors, I have a addition to the above process that I find more efficient.

1.  Save a second copy of your library under a different name

2.  Open this copy, along with the original.  

3.  Use the “Windows” menu to tile the two libraries vertically.

4.  Use the method specified in Jason’s post above in the copied library.

5.  Remove duplicates using the original library. 

This allows you to work with the original library, while maintaining the highlighted duplicates in the copied library, thereby eliminating the need to keep rehighlighting.

Yes, that’s another way to make sure which one of the duplicates to be deleted or to be left.

I still prefer the way it used to be, that is, duplicates showed up as “duplicates”. It seemed the most intuitive thing for me, and didn’t understand what was the reason to change that intuitive feature to the strange method in X2. I mean, who requested that change, and what was the rationale (just curious).

Anyway, I am still a strong believer; the older way is the way it should be. And if possible, “duplicated records” show up as a smart group such that I can monitor duplicates all the time, or just temporary “search results” of duplicated records but showing all the “duplicates”, not “one of duplicates”.

It just doesn’t make sense, to delete duplicates, why I need to go: “sort library by title” > “find duplicates” > “click one of them” > “click All references” > “scroll up and down 1000s of records to find the one” > “open duplicated ones to make sure” > “go back and check another one”

I am actually using another machine that has Endnote X1, when I need to remove 20 or 30 duplicates from my master library. It’s much faster and simpler.

Totally agree.  Plus, I really like your idea of having duplicates show up in a smart group.  Hopefully, someone from Endnote will read these suggestions.  

We read all of the suggestions and are working on improvements to the “Find Duplicates” function for EndNote X3.

Jason Rollins, the EndNote team

That’s good news! Thanks a lot to the EndNote team - I hope you find a good solution!

I have the following suggestions for improving the identification of duplicates:

At the moment, EndNote checks the whole field “Author” when performing a search in this field. The problem is that after importing references from different sources (Medline, Embase, Biosis etc.) there are often minor stylistic variations in the list of authors – and then EndNote is not able to identify the duplicates from the different sources. Checking just the first author’s name in combination with additional fields could lead to an improvement in sensitivity.

A similar problem occurs within the field “Pages”. Here too, EndNote is not able to find duplicates when there are stylistic variations (e.g., page 112-114 or 112-14). It might be an option just to check the first page number or to complete the second number.

How about reducing stringency of duplicate check, by choosing just Title, and select “ignore punctuation” option?

You may hit something that are not duplicates (articles or book chapter with the same title), but you may pick up something you missed with stringent criteria.

In any case, current “find duplicate” which doesn’t show “duplicates” is not good if many duplicates show up with less stringent criteria.