Merging Duplicate Records

Maybe there is already a way to do this.  I wish I could find it. When removing duplicates records in an Endnote database where both record numbers have been used in citations I’d like to be able to leave a pointer at the deleted RefID that indicates the RefID of the record that was kept.   So, for example, if reference #50 and #60 were duplicates and I chose to delete #50 Endnote could insert a field at RefID #50 which says in effect “see #60.”  That way, references in documents that used to refer to the deleted record would simply be redirected to the duplicate version which was kept.  This would allow older documents to open and find their references even after substantial clean-up has been done in a database. 

If the goal of this suggestion is to avoid having to re-verify or re-insert many references in a Word document after de-duping or cleaning up an EndNote library, this is already possible using version X4. In X4 we added substantial new intelligence to CWYW that enables EndNote to essentially ignore the record number specifically to support the workflow described here.

Jason Rollins, the EndNote team

That’s exactly the reason for my request.  We’ve recently upgraded to X4, so I’ll start looking back through the documentation now.  Could you provide a little guidance where I should look to get a handle on the new functionality?

This new CWYW logic is purposefully not really well documented as it is ideally supposed to be invisible to the user. In EndNote Help under “What’s New in EndNote X4?” we do note: “Traveling Library enhancements recognize references when sharing Word documents for better collaboration” - but this is really just marketing language.

This improved behavior is very easy to test by doing the following:

  • With a small test EndNote library and Word document, insert a record using CWYW in Word and format it using any style. Let’s say this is record #1 in EndNote.
  • Next, in EndNote copy this record (#1) and paste it as a duplicate record in your test library (let’s say it is now record #20)
  • Next, delete the original record (#1) from your EndNote library and empty the Trash to be sure it is really gone. The copied record (#20) should still be in the library.
  • Now over in Word, click the “Update Citations and Bibliography” button on the CWYW tab. Nothing should really happen – which is good as it means that EndNote X4 knows that #20 is really the same as the now-deleted #1 record. In older versions of EndNote, you would have been prompted to choose a new record with the “Find Matching Citations” dialog.
  • But, just to be sure, in EndNote, next make a change to some aspect of the core bibliographic data in the #20 record – like the date – save this in the EndNote record.
  • Back over in Word, again click the “Update Citations and Bibliography” button on the CWYW tab and see the new date appear in the citations/bibliography. Again, in the past, changing a key element like this would have forced CWYW to prompt you to verify or insert a matching reference.

In this simple example of just one or two references, these changes might not seem like a big deal but in longer documents or in cases where collaborators are sharing documents and references and sometimes relying on the embedded Traveling Library, the new X4 logic can help to save tons of time and lots of frustration.

All this being said, I imagine there are still areas that we can tweak and improve so if you run across these, please let me know and we will look into them.

Jason Rollins, the EndNote team

Will this work also on libraries converted from Reference Manager to EndNote?

If not - is it better to delete duplicates in RefMan before converting or in EndNote after converting the library (when converting to retain RefIDs for Word documents)?

The in-depth instructions for maintaining Word document compatibility and RefIDs advise to import the database into EndNote with duplicates, but there is no explanation as to why we should do this.

Annika:

That’s an interesting question. I personally tend to espouse cleaner data from the beginning, so my own gut reaction would be to clean out duplicates prior to transfer. However, since Reference Manager doesn’t have a merge feature like EndNote’s, there is a certain advantage to waiting until after transfer.

Either way will get you a similar result, so it shouldn’t really matter.