Removing duplicates while retaining group membership

Hi Folks,

I’ve asked this question once before (  Many of us who have libraries combined from older versions of endnote now have many duplicate references each of which are associated with different groups.

Has anyone come across a way (a script perhaps?) to remove duplicate references while maintaining their various group memberships?  For example, reference1 is the same as reference2.  reference1 belongs to group1, and reference2 belongs to group2.  I want to delete reference2, and make reference1 a member of both group1 and group2.

I could do this manually, but it would probably take 3 days of work.  So, that’ll never happen.  My library remains a mess, ever since the advent of groups.



I understand what you want to do, but how script, or somebody else other than you, know which group one particular reference should belong to? If one group is completely unnecessary, you can tag references that belong to the group and delete all of them, but that doesn’t solve the problem. This problem is not possible to automate…inherently.

And, if you want to “merge” group membership, a function that export/import group membership is necessary, but current EN doesn’t have that.

Sorry, only 3 days of manual work would be the answer.

(For this reason, sticking with one master library is a better idea, if someone start using Endnote)

Message Edited by myoshigi on 11-07-2009 09:06 AM

@myoshigi - I don’t think you understood my question.  My duplicate references already belong to different groups.  There is nothing inherently un-doable about that.  I’m not asking for any sort of automatic judgement from the computer.  Did you read my example above carefully?

ref1 belongs to group1.  ref2 belongs to group2. ref1 and ref2 are identical, apart from their group memberships.  A script would delete ref2, and make ref1 belong to both group1 and group2.

Pretty ***edit*** simple.

And yes - i have one master library.  And yes - i started with many, before groups came along.  Which i stated before.

Then, who decides ref2 to be deleted, and ref1 to be kept, and retain group membership? Record # should be the only difference between two references, so is it all right to keep the one with larger record number?

The point is, your duplicates may not have such a uniform duplicates. When you describe your library contains “mess”, I imagine some duplicates don’t belong to any group (ref 1 to group 1, ref to to nothing), or some duplicates belong to redundant groups (ref 1 to group1 group 2 and ref2 to group 1 and group 3). If you have the one like the latter case, and then you want to keep ref1 instead of ref2 because ref2 shouldn’t belong to group 3, who tells script such information? Are you okay to merge all the group 1 2 3 to ref 1?

In any case, EN deals with the group membership as not exportable/importable information. So the scripting is not possible, at least in the Win platform. Mac Endnote started to support AppleScript, but what level…I have no idea.

Btw, these redundancy and inconsistency of group membership can be organized, if we have simple “AND” and “OR” search function of group membership. For example, when we do duplicate search, EN creates a temporary “Duplicates” group. If we know which duplicates belong to “Unfiled” group, we can easily delete them because they are probably imported again.

During the beta testing of X2 and X3, I always kept saying “Please develop search function of group membership” and “clicking groups with Alt/Shift/Ctrl keys to do simple AND/OR/NOT boolean algorithm”. I included this request as #1 in X4 Wish list. Revising tree structure is one thing, but robust search of group membership is probably more like a problem-solver.

Message Edited by myoshigi on 11-07-2009 01:20 PM

The decision can be random, or all duplicate refs could be added to all groups, and all duplicate refs minus one selected, making it easy to delete.  EN already has a “find duplicates” function.  How does _ it _ decide which one to highlight?  You are making this way more complicated than it needs to be.  Much of the functionality already exists in endnote.  They just need to push it a bit further.  I imagine you’re right about it not being able to be scripted.  It’s something the developers will have to write.  And that means waiting.

In X3 you get to choose which duplicate you want to keep. It is no longer automatic. We have already asked for the option of copying information from one record to the other in a future version during this review process. So I think what you need to request, in addition, at that decision point is for the group information to be clearly visible and for the additional option to edit group membership to that dialog in the next version update. 

 Thanks Leanne - is there a thread where I can contribute that suggestion (instead of starting a new thread)?

there are two ways.  I think it is better to have a stand-alone thread in the suggestions forum here.  It makes attacting “me too” or letting people add “Kudos” easier than having a message in the X4 wish list, I think.  – but there is already the groups thread there, so you could just add to it. Or perhaps this is the right thread

But - Also, go to the and use the suggestion link there.  I think this formalizes the “suggestion” and might help to get it into thier job list.  Be very specific (and then copy that to the suggestions thread?).

Message Edited by Leanne on 11-10-2009 09:54 AM

Or perhaps this is the right thread