automatically update references

I have been an endnote user for quite a while and I thus have a library with Pubmed entries that are quite dated. It would be very useful to update all newly added fields automatically. Currently, this can be done by selecting all refs and then selecting update. However, for each reference I am asked to first select “empty fields only” and then “save update”. For a library with over 2000 entries this is prohibitive. In fact, I got tired of it after 10!

So an option where the fields to be updated is selected once and then hit go would be greatly appreciated. It seems this should be trival to program but would be amazingly useful for many researchers now having to add new fields for the NIH grant proposals.

THANKS!!

Hello, felixschweizer:

I’ve spoken with a number of users here in Support who have asked for a feature like this. I can’t speak for the product managers, but what I always point out to those customers that while that feature certainly would be helpful, it could also be highly dangerous.

At the moment, you are prompted individually for a very very good reason. Let’s say you’ve been working with your references and adding extensive notes into a custom field, such as notes or research notes. This is a pretty common workflow for people, and sometimes they invest large chunks of time into adding this custom information.

If we had an automatic update wherein it did not compare the references, it could possibly – without notification – wipe out the custom information you added. That is why, right now, it asks you very clearly and carefully for each individual reference whether it should overwrite fields or only update the empty fields… the first option could have the potential of removing some of your custom information, and the second could miss valuable new information. And one record may be better with one solution, and another the other.

I still think there are likely ways EndNote could do a better job with some of these things than it does right now, so this is certainly a valuable thread to have here in the Suggestions forum. However, I wanted to make sure it is clear why it does what it does right now, as tedious as it may seem – keeping your data safe is our paramount concern.

This is an annoyance, but here is a solution of sorts.  Usually once the publisher updates the record with keywords and such, the record doesn’t change further.  It CAN happen of course, but not often.

If I grab a newish article, perhaps from the “Early Online” section of a journal, there will be a Pubmed record that will later get updated.  My work around was to create a custom group called “Update” that contains a subset of my database for any reference that is missing the accession number, volume/issue number, or keywords.  If the journal or item is not going to be indexed in Pubmed (like unpublished papers I manage to obtain), I add the words “Not indexed in Pubmed” into the accession number field and those are excluded from my custom group.

Presently I have about 2,800 articles in my Endnote, but only 25 or so in that custom group.  When I need to update I only have to scan the 25 that are missing full information and as those records get updated will full information they drop our of the group automatically.  Some publishers update with a month or two, some take up to a year (certain IngentaConnect publications are the worst).  Finally, in order to avoid having to confirm the updates manually I wrote an Autohotkey macro that does it for me. 

One I have the macro loaded, I select all the records in my “Update” group and then click on the “Find Reference Updates” and Autohotkey will do the mouse clicks necessary to confirm any updates available.  With only 25 records  I can usually scan all my incomplete references while I refill my coffee, and I only run the macro once every few weeks.  Autohotkey is a free program (Google it) but it takes a bit of practice to get it to work.  The mouse click coordinates positions are specific for YOUR monitor dimensions and screen resolution, and the sleep timer depends on how fast your computer is (you might need to make it longer if you have a slow computer).  Also note that Autohotkey “reads” the window titles so Endnote must be on top while this is running (i.e. you can’t start the macro and then go web browsing while it runs).  With those caveats, here is the macro in all its glory

settitlematchmode, 1
start:
WinWaitActive Review
click 430,130
sleep 100
Click 615,415
goto start

The problem with having Endnote do an automatic update, aside from overwriting custom data as was mentioned, is that Endnote / ThompsonReuters doesn’t control the Pubmed database.  If you have over 2,000 entries then Endnote would have to connect to Pubmed 2,000 times and request each record one by one every time you wanted to update.  Even with a fast connection that would take many hours to scan your full database, as it did mine, which is why I created the custom group to limit the effort to just the records that need attention. 

Note that my macro is fully automatic, so it will overwrite any custom notes you have added so use it at your own risk.  It works for me, but a wise person would test it out on a COPY of their database.

Warning:

One thing to be careful of with in reference updates and find full text calls to Pubmed, is that Pubmed may interpret high numbers and quick repeat requests as Denial of Service attacks and block your IP address, so running something like this on 2000 records is a bit dangerous anyway.  

Hodag1, your script sounds like a good interim solution.  Can I just enter the text into AHK as you have it?  Is that the exact script? Are there headers?

Tried script on my windows 7 box using EndNote x7.02 and it doesn’t do anything.  Are there other lines that need to be there?  The script is currently just the default with the commented lines left in and the uncommented one replaced with your script. (I used the zip archive to install)

; IMPORTANT INFO ABOUT GETTING STARTED: Lines that start with a
; semicolon, such as this one, are comments.  They are not executed.

; This script has a special filename and path because it is automatically
; launched when you run the program directly.  Also, any text file whose
; name ends in .ahk is associated with the program, which means that it
; can be launched simply by double-clicking it.  You can have as many .ahk
; files as you want, located in any folder.  You can also run more than
; one .ahk file simultaneously and each will get its own tray icon.

; SAMPLE HOTKEYS: Below are two sample hotkeys.  The first is Win+Z and it
; launches a web site in the default browser.  The second is Control+Alt+N
; and it launches a new Notepad window (or activates an existing one).  To
; try out these hotkeys, run AutoHotkey again, which will load this file.

settitlematchmode, 1
start:
WinWaitActive Review
click 430,130
sleep 100
click 615,415
goto start

; Note: From now on whenever you run AutoHotkey directly, this script
; will be loaded.  So feel free to customize it to suit your needs.

; Please read the QUICK-START TUTORIAL near the top of the help file.
; It explains how to perform common automation tasks such as sending
; keystrokes and mouse clicks.  It also explains more about hotkeys.

No, that is the entire script.  I have added one additional line since to make it run more slowly, and was more verbose on the WinWaitActive command but that really shouldn’t matter:

Settitlematchmode, 2
start:
WinWaitActive Review Available Updates
click 430,130
sleep 250
Click 615,415
Sleep 1000
Goto start

What is likely your problem is that either you did not update the mouse click coordinates for YOUR screen, you did not update the sleep parameter for YOUR computer, or you made another error.

Play with the AHK “window spy” and make sure that the mouse coordinates specified in the script match where the boxes appear on your screen.  Personally, I have a Lenovo T420 laptop with a 14" display.  If you have any other size display then your mouse will be in the wrong spot and you will need to update the coordinates.  Ditto if you sometimes run in a window instead of full screen, the coordinates will change on you as you resize the window.  I only use EN in full screen so I don’t have that issue, but I did have to tweak the macro when I bought this computer and retired my older one with a 12" display.

Second issue is that the script runs too fast for your computer.  If yours is slower than mine, then you may need to have more “Sleep” units between commands to give the computer a chance to catch up to the script.  This is something you do by trial and error, just add a Sleep 5000 command for example to give it a bit of pause (each sleep unit is one millisecond, so 5000= 5 seconds) and if that is taking too long reduce it to Sleep 3000 or 1000 as necessary.  You have to customize it to the speed of your computer processor and your Internet link.

Finally, are you loading the AHK macro into memory, highlighting the records you need to update, and then using the “Find Reference Updates” on the set?  Just try to get it working on two records, and if that works then you are good to go with more (although you may need to tweak the “Sleep” a bit more).  Also, EN must be the window on top.  You can’t start the macro and then go Internet surfing; EN must be the program showing in the main window and it must be on top.

As for the user that cautioned running too many requests at once, they are correct.  However, you can do small subsets (like 25 records at a time) until your database is fully ready, or you can insert a one minute delay into the macro with Sleep 60000 so it only hits PubMed once a minute.  Leave it running overnight and you can do several hundred records a night for a few nights.  Personally, I do record updates of 25-50 all the time without any issues, but I am the only person that uses this IP address to contact PubMed.  I am not sure how it would look to Pubmed if, for example, you were on a shared IP from a university network with lots of students sharing a single routable address from behind a router and you all used PubMed at the same time.

so does that script need to be part of the default autohotkey script?  I was trying to make a new one.

With a little help from colleagues I created this script to auto update my records.  The procedure is as follows:

Download AutoHotKey

Install AHK

comment all lines out of the default script (if you aren’t going to make a new one)

add the following lines in where the script was

Loop
{
  SetTitleMatchMode,1 ; title must begin with string
  IfWinExist,Review Available Updates
  {
    WinActivate
    Send {Enter}
    Sleep,100
    Send {Tab}
    Sleep,100
    Send {Enter}
    Sleep,3000 ; wait 3 seconds (parameter is milliseconds)
  }
}

So the default script would look like this (feel free to copy and paste any of this post):

; IMPORTANT INFO ABOUT GETTING STARTED: Lines that start with a
; semicolon, such as this one, are comments.  They are not executed.

; This script has a special filename and path because it is automatically
; launched when you run the program directly.  Also, any text file whose
; name ends in .ahk is associated with the program, which means that it
; can be launched simply by double-clicking it.  You can have as many .ahk
; files as you want, located in any folder.  You can also run more than
; one .ahk file simultaneously and each will get its own tray icon.

; SAMPLE HOTKEYS: Below are two sample hotkeys.  The first is Win+Z and it
; launches a web site in the default browser.  The second is Control+Alt+N
; and it launches a new Notepad window (or activates an existing one).  To
; try out these hotkeys, run AutoHotkey again, which will load this file.

;#z::Run www.autohotkey.com

Loop
{
  SetTitleMatchMode,1 ; title must begin with string
  IfWinExist,Review Available Updates
  {
    WinActivate
    Send {Enter}
    Sleep,100
    Send {Tab}
    Sleep,100
    Send {Enter}
    Sleep,3000 ; wait 3 seconds (parameter is milliseconds)
  }
}

;^!n::
;IfWinExist Untitled - Notepad
; WinActivate
;else
; Run Notepad
;return
;
;
; Note: From now on whenever you run AutoHotkey directly, this script
; will be loaded.  So feel free to customize it to suit your needs.

; Please read the QUICK-START TUTORIAL near the top of the help file.
; It explains how to perform common automation tasks such as sending
; keystrokes and mouse clicks.  It also explains more about hotkeys.

You can change things if you want to only update empty fields (instead of the Send {Return} command first precede it with a Send {Tab}), or if you wish you can set a longer period at the end (instead of sleep,3000 - 3 seconds, you could set it for 120000 - 2 minutes)

I tried the Send {tab} option out and it doesn’t work.  If there isn’t the option to update only the empty fields it skips that and acts differently.  So you would have to query if the button was active first.

I have sontinued playing with AutoHotKey to automate EndNote and have come up with this script to automate the changing of fields and copying of others after import:

SetTitleMatchMode,1 ; title must begin with String
IfWinExist,EndNote X7 - [aecclib] ; set the library to what ever your library is - mine is aecclib
{
   WinActivate
   Send !t ; send alt t to access the tools menu
   Send {c 2} ; send to c keystrokes to access the change submenu
   Send {ENTER} ; send enter key to choose menu
   Send {c 7} ; send the c key 7x to choose the Custom 8 field
   Send {TAB 2} ; send the tab kay twice to position in change area
   Send {Down 2} ; send down key twice to choose “replace whole field with:”
   Send {TAB} ; to get to text field
   Send aftergrant2012_09 after2014SFURequest after2014Progressreport ; enter text for excludes in custom 8 field
   Send {TAB}
   Send {ENTER 2}
   sleep 5000 ; sleep 5 seconds to allow updating
   Send {Enter}
; finished section 1 - entering excludes into custom 8 field
sleep 1000 ; sleep for 1 second
   Send !t ; send alt t to access the tools menu
   Send {c 2} ; send to c keystrokes to access the change submenu
   Send {ENTER} ; send enter key to choose menu
   Send {r 4} ; send the r key 4x to choose the Reviewed Item field
   Send {TAB 2} ; send the tab kay twice to position in change area
   Send {Down 2} ; send down key twice to choose “replace whole field with:”
   Send {TAB} ; to get to text field
   Send 2014 from pubmed ; enter text in Reviewed Item field
   Send {TAB}
   Send {ENTER 2}
   sleep 5000 ; sleep 5 seconds to allow updating
   Send {Enter}
; finished section 2 - entering origin of pub (pubmed) and date into reviewed Item field
sleep 1000 ; sleep for 1 second
   Send !t ; send alt t to access the tools menu
   Send {c 2} ; send to c keystrokes to access the change submenu
   Send {ENTER} ; send enter key to choose menu
   Send +{TAB} ; access menu one above
   Send {Right} ; move right one tab
   Send {TAB} ; move to from field
   Send {a 4} ; press a 4x to choose accession number
   Send {Tab}
   Send {L} ; choose label field
   Send {TAB}
   Send {Down 2} ; choose replace entire field
   Send {TAB 2} ; two tabs if the former field is chosen
   Send {Down} ; choose to copy field
   Send {TAB} ; to get to next field
   Send {ENTER}
   Send {ENTER}
   sleep 5000 ; sleep 5 seconds to allow updating
   Send {Enter}
; finished section 3 - copying pmid to label (copy accession number field to label field for duplicate checking)
sleep 1000 ; sleep for 1 second
   Send !t ; send alt t to access the tools menu
   Send {c 2} ; send to c keystrokes to access the change submenu
   Send {ENTER} ; send enter key to choose menu
   Send +{TAB} ; access menu one above
   Send {Right} ; move right one tab
   Send {TAB 4} ; go to move/copy selection
   Send {Down} ; ; choose to copy field instead of moving
   Send {TAB 4} ; move from copy field to from field
   Send {r 3} ; press r 3x to choose record number
   Send {Tab} ; move to “to” field
   Send {c 8} ; press c 8x to choose call number field
   Send {TAB} ; move to next field
   Send {Down 2} ; choose replace entire field
   Send {TAB 3} ; three tabs if the former field is chosen
   Send {ENTER 2}
   sleep 1000 ; sleep for 1 second
   Send {Enter}
; finished section 4 - copying record number to call number field in case of record number change
}

The down side to this script is that it starts acting as soon as it is invoked if the EndNote window and library are open.  So the best way to use this, I have found, is to import new records - or select and show only those you wish to update, then invoke the script.  This will update all records being shown in the endnote window.

I have also attached the script - in case you want to try it out (just change the extension from .txt to .ahk).

ALWAYS make a backup before trying anything like this.  I make a backup of my library using the File->Compressed library (.enlx) … command before any change and at least once a day.

update new records on import.txt (3.01 KB)

Although it is probably justified to ask the user to accept one by one for the updated references to avoid data loss, it is not actually well designed still. Actually, it is annoying. At least the button “update all fields” and the button “save updates” can be a bit closer, right? Or how about ask the users to accept one by one (if we assume endnote really cannot make a backup), and save them all together at the end. This could save half clicks. Please just admit it, there are some place to improve this fantastic feature.

Although it is probably justified to ask the user to accept one by one with the updated reference, avoiding data loss, it is not actually well designed, it is annoying. At least the button “update all fields” and the button “save updates” can be a bit closer, right? Or how about ask the users to accept one by one (if we assume endnote really cannot make a backup), and save them all together at the end. Please just admit it, there is a plenty of room to improve this fantastic feature.

I know this thread is really old, but unfortunately the problem persists! See below for my AutoHotKey code to run Find Reference Updates and press only Update Empty Fields when available. Since I spent enough time on it, I figured I should share in case anyone else is having the same problem. 

; This AutoHotKey script performs the following actions:
; 1. Checks that EndNote is open
; 2. Initiates “Find Reference Updates” for selected publications
; 3. Checks for the existence of a “Review Available Updates” window, then activates that window
; 4. Automates the acceptance of empty field changes for each record, or skips records with no “Update Empty Fields” option
;
; To use:
; 1. Turn off EndNote’s A)“We were unable to find a reliable match for this record” error, and B) “No updates were found at this time.” error
; 2. Select desired publications to update
; 3. Open or reload this AutoHotKey file
; 4. Press ctrl-shift-f (f for find!)
;
; This script written by Melissa Day                                  

^+f::

SetTitleMatchMode,1 ; title must begin with String
IfWinExist,EndNote  ; make sure EndNote is open first
{
  Send !r      ; press alt r to access the references menu
  Send {f 3}   ; press f 3x to access “Find Reference Updates”
  Send {ENTER} ; enter to choose
  sleep,200    ; wait

  Loop
  {
    SetTitleMatchMode,2 ; title must contain string
    IfWinExist,Review Available Updates       ; wait for correct window
;    WinWaitActive,Review Available Updates   ; alternative
;    If WinActive(“Review Available Updates”) ; alternative
    {
;      MsgBox, Active-Review Available Updates          ; checkpoint
      WinActivate
      ControlGet, status_UEF , Enabled, , Update Empty Fields ->
      ControlGet, status_skip, Enabled, , Skip
;      MsgBox, UEF #%status_UEF% & skip #%status_skip%  ; checkpoint

;;;;; If Update Empty Fields button is available, click it & save
      If (status_UEF = 1) {
        ControlClick, Update Empty Fields ->, Review Available Updates
        sleep,1000       ; wait (length is important here)
        ControlClick, Save Updates, Review Available Updates
        sleep,1000       ; wait
      }

;;;;; If Update Available Fields isn’t available, click Skip instead
      else if (status_UEF = 0) and (status_skip = 1) {
        ControlClick, Skip, Review Available Updates
      }

;;;;; If Update Available Fields AND Skip aren’t available, click Cancel
      else if (status_UEF = 0) and (status_skip = 0) {
        ControlClick, Cancel, Review Available Updates
      }

      sleep,3000        ; wait for entry to advance

;;;;; To thwart screensaver in case of long runs
      MouseMove,  1, 0, 1, R ; Move the mouse one pixel to the right
      MouseMove, -1, 0, 1, R ; Move the mouse back one pixel

    }
  }
}

Hi Leanne,

I can’t update references.  Any suggestions?

Regards,

M

Hi Leanne,

I can’t update references.  Any suggestions?

Regards,

M

No suggestions other than calling Tech Support (but perhaps not the day before Thanksgiving …on this occassion you might try submitting the problem online).  I am taking off for the holidays! (but I don’t work for TR, so don’t blame me!)  – perhaps anyone else who can help might post on your new thread here.