The task to me seems simple. The NIH is now requiring that we have Grant info for each publication. I track around 150 researchers publications (about 40/week). There is a GR (grants field) in medline downloads and in the past it has been mapped to notes. This means that all the information already in my database does NOT have the grant numbers separated from the notes fields or other info in the GR field. I recently corresponded with tech support and they provided me with a modified pubmed import filter that imports only the grant number into the Translated Title field. My problem is not going forward, but rather what to do with the 100s of references I have for the past year that have that grant info in the notes field. TR has said that the only thing to do is use the move/copy function, but agreed with me that that will move the whole field, not just the part I want.
Anybody worked this out yet, since ALL NIH funded projects will have to use this by October???
There is more than one problem here. First, yes I can reimport them (I keep all the medline files for just this purpose), but the records have been customized significantly and my boss, each week, marks some as not relevant or deletes some. So a strict reimport would not only be labor intesive (1 file per week dowmloaded for a year is 52 files per year) for the importing, but would delete all the work we have gone through to get the library to this point. Second, why? This is a rather intuitive operation. One should be able to do this easily, why can’t I - if I was using Zotero I would be able to do so! (frustrated). Third, updating records uses the connection file, not the import filter. I have recently found that when the tech at TR messed with my pubmed connection file about a week ago he messed it up and now when I go to find reference updates it compares to completely different publications (except for some odd reason they have the same PMID). For this reason I have reverted back to the previous connection file (not import filter). I fear that if I ask them to alter something else, something even worse may happen.
You say you would be able to do what you want in Zotero; how would you do that? That might give us some suggestions how to solve your problem in Endnote…
(Probably, you have already thought about the possibility (?) to export your Endnote library to Zotero, make the necessary changes there, and reimport the library to Endnote.)
I have indeed thought about it. Although my speculation on how it would be easier in Zotero is based on my brief experience of exporting my library to zotero and fiddling with it there (I almost changed over a couple of years ago). The problem comes down to something rather simple. Why can’t I put in a query like:
If {Notes contains N.I.H.}
Then
copy all text starting from R01 to “/” to Translated Tilte
Else
Put Not Funded into Translated Title
EndIF
This can be repeated for each type of funding (NIAID, DOD, etc). I mentioned Zotero because when I was playing with it I noticed I was able to do a lot of the boolean queiries and manipulations that endnote would not let me do. It seems to me this should be able to be done on the back end, they have manipulated my library before, when I originally changed over from Reference Manager 12 to EndNote x2.
Unfortunately, as you already know well, you can’t run a query like that unless you have access to the back end database (and, who knows, in this case perhaps not even then). Via the GUI you can’t even change/move/copy fields on just a subset of references.
What you can do, perhaps, is to create one smart group of references for each grant number. Combine these groups into a group from groups, to select by exclusion all references without a grant number. Export each of these groups of references to separate Endnote libraries. Update the field Translated Title accordingly in each of those libraries. Finally, combine all libraries into one large library again.
That’s a tedious procedure, yes, and perhaps just seemingly straightforward, if more than one grant is contributing to certain publications. But judging by your query example, that does not seem to be the case, hopefully …
I actually do not think this is such a straighforward problem to solve but could potenitaly be addressed by leveraging the EndNote SDK - writing a custom plug-in that could search, find, and move specific text from within one field to another.
Based on the info the NIH provides here http://www.nlm.nih.gov/pubs/techbull/mj06/mj06_grant_numbers.html - it looks like this is not simply a matter of matching a specific string or pattern but rather one that would require some complex fuzzy logic to cover all/most variations of the Grant number / funding agency patterns.
Alternately, one might write a script using something supported by the operating system [Window or Macintosh] to automate this. But, again, the multiple variations of the Grant number / funding agency patterns woud need to be accounted for.
Even if the problem is complex, Thomson Reuters should be addressing this NOW. All grants will be REQUIRED, if they have any NIH funding, to use the MyBibliography section of MyNCBI. This means that everyone will need to have grant information associated with every publication, or the publication will not be allowed to be cited in the list of publications at the end of the progress report or competitive grant proposal. Currently EndNote does NOT allow me to do this or if the SDK does than it is not easy.
I am only going through the last 700 records, because we submitted a competitive renewal last year. Otherwise I would be stuck going through thousands of records (I currently have over 7000 in the library). Changing the pubmed import filter was simple, why is this so much more difficult?
BTW, I had to change the modified pubmed import filter the tech sent me so that PMC and NIHMS IDs were imported in UPPERCASE instead of sentence case.
Forogt to ask: How difficult is it to use the SDK and develop plugins?
…This means that all the information already in my database does NOT have the grant numbers separated from the notes fields or other info in the GR field…what to do with the 100s of references I have for the past year that have that grant info in the notes field…[extract] just the part I want.
Is there any discernable pattern in the way the grant numbers appear in the Notes field? Could you provide a sample EndNote export (.rtf) of a few records using the Refman (RIS) Export output style? (Using the Refman (RIS) Export output style adds “tags” to enable manipulating the different fields and corresponding data – see attached image,)
Hard to say without seeing the format of the grant numbers but absent an EndNote plug-in (which would be ideal) a tedious workaround might be to insert a delimiter to the grant numbers before or after exporting the records from EndNote. Then the grant numbers in the exported file could be parsed from the Notes fields into a GR (grants field) then imported into EndNote. There are a few caveats to this method:
1.It’s dependent on the format of the grant numbers in the Notes field (permitting isolating the info).
2. EndNote will import the records as new records each having a new record number. If 1) retaining the original record number is needed, or 2) the original records have file attachments then merge the original library with the imported library. This will create duplicate records. Depending on what version of EndNote you’re using, the “Find Duplicates” feature will allow side-by-side comparison where the grant number can be manually copied from the “imported” record into the “original” record.
If retaining the original record number and file attachments isn’t an issue then only importing the modified file onto EndNote and discarding the original records is needed.
So here is the exact problem. I have tons of records where the notes field contains this info (without the GR - )this is an excerpt from a medline textfile (I have all the medline files I have ever imported):
GR - R01 DK-058547/DK/NIDDK NIH HHS/United States
GR - R01 GM-067081/GM/NIGMS NIH HHS/United States
GR - R01 GM067081/GM/NIGMS NIH HHS/United States
The actual medline file is attached. Are you saying that if I export everything it will keep my modifications when I import it back? I already have a Pubmed import filter modified from TR that will import only the grant number from the GR field into my Translated Title field. But that is only going forward. For instance, now that same record imports only the grant number like this:
R01 DK-058547
for each grant number.
As I said it would take weeks to recreate the library as it is, due to the changes I have made and the changes my boss has asked me to make.
I’ve already taken all my medline files from 2013 forward (about 60 files) and combined them into one using the DOS copy command. I may or may not use this to reimport into a new library. 14-02-18.txt (47.8 KB)
TZ-S: Are you saying that if I export everything it will keep my modifications when I import it back?
A: Some testing might be needed given the significant modifications you noted but as long as 1) the import filter has been modified to match the field “tags” in the imported file; and 2) those “tags” correctly correspond to the appropriate field in the EndNote reference type template (e.g., journal article, book, book section). This is particularly important if you’ve customized the Reference Type fields,
I also reiterate my prior caveats concerning issues if you want to retain the original record number and if file attachments are involved.
TZ-S: …But that is only going forward. For instance, now that same record imports only the grant number like this:
R01 DK-058547
for each grant number.
A: What do you mean “that same record imports only the grant number”? Are you using the TR modified pubmed import filter to your “older” records where the grant numbers grant numbers appear in the notes fields?
This situation is different as it calls for modifying the import filter to extract the grant number from the Notes field. How much info is extracted depends on 1) how the grant number appears in the Notes field and 2) if the import filter was coded correctly.
If only “R01 DK-058547” and not “R01 DK-058547/DK/NIDDNIH HHS/United States” is being imported it could indicate problems with the grant number in the Notes field (e,g, spacing, hidden codes, etc,) that’s unaccounted for in the filter.
Would need to see 1) a few “older” records exported from EndNote using the Refman (RIS) Export output style in an ,rtf format, and 2) the import filter. If you cannot provide complete records just make a new library, copy a few “older” records then edit them down as I just need to see the notes field, and finally export them from EndNote using the Refman (RIS) Export output style
By the way - sorry it has taken so long to reply - work gets in the way.
I do need to keep the same record numbers and I know this is a problem.
The older records are not affected by the new modified import filter, which is working correctly - for the most part. The modified import filter is working correctly in that it is importing the GR field to the Translated Title field now. The only problem is that it imports one grant per line making it necessary to edit each record so that all grant are comma delimited on one line (see attached).
I have attached a couple of files that are RIS exports from the database. they are as follows:
refs with comma in translated title.txt contains records in which the grants field was already imported to the Translated Title field or I moved it there. I then deleted spaces and replaced line feeds with commas
refs with grant in notes.txt - This file is the original way grants were imported into the notes field - true of all files before I started using the modified pubmed import filter.
refs with grant in tranlated title field.txt - This file shows the result when using the modified pubmed import filter.
