How to best organize libraries, groups, and large document collection?

[chibob]

I’ve used Endnote for several projects and have almost 1000 documents (mostly PDFs) indexed, with meta-data, and my own annotations. I have another ~1000 documents I would eventually like to put into this database and of course that will grow over time. These documents are mostly PDFs, but can be other formats - some entries don’t have a document (e.g., hard cover book.) My annotations and notes are very important.

I call a research report/paper a “project”. In the past, I’ve used separate .ENL files/folders for each project. I’m now seeing overlap of some documents and projects. I’ve just started several new projects, and the document collection is growing. I need to share documents, notes, and meta-data them with co-workers, but selectively.

I’m trying to puzzle out how to use best Endnote’s libraries, groups, and online sharing to support multiple projects and co-worker contribution. Here are my questions.

1. I’m wondering what makes sense for a large collection of documents - is there a practical limit on the size/number of docs for an .ENL?

2. Should I manage all of my documents with one .ENL master database, using groups to separate them by project and/or subject?

3. Any problems with using a Group Set containing  projects, and Group Set containing  subjects, etc., all pointing into one ENL data set?

4. If one master database makes sense, how can I merge several separate .ENL data sets?

5. What is the relationship between libraries and groups and share-ability?  It looks like there may be some arbitrary restrictions on this. For example, I’d like to share a project group among several co-workers, with each person adding notes/meta-data which is merged into a master library.

6. If there any example or publications on best-practices for this, would appreciate URLs, etc.

Thanks in advance …

Bob

[crazygecko]

1) I’m wondering what makes sense for a large collection of documents - is there a practical limit on the size/number of docs for an .ENL?

 While there is a recommended limit of 50,000 references in an EndNote library (which conceivably would allow for one or more file attachments per reference), there doesn’t appear to be documentation concerning size/number for file attachments (e.g., Word docs, PDFs, etc.). However, presumably key limitations will be guided by: 1) the number of records (50,000 recommended), and 2) sufficient disk space on the computer. Also the larger the library, the longer it will take to sync to your EndNote online account.

2) Should I manage all of my documents with one .ENL master database, using groups to separate them by project and/or subject?

Working from one master library offers advantages such as: 1) maintaining all records and documents in one “location” which facilitates searching, retrieving, and managing/updating records; 2) lessening the likelihood of duplicate records; 3) ensuring continuity by “setting” the individual reference records’ ID number (as copying reference records between EndNote libraries changes the numbering sequence to follow that of the “receiving” library, so in-text citations/references in papers written using the “old” EndNote library will not be able to be generated using the “new” EndNote library).

The disadvantage of maintaining one master database includes: 1) slowed sync times to the EndNote online account; and 2) potential loss of the database due to corruption of one or more files, records, or the library file itself - but this is easily mitigated by maintaining ongoing updated backup copies (including an .xml file).

Having one master database enables parsing out records/references across multiple projects/subjects via EndNote’s “Group” function. However, the “Group” function is limited to 2-levels so you may need to develop a strategy for naming if you wish to incorporate subgroups. Refer to this thread for a workaround to create a faux hierarchy for subgroups:

http://community.thomsonreuters.com/t5/EndNote-How-To/Organizing-Group-Sets-and-Groups/td-p/24041

3) Any problems with using a Group Set containing projects, and Group Set containing  subjects, etc., all pointing into one ENL data set?

One issue might be ease-of-use as the number of groups (maximum number of groups that is 5000) within the master library start to increase. So, as mentioned previously you may consider developing a strategy for naming/categorizing and tracking groups relevant to the project or the topic. Besides a naming strategy, you might also consider placement of the groups as they can be rearranged within the list.          

4) If one master database makes sense, how can I merge several separate .ENL data sets?

You can merge separate libraries by: 1) copying references from one library into the other; 2) dragging and dropping references from one library into the next; and 3) importing references from one library to the other. Note that methods #1 and 2 (unlike method #3), does not automatically filter out duplicate records so you should run EndNote’s “Find Duplicates” function after copying/dragging-and-dropping in order to locate, review, and delete duplicated records. In contrast, method #3 of importing an EndNote library into another EndNote library enables filtering-out of duplicate records during the import process. (Refer to the online user guide for detailed info on merging libraries.)

One big caveat for merging libraries is that the records being merged into the receiving library will have their individual ID numbers changed to follow the numbering sequence of the receiving library. (EndNote uses a record’s ID number to generate in-text citations and corresponding entries in the bibliography.) Therefore, with the new numbering sequence it won’t be possible to update documents using the new library if they contain in-text citations and references from the “old”/pre-merged library. So you may consider maintaining copies of your old/pre-merged libraries in the event you may need to use prior documents.

  

5) What is the relationship between libraries and groups and share-ability? It looks like there may be some arbitrary restrictions on this. For example, I’d like to share a project group among several co-workers, with each person adding notes/meta-data which is merged into a master library.

For a quick overview of EndNote’s library sharing and syncing capabilities suggest you start at the EndNote website which has a page devoted to the topic and includes a short video: http://EndNote.com/product-details/library-sharing . As noted in the summary you can share the entire library (with references, PDFs, and annotations); everyone your sharing with can “add to, annotate and use the library – at the same time”; and there’s no charge for sharing or size limit.

Note that you can have multiple libraries on your desktop but can only sync one library to your EndNote online account. (You can subsequently elect to use a different library but will have to first reset your EndNote online account.) Although you can sync only one library to your EndNote online account (and share this library with designated co-researchers) you could also share different groups/subsets of references by using the Group function then sync the group(s) to your online account then invite your co-researchers to access the specified grouped library of references.

Refer to the Endnote training video “Sharing Groups of References”: https://www.youtube.com/watch?v=hSExPuxdbcM&index=18&list=PL7FCC6F78D0B80B91

and “EndNote Sync: EndNote X7.1 and Later”.

https://www.youtube.com/watch?v=YIzbdvRNz8Q

6) If there are any examples or publications on best-practices for this, would appreciate URLs, etc.

Don’t recall any publications or best-practices but understanding how the EndNote features work and may be doing a trial run may help. Detailed information is available in the PDF manual (which comes with the software), or more conveniently go to the Help section of the Endnote toolbar which links you to several resources (i.e., user’s guide, searchable knowledge base, training videos and live webinars, user community, etc).

[leanne]

I don’t have much to add, apart from emphasizing some point that CG made:

You can only sync one library to the endnote “cloud” so sharing beyond that library is difficult.  

you know the record is in the “one”  library - you don’t have to hunt for the right library. Each record can be associated with multiple groups or subgroups, but if you add a note, or correction, or attachment, you only have to do it once, rather than find all libraires with that record to update.  

Combining libraries by importing one library into the the other, and discarding duplicates is preferred but you don’t maintain any group structures, To reassign them to one or more groups,  I use the tool Change/Move/Copy to edit the keywords of the contents of a library I am about to import and also to each of the records in each group first to include unique keyword(s) athat allow me to use smart groups to reassign them to the correct groups in the combined library, I then copy the contents of smart groups to a hardwired groups/subgroups (be sure subgroups have unique names) to ensure they are properly synced to the cloud version.   Some of this is described in the resources CG pointed to in his reply. Smart groups are wonderful!  I never delete them.  I just notice when the number in smart group is larger than its matching hard wired group, I select them all and copy to the hard wired group and the new ones are added there.  Smart groups are not backed up to the cloud library (unless that has changed in a recent update).  

I have been using Endnote for many many many years (since about 1987).  I still only have about 7000 records.  But I am selective and tend to add records I use or might use, and have at least reviewed their content.  Having a record in my library that I am not familiar with, means I am unlikely to cite it.  It probably wouldn’t be as big as it is, if I didn’t have students/post-docs who have utilized the library over the years, who aren’t as robust on this as I am.

Once you have written a paper, you can move the documents citations to a new group, if you like.  Records that never make it into a group, are likely candidates for deletion from the library.