I have ten different libraries for different purposes, but one set of display fields applies to all when viewing the library window. This means that whole columns are unused in the display of some libraries; in others it would be useful to adjust the sizes of some columns according to the content, but this cannot be done without changing their breadth for all libraries.
I have a similar issue. I do different projects for different customers who have different needs. It would be very helpful to have contextual display fields. Also, would it really be so hard to put the reference types in alphabetical order?