Research evaluation - is it our business?

On Wednesday I attended and event laid on by the JIBS user group (users of bibliographic software). Somewhat provocative title but the aim of the day was to look at librarians role in research evaluation. There were some interesting presentations, ranging from evaluation of using Google Scholar for bibliometric data to implementing research information systems like PURE or Symplectic.

The event was introduced by Fiona Bowtell (chair of JIBS) and the day was chaired by Kara Jones. Here's my notes on some of the presentations:

Andria McGrath - King's College London - Keynote

Andria made the point that research evaluation is a "brave new world" and I guess that this is an emerging specialism for librarians. In the past research support was likely to be done by subject librarians but it was interesting to see quite a high number of specialist Research Support Librarians at this event. Andria went on to discuss the decreased importance that the REF 2014(?) places on bibliometric data and that impact (in terms of the REF) is a more general concept. Demonstrating some societal change is (quite rightly in my opinion) more important than citations. But there are areas where bibliometrics will inform panels. Andria described some initiatives that should make getting at bibliometric data easier - in particular the CERIF data model, which is a European standard for CRIS (Common European Research Information Format). This is an xml-based model that goes beyond metadata - allowing deeper description of the information needed to manage research. A CERIF4REF schema has been produced for ePrints that will allow easy porting of data in repositories for REF submissions.

Andria was the first of many speakers to mention that librarians are not always involved by their institutions and that we need to push ourselves forwards. She mentioned a RIN report "Value of libraries for research and researchers" that looks useful.

So CRIS, CERIF, new ways of using bibliometric citation APIs, discovering other systems (like grants) and bibliometrics products and principles are all new areas for librarians. Some institutions have engaged with this and there was discussion in the room of an institution (Surrey?) that have employed a bibliometrician in the library staff. There is some evidence that academics are starting to get interested in bibliometrics (I've certainly seen some interest around generating H index numbers!) and that some are using advanced techniques like using Essential Science Indicators or InCites to normalize results.

Andria also listed some of the things people want from this area: they do want a Research Information System, they do want a dashboard or simple interface that has all the information they need for grant bids; they want to capture data easily (pulling in bid data from databases and only entering data ONCE!); and they want integrated bibliometrics. All of these are things that the library can help with!

Jeremy Upton - St Andrews - Librarians involvement with CRIS

Jeremy made it clear from the outset that his involvement with CRIS has been positive and that it may not match everyone's experience. One of his first slides was about how CRIS represents the convergence of a number of areas that libraries support: OA; Impact; Research profile; and digital communication. An important point (I think) was that although Jeremy is positive about these things, developments have been from genuine desire from the academic community (rather than evangelising: it is based on dialog and recognition of a need). St Andrews was one of the first institutions to use the publications database as the front end for getting data into the repository (talking to people over lunch and coffee this now seems to be what alot of folks are working towards). I think the main point Andrew was making though was that partnerships are very important. Previous involvement with the research community at St Andrews (e.g. on digital thesis) meant that they came to him when they realised that they needed support implementing a CRIS, and as a result of the work, library staff have been transferred to the research office to support the REF (as they realise that librarians have skills that are needed). Along these lines academic liaison staff are seen as an important part of the process of rolling out and supporting these systems with academics - one of the hooks for this is by showing the academics stats of how their material has been accessed via the system - the thought of increased citations is a powerful motivator.

There followed some brief demos:

Sue Cumberpatch - York - PURE

This is not quite live at York but has been released to academics, who are currently adding publications. Academic liaison librarians are currently supporting academics asked to put material on at very short notice. PURE includes publications, media outputs like radio programmes, students supervised, etc. It's been populated by the repository and web of science (they bought the InCites API to support this), and has also brought together many sources (e.g. academics records) as York didn't have publications database before this. Librarians are also supporting inputting data from reference manamgnent software (e.g. endnote). Sue had a slide showing the whole system and identifying the areas where librarians have contributed and supported. A very useful slide! Unfortunately there are not many databases that can populate PURE (just Arxive, WoS, and pubmed) and social sciences are not well covered by these. PURE can import RIS files although the import filters can be a bit tricksy.

Andria McGrath - InCites demo

Nadine demoed Thomson Reuters InCites product that extends what you can do with the citation data in WoS. You can create your own data sets within InCites. King's have set one up based on author affiliation in WoS but also a more bespoke one based on documents in their CRIS. This uses the UT number of the record in the CRIS to link to the citation data. They've found that the tool acts as a real incentive for people to get their work into the CRIS (because they can then instantly access the citation data). It's useful for the REF as you can create subject area schemes that provide data for specific UoAs for the REF but it's also useful in other ways. You can compare your outputs to other institutions for example, which can be useful in future bids.

Nadine Edwards - Greenwich - e-resources and repository manager - survey on librarians and research.

Nadine presented the results of a JIBS survey. There were no responses from repository managers - which indicates that this is often tacked on to other roles (e.g. subject librarian). Responses showed that librarians had been consulted in less than 50% of CRIS acquisitions.

Half of the librarians surveyed support bibliometrics. Most popular form is 1:1 support with academics. Majority of people don't have InCites. I later found this surprising as it emerged that InCites can be used to normalise bibliometric data. Without a product like InCites it is very difficult to do this and, as bibliometric data varies greatly between fields and even from year to year it is meaningless without putting it into context by normalizing.

Kate Bradbury - Cardiff - Research support

Kate listed some drivers for research evaluation: external ones like: REF2014; impact/ value for money; performance/ league tables; grant applications/ awards, and internal factors like: promotion/ pay progression; evaluating initiatives; departmental performance; funding allocations. (I'd also add recruitment to this list). Helping to find evidence of impact might involve searches for researchers; verifying outputs for the REF; liaison; training; and advocacy. As a skillset it helps to have: knowledge of the research landscape (e.g. keeping on top of the many HEFCE REF reports!). Kate made the point that it is hard to provide normalized bibliometrics without a product like InCites.

Afternoon session

Jenny Delasale - Warwick - Supporting bibliometrics

Jenny explained her organisational context and made an interesting point about how their support for the publications database had been communicated to academics i.e. that they were introducing a publications service to support academics with the university's publications database. Very clever marketing.

Jenny made the point that there are lots of routes to researchers and recommended Web of Knowledge bibliometrics sessions through Mimas to develop skills in this area. There was a useful slide on how librarians can support the publication process. The bit I found really valuable about this presentation was Jenny's explanation of the h index and in particular the assertation that context is everything! Basically, bibliometrics depend so much on context that you often can't provide a yes or no answer to academics questions. The example given was is it better to have 1 paper cited 100 times or 10 papers cited 10 times? All depends on context.

Isobel Stark and Michael Whitton - Southampton - Google Scholar: Can it be used for bibliometrics?

This was an interesting talk. First was a discussion on the pro's and cons of using Google for bibliometrics:

Pros:
Easy to use and free. Wide range of articles e.g. book chapters and conference proceedings so it's especially useful for law, humanities, social sciences. Metrics tend to be a higher number because more material is indexed but also there is some duplication.
Cons:
Data can be poor quality and it is not transparent where the data comes from; there's a lack of de-dupe; there are big gaps in subject coverage; the indexing (esp. subject indexing) is naff and this makes it difficult to narrow to a particular authors work; citation matching can be flaky (relies on algorithms to do this).

Isobel listed some services that are good with scholar: Quadsearch; Scholar H-index calculator (FF add-on); Scholarometer. Publisher or perish.

There was also some useful references to the literature on the subject:
Bar-Illan (2008) - Israeli highly cited researchers - differences between Wos and scopus.
Jasco (2008) - problems with the data, issues around transparency.
Franceshet (2009) - computer science metrics higher as computer scientists publish in conference proceedings.
Lee (2009) - neurosurgery - Google scholar and scopus very close.
Mingers (2010) - Business and management - google metrics higher.

Isobel made the point that you need to know the work well as Scholar doesn't deduplicate and doesn't have institutional affiliation.

Southampton bibliometrics guides are open access www.soton.ac.uk/library/research/bibliometrics - interestingly the Google Scholar guide has been accessed far more than others!

The discussion on whether to use Google Scholar was useful. I thought it was an interesting point that if someone asks for a H index without specifying where it's come from then they probably don't understand what they are asking for - is it then fair game to give them the highest figure?

As an interesting aside the Thompson Reuters rep in the room disclosed that they are releasing a book chapter citation database in November.

All in all an interesting day and I learned alot- but my brain was a bit frazzled by the end :-)

Some things I learnt today

ResearchGate looks like an interesting source. I stumbled across it after (I'm ashamed to say) doing a Google search for a thesis from University of Utah. The result showed as a page in ResearchGate's database which includes info from: Pubmed, Arxiv, Pubmed Central, IEEE, RePEc, CiteSeer, NASA library and DOAJ. In this case it linked out to the full text in University of Utah's institutional repository - which was nice. Aside from the database it also allows researchers to network using social media features. They've got a tour here and registration is free.

Resourcing reading lists takes quite alot of time and money. You probably won't get much thanks for doing it but will probably get quite alot of complaints if it isn't done. Ho hum.

Sometimes the day just gets away with you. I've been meaning to have a look at the latest edition of Journal Citation Reports for ages but still didn't get around to it today.

This is a Click Beetle (I think). It's quite freaky when one flies down and hits you in the face!

Hootsuite just keeps on getting better with Hootsuite5 being released. Now with GeoSearch, drag and drop upload and enhanced FaceBook features among other things. Oh yeah - apparently Dolly Parton uses it too.

Bridging the Skills Gap: Developing Innovative Library Support for Researchers

Oh yeah. I just remembered this blog was supposed to be a record of professional development. This one's about the Developing Innovative Library Support for Researchers conference at Sussex Uni on Tuesday 8th June 2010. It's rough and ready but here's my record (largely taken from Tweets so apols for the grammar)...

Welcome
Bob Allison, Stephane Goldstein, Alison Mitchel

Bob kicked off with a reminiscence of Philip Larkin while he was an undergraduate  at Hull and a tale about graffiti in the library - but on a more serious note he recognised the role that libraries and librarians play in research with a reflection on a conversation with a librarian after a successful RAE in 2001. The librarian felt part of the research process and was glad that the RAE was a success, something which stuck with Bob. There was also some mention of the fact that library buildings and spaces are changing to accommodate researchers and that libraries are one of the most rapidly changing areas of HE at the moment.

Stephane Goldstein, with the 1st keynote highlighted some of the work of RIN. RIN look at the interface between researchers and information - including looking at researchers behaviour - they've found that digital literacy is often inadequate. RIN's main study "mind the skills gap" http://ow.ly/1VwwY found that a gap between researchers perceived info skills and their actual skills. Researcher Development Framework is an opportunity to coordinate joined up approach to research support. A JISC call for proposals to promote data management training, funding 6 projects, was also mentioned.

Alison Mitchell delivered the 2nd keynote on researcher development. Expressed importance of researcher's information skills being transferable to jobs beyond HE. Vitae aims are about developing researchers skills rather than geared towards specific research. Roberts funding is now coming to an end - now it's up to institutions to decide how research is embedded in the organisation. There's a need to develop the marketability of researcher's skills in the employment market and info. skills are part of that skillset. On the Vitae website the "rugby team impact framework" (http://ow.ly/1VwWd) includes a tool for measuring the impact of training workshops. The good practice part of Vitae site will include examples of training materials. Alison mentioned a 100K "Innovate" fund for innovative ideas for researcher development projects.

Gearing up for e-thesis
Isobel Stark - I didn't go to this session.

MI512: Information support for researchers at LSE library
Rowena-Macrae-Gibson
Rowena Macrae-Gibson spoke about MI512: Information Literacy tools for researchers - (blogs of the breakouts are on the ning site). The MI512 researcher information literacy programme is non-credit bearing - researchers use their own research subjects so have useful outputs. The programme integrates skills e.g. the referencing and citation workshop is linked to endnote session and they help to make sense of each other. Referencing is one of the most active, and best attended sessions. NB: orcid (sp?) open source version of researcher id. LSE decided not to include their federated search tool in MI512 as they believe it isn't useful for phd level searches. MI512 was judged to be a success - good numbers of research students are booking onto programme - despite it being non-credit bearing. Good example of cross-departmental working (IPE element?). It is important to measure impact of training and good feedback from researchers helps to maintain management support for programme.

The Research Liaison Team at Sussex: an innovative model
Joanna Ball, Helen Webb

Waving and definitely not drowning!: Offering library support for academic research in a Google world
Annamarie McKie - i didn't go to this session.

Supporting the REF at the University of Brighton
Suzanne Tatham, Helen Woodward
Suzanne Tatham, Helen Woodward Spoke about supporting the REF at
Brighton (actually they changed the title from REF to Research in light
of recent news). They found that one of the biggest challenges is
getting message of what's available e.g. workshops/tutorials etc. over
to researchers.Their information skills workshops are administered by
the university wide workshop booking system and appear alongside IT,
numeracy and all other workshops - so researchers have one place to go
for all of their skills. Another benefit is that admin/ bookings are
done by the training admin team. All workshops are also available
online for those that can't make it in person. As part of their
researcher skills offering, Brighton run a bibliometrics workshop. It's
attended by staff, research students, RA's, librarians. One comment
from bibliometrics workshop was that some didn't know what a "citation"
was. Can't assume prior knowledge of basic researcher skills. Scemago
(sp?) was mentioned as an open access alternative to scopus - also
Ulrich's show's where journals are indexed if they are not in WoS.
Brighton's workshop booking system includes online form for session
feedback - can email form to all attendees. Brighton also do a Google
Scholar for citation analysis for subjects not well covered by WoS or
Scopus - they highlight the downsides but also that it covers books/
conference proceedings etc. There followed some interesting comparison
of different results for no's citations using WoS, Scopus and Google
Scholar. In the questions at the end, the common need to support researchers
with which journals to publish in and making sense of the REF for
researchers were identified as challenges.

Information skills training for researchers: the Surrey experience
Gill Downham, John Baxter
Gill Downham and John Baxter on "Information skills training for researchers: The Surrey perspective". The organisational structure at Surrey includes 2 teams - Researchers development team, Institutional repository team and Academic liaison team. The space is also important - Surrey cleared a floor of print journals to make flexible learning space for researchers and undergrads. Use tried and tested methods for developing researcher info skills: new student inductions, specific workshop sessions, 1-2-1's. Surrey's experience is that students arrive with a diverse range of skills. Their generic induction for research students - includes a recommendation that students make an appointment to see their AL librarian for a 1-2-1. Surrey include a presentation on using their repository for increasing citations in their "Getting Published" workshop - so that use of the repository is grounded & in the context of research support.

Kitty Inglis summed up... key message thatlibraries and librarians play central part in research - huge potential to collaborate & collaboration is essential in a climate of having to do the same or more with the same or less. The event evaluation was done using clickers - which was more fun than filling in a form.

Good advice about choosing which journal to publish in

Further to my last couple of posts, I've just come across some good advice on selecting which journals to submit articles for publication in an article from Pediatric Radiology (Griscom 1999). The main point of the article is to select realistic journals. The example given is the New England Journal of Medicine and JAMA. Both have extremely high standards, but also have generalised readerships. They are only likely to publish articles that are broadly applicable or transferable across the whole medical profession.

So, it may be more realistic to publish in journals that are more specific to the field of your research. I guess one question that's worth asking is, are you more likely to be cited by publishing in a journal that has a large generalist readership (where only a small set of those readers are interested in your subject) or in a journal with a smaller readership, but one that is closer to your research?

Griscom suggests authors ask a number of questions about a publication when shortlisting journals to publish in:

"Are there articles like yours in the journal you are considering?

Do you like the articles that you read there?

Are the editors and readers of these journals likely to be interested in your work?

Is this the audience you hope to reach?

If you answer "yes," turn to their instructions
for authors and read them carefully. Can you adapt
your material to fit those instructions (which you must
follow) without too much strain?" (Griscom 1999)

References:

Griscom, N.T., 1999. Your research: How to get it on paper
and in print. Pediatric radiology, 29, 81-86. Available from: http://www.springerlink.com/content/38cmdxn17l2a5y6e/fulltext.pdf

Sources for bibliometrics in the REF

This is the second of a series of posts summarising what I'm reading about the use of bibliometrics in the Research Excellence Framework. The first post looked at how influential bibliometrics are likely to be in informing the REF in the health fields.

So, working on the assumption that citation analysis is going to at least influence expert review in the health field, what are the best sources to use to find journals to publish in? As part of the REF, a lot of research has been carried out to assess the pros and cons of the various citation databases on the market.

This document: Appraisal of Citation Data Sources: A report to HEFCE by the Centre for Science and Technology Studies concludes that Scopus is a serious alternative to WoS in the science fields. There are gaps though (for example Scopus does not index journals from cover-to-cover like WoS, so there are some articles and reviews that might be indexed in WoS but are not available in Scopus). Swings and roundabouts though as Scopus is judged to be more accurate in some areas. One example is that it preserves author affiliations for all co-authors, wheras WoS only lists the institution of the first author.

There is a subject element to this as well:

"The comparison of WoS and Scopus coverage of the ‘best’ publications submitted to the 2001 RAE showed that Scopus coverage is especially better in the Subject Group Subjects allied to Health, as well as to a lesser extent also in Engineering & Computer Science and Health Sciences." (Meod and Visser 2008)

So Scopus can't be ignored - especially in the subject areas mentioned above. But I think maybe it's best not to get too hung up on the bibliometrics side of things. The latest REF guide from HEFCE clearly states that citation analysis will only be used only in those subject areas for which citation data is considered to be a strong indicator of outputs and that other methods will also be used (HEFCE 2009). It's highly likely that other indicators will be taken into account in the health sciences fields.

References:

HEFCE, 2009. The Research Excellence Framework: A brief guide to the proposals. Bristol: HEFCE. Available from: http://www.hefce.ac.uk/research/ref/resources/REFguide.pdf [Accessed 15 February 2010].

Moed, H.F. and Visser, M.S., 2008. Appraisal of Citation Data Sources: A report to HEFCE by the Centre for Science and Technology Studies, Leiden University. Bristol: HEFCE. Available from: http://www.hefce.ac.uk/pubs/rdreports/2008/rd17_08/rd17_08.doc [Accessed 15 February 2010].

Publishing for the REF?

I'm increasingly being asked to support our staff and PhD students in the field of publishing. I guess from two angles:

1) What sources can we use to determine good publications to target - especially in the context of the REF.
2) Are there any good sources of guidance on writing for publication.

For the first I'm going have to develop my knowledge of the REF a bit. It's all very well being aware of the sources that folks can use (e.g. JCR, Web of Knowledge, Scopus) but these things need to be taken in the context of what the REF is going to be assessing. Are measures such as impact factors the 'be all and end all', or will the REF take other factors into consideration?

To help me with this I'm taking a look at this document:

HEFCE, 2009. Report on the pilot exercise to develop bibliometric indicators for the Research Excellence Framework. http://www.hefce.ac.uk/pubs/hefce/2009/09_39/#exec [Accessed 12 February 2010].

It outlines the pilot study to 'develop the bibliometrics element of the Research Excellence Framework'. I think it's worth emphasising that bibliometrics is just one element of the REF and other factors may need to be taken into account when choosing where to publish. One of the key points of the study is the conclusion that bibliometrics are not robust enough to replace expert review... but citation analysis may inform the review. Another finding is that the effectiveness of bibliometrics varies depending on your subject area (i.e. bibliometrics are going to be more effective in those subject areas that rely most heavily on journals for dissemination).

So how do you know how important bibliometrics are going to be in in informing expert review in your subject area? Annex H. of the document above is a study on the availability of citation data in different subject areas and offers some analysis of how important citations might be. For my area (Health and Social Care) it's mixed news!

‘Health sciences’ is a mixed area. Many staff are professionally engaged and used
practitioner journals not well covered by commercial data. Others are in areas that
provide excellent bibliometrics. This is therefore an area in which bibliometrics may be
seen by some commentators to provide useful information but which in practice cannot
be assumed to provide sufficient information without support from peer review. (HEFCE 2009: 154)

This suggests to me that expert reviews in health sciences will not rely solely on citations (as they recognise that many practice based researchers will be disseminating research in professional journals). I think the holy grail as far as choosing where to publish goes, will be to find journals that do well in JCR and on Scopus but are also well read by practitioners.

References

HEFCE, 2009. Report on the pilot exercise to develop bibliometric indicators for the Research Excellence Framework. Bristol: HEFCE. Available from: http://www.hefce.ac.uk/pubs/hefce/2009/09_39/#exec [Accessed 12 February 2010].