Thursday, April 19, 2012

Total impact: a critical perspective

Total impact is a set of alternative metrics for scholarship. Total impact looks at a number of factors, including social networking tools such as Facebook and Twitter. As a research tool, Total Impact has a lot of potential, to explore how works are used and what impact they have. As an alternative metric to evaluate the quality of scholarship - an alternative to impact factor - there are a number of potential serious problems to consider.

First, there is the question of whether the current push towards quantitative metrics makes any sense at all. I have talked about the problems of this kind of instrumental rationality is my book chapter,  The implications of usage statistics as an economic factor in scholarly communication https://circle.ubc.ca/handle/2429/954 (In brief, usage stats are likely to have significant negative impacts, from discouraging use to discouraging important but not necessarily popular entire fields of research); and the second chapter of my draft thesis http://pages.cmns.sfu.ca/heather-morrison/chapter-two-scholarly-communication-in-crisis/ (search for irrational rationalization).

One example, from an interview study of scholarly monograph publishers I did recently, is the impact of pushing scholars to publish more books to obtain tenure. This pressure is not consistent with the time it takes to write books that are really worth publishing and reading; so in this instance, we have a quantitative metric intended to improve quality and productivity (of our academic staff) which appears to be lowering quality (more mediocre books, more book production when the problem for all of us is not enough reading material, but rather not enough time to read).

Research, both quantitative and qualitative, is my recommendation before putting too much stock in Total Impact or other alternative metrics as a replacement for Impact Factor in evaluating the quality of scholarship. Note that I do not recommend retaining Impact Factor, but rather minimizing or eliminating quantitative approaches to evaluating the quality of scholarship.

The use of social networking tools may be particularly problematic. One research area that I recommend is examining the effects of traditional biases. I would hypothesize that social networking tools would be likely to exhibit the following biases prevalent in modern society:
  • men would show more impact than women
  • minorities would have lower impact
  • developing country authors would have lower impact
  • introverts would have lower impact than extraverts
  • authors adept at social networking tools would have more impact than authors less comfortable with these media
In addition, I would suggest that metrics based on social networking tools could easily be manipulated, not only by authors but also by interested others. It doesn't take much to imagine corporate polluters in favor of climate denial upping the impact of their preferred pseudo-science, or for drug companies to drive up the impact of studies making the drugs that they sell look good.

Another area to consider (for all kinds of tools, not just social networking tools), is the impact of funding considerations. If universities are relying more on corporate funders at a time in society such as today when the political will seems to be on the side of the wealthy who wish to cut spending on social justice, it is reasonable to hypothesize that there will be a smaller proportion of funding for social justice issues. What happens to research on poverty issues and its impact if there is less funding for research and academic positions in this area (hence fewer researchers), at the same time that there is less funding for government services and social workers (fewer potential readers and tweeters), and many of those poverty research is meant to help have lost their homes and jobs and may find it difficult to get internet access?

7 comments:

  1. I completely agree with all your caveats and the emphasis on efforts to reduce metrics. However, given the current oversupply of candidates and diversity of the research, it's hard to see any other practical solutions than metrics. In other words: as much as we might despise such metrics (and the effects they can cause, when ill-devised), unless someone comes up with a superior alternative, we have no choice. Thus, we are forced to develop the best possible (or least bad!) metrics to minimize any ill effects caused by their introduction. Therefore, I support initiatives such as altmetrics. In fact, at least for now, I support all initiatives to develop new metrics as an inflation of metrics not only allows user to chose the optimal set of metrics for their own purpose, it also reduces the chances of gaming all of them and, last but not least, inflation reduces the value of any single metric and thus any negative impact it might have.

    ReplyDelete
  2. Bjoern, many thanks for your comments. Two thoughts:

    Is there an oversupply of candidates, or an undersupply of jobs, both in academia and in society as a whole? I would argue that the latter is the case, and that the underlying problem is that the political will in western society over the past few decades has prioritized private profits over everything else, including jobs for people.

    From another perspective - are there really too many smart, capable people with advanced degrees eager to contribute to society as teachers and researchers, if only they could get a position with at least a reasonable salary and job security? I argue that this is not the case at all. We on this planet have some very serious problems to deal with, ranging from climate change to figuring out how so many of us can live together, preferably in peace, without completely destroying our environment. If we accept my argument that western society needs to shift priorities and values so that jobs for people, quality of life, and a sustainable environment are the priorities that I and many others think they should be, then there is much that needs to change to make this happen - from politics to how we think about things like economics. This takes deep thought - we need our scholars, in the humanities and the social sciences. What about arts and culture? Is this not the very best of what we humans do? Is it possible to have too many artists, musicians, dancers, great literature and movies?

    Finally, Bjoern, I see that your article was featured in Peter Binfield's presentation on altmetrics, a great example of an article with a tremendous number of downloads. Congratulations!

    What advice would you have for someone like me who sees it as the duty of a scholar to speak the truth, even when it is not likely to be popular? I have no doubts that my "total impact" would increase greatly if, instead of presenting my critical perspective, I were to uncritically applaud the approach. Should I aim to speak the truth, or to go along with what is popular to increase my "total impact"?

    ReplyDelete
    Replies
    1. To your first point: there are two reasons why I can't answer that question. The first is ignorance: I simply don't know how many scientists (in the widest sense) a society needs or should afford/invest in. I'm a biologist and not a sociologist. The second is bias: I'm a scientist, so I have a vested interest in positions for scientists. Thus, I have to refrain from calling for more money being spent on me and my buddies, no matter how much I might think it would be worth it, no matter how much I might agree with you. There is a mismatch between supply and demand and if nobody creates new positions, it's up to us to not train people in the face of such bad job prospects.

      Thanks for the kind words on our paper, I'm sure all the downloads only come from Pete's presentations :-)

      As a scientist, I try as best I can to avoid the word truth, as truths tend to be quite personal in many cases. Speaking your mind is both a virtue and a luxury. As a virtue, I'd of course advise all to follow it and nurture that virtue. I'm hesitant to ask or urge people who cannot afford that luxury to speaking their minds - which is why it is even more honorable and valued if someone speaks their mind irrespective of the consequences. In other words, I won't advise people to take positions which are against their own self-interest, but will commend and value such acts.

      Finally, the data shows that it's also the controversial discoveries or hypotheses which garner 'impact', see e.g. #arseniclife. I'm in the process of reviewing literature on journal rank and there the outcome will be very counter-intuitive and controversial, contact me directly if you'd like to see that manuscript.

      Delete
    2. Thanks for your comment, Bjoern. There is a world of difference between speaking your mind, and a commitment to seeking the truth.

      Universities and society in general has work to do to match the talents of our population with the work that we need to have done and would like to have done. It is important to keep in mind the broader societal picture. If there were more jobs with decent pay and job security available to people right out of high school, this would impact whether and why people go to university.

      However, I would caution against equating education with developing people for a labour market. For example, in Canada our government has practically declared war on the environment (to use my MP's phrasing). How should we react? Should all environmental programs that are designed to figure out how to actually care for our environment (as opposed to greenwashing) be cancelled or cut back? There certainly won't be as many jobs.

      There are times when we do need to focus education to meet both societal and labour market needs. British Columbia's expansion of medical education in recent years designed to resolve a problem of lack of supply of doctors, nurses, and other health care practitioners is a good example

      Delete
  3. Possibly of interest: Melissa Terras: is blogging or tweeting about research papers worth it? http://blogs.lse.ac.uk/impactofsocialsciences/2012/04/19/blog-tweeting-papers-worth-it/ Melissa finds that when she blogs or tweets about her articles in the institutional repository, downloads go up by quite a bit.

    Comments:

    This is interesting, and suggests a research angle for altmetrics types. Worth noting that Melissa is a regular blogger / tweeter with 2,000 followers on twitter. Based on her description of her social media interactions, I suspect she is an extravert. It may very well be the case that scholars who are regulars at social media and use this as a venue for communicating their work will have more impact - just as scholars who are more comfortable with outreach are currently more like to receive awards for service to media - but is their scholarship of greater value?

    What happens if (or when) others see this as an opportunity? If well-funded journals like Scientific American can afford to hire active bloggers like Bora Z., will this become an imperative, further driving a divide between the wealthy have and the have-not publishers? Will scholarly blogging for journals become an imperative - and add to the costs of publishing? Will universities and departments get into the act? Maybe they will hire bloggers. That's nice - jobs, presumably for people with some academic background - but won't this just take money out of the pot available to hire faculty?

    Thanks to Keita Bando on google g+.

    ReplyDelete
  4. "It may very well be the case that scholars who are regulars at social media and use this as a venue for communicating their work will have more impact - just as scholars who are more comfortable with outreach are currently more like to receive awards for service to media - but is their scholarship of greater value?"

    Very interesting! I guess that you were asking a rhetorical question here, assuming that the obvious answer is no.

    ... But is it? If research reaches more people, then maybe it is more valuable. That sort of has to be true, because otherwise in the limit case paper that no-one ever read would be valuable. (Shades of @ClosedAccessJ!)

    So I'm not sure what I think about all this. I do share your reservations about how easy it is to game some metrics. Probably the biggest thing Altmetrics has going for it right now is that it's not Impact Factor. And frankly, that might be enough.

    ReplyDelete
  5. Thanks for your comment, Mike. Here is a highly probable scenario that illustrates how the kind of measures of impact altmetrics is looking at can work against scholarship:

    Richard Smith, in his book The trouble with medical journals, discusses the publication planning of the pharmaceutical industry - planning for dissemination of results starts before the research. The purpose of dissemination for the pharmaceutical industry is to advance their profits, not the knowledge of humankind. These companies have lots of money to manipulate social media.

    If an honest scholar is doing their best to disseminate work that shows a reason for concern with pharma's drug, what chance do they have of equal impact with a corporate PR machine?

    I would argue that we need mechanisms to counter this sort of influence. Altmetrics, as a suite of research tools, could do a lot to track the problems so that they can be remedied. However, altmetrics as a measure of scholarly worth trades one set of problems (impact factor), for another set that could be much worse (amplifying corporate distortion of knowledge(.

    ReplyDelete

Thank you for your comment. Comments on IJPE are moderated.