Remarks from Panel on "Scientometric Analysis of the CHI Proceedings" at CHI 2009

danah boyd
MacArthur Forum
7 April 2009

[This is a rough unedited crib of the remarks I actually made. The panel concerned the paper "Scientometric Analysis of the CHI Proceedings" by Christoph Bartneck and Jun Hu. The other panelists were Gilbert Cockton, Robert Kraut, and Louise Barkhuus.]

Citation: boyd, danah. 2009. "Remarks from Panel on 'Scientometric Analysis of the CHI Proceedings' at CHI 2009." CHI 2009, Boston, Massachusetts, April 7.

I would be remiss if I didn't begin by pointing out the irony of me sitting up here commenting on this paper. I am an outsider. I have never had a full paper accepted at CHI and, depending on how you define HCI, my work may or may not be included. That said, I suspect that the reason that I was asked to comment here today is because I have chosen a different approach to the dissemination of research than many in this room.

Like many of you, my work is fundamentally interdisciplinary and the CHI conference stems from interdisciplinary roots. Yet, people take different approaches to being "interdisciplinary." Some choose to help create conferences and journals that are the product of natural intersections so as to create a "home" for researchers who are mixing it up. Others choose to move between different worlds, trying to create bridges and make connections that didn't previously exist. These folks have no home field, no home conference, no home journal.

There are costs and benefits to both approaches. One issue, as is pertinent to the conversation today, has to do with "impact." When we think about impact as scholars, we normally think about "impact to the field." Well, when you're super interdisciplinary, what is the field you're trying to impact? What is the best way to have impact? And what is the best way to measure said impact?

Today's intriguing paper raises a couple of key concerns about impact. First, in analyzing the relationship between publication and country/institution of researcher, they are asking: "is one's ability to have impact biased based on their location and affiliation?" Second, by questioning the citation value of "best paper" awards, they call attention to the biases of the process.

These are fair questions, but there are also implicit assumptions baked in here. First, there is an assumption that the best way to make an impact in the field of HCI is to publish in the CHI conference. Second, while they struggle with citation measurements in their opening, there is still an assumption that citation count implies quality or impact. Since we've already gotten into the nitty gritty of the paper, I want to focus on the high-level issue of impact.

As a field, HCI has been experiencing an identity crisis since its origin. CHI is an annual manifestation of that and, in every year I've ever attended CHI, conversations have swirled about how HCI can be valued within specific institutions or academia at-large. There's also always been a tension between research and practice, and a question about what role industry plays in this conference. There are also some very real fights as individuals in this community seek validation and legitimacy to guarantee their jobs and the future of their research. Questions of impact are tightly entwined with these other battles and my comments today are not meant to dismiss this.

But here's where we're at a cross-roads, both with regard to this field and with regard to academia and "impact." We are living in a technologically-mediated society where information is flowing in radically new ways, where specific journals or conferences may not necessarily be the best venue to reach the relevant audience. While it's easy to argue that publishing at the top journal or conference is the best way to keep an academic job, can we honestly say that publishing in these venues is the best way to have impact? Especially when these articles are locked down and made hard to access?

In my area of research, I would argue that the answer is no. I am not an applied researcher, but my research has immediately implications and applications. I want to be in conversation with scholars, to be informed by academic thinking and critique, but I also want my research to get into the hands of those who can put it to use. In my case, that means getting my research out to parents, educators, policy makers, technology developers, and youth, groups who would never think to turn to CHI to learn something. The "impact" of my work cannot be measured by citations or "best paper" awards. I measure my success based on my ability to stop insane pieces of legislation or to shift the path of technologies that could have a negative impact on society and youth or by the letters I get from parents and teens whose lives I've helped. My goal as a researcher is to get my work into the hands of all who find it relevant. In this way, I am an activist with research as my toolkit.

I am not on this panel because of my publication record or my citation ranking. I am proud of the work that I have published in academic journals and conferences, but that is only a fraction of my output and, arguably, the least impactful of it all. I have been blogging since 1997. I write essays online and in mainstream venues. I speak regularly in front of broad audiences. I make available the cribs of my talks. I share all of my academic work and have stopped publishing in venues where I can't make my articles available online. I embrace new forms of social media as a tool for getting my research out there. These are just a few of the ways in which I try to get my work out there.

I am not alone here - many of you are also embracing new ways of disseminating research. Yet, whenever we're in a room talking about public output of research, a question always emerges: but how are you going to get tenure? Herein lies the irony of the conversation around impact. Do we really want to make an impact or do we simply want to gain stature in our field?

The answer is really both. And the lack of alignment between the values of academic institutions and the relationship we have to impact is a tremendous source of tension for many in this room. There are all sorts of folks here fighting to get different contributions recognized by our ancient institutions. I commend you for fighting the good fight.

But as we sit here thinking about an article concerning the publication and citation practices of CHI, I think that we also need to step back and question our intentions. If you didn't believe in the value of research, my guess is that you could game the system to maximize citations and publications. There are plenty of folks out there who do indeed write crap that they don't believe in so that they can stir up controversy and get people to pay attention to their work. These people's work is historically dismissed, but their short-term "impact" is high simply by being a troll. My guess is that most of you don't think that this is a process we should optimize towards.

So then what do we do about impact? This is a question for individuals and for the collective. It's also tricky because impact is not necessarily about the public at large. Not all research is meant to be public and not all research needs to be in the public's hands to have "impact." But who is your intended audience and what are you trying to do with your work? How do we measure impact when the intended audience can vary tremendously in scope and scale? While citations may once have been a pointer to ideas that got folks thinking, I'm not convinced that this is currently the case. Heck, the larger the echo chamber, the greater the citation ranking.

I'm glad that folks are looking into the biases of this field - it is important to keep things real - but my hope is that we are able to leverage this work to start a conversation and move past the current status quo. My biggest fear is that the more we obsess over the ways to measure value in our field and the more people optimize for quantity and citation, the more we lose track of the big picture ideas, the long-lasting, high-impact ideas that shape the future.

Impact doesn't necessarily have to be about the public, but it does have to be about the future. It is fundamentally about getting people to think and see the world in new ways. My hope is that we can find a way to get beyond discussing impact and generate research that does impact.