Composition Forum 20, Summer 2009
http://compositionforum.com/issue/20/
Three On a Match: Gary A. Olson on Rigor, Reliability, and Quality Control in Digital Scholarship
Abstract: This interview examines the relationship between digital scholarship and the politics of higher education. In doing so, it advances a series of recommendations that aim to help digital scholars and digital scholarship achieve an increased level of stature in the academic community.
Rhetoric and composition scholars who publish in electronic venues are all too familiar with the struggle to have their work validated by fellow colleagues and university administrators. The sobering reality is that in spite of the massive investments of time, energy, and creativity that writing and publishing webtexts require, many skeptics remain convinced that electronic scholarship is less serious, less rigorous, and thereby less valuable to the academy. Of course, those of us invested in composing digital scholarship want this situation to change; we not only want skeptics to understand the difficulty of integrating text, image, video, and sound, but to recognize how this form of scholarship expands our intellectual horizons. Still, the question remains: what steps must be taken in order to convince our peers and university administrators that electronically based scholarship is vital to our field’s intellectual growth?
To gain some perspective on how to answer this question, I recently sat down with someone who not only holds an administrative position as the Dean of Arts and Sciences at Illinois State University,{1} but who is familiar with the struggle to make an emerging discourse that pushes boundaries of rhetoric and composition studies a legitimate site of scholarly inquiry: Gary A. Olson. Because Olson has also written extensively on academic publishing and the politics of higher education, he possesses a crucial perspective on the tactical steps necessary for convincing university administrators that electronically based webtexts are viable forms of scholarship. The following interview is comprised of a series of follow-up questions that invite Olson to elaborate on answers he gave in a recent interview published in Composition Studies entitled, “Returning to the Table: A Conversation with Gary A. Olson.” Whereas in the previous interview, Shelley DeBlasis and I asked Olson to comment on a broad range of scholarly topics, this particular interview is focused on questions concerning the future of digital scholarship and the role it might play in shaping the discipline of rhetoric and composition studies. For example, I ask Olson to comment on a range of issues related to electronic publishing, including: the necessary directions that scholarly websites must take in order to be valued as scholarship, the role that blogs should play in our disciplinary discourse, the Open Access movement, and the value of Wikipedia. Although Olson’s recommendations are specific to each question, many of them center on three key issues that in his mind must be addressed in the effort to legitimate electronic scholarship: rigor, reliability, and quality control. When discussing the importance of verifying scholarly websites, for example, Olson contends that a rigorous peer-review process is crucial for establishing the level of quality and reliability that these texts must possess in order to be valued by scholars and university administrators. When discussing the peer-review process of electronic journals, Olson returns to these issues again, arguing that e-journal editors must become more creative in how they cultivate an awareness of their peer-review practices. Until the issues of rigor, reliability, and quality control are addressed on a scale that exceeds our own subdisciplinary conversations, Olson expects that we will continue to “spin our wheels” unproductively.
Although several of Olson’s responses are quite provocative, we should keep in mind the seriousness with which he makes his recommendations. For example, some may disagree with Olson’s comparison of blogs to conversations that take place in a barroom or coffeehouse, and many will take issue with his characterization of image, video, and sound integration as digital “bells and whistles”; however, as he claims repeatedly throughout this interview, digital scholarship offers significant contributions to our field specifically and to the academy at large, and for that reason he wants it to gain the level of legitimacy that those invested in publishing electronic scholarship desire. To that end, Olson’s recommendations should be read as thoughtful encouragements to scholars whom he wants to succeed, not antagonism from someone who resists this form of scholarship from the starting gate. Our hope is that this conversation will spark the crucial dialogues that will lead to the type of change we all want to see.
Kyle Jensen (KJ): Steven Krause recently published “Where Do I List This on My CV? Version 2.0,” which examines the promises and complications of viewing self-published websites as a valued form of scholarship. Krause refers to your presentation “The Value of Virtual Scholarship in the Academy's Printcentric Economy,” from the 2001 MLA Convention, as a key perspective in developing long-term solutions that make this form of scholarship valuable in the eyes of university administrators. For those who are unfamiliar with your presentation, would you give a synopsis of your argument, discuss what motivated your interest in this area, and explain why your argument is still relevant in the current moment?
Gary A. Olson (GAO): What I was talking about in that piece, which was part of a special panel put together for MLA on the future of scholarly publishing in the digital age, is that the scholarly website doesn’t have the advantage of having gone through any kind of vetting process in the way that a book manuscript does when it goes through the editorial process at, say, a university press. Now, I am not talking about e-journals here; I’m talking about a scholarly website that a person might put up about a particular literary figure or artist, or on some other subject. Because a scholarly website does not go through a traditional vetting process, there’s no way to certify, if you will, the quality of the website. Now, you can argue that some books get published whose quality is not as good as it ought to be, and that would be true. However, at least in that process, and especially among the better presses, there is a very keen preoccupation with quality control. So, for example, when you send a manuscript to Johns Hopkins University Press, you’ll know that first it gets screened by the in-house editors, who will know a lot about the subject matter and who will decide whether they even want to bother sending the manuscript out to external reviewers. Then they will send it out, often to three distinguished scholars in the discipline who will give it a careful read because their reputations are involved. And then, if the process follows the typical pattern, those reviewers will have suggestions for revisions, sometimes major revisions. The author will then revise, and the manuscript will probably still get vetted yet again. This process will go through several cycles before the manuscript sees the light of day in print. Nothing like this happens in the “publishing,” if you will, of a scholarly website on the Internet. It goes through no review of any sort and no independent editing; it is simply posted on the web in whatever form the author wants it to take.
Now, what that means is that scholars can’t rely on any given site to be accurate, truthful, or knowledgeable about the subject matter. You don’t know who put it up: an undergraduate, a high school student, a graduate student, a distinguished professor, an associate professor, a non-academic—you just don’t know. For example, a few years ago I wrote a monograph on a contemporary literary figure, and at the time there was a website devoted to this person’s work that claimed to offer the most comprehensive bibliography of his scholarship. Well, it was not comprehensive. It was ridden with inaccuracies, mistakes, omissions, and incorrect information. It was not reliable at all, but because it was one of the only websites devoted to this figure, I suspect a lot of viewers did rely on it. As it turns out, this website was run by a student, and while I think there are plenty of bright graduate students out there who might even do a better job than some faculty, that’s beside the point. When you don’t know how credible the website’s information is, that introduces a level of uncertainty that we should not have to deal with as scholars.
The proposal in the MLA presentation was that we develop a process of certifying scholarly websites. What seemed to me to be the most feasible and workable process was to have the major professional organizations in English studies—MLA and CCCCs, let’s say—devise a certification process for these kinds of websites. So, suppose you had a scholarly website on some figure—on Aristotle, maybe. You could voluntarily apply (and perhaps there’d be a modest fee) to have it reviewed by a panel of experts that the organization would select. If it passed their inspection, then you will have earned the right to put a seal of approval on your site from MLA or CCCC (or whatever major organization) that is clearly visible to all who visit the website. That seal would certify that the site had undergone a vetting process, that a panel had found it to be of a level of quality that other scholars could rely upon. I also recommended that the certification be time dated because, as you know, websites can be transformed and updated so rapidly that they can be quite different a year or two later from what they were a year or two earlier. So, to maintain the certification, you would have to reapply for it every so often, and we would have to work out how often is often enough. The idea was that this plan would provide a process that would enable the scholarly community to start introducing a level of reliability into this kind of publishing that would make it a worthy rival to exclusively print-based forms.
As you know, digital publishing can do substantially much more than just print. Even before you start talking about audio and video streams and so on, even if you just stay with the use of text with visuals, you can do so much more on a website. Websites are actually better suited than print for a number of types of scholarship. However, if you don’t know who constructed it, what his or her background is, and if there is no process of review, then scholarly websites are never going to achieve the level of reliability that is currently enjoyed by print forms.
Even though I made that proposal a few years ago, I would still make that proposal today. The other recommendation I made is that each of the associations maintain a registry of certified sites. If you are in rhetoric, you could consult the CCCC’s registry and find out if a certain site on computers and composition, let’s say, has been certified by the organization.
Since I made that proposal, some people have responded, “Well, we don’t use that kind of certification process for university presses or for scholarly journals, why should we have to use it for web-based publishing?” The answer to that is simple: there is already a built-in process in the journals and presses, but there is no such process for scholarly websites on the Internet. What I am trying to do is devise some kind of process that’s workable and, once again, adds that element of reliability so that then we can move forward with digital scholarship. What I fear is that we’re not really going to move forward as far as everybody wants to with digital scholarship. We’re never truly going to get there until we arrive at the point where we can introduce that level of reliability. It may well be that I don’t have the best solution and that there may be a better solution, but there has got to be some solution. Otherwise, there is going to be a lot of wheel spinning for a long time.
KJ: Have you received positive responses to your proposal, particularly the version that was recently published in The Chronicle of Higher Education?
GAO: I have received positive responses since the article was published. In fact, a few people in disciplines other than English have asked how I would go about setting such a certification process up in their fields. There is a general sense in a lot of disciplines that the question of certifying the quality of scholarly websites needs to be addressed. Although there are different and differing notions about how to go about this work, we are moving toward a consensus that there needs to be some way to address the issue of quality control.
KJ: As someone who has published several scholarly websites, you clearly understand the value of this type of work. Where do you list these websites on your cv? Do you view the work that goes into these websites as a form of service, or do you consider websites such as The Theory Project a form of scholarship?
GAO: You’re not going to like my answer to this question. Scholarly websites are not “scholarship” in the commonly accepted sense of the term, although composing them is a scholarly activity. We talk about scholarship as developing new knowledge in a field, of working within a tradition—the metaphor I have often used is contributing to “an ongoing conversation”—and adding to a field’s conversation or knowledge, disputing parts or maybe all of it, but working within a conversation and trying to contribute in one way or another. Now, you can engage in activities that further scholarly work but that are not scholarship, and this is what some people do not understand. The Theory Project website that you mention is a good example. It is a website archiving the audio interviews that I and others have published in JAC over the years. This website allows you to listen to the original, raw recordings of what got very carefully edited and published in print form over a two-decade period. The Theory Project is an aid to scholarship that may help scholars do one thing or another; it may help them revisit what Derrida said in the interview and find out that he really didn’t say it quite that way, that he really meant something different. In that sense, the website might lead to some interesting scholarly work. However, it’s not a piece of scholarship in and of itself, and that’s a distinction people need to make. There are scholarly related activities that do not, or should not, count as scholarship per se.
Doing work on a journal could be considered a scholarly activity. Consider a book review editor, for example. Is a book review editor doing scholarship? No, he or she is not doing scholarship. That person may be engaging in a scholarly activity that is furthering scholarship by soliciting and publishing useful book reviews on scholarly topics, but that editor is not engaging in scholarship.
Now, there are all kinds of activities that we do as faculty that fall into this category, and too many people want to lump most of these activities together under the heading “scholarship.” However, it just doesn’t work if you’re being honest with yourself. Being the president or secretary of your professional organization, being a manuscript reviewer, being an editor, or putting up a website like The Theory Project —these are all activities that help further scholarship, but they are not scholarship, and that fact is really a thorn in the side to some people. Nonetheless, it’s a distinction that must be made because otherwise we lose the definition of scholarship altogether. If more and more things get lumped together under the category “scholarship,” then “scholarship” is going to become so capacious that it will mean anything and everything, and it will therefore mean nothing. I maintain that we really need to draw distinct lines, even though there will be grey areas between the kinds of scholarly activities that we engage in.
So, where do I list the websites on my cv? I list them under a heading that simply says “websites,” and then let other people decide what kind of value they want to give them. By the way, just so this doesn’t sound like a matter related just to digital scholarship, there are plenty of activities back in the print world that call for this same distinction. For example, I could write and publish works that are not scholarship but that may have some relation to scholarship. If I write an article on how to publish scholarship, let’s say, am I producing scholarship? I don’t think so. I am engaging in scholarly service. And by the way, that is the category on a vita where a lot of these activities should be listed. As you know, professors get evaluated all the time in three key areas: teaching, scholarship and service. Service has several different subcategories: there’s service to your institution, as when you serve on university-wide or college-wide committees; there’s service to your department, maybe you’re on a tenure committee; there’s community service; and there’s also service to the discipline. It’s the latter category where a lot of these activities that have a bearing on scholarly work but that are not actually scholarship go. I should admit that years ago when I served as a journal editor I might have been reluctant to make this distinction between scholarship and service to the discipline so starkly, but this is in fact the commonly held value system in the academy.
KJ: But, clearly, you are able to “let others decide what kind of value they want to give” these websites because you are a tenured professor and a university administrator. So, what should young scholars such as myself do if we are interested in producing similar types of work? Obviously, as someone who is untenured, I can’t publish these sites with the same level of comfort that you just described.
GAO: A tenure committee is always going to determine the value of a particular kind of work. Hopefully, that decision is based on the written requirements that the department has already laid out. Your question has to do with how to list it on your vita. With these new kinds of scholarly modes, the best course of action is to compose topic headings that accurately represent what is being listed. For example, saying something is a “scholarly website” rather than a “publication” differentiates what type of work you have done. Then, department guidelines and tenure and promotion committees will decide the relative value of a scholarly website, on the one hand, and a printed scholarly monograph, on the other. My advice to anybody is to be as clear as possible when specifying the various kinds of work you include on your vita—especially when it is experimental work—because you don’t want to look like you are padding. The vita should simply state facts; let the tenure committee determine value.
KJ: In our earlier interview in Composition Studies, you spoke about the Open Access movement, arguing that one of the crucial issues in this conversation is the role that a rigorous peer-review process plays in the reliability of electronically published texts. To make your point about the realistic possibilities for quality control, you drew upon the example of Museum Anthropology Review, an open access journal that uses the same editor and peer-review process as the print-based journal Museum Anthropology. In your professional opinion, will administrators in charge of tenure and promotion view publications in open access e-journals as equivalent to publications in a print-based journal even when the editor and peer-review process is the same? If not, what steps need to be taken in order to legitimize the value of e-journals that employ a rigorous peer-review process and whose clear focus is the distribution of credible scholarly work?
GAO: Now we’re on similar ground to what we just talked about with regard to scholarly websites. The hurdle that digital journals will need to get over in order to gain the legitimacy that you and most of the readers of this journal want them to have is the issue of quality control. Quality control when it comes to e-journals—open access or otherwise—has to do, again, with the peer-review process. Take the Journal of the American Medical Association as kind of a paradigmatic example. Publishing in JAMA may well be the crowning moment in your career, and to publish there your article has had to survive the most rigorous scrutiny that you can ever imagine for scholarly work—as well it should, since you are dealing potentially with people’s health and sickness, and possibly life and death. JAMA is not going to risk its reputation by slipping up when it comes to the rigor of its review process, nor are the reviewers going to slip up, because their reputations are on the line. Certainly, those of us in English studies couldn’t pretend to be producing scholarship of the level of importance and consequence of the work in JAMA, but, nonetheless, if that is the paradigmatic example of how scholarly work gets vetted and then published, a weaker but parallel version of that process is in play in our established journals. This is why it makes a difference to a tenure and promotion committee whether you are publishing in a peer-reviewed journal or a non-peer-reviewed journal. Some institutions won’t even consider the articles you published in non-peer-reviewed journals, or they will consider them to be so low down in the value scheme that they’re practically worthless. This is also why most institutions will consider top-tier journals—College English or JAC in our discipline, let’s say—as of much more value in the process than lower-tiered journals.
There is no equivalent to that process yet when it comes to e-journals. As I said before, there is a mimicking of that process in the example of Museum Anthropology and Museum Anthropology Review, because you have a very established scholarly print journal with a very good reputation for rigor that sponsors a parallel journal that uses the identical process of review, the identical reviewers, and the identical editor making very similar kinds of decisions. So, you have a move in the right direction with that open-access journal, a move that all digital journals are going to have to make in order to gain the legitimacy and value that they want to have. In general, however, there’s a long way to go.
Now, I know that there’s already a kind of folklore about some journals being more rigorous than others. Kairos, for example, has a very good reputation, but right now (and this is not meant as a slight) its rigor is still in the realm of folklore and will remain so until we find a way to establish once and for all the quality of the review process in this and all other e-journals. This is going to take time, in the same way that a print journal builds its reputation over time. Most journals start out being a little more loosey-goosey in the beginning as they get established, and then as the journal is more in demand and more high-level scholars want to appear in the journal, then the journal can be more selective. This process takes time. Right now, all of the e-journals are a bit too young for that process to be well-developed.
I know that journals like Kairos, Computers and Composition Online, Composition Forum, Enculturation, and The Writing Instructor all pride themselves on their peer-review process, but it’s one thing to talk among ourselves about the legitimacy of a journal’s process, and it’s an entirely different thing to establish that legitimacy for people outside of our circle. This is what I fear people are not doing. I think we’re talking to ourselves, “Oh, we know this journal is rigorous, while this other one’s not quite as rigorous,” and that discussion is just going on among ourselves. This is where the danger lies: we are all convinced, but are the tenure committees convinced? A lot of them are not. Credible scholarly work needs to have that peer-review process, and quite honestly I think the digital journals are going to have to be a little more creative in how they manage the process and how they get the word out about how the process works.
KJ: Do you have any suggestions on how electronic journals should “get the word out?” Is it enough simply to make the details of your peer-review process available on the journal’s webpage?
GAO: Clearly articulating the process is the first step. However, electronic journals should probably do much more. For example, it would be a good idea to have statements in your standard correspondence with authors about how the journal operates and that its peer-review process aligns with certain principles. Of course, it also behooves a journal editor to do as much PR work as possible in getting the word out about how the journal operates. If you think about it, this is a matter of convincing people that electronic journals operate—in its submission and decision making process—in a manner similar to how print journals operate. Let’s face it, at least for a while, you are going to have a lot of skeptical people in the academic world, and you want to convince as many of those skeptical people as possible to be comfortable with your particular review process.
KJ: In that same interview, you also expressed concern over scholars who spend a considerable amount of time blogging and/or writing on listservs, expecting such writing to be counted as a scholarly activity. In light of this argument, I wonder what you think about CCCC’s recently launched blog, which boasts a host of entries from well-established scholars such as Mike Rose, Paul Kei-Matsuda, Malea Powell, Krista Ratcliffe, and Victor Villanueva. What kind of value do you place on this type of writing, especially since it is organized by the discipline’s largest professional organization? Do you think that blogging can be a way for scholars to become more active as public intellectuals?
GAO: Now you’re playing right into one of my pet peeves. You know, it’s one thing to talk about the value of e-journals, or let’s say the future value since we still have a ways to go. I am all for digital scholarship. I am for those websites that I talked about before, where you could do really creative things in an effort to further knowledge in the field. And I am really for moving toward more electronic journals, but we’ve got to get over those hurdles that I have been talking about. And I’m optimistic that we will. Other kinds of activities on the Internet, however, do not rise to the level of these forms that we’re talking about, and blogs are one of them. This is not to say that I am dismissing blogs as a useless activity, although it’s certainly not an activity that I would want to engage in, but what I am saying is that we’ve got to draw distinctions between what gets done in the pages of Composition Forum, on the one hand, and what gets spewed out in some of these blogs, on the other. Scholarly e-journals are attempting a very noble experiment. They’re trying to advance scholarly work to a new level, and let’s face it, digital scholarship can have all kinds of advantages: it’s searchable and, well, you can name more advantages than I can. So, I see these as very laudable moves.
Blogs, I think, perform a service to those who engage in them, in that they serve as forums where people can vent about issues and engage, I guess, in some debates. But, as I said before, you can have quite interesting debates and discussions and vent in the barroom, or over tea and coffee in the coffeehouse, but those activities, while they may be on scholarly subjects, do not rise to the level of scholarship. Yet, time and time again, people try to count them as scholarship. I’m sorry, I’ll simply repeat what I said before: if we count anything and everything as scholarship, then the word “scholarship” means nothing.
This is not to say that blogs are useless, or that scholars—even highly visible ones such as those you just mentioned—shouldn’t be bothered with them. What I do fear, however, is that so many younger faculty and graduate students—and by “younger” I mean newer to the field—will spend so much time venting and discussing things on blogs that they will expend a huge amount of their creativity and time on something that ultimately will not get them very far in their careers when they could have used that time much more wisely by investing it in producing scholarly works that will count for something at the end of the day. This redistribution of time away from producing scholarship and toward these more informal discussions is a great worry to me because I am afraid that this will so cut down on some scholars’ productivity that they will have trouble producing a body of work that is considered qualitatively and quantitatively sufficient to further their career, especially when it comes to tenure and promotion.
At the risk of seeming hopelessly old-fashioned, let me repeat the point that I made before: you can be right about the way that things are moving historically, and dead wrong when it comes to your own personal career. What you don’t want to be is the person who was denied tenure and who is then saying, “But I was right: the world is going to go in this direction sooner or later.” You may be right, but you’re out of a job.
KJ: Couldn’t blog entries in prominent disciplinary forums give doctoral students such as myself a different kind of exposure to the field that wasn’t necessarily available before digital publications? In other words, can’t blogging for CCCCs, and joining conversations with established scholars in a sanctioned digital forum be viewed as a good way to get your name out there, to gain an edge?
GAO: Yes, absolutely, but there are all kinds of valuable activities related to the scholarly enterprise—there were such activities even before the days of digital technology. The question is: which activities constitute scholarship that can and should be used in a tenure evaluation for a faculty member, and which are simply helpful in the professional development of a faculty member? Often, I think that line gets confused. As I said, you can have wonderful scholarly discussions in coffee shops, but it will never be considered a piece of scholarship—it is just a conversation about scholarship. What we usually look for—in traditional kinds of scholarship, anyway—is whether or not the scholar is creating knowledge or furthering the scholarly conversation. And although, yes, you might come up with great ideas in a blog, great ideas that are even contributing to the “conversation,” is that a sufficient basis on which to grant tenure? I don’t know that many places have gotten to the point where they would answer “yes” to that question.
Let me take this out of the context of digital technology for a moment. If as a young faculty member, when I was coming through, I spent all my time going to scholarly conferences and sitting in the bar area holding wonderful, complex discussions with the great leaders of the field, that would have had a wonderful effect on my personal and professional development. However, although the subject matter is scholarship, the activity that I would be engaging in, in this hypothetical example, is not scholarship. I am not producing a text that enters into the formal disciplinary conversation and moves it forward.
By analogy, I see the same thing happening with blogs. These are wonderful avenues for professional development, for engaging in the exciting discussions of the time. However, at the end of the day—to prove yourself a scholar—you will need to produce scholarship. Although some people want to say that blogs are scholarship, I just can’t see it, and I think that many people agree with me. So, it is much better to use your time not sitting in the lounge at the conference exclusively; yes, do that a bit, but don’t forget to spend valuable time sitting in front of your computer creating a scholarly work.
KJ: For the e-journals that adopt a print-centric mode of presentation, the concept of rigor is more easily understood because such journals can rely on a traditional set of peer-review and formatting practices that are familiar to college administrators. However, for journals that publish new media webtexts, the traditional understanding of rigor does not apply as readily because such texts require a set of composing practices and peer-review processes that are largely unfamiliar to a majority of administrators, if not the university at large. Kairos, for instance, employs a three-tier review process: the editors first decide whether a piece is worth sending to the editorial board; the editors then send the webtext to the entire editorial board and invite a broad, two week discussion on the promises and limitations of the piece; after that discussion occurs, three members of the editorial board are asked to compose a traditional review of the piece, offering suggestions for revision and recommendations regarding the potential for publication; if the piece passes through this stage, the editors then begin to work directly with the web-text author(s) to revise the piece both in terms of content and design. The idea is that because the composition of web-texts requires a diverse range of skill sets, the review process needs to bring in a range of expertise that can address not only what we traditionally describe as the text’s argumentative content, but address the argumentative content performed by the design of the text as well. In your view, how might e-journals that publish new media scholarship help college administrators understand the unique and clearly rigorous processes that these published web-texts undergo? How can university administrators break with their traditional understanding of “rigorous” scholarship so that these innovative features of electronic scholarship may be valued?
GAO: I think that there is a certain amount of confusion in the question, because the process that you describe, the three-tier review process that Kairos uses, is very much in keeping with the traditional review process in many journals. That would not be unfamiliar at all to administrators. The fact that you have more than text—images and so on—doesn’t change anything appreciably at all. In fact, we have that in print today, maybe not in English studies, but in certain kinds of art studies journals and so on. So, none of this is new. The process you are describing is perfectly in keeping with the kind of rigor that traditionalists are after.
What is confusing is to talk about the process in terms of a range of “skill sets” and so on, or in the “time invested” in a project. The rigor that we are talking about vis-à-vis the review process is not a test of your skill; the rigor has to do with whether you have done your homework on the subject you’re dealing with. Does your article enter and then further an ongoing conversation in the field? Have you framed your research problem in an intellectually legitimate way? Are you attempting to establish something that has already been established conclusively? Are you drawing on the appropriate sources? Do you make inferences that flow from the facts? That’s the kind of rigor that we’re looking for. It’s not in how well you execute the style of your article. There is no prize for good writing and good integrating of texts and visuals. The question is: are you truly contributing to the conversation going on in your discipline and thereby adding to knowledge? The kind of review that Kairos apparently follows is clearly in keeping with that, and so I applaud the editors.
Now, you talk as if administrators are simply not aware of the differences between print and digital journals and that you’ve got to train them. Well, I think a lot of administrators want the move toward digital scholarship to work; I certainly want it to work. However, the burden of proof is on you and those who are going down this path. This is what I was trying to do in the piece on scholarly websites. There needs to be a way, and I am not exactly sure what it is, to convey to all concerned the level of rigor that goes into the review process at electronic journals committed to rigor.
Obviously, the first thing to do is apparently what Kairos and some of the other e-journals I’ve mentioned are already doing—that is, advertising widely and clearly exactly what the review process is. This needs to be done over and over ad nauseam because there will always be skeptics. There are plenty of old-fashioned people (perhaps I’m one of them) who will always be suspicious of digital scholarship. However, the more that you get the word out there, the better. Quite honestly, one of the key ways that print journals have established their reputations for selectivity is by their acceptance rate, or more likely by their rejection rate. Every e-journal should clearly publish its annual acceptance rate. Is it twenty percent, forty percent, eighty percent, or ten percent? It makes a difference. Widely publishing that information, along with providing a clear description of the review process, will help quite a bit. For some people, that will not be sufficient in and of itself. What beyond that needs to be done, however, I don’t know. Maybe over time, in the same way that print journals needed time, various e-journals will rise to the top, others will stay in the middle, and some will languish at the bottom, but that process will probably take another decade.
KJ: I guess the point that I was trying to make is that electronic scholarship, particularly the type of new media scholarship that appears in journals such as Kairos and Vectors, requires a different kind of review process where the rigor of the argument proper is one concern among many. As an editorial board member for Kairos, for example, I know that when considering a webtext for publication, I not only have to factor in the rigor of the argument, but the visual composition and execution of the piece, which requires a different kind of rigor altogether—a rigor that is more than how we typically conceptualize “style” in print-based manuscripts. I understand what you mean when you say that the work one puts into any piece, whether print-based or digital, does not factor into tenure considerations, but isn’t more value given to more ambitious projects? Isn’t a book valued more than an article? An article more than a book review, and so on? Is there a way to account for the ambitious undertaking of creative digital projects, particularly when we know how much they can enrich our scholarly conversations?
GAO: Sure, and this is precisely where disciplines and individual departments will need to be much more precise about how they will evaluate new modes of scholarship in the future. I have no problem with a lot of new forms of technological scholarship; however, because they are new, it is very important that we establish clear guidelines—and by “we” I mean people in the disciplines as well as in departments—as to what will be evaluated and why, and where the value in a particular kind of forum should lie and why. That is not something that should be legislated from above, but needs to be discipline specific. If we could get disciplines analyzing these new kinds of forums and coming up with clear resolutions about where particular kinds of scholarly modes should fit within the scholarly work done in the discipline, this would be immensely helpful to departments on the local level—particularly when they are trying to revise their guidelines for tenure and promotion to account for these new forums.
I know this all sounds very legalistic, but it is very important to have these discussions and to extend the type of analysis that leads to new guidelines. I make this argument because new guidelines will serve as a basic protection to faculty in the tenure and promotion process. What you don’t want is a situation like you have now, where certain faculty believe that because they are dealing in new digital forums, they are on the cutting edge; then, when tenure and promotion time comes along, they discover that their assumptions about how something was going to be valued were wrong. So, I support accounting for these new forums, but it needs to be clearly articulated in the formal guidelines both of the discipline and each department’s tenure and promotion guidelines.
KJ: As numerous writing instructors will no doubt attest, Wikipedia has become a central influence on how contemporary students conceptualize the work of research. Oftentimes, conversations about Wikipedia center—once again—on the issues of reliability and rigor. Clearly, however, the issues caused by Wikipedia are much larger than how our students conceptualize the work of writing and research. For example, in a 2006 article “Know It All: Can Wikipedia Conquer Expertise,” published in The New Yorker, Stacy Schiff reports that “Senators and congressmen have been caught tampering with their entries” and that “the entire House of Representatives has been banned from Wikipedia several times” (2). She also reports that at the time of publication, July 31, 2006, the entry on the “2006 Israel-Lebanon Conflict’ ha[d] been edited more than four thousand times” (1) since it was put up six hours after the kidnappings; the article was created on July 12, 2006. As someone who has his own wikipedia page, and who has published several times on the perils of “the rhetoric of assertion,” what are your views of this online project? How should scholars and students in the academy view Wikipedia, especially considering that it is a non-profit organization that disseminates free information in more than 250 world languages?
GAO: This issue is another real worry of mine—and, again, I hope that the various things I’ve been saying in this interview will be understood in a nuanced sense. Wikipedia serves a certain function, and I certainly wouldn’t want to ban it from the Internet, but I fear that it is enabling a whole generation of people, even scholars, to be less critical, in the sense of critical thinking. Wikipedia encourages a process of collaboration that works in a democratic spirit, and that’s a noble experiment, but I question whether it should be a source for scholars doing serious intellectual work. I also question whether it should be a source for undergraduates attempting to learn how to do academic work, especially when the temptation is simply to go to Wikipedia and not consult more legitimate sources such as real encyclopedias or primary texts. I’m afraid that Wikipedia will become the place where everybody goes for information; that would be a catastrophe, especially when we are trying to teach students to become intellectuals, or at least critical thinkers. At the end of the day (and here we go again with that terrible word), we can’t rely on a source like Wikipedia in the same way that we can rely on more authoritative texts, digital or otherwise. The information we find there may be accurate in some entries, and flat out wrong in others. So, while I wouldn’t want to do away with Wikipedia, I think it does the academy, and perhaps the culture, a disservice.
Closely related to this issue is a subject I’ve written about in several places. In recent times, our culture has arrived at the point where we seem to believe that all opinions are equal, and this is a terrible development. People seem to be confusing the fact that in a democracy you have “the right to express an opinion” with the belief that “all opinions are equal.” However, this conflation is a non sequitur; it does not follow. Having the right to express your opinion is one thing; assuming that just because you have an opinion that it’s one worth having is another. If I express a racist opinion, is that opinion equal to someone’s nonracist opinion? I think not. What is being bred—not just by Wikipedia, but by the proliferation of blogs—is that people believe they can simply state opinions and that those opinions are necessarily worth listening to. That’s just not the case. If we continue going down this road, we’re going to lose our ability as a culture to make real distinctions, to back up assertions, to have, in effect, logical arguments that have some weight; we’re just going to be left with opinion as a default, and that would be a disaster, especially for those who wish to be academics. If academic life is about anything, it is about understanding a subject and having some authority behind your understanding so that the pronouncements that you do make about the subject have some purchase with other people.
We need to be very careful with how much weight we give to experiments such as Wikipedia (and, let’s face it, they are experiments) because we may well be eroding the very thing we stand for as academics: knowledge and some level of certainty about it.
KJ: It surprises me that you make this argument given your scholarly commitment to rhetoric and the promotion of traditionally silenced voices. Do you think that Wikipedia necessarily fosters the lax thinking you describe? Couldn’t it also provide a site for knowledge that productively disrupts the traditionally authoritative discourses that have dominated the way we currently represent historical events? Highlight the need for revision more clearly?
GAO: Sure, it could have that effect—there is no doubt about it. However, what I worry about is that with open source forums like Wikipedia, which can be perpetually revised by anyone, you can never really rely on what it is you are reading. The chances are that a particular entry might be exactly correct—and by “correct” I mean that it may correspond with the way things actually happened at that time (in the case of an event, say)—and then again, it may not be. The uncertainty about how much we can rely on the particular description, as it is presented at this point in time, is a real problem.
Now, I also recognize the irony that over the years I’ve espoused a form of postmodern thinking that tends to look askance at “certainty.” However, when I argue against assertions of certainty, I am referring to arguments that assert truth with a capital “T.” The fact of the matter is that even while espousing a move away from Enlightenment rationality thinking, there is still a level of factualness that we have to rely on just to get by in our daily lives. In my view, experiments such as Wikipedia work against us in this regard.
KJ: In the recently published video production “Remembering Composition” composed by Todd Taylor and Bump Halbritter, you claim that the incorporation of audio and visual material forces writers to become much more conscious of audience than they might have been be if they just played with the text. You go on to argue that composing with audio and video can be particularly useful in helping student writers develop stronger rhetorical sensibilities. Would you comment further on your position? Does your argument imply that instructors should ask their students to compose multimodal texts in order to help them develop a keener sense of how rhetoric works? Do you see the composition of new media texts as the future of both our field’s scholarship and our future teaching practices?
GAO: This is a difficult question in a certain way because what introducing audio and visual into a writing class can do, as you just said, is help students understand that true rhetoric is about addressing a particular audience, for a particular reason, in a particular context. The extent to which you understand this premise is the extent to which you are a good writer; everything else is a side issue. Because audio and video introduce more elements that help you keep your audience and purpose in mind, they could be instructive. Of course, as we move deeper and deeper into the so called “digital age,” more and more communications are going to be multimodal anyway, so it makes sense to have a command of all the options.
With that being said, and again at the risk of seeming terribly old-fashioned, there is real value in learning how to do the really basic work before learning all the bells and whistles. For example, I think it’s extremely important that you first know how to add, subtract, multiply and divide by hand, with a piece of paper and pen, before you simply rely on an electronic calculator. Having mastered that skill gives you an edge, because you understand it conceptually much better than you would if you never learned it and were simply presented with a calculator and told how to use it. When the batteries on the calculator die, then you’re lost. Somebody could give you a piece of paper and pen and say, “Figure out your salary for the year,” and you won’t be able to do it. So, knowing the basics helps.
I would say that the same is true when it comes to writing. Knowing how to address a particular audience, for a particular reason, in a particular way, without audio, video, computer, or other device puts you at an advantage. It is much better to make sure that you understand these basics first, because it puts you in a real advantageous position—and when I say basics, I’m not necessarily referring to understanding how commas or subject/verb agreement work, although I really do hope that that is part of it. After you’ve mastered the basics, then, it’s time to introduce some of these other modes, because you are going to do much better even with audio and visuals once you know the basics.
So, I guess all I am saying is that I would really hope that we don’t lose sight of some of these basic skills in lieu of the bells and whistles.
KJ: Do you think that the “basics” and technological “bells and whistles” are necessarily at odds with one another, or is it your sense that our current preoccupation with technology and teaching with technology is causing us to ignore or, perhaps, marginalize some of the fundamental issues that characterized writing and rhetorical instruction prior to the emergence of digital technology?
GAO: I don’t think they are necessarily at odds, and I do believe that audio and visual components make an argument much more effectively than one might be able to make without them. However, at the end of the day, you have to know how to make an argument first. You may have a wonderful presentation, but your argument does not follow the rules of logic; maybe your argument is a non sequitur. You need to be able to begin with the basics of how to present something rhetorically, and then use the audio and video components to enhance that argument. If the argument is done well, then both facets should work hand in hand. If you can’t do the first, if you can’t make the argument rhetorically and articulately, then the audio and visual is not going to help you very much.
KJ: What, in your mind, are the changes that must occur in electronically based publishing, and what future actions would you like to see as a university administrator invested in these conversations?
GAO: In a way, I’ve already touched on these issues, but I can recap since this seems to be a good place in the interview to do so. We have to solve the one, overriding problem that is inherent in e-journals, open-access journals or articles, scholarly websites, and other forms of digital publishing: how do we address the issue of rigor, reliability, and quality control? Those of us who believe in digital publishing have got to prevent ourselves from simply asserting that we’re on the right path and concluding that everybody else needs to catch up; or, to just say, “My journal is really rigorous, so there!” Making these assertions makes you feel good at the time, but it doesn’t get you anywhere with an audience that is already skeptical. There’s got to be a way to establish the reliability, the quality, and the rigor of all of the work that we do for the Internet. Otherwise, what’s going to happen is that we’ll have a bunch of people just talking to themselves about how rigorous their journals or websites are, and a lot more people not believing them, period. If this situation continues, we will not have progressed much at all.
What we need to do is something like what I was approaching in the MLA presentation. What I tried to do in that paper (and I recently published a version of that paper in my Chronicle of Higher Education column as “Certifying Online Research”) is to start the kind of discussion that I think has to happen before we are going to achieve the goals that we really want to achieve.” You can’t just assert how good you are; you have to demonstrate the quality and rigor of your work to those people out there who would rather not see you survive. That’s going to take some effort.
If we start the discussion about what to do about scholarly websites, and there are an awful lot of wonderful ones out there, we need to begin by asking: how do we establish some kind of mechanism where people can rely on them? Maybe the way outlined in my piece is not the way to go; maybe we don’t want MLA and CCCCs anywhere near our websites. Perhaps we need to create a completely independent certifying body. Who knows? Do remember, however, that in my proposal the certification process is voluntary; nobody is required to seek certification. Regardless, there’s got to be something akin to that certification process where there is a generally accepted standard and way to know that this William Blake website is sanctioned and reliable, whereas this other one was just put up by a bunch of hacks who don’t know what they’re doing.
I would say the same thing for e-journals. We may be certain of the quality of a certain e-journal, but right now we’re just talking to ourselves. There are plenty of tenure committees in English departments, not to mention college-wide tenure and promotion committees made up of faculty from other disciplines—and don’t forget you have to go through that hoop as well—where the quality and reliability of this work is not so certain. We need to devise a review process, whether it’s through MLA, CCCCs or some other independent panel, that will have some kind of purchase across the discipline and that will certify e-journals, as well. So, I would say in the most general sense that is the change I would like to see. Until we get an acceptable and accepted certification process in place, we will witness more of the same wheel spinning, and more of the same talking to ourselves and patting ourselves on the back. We’ve got to convince the skeptics as well.
Notes
- At the time of the interview, this was Olson’s administrative position; subsequently, he has accepted the position of Provost at Idaho State University. (Return to text.)
Works Cited
Halbritter, Bump, and Todd Taylor. “Remembering Composition (The Book): A DVD Production.” JAC 26.3-4 (2006).
Jensen, Kyle, and Shelley DeBlasis. “Returning to the Tables: A Conversation with Gary A. Olson.” Composition Studies 36 (2008): 110-23.
Krause, Steven D. “Where Do I List This on My CV? Considering the Values of Self-Published Web Sites. Version 2.0.” Kairos 12.1 (2007). 29 Dec. 2008. <http://kairos.technorhetoric.net/12.1/binder.html?topoi/krause/index.html>.
Olson, Gary A. “Certifying Online Research.” The Chronicle of Higher Education. 6 June 2008. 29 Dec. 2008. <http://chronicle.com/jobs/news/2008/06/2008060601c.htm>.
———. “That’s ‘Your’ Opinion.” The Chronicle of Higher Education. 13 Feb. 2007. 1 March 2009. <http://chronicle.com/jobs/news/2007/02/2007021301c.htm>.
———. Justifying Belief: Stanley Fish and the Work of Rhetoric. Albany: State U of New York P, 2002.
———. “Jacques Derrida on Rhetoric and Composition: A Conversation.” JAC 10 (1991): 1-21.
Schiff, Stacy. “Know It All: Can Wikipedia Conquer Expertise?” The New Yorker (July 31, 2006). 29 Dec. 2008. <http://www.newyorker.com/archive/2006/07/31/060731fa_fact> .
The Theory Project. Ed. Gary A. Olson and Lynn Worsham. 2007. Illinois State U. 29 Dec. 2008. <http://lilt.ilstu.edu/theory/>.
“Three On a Match:” A Conversation with Gary A. Olson from Composition Forum 20 (Summer 2009)
Online at: http://compositionforum.com/issue/20/gary-olson-interview.php
© Copyright 2009 Kyle Jensen.
Licensed under a Creative Commons Attribution-Share Alike License.
Return to Composition Forum 20 table of contents.