This article comes courtesy of George Siemens  via Paul Coyne

Scholarship in an age of participation

George Siemens
March 27, 2007

This work is licensed under a Creative Commons License

Journals are an essential and trusted aspect of knowledge growth and dissemination. New discoveries, advances in disciplines, and critical solutions to complex problems find their home in academic journals. The future language and concepts of science and society often find their first life in peer review and formal publication. Citations are the heart of the process – establishing reputations and providing the infrastructure for knowledge growth and information integrity.

Eugene Garfield’s Journal Impact Factor (JIF) utilizes citation analysis to determine (obviously) the impact of the journal. Larry Page’s (of Google) insight into the value of links (backrub as was his initial term) as a means of determining authority altered web search and online information access (John Battelle). By treating each link as a citation, and determining value of a web page based on incoming links, web sites are assigned a “page rank” similar to Garfield’s JIF.

Citations and weblinks are the lubricant of knowledge growth. Yet journals – the vehicle of citations – possess a weakness derived from the structured process of review and publishing. The methodical process of submission – editor evaluation, numerous reviews, and finally (possibly) publishing – is intended to filter those ideas lacking solid research or possessing faulty reasoning. The process is time consuming. Publication can take from six to twelve months (in some cases even longer).

The process itself can be frustrating for authors, with limited opportunities to address faults of, or engage in dialogue around, anonymous reviews. Peer review suffers challenges similar to any aspect of society – where power and knowledge aggregate, there is room for abuse and misuse. What peer review does offer is an element of transparency and authority from experts. The informalization of information, as evidenced by increased use of informal citations in student papers, growth of Wikipedia, and use of Google for research, presents new challenges. The basis of peer review and the architecture of citations are critical for academic discourse. Our challenge is one of preserving the value of traditional approaches, while utilizing the best of emerging approaches.

Trends influencing formal publication

Four significant trends are creating conditions of change in academic journals:

• Growth of information,
• Expectation of participation,
• Increased openness,
• Two-way flow

Growth of Information
The growth of information hardly requires proof – we feel it in our daily lives. The growth of multi-media, internet, information management systems, advanced search engines, and academic contributions from emerging economies are only a few of the changes making their presence known in our personal lives. Our personal experience is validated with numerous studies. A research project at University of California at Berkeley stated that the global information base grew 75% from 2000 to 2002. A recent IDC report predicts a six-fold increase in digital information between 2006 and 2010.

The argument for change is simple: When characteristics and context of knowledge – the core element of the journal process – change, the processes, tools, and institutions which interact with knowledge must change as well.

Expectation of participation
Late last year, Time Magazine declared “you” – the amateur journalist, podcaster, blogger, wikipedia editor, and those who contributed to, and created the current participatory culture – its Person of the Year. The digital habits of many online participants have changed. No longer are we satisfied to simply consume the content of others. We desire to create, to participate, to collaborate, and to be involved. In many cases, content consumption is blended with content creation – a culture of create, co-create, and re-create.

Increased openness
Growing concern about the public “paying twice” for information (once in the research dollars to fund the research and again in reading the research in a journal) is driving a shift in open access in educational materials.

Peter Suber states that:

Open Access (OA) “is compatible with copyright, peer review, revenue (even profit), print, preservation, prestige, career-advancement, indexing, and other features and supportive services associated with conventional scholarly literature. The primary difference is that the bills are not paid by readers and hence do not function as access barriers.”

Two-way flow
Tim Berners-Lee’s intent with the web was not to create a broadcast medium, but instead to create a read-write medium. While this vision languished for many years, partly due to the complexity of technology and publishing and partly due the “architecture of participation” being unformed. The last five years have largely attended to these challenges. Social software enables anyone to setup and publish his or her ideas. Many news sites now offer discussions around articles and audio and video files. As Public Library of Science demonstrates, formal, peer reviewed journal articles benefit from annotation and commenting features. Essentially, the two-way flow, read-write nature of scholarly communication removes “established knowledge” from the pedestal where only select few can comment.

Implications
For many, the citations of formal journals have given way to the page rank of Google, or the tags of Technorati. Journal citations are the specialty of a small segment of society – the academically proficient. Technorati, Google, and del.icio.us are the citation machines of the masses. The filtering performed by journals – through editorial and peer review – tests information before it enters the public sphere. Today’s online publishing tools enable anyone to publish, and testing and validation of information occurs through the actions of many (links, comments, blog posts, or social bookmarks).

The rigid, sometimes restrictive, nature of journals results in learners often soliciting more accessible and less complex sources of information. A large part of the challenge stems from lack of learner familiarity with the process of peer review – a key information literacy weakness. A process more in line with the spaces and tools of learners today – situated in a community-based environment – may prove to be an important resource in setting the foundation for the next generation of researchers and academics – blending the value of emerging tools with the proven model of review and citations.

Existing in two worlds
Numerous disciplines are facing a foundational shift in their method, process, and end user. Music, newspaper, television, radio and movie industries are embroiled in core redefinition of how they relate to their customers. Google has altered basic information search, and now threatens to alter academic search as well (through Google Scholar and Book Search).

Against this backdrop of changing end-user expectations, developing technologies, and changed flow of information, academic journals must adjust to retain their relevance. The changes moving forward require a balance of honoring what has worked well with journals – peer review and the citation model in particular – and adopting those democratic elements revealed in Amazon’s reviews (though anonymous), Digg’s rating, and Wikipedia’s collaboration. We need to begin experimenting with our scholarly routines to reflect the needs of today’s researchers, learners, and society.

The value of journals is not in question. The process and pace of journal development, however, is experiencing increasingly difficult challenges. The pace of journal publication is too slow in many fields. Ideas that have been discussed at length in online forums, blogs, and conferences often only appear in journals several years later. The process concerns are based on blind review and lack of community participation and discussion.

The fault lines of “expert vs. amateur”, “genius vs. community”, formal vs. informal, need not be drawn thickly. Instead of separate and opposing camps, a gradient model of shades perhaps best reflects a suitable model for moving forward. By keeping our feet in two worlds – citation and review of traditional journals as well as participative, open emerging models – we are able to attend to broad range of needs for academics and today’s learners.

Academic Scholarship Today
Blending the best of traditional journals with emerging tools of managing high levels of information presents unique opportunities for moving journals forward as a cornerstone for information creation, dissemination, and sharing.

The following are guiding principles are suggested:

1. Two-fold model: peer-reviewed and informal commons
2. Open reviews
3. Meta-Reviews
4. Discussion
5. Annotation
6. Journal as community

Two-paths
Our need for scholarly work runs on varying gradients between formal and informal. The easy access of search engines and sites like Wikipedia, provide a simple access point to “quick and dirty” information. More involved research (such as writing a thesis or submitting an article for formal publication) requires greater use of traditional scholarship. Our knowledge need drives the tool we require. As many bloggers have discovered, peer review can help to shape and create ideas prior to publication (Chris Anderson’s book Long Tail).

To attend to this dual need for information, a journal should permit traditional peer-review, as well as the informal review of the commons. As detailed in Figure 1 of a proposed flow of a “current journal”, an author has the option of submitting a document for either formal review or commons review (though even the formal article ends in the commons after review). Articles that initiate in the commons can be moved through the formal peer process if the author chooses (and the community rates the article sufficiently well). Readers of the journal will rate articles posted into the commons (similar to Stumbleupon or Amazon rating or the Digg metric of raising the profiles of articles ranked by the community). Articles that are established are then published in the online journal as well as a paper journal. OJR forms the base of the system.

Open reviews
Anonymous review is frequently criticized as a limitation of journals. Journals need to make the comments of all reviewers public in order to form the basis of deep dialogue. No source of information should receive a privilege status. All information is available to democratic dialogue.

Meta reviews
Healthy systems permit feedback. Members of a community require the ability to “review the review”. This may be a controversial approach – the anonymity of reviewers enables expression of ideas that may be difficult in open public forums. As a democratic model, however, the ability to rate the value of each review is important. Even experts are not immune from changing pressures to the creation and dissemination of information. Editors, journalists, researchers, and others are subject to the back channel models of evaluation.

Discussion
Articles, which have gone through the commons or the formal review process, are subject to annotation and discussion. Any member of the journal community has the ability to comment on the articles, and engage the author and community members in discussion. Discussions are appended to each article. Discussions of a more general or cross article nature can be held in separate forums.

Annotation
Annotations differ from discussion in the granularity of focus. Annotations focus on or address a single idea – a statistic, citation, or comment. Public Library of Science uses an annotation system where a blue asterisk is placed inline to alert readers to an annotation.

Journal as community
A journal is an opportunity to move beyond content or information consumption. While “community” and “journal” may not appear to fit together well, journals typically bring together the prominent thinkers and interested stakeholders of a discipline. Enlarging the conversation of journals to include deep discourse on articles and annotation throughout, sets the basis for a democratic, social model of scholarship.  

Challenges
The established structure of peer review and academic publication is a difficult place for experimentation of new ideas. Nature’s experiment with open review resulted in poor researcher and reader involvement: of 1369 papers, only 5% agreed to open peer review. Of those, only 54% received comments. The poor showing of articles submitted to open review led the publishers of Nature to conclude: “[We] will continue to explore participative uses of the web. But for now at least, we will not implement open peer review.”

Poor performance of Nature’s foray into openness could be due to numerous factors: apathy on the part of community members (wishing to read instead of participate), uncertainty of the process, researcher’s reluctance to put ideas into public spaces before peer review, the nature of the discipline (fields of philosophy and psychology, for example, may foster more formative discussion as compared with hard sciences), and general lack of familiarity with participative processes. Nature’s experience provides important consideration in continued experimentation to revise the nature of scholarship. For the proposed journal discussed next, it is hoped that greater reliance on community will serve as the crucial element in increasing dialogue.

Next Steps
Theory finds its fullness in application. The interplay between theoretical constructs and the lessons learned in application require a malleable approach to journal formation. Issues of identity, fairness, civility, and engaging in democratic environments, require dialogue and an adaptive approach. Quite simply – as academics, we do not have a clear model of implementation for scholarship in light of current online trends. We need to adopt and experimental approach of sensing, evaluating, and responding to trends. Beyond being a community, the proposed journal is emergent – reflective of, and responsive to, the community it serves.

The dramatic changes to how information and knowledge are created, disseminated, and consumed are forcing traditional industries (and any information structure) to change as well. The experiences of newspapers, journalism, the music and movie industries, can serve as an indication to the types of changes academics face. Perhaps, instead of banning participatory sites, we can avoid mistakes of others, and begin experimenting with models of adaptation that preserve the best of tradition, while simultaneously incorporating new approaches to knowledge creation and dissemination.

Your Involvement
The principles discussed in this paper form the basis of a collaborative project between North American and European researchers and academics. We are requesting involvement from individuals interested in contributing to the development of the journal – enlarging its representation to include a global audience. If you would like to participate in discussions to shape the journal itself, or subsequent involvement in the community, please let us know. The application of ideas, of course, tests, informs, and revises theory. The journal will focus on emerging trends in educational technology and pedagogy, exploring fields of social software, connectivism, and networked learning. The current group of journal founders has established a conversation space (you are invited to create an account and contribute to the conversation. If you have specific questions, please email me). We will use Open Journal System as the base of the journal, with modifications to allow for dialogue, annotation, and the process detailed in Figure 1 of this article. Our intent is to provide a journal free of charge to authors and readers (many open journal models charge authors or institutions, not readers – while this is a viable model, our experiment is focused on volunteer efforts).