This memo is available in the following formats, besides the HTML version that follows below:
The patterns of scientific communication are changing rapidly. The importance of electronic preprint archives is growing rapidly, including both centralized archives like the Ginsparg archive in Los Alamos, and distributed system like the NCSTRL (Network of Computer Science Technical Report Libraries). The American Association of Universities (AAU) has recently issued a proposed initiative for "streamlining scientific publication", including a proposal for "decoupling of certification", meaning that peer review is done separately from the publication in the sense of `making public' or `putting on-line'. The Association for Computing Machinery has an ambitious electronic publishing policy, and has appointed a committee, chaired by Joseph Halpern, for working out its detailed strategy.
In this paper I shall describe the experience of the ECCAI (European Coordinating Committee for Artificial Intelligence) in developing an electronic publishing medium called the ETAI (Electronic Transactions on Artificial Intelligence). The evolution of the ETAI since it was announced in May, 1997 seems to indicate that the changes to scientific publication can be much more dramatical and thorough-going than what has been projected so far. In fact, we seem to witness the emergence of new forms of scientific communication, and not merely of publication, with profound implications for how research is performed and how research is organized, and not only for how its results are reported.
At the outset, ETAI was planned as an electronic journal that would differ from conventional journals in several important respects. The fundamental idea was open reviewing, where a large part of the reviewing process should be an open discussion, similar to the question period after a conference presentation, or to a public Ph.D. defense: the questions and the answers as well as the identity of the discussants are open to the public, although in our case they would appear in written form and over the net. Only the final part of the process, leading to a decision of acceptance or non-acceptance should be confidential like in the conventional system. It was expected that feedback to the authors, controversies, and requests for clarification and improvements all take place during the open part, the reviewing. Then the final, confidential part, called the refereeing could be limited to a simple pass/fail decision.
This choice of approach necessitated a number of other choices:
Today, the situation is radically different. While an article is still in its reviewing stage, it may be accessible to the whole world by the click of a mouse. It's only natural to admit that it has been published then, with the protection of scientific priority that comes with that concept.
Another necessary consequence of ETAI's publication concept was that we had to reach the primary readers of each article. By primary readers, we mean the active researchers in the same speciality: those who work on the same research issues as the article addresses, who are prone to using the article's results, and who are also the ones likely to ask questions and to critique the article. The open discussion or `defense' of the article will naturally take place in this community of primary readers.
For the reasons that have now been described, ETAI was initially defined for a two-level publication scheme, consisting of a publication layer and a reviewing layer. These are different layers in the sense that they are organized in completely different ways.
The purpose of the publication layer is easy to summarize: to make the article easily available, independently of space and time. The basic service is of course one of making each submitted article available on the net, or more precisely, of making articles available on the net so that they can be submitted. The reviewing layer will only receive articles that have already been published using the publication layer. This principle is diametrically opposite to the policy of conventional journals who maintain that they only accept "previously unpublished work".
Besides making articles freely available on the net, the publication layer must also guarantee that articles can not be changed. Unlike in ordinary file systems, it must in particular guarantee that the author himself or herself is not able to change the article after the stated publication date. What is more, the publication layer must be organized in such a way that everyone is convinced that noone is able to tamper with the article after it was published. This calls for both technical and administrative safeguards.
The reviewing layer contains both the open reviewing and the subsequent, confidential refereeing, as outlined above. In fact, there are three points where the two layers interact:
First publication can not consist merely of the author putting the article on her or his own web page, since then the integrity of the article over time could hardly be guaranteed. For similar reasons, we could not say that any article that has been published as a departmental technical report qualifies as published for the purpose of ETAI reviewing. On the other hand, it was not possible (especially not on the European scene) to assume a centralized repository for all articles. We therefore decided to define a concept of First Publication Archives or FPA's, that is, organizations that are able to accept vthe required responsibilities.
The publication layer is thus organized as a family of First Publication Archives. We would expect every major university to run its own FPA, just like it has a university library. In fact, Linköping University Electronic Press, which was the first FPA, is organized in close cooperation between the university library and the computing center of its university. The ETAI itself does not participate in the FPA's: it specifies a number of requirements for authorization of FPA's, but that is all. The basic requirements are:
FPA publication works fine for first publication and revised publication as defined above. But what about journal republication? We chose the following solution: The ETAI prescribes a standard style (in Latex) for submitted articles. This style is chosen so that the journal version of the article can be obtained by removing a few pages at the beginning and replacing them by a few others. These operations can be easily done on the level of the postscript version, and are in fact almost entirely automated. Therefore, the reviewing layer is able to create the journal republication version of the article by an easily managed transformation on the FPA published version. The reader is invited to visit
| http://www.ida.liu.se/ext/etai/... |
The publication layer and the reviewing layer are partitioned along opposite axes: each university may have its own FPA, the present ETAI can review articles from any of its approved FPA's, and when there are several journals similar to ETAI, each FPA may serve any of them. In addition, however, the ETAI is organized as a number of research areas, such as "planning and scheduling", "reasoning about action and change", "intelligent human-computer interaction", and so on. Articles are submitted to research areas, discussed within them, and refereed within them. This is the only way an article can be accepted by the ETAI. If it does not fit into any of the existing areas, then it can't be received.
Each ETAI area has an area editor who organizes the work, and an area editorial committee which is in charge of the refereeing. However, the area must also have a "constituency", that is, a list of researchers who identify with its choice of research problems and research methods. The active participation in the open reviewing must come from the constituency.
First of all, the discussion is posted and preserved. It would not be sufficient to only run it as a conventional mailgroup. This is for several reasons:
In our experience, it was very difficult to get off the ground with that approach: we ran into a vicious circle of few discussion contributions, few visits to the web page, and accordingly a low motivation to contribute to the discussion. The solution that evolved fairly soon was to create a Newsletter, which can be described as a somewhat edited variant of a mailgroup. It works as follows: contributions are sent to the area editor, presently using a plain E-mail message. One or more contributions are collected into a newsletter, which is sent out to the constituency. As far as possible, newsletters are sent out the same day as the contributions come in, but never more than once a day. (This provides a protection against overly intense interactions). All newsletters are accumulated without change to a webpage structure, so a subscriber never needs to save arriving newsletters - they are retrievable from the webpage of the ETAI area in question.
Besides retaining the plain newsletter in .txt format, there are also some other operations. The text is reformatted into .html for more convenient reading. Also, the constituent messages are organized into the pages for the respective articles that they are commenting about, and in a structure where one can easily follow what is the answer to what question. Finally, the article discussions are also reformatted into latex and used for generating a postscript file that reads conveniently on paper.
Now we have described everything that is required in the reviewing layer. The area editor has to organize a web page for the area, and he or she must have a mailing list for sending out newsletters. When an article is submitted to the area, an article interaction page must be set up, and the constituency must be informed in a newsletter issue. When discussion contributions come in, they must both be circulated using the newsletter, and accumulated to the interaction page. After three months of discussion, the area editor asks the authors whether they wish to revise the article, awaits the URL of the revised article, then nominates three referees who decide whether to pass or fail the article, posts the verdict, and informs the editor-in-chief. The latter maintains a web page for the ETAI as a whole, where accepted articles are posted in journal look-and-feel format.
In principle this is all. In practice, there are a few more aspects to it that must be mentioned:
In a longer perspective, one may consider extending this scheme so that there is always one or two "opponents" for each submitted article, similar to how Ph.D. theses are defended in some countries, or even similar (in some ways) to a court proceedings. It is important, in these cases, that designated opponents shall never be asked to be referees for the same article.
Being an area editor in this sense need not be very much work, in particular if one has adequate software support for the various operations involved. (A forthcoming article will describe the software support system that has been developed for this purpose). It does however require responsiveness: the arrangement is only credible for the constituency if incoming messages do get forwarded and administered the same day, with no or minor exceptions. Anyway it is certainly possible for this task to be performed by the research community itself. The only function that is performed by journal publishers today and that the ETAI system is not set up to cope with, is to handle articles from authors that do not have access to contemporary computer and Internet technology. In our context, some special solution has to be found for them.
The following is one example of how the ETAI reviewing layer has changed the reviewing practice. (... example ...)
The two-layer system consisting of a publication layer and a reviewing layer corresponds fairly well to our initial design hypothesis at the time the ETAI was started. Only the emergence of the newsletter with direct E-mail of daily issues (when needed) was an extension of the original concept. However, once this basic structure was in place, we began to see other activities that could also be run over the same structure. To repeat, the review layer provided an ETAI area with:
These additional uses of the mechanisms that were created by and for the reviewing layer, have now become so varied that they are best viewed as a third layer of communication activity, the colloquium layer. We talk of an electronic colloquium, keeping in mind that the traditional academic colloquium is a forum where researchers meet in order to hear the latest news in their area of interest, to listen to results, to ask for advise about where to find what information, and so on. The electronic colloquium serves many of the same roles, but in a way that defies distance and that is also preserved over time.
This is already enough for talking about a new form of communication between researchers. Distance education is already a fact on the undergraduate studies level; the ETAI experience suggests that a similar development may be within reach for graduate study and for research. With the ETAI electronic colloquia, grad students anywhere can follow the discussions between senior researchers in their field of interest, and they can also participate in those discussions.
However, the scenario offers additional possibilities, which we have barely started to tap into. One such possibility concerns the size of written contributions; this is a key to new ways of publishing systems research, in particular. Until now, articles in computer science have tended to be fairly long: long in comparison with some other fields, such as physics, and long in the sense that reading an article often requires a considerable investment of time. At the same time, authors often wanted to write even longer articles; journals editors are known to require authors to cut down the size of the articles.
In the ETAI scheme, we can extend the size range both upwards and downwards. Long articles are not a problem in the sense that it's the author institution that pays the cost anyway; it is possible to cover the entire spectrum from article-length to book-length publications. Conversely, short notes have never been widely used in our literature, presumably for a number of reasons:
In its present form, the ETAI provides a system for handling conventional research articles and information about those articles: questions to the author, commentary by the author, and so forth. It defines a communication channel between reseachers in an area. This channel was initially set up for the purpose of the open reviewing process but, once in place, it is also being used for many other communication purposes.
This experiment has only existed for less than a year, but even during this short time it has developed some features and facilities that were not anticipated at the outset. Therefore it is likely that additional developments will emerge during the coming years, and for the same reason it is very difficult to predict the direction of those developments. However, we shall anyway venture to speculate on some of the obvious possibilities.
Developments in this area will only happen in response to a perceived need: a perceived failure of the existing system. Consider, therefore, the question of how systems oriented research publishes its results. For a long time, there have been complaints about how this works. Conference participants request more systems papers and fewer theory papers. Conference organizers answer that they really try to get systems papers, but few are submitted, and many of them have insufficient quality. Authors of these papers complain that their work is only weighed for its theoretical contents, and that it is therefore too harshly treated in the reviewing process.
It may be appropriate, therefore to question whether the conventional-size article is a natural way of communicating the results of a systems oriented research project. Is it reasonable to squeeze the description of a large project into a short article? Is it reasonable to require the system description to also contain an analysis of, and comparison with alternative approaches? And of course: is it reasonable to have articles that do not allow the reader to test-run the system being described?
For the sake of the argument, consider the following alternative scenario for publishing the results of systems-oriented reseach, in an ETAI-style constituency where several of the participating groups build comparable systems. First, each group develops a publication structure that describes their system, and maintains it on-line. Thus, they do not necessarily generate a sequence of papers with more or less overlapping contents; instead, the group makes a serious job of maintaining an on-line documentation structure consisting of a number of parts that can be exchanged asynchronously, and which at each point in time is a fair description of their system in terms of motivation, principles, and architecture as well as details.
Then, the on-line discussion mechanism of the ETAI system is used for comparing notes about these systems. Their performance for specific examples or specific classes of problems is compared in debate mode. Also, a few researchers from different groups may cooperate in making a detailed comparison of their respective systems from some specific point of view, and publishing the result, possibly sparking additional discussion. News Journal notes may be used for contributions that are too long or too substantial to be edited into a plain discussion protocol. Throughout the discussion, the primary publications of the respective groups are used as framework and reference for the discussion. However, those readers who are not themselves actively involved in exactly the same kind of research are likely to profit much more from reading the discussions than from trying to read the full articles from each project. - Let us use the term comparative publication for this way of communicating research results.
It may well be argued that comparative publication is more adequate for systems oriented research than the traditional paper format is. But if we think so, there are again a number of issues that must be addressed. In particular, how do we assign merit for research that has been done and communicated in this way? The traditional merit scheme is based on identifying influential articles, often also counting the number of articles, taking into account the prestige of the journal where the article was published, and counting the number of citations to the article from other, later articles. Most of these criteria become inadequate and misleading in the comparative publication style.
So how will reseachers that subscribe to comparative publication obtain their promotion? First of all, the members of a promotion committee are likely to find the discussion protocols for the debates between the research groups more informative, and easier to understand and evaluate, than the orginal articles of each of the groups. An ambitious evaluation of research quality should maybe look less at the praise made by peers in confidential letters of recommendation, and more at how the project was able to present itself and to assert itself in open, technical discussions.
In addition, the open "colloquium" environment might organize special events where different systems compete with each other. An obvious kind of competition event is the championship, where participants are asked to use their systems for solving some concrete tasks. Another possibility is to have duels, where the group that developed system A challenges the group of system B with a concrete task or set of tasks, proposing that a closed competition be set up with only systems A and B, and for solving the problem(s) proposed by the A system team.
It will be necessary to add a few recommendations to the rules for these competitions. For example, in the interest of fairness one may rule that for every task proposed by the challenging team in a duel situation, the challenged team shall be able to responed with another task of their choice.
The use of competitions for evaluating research results differs in important ways from the classical criteria, where research is given credit for the number and the popularity of its original results, that is, results where the author or the authoring team are able to claim priority. In the competitive scheme, priority is not directly relevant. If team A has developed a powerful new concept and incorporated it into their system, and team B picked up the idea and were able to introduce it into their system in time for the next competition, then the historical priority of team A does not matter. On the other hand, team A had a head start, and if they are able to use it well they can proceed more rapidly to the next innovation. This arrangement may seem very unsatisfactory if you take priority of results as the ultimate touchstone of scientific success. On the other hand, if you feel that priority is an ill-defined concept since new ideas tend to come up concurrently in several laboratories, then you may argue that competitive evaluation makes more sense than publication counts and priority rules.
Competitive evaluation also has its problems. In its extreme form, it might lead to excessive secrecy, since a research project will not only strive to keep its results secret until they are properly published; they may have an interest in not telling at all about some of their developments, in the hope of being able to use them during forthcoming competitions during an extended period of time. This risk may be reduced by a combined system, where competition and conventional publication are used together. Then, a research project that hides some of their results may win in competitions, but they may also lose in publications since someone else will publish the method sooner or later. Yet another idea may be to impose explainability requirements on the competing teams: the merit value of a victory is qualified by the extent to which the winning team is able to explain why and how they won. These issues will no doubt have to be worked out in terms of the special characteristics of the discipline and the type of system that is concerned.
The ETAI scheme of publication has developed over less than a year. Similar approaches have reportedly been tried in some other disciplines, although not yet in computer science, to the best of our knowledge. This is actually a bit surprising, since the operation of ETAI depends quite heavily on software support, which means that it ought to be particularly easy to realize it in a computer science environment. This applies in particular for the third, colloquium layer, but also for the first two layers. Anyway, now that a concrete example exists, we hope and believe that similar systems will be set up for many other specialized areas as well.