Paper rating website

Post Reply
Sarah Bridle
Posts: 144
Joined: September 24 2004
Affiliation: University College London (UCL)
Contact:

Paper rating website

Post by Sarah Bridle » July 07 2005

Thanks to Steven Fuerst for pointing this out to me: http://bororo.physics.mcgill.ca
Seems relatively well thought through.
A bit of a hurdle to get started, as you have to have a certain number of your own papers rated by others before you can become a "certified user", and see scores of other papers.
I thought I'd try out grading another paper, so tried to think of a good recent paper I'd read (Eisenstein et al wiggles, in case you were wondering), and gave it a high score. But the first 5 papers you score must have medium to low grades, to guard against grade inflation. I can't really be bothered to think up a load of mediocre papers I read recently so I've stopped playing with it for now.
But it is an interesting idea and if it took off it could be useful.
Do other people think we (in cosmology) should make the effort to make it take off (in this area)?
What improvements could be made to make it more likely to take off? / more useful?
(e.g. I sent in a comment suggesting it would be useful to have a feature that allows you to browse papers, rather than having to think up a paper and search for it.)

Simon DeDeo
Posts: 44
Joined: October 26 2004
Affiliation: Santa Fe Institute
Contact:

Paper rating website

Post by Simon DeDeo » July 07 2005

It seems a little mean-spirited to rank or "rate" papers, and not quite germane to the process of advancing research -- i.e., it would be great to know what others thought were the terrific papers in the field, especially when one was starting out, but I don't see why it is useful to know what the "bad" (or "unoriginal") papers are.

Especially since there are some papers that may not make an original contribution but that are exceedingly useful nonetheless -- just because they simplify or organize what others have done.

It seems to me, in other words, that the best way to run the site would be to get rid of the "grade inflation" requirements. It doesn't seem to help to require users provide negative feedback as a condition of participating.

I wonder also what this rating process does that Citebase doesn't -- I mean, the best way to get "votes" on how good a paper is is to see how many people cite it.

Syksy Rasanen
Posts: 119
Joined: March 02 2005
Affiliation: University of Helsinki

Re: Paper rating website

Post by Syksy Rasanen » July 08 2005

It's not clear to me what useful purpose the rankings would serve. It's indeed nice to have good (or maybe just stimulating) articles pointed out, but the rankings are not necessary for that. The Introduction says that "This service has the goal of encouraging the writing of outstanding papers in astrophysics.", but I don't see how it contributes to this end.

It's also bizarre that on the scale between 0 and 20, scores 4-20 are reserved for different grades of the "top 4.4%". How can one possibly evaluate whether the paper is in the top 0.6%, 0.28% or 0.16%! If a grading system is wanted, a simple 1 to 5 would be better.
Simon DeDeo wrote:I wonder also what this rating process does that Citebase doesn't -- I mean, the best way to get "votes" on how good a paper is is to see how many people cite it.
The number of citations depends also on other factors, such as how fashionable the subject is, the size of the community working on it and the prestige of the authors. There are papers that I have found good and interesting which are poorly cited because not many people are working on the topic. Conversely, it is not the case that I would consistently find highly cited papers exceptionally good.

For example, the proposal by Kolb et al has gotten 23 citations in a short period of time (regardless of the fact that everybody seems to think it's wrong!), while some good and careful papers on backreaction haven't gotten the same number of citations over several years.

Boud Roukema
Posts: 87
Joined: February 24 2005
Affiliation: Institute of Astronomy, Nicolaus Copernicus University
Contact:

Paper rating website

Post by Boud Roukema » August 26 2005

IMHO, a numerical rating system is likely to provide exponentially growing biases in favour of popular papers - the more people think that paper X is good, the faster the judgments of its quality grow. Don't we already have enough positive feedback processes for good, popular papers? i.e. citations? Do we really need to make the positive feedback loop even stronger?

IMHO, the structured discussion provided here on cosmocoffee makes much more sense. Good papers can be complimented. Less popular but interesting and original, correct papers can be commented. Constructive comments can be made about papers which hint what would need to be corrected, what assumption is wrong in them, etc.

If we wanted to be even more structured, we should wikify:
http://en.wikipedia.org/wiki/wiki

and to some extent that has started already, e.g.
http://en.wikipedia.org/wiki/physical_cosmology

A web of corrections to papers makes more sense to me than having popularity contests.

My guess is that the site is more about trying to find/create superstars rather than trying to encourage understanding of the Universe.

If we want to create popular pressure for the best practices in cosmology, then we could create a site where users rate the degree to which the software in a paper is available as free software (free as in speech). (Since data is mostly very public in cosmology, we probably don't need extra social pressure here.)

Of course, simply saying that people are bad for not providing free software for their paper is not terribly constructive. Again we get back to a more constructive feedback process where people could say, hey, you can use such and such a package which is GPL even if the author himself/herself was not aware of this. That way the knowledge itself becomes free (as in speech).
[/b]

Sarah Bridle
Posts: 144
Joined: September 24 2004
Affiliation: University College London (UCL)
Contact:

Paper rating website

Post by Sarah Bridle » August 26 2005

Hi Boud, I agree with most of what you say. At one time we did think a bit about the relative pros and cons of wikis versus discussion forums.

As I see it they lend themselves to slightly different types of topic.

e.g. for discussing contentious issues I think it is valuable to have each comment attributed to a given author, and since debate then often follows, it is useful to have comments in chronological order.
Although it would be possible to do this in a wiki, it would not be so reliable and could get confusing. e.g. at first cosmocoffee was set up so that authors could at any time edit their posts. But there were then instances where authors modified their post, but then the replies to the post didn't make sense.

On the other hand, sometimes you just want the bottom line, final conclusion. And then I think a wiki can be ideal. e.g. a table of the latest proposed cosmic shear experiments.

Boud Roukema
Posts: 87
Joined: February 24 2005
Affiliation: Institute of Astronomy, Nicolaus Copernicus University
Contact:

Re: Paper rating website

Post by Boud Roukema » August 29 2005

Sarah Bridle wrote:Hi Boud, I agree with most of what you say. At one time we did think a bit about the relative pros and cons of wikis versus discussion forums.

As I see it they lend themselves to slightly different types of topic.

e.g. for discussing contentious issues I think it is valuable to have each comment attributed to a given author, and since debate then often follows, it is useful to have comments in chronological order.
Although it would be possible to do this in a wiki, it would not be so reliable and could get confusing. e.g. at first cosmocoffee was set up so that authors could at any time edit their posts. But there were then instances where authors modified their post, but then the replies to the post didn't make sense.

On the other hand, sometimes you just want the bottom line, final conclusion. And then I think a wiki can be ideal. e.g. a table of the latest proposed cosmic shear experiments.
Hi Sarah,

IMHO this is where the mediawiki solution comes in
  • each theme has both an "article" page and a "discuss" page
  • on the "article" page, people do not sign their names
  • on the discuss page, it generally is a good idea to sign each comment
  • on the discuss page, there's a very easy syntax to make single, double, triple indents in order to show the discussion thread
The advantage of chronological discussion on controversial issues remains in this structure: it's separated from the structure "results" page. Even though it is technically possible to modify previous comments and therefore lead to a confusing discussion, in practice, if people modify their comments in a way which leads to confusion, they tend to discredit their own arguments, or else other people complain, or even revert the text. Thinking more positively, it's the complementarity between the article page and the discuss page, even though both have essentially the same technical wiki structure which discourages confusing edits on the discuss page.

In any case, this forum seems to be functioning fine as it is at the moment - nothing stops us thinking of a wiki at some time in the future when we're ready for that. IMHO it will sooner or later become a question not of if but of how and when. ;)

Sarah Bridle
Posts: 144
Joined: September 24 2004
Affiliation: University College London (UCL)
Contact:

Paper rating website

Post by Sarah Bridle » August 30 2005

Hi, thanks for reminding me about the discussion part. This is a good point. I find having the two different pages slightly confusing, but I agree that it should in principle be a nice feature.

Looking for an example of this in action I looked at the wikipedia cosmology page. I see that it works quite well. I find the layout of the discussion page a bit hard to scan, but I guess this could be improved very easily (hey, I could even go and edit it myself).

Will look into it. I have installed MediaWiki OK.

Boud Roukema
Posts: 87
Joined: February 24 2005
Affiliation: Institute of Astronomy, Nicolaus Copernicus University
Contact:

throwing ideas around...

Post by Boud Roukema » August 31 2005

Sarah Bridle wrote:Hi, thanks for reminding me about the discussion part. This is a good point. I find having the two different pages slightly confusing, but I agree that it should in principle be a nice feature.
Hmmm... let's play around with the idea of separating out the typically 7 standard sections of an astro journal article - each has a different role and probably should have a different level of wikification. This would be rather than just having a complement to standard journal articles, to have publication of research articles themselves, partly using wiki type techniques... Warning: radical ;)
  • abstract -Maybe
  • introduction - should non-authors be allowed to add more background stuff? better explain the idea? -Maybe
  • method, data - probably not wikified, except possibly for the authors to correct - we want to know what the authors did, how they did it, or at least how they claim they did it - No
  • results - should other people be able to make minor changes, or improvements to the method, or corrections to the data, and then put in the corrected results? -Maybe
  • discussion - definitely wikifiable -Yes
  • conclusion - maybe an authors' conclusion + a wikified conclusion that others can modify? -Maybe
  • bibliography -Probably
We could then add on a peer-reviewing mechanism. This would require:
  • some way of authorising people to be reviewers - could be more objective than the present methods
  • a button to indicate whether anyone has peer-reviewed the article yet
  • buttons to allow anyone (who is authorised) to propose to review it
  • more difficult questions:
    • do we make the reviews public?
    • are the reviewers' identities made public? or do we stick to the standard system, given that there do exist some reasonable arguments for anonymous peer-reviewing
      • the advantage of (at least optionally) non-anonymous reviewing is that those people who do a lot of reviewing can get credit for it, or at least, the info is there for people who want to know, while in the standard system, only the journal editors know who does this work
  • meta-pages (like Category: pages on mediawiki sites), with lists of articles that have not yet been peer-reviews, that way anyone wanting to contribute to the community by reviewing has the opportunity to do so
At the same time, one argument against considering wiki-type peer-reviewed journal articles is for cosmologists in poor countries, with slow internet connections: when I was in India 1998-200, cosmologists in New Delhi University found astro-ph, gr-qc etc to be one of their main sources of information, since they more-or-less couldn't pay for most of the printed journals, but they accessed this mostly through email. With a bad internet connection, email is still fairly functional. Wikis, where you need to be online a lot (or where it's much more natural to do things online, unless you prefer a file locally and later on copy/paste), disadvantage those with poor internet connections. Maybe by now NDU is in better shape, but i'm sure there are new places falling into the category of poor internet connection + good physicists.
Looking for an example of this in action I looked at the wikipedia cosmology page. I see that it works quite well. I find the layout of the discussion page a bit hard to scan, but I guess this could be improved very easily (hey, I could even go and edit it myself).
The whole point is that it's a scratch pad for deciding what to put on the main page - it's not meant to be the place where people say "Sandage claims 42 and Tully claims 85" - the point is whether or not people agree that that is what Sandage and Tully claim - and if people agree that this is what the different groups find, then the statement "Sandage claims 42 and Tully claims 85" can uncontroversially get put on the article page. (Well, i'm deliberately putting an out-of-date debate here ;)

So it's not quite the same as the discussion section of a journal article.
Will look into it. I have installed MediaWiki OK.
You're fast. :)

Sarah Bridle
Posts: 144
Joined: September 24 2004
Affiliation: University College London (UCL)
Contact:

Paper rating website

Post by Sarah Bridle » September 02 2005

Blimey! I agree, v radical re writing a paper using a publically accessible wiki!
I would absolutely love it if research were done like this.
But I don't see how it can work within the current funding system.
We all need to get funding and the amount of funding we get is based on our output of refereed papers.
How would funding bodies ever rate our output in the wiki system?
Don't get me wrong, I would love it to be as you suggest but we have to win over the funding bodies at the same time, as otherwise how can people afford to spend their time on this?
The whole point is that it's a scratch pad for deciding what to put on the main page - it's not meant to be the place where people say "Sandage claims 42 and Tully claims 85" - the point is whether or not people agree that that is what Sandage and Tully claim - and if people agree that this is what the different groups find, then the statement "Sandage claims 42 and Tully claims 85" can uncontroversially get put on the article page. (Well, i'm deliberately putting an out-of-date debate here ;)
OK, thanks. I guess I was contrasting the wikipedia cosmology discussion page with a discussion forum, and saying that I find the phpbb (e.g. cosmocoffee) layout easier to read.

Boud Roukema
Posts: 87
Joined: February 24 2005
Affiliation: Institute of Astronomy, Nicolaus Copernicus University
Contact:

Re: Paper rating website

Post by Boud Roukema » September 05 2005

Sarah Bridle wrote:Blimey! I agree, v radical re writing a paper using a publically accessible wiki!
I would absolutely love it if research were done like this.
:))))
But I don't see how it can work within the current funding system.
We all need to get funding and the amount of funding we get is based on our output of refereed papers.
How would funding bodies ever rate our output in the wiki system?
Don't get me wrong, I would love it to be as you suggest but we have to win over the funding bodies at the same time, as otherwise how can people afford to spend their time on this?
In most countries, once people have tenured faculty positions, there is no direct link between number of refereed papers and salary, there's only peer pressure, and (at least in France) there is probably a big majority opinion that publishing 5 papers per year is not necessarily that much more useful for good research than publishing one paper per year. The USA, UK, Canada and Australia - the countries where people take the longest to get tenured positions - are not the only countries where cosmology is done. In France, Spain, India, Japan, ... people still get tenured positions relatively young, around 30-35ish, though the pressure is pushing this to older ages.

In any case, if cosmologists (and other astronomers) organise, we don't have to accept the failed-economic-ideology model which is being pushed on universities and research institutes. Exactly how we should do this, or whether or not it is a priority, and whether or not we as a community are ready to do this, is another question. However, IMHO it is definitely something that can be done.

The French research community - going across the whole spectrum from sociology and philosophy to cosmology and particle physics - has been organising for the past year and a half to tell the government that decisions on research spending, including reverting the destruction of faculty positions, but also the creation of new faculty positions, should be made rationally and reasonably and based on the decisions of researchers together with participation from the general public. Concrete demands have been made, all sorts of different campaigning techniques have been used, and the government has been forced to give back a few billion euros (AFAIK). Since i'm not in France at the moment, i haven't followed this in detail, so don't quote me ;) on the specific details above. The main web site is:

http://recherche-en-danger.apinc.org/

All sorts of communities around the world are starting to better self-organise http://en.wikipedia.org/wiki/Self-organisation and force pseudo-democratic governments (USA, UK, Australia, ...) to act more like representatives and less like authoritarians, in part thanks to the internet, though also IMHO due to the inherent inconsistencies of the communist and capitalist economic systems with human rights.

Anyway, independently of my generalisations, the fact is that the French research community is acting very effectively to self-manage and impose this on our pseudo-democratic government.

But getting back to the idea of a partly-wiki journal, the practical question is how many cosmologists would be willing to try this to start off. More or less by definition, it's only going to be people with tenured positions, who tend to be older. And older people tend to be less confident in new things like wikis... Hmmm.

BTW, so far i'm only throwing-ideas-around(TM), (no, it's not really a TM AFAIK ;), but there's no harm chatting about this to see where it goes...

An argument in favour of the wiki journal - over a few-year time scale, is that just as astro-ph articles get cited twice as often as non-astro-ph articles astro-ph/0503519, we could expect that wiki papers will get an extra boost in citation rates, especially as wikis generally get very highly rated by google, because of their innate nature of extensive cross-referencing. So the committees that judge the impact factors of journals would sooner or later have to accept that articles in the journal have a high impact in citation rates. The wikipedia is now taken very seriously by many people, but it didn't exist 5 years ago.

And there's no need for anyone to commit to only publishing on our new journal. It would be enough to get a few articles there and as their high quality becomes obvious (due to the continuous correcting by people annoyed at seeing errors and imprecise statements), people would start to cite them and there would be a postiive feedback process.

If there are more people who think that in principle this is a good idea and are willing to think through or invite discussion on the design questions (e.g. what level of wikification for different sections, peer-reviewing algorithm), and if we have enough people willing to cooperate on this and each willing to, e.g, commit to publish a few of their own articles on it, but we also need a few people to maintain/develop the software, maintain the server(s), etc., then IMHO it's feasible. Techies from the mediawiki and general wiki community would probably be willing to help, but we would need a few cosmologist-techies (e.g. you and i?) willing to commit the time as well.

For the moment, i have plenty of other interesting things to do, so i wouldn't put this at a high priority right now. But if there's a big interest and willingness to try, i'd be interested in supporting it. :)

On the other hand, wikipedia and wikibooks are tending towards the directions of review articles, hmmm.... maybe we could start with just a review journal?

There's already http://relativity.livingreviews.org/ but this is an original-author only wiki, and the same web site hosts http://europeangovernance.livingreviews.org/ which is funded by Connex which is part of Veolia Environnement which is the new name of Vivendi Universal, which had a reputation of allegedly being one of the most corrupt French water companies and also including Universal Studios (yes, the film group in the USA). And last time i looked, Connex/Veolia Environnement had 13 billion euros debt... While in cosmology we have to make lots of compromises to get funding, livingreviews.org seems to have made more compromises with authoritarian organisations than are really necessary...

Anyway, IMHO there would be no serious competition with livingreviews.org - a wiki review journal would quickly attain much higher quality than a journal limited to a small select group of authors.

Sarah Bridle
Posts: 144
Joined: September 24 2004
Affiliation: University College London (UCL)
Contact:

Paper rating website

Post by Sarah Bridle » September 07 2005

I think you are right that wiki reviews would be the most natural thing to start with. It would certainly be useful to read. I wonder how many people would want to spend their time editing it. Only one way to find out I guess! It would be easy to build this up v gradually too, which makes it more feasible.

I disagree with your implication that once people are tenured then their publication record is not very important. I am only just beginning to find out about these things, but it does seem that the amount of money available for students, postdocs and colleagues is critically dependent on measurable factors such as refereed publications (e.g. see the RCUK report Sam already pointed out in http://www.cosmocoffee.info/viewtopic.php?t=280). And fun though it is, chatting via cosmocoffee and email, life would be pretty dull without direct interaction too!

Boud Roukema
Posts: 87
Joined: February 24 2005
Affiliation: Institute of Astronomy, Nicolaus Copernicus University
Contact:

Re: Paper rating website

Post by Boud Roukema » September 07 2005

Sarah Bridle wrote:I think you are right that wiki reviews would be the most natural thing to start with. It would certainly be useful to read. I wonder how many people would want to spend their time editing it. Only one way to find out I guess! It would be easy to build this up v gradually too, which makes it more feasible.
Agreed :).
I disagree with your implication that once people are tenured then their publication record is not very important. I am only just beginning to find out about these things, but it does seem that the amount of money available for students, postdocs and colleagues is critically dependent on measurable factors such as refereed publications
This is a community choice: the research community can either accept this (because it is insufficiently self-organised), or it can reject it in favour of something more meaningful.

UK is not the only cosmology place in the world. It is a fact that the French research community has self-organised and (so far) had a fair degree of impact (measured in Giga-euros, if you like) against irrational government policies. There are good cosmo communities in India, Japan, etc etc.
(e.g. see the RCUK report Sam already pointed out in http://www.cosmocoffee.info/viewtopic.php?t=280).
i don't quite see where it talks about postdoc, travel etc funding being directly related to numbers of papers.

(BTW: in CAMK http://www.camk.edu.pl in Poland, there is, apparently, a direct relation between salary and numbers of refereed publications, conference proceedings and even... seminar presentations. My sarcastic response is that they forgot to include credit for corridor/coffee conversations.)
And fun though it is, chatting via cosmocoffee and email, life would be pretty dull without direct interaction too!
Sure. :)

Boud Roukema
Posts: 87
Joined: February 24 2005
Affiliation: Institute of Astronomy, Nicolaus Copernicus University
Contact:

open/wiki astro/cosmo journal threads

Post by Boud Roukema » September 07 2005

i just want to link these three threads together, hopefully the software won't complain about self-links :P
My own problem with http://www.arxivjournal.org/ is that it says nothing about a wiki style opening up of sections of the articles, it says nothing about encouraging free (as in speech) software, and it ignores the empirical experience of the wikipedia http://en.wikipedia in which completely open editing (+ many meta/feedback processes, themselves also wikified) generates higher quality articles than those on, e.g. Britannica - simply because more people correct errors.

Probably someone (e.g. me?) should copy/paste the main arguments from these different threads on... a wiki and then try to sort them into something coherent... Any cosmologist wishing to use the http://cosmo.torun.pl/cosmo twiki for this purpose (e.g. as a scratch pad wiki for cosmologists) is welcome to do so.. Hmmm.... i've already finished my coffee for today, but watch this space...

Post Reply