Re: [tlug] Open Access Journals

[Raymond Wan <rwan.kyoto@example.com>]

On Mon, Mar 24, 2014 at 11:09 AM, Darren Cook <darren@example.com> wrote:
>>  > In some areas in CS, they have annual "competitions" to see which
>>  > method is best.
>>
>> URL?  (I have no idea what you're referring to here, I'd like to see!)
>
> In computational linguistics, there are often tasks/challenges. e.g.
> here is just the first one I found scrolling back up the corpora list
> archives:
>   http://challenges.2014.eswc-conferences.org/index.php/RecSys
>
> (I'm guessing that is the kind of thing Raymond was referring to.)


Yes, indeed it is, though I don't know of that one specifically.
Others I know of include:

1)  CAMDA for bioinformatics data
[http://camda2014.bioinf.jku.at/doku.php/contest_dataset]

2)  KDD Cup for data mining, but it seems to have ended in 2010
[http://www.sigkdd.org/kddcup/index.php]

3)  TREC for information retrieval [http://trec.nist.gov/]

I've only participated in TREC (a while ago), but you run your systems
on a data set where no one knows the answer.  All systems' results are
combined and these are evaluated by humans.  From this "gold
standard", the systems are evaluated and ranked.  Then, at the
conference, you find out how you performed.  So, you may fly far away
to Washington DC, to find out you're at the bottom...

You do write a paper and include it in the proceedings, but it isn't
peer reviewed in the traditional sense.  The competition is the "peer
review"; anyone who participated can write a paper...the point is,
your paper can say why you did well or even what you did wrong.

The (IMHO) nice part is that if you're in the research community,
having participated in this conference counts for something.  No, not
as great as a journal -- true -- but still something.

There is an equivalent to TREC for East Asian languages which has been
running for 11 years (???).  It is hosted by NII in Tokyo:
http://research.nii.ac.jp/ntcir/ntcir-11/index.html .  Some of you
might find it of interest...

See:  http://research.nii.ac.jp/ntcir/data/data-en.html  for the types
of data available...

Ray