[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 112: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 112: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 112: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/functions.php on line 4586: Cannot modify header information - headers already sent by (output started at /includes/functions.php:3765)
[phpBB Debug] PHP Warning: in file /includes/functions.php on line 4588: Cannot modify header information - headers already sent by (output started at /includes/functions.php:3765)
[phpBB Debug] PHP Warning: in file /includes/functions.php on line 4589: Cannot modify header information - headers already sent by (output started at /includes/functions.php:3765)
[phpBB Debug] PHP Warning: in file /includes/functions.php on line 4590: Cannot modify header information - headers already sent by (output started at /includes/functions.php:3765)
AI Challenge Forums • View topic - Request: A better scoring system

It is currently Wed Jan 24, 2018 11:33 am Advanced search

Request: A better scoring system

Code won't compile? Found a bug? Post here!

Request: A better scoring system

Postby jmcarthur » Fri Feb 12, 2010 6:26 pm

jmcarthur
Colonel
 
Posts: 80
Joined: Fri Feb 05, 2010 3:41 pm

Re: Request: A better scoring system

Postby Janzert » Fri Feb 12, 2010 7:25 pm

For those interested in rating systems you might like to check out .
Janzert
Contest Organizer
 
Posts: 271
Joined: Sun Feb 07, 2010 1:59 am

Re: Request: A better scoring system

Postby jmcarthur » Fri Feb 12, 2010 7:35 pm

jmcarthur
Colonel
 
Posts: 80
Joined: Fri Feb 05, 2010 3:41 pm

Re: Request: A better scoring system

Postby jmcarthur » Sat Feb 13, 2010 9:40 pm

Can any admins comment on this, please?
jmcarthur
Colonel
 
Posts: 80
Joined: Fri Feb 05, 2010 3:41 pm

Re: Request: A better scoring system

Postby jeff.cameron » Sat Feb 13, 2010 10:55 pm

Good post. We have been trying a number of things with the scheduling algorithm and the ranking algorithm in the last few days. But have no fear, we have learned a great deal about which things work and which things don't, and we will have more precise rankings before not too long.

We have actually tried a number of different ranking algorithms, including one that we invented. If anyone wants to take a crack at implementing other ranking algorithms, we will definitely try them out. It's relatively easy for us to plug in any old ranking algorithm. The general problem is to take a list of (possibly inconsistent) pairwise rankings and then come up with the best possible total ordering of the submissions. Each pairwise ranking takes the form "submission A beat submission B", where A and B are integers. The problem gets tricky when you consider that many of the pairwise rankings are inconsistent. That is, it's possible to have pairwise rankings like A>B, B>C, and C>A at the same time.
jeff.cameron
Contest Organizer
 
Posts: 91
Joined: Sun Jan 31, 2010 4:06 am

Re: Request: A better scoring system

Postby a1k0n » Sun Feb 14, 2010 8:44 am

Yes, I think an Elo-ish model is the way to go, but it need not be as complicated -- there is no skill variation whatsoever for these bots until they are replaced with a new version, at which point their ranking goes away anyway.

I would go about this with an Elo-like model where each players' skill is normally distributed, and to get the probability that player A beats player B you plug the difference in skill into a logistic function. Working backwards from there, it should be possible to either find a closed-form posterior distribution, or a sampled approximation, for every bot's skill, given the entire game history (as long as the history only includes the current versions of every bot). Then you'd take the mean of each to do the ranking, but it'd be useful to know the standard deviation (or volatility) of the ranking as well. This is of course computationally expensive and requires a background in machine learning to implement, but this *is* an AI contest.
a1k0n
Colonel
 
Posts: 90
Joined: Fri Feb 12, 2010 3:51 am

Re: Request: A better scoring system

Postby Janzert » Wed Feb 17, 2010 11:32 pm

Probably the easiest thing to do while giving a large improvement over the current system would be to use . This would just require writing a script that could process the current game results into a pgn format that bayeselo can understand.
Janzert
Contest Organizer
 
Posts: 271
Joined: Sun Feb 07, 2010 1:59 am

Re: Request: A better scoring system

Postby aipcaish » Mon Feb 22, 2010 9:29 pm

aipcaish
Lieutenant
 
Posts: 10
Joined: Wed Feb 03, 2010 6:43 pm

Re: Request: A better scoring system

Postby montanalow » Mon Feb 22, 2010 9:44 pm

I've posted a suggestion over in viewtopic.php?f=8&t=171&start=20#p1456
Perhaps we should merge these threads, or lock one?
montanalow
Lieutenant-Colonel
 
Posts: 42
Joined: Thu Feb 18, 2010 6:50 pm

Re: Request: A better scoring system

Postby jeff.cameron » Fri Feb 26, 2010 7:30 am

We have now switched to an Elo-based ranking system. Let us know what you think!
jeff.cameron
Contest Organizer
 
Posts: 91
Joined: Sun Jan 31, 2010 4:06 am

Next

Return to Technical Issues

Who is online

Users browsing this forum: No registered users and 1 guest

cron