Any ideas how to make ranking less random?

[krokkrok]

Note that this is not a particular flaw of Elo ratings

Correct, however the matching system used by this contest (and many others) can magnify the effect. Cheating straight round-robin can work, but it can be done easier with this matching system because the way it pigeon-holes bots to play against bots of similar ranking.

Head-to-head competition emphasizes anticipation, adaptation, and trickery, in addition to pure optimization.

Therein lies the catch: if a bot is encouraged to use trickery, what are the bounds of trickery it is allowed to use? The truth is that cheating is called winning when it isn't detected.

I would much prefer the contest organizers take strides to detect and discourage collusion and ranking manipulation

This could not be done perfectly, but a huge step forward would be to validate accounts by requiring credit card information. As it stands now there is no way to trust the identities of the players; the claims by the competition runners are hot air, there is no way to detect a smart cheater.

[Mistmanov]

This could not be done perfectly, but a huge step forward would be to validate accounts by requiring credit card information. As it stands now there is no way to trust the identities of the players; the claims by the competition runners are hot air, there is no way to detect a smart cheater.

I can think of a few people who would be pretty hesitant to hand over credit card info to a bunch of university students who're organizing an AI contest (e.g.: me). Hell, many people don't even have credit card info (e.g.: me)

Someone who cheated in order to get position 200 while he only deserved position 220 will probably go undetected, yes.. but does anyone really care about that?

The important bit is the top10, and the really important bit is the winner. And if there's something weird going on with the winner, people will notice. If the #1 lost masses of games against some players A, but always won against some other players B, people would go "hey.. what's up with that?". If they then noticed that the tactics that B use against #1 are wildly inferior to the tactics that B use against other players, people would get suspicious.

[Astek]

Maybe this will be interesting information so here it is:

Last day of conquest I made few experiments on test server. I was looking for best way to refine parameters. Final effect of one of the tests was very disappointing for me. I wanted to make sure how random are Elo rankings on test server. So I did run 4 copies of SAME bot in parallel. After few hours (2-3) this 4 bots ranking was very strange: 3, 17, 33 and 70!!! Conclusion was clear: few hours of testing was not enough and results was almost random (most important was effect of first few fights), so I stopped using dhartmei server in this case.

Hoverer at the finals each of top 1000 bots made more than 500 games, so its rather hard to compare my results from test server with results of finals. There is few ways to find the answer on the question "How random rankings may be?". One of them is to simulate finals again using same maps and compare final ranks. Maybe additionally make few copies of some bots and check how far they will be ranked from each other.

I am not saying that finals was random, I am convinced that there will be same winner again. I am treating this as interesting academic problem of Elo score effectiveness in semi random games like this one.

Lets point out which rules are making this game random:
- generating random maps for finals (different set of maps will generate different rankings)
- random picking opponents (some bots plays better against others)
- randomizing map selection (bots are optimized for specific maps, some plays great when starting distance is high, some other has great strategies for close combat maps etc)

Regardless of how random the results are I think that Elo score was very useful and fulfilled its role.

[amstan]

Maybe this will be interesting information so here it is:

Last day of conquest I made few experiments on test server. I was looking for best way to refine parameters. Final effect of one of the tests was very disappointing for me. I wanted to make sure how random are Elo rankings on test server. So I did run 4 copies of SAME bot in parallel. After few hours (2-3) this 4 bots ranking was very strange: 3, 17, 33 and 70!!! Conclusion was clear: few hours of testing was not enough and results was almost random (most important was effect of first few fights), so I stopped using dhartmei server in this case.

Hoverer at the finals each of top 1000 bots made more than 500 games, so its rather hard to compare my results from test server with results of finals. There is few ways to find the answer on the question "How random rankings may be?". One of them is to simulate finals again using same maps and compare final ranks. Maybe additionally make few copies of some bots and check how far they will be ranked from each other.

I am not saying that finals was random, I am convinced that there will be same winner again. I am treating this as interesting academic problem of Elo score effectiveness in semi random games like this one.

Lets point out which rules are making this game random:
- generating random maps for finals (different set of maps will generate different rankings)
- random picking opponents (some bots plays better against others)
- randomizing map selection (bots are optimized for specific maps, some plays great when starting distance is high, some other has great strategies for close combat maps etc)

Regardless of how random the results are I think that Elo score was very useful and fulfilled its role.

There are many reasons that could have happened:

• is your bot deterministic?
• how many other bots were playing at the same time?