[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 112: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/functions.php on line 4586: Cannot modify header information - headers already sent by (output started at /includes/functions.php:3765)
[phpBB Debug] PHP Warning: in file /includes/functions.php on line 4588: Cannot modify header information - headers already sent by (output started at /includes/functions.php:3765)
[phpBB Debug] PHP Warning: in file /includes/functions.php on line 4589: Cannot modify header information - headers already sent by (output started at /includes/functions.php:3765)
[phpBB Debug] PHP Warning: in file /includes/functions.php on line 4590: Cannot modify header information - headers already sent by (output started at /includes/functions.php:3765)
AI Challenge Forums • View topic - Final Tournament Rules

It is currently Sat Oct 20, 2018 4:09 pm Advanced search

Final Tournament Rules

Random stuff about the contest, posts that don't fit in the other forums.

Re: Final Tournament Rules

Postby zugzug » Tue Feb 16, 2010 9:57 pm

Results within 30 minutes are not at all important to me either. I've spent enough time on this, that I'd much rather have my work judged slowly and accurately, than quickly and inaccurately. The high volatility of current ranking/matching algorithm, especially within the top 10, shows that it is precise, but it is not accurate.

Perhaps you could post qualifying rankings in 30 minutes, using the current matching and scoring methodology.
The top 100 qualifiers would become Semi-Finalists, and play against every other Semi-Finalist. 5,050 games total, 99 games each.
Then the top 10 would advance to a Finalist round. 55 games, 9 games each.
Any ties in the rankings should be resolved with sudden death competitions between them (first to win a map, wins the rank).

Not only does this give much more accurate results where it matters, it shouldn't take much longer, and it has the added advantage that you get to anticipate results 3 times, instead of just once. There is also some psychological consolation to be able to claim "finalist" instead of #10, especially when you can watch you bot play and lose the most against all the other Finalists.
zugzug
Cadet
 
Posts: 2
Joined: Tue Feb 09, 2010 8:31 pm

Re: Final Tournament Rules

Postby callinwire » Tue Feb 16, 2010 11:07 pm

Looking at the current rankings, I've noticed that, when even a single loss or tie can really change your rank, the maps used have way too much influence on rankings.

In short, your rank is too dependent on which maps you play your opponents on. Some are more likely to produce ties than others.

For the final tournament, I hope that these round-robins will be played on the same maps. Of if that's too much, then once the top 50% of players are found, then have them play every other person on the same couple of maps. So PersonA will play PersonB 5 times, once on each map, and this will be the same for everyone.
callinwire
Lieutenant
 
Posts: 16
Joined: Thu Feb 11, 2010 8:54 pm

Re: Final Tournament Rules

Postby stefan » Tue Feb 16, 2010 11:28 pm

My suggestions would be (most of them have already been made):

1. Time does not matter, accuracy does! So please, take all the time it needs to complete the final rankings.
2. Use better maps. Obviously they need to be bigger, and they should test the AI's for all kinds of properties. So make sure you inlcude some random maps, some mazes, some big open fields, etc.
3. Don't let the ranking be influenced on map choice. So please make sure that any two bots play against each other on ALL maps. Or if this is not possible (though I cannot see why it wouldn't; remember, time does not matter), at least do so for the top N bots.

If you really feel like it is important to deliver the results fast, please at least ensure that the top N bots have a fair tournament (with good maps, and lots of matches).
stefan
Lieutenant
 
Posts: 13
Joined: Mon Feb 08, 2010 10:38 pm

Re: Final Tournament Rules

Postby encloser » Wed Feb 17, 2010 1:49 am

I agree with bruudruuster's proposal of:

- selecting a fixed set of random (or at least good ones but not used before) maps to be played between all players. But 10 may be a bit too much, perhap 5 or so, with each player playing both sides on each map. Having every player play the same set of maps is the fairest.

- like everyone mentions, time is not important. I believe the "typically range from 15x15 to 30x30" maps is a conern for time, so if time is not the contestants' concern, then have the final maps be larger maps of 30x30 to 50x50. Let the machines run for days or even 2-3 weeks if need to. Maybe have the current scoreboard be updated realtime for the unofficial scores. To save time, if submissions have not been altered from the initial packages, don't run them (I heard they are out the current scoreboard already).

- what is the official scoring function? Like others mentioned, ties should not to punished as much since it's usually a position move to avoid a loss.
encloser
Cadet
 
Posts: 1
Joined: Wed Feb 17, 2010 1:32 am

Re: Final Tournament Rules

Postby stumac » Wed Feb 17, 2010 7:18 am

I agree with the previous posters - it is of no consequence whether the final tournament is completed in 30mins - the real test is to find the best bot.

A poor final tournament design (one that introduces any randomness) will ruin all the good work that's been put into organising this competition so far. The true test is that if the tournament was conducted again, you should get exactly the same results (with a small variance due to bots that facilitate the random behaviour).

Due to the random nature of the maps for the current competition, you can see that there are some players with no losses sitting around 70th. This is the type of outcome that should be avoided in the final tournament.
stumac
Cadet
 
Posts: 1
Joined: Wed Feb 17, 2010 6:16 am

Re: Final Tournament Rules

Postby GarySWest » Wed Feb 17, 2010 9:39 pm

GarySWest
Major
 
Posts: 34
Joined: Mon Feb 15, 2010 3:45 pm

Re: Final Tournament Rules

Postby Accoun_ » Thu Feb 18, 2010 7:52 am

by special map selecting you may make any bot to win.
please publish final map set
for justice (fairness)
Accoun_
Lieutenant-Colonel
 
Posts: 45
Joined: Mon Feb 15, 2010 12:29 am

Re: Final Tournament Rules

Postby Accoun_ » Thu Feb 18, 2010 1:38 pm

please publish percent of :
1. large maps
2. closed maps
3. large and closed maps
need for bot adjusting (customization)
-------------
Some members work in universities and may conduct extensive testing of their bots with one another.
single well can not.
make a map in a prior competition close to those that are in the final.
Accoun_
Lieutenant-Colonel
 
Posts: 45
Joined: Mon Feb 15, 2010 12:29 am

Re: Final Tournament Rules

Postby Fritzlein » Fri Feb 19, 2010 12:53 am

Like other posters, I would suggest deprecating the importance of playing a round robin among all entrants.

The Swiss format, as suggested by many, has the virtue of quickly sorting the players so that more attention can be paid to the top bots. If there are 1000 bots entered in the final tournament, a round-robin would do just as good a job at discriminating between 999th and 1000th place as it does discriminating between 1st and 2nd place. I personally don't care nearly as much about the accuracy of rankings at the bottom of the list as I care about the accuracy at the top.

If we do care about accurately distinguishing the top two players, it is far more important to have them play each other twenty times than to have each of them play ten opponents in the bottom half of the draw. In ten games against weak opponents, the top two bots might go 10-0 and 9.5-0.5, or probably even 10-0 and 10-0, providing half a point or no points of discrimination. Mismatches are just wasted games that prove nothing. Playing against each other, however, the top two bots might go 14-6, providing eight points of discrimination and a very clear indicator of who is better.

Therefore I recommend successively shrinking the field, as suggested by other posters. In the early rounds, I disagree with the idea of playing multiple maps between each pair of players. It is much more fair to play twenty different opponents, each on a random map, than to play two different opponents ten times each. Drawing an unusually strong opponent is much worse luck than getting a "draw map" that doesn't permit discrimination between stronger and weaker players, and also worse luck than getting a type of map that an equal opponent just happens to play better than you. We should mix up the players in the early round for a fast, accurate sort, and save the extra maps for the finals.

If we are contemplating a 1000-player round-robin of only one game per player, we would need 499,500 games. Consider what wonderful things could be done in a Swiss for just a fraction of that! For the same hypothetical 1000 players, 50 rounds of Swiss play would take only 25,000 games. After 50 rounds there would be about 500 contestants with a record of 25-25 or better and 500 contestants with a worse record. I would argue that there is a statistically insignificant chance that be best overall bot is not in that top half after a 50-round Swiss, so we could eliminate the bottom half without worrying that we didn't give some players a fair shot.

Among the 500 remaining players, we could start permitting opponents to play for a second and third time, carrying forward records and carefully having them play on a new map each time. Another 100 rounds would be only 25,000 more games, after which we could quite comfortably reduce the field to 200 players without fear of eliminating the best.

We could use our next 19,900 games on a full round-robin on those 200 players, which for some pairs will be their fourth meeting, always on a different map. At the end of that round-robin each remaining player will have played 349 games, so they certainly would have had ample opportunity to shine. Cutting the field to top 50 at that point couldn't hurt anyone's feelings.

A round-robin among 50 players needs only 1225 games, so for our next 7,350 games we could play a sextuple round-robin, bringing most of the pairings up to ten different maps played against each other. After that each survivor will have played 643 total games, and anyone who is not in top twenty can retire feeling fairly beaten.

When the final twenty play a twenty-fold round robin, it will be a mere 3,800 games, bringing the grand total of games needed for the tournament to a mere 81,500, less than a fifth needed by the full round robin. Each of the top players will have played 1023 games, i.e. just about as many as they would have in a full round robin, but those games will have been much more meaningful, and done a much better job picking the top player. We get the best of both worlds, i.e. fewer games and a truer (less random) champion.

This format would require 30 different maps, but with the 15x15 size restriction eased, it should be possible to have that many different reasonable designs without difficulty.

To summarize my proposal:
1) 50-round Swiss, no repeat pairings
2) Eliminate anyone with a losing record, carry records forward
3) 100-round Swiss, allow second and third match for each pair, no repeat maps within each pair
4) Reduce field to top 200, carry records forward
5) Single round-robin, may be up to fourth match for each pair, no repeat maps within each pair
6) Reduce field to top 50, carry records forward
7) Sextuple round-robin, may be up to tenth match for each pair, no repeat maps within each pair
8) Reduce field to top 20, carry records forward
9) Twenty-fold round-robin, may be up to thirtieth match for each pair, no repeat maps within each pair
10) Best record wins!

I am going to bet a nickel that anyone with a bot that is anywhere in contention for the title will prefer this format to a full round-robin, because successively shrinking the number of players reduces randomness and gives the best bot a far better chance of winning.
Fritzlein
Colonel
 
Posts: 81
Joined: Thu Feb 18, 2010 9:20 pm

Re: Final Tournament Rules

Postby GarySWest » Fri Feb 19, 2010 2:28 am

I would vote for it
GarySWest
Major
 
Posts: 34
Joined: Mon Feb 15, 2010 3:45 pm

PreviousNext

Return to Misc

Who is online

Users browsing this forum: No registered users and 2 guests

cron