How did you test your bot ?

[Zaph]

I'm interested in hearing how people tested their bots, especially if anyone had unit tests / sanity checks which would guarantee they hadn't gone backwards while "fixing" things.

I used two harnesses to test my code, ignoring simple adhoc tests:
1. Hand-built 'run the gauntlet' simulator which would run a series of bots (e.g. the last three versions) against a known set of other bots (all combinations) across a decent number of maps:
e.g. if I wanted to compare A, B and C I would do it against D,E,F,G
it would play out like this:
map 0 A,D,E,F
map 0 A,D,E,G
...
map 0 B,D,E,F
...
etc
I could then see how each bot did under the same set of opponents (more or less).
I could also disable the running of the older bots once I had set the benchmark, but would often change the maps or the opponents and run the gauntlet again. This took many hours to run, especially in the latter stages once I went from 50ms per turn to 300ms per turn (on purpose)

2. VM of the contest site, with many versions of my code running
This was awesome, someone posted a VM image of the contest which could be run with the free VMware Player. I would create lots of accounts in it, upload all the different versions of my code (plus some starters and the tutorial) - then just leave the VM playing in the background.
I made great progress while this worked... and then I broke it, and never managed to recover it. I would have had to upload all the bots again from scratch and just couldn't be bothered, but it was great while it lasted.


What I *didn't* do, but wanted to:
a) Create a set of sanity-check maps+situations to test against.
for example:
- Given a specific map with a static enemy, make sure I didnt take too long to gather food and explore
- Given a map with no food, but some static enemies and two starter ants for me, make sure I could win the game without losing an ant (good for 2-on-1 combat testing)
- Some kind of smoketest to make sure I never let a lone ant wander in to my base
- Basic sanity checks for illegal moves, suicides (two of my ants colliding), etc

b) create some combat and hill-taking situations and make sure my bots all won the combat and took the hills
c) create a super-slow N-ply combat bot to test against, where it had no time limit but I still obeyed the 500ms limit
d) create some timeout unit tests, where I get a pre-populated map with hundreds of ants, lots of food and enemies in a way that would stress out my bot and possibly cause a timeout.

Did anyone do any of those things, either in part or in full ?
How about any even smarter testing utilities or methods ?

[DrClaes]

"Not nearly enough" is my short answer... I became too engrossed in details--first in exploration, later in combat--to spend any significant time testing locally. I would run the bot some dozens of times against the starter bots, just to make sure it didn't crash, then upload to the main server and/or run it on the tcp servers. I ended up not having a lot of time for the challenge for about a month, which I could probably have spent doing some optimization of my exploration parameters, but that didn't happen...

Probably some test cases would have been valuable as sanity checks while developing the code. As many have pointed out, it's non-trivial to determine which version of your bot is going to be superior, and as the incremental strength shrinks, the number of games required to document the difference increases... probably one of the biggest obstacles to those wanting to use genetic algorithms or similar to optimize parameters was game length... when each game on tcpants takes 3-5 minutes, you're going to have a long time between generations...

As my bot ended up, I submitted with a serious bug which has caused a lot of timeouts in the finals, and which I was able to identify and fix in <10 minutes (of course after the deadline).... very annoying, and a valuable lesson for the next challenge, assuming I have the time then too... the challenge has been a lot of fun and very much a learning experience for me, though!

[Zaph]

[Hippo]

I didn't test my bot by any unit tests / regression tests ...
I made huge logs and made a local turnaments testing all maps (with different turn time limits) and randomly chosen older versions of my bot (And one version of standa's). Studying logs and watching the games to find wrong habbits ... So codding roughly from 11:00 pm till 2:00 am, running turnaments at night. (Fortunately I had 2 Saturdays for deeper log study). Sometimes I run tcp version of my bot, but the time issues could not be tested well with it.