I'm interested in hearing how people tested their bots, especially if anyone had unit tests / sanity checks which would guarantee they hadn't gone backwards while "fixing" things.
I used two harnesses to test my code, ignoring simple adhoc tests:
1. Hand-built 'run the gauntlet' simulator which would run a series of bots (e.g. the last three versions) against a known set of other bots (all combinations) across a decent number of maps:
e.g. if I wanted to compare A, B and C I would do it against D,E,F,G
it would play out like this:
map 0 A,D,E,F
map 0 A,D,E,G
...
map 0 B,D,E,F
...
etc
I could then see how each bot did under the same set of opponents (more or less).
I could also disable the running of the older bots once I had set the benchmark, but would often change the maps or the opponents and run the gauntlet again. This took many hours to run, especially in the latter stages once I went from 50ms per turn to 300ms per turn (on purpose)
2. VM of the contest site, with many versions of my code running
This was awesome, someone posted a VM image of the contest which could be run with the free VMware Player. I would create lots of accounts in it, upload all the different versions of my code (plus some starters and the tutorial) - then just leave the VM playing in the background.
I made great progress while this worked... and then I broke it, and never managed to recover it. I would have had to upload all the bots again from scratch and just couldn't be bothered, but it was great while it lasted.
What I *didn't* do, but wanted to:
a) Create a set of sanity-check maps+situations to test against.
for example:
- Given a specific map with a static enemy, make sure I didnt take too long to gather food and explore
- Given a map with no food, but some static enemies and two starter ants for me, make sure I could win the game without losing an ant (good for 2-on-1 combat testing)
- Some kind of smoketest to make sure I never let a lone ant wander in to my base
- Basic sanity checks for illegal moves, suicides (two of my ants colliding), etc
b) create some combat and hill-taking situations and make sure my bots all won the combat and took the hills
c) create a super-slow N-ply combat bot to test against, where it had no time limit but I still obeyed the 500ms limit
d) create some timeout unit tests, where I get a pre-populated map with hundreds of ants, lots of food and enemies in a way that would stress out my bot and possibly cause a timeout.
Did anyone do any of those things, either in part or in full ?
How about any even smarter testing utilities or methods ?