[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 483: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 112: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 112: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 112: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 112: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 112: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/bbcode.php on line 112: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead
[phpBB Debug] PHP Warning: in file /includes/functions.php on line 4586: Cannot modify header information - headers already sent by (output started at /includes/functions.php:3765)
[phpBB Debug] PHP Warning: in file /includes/functions.php on line 4588: Cannot modify header information - headers already sent by (output started at /includes/functions.php:3765)
[phpBB Debug] PHP Warning: in file /includes/functions.php on line 4589: Cannot modify header information - headers already sent by (output started at /includes/functions.php:3765)
[phpBB Debug] PHP Warning: in file /includes/functions.php on line 4590: Cannot modify header information - headers already sent by (output started at /includes/functions.php:3765)
AI Challenge Forums • View topic - Goals for Backend Rewrite

It is currently Sun Sep 24, 2017 8:41 am Advanced search

Goals for Backend Rewrite

Place where the Contest Admins can talk about the running of the contest.

Goals for Backend Rewrite

Postby Zannick » Sun Jan 16, 2011 12:42 am

I figure I should explain what I'm aiming for in the backend rewrite that I've already started.

Simplicity - The majority of the python code should be readable and it should be easy to recognize what the code intends to do, with minimal cross-referencing. For example, look at the compile_anything.py script from the end of Planet Wars. It had three dictionaries, and the languages one often referenced the other two, in addition to being collections of arbitrary-looking tuples. I believe the new backend/submissions/compile.py to be an improvement upon this.

Testability - The code should be easily testable. There should be tests for the code checked into the repo (perhaps in their own directory), easily modifiable to add more tests. This is especially important because we don't want to find errors by having users run into them.

Furthermore, the code and its tests should be runnable for anyone with the proper Python version without any changes. Last contest I commented out specific imports for modules I didn't have while testing, and uncommented them for committing again. These were MySQLdb and server_info, and code that used these dealt with connecting to the server's main database. I aim to cut that functionality out of the main scripts, putting it in the glue scripts that will bind everything together on the server.
Zannick
Contest Organizer
 
Posts: 25
Joined: Wed Nov 17, 2010 9:18 pm

Re: Goals for Backend Rewrite

Postby jeff.cameron » Sat Jan 29, 2011 6:31 pm

I heartily agree with all the above. Lately I have become a huge fan of Python's unittest module which allows us to create unit tests really easily.

Another aspect of testability is making the contest software easy to deploy. I think that it would be a lot easier to work on the contest if it was trivial to deploy a contest instance on people's own machines. We had a few bottlenecks last time where we wanted to make a change, but could not realistically test it anywhere other than on the live production server.

amstan came up with the idea of having a contest .deb file for installing on an Ubuntu machine. I tentatively think this is a great idea. For testing new changes I could kick off a new contest instance inside a VM in seconds just by installing the .deb.
jeff.cameron
Contest Organizer
 
Posts: 91
Joined: Sun Jan 31, 2010 4:06 am

Proposed Zeta architecture

Postby jbroman » Sun Jan 30, 2011 9:20 pm

As the rewrite project "Codename Zeta" spools up, I thought I'd share my thoughts about how I think the architecture should work. (I'm going to use the present tense so I don't have to say "would", "should", and "could" as frequently.)

Zeta consists of three major components: the website, (tournament) manager and worker.

Website

The website is a Django project which presents the interface users deal with. Most importantly, it accepts submissions and displays rankings and previous games. The rankings and games are displayed using content from a MySQL database (either local to the web server or, preferably, on Amazon RDS so that we get isolation and trivial DB backups).

When a user submits their bot, the website computes its SHA-1 checksum and moves it to Amazon S3 under a key containing the checksum (e.g. aichallenge-submissions/3e2db1dde4b865685e2d9f1f99f66e85eb81ae9e). It may remove old submissions, mark them as inactive, or leave them be.

Manager

The manager is responsible for the operation of the tournament itself. It chooses who plays whom and computes rankings based on the results. The manager may run on the same host as the website, but especially if the website experiences high load, it may be run on a separate host instead.

The manager is responsible for populating the game queue, which is maintained on Amazon SQS (Simple Queue Service). When the game queue approaches exhaustion (i.e. fewer than some predetermined number of scheduled games remain on the queue), it uses some matchmaking algorithm to schedule additional games by placing messages on the game queue. Each message contains the SHA-1 sum of each submission to play, and any additional game parameters necessary (e.g. map to be used).

The manager also reads messages from the game result queue. This contains messages which contain the same information as the game message, plus information about the result (winner, game history for visualization). The manager is responsible for storing this information in the database and updating the rankings (according to another forum post, Trueskill can do this iteratively, which sounds nice).

Worker

The worker is responsible for hosting the games. A worker first gets a message from the game queue. For each submission, it checks to see if that submission exists in its cache already. If not, it fetches the archive from S3, then decompresses and (if necessary) compiles it. The worker then runs each submission under a restricted environment (separate UID, limited CPU time, limited memory, limited/no forking, no network access) and judges the result. It puts a message in the game result queue and repeats this process. Multiple workers would likely run on a worker host.

Advantages

It is useful to understand how SQS works to see its benefits. SQS allows storing short messages in a queue. When a message is read from the queue, it is not removed but only hidden for a predetermined amount of time. When the message has been processed successfully by the recipient, then (and only then) does the recipient remove the message permanently from the queue. If the recipient fails to either remove the message or extend its lease within the allotted time, it is assumed that the recipient has failed. In this case, the message once again becomes visible and is picked up by the next access to the queue.

By using this architecture, we get many advantages, especially in separation of duties and fault tolerance. Consider the following failure scenarios (assuming MySQL is on RDS; adjust as appropriate if not):

1. The website host experiences a denial of service or other critical failure.

Users can no longer submit bots or view results until service is restored. However, no data is lost. Contest organizers can terminate the website instance and replace it with a new instance of the same or larger instance size. If necessary, organizers can start multiple instances behind a load balancer (e.g. Elastic Load Balancing) – since the database is on another host, all website instances will operate correctly and refer to the same authoritative database. The tournament continues to operate despite the failure. The manager continues to schedule games and the workers continue to run them. Updates results will be available once website service is restored – failure is isolated to the website only.

2. The manager experiences a critical failure.

No new matches are scheduled and rankings are not updated with new match results. However, matches which have already been put into the game queue will run even while the manager is down (e.g. while an organizer terminates the manager instance and spawns another). Results from matches which finish while the manager is down accumulate in the queue, and when manager service is restored, the result backlog is processed (no special logic is needed to do this, it's an result of the architecture) and rankings are updated.

If the exhaustion threshold is correctly chosen, the manager will keep the game queue sufficiently populated as to give organizers time to restart it with no interruption to game operation and no loss of results.

3. One or more workers experience a critical failure.

The games that the worker was running are made visible in the queue again and will be run by another worker after a slight delay. The worker can be terminated and possibly replaced by a new instance, and any games it was responsible for will be handled by another worker node.


Thoughts?
Jeremy Roman
Student, Computer Science
University of Waterloo
jbroman
Cadet
 
Posts: 8
Joined: Mon Feb 01, 2010 5:39 pm

Re: Goals for Backend Rewrite

Postby Janzert » Mon Jan 31, 2011 3:29 am

For clarity of anyone who doesn't know, the current system also builds a queue of games to play but uses the database and an http api to hand out the games to workers. While I like the idea of using SQS from the perspective of having less contention around the main database. I'm not very familiar with SQS and there are a couple of things to consider that I'm unsure if they are possible under SQS.

The pairing program needs to know not just the number of games queued for play but also what all of those pairings are, including the in progress ones. This is so it knows what players haven't played recently but are actually already scheduled to play.

The second problem is that between the time a game is scheduled and a worker pulls it off the queue to play, one or both of the submissions may have been replaced by a new submission. In the current system this is a simple check on the main server that both submissions are the latest for each player and dropping the pairing if they aren't. This is a little less important since the only problem caused by not having it is the wasted resources on a game that won't count for anything.

One other thing I didn't see noted above is it would be good for initial compilation and functional testing of submissions to be handled by workers as well. Possibly just by using another task type in the queue the games are pulled from.

Also not explicitly mentioned, but it would be really good to get the game replay data stored outside of the database. It should be possible to get it either served directly as static files by the frontend webserver or possibly through something like CloudFront. This should cut down on the database load tremendously.

Janzert
Janzert
Contest Organizer
 
Posts: 271
Joined: Sun Feb 07, 2010 1:59 am

Re: Goals for Backend Rewrite

Postby jbroman » Mon Jan 31, 2011 4:12 am

Jeremy Roman
Student, Computer Science
University of Waterloo
jbroman
Cadet
 
Posts: 8
Joined: Mon Feb 01, 2010 5:39 pm

Re: Goals for Backend Rewrite

Postby Janzert » Mon Jan 31, 2011 7:35 am

Janzert
Contest Organizer
 
Posts: 271
Joined: Sun Feb 07, 2010 1:59 am

Re: Goals for Backend Rewrite

Postby jbroman » Mon Jan 31, 2011 2:22 pm

Jeremy Roman
Student, Computer Science
University of Waterloo
jbroman
Cadet
 
Posts: 8
Joined: Mon Feb 01, 2010 5:39 pm

Re: Goals for Backend Rewrite

Postby jeff.cameron » Mon Jan 31, 2011 10:28 pm

jeff.cameron
Contest Organizer
 
Posts: 91
Joined: Sun Jan 31, 2010 4:06 am

Re: Goals for Backend Rewrite

Postby jbroman » Tue Feb 01, 2011 12:29 am

Jeremy Roman
Student, Computer Science
University of Waterloo
jbroman
Cadet
 
Posts: 8
Joined: Mon Feb 01, 2010 5:39 pm

Re: Goals for Backend Rewrite

Postby dmj » Tue Feb 01, 2011 1:38 am

dmj
Lieutenant-Colonel
 
Posts: 41
Joined: Thu Feb 11, 2010 11:36 am

Next

Return to Behind the Scenes

Who is online

Users browsing this forum: No registered users and 1 guest

cron