Asura is an environment for game-based assessment of programming challenges. Its main features include the graphical feedback, the tournament-based assessment, and the framework and tools that it offers regarding the simplification of the process of building Asura challenges. This environment leverages the Software Agent (SA) evaluation on Mooshak 2’s program analysis, extending it to support multiple submissions running in the same evaluation environment. This allows students to code their SAs in any programming language supported in Mooshak. On the client side, the GUI is embedded into Enki, allowing any course created in Enki to provide Asura challenges out-of-the-box.
The architecture of Asura, present in the figure below, contains four components: Builder, Viewer, Tournament Manager, and Evaluator. The Builder, Viewer, and Tournament Manager are completely new components. The Evaluator is an extension of Mooshak 2 program analysis to support game assessment. In this architecture, Enki makes the bridge between the server and the client, requesting evaluation to the Evaluator and loading the feedback into the Viewer.
Evaluator
The Evaluator component of Asura inherits the static analysis of the Mooshak’s evaluator engine. The only difference is the compile command line can include language-specific player wrappers, present in the game.jar file. However, the dynamic analysis is completely new. Instead of test cases with input and output text files, Asura Evaluator receives as input a list of paths of the selected opponents’ submissions that generate the input and consume the output of each other.
Since these submissions are already compiled (when necessary), the component just initializes a process for each of them. After that, it organizes matches containing the current submission and a distinct set of the selected opponents’ submissions. The length of this set depends on the minimum and maximum number of players per match, which are specified by the game manager. At this point, the evaluation proceeds on an instance of the specific game manager, instantiated from the game.jar. The game manager receives the list of player processes indexed by the player ID and starts the game.
The game manager controls the execution of the game, and keeps the SA’s aware about the state of the game, querying the SAs for their state update at the right time, ensuring that the game rules are not broken, managing the state of the game, and classifying and grading submissions. If an SA fails to comply with the rules, the match ends and the SA receives a “Wrong Answer”.
During the game, any updates to the state object propagate to the movie builder. The state object provides the following methods that can be used by the manager to control its lifecycle: prepare(movieBuilder, players) which initializes the state and sets up the metadata of the movie, execute(movieBuilder, playerId, playerUpdate) that updates the game state with the action of a certain player, endRound(movieBuilder) which ends a round of actions, and finalize(movieBuilder) which finalizes the game, adding the submission results in the movie, among other things.
Finally, the status containing the observations, mark, classification and feedback compiles into a single status, and goes to the client.
Builder
The Builder component is a Java framework for building Asura challenges, providing a game movie builder, a general game manager, several utilities to exchange complex state objects between the manager and the SAs, and general wrappers for players in many different programming languages. The framework is accompanied by a Command-Line Interface (CLI) tool to easily generate Asura challenges and install specific features, such as support for a particular programming language, a default turn-based game manager, among others.
To simplify the task of creating graphics, Asura introduces the concept of game movie. A game movie consists of a set of frames, each of them containing a set of sprites together with information about their location and applied transformations, and metadata information, such as title, background, width, height, fps, and players (i.e., player names indexed by their ID). Furthermore, the concept of game movie is formally defined in a JSON schema. To allow the manager to easily build the JSON data, the game movie builder provides several methods, such as: setters for each metadata field, addFrame() which adds a frame to the movie, addItem(sprite, x, y, rotate, scale) which adds a sprite to the current frame in position (x,y) with the given rotation and scale, addMessage(playerId, message) which adds a message to the player, setters for observations and classification, and saveFrame()/restoreFrame() which allow to add a frame state to a stack to restore it later.
The exchange of state updates between the SAs and the manager uses JSON messages. Depending on the programming language, this can be a hurdle and make it very complex for SA’s to process it. For that, there are wrappers for players providing several methods to process JSON. Moreover, each game can define its own wrappers for SAs providing methods specific to the game.
The documentation for the very first release of Asura Builder is available online and has already been followed by some peers who volunteered to test the system.
Tournament Manager
The Tournament Manager is the component responsible for managing and running tournaments. A tournament can have any number of stages, each with its own type (pools or knockout) and format (e.g., round-robin, swiss type, knockout, double knockout, among others). Stages contain with a set of players, those who qualified in the previous stage, which will compose the matches of every round of the current stage. The assignment of players to matches is a task of the class implementing the specific tournament format. These tournaments can be organized in the administrator GUI of Mooshak 2, through a wizard developed specifically for this task. This wizard allows to select players individually, add/remove stages, and set some properties of the tournament, such as the number of players per match, the number of qualified players in each stage, and the points awarded to a win/draw/loss.
Viewer
Asura Viewer is a GWT widget with two modes: match and tournament. Figure below presents both modes side-by-side. In the tournament mode, it expects a JSON file following the Tournament JSON Schema. This data contains a reference to each match’s movie, organized by stages and rounds, as well as partial and complete rankings of each phase. The tournament mode widget consists of an interactive GUI, allowing students to navigate through the stages of a tournament, visualize specific matches or the whole course of a player, and check the rankings of each stage.
The match mode displays the game movie JSON produced during the evaluation phase. The GUI mimics that of a media player, containing a slider, a play/stop button, buttons to navigate through the current play-list, a full-screen button, a box to display messages, and a canvas where the movie appears. The movie is completely resizeable keeping the original aspect ratio.
Overview
The video below presents Asura in action, in a ES6 course.