Project INCESSANT: INteraCtivE Simulation optimiSAtioN laboraTory

Open Human-Computer Interaction Computer Science Games Engineering Bachelor Master Project Thesis Games Research Lab

Background

A central challenge in games engineering lies in building models of various phenomena that we see around us, e.g. the way buildings look like or the way persons and animals behave and interact with us. If we setup basic models that resemble the real phenomena in some ways, there are typically aspects that have not been considered in detail which makes us realise that the virtual model is quite different after all, sometimes disturbingly so. Such details can, e.g., be the gait of a virtual non-player character, its blinking patterns, or the way a frog hops across the street. In order to find the proper configuration of these and other models, we may run optimisation algorithms that might either be fed with real-world examples of frogs crossing a street and simply minimise the difference between the simulated and the real behaviour. One might also assume that the frog’s movement follows a certain pattern that has yet to be found out and its degrees of freedom in the virtual world are accordingly reduced to ensure “natural” behaviours.

Traditionally, one would state the goals/ideas associated with the frog’s behaviour first (or “define some requirements”), run simulations with random behaviours next and let its parameters change automatically until the goals are fulfilled, or the “requirements are met”. For instance, one may ask the frog to cross the street and to do so with minimal effort, etc. Based on these expectations, a frog model could be optimised to only move upwards as in the game classic Frogger and to only rely on short leaps, for as long as it is not endangered by traffic.

Human-in-the-loop systems implement the idea that users could interfere at any time during the optimisation process and change its course. In particular, users could proactively introduce certain changes in the model or its configuration, e.g. consider power buffs that had not been there before, correct or extend the optimisation criteria on-the-fly, e.g. to also achieve a high score, and alternate or trigger specific optimisation steps, for instance introduce far leaping frogs to see how they compete with other solutions.

In the end, one experiment does not only yield one optimal solution for one well-specified problem, but a series of interactions leads to a series of new models and series of optimisation histories. Instead of a black box with a 1:1-mapping of problem to solution, the human-in-the-loop simulation results in a tree of models and optimisation paths, all of which should remain available for analysis and comparison.

Project Goals

In project INCESSANT, the students are (1) asked to implement a simple data visualisation such as autoglyphs or stick figure icons, which can be interactively configured to specify target model configurations or change individual solutions. (2) An optimisation algorithm such as gradient descent or a heuristic such as evolutionary algorithm (EA) is deployed to complete an interactive optimisation cycle: The user can specify a target configuration, the optimiser finds one or several according solutions. In addition, in order to realise a full human-in-the-loop system, the user can interfere at any point with the optimisation target, the current solutions in the system, and the optimisation steps. (3) An infrastructure is designed, implemented and validated that supports human-in-the-loop simulation and optimisation. Aspects that need to be researched include (a) visualisations of the interaction/simulation/optimisation history, (b) user interfaces for organisation and maintenance of simulation/optimisation experiments (e.g. virtual shelves), (c) user interfaces for driving optimisation processes (e.g. introducing mutations, performing recombinations, etc. in EA), (d) back-end infrastructure for persistence and distributed computation, or (e) embedding contextually relevant geometrical, geographical, or otherwise relational, visual information.

Contact Persons at the University Würzburg