Unreal based tabletop framework for hybrid reality games

Closed Human-Computer Interaction Bachelor Thesis No Affiliation

Motivation

Board games are an integral part of everyday life. The market has been rapidly growing over the last few years (Piotr, 2016). Examples for boardgames are Dungeons and Dragons (Wizards of the Coast, 2017) as well as Risiko (Hasbro, 2015). Those games need at least two players and are played by moving figures on a board according to specific rules. Because of this, boardgames are played in groups and provide a social component to the game. However, the rules need to be checked by a designated player. To achieve this, boardgames are normally turn-based and are thus slow paced. In a typical tabletop roleplaying game one of the players plays as “gamemaster” (Dormans, 2006). It is his responsibility to explain and lead the story and to check if the gamerules are executed correctly.

Another important area of the entertainment industry are computer games. The market in this sections booms and is steadily growing (The Entertainment Software Association, 2016). In computer games, the rules are checked and executed by an AI, which makes cheating harder and gives the user more time for other aspects of the game. Since the rules are executed automatically, computer games are seldom turn-based, but rather executed in real time. Contrary to board games, computer games can be played on multimedial and multimodal systems, which appeal to multiple senses. This allows the system to render complex graphics using the third dimension and change the graphics during runtime, which a boardgame is not capable of. Computer games also extend the user experience with sounds and music. However, computer games need to be played on a console or PC and thus do not provide a social component by themselves. The social component of computer games can often only be achieved with online communities (Lee & Lee, 2010) or multiplayer. This aspect can still be improved since those methods work remotely and do not favor real life interaction.

An approach to combine board games and computer games is by using mixed reality. Milgram and Kishino (1994, p. 2) define mixed reality as a “subclass of VR related technologies that involve the merging of real and virtual worlds”. They define six different subclasses, in which the systems in class six are defined as “completely graphic but partially immersive environments (e.g. large screen displays) in which real physical objects in the user’s environment play a role in (or interfere with) the computer generated scene, such as in reaching in and “grabbing” something with one’s own hand” (Milgram & Kishino, 1994, p. 3). These systems are categorized as “Hybrid Reality (HR)”.

By combining classical boardgames with computer games into a HR game this project aims to integrate a social component to the game while still maintaining automatic rule execution in real time. This also allows rendering and changeable graphics as well as enhancing the interaction with the use of real world figures. This will be accomplished by developing a game development framework utilizing reacTIVision input to connect real world objects placed on a digital surface with an in-game representation. This allows the players to interact socially by being physically located around the play area while the rules are executed and visuals are rendered via a game engine.

Related Work

Many boardgames were ported to phones or tablets (Sheldon, 2016). However, those games are still turn-based most of the time and do not utilize the social component since multiplayer is still done over the internet. Also, the hardware of phones and tablets is limited in order to maintain mobility. By developing a framework for a digital table mobility is restrained, but in return the social interaction is still accomplished in the real word. When combined with a digital surface that allows rendering on a separate machine, the technical capabilities can be extended.

An example for using the hybrid reality approach are chess computers. Those are devices where one side of the chessboard is played by the player whereas the other side is played by an AI. Typically the computer recognizes chess piece position and gives the player instructions on how to place the pieces. The first stand-alone chess machine was developed in 1977 by Fidelity (Schachcomputer.info, 2017). Those chess machines however mostly only offer a small display and speech output for optimal turns and can only be used to play chess. Developing a broad and general framework will allow the development of other genres and tabletop games. Different rulesets and optimal turns can be implemented using such a framework as well.

STARS is a hybrid tabletop game developed by Magerkurth, Memisoglu, Engelke and Streitz in 2004. It combines a custom multitouch table called InteracTable and PDAs to create a hybrid reality role-playing table-top game. Game pieces were recognized using RF-ID whereas the interaction was implemented with touch interfaces on the table and the PDAs. The PDAs were also used to display character specific information. A big problem of the implementation was that it only allows one single touch event simultaneously. A reacTIVision enabled application will not only be able to use multitouch, but also send recognized markers over the same protocol instead of RF-ID. This also allows the use of tangible interfaces instead of PDAs for character stat information.

Previous versions of a multimodal boardgame were developed by Giebler-Schubert, Zimmerer, Wedler, Fischbach & Latoschik in 2013 with the use of a scala based framework called Simulator X (Latoschik & Tramberend, 2011). They called the system XRoads (Cross Reality On A Digital Surface). Over time, the multimodal boardgame based on Quest: Zeit der Helden was expanded with a gesture- and speech recognition system (Fischbach, Zimmerer, Giebler-Schubert & Latoschik, 2014) as well as with a support for gaining rewards when the player visits real life locations (Fischbach, Lugrin, Latoschik & Fendt, 2014; Zimmerer, Fischbach & Latoschik, 2014). The system features a gamemode where four players need to defend a city against a gamemaster. Their goal is to kill the gamemaster and defend their town against his minions. Another present gamemode is a story-driven one. The system was developed to be played on a Samsung SUR40 table based on reacTIVision (Samsung, 2012). This table has build-in hardware, but since the table’s original release was in 2012, the hardware is already outdated. By using a digital surface with expandable hardware, this problem can be avoided easily. By developing a new framework, the code base can be changed from scala to a game engine like Unreal or Unity to improve rendering and game performance.

Requirement Engineering

Elicitation

Personas

There are three personas the system is developed for: Ellen, David & Florian, which fulfill roles in the games development.

Florian Smith is 28 years old and a consumer of the games developed by Mixed Realities. He is good friends with David and often tests their implementations since he plays tabletop games in his leisure and favors role playing games.

David Fernandez is 29 years old and founded the startup by himself two years ago. He is an experienced C++ developer and the lead programmer of his team. He needs base classes to build upon and to quickly get started with development. His task is to provide functionality which can be called by the application developers via scripting.

Ellen Jones is a 27 years old level and interface designer at Mixed Realities, a small startup for hybrid reality boardgames. Her task is to link certain trigger conditions with certain events. Since she has no programming experience, her preferred way to accomplish this task is via scripting.

User Stories

Florian is preparing a tabletop game for his friends. He places the board on the table and sets the figures / tangibles into their starting position. Those pieces will be moved when the game starts and will be able to interact with each other. He places cards besides the board. These cards trigger different effects when played. In the package Florian also finds a handbook describing the rules and winning conditions of the game.

David needs basic TUIO input to connect the digital surface with the engine. To link input to a certain feature, he needs a method to assign markers to specific classes and touch events to specific locations on the interface. He also needs some debug functions to prove the connection is working correctly. Since recognized markers of a boardgame can be classified as tangibles or cards, they have different properties or effects assigned to them. In order to provide variety, he needs to define multiple subclasses derived from a baseclass. David would like to provide multiple game modes to switch between different rulesets and winning conditions easily. As the main developer, David also cares about providing an error-free and stable solution with low latency and high framerates.

Ellen is tasked with implementing the level and the content itself. She assigns a visual reprasentation to the figure class and adds components defining the behaviour and properties of the figure before placing them on the board. She also defines interactions between and effects triggered by different tangibles to allow e.g. attack systems, trade and playing cards with effects. After selecting one of the preexisting base game modes, she adds specific conditions, like e.g. winning by score. Ellen accomplishes this by linking e.g. trigger boxes with preexisting methods via visual scripting.

Ellen, David and Florian play a roleplaying tabletop game in the evening and discuss plans to improve the current system with optional content in the future. They play different heroes with different races and classes, which influence their player stats and skills. They track their player stats by writing it on a character sheet and also specify which armor and weapons they use on this sheet. One of them is a gamemaster and gives the other players quests. The group needs to fulfill different objectives to gain a reward.

Analysis

David needs a way to convert placed tangibles into a digital representation. A common approach is using reacTIVision (Kaltenbrunner & Bencina, 2007), which uses infrared cameras to track touch events and recognize marker input. An example would be Tangible Optical Chess by Joyner, Wu, and Yi-Luen Do in 2009. They compared a game of hybrid reality optical chess to one based on point and click. Their results indicate that interaction was perceived as simple and intuitive as with a classical boardgame. They also showed that displaying more information mixed with real pieces helps players to plan out their strategies.

Ellen needs high computational powers and rendering cabalities to provide high end level design and interfaces while David strives to achieve low latency and high performance for all his games. Multitaction (MultiTaction, 2017) creates reacTIVision based cells that can be connected together. This cell can be expanded to a table by placing it in a scaffold. Those cells allow the use of a PC for rendering and computing power. Therefore this table is able to provide better graphics and more rules than the system the original XRoads was based on.

Ellen needs a framework providing an easily accessible and usable API as well as base classes to implement the application itself. A scala based framework called Simulator X (Latoschik & Tramberend, 2011) is used as basis for XRoads. Although Simulator X is improving every year, the focus of the system is based on actors in a real-time interactive systems. With this focus on AI and non-functional requirements, the rendering capabilities can still be improved. The main drawback is that Simulator X was developed primarily for research applications and is therefore seldom used for serious game development. Since this is a limitation to the goals of Ellen, she would like to prefer a native solution.

Other game engines like e.g. Unity (Unity Technologies, 2017) or Unreal (Epic Games, 2017a) provide far better rendering options. The Unreal engine is state of the art and is used in a lot of recent AAA games (Wikimedia Foundation, Inc., 2017). With it’s difference between original C++ code (aimed at developers, which pleases David) and a visual scripting called Blueprints (aimed at level designers like Ellen), functionality can be predefined in a framework and used in Blueprints to easily design functionality. The official side provides detailed tutorials for blueprints, which help developers to get started. Developers can also add plugins to the engine to extend functionality. In contrary to Simulator X, the rendering capabilities of the Unreal Engine provide more and better options. It is easy to define and add particles, fog effects, post process rendering and more (Epic Games, 2017b).

To make systems like the table by Multitaction more appealing for game developers and customers, a framework for the Unreal engine is useful. Currently there is no official reacTIVision support for the Unreal Engine. TUIO provides a C++ dll, but this dll redefines macros the engine itself uses and including it leads to corrupting the Unreal project. There is however a plugin utilizing OSC in- and output. This plugin can be used to develop a plugin providing reacTIVision input to all users of the engine. A framework can be put on top of the plugin to support hybrid reality game development.

Specification

TUIO Plugin

• F-TUIO-1: Receive OSC events and translate them into (2D) Touch & Marker classes. Communicate this via events to the TUIO manager.
• F-TUIO-2: Provide a TUIO manager class where listeners can be registered and deregistered.
• F-TUIO-3: Provide touch and marker structs to easily access information.
• F-TUIO-4: Provide touch and marker listener interfaces.
• F-TUIO-5: Provide a function library for e.g. converting TUIO coordinates to world coordinates.

Tabletop Framework

• F-FW-1: The framework translates received markers into tangibles. Those tangibles represent e.g. figures, obstacles, cards and stat overviews.
• F-FW-2: The framework is able to use touch input provided by TUIO.
• F-FW-3: The framework will provide interfaces / a base class for what a movement system should provide.
• F-FW-4: The framework will provide interfaces / a base class for interaction between tangibles.
• F-FW-5: The base rules, winning and losing conditions will be available via a custom expandable gamemode class.
• F-FW-6: The framework provides a basic AI-Controller for characters that are not associated to an marker.
• F-FW-7: The framework provides easy access to scores & board size via a board class.
  
  
• F-Figure-1: The framework defines a base class for tangibles.
• F-Figure-2: The figure position is recognized with markers.
• F-Figure-3: The player character is represented with a visual representation.
• F-Figure-4: A figure is able to use different movement systems.
• F-Figure-5: The figure classes will be able to have different properties.
• F-Figure-6: The figure classes will be able to interact with another.
• F-Figure-7-O: Visualize the movement system by showing a growing ring of possible movement options.
• F-Figure-8-O: Provide sample properties like classes, races and armor which have an effect on the character stats.
  
  
• F-Card-1: Played cards will trigger an effect defined in various subclasses.
  
  
• F-Stat-1-O: The players will be able to see detailed character stats of their owned figures.
  
  
• F-Quest-1-O: Baseclasses providing functionality to define quest text, objectives and rewards will be provided.
• F-Quest-2-O: Multiple rewards can be received upon quest completion.
• F-Quest-3-O: At least one objective has to be completed to complete a quest.
  

Non-Functional Requirements

• NF-TUIO-1: The plugin will not crash.
• NF-TUIO-2: The plugin will not cause any errors.
• NF-TUIO-3: The plugin will have low impact on performance.
• NF-TUIO-4: The latency between recognizing input and throwing events will be low.
• NF-TUIO-5: The plugin follows the TUIO v1.1 specification to translate 2D TUIO events.
  
  
• NF-FW-1: The framework will not crash.
• NF-FW-2: The framework will not cause any errors.
• NF-FW-3: The framework will have low impact on performance.
• NF-FW-4: The framework will be available as a template for the Unreal Engine.
• NF-FW-5: The framework will support multiple maps and will work even with bigger and smaller maps.

Task

The goals of this project are to

• Provide reacTIVision support to the Unreal Engine as a plugin.
• Provide a framework on top of the plugin to support (tabletop) game development.
• Measure benchmarks of the framework and develop a proof of concept.

After the development, a short pre-study will be conducted comparing the usability of the different interaction techniques the hybrid reality table provides. The study will also be used to unveil potential bugs.

In the future, the system can be expanded and used to research how multimediality in games influences game experience, the social context of the game and how to design user interfaces for multimodal tables.

A requirement analysis was concluded beforehand to generate requirements based on how typical tabletop games work.

Concept

The system will be developed using the Unreal Engine v4.16 and will run on a multimodal table. The TUIO protocol (Kaltenbrunner, 2014) will be used for communication between the table and a workstation. Code will be written in C++ and via Blueprints.

The project will be developed during a schedule divided into five phases. A main goal is to develop an extensible framework providing base functions and easy to use interfaces.

Phase 1 - Plugin Development

Phase duration: 2-3 Weeks

In the first phase the plugin will be developed. It will provide the following API:

TUIO-Plugin API

Phase 2 - Tabletop Framework

Phase duration: 2-3 weeks

The framework is based upon typical tabletop games. Expanding the plugin towards roleplaying tabletop games is optional and will be done if enough time is available.

Tabletop Framework API

Phase 3 - Proof of Concept & Benchmarking

Phase duration: 2-3 weeks

Benchmarks will be done after framework development. Benchmarking will focus on measuring the influence on GPU, CPU & reaction times.

The proof of concept will include following simple mechanics:

Three areas

The map is divided into three areas: Two play-areas on opposing sides where the players are able to move via moving their marker and a battlefield inbetween. The players are not able to move into the battlefield.

Attack system

Four simple physics based attack systems will be developed and can be changed in order to enable the study. These will be projectile based - the difference between these systems will be how the projectile flight path is manipulated.

Playing cards

There will be two different effects when playing a card:

• Half of the cards are mapped to a blockage. Using this card, a permanent blockage can be established on the battlefield.
• A short-lived shockwave can be placed which pushes projectiles in the opposite direction.

Winning & Losing Conditions

This will utilize the base game mode. The base mode will only include winning & losing conditions: The player with 0 hitpoints loses whereas the other player wins.

Basic AI-Controller

One or both of the players will be controllable via AI to enable single player gaming.

Stat Overview (optional)

Players will be able to see their hitpoints when they place their stat overview on the table.

Second map (Optional)

A second map with a different battlefield will be made available.

Quests (Optional)

A quest where the objective is to hit the enemy X times could wield gold rewards.

Gold (Optional)

Gold will be obtainable via quests and will automatically restore a shockwave card when enough gold is collected.

Phase 4 - Pre Study

Phase duration: 1 week

A short pre-study will be conducted after the prototype is finished. The study will involve playing a round of the full game and will use the Game Experience Questionnaire (IJsselsteijn, de Kort & Poels, 2013) to measure immersion and game experience as well as the QUESI (Hurtienne & Naumann, 2010) to measure the intuitiveness of the system. The main focus will be on which interaction method is suitable for a hybrid reality attack system and to test the implementation. The system will have one condition with four levels: How the attack system and the projectiles flight path is controlled.

Following four interaction methods will be tested:

• Throwing a card on the table
• Drawing a flight path via touch interaction
• Rotating the hero game piece
• Moving a slider

Phase 5 - Report Writing & Refinement

Phase duration: 3 weeks

The report will be written mostly during project development and refined during the last two weeks. The report language will be English, the study results will be analyzed using R (The R Foundation, 2017).

Schedule

References

Dormans, J. (2006). On the Role of the Die: A brief ludologic study of pen-and-paper roleplaying games and their rules. Game Studies, 6(1).

Epic Games (2017a). Game Engine Technology by Unreal. URL: https://www.unrealengine.com. Accessed 16 June 2017.

Epic Games (2017b). Rendering and Graphics. URL: https://docs.unrealengine.com/latest/INT/Engine/Rendering/index.html. Accessed 16 June 2017.

Fischbach, M., Lugrin, J. L., Marc Erich Latoschik, M. E. & Michael Fendt (2014). Picture-based Localisation For Pervasive Gaming. Virtuelle und Erweiterte Realität, 11. Workshop der GI-Fachgruppe VR/AR.

Fischbach, M., Zimmerer, C., Giebler-Schubert, A. & Latoschik, M. E. (2014). Exploring multimodal interaction techniques for a mixed reality digital surface (demo). IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

Giebler-Schubert, A., Zimmerer, C., Wedler, T., Fischbach, M. & Latoschik, M. E. (2013). Ein digitales Tabletop-Rollenspiel für Mixed-Reality-Interaktionstechniken. In M. E. Latoschik, O. Staadt, F. Steinicke (Eds.), Virtuelle und Erweiterte Realität, 10. Workshop der GI-Fachgruppe VR/AR (pp. 181-184). Shaker Verlag.

Hasbro (2015). Risiko Spiel. URL: http://www.risiko-spiel.de/. Accessed 03 June 2015.

Hurtienne, J. & Naumann, A. (2010). QUESI - A questionnaire for measuring the subjective consequences of intuitive use. In R. Porzel, N. Sebanz & M. Spitzer (eds.), Interdisciplinary College 2010. Focus Theme: Play, Act and Learn, 536. Fraunhofer Gesellschaft, Sankt Augustin.

IJsselsteijn, W. A., de Kort, Y. & Poels, K. (2013). The Game Experience Questionnaire. Eindhoven: Technische Universiteit Eindhoven.

Joyner, D., Wu, C. & Yi-Luen Do, E. (2009). Tangible optical chess: a laser strategy game on an interactive tabletop. Proceedings of the 8th International Conference on Interaction Design and Children (IDC ‘09), 278-279. ACM, New York, NY, USA.

Kaltenbrunner, M. & Bencina, R. (2007). reacTIVision: A Computer-Vision Framework for Table-Based Tangible Interaction. Proc TEI07. ACM Press.

Kaltenbrunner, M. (2014). TUIO.org. URL: https://www.tuio.org/. Accessed 16 June 2017.

Latoschik, M. E. & Tramberend, H. (2011). Simulator X: A Scalable and Concurrent Software Platform for Intelligent Realtime Interactive Systems. Proceedings of the IEEE VR 2011.

Lee, J. & Lee, H. (2010). The computer-mediated communication network: exploring the linkage between the online community and social capital. New Media & Society, 12(5), 711-727.

Magerkurth, C., Memisoglu, M., Engelke, T. & Streitz, N. (2004). Towards the next generation of tabletop gaming experiences. Proceedings of Graphics Interface 2004 (GI ‘04), 73-80. Canadian Human-Computer Communications Society, School of Computer Science, University of Waterloo, Waterloo, Ontario, Canada.

Milgram, P. & Kishino, F. (1994), A Taxonomy of Mixed Reality Visual Displays. IEICE TRANSACTIONS on Information and Systems, E77-D(12), 1321-1329.

MultiTaction (2017). Collaborative software solutions from MultiTaction. URL: https://www.multitaction.com/. Accessed 16 June 2017.

Pegasus Spiele (2010). Quest: Zeit der Helden. URL: http://www.pegasus.de/quest-zeit-der-helden/. Accessed 8 June 2017.

Piotr K. (2016). Re: How can I get a list of (or number of) board games by year range? URL: https://boardgamegeek.com/article/21815077#21815077. Accessed 16 June 2017.

Samsung (2012). 40” SUR40 SMART Signage. URL: http://www.samsung.com/uk/business/business-products/smart-signage/professional-display/LH40SFWTGC/EN. Accessed 16 June 2017.

Schachcomputer.info (2017). Fidelity Chess Challenger. URL: https://www.schach-computer.info/wiki/index.php?title=Fidelity_Chess_Challenger. Accessed 16 June 2017.

Sheldon, S. (2016). AppList. https://docs.google.com/spreadsheets/d/1M824hQ0kgksiNuEmBMGFsAEsVPBo2VD5FvpEvYj6UVI/edit#gid=693775771. Accessed 8 June 2017.

The Entertainment Software Association (2016). Essential Facts About The Computer And Video Game Industry URL: http://essentialfacts.theesa.com/mobile/. Accessed 16 June 2017.

The R Foundation (2017). The R Project for Statistical Computing. URL: https://www.r-project.org/. Accessed 16 June 2017.

Unity Technologies (2017). Unity - Game Engine. URL: https://unity3d.com/. Accessed 16 June 2017.

Wikimedia Foundation, Inc. (2017). List of Unreal Engine games. URL: https://en.wikipedia.org/wiki/List_of_Unreal_Engine_games. Accessed 16 June 2017.

Wizards of the Coast (2017). D&D Official Homepage. URL: http://dnd.wizards.com/. Accessed 16 June 2017.

Zimmerer, C., Fischbach, M. & Latoschik, M. E. (2014). Fusion of mixed reality tabletop and location-based applications for pervasive games. Proceedings of the 2014 ACM International Conference on Interactive Tabletops and Surfaces. ACM.

Contact Persons at the University Würzburg