Traversing the pass: A serious game teaching students how neural networks learn

Closed Human-Computer Interaction Master Thesis No Affiliation

Motivation

Machine learning is becoming more and more prevalent in our daily life. It is used by many companies, like Google, Amazon, or Netflix, to improve and analyze user interactions. It has also grown to be one of the most important topics in science. Over 130k articles have been indexed on Google Scholar in 2021 alone, with over a million articles being published since 2017. Thus, the demand for machine learning experts is ever-growing. Many schools and universities often teach it as part of their curricula. However, students perceive machine learning as hard to learn and understand. A survey among ten instructors of machine learning courses reveals that higher learning goals, like integrating the aspects of a neural network or analyzing and comparing a network to another, were the most challenging to teach [1]. This is supported by the findings of [2], who interviewed thirty computer science students that attended a machine learning course at university level. The author concluded that students found evaluating and comparing different models hard to understand. This is challenging even for experienced researchers [3]. Often, programmers treat neural networks as black boxes and experiment with the parameters until they get the desired result. It shows a need to better understand how these networks learn and operate in detail, especially for students from other backgrounds.

Teaching these concepts could be assisted with Serious Games, which are games that are used for purposes other than mere entertainment, e.g., learning [4], [5]. These games envelop the concepts they want to teach in-game mechanics [6], [7] and require the players to repeatedly apply them, supporting learning by repetition [8], [9] and generating flow. Furthermore, these games’ repeated early successes and overall interactivity foster learning [10], [11] and improve the high-level understanding of the enveloped concepts [12]. Serious games have effectively been used in teaching complex topics, like AI [13], logistics [14], computational thinking [15], debugging [16], and affine transformations [17].

Multiple serious games teaching various machine learning concepts are already available from scientific and commercial sources (some examples in figures 1-5. QUBE [18] and ArtBot [19] teach reinforcement learning by letting the players navigate an avatar through a dungeon by changing learning parameters and placing obstacles. A subsequent study of the former by [12] found that it successfully generated a high-level understanding of the concept, but participants struggled to understand how the network learned, while the latter’s evaluation yielded similar results. Beneath these scientific implementations, While True: Learn() [20] and Learning Factory [21] teach machine learning concepts by enveloping them in increasingly more complex puzzles and logistic problems, respectively. Although the player receives basic information about these algorithms when unlocking them, how they work and learn in detail is only available with external text and video tutorials.

All these games have in common that they treat the inner workings of neural networks, i.e., how these networks learn and what influence changing a parameter has, as a black box. This is highlighted by the findings of [22] and their game ML-Quest, which teaches supervised learning, gradient descent, and K-nearest neighbor. Their evaluation showed that only 42% of the interviewed students knew anything about the underlying processes, highlighting the importance of teaching these concepts. The students would have also preferred to play the game on their phones, making it easier to integrate learning into their daily lives.

Goal

This work attempts to open up the black box of how neural networks learn and make it less challenging to understand. Following a user-centered design approach [23], I analyzed the needs and requirements of the end-users, which in this case are students first learning about machine learning. The reviewed literature showed that the most challenging part about learning the basics of machine learning is how the inner workings, e.g., forward and backward propagation, work and learn.

I will develop a serious game where students playfully learn how a neural network learns using game mechanics that foster understanding. After development, I will confirm if my implementation is addressing the problems the correct way by letting students that have already completed the machine learning course at the University of Würzburg try the prototype. Then, I will adjust the used mechanics accordingly to the received feedback. In the end, I will evaluate the game with students that are new to machine learning.

Game Concept

The game will be developed using the Unity Engine 2020.3.21f1 [24] and run on state-of-the-art Windows computers. It will teach the concept of for- and backward propagation, e.g., how a neural network learns. In the typical fashion of serious games, it will envelop the concepts in game mechanics [6], [7]:

In the turn-based game “Traversing the pass”, the player will control a tourist company trying to predict which mountain peak each hiking group will prefer. At the bottom of the map, the players see the number of hikers and their available rations (representing the inputs of a neural network). Above that, they will see different campsites on the map (representing the neurons) and the trails between them (representing the connections). At the top of the screen is the mountain peak, where all hikes end (representing the output).

In the first phase (forward propagation), the players will be presented with the number of hikers and how many rations they have available. They will now have to predict how long it takes for that specific group of hikers using the estimate of how long each trail takes (representing the weights). Afterward, the players enter the second phase (backward propagation). They will be told how long it took the group to reach the mountain peak. Using that, they now have to update the trail estimates. The round ends when all trail estimates are updated. The players will receive feedback about correct and wrong calculations and suggestions on fixing them. Afterward, the next round begins, starting again with the first phase. This will go on for multiple rounds before a new mode opens up: The players will be able to “deploy” their system (live prediction). After deploying, the system will run automatically, predict new groups of hikers every few seconds, and generate money. The player can always turn off this mode and train the system more, if needed.

In addition to the gameplay loop itself, I will include reward systems to increase motivation [25]. In deployment mode, the players receive money depending on the number of updated trails and the current loss of the system. Thus, they will gain an increasing amount of money over time. They will be able to spend the money earned to buy skins for their campsites and unlock more campsites, hiking trails, and supplies. This will make the game progressively harder by adding more inputs, neurons, and a second hidden layer. Gradually increasing the difficulty helps to maintain flow [26] and requires the players to repeatedly apply learned concepts [8], which eases learning [9].

Further, after unlocking every neuron on the map and some rounds of practicing the for- and backward propagation, the players will unlock automated training. When bought, the game will automatically train the neural network in deployment mode, which allows them to observe how the algorithms are applied correctly during play, which - in theory - further increases understanding [8], [11], and now passively create income (akin to cookie clicker games) while increasing the networks’ accuracy. At the same time, the players unlock different maps, where they need to use different activation functions or multiple outputs, generate training data, and even solve a basic recurrent neural network. The players win the game by reaching the number one spot of tourist companies, which they achieve after unlocking every neuron and fully automating every map.

Methods

After finishing the development phase, I will evaluate the game to see if for- and backward propagation are easier to understand. First, I will invite students that finished the machine learning course at the University of Würzburg. They will play the game while thinking aloud and fill out a questionnaire about typical HCI measurements like usability, user experience, and gaming immersion. I will then adjust the prototype according to the received feedback.

In a second study, I will recruit students that are new to machine learning and split them into two groups for a three-day-long exercise session. The first group will learn the concept in a conventional way, i.e., using exercises printed on paper, whereas the second group will learn the concept using the game. Each day, the participants receive new exercises or unlock new levels in the game, respectively. After the training sessions, the participants will have to complete a test to reflect on what they learned and a questionnaire on their experienced task load. They will then fill out questionnaires about learning aspects, like motivation, task load, and acceptance. By splitting the participants into two groups, each participant will learn the basics of machine learning with only one method, not both, which is necessary to attribute the learning success to either the game or the traditional method. The outcome will further be evaluated in a second study two weeks later, where the students repeat the test without finishing the exercise first. After finishing this thesis, I plan to further validate the learning effectiveness of the game by evaluating it over a whole semester complementary to a machine learning course.

Timeline

[1] E. Sulmont, E. Patitsas, and J. R. Cooperstock, “Can You Teach Me To Machine Learn?” pp. 948–954, 2019, doi: 10.1145/3287324.3287392.

[2] N. Lavesson, “Learning machine learning: A case study,” IEEE Transactions on Education, vol. 53, no. 4, pp. 672–676, 2010.

[3] K. Patel, J. Fogarty, J. A. Landay, and B. L. Harrison, “Examining difficulties software developers encounter in the adoption of statistical machine learning.” in AAAI, 2008, pp. 1563–1566.

[4] T. Susi, M. Johannesson, and P. Backlund, “Serious games: An overview,” 2007.

[5] S. De Freitas and F. Liarokapis, “Serious games: A new paradigm for education?” in Serious games and edutainment applications, Springer, 2011, pp. 9–23.

[6] C. B. Eiben et al., “Increased diels-alderase activity through backbone remodeling guided by foldit players,” Nature biotechnology, vol. 30, no. 2, pp. 190–192, 2012.

[7] S. Oberdörfer and M. E. Latoschik, “Predicting learning effects of computer games using the gamified knowledge encoding model,” Entertainment Computing, vol. 32, p. 100315, 2019.

[8] J. P. Gee, “What video games have to teach us about learning and literacy,” Computers in Entertainment (CIE), vol. 1, no. 1, pp. 20–20, 2003.

[9] M. Csikszentmihalyi and M. Csikzentmihaly, Flow: The psychology of optimal experience, vol. 1990. Harper & Row New York, 1990.

[10] D. Smilkov, S. Carter, D. Sculley, F. B. Viégas, and M. Wattenberg, “Direct-Manipulation Visualization of Deep Networks,” 2017,Available: http://arxiv.org/abs/1708.03788

[11] A. A. De Freitas and T. B. Weingart, “I’m Going to Learn What?!?: Teaching Artificial Intelligence to Freshmen in an Introductory Computer Science Course,” SIGCSE 2021 - Proceedings of the 52nd ACM Technical Symposium on Computer Science Education, pp. 198–204, 2021, doi: 10.1145/3408877.3432530.

[12] C. M. Myers, J. Xie, and J. Zhu, “A Game-Based Approach for Helping Designers Learn Machine Learning Concepts,” 2020,Available: http://arxiv.org/abs/2009.05605

[13] N. Sintov et al., “From the lab to the classroom and beyond: Extending a game-based research platform for teaching ai to diverse audiences,” 30th AAAI Conference on Artificial Intelligence, AAAI 2016, pp. 4107–4112, 2016.

[14] C. L. Liu, “Using a video game to teach supply chain and logistics management,” Interactive Learning Environments, vol. 25, no. 8, pp. 1009–1024, 2017, doi: 10.1080/10494820.2016.1242503.

[15] B. Law, “Puzzle games: A metaphor for computational thinking,” Proceedings of the European Conference on Games-based Learning, vol. 2016–Janua, pp. 344–353, 2016.

[16] A. S. M. Venigalla and S. Chimalakonda, “G4D - a treasure hunt game for novice programmers to learn debugging,” Smart Learning Environments, vol. 7, no. 1, 2020, doi: 10.1186/s40561-020-00129-4.

[17] S. Oberdörfer, D. Heidrich, and M. E. Latoschik, “Usability of gamified knowledge learning in VR and desktop-3D,” in Proceedings of the 2019 CHI conference on human factors in computing systems, 2019, pp. 1–13.

[18] J. Xie, J. Zhu, and C. M. Myers, “Interactive visualizer to facilitate game designers in understanding machine learning,” Conference on Human Factors in Computing Systems - Proceedings, pp. 1–6, 2019, doi: 10.1145/3290607.3312851.

[19] I. Voulgari, M. Zammit, E. Stouraitis, A. Liapis, and G. Yannakakis, “Learn to Machine Learn: Designing a Game Based Approach for Teaching Machine Learning to Primary and Secondary Education Students,” Proceedings of Interaction Design and Children, IDC 2021, pp. 593–598, 2021, doi: 10.1145/3459990.3465176.

[20] Luden.io, “While true: learn().” Jan. 2019.Available: https://store.steampowered.com/app/619150/while_True_learn

[21] Luden.io, “Learning factory.” Feb. 2021.Available: https://store.steampowered.com/app/1150090/Learning_Factory

[22] S. Priya, S. Bhadra, and S. Chimalakonda, “ML-Quest: A Game for Introducing Machine Learning Concepts to K-12 Students,” vol. 1, no. 1, pp. 1–13, 2021,Available: http://arxiv.org/abs/2107.06206

[23] J. J. Garrett, The elements of user experience: User-centered design for the web and beyond. Pearson Education, 2010.

[24] Unity, “Unity engine.” 2021.Available: https://unity.com

[25] K. Seaborn and D. I. Fels, “Gamification in theory and action: A survey,” International Journal of human-computer studies, vol. 74, pp. 14–31, 2015.

[26] J. McGonigal, Reality is broken: Why games make us better and how they can change the world. Penguin, 2011.

Contact Persons at the University Würzburg