Categories
Uncategorized

COMP704 – Concept and Research

For my first journal entry I will be explaining the game my AI will be playing and the algorithm picked for its learning capabilities.

The Concept

For this project I will be creating an AI that will learn to play Snake (1976) using Q-learning, a reinforcement learning method, to learn and improve over time via trail and error. When the snake moves around it will receive a reward of 0, while when the snake hits itself it will lose a point (-1) and when it hits a fruit it will get a point of 1.

The game

The AI’s environment will be Snake (1976) as it is a simple game to play with some challenges such as the snake getting bigger over time and the fruit spawning in random positions. Because of this I believe it will fit for my scope of 5 weeks. A picture of Snake (1976) is shown below in fig 1.

Snake through the ages — Nokia phones community
Fig. 1: Unknown maker. ca. 2019. No title [photo]

However, to ensure that the AI can learn from it mistakes I want to see if I can give the fruit a set position so that the fruit will spawn in the same position each game unless it’s going to spawn where the snake currently resides in which case the position will be random. In addition, while Snake (1976) is arguably endless which could be problematic, so to ensure their is a reach able goal, all the AI needs to do is collect 10 pieces of fruit and the game will be over.

The algorithm

The algorithm that I will be using is Q-learning, a reinforcement algorithm that is off policy and uses Temporal Difference (TD) control. Because it is a reinforcement algorithm it will be using trial and error to learn and won’t rely on an existing model or data to determine its decisions. This would save time as I won’t need to collect data from users to make the AI work during the first attempts at training, however it will mean that’s the first bit of training will mainly be it moving in random directions and losing the game (S. Sutton and G. Barto 2018).

Q learning uses rewards to make decisions on which action to preform and since it incorporates the greedy policy it will pick the reward with the highest value. However, to prevent it from making mistakes such as ignoring paths that might have a higher reward 10% of the time instead of 100% of the time, the algorithm will use a random event to decide if it should explore (pick an action at random) or exploit its environment (use Q learning algorithm and sampled action/state). The random event will be based on the epsilon value which is a part of the greedy policy. At the beginning of training epsilon will be set to 1 meaning it should explore the environment 100% of the time, however over time the number should go down to increase the chances of it picking exploit rather than explore (Lee 2019).

As mentioned earlier the AI uses rewards, received from doing good or bad actions, this is used in the Q learning equation to ensure that a higher quality value is calculated and stores in the q table so that is will likely be picked again in later steps. The reward value will increase or decrease during the training and will be set back to zero at the start of each training session (S. Sutton and G. Barto 2018). A diagram of the Q learning equation is shown below in fig 2.

Fig. 2: ADL. ca. 2018. No title [diagram]

Q-learning uses a table to store its actions and states, this stores the data it has gain through its learning and will use when making decisions on which action to pick (N. Yannakakis and Togelius 2018). For the AI to play snake it will have at least 4 actions, those being: up, down, left and right. These are based on the values assigned to its state, which in this case will be the position it’s currently in, then it will pick the action with the highest value. An example of what the table might look like is shown below in fig 3.

Fig. 3: Oates 2022. example of AI’s states and actions table

In addition, other actions could be added on if need be such as an action for fruit which should have a very reward value and a death action which should have 0 or a very low reward value in most columns of the table

A limitation of Q learning is to do with the table as depending on the chosen action and state the number of row and columns could be become incredibly large. Due to this the memory and computational requirements could become very high and thus increasing the amount of time per decision, affecting the quality of the results (N. Yannakakis and Togelius 2018).

There are two ways to avoid this the first is by being careful about the choice of action and state to keep it as small as possible. The other option is to use neuroevolution due to it being able to sensibly pick actions and states for a system while still remain companionly friendly (N. Yannakakis and Togelius 2018).

Since I only have 5 weeks I aim to go with the former as mentioned earlier when it comes to actions there are only 4 to pick from. As for states they will be the positions of the grid, but each axis should be stored as a whole number. In addition, the grid making up the environment will be 15 x 15 meaning in total there will be 225 states, which is small compared to the possible billions of states that a more complex game could have in terms of positions.

Potential problems

While I have been careful on what game and AI solution to pick, I am nervous about how unforeseen bug and problems could get in the way and affect development. For example, a potential problem could be compatibility between the AI and the environment resulting in errors that the AI can’t handle and removing time required for development.

However, if these problems do occur, then I would have to adjust my development scope and features to meet the new availability I have to build, train and polish the AI.

Training environment

Since I only have 5 weeks I will be using open AI Gym to create the training environment. While the current Open AI Gym doesn’t have Snake there is a plugin that I found which allows it to have that environment (Grant and Rishaug 2021).

Research and Development timeline

As mentioned earlier I have 5 weeks to research, develop and train the AI, I will aim this week to solely research for other the AI and it’s environment. I will then use the next 2 weeks developing the AI to ensure that it works properly. The last two weeks will be spent polishing and training the AI to ensure that it can collect 10 pieces of fruit. In fig 4 you can see a Gantt chart of my research plan and how I will spend these 5 weeks.

Fig. 4: Oates. 2022. Table of development timeline

Further enquires

During the training session I need to keep track of the AI’s reward and its parameters to ensure I get good results and training data, however this will take some time to find out the correct set of parameters to get the best results. Consider how I have a shot scope I’m curious if a simple Q-learning system or linear regression system could be used to make a prediction on what kind of score the AI will get based on it’s current parameters. For example, if a multi-variable linear regression system used collected training data and using these results I could compare what the score the AI got, compare it to the predicted score and then adjust the parameters to see if the AI could improve to reach the predicted results (Tsoukalas and Fragiadakis 2016).

Of course I would need to make sure the predictions are correct which would means the linear regression would requires lots of training data generated by the AI to ensure that it matches. But having the score something to compare to would allow for accurate improvements rather then lots experimentation with little results.

Reflection and conclusion

Reflecting on my research, I am quite excited to use Q-learning and build one from scratch as it will allow me to gain more experience as an AI programmer. By building this AI from scratch I will have a great portfolio piece and while time is short the equation required for the AI is quite simple and shouldn’t be too hard to implement. That being said one of the complication I feel I will face is a seeing if positions can work as the AI’s states, however that will be figured out over time. From here I aim to being development on the Q-learning AI and at least have the structure of the AI set up and ready for functionality.

Through out my journal I will be reflecting on my work with Gibb’s reflective cycle, which contains 5 states: Description, Feelings, Evaluation, Conclusion and Action (Ashby 2006).

Bibliography

ASHBY, Carmel. 2006. ‘Models for reflective practice’. Practice Nurse : The Journal for Nurses in General Practice, 32(10), 28.

GRANT, Satchel and Joakim RISHAUG. 2021. ‘Gym-Snake’. Available at: https://github.com/grantsrb/Gym-Snake. [Accessed 26/1/22].

LEE, Dan. 2019. ‘Reinforcement Learning, Part 6: TD(Λ) & Q-Learning’. Available at: https://medium.com/ai%C2%B3-theory-practice-business/reinforcement-learning-part-6-td-%CE%BB-q-learning-99cdfdf4e76a. [Accessed 24/1/22].

N. YANNAKAKIS, Georgios and Julians TOGELIUS. 2018. Artificial Intelligence and Games. Springer.

S. SUTTON, Richard and Andrew G. BARTO. 2018. Reinforcement Learning an Introduction. (Second edn). United States of America: Westchester Publishing Services.

Snake. 1976. Gremlin Interactive, Gremlin Interactive.

TSOUKALAS, V. D. and N. G. FRAGIADAKIS. 2016. ‘Prediction of occupational risk in the shipbuilding industry using multivariable linear regression and genetic algorithm analysis’. Safety Science, 83, 12-22.

Figure List

Figure 1: Unknown maker. ca. 2019. No title [photo]. V&A [online]. Available at: https://community.phones.nokia.com/discussion/44549/snake-through-the-ages.

Figure 2: ADL. ca. 2018. No title [diagram]. V&A [online]. Available at: https://www.freecodecamp.org/news/an-introduction-to-q-learning-reinforcement-learning-14ac0b4493cc/amp/.

Figure 3: Max Oates. 2022. example of AI’s states and actions table.

Figure 4: Max Oates. 2022. Table of development timeline.

Categories
Uncategorized

Lessons learned from my second game jam

With the second game jam coming to an end I decided to create a blog post about the lessons I learned from the this week. Mainly I will be focusing on what went well and what didn’t. I will then finish with reflecting on what I have learned and how I will aim to do better.

Since we had to form our own teams, pick a genre and pick which systems we want to use as inspiration, this lead to the first 1 -2 days being spent forming a team and coming up with an idea. Fortunately we had a discord to help form teams, and forming one was soon done after. On the next day we spent the day coming up with an idea for the game, though we got along with each other pretty well, the project was over scoped, this was possibly due to the fact we didn’t have lot of time and wanted to begin immediately so we tried to fit everyone’s ideas and leave it there, without considering how long each peace of work would take to make. In addition, I wanted to make a game that was bigger and better compared to the previous one I made which may have affected how the project was going to look and function. One of the other things that I think slowed down the project was external factors outside of the university.

In the end I feel like the cons of the project can be summed up to a lack of organisation while also trying to deal with the stress of getting the project playable and finished in 4 days, with evidenced playtesting while trying to handle external factors.

Looking at the pros of the group project, the project was definitely a step up, compared to the previous project as there was more gameplay and mechanics for the player to use. This is because the core gameplay loop was about hiding from enemies in bushes while trying to reach house to deliver letters all with in 5 minutes. In addition, the narrative was improved as well since the story was all about delivering soldier’s mail to their families to making people happy in sad times. an example of the game is show below in fig 1.

Fig: 1. Oates. 2021. Screenshot of gameplay [screen shot]

Though the art struggled a bit due to lack of experience with Unity’s materials and making 2.5D games, the aesthetic of late 1800s to early 1900s was achieved especially in the environment. This because creating a old timely map as the environment was a focus of the game. Below you can see what the map looked like without the game elements in fig 2.

Fig: 2. Pechtel. 2021. Game Background [Digital]

Another thing that went well was getting along with each team member as though organisation was lacking, me and the rest of the team got on very well and had no arguments. In addition, while coming up with the idea for our game we also spent time planning out our target market. One of the artists was able to immediately create 3 personas of our target market as they were well experienced in that area.

From play testing, most of the results most were positive with the aesthetic being the most well received area of the game. People liked the gameplay loop but as results stated improvements were needed and some thought there wasn’t enough. If this idea goes further more mechanics could be added, mainly to ensure that there is challenge and something to encourage the player to move forward. Though the game was set during a war, it only implied in the art and narrative, which prevented us from upsetting any play testers.

After the playtesting, improvements were made to the game, mainly adding a counter to show you how much mail was left to deliver and also increasing the guards speed by 0.5units every time you delivered a letter. Play tests justified this by stating it would help give them a challenge as it was generally quite easy to avoid the guards. A picture of the improved game is show below in fig 3.

Fig: 3. Oates. 2021. Screenshot of improved game [screen shot]

Conclusion

Reflecting on what has happened if I want to avoid feature creep from happening again I should look at the backlog of the tasks that need to be done in order to meet the overall goal. From there I should make decisions as to what we should focus on and where we need to make changes in order to meet the end product (Pries and Quigley 2011).

For the next group project if I became the scrum master I would use a burn down chart to keep track of the teams progress and also use it to keep track of time. Each day I would fill in how much work has been completed based on the scrum board, this would be put into a graph and then compared to a predicted graph of how much should be complete. After the stand up meetings I would state that we need to get this many tasks done my the end of the day to ensure that we stay on target. If were are facing troubles I would also bring it up and suggest that we make changes so that we can meet our deadline Reflecting on what has happened if I want to avoid feature creep from happening again I should look at the backlog of the tasks that need to be done in order to meet the overall goal. From there I should make decisions as to what we should focus on and where we need to make changes in order to meet the end product (Pries and Quigley 2011).

Despite everything that has happened, we did manage to get the gameplay loop working and the aesthetic was enjoyable. In addition, playtesters did enjoy the game and improvements have been made.

Bibliography

PRIES, Kim H. and Jon M. QUIGLEY. 2011. Scrum Project Management. Boca Raton, Fla: CRC Press.

Figure List

Figure: 1. Max Oates. 2021. Screenshot of gameplay [screen shot].

Figure 2. Jeanett Pechtel. 2021. Game Background [Digital].

Figure 3. Max Oates. 2021. Screenshot of improved game [screen shot]

Categories
Uncategorized

Modular AI

When ever I create an Artificial Intelligence (AI) I find it tedious that I have to hard code everything into the AI, this is especially the case when created multiple variations of AI as it can lead to repetition. Because of this I decided to take my second game jam as an opportunity to experiment with modular AI. Even though this game just needs one type of AI I decided to see if by making the AI modular, it would be considered designer friendly allowing designer and other non-programmer team members to work on the AI with little help from me.

Looking into Game AI pro 3 on modular AI it talks about modules being used for a specific function and should not be similar to one another. The book event suggest sub modules for modules that cover one area of the AI such as movement but have many variety’s of that area such as walking and riding a bike or car and so on. In addition, it also states using modular decision making something more akin to behaviour tress (Rabin 2017).

Unfortunately due to time constraints my AI is a Finite State Machine and only has modular actions while the states that are needed to make the transaction to different actions will need to be hard coded. However, each action inherits from an action base class allowing for more actions to be easily made. I also intended to ensure that the AI will not break if it is missing an action, which could allow for different enemies using different behaviours. For example, an AI could just have the chasing action so that the AI will only chase the player when spotted.

When planning and prototyping the AI I decided to use paper prototyping as it is a cheap and easy way to create and design features. This is shown below in fig 1.

Fig. 1: Oates 2021. Paper prototype of soldier AI [Drawing].

When creating the actions I decided to use Unity’s Scriptable objects (SO) as they are a great feature that lets you create programmable objects that can then be created like when a developer is creating a cube for the level.

When decided to use SO I was not entirely sure how to pass data to the AI or how to attach an array of SO to the AI component. Fortunately, a Unity presentation that focuses on SO was about to help me understand how to set it up, the presentation even had an example of modular AI. The presentation is called ‘Overthrowing the MonoBehaviour Tyranny in a Glorious Scriptable Object Revolution’ by Richard Fine.

Though these are not actual GameObjects that can be placed or seen in a level they are objects containing data that you can drag and drop into custom components. SOs can be used to contain data for items, weapons and so on, but for me the SOs contain the necessary data and instruction for each action that the AIs will use. This makes it designer friendly because one can be created without having to create a new script and can then be dragged and dropped into the AI component. An example of the SOs action is shown in fig 3.

Fig: 3. Oates 2021. Chase action example [Code snippet].

If I could do this again

If I were to do this again I would definitely look into creating SO states so that what triggers the actions can also be modular. In addition, one thing that wasn’t considered when designing the system was that the actions would be called every frame, which was fine for actions like Patrol, chase and Capture, however for the ‘looking’ action, since a random position was being made every frame I had to include a task system that ensured a new position wasn’t generated until it reached the current one.

Conclusion

Though the AI needs many improvements it does work and through this I feel like I have improved as a programmer specialising in AI. Reflecting on making a modular AI it feel like some aspects of it will have to be hard coded such as having variables that store data only the agent or outside sources can retrieve, implementing a task system and having code that allows for the states and actions to function properly.

When making a new modular AI I would spend an hour researching further into different modular AI systems, to try and see if I could create an more advanced modular AI with modular states. An AI like that with 5 actions and 5 decision states would take between 3 – 6 hours to plan and make. I think it would take this long as it took me 3 – 4 hours to make this current AI.

Bibliography

Fine, Richard. 2016. Overthrowing the MonoBehaviour Tyranny in a Glorious Scriptable Object Revolution. Available at: https://www.youtube.com/watch?v=6vmRwLYWNRo&t=820s [accessed 10th Oct 2021]

RABIN, Steve. 2017. Game AI Pro 3 : Collected Wisdom of Game AI Professionals. Boca Raton, FL: CRC Press, an imprint of A K Peters/CRC Press.

Figure list

Figure 1. Max Oates 2021. Paper prototype of soldier AI [Drawing].

Figure 3. Max Oates 2021. Chase action example [Code snippet].

Categories
Uncategorized

Working in a team part 2 and managing stress

As of yesterday I’m currently working in a second team and working on a new project. Though it’s good to be working with new people, learning new skills and working with another programmer rather than doing all the programming myself it does feel like I need to improve on my team working in skills, especially my communication skills and time management as well as managing my stress as the past few weeks have been quite stressful.

Firstly with communication, though there has been some progress using Hope’s techniques when dealing with my social anxiety as it has improved my confidence when speaking, however there hasn’t been much progress pass that. I’ve decided to not just incorporate Hope’s techniques for dealing with social anxiety but also looking into actual communication skills used in business. Specifically I have been looking at Rai’s book on business communication where I have learned that communication in teams is two way communication, meaning that everyone in the team is sending and receiving information. Because of this I need to remain silent and listen well when others are talking as it makes the other members in the team feel comfortable when talking and strengthen prowess when I’m talking to them. In addition, by being a good listener it can allow for potential questions to be answered while introducing new ideas to the project (Rai and Rai 2008).

When testing these skills for the brain storming session on the game it did go well letting everyone speak and get their ideas across and it even answered some of my questions without having to ask. However, due to remaining silent as long as possible to ensure everyone had said their bit, it did prevent me from stating my ideas unless asked, which did get annoying at time due to it feeling like I was being cut off before getting a chance to speak.

I feel that time management has not been one of my greatest strengths as trying to keep a plan for near future events while remaining flexible has always been a challenge for me. Looking at Brian Tracy’s book on time management it is stated that one must take into account what events are important to you and which ones are not so that you can be happy, as you are working on things that you genuinely enjoy working on. In addition, I should have a daily to do list with clear priorities so that I know what I aim to get done by the end of the day as a way to keep morals high while getting work done (Tracy 2014).

I believe that would be an effective tool to use on a daily basis as in the past when I used to do lists for work I have been more focused on my work. Using lists has also help me reduce my stress which allows me to move onto the next thing I’ll be writing about.

Dealing with stress has been problematic at times and as mentioned earlier, stress has been quite prevalent these past few weeks. While using techniques to manage my social anxiety have helped in ways stress had been a constant mainly due to how much work needs to get done with so little time, which is why I’m trying to incorporate better time management skills. That being said I wanted to try and research if there are any techniques to help manage my stress and see if I can decrease how often it comes up.

Spending some time looking over Stress management by Harry kane using techniques like meditation and stress balls could help with dealing with stress mainly due to it know effects to increase patience and calm down practitioner (Kane 2007). After attempting meditation for 4 days I have felt calmer especially after meditating despite it being hard to focus on one thing.

I have yet to try stress balls but it is stated that stress ball are used as methods for physical therapy and mainly should be used after a stressful bit of work has happened. It is also a common place item to have at work as it can help with present stressful events.

I will continue to use meditation as a way to calm down as it seems to be quite effective. However if things do change then I would give the stress ball a try to see if has any different results.

Conclusion

Reflecting on using these techniques to further my communication and stress management skills, I believe they are improving as I have seen some improvements in how I talk to others and managing to remain calm after to them. In addition, despite all the stress I have been experiencing I think I am developing better ways to deal with my stress.

In order to take these skills further I should begin by spending 30 mins writing a to do list for the next day so that I know what I need to achieve. When writing these tasks I need to priorities which task I value the most or are important to achieving the overall goal.

When using the business techniques to improve my communication I should spend a minute or 2 to remind myself to be mindful of others so that I can remain silent and listen. When listening I should aim to focus on what people are saying and not lose focus.

Finally for managing my stress. I will continue to mediate 5 -10 minutes during my morning routine. While meditating I should focus on one through and try to go back to that though if I lose focus. This in turn should help me manage my stress and patience.

Bibliography

KANE, Harry. 2007. Stress Management. (1st edn). Chandni Chowk, Delhi: Global Media.

TRACY, Brian. 2014. Time Management. (1st edn). New York: American Management Association. 

Categories
Uncategorized

Communities of practice in Software Cornwall

When it comes to events about technology or the industry in Cornwall, Software Cornwall is always there either on banders or hosting the event. They have been in network events like Kernow Dat where local tech companies come together to discuss their business, ideas or hold presentation about their journey to discovering new ideas.
However, I recently experienced one of their Tech Jams where I volunteered as an assistant to help run the event. Meeting people of various ages from children to young teens to middle aged adults, working on their personal projects or introducing them to activities where they could learn how to program electronics using Arduino or Raspberry Pi. One of the other activities was about using Python to hack Minecraft allowing them to control the environment, one of the children I helped was able to use the instructions given to them to spawn in an entire village using only the code they wrote. Their last activity was called Mission to Mars where people could work on an Arduino Mars rover to try and get it to meet the requirements that the ‘contractor’ wanted.

The rover itself was a part of a much larger event that Software Cornwall also hosted called “Mission to Mars” which was an event specifically for schools to help introduce people to the tech industry in Corwnall as well as team working skills and Agile. This is an event I will also be helping out in a few weeks’ time.

When it came to children their language and tone was inexperienced as they were quite new to code, however I had met teenagers who were working on their personal projects creating simulations using Python and Pygames, such as simulating water and sand or trying to create their own personal networks. Their tone and language ranged from being able to explain their code and theory very well without over complicating things while other went into great depth where at time it was hard to follow due to how complex their works was, but all the same they were happy to talk about their work. The same was for adults bringing in their personal projects, many being very happy to show off their work and the progress they have made. As for me and other volunteers we had to always stay positive and point people in the right directions when they were experiences difficulties.

The community was mostly about helping each other out or showing off their work, with a variety of newcomers and familiar faces. Interactions were quite mixed as mentioned before volunteers were talking to people to make sure they were fine, but besides that people usually either kept to themselves or talking to one another via friends or people who know one another from work.

Software Cornwall is partnered with a software development company called Head Forwards who tend to help host the events, but that hasn’t stopped people from other local tech companies helping out or joining in, one of the mangers from Bluefruit Software, brought their children to show them to Raspberry Pi and how to program electronic with it.
When it came to rules, their much really, when people came in, they were show activities they could do, or they were given desks for them to work on their personal project but mainly it was quite free.

As for volunteers we would have to try and interact with the people as much as possible, making sure people were doing well and help them out were needed, or try and understand their thought process when working on a new feature.

Conclusion

Looking at Software Cornwall and what they do for the local tech industry, they do a very good job at introducing people of all ages to an industry that is not very visible in the county. In addition, by using Tech Jams to introduce children to software and electronics at a young age, it can potentially introduce them to skills they have a talent for while allowing them to develop them early and get a career they really enjoy.

As for me being a volunteer has been helpful with my communications skill, despite the fact that I may not be doing well at communicating, when things do work out and the person, I am helping is happy it makes the whole endeavour worth while.

Categories
Uncategorized

Ideas from Different Systems

To help give us game ideas for our second game jam, we have been asked to research two different kinds of systems. One that is of culture or society and another that is technical or aesthetic.

Gross National Happiness (GNH)

For the culture or society system I decided immediately to use Bhutan’s Gross National Happiness which is a system introduced by the king of Bhutan back in the 1972, suggesting that progress should be measured and paved via happiness rather than with economy. This system was divided into 9 different categories so that making policies around happiness is possible (‘Bhutan’s Gross National Happiness Index’. 2021).

a diagram of GNH is shown below in fig 1.

Gross National Happiness - Measure What Matters
Fig. 1: Unknown maker. ca. 2011. gnh-domains [Diagram]

The system works by a happiness value if, an area is reported to have 50% or less happiness then they will receive benefits to ensure that those values increase. Depending on how well the system works it’s been recorded that only 10.4% of the country are under 50% happiness while 48.7% are between 55 – 65% happiness (narrowly happy) with 32.6% scoring up to 67 to 76% in happiness (very happy) and the remaining 8.3% having 77% in happiness.

Reflecting on why I chose this system, to me it’s a very unique system especially since most other countries use Gross Domestic Product (GDP), a system that measure how much money is being made and spent to determine the well being of a country (‘What is GDP?’. 2021). In addition, reading further into the why of the system it’s made clear that the king at the time cared about the well being of his people which adds a wholesome feeling to the system.

This is a similar feeling I hope to capture in the game jam as it could also get the player to learn something new about the world and how money isn’t the only thing to indicate well being. This, of course, depends on how much the style of the game is made to look like Bhutan’s culture and if whether or not the player believe GNH is an actual thing.

Procedural Content Generation (PCG)

The technical system I researched was Procedural Content Generation as it is a very interesting system that can be used as a mechanic or tool to make unique features/content with or without the designers input. This type of system is has s tendency to be used if the team consists mainly of programmers rather than artist and designs. This is one of the reasons I find this system interesting as it can be used to generate content at a near infinite possibility just by adjust its parameters to make game dev easier. The system also uses random values to ensure that each bit of content is unique (N. Yannakakis and Togelius 2018).

The type of PCG that I want to make is similar to Left4Dead’s (Value Corporation, 2008, PC) zombie system that would increase or decrease the amount of zombies in the game depending on the player’s progress. The proper name for this type of PCG system is called ‘Experience-Driven PCG’ and is used to adjust the pace of the game based on a value/model representing the player’s stress or other emotions, the pace in this case could be the design of the level, number of enemies or the type of weapons you will be given (N. Yannakakis and Togelius 2018). A diagram of experience-drive PCG can be show below in fig 2.

The main components of the experience-driven procedural content generator.  | Download Scientific Diagram
Fig. 2: Unknown maker. ca. 2018. The four key components of the experience-driven PCG framework [Diagram]

One of the problems that I would be facing is that, due to the randomness of PCG it can be hard to control the output of the system and thus affect the type of experience the game is trying to capture. In addition, when designing the game it is important to evaluate how well the content meets the intended experience as mentioned before this system could potentially generate infinite amounts of content but it wouldn’t mean much if any of it is good.

For the game jam I want to use this system to generate different types of disasters, affecting the overall happiness of the cultural system and it will be up to the player on deciding on how to fix it. I want to use a model of the player’s stress so that the as things progress, the system will either increase or decrease the amount of disasters or how greatly the disaster will impact the happiness value.

Conclusion

In conclusion, looking into two different types of systems has helped me understand how to find connections two areas that seem different from one another. I particularly found it interesting that Left4Dead use that system as it was well hidden and not mentioned as a big feature for the game.

If this idea comes to fruition, then I would spend a day researching and planning how to create the experience driven PCG. The system would be designed using flow charts to ensure that the design is easy to follow. Once that’s done I could begin prototyping the system in Unity using C#.

Bibliography

‘Bhutan’s Gross National Happiness Index’. 2021. Available at: https://ophi.org.uk/policy/gross-national-happiness-index/.

Left4Dead. 2008. Value Corporation, Value Corporation.

‘What is GDP?’. 2021. Available at: https://www.bankofengland.co.uk/knowledgebank/what-is-gdp. [Accessed Oct 12,].

N. YANNAKAKIS, Georgios and Julians TOGELIUS. 2018. Artificial Intelligence and Games. Springer.

Figure List

Figure 1: Unknown maker. ca. 2011. gnh-domains [Diagram]. V&A [online]. Available at: https://gnhusa.org/gross-national-happiness/ [accessed 12 October 2021]

Figure 2: Unknown maker. 2018. The four key components of the experience-driven PCG framework [Diagram], From Artificial Intelligence and Games pp. 181

Categories
Uncategorized

Unity’s Machine Learning

I’ve had a big interest in Machine learning for quite a while but have never really had the time to work on one. Fortunately, Unity has introduced a Machine learning tool that allows developers to create machine learning agents. In this case I will be creating my own agents and improving my knowledge of reinforcement learning as that is the method the agents uses to learn.

In this project I wanted to start simple, so I wanted the agent to learn how to reach the yellow sphere.

by using reinforcement learning it would learn through trial and error to reach the ball. Every time it reaches the sphere it’s given a reward, in this case 1 point but every time it hits the wall it will lose a point, and will aim to get a high reward by learning from it’s mistakes (Osiński and Budek. 2018).

Fig. 1: Oates 2021. picture of agent’s environment

When training the agent I needed to set up a virtual environment which required Python this was so that I could collect data on the agent’s decision making and see even see graphs to see where it was achieving high awards and where it failing. Below is a picture of me setting up the virtual environment in fig 2 and what data was being displayed during the training in fig 3.

Fig. 2: Oates 2021. virtual environment created through CMD
Fig. 3: Oates 2021. training in progress

When training the AI I decided to duplicate the agent and its environment 8 more times to speed up the training process. This provided to be quite effective as in few minutes all 9 agents where moving to the sphere with ease with occasional failures here and there.

Fig. 4: Oates 2021. using multiple agents to increase the rate of training

Below is a method which I can use to control the AI to ensure that they will move in the correct direction, this is show in fig 5. When setting up the controls, for some reason I had to swap around the directions by which I mean I set the agent’s x axis to the data required for the z axis and vise versa for the agent’s z axis. This in turn allowed the agent to move in the right direction which I find strange as at first I thought the Heuristic method shouldn’t affect the AI decision making nor was the virtual environment active to record data.

However after reading more on Unity’s Machine Learning I learned that if you’re using the heuristics method in the agent’s script, it may use the elements in the method. This dose makes sense as in fig 6 the actions used to dictate the agent’s movement is also being controlled in heuristics method.

Fig. 5: Oates 2021. controlling the agent to ensure its movement work
Fig. 6: Oates 2021. code the controls the agent’s movement

If I were to do this experiment differently I would spend more time preparing the agent and it’s environment, mainly to ensure the the agent can move in the correct directions to make sure that the input system or anything else in Unity wasn’t affecting it’s sense of direct. Looking at a blog about machine learning it’s said that the biggest challenge when using reinforcement learning is setting up the environment correctly (Osiński and Budek. 2018).

In addition, I would remove the Heuristic method from the agent script when training it, to see if anything would change the direction decides to move as inverting and swapping the input did have an affect on the direction it went during the the training process.

I could also experiment with Unsupervised Training which is a method of training where, certain metrics can be used to train the agent and make the discovery by itself without me having toe observe it (Lanham 2018).

Conclusion

After trying out this project I’m quite surprised about how simple the process is when training AI, however this probably would change if I was experimenting with training an agent to achieve a more complex task. To improve on this, I will spend a day trying to training an agent to find multiple objects in any order it decides to. For it to achieve its goal all it would need to do is find all the spheres and when it’s do so it will receive a reword. If it doesn’t find all the sphere’s in a set time frame, it will lose a point.

Bibliography

LANHAM, Micheal. 2018. Learn Unity ML-Agents – Fundamentals of Unity Machine Learning: Incorporate New Powerful ML Algorithms such as Deep Reinforcement Learning for Games. Birmingham, UNITED KINGDOM: Packt Publishing, Limited.

OSIŃSKI, Błażej and Konrad BUDEK. 2018. What is reinforcement learning? The complete guide.

Figure List

Figure 1: Max Oates. 2021. picture of agent’s environment.

Figure 2: Max Oates. 2021. virtual environment created through CMD.

Figure 3: Max Oates. 2021. training in progress.

Figure 4: Max Oates 2021. using multiple agents to increase the rate of training.

Figure 5: Max Oates. 2021. controlling the agent to ensure its movement work.

Figure 6: Max Oates 2021. code the controls the agent’s movement.

Categories
Uncategorized

Using Agile and time management

For this team project the team and I will be using Agile development to help plan out the development of our game. This is to ensure that we are aware of the game’s progress and how close we are to dead lines.

During my lecture we discussed Agile development as a means of planning your product for the user. When doing further research in Agile development, I came a cross User Stories, which are the tasks the each developer will have, however instead of simply writing the task as “Develop player movement” you need to consider four things (Measey et al. 2015):

  • Who – the player, client, etc.
  • What – they want
  • Why – do they want it
  • Acceptance criteria – a list of requirements for the feature to ensure the story is done

Both from the lecture and the research has shown that from third year to the present, I have been doing user stories incorrectly and have been simply writing tasks rather that actual user stories. Interestingly enough when I was taught how to use Agile way back in year 1, it was never mentioned that we have to write acceptance criteria. However I believe it is something for me to add when writing user stories as it allows the team to know when the feature is fully complete (Measey et al. 2015).

In addition, reading another book on Agile development by Clinton Keith has made me realise that you want to write User Stories that allow for negotiation between the team members as what you write in the User Story shouldn’t describe the complete process needed to create the feature (Keith 2010). An example of a good user story that can be negatable is show in fig 1.

Fig. 1: Keith 2010. A more negatable user story [Image]

If I was to write User Stories differently I would need to keep them short so that they are easy to understand and so that other teams members can understand negotiate with me about the feature. It would be a good idea to, consider the term SMART as it contains five good characteristics when writing a task(Wake. 2003). SMART is an acronym for:

  • Specific
  • Measurable
  • Achievable
  • Relevant
  • Timebound

I would also include an acceptance criteria to ensure that I and the team know what the feature should contain in order to be considered complete.

Conclusion

To improve on my Agile skills I will need to write better user stories, I will begin tomorrow (Wed 6th Oct) as I will need to work on a new feature for the Game. I should spend 2 – 5 minutes writing out who the user it, what they want (the feature), what needs to be in the feature to consider it done and add a time estimate for the development. Once done I should aim to continue this way writing every time I make a user story, likely 5 – 10 user stories each week.

Bibliography

KEITH, Clinton. 2010. Agile Game Development with Scrum. Boston, Mass. ;: Addison-Wesley.

MEASEY, Peter et al. 2015. Agile Foundations: Principles, Practices and Frameworks. Swindon, UNITED KINGDOM: BCS Learning & Development Limited.

WAKE, Bill. 2003. INVEST in Good Stories, and SMART Tasks – XP123.

Figure List

Figure 1: Keith Clint. 2010, pp. 126 – 127. A more negatable user story [Image]

Categories
Uncategorized

Working in a team

I’ve now been put in a team and have attempted to use the techniques to challenge my social anxiety to improve my communication. In addition, I intend to use this game jam as a way to practice my patience and planning skills, by designing the game mechanics with a flowchart before writing things in code.

When discussing my idea with the team I tried to talk slowly and look at each team member when talking.

We’ve decided to use my idea as a base and then find ways to combine everyone else’s ideas to help flesh out the design. I’m quite glad that they liked my idea as it has given me more confidence in both my game/technical design skills and my skills when working in teams.

As we’ve gone through the planning stages, I’ve had to be patient and try and not go immediately into building the game. This is especially the case as the team’s designer has been working on the narrative flow charts for the game. Once those have been finalised we will be using a third party asset called Fungus to create the dialogue system in Unity.

At the moment it doesn’t feel like I’m doing as much work compared to the reset of the team however, using Hope’s book on Cognitive Behaviour Therapy I’ve attempted to challenge my thoughts by reminded myself that I’ve done behind the scenes such as setting up the repo, the Kanban board and getting the Unity project ready (Hope et al. 2006).

In addition, communicating to the team has been mixed as though it feels like I haven’t been communicating very well, we have been making good progress on build the concept of the game.

In addition, I’ve been looking into how Fungus works and trying to see if I can reverse engineer how Katana Zero’s dialogue works. By this I mean words in the sentence change their colour and have visual affects to convey how the NPC feels and their tone of their voice (Wiltshire 2019). An example of the dialogue is show below.

(Wiltshire 2019)

Looking into Fungus’s documentation, I can change the font’s size and colour and cause the textbox to shake, this is shown below in fig 1.

Fig. 1: Oates 2021. example of Fungu’s changing dialogue

However, trying to recreate the effect where the words jitter is proving to be hard. Looking on their blog it shows you how to create a jitter effect on a specific word, but trying out their code shows no results, an example is shown below in fig 2 and 3.

Fig. 2: Oates 2021. example of the code for jittery words
Fig. 3: Oates 2021. example of jittery words not working

Due to the dates on their documentation it’s possible this solution is out of date and no alternatives have been shown, which has affected my hopes in using the effect in our team game (‘Blog’. sept 10, 20). Despite all this I will continue to look into this as I believe it will help add to the horror elements in the game.

Conclusion

In conclusion, I’ve continued to try out Hope’s techniques in dealing with my social anxiety, to help improve my communication, though it feels like some improvements have been made I should still continue, with the techniques for the rest of the week to see if more improvements can be made. The same can be said about my patience as it felt hard to stay still during the planning phase.

Next week we will be working on the game and will be using Fungus to create the dialogue system and mechanics, whether visual affects will be added to parts of the dialogue is unknown, however I will be continue researching tomorrow (Friday )to see if there is away to get them to work.

Bibliography

‘Blog’. sept 10, 20. Available at: https://fungusgames.com/blog. [Accessed Sep 30,].

WILTSHIRE, laex. 2019. ‘How Katana Zero Brought Action into Cutscenes’. Available at: https://www.rockpapershotgun.com/how-katana-zero-brought-action-into-cutscenes. [Accessed Sept 29].

HOPE, Debra A., Richard G. HEIMBERG and Cynthia L. TURK. 2006. Managing Social Anxiety: A Cognitive-Behavioral Therapy Approach Therapist Guide. Oxford University Press, Incorporated.

Figure list

Figure. 1: Max Oates. 2021. example of Fungu’s changing dialogue.

Figure. 2: Max Oates. 2021. example of the code for jittery words.

Figure. 3: Max Oates. 2021. example of jittery words not working.

Categories
Uncategorized

Character controller

Recently I’ve been in a pair and been busy creating a 3D version of MAD’s album cover for Our House in Unity and blender 3D.

I was was the one making the First Person character controller that the player can use to walk around the environment, how this time things where done a little different. Instead of using a Rigidbody component and controlling its physics through code, I decided to use Unity’s Character Controller component as unlike Rigibody it had the following benefits:

  • Slope limit – preventing the player from going up slopes above a certain angle.
  • isGrounded – a build method that can detect is the player is on the ground. This prevents me from having to manually build one myself.
  • Step Offset – allowing the player to automatically move to the top an object if it isn’t above the set height when compared to the player.
  • Built in collision detection – preventing me from needing to use Unity’s collision detection methods.

More information can be found here: https://docs.unity3d.com/Manual/class-CharacterController.html

Though I did need to program in my own gravity for the controller, it was a small price to pay as what could have taken over a day to fully build and polish was now accomplished in 10 minutes. A snippet of the code for the gravity and controls is shown below in fig1.

Fig. 1: Max Oates. ca. 2021. Example of character controller script.

Since the player can not jump then the downwards velocity is constantly set to -9.8 units.

There is also another problem with the Character Controller component as the Gameobject can’t rotate on the x or z axis, meaning if you want the player to simulate changes in gravity’s direction or have realistic falling over e.g. the collider rotate with the player’s mesh and Gameobject, then this can not be achieved. However, this wouldn’t affect the this project as these mechanics aren’t.

As mentioned earlier, I was quite new to the Character Controller, so I had to do some research to understand how to use it. fortunately, the video below in fig2 was quite helpful explaining how the component works.

Fig. 2: iHeartGameDev. ca. 2021. How to Move Characters in Unity 3D: Built-In Character Controller Explained [#1].

A challenge that I did have to overcome, was not from the movement side of the controller, but what to do with the mouse controls for the camera. Getting the player to rotate on the Y-axis with the mouse was fine, but trying to get the camera to rotate on its X-axis proved to be a challenge as I had to include limits to ensure the when looking upand down camera couldn’t be rotate into the body of the Gameobject. To make this work I used Mathf.Clamp to limit the rotation, however for unknown reasons when the camera reached the center of the screen it would instantly look down.

Eventually I simply added a variable that would increase or decrease by the the value given by the mouse’s y Axis. In this case the value was between 0.25 and 1 allowing for smooth rotation. I then used Mathf.Clamp to prevent the variable from going over 90 or under -45. A snipped of the code is shown below in fig3.

Fig. 3: Max Oates. ca. 2021. Example of camera rotation control script.

Conclusion

After using the Character Controller component for this project, I hope to use it more often for future projects. In addition, I could also spend an hour experimenting with manipulating the y axis of the player to simulate a jump and see how the Character Controller component works with root motion, allowing for more smooth and realistic movement and actions.

In addition, through out this week I will doing a game jam, which should give an opportunity to try out the mechanics mentioned above while also creating an enjoyable game in the process.

Bibliography

Character Controller – https://docs.unity3d.com/Manual/class-CharacterController.html

How to Move Characters in Unity 3D: Built-In Character Controller Explained [#1]

Figure List

Figure 1. Max Oates. ca. 2021. Example of character controller script [Digital image].

Figure 2. iHeartGameDev. ca. 2021. How to Move Characters in Unity 3D: Built-In Character [YouTube user-generated content]. Available at: https://www.youtube.com/watch?v=UUJMGQTT5ts&feature=emb_imp_woyt [accessed 27 September 2021]

Figure 3. Max Oates. ca. 2021. Example of camera rotation control script [Digital image].