Interactive AR Stories - Senior Capstone Project and Paper

Download Alex's New Castle on the iOS app store!

You can download the e-book from the app if you do not have a physical story kit!

Highlights Video (full length demo below):

Summary

Alex’s New Castle is an interactive educational augmented reality children’s story that fosters creativity through activity based learning. The AR app paired with a physical story kit creates a captivating adventure for elementary school aged children and their parents that combines creativity and fun to give them the best possible learning experience.

Elevator Pitch

Enter the fairytale world of Alex the dragon where you can interact with a combination of augmented reality and physical objects and participate in activities to help Alex feel more comfortable in their new home. Make your own choices to progress in this choose your own adventure style story, while also learning topics in math, science and art from your experiences. Come play and see what you discover!

Abstract

Alex’s New Castle is an interactive educational augmented reality children’s story that fosters creativity through activity based learning. The AR app paired with a physical story kit creates a captivating adventure for elementary school aged children and their parents that combines creativity and fun to give them the best possible learning experience. A key feature of the story is the ability for the child to have autonomy. After entering the first room for the introduction it is completely up to the child where they want to go and what they want to do, or not. The rooms are laid out in the shape of a castle so that there is no specific order children are supposed to go in like when reading a normal book. Audio narration and subtitles help guide them to certain features that can help them progress further in the story, but there is no right and wrong way of doing things, they can go back to each room as many times as they want, and they can play as long as they like. Creative thinking is an essential skill for the children of today, however it is severely lacking in most current digital education. The activities in Alex’s New Castle are designed to teach academic concepts in a creative way, allowing for personal expression and catering to different ability levels. The children’s actions have an effect on their environment and they learn through their experiences rather than just hearing and memorizing. Parent-child collaboration is also important as the parent is there to help and encourage the child as they go through the activities, however they need to keep in mind the child’s autonomy to make the decisions about how they progress through the story and figure things out on their own. Alex’s New Castle focuses on topics in math, science and art, taken from a standard first grade curriculum. Learning takes place in the contexts of purchasing items from a market (learning about counting money), gardening (learning about plant growth), and painting (learning about color theory). Each activity happens in a different room of the castle, where the physical page for the room is scanned by the app and shows the corresponding activity in AR. Although the activities are separate, the storyline is intertwined with actions from one room affecting another, for example the child cannot buy items from the market until they have found the money in the treasure chest. There are only three main activities for now, however there is a lot of room for expansion. For example including other subjects as well as furthering activities in the existing rooms, like being able to buy more different types of seeds from the market to plant in the garden. Alex’s New Castle uses image target AR software through Vuforia but I am also currently experimenting with ARKit for the future. ARKit uses plane based AR which could provide a more stable and widely available experience. It would also allow the characters and scenes to appear life sized, which is what I originally planned but could not do due to the instability of the image target recognition.

User Research and Storyboarding

My capstone was split into two semesters with the first semester being used for research and the second semester for development.

During the research phase I decided on a target age range for Alex's New Castle (5 - 8 years) and studied types of educational apps they use and did a comparative audit with other current educational AR apps. I found a market gap in educational apps that focus on creative thinking and decided to use that as a central feature of my story.

I interviewed a children's book author for advice on how to write a captivating story and did background research on 1st grade curriculums before picking a few specific topics to focus on.

My research also explored parent-child interaction when using educational apps, as many parents are also involved in the process. This allowed me to consider aspects where the parent and child could work together in my story.

Full research paper:

HopeMyers-CapstonePaper.pdf

Download PDF • 1.20MB

One minute video summary:

3D Modeling

All the 3D models in Alex's New Castle were made by me. I used tilt brush, a VR painting app, to hand draw over 40 models. I chose to do this to make sure the models had a cohesive style that fit the vibe of my story.

Artwork

I drew the story art myself, using screenshots of the 3D models and adding backgrounds based on the setting for each page.

Fabrication

To make the physical pages I laser cut wood into shapes corresponding with the pictures. After printing the pictures, and used the laser cutter again to cut them out, I used mod podge to glue the pictures to the pages. I also printed out a paper mat with the pictures on it so that kids could place the pages down neatly. I also hammered on mini hinges to the castle doors so that they could be open and shut to display the inner picture. After making some tests and adjustments I made eight copies of the kits.

Complete Physical Kit - Box with contents including:

• 1 castle mat

• 5 castle pages with 6 scannable AR scenes (castle doors open to see another page inside)

• 1 wooden base, 2 dragons and 1 white board that can be put together to make a stand for the pages

• 1 mini treasure chest

• Fake coins (6 quarters, 10 dimes, 10 nickels, 10 pennies)

• 1 cloth money bag

Coding and App Development

I used the game engine Unity and programming language C# to code the Alex's New Castle app. After importing all the 3D models that I made in tilt brush I arranged each scene to look like the pictures on the pages. The Vuforia plugin for Unity allowed me to display different AR scenes by using the castle pages as image targets. I also had to alter the base Vuforia code to register how many times each image target has been recognized because some of the activities change depending on the amount of times you visit it (for example the plant grows each time you go back to the garden after watering it). I carefully considered all the possibilities when coding because since it is a choose your own adventure style story all the activities affect each other so the star stamp doesn't show up in the bedroom until you buy it from the market, and you can't buy things from the market until you get the money from the treasure chest. In addition I also had to add in many different audio narrations and subtitles that would play differently depending on the choices made. One of the hardest parts was making the painting activity so that the user can dip the brush in the paint buckets to change its color and having the dots and stars show up where they paint on the wall. The painting mechanics are still not the most user friendly and need to be improved in the future. Once all the activities functioned I created the user interface, adding instructions to help with kit set up in the beginning as well as pop up windows with specific instructions for some activities. After getting an apple developer license I learned how to go through the app publishing process. I had to optimize for mobile and tablet, write descriptions for the app store, a privacy policy, take screenshots and demos and submit it for apple testing and review before it was approved to be published.

User Testing

Midway through development I conducted a user testing session with five children between the ages of 6 and 9 and one of their parents. I found that my book was quite successful in the key areas outlined in my abstract and the kids gave me feedback about the usability and difficulties they had, which I improved upon as I continued development. Most importantly the kids enjoy the story and activities and had fun! I am hoping to do further user testing in the future.

Final Showcase

Our capstone final show was up for a week so in addition to presenting to visitors how my project worked, my main goal was for guests to sit down and actually play with it themselves. In the industrial style studio space I created a little area with a home play room aesthetic so that people could relax, take their time and put themselves in the mindset of how a child would play with Alex's New Castle.

Future Steps

Features to change:

• Experiment with non image-target based AR to increase stability of the models

• Simplify setup instructions (this will also be easier with a different method of AR)

• Rework user interaction with the painting activity

• Let the user click to control the subtitles instead of auto-playing. Improves reading comprehension and attention span

Features to add:

• Expand upon the current activities (ex. more items to buy in the market, being able to plant multiple types of seeds)

• Add activities for other educational subjects. Alex's New Castle currently focuses on math, science, and art. New activities could focus on reading, spelling, etc.

• Make a bilingual version of the story

Full Demo Video

Walk through of the entire story play process including set up and all activities

App Privacy Policy: No personal data is collected from the Alex's New Castle app