When you are faced with something entirely new, one of the best ways to explore is through play. The power of play came to mind when our friends at Google brought us a new piece of technology inside a prototype of the Pixel 4 smartphone. They showed us a new way that people could interact with their phones, not with touch but with gestures in the air. We saw an opportunity to create a new paradigm for mobile interactions, and we were inspired by how we could combine playfulness and UX design to make this breakthrough technology both enjoyable and useful.
This new technology, named Soli, had been in R&D for a while but now it would be in the hands of consumers, providing a new way to interact with their phones. Google asked us to explore what a gaming experience could look like and how to help people become familiar with a new method of interaction. Faced with a novel format, we were keen to find a fresh solution that would create something very playful.
In this article, I’ll share how we conceived Headed South as a magical journey, one where you are part of a collective journey, scouting out new landscapes from forests to a volcano, drawing the user in with brightly colored characters and scenes, and combining screen contact and gestures as a new way to navigate the mysterious landscape.
Fundamental Experiments
From the beginning, we understood that the only way to tackle this challenge was to have designers and developers working closely together in a very open and explorative process, all tailored to gather insights from users.
We knew that the device would launch with two main interaction inputs: directional and omni-based swipe gestures, read by radar and performed in mid-air, above the phone. It felt a lot like magic and so we thought about ways to capture that magical feeling in a gaming experience.
We came up with 27 different prototypes that explored different game mechanics. In these early prototypes we worked out how you could jump, shoot, slap, fly, freeze time, solve puzzles, move a character, select things, move objects around, push, pull and rotate objects. That’s a lot of moving around and many possibilities for play.
Gesturing above your phone is a “bigger” interaction than the familiar act of tapping on your screen. We knew we needed to find a core game mechanic that felt perfect before exploring the concept that would tie it all together, so we set our focus on three main ideas -
Interacting with a digital character, like a digital pet by using different gestures.
Solving puzzles through gestures that control certain superpowers like slowing down time.
Gesturing to manipulate the wind and control a kite, flying through in an open landscape.
Flying through a beautiful space
In our playtesting, the kite mechanic was the most engaging and aligned with work already done by Google to create gestures that felt airy and smooth. To make the gesture-based interaction feel right, we used a combination of visual, sound and haptic feedback all at once, and then addressed the challenge of making that feedback return a sense of control and power without being overwhelming.
Alongside the mechanics, we were also developing design concepts that worked with this sense of movement. We started designing different landscapes to create a sense of progression while flying through the world. You could fly through forests, follow rapids down waterfalls, fly through canyons with caves, and through tropical jungles with volcanoes. We wanted to create a style that was between something realistic and abstract. It should feel like nature but with a fantastical flavor to it. Our main inspiration sources were a combination of real nature photography and art portraying nature. One big art influence was how painterly brush strokes in textures can create a sense of movement in the landscapes around you, as you fly through them.
From here we landed on the design concept of migrating birds and the journey south. We were conscious this was an experience that needed universal appeal, designed for a broad audience. The nature-based theme felt right as something inclusive that defines the meaning of a game in terms of play and navigation. This is where Headed South was born.
As players explore the game they can learn tricks and new gestures as they fly with different flocks of birds. This helped to create a real sense of satisfaction through play; learning to do a barrel roll by gesturing in the air feels more physically connected than endlessly tapping at a screen.
The free soaring and flight expression is controlled in a new way, combining screen contact and gestures that is matched by brightly colored scenes. Only with design and engineering working in tandem could we manage to blend technology to create something as appealing and inclusive each step of the way.
Creating Headed South became an exploration in game design and a study of the potential of an emerging technology. It gracefully introduces people to a new way of interacting with a smartphone, while not focusing on the technology itself. There is something fundamental about physical play and digital gaming that this new technology enables. Instead of relying on binary taps on a screen, we can now create experiences that react and connect to physical gestures. Investigating this field has made it possible to create a more direct and fluent relationship between physical and digital worlds.