Mobile 360° video is giving viewers a new perspective of what future videos can be like. But the technology is still in its infancy. By overcoming a few technical challenges, like how to create software that can detect gestures and motions with greater accuracy for a more seamless viewer interaction, 360° video has the potential to open the doors to new levels of engagement for app developers, media outlets, advertisers, and most importantly to the consumers.
Before we get into that, let’s cover some background. The appeal of 360° videos (like this one of a dinosaur or this one of the Golden Gate Bridge) is they put you smack in the center of the action. You’re in charge. You can pan around to look up, down, behind you, and discover new details every time you watch a video. The viewer is no longer a passive observer, but actively engaging with the medium.
If you’ve got a headset like the Google Cardboard or Oculus, you can engage by moving your head to pan a scene. If you don’t have a headset and you’re watching from a desktop, you can use your mouse to pan the virtual world. Or, if you are on a smartphone, you can simply rotate, tilt, or otherwise move your device to interact with the 3D world before you — all of that is powered by sensors in the mobile device.
Since this article about “lightweight VR” appeared in VentureBeat last year, 360° videos have gone mainstream. Now, social media titans YouTube and Facebook both offer capabilities to upload and share virtual reality videos with the world at large.
Anyone with the right equipment can create their own videos. You just need a spherical camera rig (outfitted with six or more cameras) to capture what is happening from all directions. Next, you need a software like Stitch that can turn the individual videos into one seamless, panoramic view. And don’t forget to add your own metadata. (Some cameras do this for you automatically, but not all.)
For advertisers, 360° video is a watershed opportunity. Over the years, consumers have been bombarded with so many banner ads, 2D videos, and the like that now it’s almost impossible to get through to them. Overcoming ad desensitization is a growing problem for marketers and 360° video may be the solution. The new medium has huge potential to breath life back into advertising through engagement. Because when a consumer engages with an ad, you know you’ve got their full attention.
Now the question for the advertiser becomes: How do you keep 360° videos interesting? How do you create new interactions that are quirky and stimulating enough to grab and hold a viewer’s attention? The answer: make those interactions application like. That requires overcoming a few major technical challenges.
Hurdle #1: Improving the framework for broad distribution
Right now, distribution on 360° video is not great, because the current formats are not designed for cross devices and browsers. As mentioned, the two leading ways to browse, discover, and view 360° video content are through YouTube and Facebook. Both have limitations. On a desktop computer, for example, only certain browsers support 360° videos. Chrome and Opera are compatible, but Safari and Firefox are not. The perspective controls via the browser (which involve clicking and dragging on the video while it’s active) are also less intuitive and immersive than the headset or motion controls using a mobile device.
Additionally, on a mobile device, you can only view 360° videos on Facebook and YouTube in-app. This makes it difficult to reach a broad range of viewers. It is very difficult to create an app that works perfectly on a vast range of devices and browsers — in fact, it’s impossible due to poor hardware and/or software quality. Apps are always limited to working well in some environments and screen sizes, and not so well in others.
Hurdle #2: Getting better data from motion sensors
When it comes to viewing on a mobile device, another challenge with interactive 360° video is making sure interactions (such as rotating the device, pitching the device, tilting the device, or walking with the device) are as smooth and as seamless as possible, with minimal latency. If you rotate your smartphone to view your virtual world from a different angle, you want the video to move as you move.
When you begin to interact with a 360° video on the level of a game, the quality of motion processing technology becomes paramount. The only way to obtain accurate orientation and motion measures are by combining outputs from at least three sensors: the accelerometer, the gyroscope and the compass. All three come innate in most modern mobile devices. The problem is these sensor readings are often inadequately accurate when used separately and result in poor user interactions.
Hurdle #3: Inventing cool new interactions to engage the viewer
One of the ways brands can make 360° videos more interesting to viewers is to incorporate creative and new types of engagements. For instance, a user might interact with objects in a 360° video that can allow the user to engage in a multitude of distinct scenes and different worlds, creating countless unique experiences. Or, brands can incorporate new types of motions in their experience so that, for example, taking steps in the physical world corresponds to the user stepping through the virtual world. Haptic or multi-dimensional feedback can also be used to guide the user to interact with a 360° video using a particular gesture or in another specified way. Or, the user might be able to interact with the video space using the Air Pencil. The 360° video might even give us a whole new social networking medium. The possibilities are endless and you can come up with lots of ideas just by paying attention to what’s happening in the mobile hardware and VR hardware space.
But the key to developing any new type interactions lies in tapping into the innate sensors in a mobile device to detect a viewer’s motion with a high degrees of accuracy—and then make sure the virtual world responds seamlessly to those actions. All of this requires high quality motion sensor processing framework, machine learning, sophisticated algorithms, and design that are all carefully tied into one beautiful user experience.
We are not there yet, but the future of 360° videos and immersive content is very promising. And it may bring a lot more inspiration to mobile applications and advertising by engaging with consumers in a way that pulls them in, instead of putting them off.
If you want to keep up with the latest developments, follow us on Twitter or send us an email to firstname.lastname@example.org .
PS: We are working on something called “360° Motion Video” and “Flightpath” —You’ll hear more from us on that soon!
What does the father of modern physics have to do with the tiny MEMS motion sensors in your smartphone? Plenty. A mathematician known for his pioneering observations of nature, Galileo (1564-1642) was the forefather of much of what we know about motion today.
The ultimate disrupter of his time, Galileo challenged Aristotelian theories people held true for centuries. For example, Aristotle claimed a rock fell to the ground because the two were made of the same element: earth. In contrast, Galileo studied quantifiable entities like time, distance and acceleration to explain what makes objects fall, break, and bend.
Galileo showed that force causes acceleration. On the basis of the law of parabolic fall, Galileo found that bodies fall at a constant acceleration, and that gravity is a constant force. In other words, a constant force does not lead to constant speed but to constant acceleration. He also developed the concept of inertia, which states an object in motion only stops due to friction. A hundred years later, Isaac Newton (1642-1726) built on these ideas to develop his first law of motion.
Later researchers expanded on the discoveries of Galileo—and Newton— to develop mechanical devices like the gyroscope and the accelerometer, instruments that play a critical role in navigation systems. The problem was these early mechanical devices were bulky and expensive. So scientist continued working to make them smaller and smaller.
Today gyroscopes and accelerometers have evolved into tiny devices called MEMS (micro-electro-mechanical systems). Built on silicon wafers alongside the circuits that control them, these are the smallest machines ever made.
It is pretty amazing to think that some of the MEMs motion sensors today are rooted in the principles of physics discovered centuries ago. Let’s take a look at four motion devices on the smartphone, how they work, and their history.
If you’ve ever played with a top, you know that a spinning top somehow has the power to stand upright. Without torque to change direction, a spinning wheel will always remain pointed in the same direction. A gyroscope is essentially a spinning top mounted on a gimbal, so the top’s axis is free to orient itself anyway it wants.
The first gyroscope was invented in 1852 by French physicist Leon Foucault to demonstrate the Earth’s rotation. Since our planet rotates, the gyroscope’s axis completes a full rotation once every 24 hours.
Electric motors in the 1800s made it possible for gyroscopes to spin indefinitely. And today, gyroscopes play an essential role in inertial guidance systems on ships and aircraft.
Many types of MEMS gyroscopes exist. Each type has some form of oscillating component for detecting directional change. All MEMs gyroscopes take advantage of the (Coriolis effect)[https://en.wikipedia.org/wiki/Coriolis_effect. And typically, the one on your smartphone is a three-axis gyroscope for measuring roll, pitch and yaw.
Have you ever wondered how your smartphone knows how you are holding it? It uses a device called an accelerometer, which measures g-force. The first accelerometer was invented by George Atwood in 1783.
If you want to know how an accelerometer works, picture a box with a metal ball inside suspended by a spring. If you move the box up, the ball lags behind, stretching the spring. By measuring the force on the spring, you can measure acceleration.
While the MEMs accelerometer in your phone is more complex than the simple ball and spring model, it uses the same fundamental principles. Inside the chip, engineers have created a tiny triple-axis accelerometer out of silicon that measures acceleration in the x, y, and z dimensions, so your phone always knows which way is down.
Used for navigation, the traditional compass has a magnetic needle that points to the North Pole. The earliest compasses were most likely invented by the Chinese in 206 BC and used for fortune telling. Later, compasses were used for navigation so sailing vessels could set their direction without having to rely on the stars.
The magnetometer sensor in your smartphone doesn’t use a magnetic needle. Instead, it uses an analog transducer to create a miniature Hall-effect sensor that detects the Earth’s magnetic field along the x, y and z axis.
The Hall-effect sensor produces voltage proportional to the strength and polarity of the magnetic field along the axis each sensor is directed. The voltage is converted to a digital signal representing the strength of the magnetic field.
The magnetometer is enclosed in a small electronic chip that often incorporates a three-axis accelerometer to determine which way is down.
The GPS receiver on your smartphone is a more modern invention—and technically it is not a sensor—but it works with the motion sensors in your device to more accurately determine movement and location.
Originally created by the US for military purposes, GPS (global positioning system) is a space-based navigation system made up of 24 satellites that circle the earth twice a day in a precise orbit. The satellites are spaced so that at least four are visible from any point on Earth at a given time.
Your GPS receiver requires data from at least four satellites to fix a position. It uses three satellites for trilateration to reduce your possible location to two points. The timing code from a fourth satellite is used to narrow down your location to one of those two points. (GPS satellites, by the way, tell very accurate time.)
Working together, these three MEMS sensors (gyroscope, accelerometer, compass) and the GPS receiver in your smartphone provide accurate data to determine navigation and a data on range of different user motions. Next time, you use your mobile device, you can thank Galileo, who opened up humankind to a new age of discovery and ultimately contributed to much of the smartphone’s innate intelligence.
Introducing Project Galileo.
The Adtile Technologies team is planning to release a cutting-edge motion processing dev kit (Project Galileo) that will allow developers to create new interactions and applications. Please sign up for more updates.
Is it possible to reimagine Picasso and Mili’s work with a smartphone? Light painting enters the 21st Century.
The year is 1949. In a darkened room in Vallauris, South of France, Pablo Picasso is working with a small electric light. The 68-year-old artist’s moves are swift and athletic as he draws lines, curves and shapes on a canvas of thin air. Those images of pure light were captured by the photographer Gjon Mili for Life Magazine.
Mili, an engineer by trade, made a name for himself with light paintings and photoflash photography. Previously, he had attached tiny flashlights to the boots of an ice-skater and recorded trailing lines of light as she waltzed through the air. He showed that work to Picasso who agreed to an experiment of combining art with Mili’s technical expertise.
So fascinated was Picasso by the results of their initial work together he agreed to five further sessions with Mili. The photos generated by those sessions equated to 30 images, all preserved by the camera. Featured were familiar Picasso motifs: bulls, centaurs and Greek profiles.
Capturing light and movement through art was nothing new. It has been repeated in various forms since 1889, but Picasso’s pictures, which appeared in Life, were iconic and transformative. (You can read more about the history of light painting photography here.)
At the time, Mili was using cutting edge equipment and technique to construct his photos. His work of capturing motion on film was a combination of high-tech, skill and finely honed talent.
Today, you do not have to be a Mili to create these types of images. Adtile Technologies, a company known for its sensor-enabled Motion Ads, has made it possible to capture motion with a smartphone so that anyone of any skill can create these highly technical images.
Introducing the Air Pencil
Today, we’re introducing something very cool that has never been done before—we call it the Air Pencil. Air Pencil (now in beta) that lets anyone capture freeform movement in space using their mobile device. How does it work? Think of your phone as Picasso’s electric light. Move, swing, glide the smartphone through the air and your motions will be captured, not on camera, but on screen.
Additionally, while Picasso and Mili’s light paintings were in 2D, Air Pencil lets you capture motion in 3D, as beautiful lines, curves and shapes you can literally move through and explore in ways never imagined possible in the time of Picasso.
Air Pencil is intuitively easy to use. It is a lightweight app that runs on a mobile web browser. Since it is a web app, the majority of the code powering it runs on a remote server. All you need to get started is a URL.
When you initiate the app, you’ll see a screen with a three-axis helper and a small red position indicator at the origin. On the bottom left and right of this screen are joysticks for panning and zooming. These items are all within the web-browser’s viewport, which you can think of as your virtual camera in the 3D space.
To begin recording your movements, simply press down on any point in the the screen and move your smartphone through the air. Release the screen to stop or pause your recording. Want to share your work? You can easily send a recorded 3D space file to anyone as a link in a text message or put it on social media.
As you can imagine, creating an app like Air Pencil requires serious technical engineering. You have to be able to capture three-dimensional motion with a high degree of accuracy, whether the user is swinging the phone in a large elaborate swirl or drawing a tiny circle.
To do this, the Air Pencil taps into a smartphone’s native micro-electro-mechanical systems (MEMS) — namely the three-axis magnetometer, three-axis accelerometer and three-axis gyroscope. It then calls on sophisticated machine algorithms to reliably infer the precise movements of the user based on sensor data.
Where would you use such an application? The answer: education, art, and collaboration. For example, you might use Air Pencil in a physics class to teach students about flight dynamics, or how objects move through space. As a collaboration tool, several users can draw independently with different colors and then combine those images on screen. Finally, the application has huge potential in the art world as the images you create can be shown on any size screen.
An art-inspired technology
Picasso and Mili were the inspiration for Air Pencil. Picasso, the father of modern art, liked to experiment with a plethora of media. And Mili brought a high tech element into that instinctive world or art.
The truth is, I wanted to go back in time and recreate the flashlight and camera technique with a phone and see what people would do with it. What kind of art will they create?
At a basic, intrinsic level, art inspires technical innovation. The two are inextricably combined. In fact, I got the inspiration for Adtile in 2013 when I was visiting the Alexander Calder exhibit at the LACMA. At the exhibit, quotes from the sculptor Calder lined the walls: “Just as one can compose colors, or forms, so one can compose motions.”
That was my aha moment. That quote influenced me to start working on sensory-enabled motions for mobile devices and turn Adtile into a motion technology company.
When you get down to it, art is about taking something technologically or emotionally complex and turning it into something simple, functional and beautiful. It’s also about making people smile. Or as Calder said, “Above all be happy.” And that is what I hope to do with Air Pencil, make people smile.
Smartphones have gone through an incredible evolution in the last decade. We are moving to an era where our smartphones are becoming more like personal assistants, monitoring our behavior, tracking our movements and anticipating our needs. A large part of this evolution is enabled by sensor technology.
Sensors bring intelligence and awareness to our smartphones. Today’s mobile devices are packed with nearly 14 sensors that produce raw data on motion, location and the environment around us. This is made possible by the use of micro-electromechanical systems (MEMS). MEMS are mechanical systems built into tiny semiconductor chips.
Let’s take a look at some of the major sensors in the typical smartphone.
Magnetometer and GPS
Your smartphone comes equipped with a magnetometer, otherwise known as a compass. With its ability to sense magnetic fields, this MEMS device detects compass heading relative to the Earth’s magnetic north pole. In conjunction with GPS, it determines your phone’s location. GPS is another type of sensor in your mobile device. It relies on satellites to determine location. Originally developed for the military, GPS was made available for everyone in the 1980s.
A three-axis gyroscope determines if your device is twisted in any direction. Using rotational force it measures angular velocity around three axes. The absolute orientation of your phone, represented as the angles yaw, pitch, and roll, is detected by a combination of the accelerometer, compass, and gyroscope.
A three-axis accelerometer in your smartphone reports on how fast your phone is moving in any given linear direction. The accelerometer has the ability to detect gravity as a static acceleration as well as dynamic acceleration applied to the phone. There are various types of MEMS accelerometer hardware available, such as microscopic piezoelectric crystals that change voltage under stress when vibrations occur, or differential capacitance caused by the movement of a silicon structure. The magnetometer, GPS, gyroscope and accelerometer on your phone all work together to create the perfect navigation system.
Comprised of an infrared LED and an IR light detector, a proximity sensor detects how close the phone is to an outside object, such as your ear. This sensing is done to reduce display power consumption while you’re on a call by turning off the LCD backlight. It also disables the touch screen to avoid inadvertent touches by the cheek.
More advanced smartphones have a chip that can detect atmospheric pressure. But to use it, the phone needs to pull down local weather data for a baseline figure on barometric pressure. What’s more, conditions inside a building, such as heating or air-conditioning flows, can affect the sensor’s accuracy. Barometers are best used in combination with other tools, including GPS, Wi-Fi and beacons.
Your smartphone also has an ambient light sensor to adjust brightness levels in dark environments. A fingerprint sensor can enable secure device and website authentication as well as mobile payment. Add to that list, microphone and camera sensors. Samsung’s Galaxy S6 even has an integrated heart rate monitor.
Sensors raise the consciousness of our smartphones. With mobile sensors becoming smaller and more sophisticated—and new types of sensors coming onto market—what we’re seeing today is only the beginning in the era of self-aware devices. More is waiting around the corner.
Machine learning will play a potent role in the future of mobile devices
As we talked about in a previous blog post, your smartphone is replete with dozens of sensors that collect all kinds of information on three dimensional device movement, positioning and the outside environment. But most of the data those sensors collect comes in a raw form. It has no practical meaning on its own. And that is where machine learning steps in.
What exactly is machine learning? The field is immense with lots of different categories and subdivisions. But let’s start with a common, layman’s definition: Machine learning is a discipline of artificial intelligence that focuses on the development of algorithms that learn from and make decisions based on data.
Or, as machine learning pioneer Arthur Samuel defined it, machine learning is a “field of study that gives computers the ability to learn without being explicitly programmed.”
You may not be aware of it, but almost everything that happens online is driven by a type of machine learning algorithm. When you do a search, machine learning chooses the results you get. Amazon uses machine learning to recommend products. Netflix uses it to recommend movies. And Facebook and Twitter use it to choose which posts to show you.
Additionally, both Google Now and Siri rely on machine learning to recognize speech input and respond quickly to user commands. And facial recognition software Affectiva also uses machine learning.
Machine learning plays a big part in how intelligent our mobile devices are and how they interact with us. We believe machine learning will play an even bigger role in the future of mobile devices — and that is some of what we are working on at Adtile.
As machine learning algorithms become increasingly sophisticated, they will change how our mobile devices interact with us — our mobile devices will recognize gesture, motions and movement to higher degrees of accuracy and respond to our needs in ways we never imagined possible.
Machine learning algorithms have the potential to give personality to our mobile apps and devices. In future blog posts, we’ll talk more about how machine learning and sensors are working together to improve people’s lives, their health and their mobile experiences.
I get a lot of questions about how Adtile Motion Store differs from HTML5 ad builders, so I thought I’d address those in a blog post.
If you’re not already familiar, an HTML5 ad builder is for creating display or rich media cross-screen ads — the static, animated, and video ads you see in mobile apps and Web browsing. You assemble the ad by dragging and dropping simple shapes, imported media, and HTML widgets and gizmos onto a blank canvas.
Now let’s address the Motion Store, why it’s a completely different animal, and why sensory Motion Ads are the future of mobile advertising.
Tapping into sensors
HTML5 ad builders are great for building standard rich media ads that run across screens, on desktop and mobile. But if you want to create a sensor-enabled ad specifically built for the mobile environment, look to the Motion Store.
The Motion Store is a do-it-yourself platform that makes it possible for anyone, regardless of their programming ability or design knowledge, to build an elegant, sophisticated Motion Ad.
A Motion Ad behaves more like a game or mini app than an ad. It does so by leveraging the various motion and location sensors within a mobile device to accurately detect a user’s motions.
When the user moves, whether to push, pull, rotate, twist, shake or tilt his or her smartphone or tablet, the ad responds in real-time, creating a coordinated seamless interaction that is completely intuitive to the user.
Adtile templates ensure high quality UX
With an HTML5 ad builder, you drag and drop objects. The Motion Store, on the other hand, brings an ‘app-store’ style approach to ad building. Only instead of selecting an app, you’re selecting a sensory Motion Ad template.
So far we have over 200 templates in our Motion Store, and we are adding new ones all the time. Each template is a fully coded, fully designed experience, one built on the Adtile Motion Framework that follows our proprietary design language.
You can customize a template with your own creative and storytelling to create a brand experience unlike anything else that’s out there on mobile.
Adtile templates offer several advantages. They save time by eliminating the countless hours of iteration and experimentation typically associated with complex ad building. We only provide proven templates that we have tested live multiple times and they capture strict design constraints to ensure the highest quality user experiences.
The numbers speak for themselves. Adtile Motion Ads generate on average over 30 percent engagement rates, more than 24 second participation rates, and click-through rates of more than 6 percent.
Built for a fast-paced programmatic world
HTML5 ad builders are useful for creating simple rich media ads but if you want to add device specific experiences, like animation, you quickly get into custom scripts. And, because HTML5 ad builders aren’t supported by a sensory framework or design guides, adding any type of even primitive motion detection abilities requires extensive manual coding with poor results.
Due to this, building a complex experience with an HTML5 ad builder can take weeks of back and forth and testing, driving up costs.
In contrast, the Motion Store lets anyone create a sophisticated Motion Ad within minutes. Once you’ve finalized your design, the platform generates a responsive HTML5 build for iOS and Android. You can request standard mobile Web or MRAID tags for your ad server, and analytics are baked right in.
True native experience
While HTML5 ad builders let you create rich media ads, Motion Ads are truly native.
Native ads blend in with the organic experience of the app or Web page they appear on but Motion Ads are also built on a native mobile framework.
The Adtile Motion framework takes advantage of the innate sensor technology found in all smartphones and tablets. It incorporates platform agnostic native code and application agnostic design and it delivers a responsive storytelling experience for both iOS and Android.
One or two of those elements alone would contribute towards generating a successful cross-screen ad. But the Adtile framework brings together all four to create a delightful and highly personalized experience for the user.
Solving the entire problem
You’d also have to grasp iOS and the vast Android device and browser landscape. You’d have to hire a unique mix of software engineers, product people, technologists, designers and mathematicians for the undertaking. And even that won’t guarantee a workable solution.
Fortunately, the Motion Store solves the entire problem for you. For the first time ever, building sensor-enabled mobile creative is seamless, simple to execute, and completely self-service.
Keeping it simple
The reason why Motion Store works and Motion Ads are so successful is due to their simplicity. We strip everything down to create a minimalistic, straightforward approach that makes sense to both the brand creating the ad and the consumer engaging with it.
We realize you can’t remove everything from the brand experience and some Motion Ad designs are inherently more complex than others but we strive always to follow the advice of Albert Einstein who said, “Make everything as simple as possible but no simpler.”
If you haven’t already, we encourage you to check out our Motion Store and find out why simple makes so much sense.