Miraisens transforms gaming: exciting haptics technology(2/3)

Virtual reality (VR) and 3D imaging technology have come to be used for games and other kinds of content. Even with the apparent improvement in the sense of realism compared to the flat visuals we have seen up until now, it can also be said that many find this technology awkward or unnatural. In part 2/3 of this series of articles, the developers themselves explain the chronology of the development of 3DHaptics. Through doing so, we also explore the merits of using haptics in presenting content and the principles that make realistic haptics possible.

From the left, Norio Nakamura, Miraisens founder and CTO, Senior Researcher at the National Institute of Advanced Industrial Science and Technology, and Natsuo Koda, Miraisens co-founder and CEO

The paradox in the fact that the more realistic the imagery, the less it is seen as realistic

--What kinds of effects that were not possible with conventional methods of content presentation appealing to the senses of sight and hearing can be expected with the use of haptics technology to appeal to the senses of touch and force (haptics feedback)?

Nakamura: Implementing haptics technology would allow us to provide the user with a more realistic experience. This technology is likely to be beneficial when used in not only video games but also in all of the different applications of VR technology. I’ll talk about how it would be beneficial as I explain what drove me to begin research on haptics technology.

The International Exposition, Tsukuba, Japan (Tsukuba Expo) was held in 1985. There, I was able to witness everyone from young children to the elderly experiencing exciting holographic effects for the first time. I remember seeing them reaching out into the air to touch what they believed they were seeing. It was then that I realized that the more realistic the visual and audio technology, the more people would instinctively want to try to touch what they are seeing.

However, in the case of holograms, even if you can see them, it’s obvious that you cannot touch them. In the real world, if you could see something, you would almost unmistakably be able to touch it. Realistic images look so real, it feels unnatural when you are not able to touch what you are seeing.

The challenge when it comes to VR/AR/digital content

Haptics sensation is a sense that is closely associated with the workings of the mind and body

--That is an unfortunate repercussion to what is supposed to be a technological advance.

Nakamura: It is. When the images and sound weren’t as realistic, I think your mind was able to identify what you’re seeing as something that is made-up, and hence, you experienced the content as so. That is why you don’t think to try to touch it. On the other hand, once you come out with visuals that are so realistic you can mistake what you see for the real thing, I think it becomes easier for you to feel let down when you can’t touch it, or feel like something is missing.

 Even if we can’t see it, we can sense the presence and state of an object through our sense of touch, such as feeling the clunking sensation when you unlock the door or the sense of distance you get when you reach out to touch something at a distance. It’s true that we get a lot of information through sight and sense of hearing, but in reality, we live our lives making full use of our other senses as well. What you feel through all of those senses combined is remembered as an experience.

--So the information our minds obtain through haptics perception is slightly different from the information obtained through our senses of sight and hearing.

Nakamura: In the course of biological evolution, the first senses we acquired were tactile, with sight and hearing coming much later. Haptics sensation, a combination of tactile sensation with the feeling of something physically there, is a very primitive and instinctive sensation that is closely associated with the workings of the mind and body.

For example, even newborn babies who have not yet acquired the sense of sight can learn to feel safe and learn how to interact with their surroundings through the haptics experience of skin-to-skin contact with their mothers. There was also a time when swimmers working towards a world record practiced with a rope tied to their body and did training while pulling at the rope to experience what it would feel like to swim at world-record-setting speeds. You feel the resistance and viscosity of the water at the speed you are aiming for, which you would not be able to experience on your own at this point, and through this, you learn how your body moves once it is able to adapt to this speed, and you learn how to move at your fastest in the process. The nature of the awakening of new emotions and abilities based on sensory and physical experiences is referred to as “human augmentation through sensation and physicality”.

Commentary from XEODesign's Nicole Lazzaro:

Haptics textures and surfaces create more believability. Haptics plays an important part in this evolution towards better storytelling and richer experiences.

With each jump in quality from 8-bit graphics to millions of colors to 4K resolution to photogrammetry scans there has been an increase in game quality and engagement. Haptics delivers new sensations to increase immersion and the sense of presence.

Next-generation haptics uses Linear Resonant Actuators. These LRA’s vibrate at numerous frequencies similar to how magnets on a speaker work. This provides next-generation haptics devices a broader range of frequencies and a larger design space of touch sensations to incorporate into gameplay. Instead of just on/off next-generation haptics creates a whole universe of tactile sensations in time with gameplay.

Haptics can be pre-recorded, rendered at run time, be imbedded in a surface, or activated as the result of gameplay. Tools like Miraisens provide an essential bridge in editing and redefining these experiences.

The nature of the brain’s susceptibility to illusion is used to make the user feel tactile sensations and textures that are not actually there

--It seems haptics sensory content presentations are going to become increasingly important. However, it is not easy to make a wide variety of haptics sensory experiences possible. For example, when recreating the rough texture of brick, you can’t very well have actual bricks available to create the haptics sensation.

Nakamura: That’s exactly it. That is exactly what we are able to achieve with our “illusionary haptics generation technology”, which is at the core of our 3DHaptics technology. The workings of the human brain can be manipulated using illusionary haptics generation technology to create similar haptics sensations without having to recreate the external factors to directly trigger the haptics sensation.

The principles of Miraisens’s 3DHaptics technology

Humans perceive the state of the outside world from within their brains. We collect information through our body, such as through our eyes, ears, and skin to comprehensively assess the state of our environment, but the information obtained by each of our sensory organs merely tells a part of the whole story. Because of this, the brain picks up on the missing information and fills in the inconsistencies in the information while subconsciously creating a model of the state of the outside world. For instance, if there is an inconsistency between what your right eye sees and what your left eye sees, your brain uses a multi-dimensional mental model to make sense of the inconsistency. However, this is all but a mental model that your brain is creating on its own accord, and does not necessarily represent the actual environment. This means there may be cases where the brain is mistaken about what you are seeing.

 When using haptics sensations as a medium for content presentation and communication, the importance doesn’t lie in the recreation of stimuli from the outside world. It is making it possible to replicate the mental models created by the brain that is key. Simply put, it is tricking your brain into thinking that it is feeling the haptics sensations you want to present by taking advantage of the brain’s tendency to get tricked by illusions.

--In what ways can the brain be tricked with Miraisens’s 3DHaptics technology to feel realistic sensations?

Koda: 3DHaptics technology stimulates the sensory receptors beneath our skin with complex patterns of vibratory stimulations generated by actuators. If the right vibration stimulus patterns are chosen, mysteriously enough, it can create an illusion in the brain, such as a force sensation of being pulled by a great force or the tactile sensation of touching the surface of something. I think it can really convince you that something is there. This means that you feel something that feels real but is completely different from the stimulation you were actually given. We take advantage of this phenomenon and control the stimulus patterns applied to the fingers to trick the user’s brain into any kind of haptics sensation. That is Miraisens’s 3DHaptics technology.

Commentary from XEODesign's Nicole Lazzaro:

Haptics adds a lot of value by stimulating the sense of touch and provides a whole new palette of creative expression and opportunities for immersion and interactive storytelling. Because the sense of touch is so primal, haptics can move players emotionally without them realizing it. The impact of haptics will be like that of color and of sound.

Next-generation haptics provide the player the sensation of moving through sand or water, the texture of brick and glass, responses to their actions such as drawing a bowstring or falling to the ground. Haptics is rapidly become a ubiquitous part of the game industry.

Next-generation haptics is a core value proposition. Developers delivering on the promise of next-generation haptics requires easy-to-use tools to record, edit, and integrate haptics files into games.If developers don’t support the advanced haptics features. 3DHaptics such as Miraisens goes beyond most next-generation solutions to increase immersion and deeper experiences in games even more.

--So this is technology that tricks the brain with an illusion rather than physically recreating external stimulations and applying them to the human body.

Koda: That’s right. That is the most crucial point. If we tune the stimulus patterns correctly, we can realistically simulate all kinds of haptics sensations, like the way pushing or pulling something, weight and presence feels, or the hard yet soft feel of something made of rubber when it is stretched, the texture of the material on the surface of something, the roughness of it, and so on.

Nakamura: Illusionary haptics generation technology made it possible to design haptics sensations and create a sense of substance or tactile sensation of an object that is not actually there. What’s more, using technologies like artificial intelligence (AI) technology would make it possible for us to clearly associate and categorize the sensations with emotions they evoke in the brain, such as the sensations that feel pleasant or unpleasant, or sensations that feel soothing. Implementation of technologies like this is sure to help us freely design not only the haptics sensations, but also the emotions and experiences we want to evoke in the user.

Articulating a clear realism through a game controller

--That sounds like a very attractive piece of technology for content creators and communication device developers. However, because this is stimulation that is applied to the fingers, it doesn’t sound like very strong stimulation. Is this really enough of a haptics stimulation to give users an immersive experience and convey to them a sense of realism?

Koda: That is another point in our technology. Haptics technology up until now required large, heavy equipment generating strong external stimulation to create powerful haptics sensations. Now with 3DHaptics technology, a small device like a gaming console controller can create the sensation of external stimulation as if it were being created on a large device. Of course, this also means we can build the equipment at a low cost. This benefit is what makes it possible for us to use this technology in consumer devices.

Problems with haptics up until now

Nakamura: When I first started researching haptics, we tested an approach where we recreated physical forces to make the user feel haptics sensations. However, the device we got to do that was inevitably large and heavy. Even when we take the prototype to equipment manufacturers, questions like, “This is interesting, but how small would you be able to make the device?” would always come up. Even I knew that if we don’t make it smaller, lighter, and consume less power, it wouldn’t be able to be put to practical use, so it was a shock.

 Since practical implementation wouldn’t be possible with that approach, I looked back on my original area of expertise in neuroscience and went back to the drawing board for a way to practically apply haptics. What I ended up coming to was the idea that we didn’t have to physically recreate external stimulations. Instead, we could just recreate the feeling of touch sensations in the brain. This turned out to be a new approach that no one has ever tried. This was the early 2000s.

A prototype of an ultra-compact actuator that can make the user feel two-dimensional haptics sensations

--This is a revolutionary technology that is conceptually different from any haptics technology we’ve seen. Presently, what level of content presentation is it capable of?

Koda: There is a lot that this confidential, but within the limits of what we can divulge, we can say that we’ve developed a system that can recreate two-dimensional sensations of force, pressure, and touch using an actuator slightly larger than a lump of sugar.

We actually would like to have already achieved a device that can allow the user to feel stimulations three-dimensionally from any direction in space, but in the present, the development of the mechanical structure for a device like this is still difficult. This is why we have consciously decided to work with fewer dimensions and begin with creating an actuator that can be mass produced. When worn on the hands, this device can enact forces that can move the hand forward, backward, to the left, to the right, or turn your hand in different directions. The feeling of something pushing back when you perform a pushing motion, the feeling of touching something at a distance, or the roughness or bumpiness of something when you trace your finger along its surface; these are all sensations that can already be recreated by this device.

But because three-dimensional content presentation is currently not possible, it cannot stimulate sensations for upward or downward motions. If 3D presentation becomes possible, the user would become able to feel the sensation of touching something three-dimensional. We’ll be working to make this technology available in the future.

Commentary from XEODesign's Nicole Lazzaro:

Haptics are an undiscovered country in games creation. Haptics provides a new tool pallet of options for game designers to describe surface textures and directional force pulling players more deeply into the fictional world. With greater immersion players feel more involved in the game and a greater sense of agency and emotion. I know for our studio, and me personally as a designer, it was an amazing experience to design tangible 3D holograms that you could touch in VR. Imagine being able to walk through a virtual forest and feel the leaves brushing your skin. With Miraisens haptics technology can perform a Kame Hame Ha (gather and fire a ball of energy) and feel the sensation in their hands. After that, playing other games without haptics felt flat.

Other Links

Related articles