Miraisens transforms gaming: exciting haptics technology(3/3)
No matter how great the potential of a technology, it is meaningless unless it can actually be applied and used over a broad range of applications. There are many hurdles to overcome before we can practically apply 3DHaptics technology. In part 3/3 of this series of articles, we discuss the development of the environments required for haptics sensations to be articulated precisely, hardware that allows for stimulation that can be felt by the brain, in addition to the future applications of 3DHaptics technology that are in demand by creators.
Breaking down the different kinds of haptics sensations we want to convey into three components: pressure, touch, and force
--So 3DHaptics is a completely unique technology that takes advantage of the brain’s susceptibility to illusion to trick the brain into feeling a variety of realistic sensations. However, content creators currently do not have the knowledge, methodology, or skill they need to manipulate haptics sensation content as they wish. There is also the wide gap between the haptics sensation you want to create and the actual stimulus patterns you need to trick the brain into feeling that sensation, so creators may face difficulty when trying to produce the correct patterns.
Nakamura: We will need to solve that before 3DHaptics can be practically implemented and widely used. There are so many different haptics sensations that humans feel, which means there is also a great deal that we will need to recreate. In order to convert these sensations into digital data and make it possible to process and store this data and circulate it to others, we will need to figure out the basic components that make up each sensation.
The wide range of colors in displays are produced through the appropriate superimposition of three primary colors: red, green, and blue. In the same way, Miraisens proposed the three primary senses that make up touch sensations based on perception and cognition in the brain: the sense of pressure, which determines the hardness and softness of something; the sense of touch, which determines the material of an object’s surface; and the sense of force, which gives the sensation of being pushed or pulled. These three primary senses are combined in different ways to recreate all kinds of tactile sensations. When designing haptics sensations, we are putting these three primary senses together in different ways to get the specific sensation that we need.
Getting an easy-to-use development environment ready for creators
--So first off, you have reinterpreted a variety of tactile sensations and converted them into a form that is easy to process digitally. Is there any kind of a development tool?
Koda: We are preparing a tool for processing haptics sensations. We are preparing to distribute a tool to clients that allows them to easily process haptics sensations. This tool uses sound effects used to add intensity and realism to visual content as a reference.
Through a specification of the intuitive parameters for the sensations of being pulled forward or pushed backward, or the tactile sensations of touching something slightly rough or uneven, this tool instantly creates the optimal waveform patterns that need to be applied to the fingers to recreate the desired sensations. Tools such as video editing tools are used in video production to easily create the exact desired visual content. I think it’d be easy to picture if you think of our tool as the haptics version of that.
--So you are preparing a development tool from the perspective of the user.
Koda: There is additionally the need to optimize the wave patterns for the vibrations being applied to the user’s fingers for the shapes and specifications of devices such as game controllers. The mechanism that generates the stimulus patterns applied to the fingers is like a kind of acoustic device. Depending on the device on which the mechanism is being used, you need to tune the driving waveform to suit that device, just like how you would tune a violin. This requires the skill of an expert, and in some cases it can take weeks for even a Miraisens engineer to complete this “tuning” by hand. It would be a near impossible prospect if we left the tuning to the client.
That is where we decided to convert the know-how that Nakamura has accumulated into a database and created a system based on AI technology. By simply entering the shape of the device in which the actuator is being inserted or the way the user will be holding the device, the best way to adjust the waveform can be calculated in just a few seconds. Even in the case that a device’s specifications change, by simply looking up the optimal points to tune the waveform to while readjusting the parameters, the waveform can be quickly retuned.
Commentary from XEODesign's Nicole Lazzaro:
Delivering next-generation haptics experiences requires practical and flexible tools. These haptics tools need to provide efficient ways to make multiple small adjustments to expand the creative expression of designers. To deliver the promise of next-generation haptics, developers need tools to flexibly iterate on the haptics sensations. They need the ability to perform, record, manipulate, layer, and mix vibrations to create complex and interesting haptics sensations that coordinate with what players are doing and see on screen. Haptics designers of the future will have large libraries of haptics textures just as sound engineers have large libraries of sound effects and filters to manipulate them. To get there, haptics designers need editing tools that help them visualize and manipulate the sensations, textures, and directional force. These tools need both a time component as well as an interactive component added on to the textures themselves. The right tools can improve expression and ease of use allows more iteration and for developers to create more nuanced and immersive haptics experiences.
--So you are creating a system that will make the work much easier for equipment manufacturers and creators using your haptics technology.
Koda: We’re putting mechanisms into place to allow creators to focus on content creation and not worry about the technical haptics part of the process. I hear that even for sound effect synthesis about 20 years ago, people had to work incredibly hard just to create the sound effects they needed on a synthesizer. I try to consider this kind of content creation experience when implementing these new methods of content presentation we call haptics so that we do not make any of the same mistakes.
The new actuator that intricately applies all kinds of stimulus patterns to the fingers
--There are already gaming consoles that have a device built in that generates vibrations in the controller. What do devices like this need to apply stimulations with 3DHaptics technology to trick the brain?
Koda: The vibration-generating actuators used in game console controllers up until now utilize a linear resonant actuator (LRA), which can only produce simple vibrations. “Resonant” describes something that utilizes resonance. Resonance can produce a large output of certain frequencies of vibration, but cannot effectively generate vibrations at other frequencies. For this reason, the haptics sensations this type of actuator could create could only be rhythmic, like Morse code. The LRA cannot be used to trick the human brain with 3DHaptics technology because it is like a drum if we were to compare it to a musical instrument.
Nakamura: In order to make 3DHaptics technology a reality, we need the more advanced “wide-range linear actuators” (WLA) (as defined by Miraisens) that can output vibrations in a wide range of frequencies. We would need a means of creating more complex stimulus patterns.
If we were to improve the ability of even a visual content display to output more high quality and expressive content, it would be necessary to improve not only the size and brightness of the screen, but also the resolution, color count, contrast ratio, dynamic range, and response time. In order to make more advanced haptics a reality, the same would be required of actuators.
--Has there been progress in the development of the ideal actuator for implementing 3DHaptics?
Nakamura: Actuators are evolving by the minute. What we used at first was an eccentric motor that could easily produce large amplitude vibrations. However, the vibrations generated were unnatural, so we decided to switch to the linear actuators we are currently using. WLAs are also evolving, for instance, through the implementation of piezoelectric elements to become more compact.
--Is Miraisens also working on developing actuators?
Nakamura: While we originally also worked on hardware development, because our core technology is algorithm development, it goes without saying that we were limited in our ability to develop technology having to do with hardware. That was where we teamed up with Murata, whose strength lies in hardware development, to apply Miraisens’s years of expertise in developing new actuators with specifications optimized for haptics.
Establishing an “ecosystem” to make way for implementation and application
--Haptics is a brand-new frontier with technology that is different from that of visual or audio content. We will need to continue to both get it ready to be put it into practical use and make way for its development. So it will be essential to prepare and put into place an “ecosystem” of development tools, actuators, etc., for utilizing the technology.
Koda: That’s right. Miraisens has approximately 40 patents filed worldwide on 3DHaptics technology and technology for applying 3DHaptics, 20 of which are registered. What’s more, most of them are patents for core technology that can be called basic patents. These encompass all the patents for the technology needed to make it possible to practically apply haptics technology that utilizes illusionary haptics. We are bringing together Miraisens’s technological developments and the technology in which Murata’s strengths lie in addition to collaborating and brainstorming to develop the ecosystem we need in the best possible manner.
--In what industry is Miraisens working towards applying haptics technology first?
Koda: We are thinking to implement haptics in the entertainment industry first. We hope to provide our support in the production of content that makes use of game console, VR, and AR technology. That is where we plan to take advantage of Murata’s strengths in hardware to develop solutions for game console and equipment manufacturers, and make them available in a few years. The development tool for applying illusionary haptics is nearly complete. We are also preparing a hardware development prototype kit consisting of actuators, control boards, different kinds of firmware, etc.
Wide range of applications, including the remote sharing of spaces and remote surgery
--Going forward, what areas in particular do you see making progress towards effective implementation?
Koda: Haptics technology conveys information including the atmosphere of a space and the sense of distance between people that cannot be fully conveyed through visuals and sound alone. This can be used to create an innovation in telecommunications, following in the footsteps of the telephone and television, that can be referred to as “telespace”. We are already working together with companies around the world and taking steps toward applying this technology. We also believe this can be implemented in HMIs for in-vehicle devices and smart homes.
Additionally--and this has been Nakamura’s wish for some time now--we are also considering its uses in the medical field. The use of haptics would be extremely effective in making safe and precise remote surgeries possible and allowing physicians to practice performing surgery. What can be felt through the hands during surgery is said to be extremely important for genius doctors referred to as “god’s hands” in Japan. Yet, telesurgery systems that are currently in place do not provide any tactile feedback, so those using them are forced to rely solely on visual information during surgeries. Haptics will allow for remote surgeries to take full advantage of the skills of master doctors.
It would also be possible to use haptics in the remote operation of construction equipment. Workers operating bulldozers and other construction equipment determine how much soil they dug up and so on through the feel of the control stick or other aspects of operation. Implementing haptics to unmanned construction equipment could make it possible to, say, perform construction work remotely in a VR environment from an operations center in Tokyo.
--What kind of value is Miraisens ultimately looking to bring to the world as a company?
Koda: If I were to just talk about my personal aspiration, it would be playing a role in establishing haptics as a conventional means of presenting content in the content creation industry. I hope we can lead the way in opening the door to a new world of haptics content.
We are also convinced that haptics will be the technology that will bring about the new paradigm shift in how information is transmitted, following in the footsteps of other technological shifts through the spread of audio communication with the invention of the telephone and the spread of visual communication with the invention of the television. If we can use haptics to transmit information such as the feel of a space and the sense of distance between people, all on top of transmitting visual and audio information, then we can make the transmission of the actual state of a place from a distance, or “telespace” technology, possible. It would feel as if someone who is somewhere far away is right next to you, and you can shake hands with them and physically feel the place at which they are located as you communicate. We believe this will be the third big revolution in communication.
The fact that Miraisens is part of the Murata Group will certainly work to our advantage as our groundwork to guide us as we go on to provide world-leading technology. Our goal is to become the world’s leading company in human-machine interfaces, with a focus on a certain means of presenting content that unlocks new value: haptics.
Commentary from XEODesign's Nicole Lazzaro:
I am looking forward to continuing to work with Miraisens technology as it evolves. Currently the market leader in directional force and surface textures, I expect even more complex haptics sensations and streamlined editing. Next, I’d like to see Miraisens offer haptics recording and performance tools, stereo multilayer editing, a haptics library, filters to annotate the haptics files for more accurate placement in games as well as comment notes.
What I’m especially excited about is next-generation Miraisens that defines the PBR for haptics. In graphics we use PBR (Physically Based Rendering) that models the flow of light in the physical world for more realistic surface textures and lighting. Miraisens is taking a similar approach for haptics. I’m looking forward to using it when it gets there!
Imagine being in an adventure game. Pull on a canvas-bound lever covered with dried blood and feel the weight of the enormous iron chain as it lowers the drawbridge. Feel as well as hear the deep rumble as the gears turn and the bridge jerks and lowers into place. Knock on the palace gate and it feels like rusted iron. Brush the dust and leaves off the ancient inscriptions inside on the palace walls and feel the different textures. Locate and open the leather-bound book and turn the pages to find the code. Feel the weight and turn of the tumblers as you solve the codex unlocking the doors to the inner sanctuary. Feel the shape and facets of the idol’s enormous ruby and the intricate filigree of its setting. Plunge your hands into the treasure chest and feel the smooth surfaces of the gold coins as they slip against each other and fall through your fingers as you stuff your pockets full of treasure. Feel the whoosh of the fins of an arrow as it moves past barely missing you as you make your escape.
All of these are possible with next-generation haptics. Can’t wait to feel what Miraisens delivers next.
Haptics is uncharted frontier with significant potential
The implementation of the fifth-generation mobile communication system (5G) has made it possible to exchange vast amounts of data over distances between many terminals at a low latency. In turn, this has allowed us to communicate through high-definition video, audio, and even VR. If we combine all this with 3DHaptics, we will discover entirely new ways of communicating with each other without feeling like there is a distance between you and the person at the other end of the line.
Haptics technology for recreating haptics sensations is still in its infancy, and there is great potential there. We also predict that it will have an extremely wide range of applications. It can be said that the foundation for the future of content presentation and communication is currently being developed at Miraisens. Meanwhile, electronics manufacturers and content creators are getting ready to start taking advantage of this technology to make innovations with it at the earliest opportunity.
Other Links
- Notice of Acquisition of MIRAISENS, a 3DHaptics Company “Haptic Applications for Various Production Sectors”
- MIRAISENS website