Main image of Super-Five Senses Created by Combining and Utilizing Sensors and AI

Bringing humans and machines even closer with sensors and AI

Super-Five Senses Created by Combining and Utilizing Sensors and AI

The perceptual abilities to be aware of the state of the outside world are indispensable for all living things to keep on living. Humans gather information through the five senses of sight, hearing, taste, smell, and touch. We use that information to recognize and understand the situation surrounding us with our superior intelligence. We then perform advanced actions and judgments. Looking at it comprehensively, the perceptual abilities of humans are probably among the best in the living world.

However, if we focus on individual perceptual abilities, we find that there are many living things with abilities superior to those of humans. We are no match for the compound eyes of dragonflies when it comes to a sense of sight, whereby the movements of things are perceived in the surroundings. Furthermore, we can't compete with dogs in terms of our senses of hearing and smell. There are even living things that have the ability to capture information that cannot be detected by humans in the first place. Examples of these include migratory birds that determine their direction of flight by perceiving geomagnetism. If we stretch our imaginations further, there may even be perceptive abilities that capture information that living things in existence cannot detect.

If we could acquire perception beyond the five senses of humans with the assistance of machines, how would it change our lives and society? We introduce trends in the development and utilization of technologies that realize the super-five senses.

Extending Our Perceptual Abilities Utilizing Evolving Sensors and AI

Sensors are electronic devices with the function to acquire information from the outside world. There are already a wide variety of sensors that can reproduce the five senses of humans. Image sensors, microphones, accelerometers, and temperature sensors are examples of those. There are also many sensors that have performance that exceeds the abilities of humans. For instance, there are infrared ray image sensors capable of detecting the existence of things in the dark. Furthermore, there are also sensors that can acquire information that cannot be detected by humans. Examples include geomagnetic sensors and CO2 sensors.

Advances in sensor technologies have made it possible to realize the ability to gather information far beyond the abilities of humans. Nevertheless, even if the range of information we can obtain expands and diversifies, it does not mean it is possible to extend the perceptual abilities of humans. We cannot improve our perceptual abilities without also improving our ability to appropriately process the information we have collected at the same time. 

For example, facial recognition technology has come to be used for the personal verification of visitors when holding sports and other events. It has also become possible for visitors to gain admission to venues using a face pass without the need for a ticket by registering their image data in advance. Furthermore, in recent years, it has become possible to simultaneously recognize those who can enter and those who have not purchased a ticket by collectively recognizing images of crowds captured by cameras. This has made it possible to smoothly guide people into the venue without lines. In addition, technology has evolved to the point where it is possible to estimate the age and gender of visitors.

Image 1 of Super-Five Senses Created by Combining and Utilizing Sensors and AI

It can be said that such advanced perceptual abilities using machines are the super-five senses beyond the abilities of humans. However, in this example, the super-five senses have not been realized by an evolution in the performance of the sensors that collect the information; or, in other words, the performance of the cameras. The super-five senses have been realized by the evolution of the artificial intelligence (AI), which is utilized in the recognition processing of the acquired images. 

Detecting Machine Failures from Noises and Vibrations and Cancer from Exhaled Breath

The advanced perceptual abilities of humans are realized by combining the detection abilities of the sensory organs like the eyes and ears and the cognitive abilities of the brain (Fig. 1). In a similar scheme, the super-five senses will be realized by advancing sensors and AI and then utilizing those technologies in combination. We would now like to introduce just a few examples from among those of the super-five senses already being put into practical use or under development.

Image of The perceptual abilities of both humans and machines comprise a combination of detection (information gathering) and cognition (information processing) abilities
Fig. 1: The perceptual abilities of both humans and machines comprise a combination of detection (information gathering) and cognition (information processing) abilities

The super-five senses are already being utilized in many cases in the field of monozukuri. Some factories that produce industrial products employ experienced workers with the ability to sense signs of failure in devices. They do that by identifying the slightest abnormal sounds mixed in with the sounds of those devices operating. This is a personal skill acquired through experience over many years. It can be said that this is an advanced perceptual ability that regular people do not possess. However, high-precision accelerometers are nowadays being mounted to such devices. This has enabled the detection of signs of failure by utilizing AI from the slightest changes in the vibrations of those devices. Moreover, technologies are also being put into practical use that pinpoint the place where abnormal sounds are being emitted on images captured by cameras through catching the slightest abnormal sounds. This is an example of the realization of perceptual abilities not possessed by humans in terms of converting auditory information into visual information.

The super-five senses are also starting to be utilized in the field of medicine. A variety of information that reflects the physical and health condition of humans is contained in the things we excrete from our bodies. One of those is the air we exhale when we breathe. It is known that exhaled breath contains components that indicate signs of cancer, appearing as a change in odor. Nevertheless, the change in odor is extremely slight. Accordingly, it is difficult to detect such a change with the human sense of smell. However, there are cases of specially trained dogs called "cancer detecting dogs" being used to detect cancer. That is because dogs have a superior sense of smell that can discern such a change. Many companies and research institutions are carrying out R&D to mechanize this ability by combining a gas sensor that detects the components contained in exhaled breath with AI and other information processing systems. If such a function is installed on a smartphone where exhaled breath is blown during a phone call, it may be possible to achieve a state as though the user was undergoing cancer screening on a daily basis. In addition, attempts are also underway to analyze acetone contained in exhaled breath and to then utilize that analysis in lifestyle-related disease prevention, dieting, and other areas of health.

HSI That Finds Valuable Information from Invisible Light

Furthermore, technologies that acquire information that cannot be obtained from visible light are being developed and utilized. This is done by combining image sensors that detect light with wavelengths that cannot be detected by humans such as ultraviolet rays and infrared rays with AI and other advanced information processing technologies. This technology is called hyperspectral imaging.

It may be possible to acquire unexpectedly high-added-value information depending on the method of processing the information obtained from visible light as well. For instance, the nutrient of lycopene contained in tomatoes absorbs blue and green light. Therefore, tomatoes that contain lots of lycopene have a bright red appearance. In other words, we can detect nutrients from color. We can acquire even more diverse information without contact by also applying techniques with similar principles to light other than visible light.

For example, it is possible to identify or otherwise recognize the state of pollution in the land, atmosphere, and water and the types of trees growing on the surface from images taken with visible light and infrared rays from an artificial satellite. In addition, it is also possible to discover contamination by foreign substances or abnormal ingredients in food production lines and to evaluate the content of specific amino acids including oleic acid, which is an umami (savory) ingredient. We can also check the health condition of people based on changes in their complexion that cannot be detected by the human eye.

There is still room for significant advances in both sensors and AI. As a result, it seems there are infinite possibilities for what kind of super-five senses humans will acquire in the future.

Related articles