Making Use of Sensor Fusion in Smart Factories

03/28/2024

What is sensor fusion?

The term “sensor fusion” refers to processing technology that fuses (combines) data collected from multiple sensors in order to extract more useful information than can be obtained from a single sensor.
Some familiar examples of the use of sensor fusion include automobile rearview monitors. The sensors in this case are a camera and millimeter wave radar detector mounted on the rear of the vehicle. The camera captures a rearview image, and the millimeter-wave radar detector measures distance behind the vehicle. The image data captured by the camera and distance data from the millimeter wave radar detector are then combined and output to a monitor inside the vehicle, enabling safe and accurate parking.
But how is sensor fusion utilized in smart factories? Below we explain why sensor fusion is essential to smart factories and present some usage examples.

Image of automobile rearview monitors — Figure 1

What about sensor fusion in smart factories?

The wave of sensor fusion has now reached smart factories deploying Industrial Internet of Things (IIoT) technology. Today, smart factories embracing IIoT are overflowing with cameras to capture images, and sensors to measure properties such as temperature, humidity, angle, acceleration, current, and voltage. The data collected by sensors is output to monitors. Action is taken after determining the situation from the values displayed on the monitors. But this way of doing things requires knowledge about the values displayed, and the transition to action takes time.

Image of Before implementing sensor fusion — Figure 2: Before implementing sensor fusion

Sensor fusion is technology that combines data output from many sensors and outputs the results as information that is easy for people to understand. This makes it possible to understand the status of equipment and devices quickly and accurately.
What’s more, by collecting data continuously and accumulating it, the resulting data can be collated and analyzed. This brings substantial advantages in areas such as monitoring the operating rate of equipment, warning of impending failures, warning of impending defects, and tracking improvements in the operating environment.

Image of After implementing sensor fusion — Figure 3: After implementing sensor fusion

Examples of utilization of sensor fusion in smart factories

Here we present some examples of the implementation of sensor fusion in smart factories where adoption of IIoT has advanced substantially.

Boosting productivity by optimizing takt time*

Raising production efficiency demands the optimization of operations and processes corresponding to a variety of types of processing. This requires monitoring the operating status of the various robots and automated machines corresponding to the “work” to be processed, which comprises materials and components with different material properties and configurations, monitoring the work status of the operators, and monitoring the flow of products, as well as appropriate layout.
To accomplish this, a camera might be mounted on the end of a robotic arm and an accelerometer might be installed on the arm, for example. The camera would collect data showing the position and orientation of the work, and this would be collated with operational speed data from the accelerometer. This would make it possible to guide the robotic arm to the operation point precisely and in the minimum time possible, and also to calculate the optimal takt time.

* Takt time: Required product assembly duration

Image of Boosting productivity by optimizing takt time — Figure 4

Detecting equipment failures early on: Implementing predictive maintenance

To implement predictive maintenance, it is essential to constantly collect data on vibrations, sounds, heat, etc., produced by the equipment, using sensors for vibration, sound, temperature, and the like, and then to use this data for comparison and time series analysis.
In predictive maintenance using sensor fusion for a motor, say, data not only from a vibration sensor, but also from temperature, current, and voltage sensors, might be combined and analyzed. This would permit early detection of abnormalities related to vibration, rotational, speed, etc., that could not be determined using only a single sensor. This would make it possible to more accurately forecast the service life of the equipment and draw up appropriate maintenance plans.

Image of predictive maintenance using sensor fusion — Figure 5

Passing down the technical expertise of skilled operators: Maintaining quality

The quality in terms of dimensions, configurations, surface finishing, etc., of work processed by robots or automated machines can be evaluated using inspection equipment, such as image sensors, dimensional measuring devices, and image processing systems. Also, quality judgments can be made through comparison with the threshold values set for various inspection data items. Nevertheless, skilled technical knowledge and experience are necessary to make quality judgments based on the data output by increasingly precise sensors.
With sensor fusion, data obtained from sensors and test devices is combined to implement inspections. For example, an image sensor can be used for configuration inspection and an image processing system for surface finishing inspection, allowing for an integrated determination of quality. This makes it possible to implement an inspection system that is not only automated, but also prevents the overlooking of faulty products by inexperienced operators.

Fusing operators’ vital signs data and spatial environment data to maintain and improve the work environment

For smart factories, improving the work environment is a key goal, alongside boosting production efficiency. This requires monitoring of the temperature and humidity, as well as the concentration of gases (oxygen, carbon monoxide, carbon dioxide, and other gases in use) and particulate matter (particles produced by the manufacturing products process, dust, etc.) in the air of the location where operators work, and detection of abnormalities in the physical condition of operators by monitoring their body temperature, heart rate, etc.
In this case, sensor fusion makes it possible to track the work environment quantitively and precisely by fusing and monitoring data from CO2 sensors, temperature sensors, etc., and vital signs data such as the body temperature and heart rate of operators. Information obtained in this way can be used to suggest work environment improvements, such as measures related to climate control, ventilation, and dust abatement.

Types of sensor fusion

Sensor fusion is technology that combines data collected from sensors and outputs information that is easy for humans to understand. It can be classified into three types based on the methods used to achieve the fusion of data. Here we discuss the advantages and disadvantages of each method.

Centralized model

This model uses a method in which data collected from multiple sensors is sent directly to the central processing unit (CPU).

Advantages: Since data is sent directly to the CPU and fusion of the data occurs on the CPU, highly precise analysis results can be obtained.

Disadvantages: To process the large volume of data sent by all the sensors, a high-performance CPU is essential. If the processing cannot keep up, the responsiveness drops. For this reason, the centralized model is generally not used in fields that require instantaneous determinations.

Distributed model

This model uses a method in which data collected from multiple sensors is first processed by each sensor and then sent to the CPU.

Advantages: Since data is sent to the CPU for fusion after being processed by the sensors, the load on the CPU can be reduced.

Disadvantages: Since the data from each of the sensors is first processed locally, it is possible that some data that could have been useful material for analysis may not be sent to the CPU. This means the analysis results are less precise than those obtained using the centralized model, so the distributed model is not used in fields that require high-precision analysis.

Hybrid model

This model uses a combination of sensors that send their data directly to the CPU and sensors that process their data first and then send it to the CPU.

Advantages: Since data is fused among the sensors and the results sent to the CPU, which then outputs the optimal solution, this model combines the advantages of responsiveness and stability from the centralized and distributed models.

Disadvantages: The system configuration is more complex, and the processing costs associated with data transfer and calculation are higher.

Conclusion

Implementing sensor fusion requires continuous and stable acquisition of data from multiple sensors. Smart factories utilizing IIoT already have in place the equipment and technology that serves as the infrastructure for sensor fusion.
Deploying sensor fusion technology in these smart factories makes it possible to achieve further productivity gains, while improving aspects such as equipment maintenance, quality assurance, and improvement of the work environment, all of which result in even smarter factories.
Developments expected to achieve widespread adoption, such as wireless sensors and new sensing technologies like edge AI involving sensors performing AI inference, are sure to bring about new innovations in smart factories when combined with sensor fusion technology.