Tara Baker: Thank you for joining us today as we discuss the evolution of sensor’s, new sensing technology and testing and certification challenges facing manufacturers that want to leverage this new technology. Now, as with most areas of technology, sensors have evolved over time, the earliest forms of sensors were able to turn a physical event into a mechanical movement with the use of electricity, electrical since the start to appear on the scene. And these sensors had the ability to turn a physical event into an electrical current. And these sensors paved the way for basic remote control and automation. Then we start to see miniaturization which highly increased the accuracy and reliability of sensors. And today we’re seeing a new step in sensing technology. We now see the communication within the sensors and the Internet of Things promises fully automated processes based on a complete digitization of physical assets. But what does this mean for companies and manufacturers and businesses that want to leverage this technology? Well, from a process perspective, filling up a factory plant with sensors to optimize production is a great idea. But the implementation has big practical issues. For example, the number of sensors needed to digitize an entire factory could easily be in the thousands. And the cost to power up and maintain all of those sensors can be restrictive for many companies. And they’re also technology issues.

For example, most sensors can’t work reliably or can’t be installed in harsh environments like the ones present in some industrial processes, for example, high or cryogenic temperatures, aggressive chemicals or salt water or high pressure or high radiation. And also, this sensors can be too expensive, they can be unreliable or they may not even exist at all. This is where an emerging technology called virtual sensors can help. But what are virtual sensors and how can they solve some of the issues present with physical sensors?

Well, joining us today to discuss these topics are Dr. Edgar Sotter, Senior Director of New Product Technology at CSA Group, Anna Zelisko, Technology Consultant at CSA Group, Rajinder Jakhu, Technical Oversight Specialist of Signal Sensing, and Trevor Perera, Product Manager for Home and Commercial Products at CSA group.

Edgar, why don’t you start by telling us a little bit more about the historical background of virtual sensors and how they are used.

Edgar Sotter: Yes, so Tara, virtual sensors or soft sensors as some people know them are inferential models that take inputs from sources like physical sensors, process information or historical data to estimate an unknown variable. Just to give you an example of what a soft sensor or virtual sensor is, we have the weight scale.

Recently, a few months ago, I bought these smart weight scale that I use my bathroom. I put my feet on, and look at my weight, I am not too happy of what I see every day, but well… It is what it is. So this weight scale, besides my weight, it gives me my fat content, my muscle content, the bone content of my body and also even my hydration levels at that point.

Now if you see the scale, it does not have one different sensor for each of those values or parameters. Instead, it only has two electrodes, where I put my feet on. And those electrodes connect to a bioelectric impedance sensor that the scale combines with my weight in my height to estimate all those other values. So this example shows us that virtual sensors usually reside in embedded systems, that most of the time also have physical sensors.

The example also shows a really good characteristic of this technology, and is that it can reduce the number of physical sensors that you need to find new parameters.

This is key for the evolution of Industry 4.0 or the industrial Internet of Things, just to give you an example to explain this concept through an example, you mentioned in the introduction about the cost of digitizing a whole factory plant. So let’s go to that example and imagine that we have to do that job for a 5000 square meter plant. And let’s say that we do our math, our estimation, and we come to the conclusion that we need around five sensors per square meter to do the whole work, the whole project.

If you do the calculations, five sensors per square meter in a five thousand square meter plant, you will need around twenty-five thousand sensors.

Those are a lot of sensors. Now, you were right on mentioning the cost, because just to change the batteries of those devices, when the battery dries out or to replace the broken sensors will require a full-time employee for that. Also, another point, let’s assume that each of those devices will require around three watts of power, which is normal, given that those devices may connect the awareness to a computer. If we do the calculations, three Watts per sensor, five sensors per square meter and five thousand square meters, you will need around 75 kilowatts of power for those sensors to work.

That is a lot of power.

You will need to get a whole, a high-power transformer just to power those sensors, which will add to the cost of the project. So now we see why just using physical sensors is not a good idea from an application standpoint. Now, let’s bring virtual sensors to this example. So from the five physical sensors were using, let’s assume that we can estimate two of those parameters using the other three sensors.

So instead of using five per square meter, we are going to need three per square meter. That will reduce the number of sensors from twenty-five thousand to fifteen thousand. That is still a high number. Still a lot of sensors. But it’s less than twenty-five thousand. Now, let’s assume that we can take we can model the warehouse, where the factory is. We can create a model of that and put it inside the virtual sensor, which means that instead of needing to put three sensors every square meter, we can now do it every 10 square meter. That will reduce the number of sensors from fifteen thousand to fifteen hundred, which is way more manageable for our company. So you see how by introducing virtual sensors, we were able to reduce the number of physical sensors from twenty five thousand at the start of the project to fifteen hundred. This is the power of this technology. Now, this is not a new technology. If you do, we do a literature research. We can find papers back from the 1990s, even before of that. But there is a reason why now this technology is getting a lot of traction. There was a survey done in 2009 by the Journal of Chemical Engineering of Japan, it was actually a paper released by that journal. And in that paper and in the survey, they mentioned two major reasons why companies were not adopting this technology that fast. One was the cost of acquiring the data from the physical devices, from the physical sensors to the computer that was analyzing the data and the other a roadblock was and the cost of creating the inferential models from scratch. It was too expensive to do that.

Those two issues have been solved with all the communication technologies that IoT brought, that the Internet of things have brought, and the inferential models are no easy to make thanks to all those machine learning tools that are available for developers today. And we can see how this is creating a lot of research and development, including virtual sensors. I did a quick search in a database recently and found several papers that were really interesting that included virtual sensors. I’m just going to mention two of them: one was the use of a virtual sensor to estimate the state of the charge of the battery of an electric car. So instead of using a current sensor, which is the standard way to measure the charge, what they did was to estimate the state of the battery by using the voltage, using the amount of time the driver step on the gas pedal. Well, in this case, it’s not the gas because it’s an electric car. But it would be like this speed up pedal or something like that. So they use those two values, plus another couple of more volumes to estimate the charge and it end up to be really, really close to what the physical sensors was providing. Another example is a collision sensor for robots. So instead of using proximity sensors to detect when the robot hits an obstacle, in this paper, they were showing how they could use the current that the engines of the robot pull when the robot hits an obstacle. So when that happened, just the force that the road needs to make pulls more current for the motors, and they could corelate that to the presence of an object. So just those two examples show how people are now using this technology to find ways and applications to provide redundancy to physical sensors or in some cases even replace physical sensors.

Tara Baker: Thank you, Edgar, sensors have certainly been around for a long, long time, and there have certainly been some major breakthroughs and because senses have been around for a long time, they’ve been able to collect a lot of data. Anna can you tell us about how different mathematical models can use this data in sensors and how these models are built?

Anna Zelisko: Sure, Tara. So as both you and Edgar mentioned, sensors have been with us since the Industrial Revolution. So we’ve had a lot of time to collect data associated to physical sensors. So that would be the environment in which that physical sensor existed, the attributes of that specific sensor and finally kind of the output that the sensor provided us over time. Now, having this information, we can study the patterns of those physical sensors or the different traits that provided us with the outputs that they gave us. And once we understand those trends and behaviors, to model, to recognize those in the real world as well. So that’s where we kind of get into building a mathematical model. Really in our environment, there is two forms or two groupings of mathematical models: the first is regression models, which really focus on producing a numeric output and classification models which focus on creating kind of smaller groupings. To really answer your question. And now you typically will pick one or the other pending on what problem you’re trying to solve or what the sensor is trying to get at. So as Edgar mentioned, kind of in his example, we saw that there was we were trying to have a virtual sensor that measured the electrical charge of a battery. When we think about that, that’s the direct question we’re asking is what charges are we currently at for the battery? So that would provide us with a percentage or a number of this is how much we have remaining. So to get that, we could put in attributes such as what kind of car are we looking at, how far is the car traveled and so forth and so forth. And we would like to get kind of a final number with which we could evaluate what to do. So a regression model would work really well in this case. And you can kind of test a variety of the regression models that are out there to see which one really fits your data set the best. Now, secondly, in the classification example, another piece that Edgar mentioned was having a wide number of physical sensors around your factory, even though you can kind of replace them with one small virtual one. So in this case, what a few of the papers I have read have done has actually they took in all the data from those sensors and actually grouped them together to say, you know, which ones actually are telling us similar information about that area or that region. So it allows us to then group that information and train virtual sensors and classification models on specific data. And if a switch is very simple, so with the charge example, really what we might be going to is do I need to stop for gas right now? The answer is yes or no. So that’s more of a grouping classification mechanism as well, which is the virtual sensor kind of informs us of a simple choice is to say yes or no answer or something even broader as well. So in either of those cases, it’s really the problem that drives the decision between the two models, but I hope that gets to answering your question as to how mathematical models can help in assisting virtual sensors become more powerful in our times.

Tara Baker: Thank you for discussing how those key models can help support virtual sensors. Given the versatility of sensors, Rajinder what certification considerations should be taken into account when using a sensor as an input in the control system of a device.

Rajinder Jakhu: Oh yeah. So I’ll go through with the existing certification model we have and how the same model we can use it for virtual sensors. So I just go over like the safety control that we commonly used in the household appliances, where the sensors are monitoring the temperature, humidity and the outputs are driving like motors, alarms, a light. In the normal mode, the controller will operate within the functional requirement of the product. But under the abnormal situation, it will act to mitigate hazards, to provide safety to persons and property by disabling the output. So sometimes the safety is achieved by safety devices, in safety critical circuits such as thermal fuse and bimetal thermostats. So, for example, if the thermal fuse is installed in the motor winding and in case the winding temperature is exceeding the limit, the thermal fuse will open to protect the motor winding against any fire hazard. So the reliability of the safety devices are verified by reviewing the construction of the product, performing the test, some of the tests are listed in the product standards and we do the test under the extreme operating conditions. So in the case of a biometal thermostat, it is checked that the thermostat will open when the temperature will exceed the set point. And also we do the endurance cycles on the thermostat and the full load, and the number of cycles are calculated based on the target cycle of the lifecycle of the product. So with the advancements in electronics and embedded software, the same thing is performed by discrete sensors and the controller will process the input signal to control the output devices under both normal and abnormal situation. So in the case of a motor, the winding temperature can be monitored by NTC type of sensors, as an input to the controller. The controller will open the motor relay when it detects the temperature is exceeding the maximum limit of the winding temperature. So, the sensor will act as input signal to the controller and by itself it will not open the motor winding circuit, but the circuit control is done by the controller.

So when the sensors are used with electronics and embedded software to achieve safety, then the reliability of the product to act safely in abnormal situation becomes difficult to achieve. There are a number of factors for that. So such as detecting the input signal correctly, the reliability of electronic component due to aging and environmental factors, hazards and risk associated with the software implementation, also electronic circuits are sensitive to mains-borne perturbances and radiated disturbances. So in these cases, to do the certification, we verify the safety of the electronic circuit by performing electrical testing, software evaluation and EMC testing.

So in the software evaluation, the systematic fault and the control which are using safety software and electronics, are avoided by implementing V-model of the software life cycle and the random faults are dealt with components FMEA techniques. Therefore, the safety system can be designed in such a way, the systematic errors are avoided and random faults shall be dealt with by a proper system configuration.

So now that virtual sensors are being implemented in the product safety as it was introduced Edgar and Anna. And CSA may be able to evaluate virtual sensor implemented in safety critical applications under attestation program, on a case by case basis.

Attestation is a non-certification service and is intended to provide manufacturers with evidence that a product has been evaluated by the CSA to the requirements indicated on the Attestation marking and in the Attestation report. Back to you, Tara.

Tara Baker: Thank you, Rajinder. Trevor, I would love to hear your thoughts on this, too. Is there anything that you’d like to add?

Trevor Perera: I would echo the same criteria that Rajinder mentioned when we are certifying. Pay attention to the safety software analysis, functional safety, EMC, because the virtual sensors are heavy on electronics, you know, getting input from different models and trying to process that or try to simulate a normal sensor. Having said that, I would like to add that we are inputting data from known sensors into the mathematical model and expect the virtual sensors to operate close to the real sensors, which definitely has a lifetime degradation and interference. So we will have to subject the virtual sensor, to the same environmental conditions when we are testing. But that alone is not enough because it’s more heavily on the electronic side. The reason I’m saying that is the virtual sensors should not degrade like electrochemical sensor or metal-oxide sensor and so forth. So their life is much, much greater from that aspect. But from the interference on electronics, you have to do due diligence to look at those. And as Rajinder said, Attestation would be the obvious path to start testing, evaluating these sensors. And the next step obviously would be a national standard, like a CAN standard or in the US an ANSI standard. That lead to more recognition throughout the World, you know. So as I wanted to point out, the sensors that we have been testing so far, there have been for detection of carbon monoxide, flammable vapor sensors and combustible gas sensors.

Those are the ones that are most widely used right now.

And there are established standards and test methods to evaluate that. So the virtual sensors are looking to those types of standards and requirements either to meet them or exceed them in their performance. Because you are, after all, from those their lifetime data, the behavior and environmental conditions, inputting all that, like you said, in a regression model, the classification model, and then coming up with something that will closely resemble the real sensor performance specifications.

So that’s what I had additional to what Rajinder said.

Tara Baker: Yeah, thank you, Rajinder and Trevor, for the providing insights into the critical importance of sensors and also the challenges of testing and certifying products with control systems that rely on the use of virtual sensors.

Edgar, in light of these challenges, it does seem particularly important for manufacturers to find a solution to assess the accuracy and the reliability of virtual sensors. Can you tell us a little bit more about that?

Edgar Sotter: Yes, Tara. This technology, virtual sensors is gaining a lot of traction lately. Manufacturers are not only using this technology to reduce cost by reducing the amount of physical sensors they need in their product or systems. They’re also using them to expand their product roadmap to design new products or systems, to put them in areas where physical sensors cannot be used, like areas with pretty high temperature or low temperatures, as you mentioned in the introduction, or areas where there is a high amount of radiation that could destroy any physical sensor. So that’s the reason why these manufacturers are using it more and more and more often. Trevor mentioned a good point too, these sensors don’t degrade as physical sensors can. Like, you have electrochemical sensors, they could get poisoned or they could degrade with time. Virtual sensors don’t have that issue. And you can see when we do our research online to find patents, we can find several patents that are related to virtual sensors in recent years. Just to mention two of them: one was for a combustion system, a combustion control system that uses sensors to assess the temperature in the chamber, in the furnace, as well as gases inside the chamber. I also found another patent from other company, very closely related to this one, there was the use of a virtual sensor to retrofit a burner control system. So companies are already thinking on using this technology, not just to improve the performance of their systems, but also to provide redundancy to sensors used for the safety critical parts of their systems.

And this is where accuracy and reliability is key, start to gain more attention, because as Rajinder and Trevor mentioned, any sensing technology that is used in these systems have to prove that it meets the same level of performance that existing sensors. So manufacturers have to come to out to a way to assess the accuracy and reliability of these virtual sensors. Now, this is not an easy task because even though the virtual sensors don’t degrade with time as physical sensors do. There is a challenge here because these pieces of software, these inferential models are expected to improve as time goes, it is expected that they will use the same data that they are acquiring to improve the model. That means that the test that we do today may not provide the same output of the test that we can do a month from now, which can be challenging if we start thinking on ways to certify this technology.

So but to start, we have to first think on a way to assess the accuracy and reliability of virtual sensors.

Tara Baker: Thank you, Edgar.

I think that it’s clear that, you know, it’s very important that we need a robust solution that can test for accuracy and reliability. Anna can you tell us about processes, data sets and models that can be used to assess the accuracy and reliability of virtual sensors?

Anna Zelisko: Sure, Tara. So an important thing here is to really go back to the model building process, right, as you build your model.

There are a few key stages in which you want to really take a step back and say, is this an accurate step that we’re doing? Is how is this actually going to influence my model in the end of the day? Because ultimately, as Trevor mentioned, we want to build a virtual sensor that is just as accurate as a physical sensor, if not more accurate.

And the first step of that is back to the data, right? We want data that has been cleaned. It doesn’t have many blanks. Those blanks have been filled in.

We want data that is an accurate representation of that realistic situation. So if there are some weird off occurrences, do we want to include those occurrences, or what is it about these specific occurrences, are they accurate of how a physical sensor may actually perform in the real world? Those are questions you kind of want to ask. And finally, you want to make sure that your data set is pretty diverse. So if you’re building one specific model to answer a key problem, you know, do we have a way to diversify that set to maybe use two or three physical sensors that are trying to calculate the same thing just to see the data across physical sensors, not only across different time frames as well.

So that’s kind of the first question, looking at a diverse set with good data that’s clean and hopefully you have a lot of it. And the second piece is actually where we’re starting to integrate all that data into a model. And so typically this would work with evaluating the types of information you’re putting in. Some information might be duplicate information of another piece you’ve already put in. Some might not actually even benefit the model at all. So like in the energy and battery situation, if I included what I had for lunch, probably not going to help the model.

So I’ve got that data, but is it helpful?

So, you know, you’ve got to, you’ve got to give that to the model and let it actually or even other algorithms.

And they’ll actually inform you of whether this is a valuable feature and you should put it in the model or whether this is a feature that maybe you should drop and leave for another occasion.

So to make sure that that process is consistent and that you haven’t even exposed some of your testing data to your model is very important because once you’ve done that, your model already knows what you’re about to show it. And it’s going to give you a 90 percent accuracy, and you are like “great model”, but actually you biased it. So that’s an important part of this process as well. Now, finally, if you’ve actually got the data, put it in the model. We know we’ve got the great features, final pieces to actually evaluate your model. And most models, regression of classification have their own form of metrics that are already built into the system and that a lot of data scientists and machine learning engineers out there use and kind of the community has confirmed they are good metrics to use. So that can be anything that’s a C-curve, an F1 score sometimes R2 score. So each of these is dependent on the type of category that you pick along with even the type of model you’re using. And sometimes it’s also highly encouraged to train some different models in the same category to see which one of these three models is the best model to actually represent this data set.

So you can easily cross compare it within categories that you want to cross, compare across categories that can be a little trickier, but still possible. So there’s a lot of opportunity to kind of step in and say to help with the standardized piece and say, you know, this is what your data should cover, should look like, this is kind of what we were thinking your process in terms of building your model should look like. And finally, let’s actually evaluate what your model score is. So that’s in the full picture, kind of how one can test the accuracy of validate whether a model is a reliable representation of a physical sense or physical world even.

Tara Baker: Thank you for that very detailed explanation.

Edgar. There are clearly challenges and opportunities, in the use of individual sensors. Can you explain what CSA group can offer to manufacturers using virtual sensors in critical systems?

Edgar Sotter: Well Tara, there is not one answer for that question, as Rajinder mentioned just a few minutes ago, we will have to go on a case by case basis here. Given that the current standards that apply to products that are using sensors or control systems like the CSA 0.8, UL1998 and the CSA/UL/IEC60730-1 annex H. They don’t consider the use of systems like virtual sensors. These are too complex for what these standards are considering. It will be hard for us to provide a certification to these standards. However, there are options that can be helpful for a manufacturer, like performance evaluation or some level of Attestation, like Rajinder mentioned. But we will have to go on a case by case basis. It will depend on the specific product and the way it is using the virtual sensor. Now, one thing to keep in mind to remember, it’s that this is not the first time that CSA is dealing with new technologies, with emerging technologies. We have done it in the past several times and we have found solutions for it. So we will do it again. This panel is just an example of how we are proactively looking for ways to approach this emerging technology: virtual sensors.

Tara Baker: Thank you, Edgar. I would like to also thank all of our panelists, Edgar, Anna, Rajinder and Trevor, for such a robust and interesting conversation today. It’s so exciting to see how much knowledge of this topic we have in-house at CSA group as we are looking into possible solutions for new technology. And to our audience, please do keep an eye out on our digital channels for more information about emerging technologies and new solutions. And if you’re interested in learning a little bit more about this particular solution, you can reach out to us via the Contact US form.

This concludes today’s session. Once again, thank you so much to our panelists and thank you for joining.