Deploying
Industrial AI At Scale:
From Theory To Production
Using advanced data-analytics technologies like AI and ML to optimize manufacturing processes has gone mainstream. The benefits are just too compelling, including reduced costs, less wastage, faster time-to-innovation, and enhanced customer satisfaction. But the reality on the ground is that the vast majority of industrial AI projects do not make it into production.
There are many obstacles that constrain the promise of AI-based manufacturing. Some common challenges are:
- The lack of a cross-team pipeline to reliably move POC models to production.
- The difficulty in finding data-science talent in general, and talent with domain expertise in particular.
- Scalability issues: Building one model does not mean that you can build the 100 models you need to meet your different production needs.
In this technical guide, we examine why it is so difficult to move AI-based manufacturing solutions into production. We then discuss how using a fully managed but flexible AI-model service such as Vanti can close AI scalability gaps and consistently optimize production outcomes.
Why Is It So Hard to Scale AI Into Production?
It takes an entire organization working together to build a successful AI model (i.e., a model that will migrate well from the lab to the production floor). As you can see in Figure 1 below, each step in the AI-model development cycle requires inputs from outside the data-science team. For example, you need business and domain expertise to get the right use case (see Step 1), and you must have seamless feedback loops between operations and data-science teams for monitoring and maintaining models already in production (see Step 6).
In other words, successfully building a single AI model for a specific production use case requires a significant investment in resources, time, and organizational structure. Imagine scaling this effort to meet all the AI needs of all production lines and sites. Now add into the mix the fact that data changes constantly in the manufacturing domain (testing schemes evolve; materials change; machines fall out of calibration; and so on). An unmaintained model won’t survive more than 1–2 weeks in production.
Given these challenges, not every manufacturing organization can (or wants to) invest in and build out AI capabilities in-house.
A Guide to Deploying Manufacturing AI at Scale
In order to demonstrate how Vanti helps manufacturing organizations close their AI gaps at both the single-model and many-model level, we use the following scenario:
The key metrics that predict a Pass or Fail result for a particular unit under production are the temperature, pressure, and angle that are applied in the 10 physical stations in the production line that are involved in manufacturing the unit. The manufacturer wants to automatically ensure that: No defective units will make it past physical station #5. Any faults in the line itself that may be causing these defects will be detected quickly.
Getting Started With Vanti
Vanti has production-ready AI solutions for three key manufacturing challenges:
Models that detect failures at a very early stage or even predict them before they happen
Models that detect defects in units as they occur
Models that do in seconds what traditional calibration stations do in 15 minutes or more
With Vanti’s self-service AI product, all you need to get started is:
- Identification of the production line you want to improve: Vanti is all about results-oriented AI that quantifiably improves production outcomes. So it is important that you have a specific production line in mind, as well as a clear understanding of the problem(s) you want to solve.
- Sufficient data to train a model: Vanti supports both tabular and image-based training data. Usually, 1,000 records in the training dataset is sufficient for image-based data. For tabular data, you should have at least 10 records for each metric. Using our scenario as an example, you would want to provide a training dataset as follows (3 metrics x 10 records each):
Vanti in a Nutshell
Generally speaking, your Vanti models can be created, configured, and trained in a matter of hours. It’s usually just a couple more days to integrate the model into the production line itself and start getting real-time predictions. Once deployed, Vanti automatically monitors and visualizes model performance, alerts to model performance issues (such as slipping accuracy), and auto-discovers new model opportunities. In addition, Vanti’s white-box approach ensures that your team can inspect and tweak the models.
Viewing and Inspecting Vanti Stations
Each Vanti station addresses a single solution (early fault detection, defect detection, or process optimization), with multiple models that aim to maximize accuracy for solving the problem. There can be multiple Vanti stations that link to a single physical station (e.g., a physical station may require early fault and defect detection at the same time). Also, a given Vanti station could point to multiple physical stations (although this is not recommended if the customer wishes to get accurate predictions for each physical station).
Figure 2, above, shows a typical summary screen for viewing and inspecting deployed Vanti stations, where:
[1] is the name of the Vanti station.
[2] is the Vanti station status, such as Live (deployed and running), Ready (trained but not yet deployed), Error (as detected by Vanti, which automatically suspends the Vanti station until corrective action is taken), and so on.
[3] is the Vanti station performance, such as prediction accuracy for live stations.
[4] is activity on the Vanti station, in graph format.
[5] is the average number of predictions per day for that station.
[6] is the last time a prediction was made.
[7] is clickable alerts and suggested actions.
[8] is an overview of all Vanti stations, including the total number of deployed stations (and how many are not deployed), the total number of prediction calls, a summary according to station status, and a summary according to actions to be carried out.
Now let’s drill down into a specific Vanti station. In the example shown in Figure 3 (below), the Vanti station deploys an early fault detection model called Model1 [3-d].
At the bottom left [1-A] of the image above, we see a graph of the prediction calls carried out by this Vanti station over the last hour. The user can change the timeline [1-E]. Just above the graph, we see how many total Predict Calls [1-C] were made to the Vanti station over the last hour. We also see the number of Feedback Calls [1-D] that were returned to the Vanti servers. A feedback call provides the actual status (Success or Failure) detected at the end of the process. Vanti uses the Predict Calls and Feedback Calls to calculate the accuracy of the early fault detection model deployed currently on the Vanti station. We can also see the number of failures that the Vanti model detected [1-B].
To the right of the graph is the Insight Feed. This is a rolling feed of Vanti-generated insights and warnings about the model. The insights reflect the logic behind the model that was created, based on the data trained. For example, the first insight shows that parts with a certain serial number are more likely to fail, an insight that can be used to make adjustments to the production process.
Adaptive AI on the Production Floor
Training an AI model to solve a manufacturing problem is challenging. But to be truly production ready, models must be able to seamlessly scale to multiple lines, as well as adapt automatically to changing data and production conditions.
At the top of the Station Live View (Figure 3), we see three important automated services provided by Vanti to monitor model performance in the production environment and adapt and optimize the model as required. These services are particularly critical in the industrial domain, where data is highly dynamic due to continuous fluctuations in equipment performance, environmental conditions,
operators, materials, customer requirements, testing criteria, and so on.
Model Self-Monitoring
Every Vanti model has the ability to monitor its accuracy and alert to dropping accuracy or other anomalies that could impact model performance. In the case of early failure detection models, the self-monitoring is based primarily on the correlation between Predict Call and Feedback Call results. The team will be alerted both here and in the Summary View (Figure 2) when a model’s performance falls below an acceptable threshold.
Model Adaptation to Data Changes
This service checks whether the model needs to adapt to a change in the environment, such as data drift. If we go back to our reference scenario, the model was originally trained on a temperature range of 45–55. If the real-time temperature values drift beyond this range (let’s say 40–60), the model’s performance may be affected because it doesn’t really know what to do with values below 45 or above 55. Similarly, the model is expecting to get three parameters (temperature, pressure, and angle). What happens if the prediction call is missing one of the parameters? Will the prediction accuracy be affected?
Vanti uses a backtesting process that runs the training set on the new (changed) data or data structure and checks if the model’s expected accuracy is still good enough to keep making predictions.
Model Optimization
This service can be triggered by one of the other two services (Self Monitoring or Adaptation), or it can run routinely at a predefined interval. Vanti constantly collects new data coming in from the production line and automatically checks for improvement opportunities by training new models based on new data or new data structures. Vanti then compares these ghost models against the existing model based on the main performance metric, such as recall and precision for supervised models or silhouette score for unsupervised models. If the performance of the new model is better, it is offered to the user.
Model Inspection and Tweaking
Each Vanti station can have a number of trained models. If you click on the name of the currently deployed model in the Live Station View (Figure 3: 3-D), you see a summarized comparison of all the trained models associated with the station, as shown in the image on the right. You can either choose another trained model to run on the station or click “Train Another Model” to create a new one. Vanti also provides advanced functions for data scientists to tweak trained models, but that’s a subject for another article—stay tuned.
Scalability
In addition to supporting quick time-to-production-ready models, Vanti supports the scalability that is essential for impactful deployment of AI on the production floor. It’s not enough to build and deploy a single AI model on a specific production line—you have to be able to scale to managing and maintaining hundreds of models across numerous and diverse production lines.
The key ways that Vanti ensures scalability are:
- Autonomous, self-aware, self-healing inference models: The model monitoring, adaptation, and optimization services described above open up one of the biggest AI scalability bottlenecks. They eliminate or dramatically reduce the need for time-consuming (and error-prone) manual model maintenance.
- Seamless duplication and deployment of models: Have you tweaked a training model so that it fits your production environment like a glove? With a couple of clicks, you can now duplicate the trained model and apply it to all relevant Vanti stations.
- Easy application of Vanti stations to other production lines: Once you have defined a Vanti station (its inference type, a set of trained models, the default active model, and so on), you can easily apply that station to all relevant production lines.
Vanti in Action: Some Case Studies
This section includes a series of case studies from manufacturing customers who used Vanti to apply AI to real production problems, with clear and quantifiable benefits.
Image-Based Fault Detection at Scale
In addition to supporting quick time-to-production-ready models, Vanti supports the scalability that is essential for impactful deployment of AI on the production floor. It’s not enough to build and deploy a single AI model on a specific production line—you have to be able to scale to managing and maintaining hundreds of models across numerous and diverse production lines. The key ways that Vanti ensures scalability are:
The key ways that Vanti ensures scalability are:
The Challenge
The process for producing millions of memory product wafers annually has hundreds of stages. At each stage, an X- ray image is taken and unlabeled data is saved. Early default detection (and remediation) is critical in order to avoid the high cost of failure in later stages. It took three months to train an image- based fault detection model for a single production stage, including the need for domain experts to manually label the training data.
The Solution
- The OEM used Vanti to build and deploy unsupervised image classification models within three hours.
- Performance was validated with manually labeled images from five unique stages.
- The model’s performance was further improved by injecting domain knowledge using data engineering and code (a one-time activity).
The Benefits
- Dramatic reduction in time-to- deployment: A high-performance unsupervised model was trained and deployed for five stages in three hours (versus three months).
- Less reliance on domain experts: Significant optimization of domain expert time to label the training data.
- Consistent high performance: The self- healing, self-adapting inference model continuously delivers a 95% true positive detection rate.
- Unprecedented scalability: The customer scaled Vanti’s fault detection model to 60 additional production stages in just under five days.
Virtual Metrology and Data Drift Recovery
The Challenge
The memory wafer fabrication process comprises more than 1,000 unique steps, during which time-series data from the production equipment is collected for each wafer. The process includes multiple expensive and time-consuming quality-control steps, such as full metrology inspection of each wafer 13 times. The process is prone to data drift due to constantly changing conditions in the processing equipment.
The customer needed to:
- Decrease the overall cycle time by reducing sampling time during each metrology inspection, as well detecting defective wafers early in the process.
- Deploy a model that is immune to natural data drifts.
The Solution
Using Vanti, the customer was able, for each step, to train a model that predicted the metrology test results based on a combination of:
- User-defined acceptance criteria
- Historical time-series data from the relevant step
- Corresponding quality test results
These models provided reliable real- time defect prediction, reducing the customer’s dependency on physical metrology testing. In addition, Vanti automatically overcame the customer’s data-drift challenge through its model monitoring, adaptation, and optimization services.
The Benefits
- Streamlined data preparation: The data engineering and cleaning capabilities built into Vanti dramatically reduced the time and complexity of preparing the model training data.
- Enhanced production efficiency: By minimizing the need for physical quality sampling during metrology inspection, Vanti was able to significantly reduce cycle times.
- Consistently high quality of service: Vanti’s proprietary drift-recovery mechanisms and auto-optimization of deployed models ensured a high quality of service with minimal human intervention.
Predict Failures After Deployment (RMA)
The Challenge
A manufacturer of power management systems produces tens of thousands of products each year, each with a long warranty period (12–25 years). Approximately 3,500 field units are returned each week for repair, incurring significant costs for technical support, repair services, time spent on RCA, and wasted material—not to mention the company’s damaged reputation. The customer wants to detect—early in the production cycle—units that have a high likelihood of failing in the field.
The Solution
Vanti built an ML model to predict units that are likely to fail in the field based on uploaded historical testing- results data for tens of thousands of produced units, along with their post- deployment performance.
The Benefits
- Accelerated RCA
- Fewer returned units (RMA)
- Reduced technical support, repair, and servicing costs
- Fewer customer complaints and warranty claims
Key Takeaways
Although it is clear that advanced data technologies such as AI and ML can—and should—play a central role in optimizing manufacturing processes, there are still significant challenges in reliably and scalably moving these solutions onto the production floor, including:
- Shortage of data-science personnel in general, and those with domain expertise in particula
- Lack of the cross-team organizational infrastructure that is essential for consistently moving POC models into production
- The sheer scale of building the dozens, or even hundreds, of models that are needed to meet diverse production needs
- The difficulty in maintaining inference models in light of the highly dynamic nature of production processes and data
Vanti’s SaaS product has been designed from the ground up to deliver production-ready inference models that are trained on customer datasets. Vanti models can be created, configured, and trained in hours, rather than months. And Vanti’s white-box approach means that these quick time-to-production-ready models can also be inspected and tweaked by your data scientists, as necessary.
Once deployed, Vanti’s models automatically monitor their own performance and alert to slipping accuracy or other anomalies. Vanti also automatically detects and adapts to changes in the production environment, such as data drift. Lastly, Vanti continuously searches for opportunities to improve the inference model and prompts the user to switch to a new optimized model where relevant.
In all of these ways, and through an intuitive user interface that doesn’t require specialized data-science skills, Vanti supports the kind of scalability you need to manage and maintain hundreds of models. The benefits include reduced costs, enhanced efficiency, and greater customer satisfaction.
Start using Vanti today to see how Vanti can consistently help your manufacturing organization reap the full benefits of AI in your production environment.
