Without data, even the most advanced AI model can only be as smart as a rock. For models to become adept at their intended use cases, they require data for model accuracy, low costs, and effective value delivery. Since the release of ChatGPT in 2022, the world has become obsessed with the new AI world that’s being built around us. While there is a lot to be excited about, it is important for organizations to be measured in their approach, and to be good stewards of their data to make sure they are fully taking advantage of it. This especially holds true in manufacturing, where organizations may hold valuable data, but in disparate locations, preventing their ability to optimize it and drive efficiencies.
Everyone is working to improve AI model architectures right now, and no one knows where it’s going. What we do know, however, is that data is the lifeblood which drives the development of AI and underpins its future. At StandardData, we see data as being the limiting factor in the market penetration of AI over the long-term, and especially over the next several years.
This blog explores the benefits of well-architected data infrastructure and data governance, and why it is important for your manufacturing organization to start now and realize the competitive advantages that come with preparing your data, when, and if, you are ready for the AI journey.
In this blog:
- The Data Scarcity Challenge in Manufacturing
- Data as a Competitive Advantage
- Benefits of Data Preparation for Manufacturing
The Data Scarcity Challenge in Manufacturing
Data is the foundation upon which AI models are built. Without clean, accurate, and relevant data, even the most sophisticated algorithms can falter. High-quality data ensures that AI models can learn effectively, make accurate predictions, and deliver reliable insights.
Despite the abundance of data we have today, according to the Wall Street Journal (WSJ), there is a 50% chance that demand for high quality data exceeds supply by Mid-2024, and 90% chance it will happen by the year 2026. Why is this so? There is a subset of an organization’s data that is likely not digitized yet (on paper), and even though a large portion may already be in the cloud, it’s unlikely the data can be utilized. This is the difference between quality data that is leverageable, and data that’s a meaningless waste of cost to store.
Data will be scarce for many years to come, but it doesn’t have to be for your manufacturing organization. Your organization has likely generated tremendous amounts of data from its activities including sensor data, telematics, image and video data, contracts, proposals, invoices, and so on. This is what constitutes the “quality” data the WSJ mentions, as it’s data that isn’t available on the public internet and is unique to your organization and its secret sauce. The good news is, even if AI is something later down the road for your organization, you can take full advantage of a modernized data infrastructure now, and there are many benefits to doing so.
Data as a Competitive Advantage
With the understanding that data shortage is going to be an impediment to AI model development as a whole over the next several years, organizations must take advantage of this by treating their data as the precious material it is, or more specifically: Data as their competitive advantage. Not only is this data most likely valuable to others in industry that are driving generative AI efforts, but also to an organization that is preparing for its future.
Rob Zelinka, CIO of Jack Henry & Associates
We’ve seen many cases where an organization is held back by its data infrastructure: it’s slow, expensive, and very disparate. Manufacturing companies often struggle with consolidation of their data stores since they are actively acquiring such great volumes of data, often in the petabyte scale. Data, like any other asset, derives a large part of its immediate value from just how liquid it is. If it’s locked away in paper archives, for example, it can’t at all be leveraged in the cloud to derive AI insights, train AI models, or contribute to an organization’s operational intelligence.
Regardless of the end-services that wind up consuming the data, whether they be Large Language Models (LLMs) for training, Machine Learning (ML) models for inference, AI-as-a-Service APIs (e.g. ChatGPT), it is important that an organization focuses on utilizing the data it already has rather than investing in new developments. We recognize these two benefits as most significant:
- Leveraging Proprietary Data: The focus should be on utilizing proprietary data effectively with advanced AI models provided by the market unless the use case specifically calls for it.
- Unlocking New Efficiencies and Insights: With high-quality data, companies can unlock new operational efficiencies and gain deeper insights, which are crucial for maintaining a competitive edge in the market.
We’ve seen those in the manufacturing industry benefit from preparing their data for AI, even if they aren’t necessarily using AI yet.
The Benefits of Data Preparation for Manufacturing
Responsiveness
We worked with an automotive telematics manufacturing supplier that stored petabytes of sensor log data in Azure cloud storage and struggled to identify safety faults in their sea of disparate data which encompassed 250,000 CSV files of different schemas, with over 30 trillion rows. Modernizing their infrastructure, we enabled them to return results to their client in a more time-efficient and resource-frugal manner. In their current state at the time, it was taking them weeks to load data into MySQL and query for VINs that exhibited safety faults. We built a data pipeline around their data lake and sped up the processing dramatically, allowing IOT device data to be sent straight to cloud storage where our processing pipeline could parse the data and return affected VINs in a matter of minutes.
We optimized their data lake, reducing the processing time from weeks to minutes, and shaved 50% off the cost of their data infrastructure, putting the company in an ideal spot for development of predictive maintenance artificial intelligence models and providing higher levels of responsiveness.
You can read more about this case study here.
Accessibility
Document extraction, specifically Optical Character Recognition (OCR), has tremendous implications for manufacturing. As an example of what OCR can achieve, for one of our clients, we took a database of historical archive images sitting in cloud storage and ran them through a document extraction engine to build a structured dataset with quality OCR to enhance the searchability of the database for end users.
To improve processing efficiency, we migrated their system to Amazon Web Services (AWS), utilizing a serverless architecture. This allowed us to distribute processing across hundreds, or even thousands of machines, working in parallel.
The results were outstanding. The clients experienced a 10X improvement in OCR quality, successfully converted previously unreadable pages into fully legible text. Thier technology costs decreased 94%, reducing from $300 per batch to $20 per batch. In addition, implementation speed improved 99%, from three weeks to one hour.
The new accessibility of the troves of records are likely to be incredibly valuable for the historian AI models of the future.
You can read more about this case study here.
Efficiency
Predictive maintenance reduces costs, drives efficiency, and enables more accuracy for manufacturers. A key challenge in adopting this technology is integrating disparate datasets. For example, we recently worked with a client to link parts eligible for maintenance and their maintenance time estimates, employing a sentence transformer model to link part names with maintenance task titles semantically. Despite inconsistent part naming and disparate datasets, the client can now automatically project time estimates for maintenance identified at the part level, resulting in great efficiencies with estimated hundreds of hours of time saved.
You can read more about this case study here.
Looking Ahead
In summary, as we move further into the AI-driven future, the manufacturing organizations that prioritize effective data management and governance today will be the leaders of tomorrow. Investing in data infrastructure is not just a technical necessity but a strategic imperative. Ensuring your data is well-organized, accessible, and high-quality will allow you to fully leverage AI technologies to drive your business forward.
Along the way, as outlined in the use cases above, we expect to help our manufacturing clients with more fundamental data issues surrounding storage cost optimization, processing, and machine learning preparation. After these building blocks are in place, organizations can move forward, when and if they are ready, for their AI journey.