CEO of InfluxData, a leading time series platform, board member for One Heart Worldwide and board advisor for Lucidworks and The Fabric.
getty
Everything around us is getting smarter.
Sensors have become ubiquitous in our daily lives. They cover our cars and factory floors. They’re present throughout smart homes and smart cities. Sensors make the world around us smarter and more connected every day, and the language they speak—time series data—holds the key to fueling AI-driven innovation.
Real-world AI, or applying AI to the physical world, requires transforming data into valuable intelligence, making sensor-produced raw data foundational for intelligent, autonomous systems. Whether it’s alerting a car when there’s an object on the road, notifying a plant supervisor that a machine may overheat or letting us know when it’s time to stand up and take a walk, the process of real-time data collection, transformation and response at its core is building intelligence.
Transforming time series data into intelligence is table stakes for creating and leveraging AI models, but it’s also a complex process. If you can harness this data at scale, you can create intelligent, self-healing systems that continuously become smarter over time.
Instrumentation: The Cornerstone AI Systems
Any device connected to the internet produces a constant stream of data. AI algorithms leverage this data to analyze historical patterns, model behaviors and even make predictions. That’s what the world is trying to do with AI—build intelligence through automated data collection so systems can predict outcomes, react to those outcomes and resolve them.
On a larger scale, if we can build intelligence around each sensor and extract insights in near real time with increasing precision, we can create intelligent—and eventually—autonomous systems.
Pulling this off requires high-resolution data—sometimes down to nanosecond precision—for real-time analytics. While not all systems require this level of granularity, having it offers benefits and enables users to find new applications for it over time.
Instrumenting systems and handling the large volume of data they create presents a challenge, however. Many organizations use analytics tools alongside their databases to visualize data according to their unique business use cases. When used effectively, this combination promotes the development of highly intelligent systems and offers opportunities for predictive analytics, forecasting and other types of real-time analysis.
Managing The Data That Powers AI
A well-known adage in tech is that AI is only as strong as the data that powers it. While connected devices and software produce large volumes of highly granular data that strengthens AI systems, managing all that data creates several challenges, including:
• Managing Cardinality: When it comes to time series data, cardinality refers to a metric that shows a high number of unique or distinct values over time. Think of a sensor that measures 40 separate data points every millisecond; that sensor produces high-cardinality data that grows exponentially every minute. The trend in addressing the challenge of managing high-cardinality data is utilizing a columnar database, which supports near-real-time querying while reducing the amount of disk space necessary to store that data. Columnar databases manage data differently than row-based, relational databases, but the underlying technology should be familiar to most developers. Users need to understand the characteristics of data workloads to optimize and improve their data processing.
• Transforming And Evicting Data: The large amount of data that sensors produce can be cost-prohibitive to store, so organizations need a strategy for handling older data. The first step is transforming the data. Returning to the example of a sensor that produces 40 separate data points each millisecond, that level of granularity likely isn’t necessary a few months down the road. Instead, an organization might summarize second-by-second analysis (rather than every millisecond) and then evict the rest to keep storage costs down.
• Compressing Stored Data: After transforming it, organizations still have a large amount of time series data on their hands. Shifting to columnar storage can allow for better data compression ratios, reduced file size on disk and better query performance. By aligning the data’s on-disk representation with its in-memory counterpart, moving data between disk and RAM is more efficient, allowing consistent query performance while reducing costs.
To unlock the full potential of sensor data, data platforms can provide a scalable, reliable and secure environment for storing, managing and analyzing the volume, variety and velocity of IoT and sensor data. These platforms must support data processing in real time to enable businesses to build and deploy AI models that can anticipate future outcomes.
It’s also important to understand the topology of your system. Where is the data coming from and where does it need to go? What are the various layers involved and where are they located? Many real-world AI applications combine edge devices and cloud-based platforms. Organizations need to understand the resources and limitations of their edge devices and optimize their performance and connectivity with costs.
Having tools that can collect data from a diverse range of sources and process it into a standardized format factors into system efficiency. Tech stacks may benefit from open-source tools and technologies that make it easier to integrate with virtually any other technology, providing greater control over data and extending its utility into new areas with less effort than what’s required for proprietary solutions.
Turning Intelligence Into Insights: A Continuous Effort
A significant challenge you can’t overlook is the fact that turning data into intelligence is a continuous process. It’s not like performing a one-time data transformation—time series data is produced continuously and never stops. It must constantly be refined, validated and transformed into the intelligence that powers real-world AI systems.
As data evolves and new data is introduced, AI models must be updated to remain current and relevant. Models must constantly adapt to new scenarios; continuous monitoring can ensure these situations are handled as they arise. Further, it’s important to regularly analyze AI model performance to verify it’s working correctly, particularly as new data is introduced.
As we move toward the sensor-driven future, our ability to anticipate future outcomes through AI will be a game-changer. It will enable businesses to make data-driven decisions, reduce risks, enhance customer experiences and optimize operations. In this era of sensor-driven insights, prioritizing time series data strategies and AI can help businesses lead the way, transform industries and shape the future.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
