Streaming data observability startup Datorios Ltd. today announced the immediate availability of a new real-time observability platform for the big-data processing framework Apache Flink.

With the new platform, companies will benefit from what the startup claims are previously unseen insights relating to streaming data processing. These insights can aid in the creation of new, real-time artificial intelligence systems that can be fully audited to ensure they don’t misbehave, the startup said.

Datorios’ founders say they have applied years of experience in the research and development of real-time military intelligence systems to create their new product, with the end goal being to help developers create more advanced AI applications that can operate in real time. The basic idea is that Datorios provides more visibility into the information that’s being fed into real-time AI systems using Apache Flink-based data pipelines.

It can be thought of as a “flight recorder for real-time AI,” the company said. By enhancing visibility into real-time streaming data, the startup reckons, it can help to simplify tasks such as troubleshooting and auditing real-time AI applications.

Apache Flink has emerged as a vital tool for creating real-time AI. It’s an open-source big data processing framework that’s widely used in industries such as video streaming, e-commerce and financial services. It enables data to be processed the moment it is created, and it has use cases in data analytics, transaction processing and AI models that need to process up-to-date information.

The Apache Flink framework is often compared to Apache Spark, which also processes information in real-time. However, Flink’s main advantage is it can perform compute on data in motion, as it’s being processed, resulting in what its creators say is true real-time data processing. That’s different from Apache Spark, which instead processes data in small chunks using a technique known as “microbatching,” where information must be accumulated in small batches before being computed. The microbatching approach means slightly higher latency, which can be a deal breaker for some types of AI systems.

Fixing real-time data disruption

Real-time AI systems rely on continuous streams of up-to-the-second information in order to automate processes such as fraud detection, product recommendation engines and vehicle routing. But although Apache Flink’s ability to process real-time information is virtually unrivaled, the streaming data it handles can still be disrupted. For instance, it can arrive in the wrong order, or it might arrive more than once, or in an unexpected format. When this happens, it can cause big problems for AI systems that need the information to be accurate and immediately accessible.

Datorios aims to solve these problems by increasing visibility into these real-time data streams, so developers can be sure their AI systems are processing the right information they need.

More specifically, the startup said its platform delivers insights into event visualization, showing the journey of the data as it flows through each processing operation. It also has event search capabilities, giving users the ability to trace specific events for auditing and compliance purposes. Its state monitoring features allow users to see how the state of data changes during each processing step, while the window analysis tools provide insights into areas such as the order in which data is processed, its lateness, discarded records and so on.

By providing these novel data observability capabilities, Datorios says it’s carving out its very own niche within the wider Apache Flink ecosystem as the only tool that can identify streaming data disruptions.

Datorios co-founder and Chief Executive Ronen Korman said the most innovative companies today are keen to use AI to run their business operations on auto-pilot, and to do that they need to ensure the accuracy and reliability of the data that powers these systems. “Datorios is their flight data recorder, showing them exactly how a system behaved and why at any point in time,” Korman explained. “Just as with aircraft, unexpected conditions can affect performance; insights from Datorios makes that clear so they can continuously improve their AI.”

Datorios said it’s making its platform available for free with almost no limitations for new users, in order to showcase how it can support their real-time AI systems.

Images: Datorios