Microsoft added three new big data services to its Azure cloud platform yesterday as part of its ongoing efforts to make Azure a leading platform for big data services and to make it ready to tackle the Internet of Things (IoT).
"Every day, IoT is fueling vast amounts of data from millions of endpoints streaming at high velocity in the cloud," says Joseph Sirosh, corporate vice president of Machine Learning at Microsoft. "Examples of streaming analytics can be found across many businesses, such as stock trading, fraud detection, identity protection services, sensors, web clickstream analytics and alerts from CRM applications. In this new and fast-moving world of cloud and devices, businesses can no longer wait months or weeks for insights generated from data."
Two of the services -- Stream Analytics and Azure Event Hubs -- are intended to help customers process data from devices and sensors in the IoT. Meanwhile the Azure Data Factory service is about information production by orchestrating and managing diverse data, especially in hybrid environments where some data sources reside in the cloud and others are on-premise.
[Related: Microsoft Outlines Vision for Azure With New Services]
Stream Analytics is an event-processing engine, akin to the Apache Storm framework, except Stream Analytics is a managed service. Microsoft announced support for Apache Storm in Azure HDInsight earlier this month.
"The scenarios are similar," says Herain Oberoi, director of product management, Microsoft Data Platform, at Microsoft. "Stream Analytics is more of a managed service whereas Storm is an additional project in the Hadoop ecosystem that you have to manage yourself. For customers that want a more managed service where the actual coding is less, the package management is less, then Stream Analytics makes more sense. We're investing in both."
Stream Analytics is meant to work hand-in-hand with Azure Event Hubs, a scalable service for collecting data from millions of events per second.
[Related: Microsoft Goes All-In on Hybrid Cloud With Azure-in-a-Box]
"With Azure Stream Analytics, businesses can gain insights in real-time from data generated by devices, sensors, infrastructure, applications and other sources," Sirosh says. "Developers can combine streams of data -- such as clickstreams, logs, metering data or device-generated events -- with historic records or reference data. Complementing Stream Analytics, Azure Event Hubs is a highly scalable publish-subscribe ingestor that collects millions of events per second, allowing users to process and analyze data produced by connected assets such as devices and sensors. Stream Analytics provides out-of-the-box integration with Event Hubs -- when connected, these two solutions enable customers to harness IoT by processing and analyzing massive amounts of data in real time."
Beyond the Internet of Things
However, Microsoft isn't focusing on only the Internet of Things. The Azure Data Factory service is intended to address a more fundamental challenge: effectively managing, coordinating and processing data from many different sources -- across geographic locations, on-premises and cloud, whether the data is structured or unstructured -- while dealing with shifting business requirements.
[Related: Microsoft Azure Continues Enterprise Push with Private On-ramps]
"One of the reasons we've seen the need for Data Factory is because so many of our customers are doing things in a hybrid environment," Oberoi says. "The need to support information production across on-premise and cloud has become fairly acute."
Oberoi likens Data Factory to data integration 2.0, providing orchestration between the myriad data stores and processing types that are increasingly becoming available.
"Azure Data Factory helps solve this problem by providing customers with a single place to manage data movement, orchestration and monitoring of diverse data sources, including SQL Server and Azure Blobs, Tables, Azure SQL Database and SQL Server in Azure Virtual Machines," Sirosh says. "Developers can efficiently build data driven workflows to join, aggregate and transform data from local, cloud-based and Internet services, and set up complex data processing systems with little programming."
Stream Analytics and Azure Data Factory are currently available in preview. Azure Event Hubs is now generally available.
Thor Olavsrud covers IT Security, Big Data, Open Source, Microsoft Tools and Servers for CIO.com. Follow Thor on Twitter @ThorOlavsrud. Follow everything from CIO.com on Twitter @CIOonline, Facebook, Google + and LinkedIn.
This story, "Microsoft adds IoT, big data orchestration services to Azure" was originally published by CIO.