Topic 4 of 6
Real-time analytics
With smart devices and internet access everywhere, a lot of data arrives as a constant stream that can be processed instantly. The exam asks you to tell batch processing from stream processing and to identify Microsoft's services for real-time analytics.
Contents
Batch and stream processing
Batch
Records are collected and stored, then processed together in a single operation. The timing can be a scheduled interval (every hour), when a certain amount of data arrives, or after some other event.
Streaming
A data source is constantly monitored and each piece of data is processed in real time as it arrives, with no waiting for the next batch.
The car example
Counting cars in batches is like parking them in a lot and counting them at rest: there's no result until you have the batch. With streaming you count them as they pass, and you can aggregate by intervals, such as cars per minute.
- Batch example: credit card companies send one monthly bill with all purchases, not one per purchase.
- Streaming examples: tracking the stock market and rebalancing portfolios; analyzing player interactions in a game; recommending properties based on phone location; triggering alarms on smoke or heat.
| Batch | Advantage | Disadvantage |
|---|---|---|
| Capacity | Processes large volumes at a convenient time | All input must be ready before processing starts |
| System usage | Scheduled during off-peak times, like overnight | There's a delay between input and results |
| Errors | — | An error, even a small one, can stop the whole batch |
Differences between batch and streaming
| Batch | Streaming | |
|---|---|---|
| Scope | All the data in the dataset | The most recent data or a rolling window (the last 30 s) |
| Size | Large datasets | Individual records or micro-batches |
| Latency | Typically a few hours | Seconds or milliseconds |
| Analysis | Complex analytics | Simple responses, aggregates, rolling averages |
Combining both: lambda architecture
Many solutions mix batch and streaming to analyze both historical and real-time data. Even without real-time analysis, streaming is often used to capture data and store it for later batch processing.
Events from a streaming source are captured in real time.
Data from other sources is ingested into a store (often a data lake) for batch processing.
If real-time analytics isn't needed, the captured data is written to the store for later processing.
If it is needed, a streaming technology prepares it, filtering or aggregating over temporal windows.
Non-streaming data is batch processed and persisted in an analytical store (a data warehouse) for historical analysis.
Stream processing results can also be persisted in that store.
Analytics and visualization tools present the real-time and historical data.
Other common architectures are delta and kappa; the latter removes the batch layer and treats all data as a continuous stream that's replayed when reprocessing is needed.
Elements of a streaming solution
Event
A sensor signal, a social media post, a log entry
Source
A folder, a table, or a queue that ensures ordering and once-only processing
Processing
A perpetual query that filters, projects, or aggregates over temporal windows
Sink
A file, a table, a real-time dashboard, or another queue
Fabric Real-Time Intelligence
A set of tools built into Microsoft Fabric for ingesting, processing, and analyzing streaming data, from data arrival through visualization and automated action, all in the same workspace.
Eventstreams
Ingest, route, and transform streaming data.
Eventhouse
A database optimized for time-series and event data, queried with KQL.
Real-Time Dashboards
Live data visualization.
Activator
Triggers automated actions when data meets defined conditions.
Real-Time hub
A centralized catalog to discover, add, explore, and share streaming data across the organization.
AI assistance
Copilot generates KQL queries from natural language questions.
KQL (Kusto Query Language) is a query language designed for fast log and telemetry analysis. It's also the language of Azure Data Explorer, a managed platform for querying timestamped data.
Spark Structured Streaming
Apache Spark splits work across many machines (a cluster) and supports Python, Scala, Java, R, and SQL, for both batch and streaming. In Azure it's available in Microsoft Fabric and Azure Databricks. Spark Structured Streaming is Spark's library for streaming data:
It connects to a streaming source, such as Azure Event Hubs, a folder of files, or a network feed.
Spark reads the data into a dataframe: a table that keeps filling as events arrive.
You write a query on the dataframe, for example events per minute or a rolling average.
The results are written to a sink: a file, a database, or a dashboard.
Delta tables for batch and streaming
The same Delta table can be a streaming sink and a source for batch queries, so you don't need separate storage for historical and live data. Delta Lake adds reliability (failed writes don't corrupt data) and schema enforcement.
Azure Stream Analytics
A PaaS solution for defining streaming jobs that ingest data from a source, apply a perpetual query, and write the results to an output. It serves standalone or hybrid scenarios outside Fabric.
- It uses a SQL-based query language extended with temporal constraints; it also has a no-code editor.
- Inputs: Event Hubs and IoT Hub for streaming, Blob Storage or Data Lake Storage Gen2 for historical data, and reference data to join with the stream.
- Outputs: Blob Storage, Azure SQL Database, Data Lake, Cosmos DB, Event Hubs, or Power BI for real-time visualization.
- It guarantees exactly-once processing (with selected outputs) and at-least-once delivery; it can also run on IoT Edge.
- You pay for the streaming units consumed.
| Window | How it works |
|---|---|
| Tumbling | Fixed time segments that don't repeat or overlap; each event is in only one window |
| Hopping | Hop forward by a fixed period and can overlap; an event can be in several |
| Sliding | Output results only when an event enters or exits the window |
| Session | Group events that arrive at similar times and filter out periods with no data |
| Snapshot | Group events that have the same timestamp |
Sources and sinks
Sources
- Azure Event Hubs: high-volume event ingestion; ordering within each partition, at-least-once delivery, and Apache Kafka compatibility.
- Azure IoT Hub: like Event Hubs, but optimized for IoT devices.
- Data Lake Storage Gen2: common in batch, it can also be a streaming source.
- Apache Kafka: open-source ingestion, often used with Spark.
Sinks
- Event Hubs: queue processed data for downstream processing.
- Data Lake Storage Gen2, OneLake, or Blob Storage: persist results as files.
- Azure SQL Database, Databricks, or Fabric: persist results in tables.
- Power BI: real-time visualizations.
Exam scenarios
| Situation | Answer | Why |
|---|---|---|
| Generate a monthly card bill with all of the month's purchases | Batch processing | They're grouped and processed together. |
| Trigger alarms and unlock doors as soon as smoke is detected | Stream processing | An immediate, real-time response. |
| Count the cars that pass per minute | Streaming with temporal windows | Aggregates over time intervals. |
| Ingest millions of events per second with Kafka compatibility | Azure Event Hubs | A managed streaming platform. |
| Ingest telemetry from IoT devices | Azure IoT Hub | Optimized for IoT devices. |
| Live dashboards and automatic alerts when a condition is met, inside Fabric | Real-Time Intelligence (Real-Time Dashboards and Activator) | Tools built into Fabric. |
| Extend existing Spark work to streaming | Spark Structured Streaming | The same technology for batch and streaming. |
| A standalone PaaS streaming job, outside Fabric, with a SQL query | Azure Stream Analytics | Input, perpetual query, and output. |
Common exam traps
❓ Is stream processing best for complex analytics over all historical data?
No. Complex analytics over the whole dataset uses batch; streaming works with recent data and simple calculations.
❓ What latency does batch processing typically have?
A few hours; streaming, seconds or milliseconds.
❓ Which Real-Time Intelligence component triggers automated actions?
Activator.
❓ Which language do you query an Eventhouse with?
KQL (Kusto Query Language).
❓ Is Azure Stream Analytics IaaS?
No. It's PaaS: you define the job and Azure manages the infrastructure.
❓ In a tumbling window, can an event belong to several windows?
No. Tumbling windows don't overlap; hopping windows can.
❓ Which architecture removes the batch layer?
The kappa architecture.
Did you get this topic?
Test what you just learned
A manufacturer wants to see its machines' temperature on a live dashboard and automatically receive an alert when it crosses a threshold, all inside Microsoft Fabric. What does it use?
Ready to test "Real-time analytics"?
Original questions, each with a full explanation.
Start a mock exam