Azure Data Factory Integration
Pipeline run metrics, activity duration, and integration runtime health for Azure Data Factory. Detect failing ETL pipelines and IR capacity issues before they delay your data consumers.
How It Works
Create Service Principal
Register an Azure Service Principal with Monitoring Reader on your Data Factory resource group. TigerOps pulls pipeline run status, activity durations, and IR metrics from Azure Monitor.
Enable ADF Diagnostics
Configure Diagnostic Settings on your Data Factory to stream PipelineRuns, ActivityRuns, TriggerRuns, and SandboxPipelineRuns categories to a Log Analytics workspace.
Map Critical Pipelines
Tag your mission-critical pipelines in TigerOps with SLA labels. The integration groups pipeline run metrics by pipeline name and data domain for focused observability.
Set Duration SLO Alerts
Define maximum allowed pipeline run duration and failure rate thresholds. TigerOps alerts when a pipeline exceeds its SLO and identifies the specific activity causing the delay.
What You Get Out of the Box
Pipeline Run Metrics
Total pipeline runs, succeeded, failed, and cancelled counts with run duration percentiles per pipeline. Track SLO compliance for your ETL, ELT, and data movement pipelines.
Activity Duration Analysis
Per-activity execution duration, queue time, and transfer rate for Copy, Mapping Data Flow, and custom activities. Identify the slowest activities in complex multi-step pipelines.
Integration Runtime Health
Azure IR and Self-hosted IR availability, node count, CPU and memory utilization, and concurrent job capacity. Detect IR resource pressure before it causes pipeline queuing.
Trigger Execution Tracking
Schedule, tumbling window, storage event, and custom event trigger fire counts and failure rates. Monitor trigger health to ensure pipelines start on time and detect missed trigger events.
Data Movement Throughput
Copy activity throughput in MB/s, rows per second, and bytes read and written per Copy activity run. Benchmark data movement performance and identify network or throttling bottlenecks.
AI Pipeline Failure Root Cause
TigerOps AI correlates ADF pipeline failures with source system availability, network connectivity, and IR resource saturation to surface the most likely root cause for each failed run.
ADF Diagnostic Settings Setup
Enable all pipeline and activity run log categories for full Data Factory visibility in TigerOps.
#!/bin/bash
# TigerOps — Azure Data Factory diagnostic setup
ADF_NAME="my-data-factory"
RESOURCE_GROUP="my-resource-group"
WORKSPACE_ID="/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.OperationalInsights/workspaces/tigerops-workspace"
ADF_RESOURCE="/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.DataFactory/factories/${ADF_NAME}"
# Enable all ADF diagnostic categories
az monitor diagnostic-settings create \
--name tigerops-adf \
--resource "${ADF_RESOURCE}" \
--workspace "${WORKSPACE_ID}" \
--metrics '[{"category":"AllMetrics","enabled":true}]' \
--logs '[
{"category":"PipelineRuns", "enabled":true},
{"category":"ActivityRuns", "enabled":true},
{"category":"TriggerRuns", "enabled":true},
{"category":"SandboxPipelineRuns", "enabled":true},
{"category":"SandboxActivityRuns", "enabled":true}
]'
echo "ADF diagnostics enabled for: ${ADF_NAME}"
echo "Pipeline run events will appear in TigerOps within 5 minutes"Common Questions
Does TigerOps support Mapping Data Flows in Azure Data Factory?
Yes. Mapping Data Flow activity runs emit execution metrics including data read and written, rows processed, and transformation stage duration. TigerOps captures these from the ActivityRuns diagnostic log.
How does TigerOps monitor Self-hosted Integration Runtimes?
For Self-hosted IRs, TigerOps reads node availability, CPU and memory usage, and concurrent job counts from Azure Monitor. These metrics are available at 1-minute granularity and alert when IR capacity is exhausted.
Can TigerOps alert me when a scheduled pipeline did not run?
Yes. TigerOps tracks expected pipeline run frequency based on your trigger schedule. If a scheduled pipeline has not run within a configurable window past its expected start time, TigerOps fires a missing run alert.
Does TigerOps support Azure Synapse Analytics pipelines as well?
Yes. Synapse Analytics uses the same pipeline and activity infrastructure as Azure Data Factory. TigerOps monitors Synapse pipelines with the same integration, with Synapse workspace metrics tracked separately.
How does TigerOps correlate ADF pipeline failures with source system issues?
When TigerOps is also monitoring the source systems such as Azure SQL Database, Blob Storage, or an on-premises database via SHIR, it links ADF pipeline failures to concurrent issues in the source system, reducing investigation time significantly.
Keep Every Data Pipeline Running on Schedule
Pipeline SLO tracking, activity analysis, and integration runtime health for Azure Data Factory. Connect in 5 minutes.