Most Important Azure Data Factory (ADF) Interview Questions & Answers
If you're preparing for an Azure Data Factory (ADF) interview, here are some commonly
asked questions along with their answers to help you succeed!
1. What is Azure Data Factory (ADF)?
Answer:
Azure Data Factory is a cloud-based ETL (Extract, Transform, Load) and data integration
service used to orchestrate and automate data workflows across different sources. It helps
in moving, transforming, and loading data into cloud-based storage and analytics solutions
like Azure Data Lake, Azure SQL, and Synapse Analytics.
2. What are the key components of Azure Data Factory?
Answer:
ADF consists of the following key components:
✅Pipelines – Group of activities that perform data movement & transformation
✅Activities – Steps in a pipeline (e.g., Copy, Data Flow, Stored Procedure)
✅Datasets – Define the structure of data sources/destinations
✅Linked Services – Connections to external data stores (e.g., ADLS, SQL, APIs)
✅Integration Runtime (IR) – Compute engine to execute activities
✅Triggers – Schedule or event-driven execution of pipelines
3. What are the different types of Integration Runtimes (IR) in ADF?
Answer:
Azure Data Factory supports three types of Integration Runtimes:
🔹 Azure IR – Used for cloud-based data movement & transformation
🔹 Self-Hosted IR – Used for on-prem or private network data access
🔹 Azure-SSIS IR – Used to run SSIS packages in the cloud
4. What are the different types of triggers in ADF?
Answer:
ADF provides three types of triggers:
1️⃣Schedule Trigger – Runs pipelines on a specific time-based schedule
2️⃣Tumbling Window Trigger – Runs pipelines at fixed intervals with dependency tracking
3️⃣Event-Based Trigger – Runs pipelines when a file is added/deleted in ADLS/Blob Storage
5. How do you move data from On-Prem to Azure using ADF?
Answer: To move data from an on-premises database to Azure, follow these steps:
✅Install Self-Hosted Integration Runtime (SHIR) on an on-prem server
✅Create a Linked Service in ADF to connect to the on-prem database
✅Use Copy Data Activity to move data to Azure (ADLS, Blob, SQL, Synapse)
✅Schedule pipeline execution using triggers
6. How does Azure Data Factory handle failures?
Answer : ADF provides several error-handling mechanisms:
✔ Retry Policy – Set retry count & interval for transient failures
✔ Logging & Monitoring – Use Azure Monitor & Log Analytics
✔ Custom Error Handling – Implement If Condition & Web Activity for alerts
✔ Try-Catch Logic – Use Execute Pipeline & Error Handling
7. What are Mapping Data Flows in ADF?
Answer : Mapping Data Flows provide a no-code, visual way to perform ETL transformations using
Apache Spark in ADF.
Features include:
✅Joins, aggregations, filtering, pivoting
✅Schema drift support (Handles changing schema)
✅Auto-scaled Spark execution
📌Example Use Case: Cleansing & transforming raw data before loading it into Azure
Synapse Analytics.
8. What is the difference between ADF and SSIS?
Answer :
🔹 ADF is a modern, scalable, cloud-native alternative to SSIS for hybrid data movement &
orchestration.
9. How can you secure data movement in ADF?
Answer :
✔ Use Managed Identity Authentication instead of storing credentials
✔ Encrypt data at rest & in transit (TLS 1️.2️, Azure Key Vault)
✔ Use Private Endpoints to restrict data access
✔ Monitor data pipelines using Azure Security Center
10. What is the difference between Copy Data Activity and Data Flow?
Answer :
✔ Use Copy Data for basic ETL
✔ Use Data Flow for complex transformations
🔹 Scenario-Based ADF Questions!
1. How would you design an ADF pipeline for daily sales reporting?
Answer:
1️⃣Source: Extract data from on-prem SQL Server
2️⃣Transformation: Use Mapping Data Flow for data cleansing
3️⃣Destination: Load transformed data into Azure Synapse
4️⃣Trigger: Schedule a daily trigger at midnight
5️⃣Monitoring: Enable Azure Monitor alerts for failures
2. How do you implement incremental data load in ADF?
Answer:
✅Use Watermark Columns – Track last modified timestamps
✅Query Only New/Changed Records – Use WHERE LastUpdated > @LastRunTime
✅Store Last Run Time – Save in Azure SQL Table or Blob Metadata
📌Example Query for SQL Incremental Load:
SELECT * FROM SalesData
WHERE LastUpdated > (SELECT MAX(LastRunTime) FROM ADF_Metadata)
🔹 Summary: Key ADF Topics to Prepare
1. ETL Pipeline: Ingesting and Transforming Sales Data
🔹 Scenario:
A retail company wants to ingest sales data from on-prem SQL Server, clean and transform
it, and store it in Azure Synapse Analytics for reporting.
🔹 Solution Approach:
✅Extract: Copy sales data from SQL Server to Azure Data Lake (ADLS Gen2️).
✅Transform: Use Mapping Data Flow or Databricks for cleansing.
✅Load: Store the transformed data into Azure Synapse for Power BI reporting
🔹 Step-by-Step Implementation:
1️⃣Create a Linked Service to connect to SQL Server (on-prem).
2️⃣Use Self-Hosted IR to securely move data from on-prem to cloud.
3️⃣Copy Data Activity → Move data to Azure Data Lake Storage (ADLS Gen2️).
4️⃣Mapping Data Flow → Clean missing values, format dates, and filter records.
5️⃣Load Data into Azure Synapse Analytics for BI reporting.
6️⃣Schedule Pipeline Execution using Schedule Trigger (runs daily at midnight).
📌Example Query for Incremental Load:
SELECT * FROM SalesData
WHERE LastUpdated > (SELECT MAX(LastRunTime) FROM ADF_Metadata)
✔ Outcome: Automated ETL pipeline keeps data updated in Azure Synapse for Power BI
reports.
2. Data Migration: Moving Data from On-Prem SQL Server to Azure
🔹 Scenario:
A financial company wants to migrate historical data from on-prem SQL Server to Azure
SQL Database.
🔹 Solution Approach:
✅Extract: Read data from on-prem SQL Server.
✅Transfer: Use Self-Hosted IR to securely move data to Azure.
✅Load: Store data in Azure SQL Database with incremental updates.
🔹 Step-by-Step Implementation:
1️⃣Install Self-Hosted Integration Runtime (SHIR) on an on-prem machine.
2️⃣Create a Linked Service to connect SQL Server and Azure SQL Database.
3️⃣Use Copy Data Activity to transfer data.
4️⃣Enable Incremental Load using Watermark Columns.
5️⃣Monitor & Log Pipeline Runs using Azure Monitor.
📌Example Query for Incremental Load (Watermarking Approach):
SELECT * FROM Transactions
WHERE LastUpdated > (SELECT MAX(LastProcessedDate) FROM MigrationLog)
✔ Outcome: On-prem data is seamlessly migrated and updated in Azure SQL.
3. Real-Time Data Processing from IoT Devices
🔹 Scenario:
A manufacturing company wants to process IoT sensor data in real-time and store it in
Azure Data Lake for analytics.
🔹 Solution Approach:
✅Ingest Data from IoT Hub using Event-Based Triggers in ADF.
✅Transform Data in Databricks – Aggregate, filter, and cleanse data.
✅Store Processed Data in Delta Lake for analytics.
🔹 Step-by-Step Implementation:
1️⃣Use Event-Based Trigger to detect new IoT data arrival in ADLS.
2️⃣Copy Raw IoT Data to Azure Databricks for processing.
3️⃣Use Databricks Notebooks to filter anomalies, aggregate sensor readings.
4️⃣Store Data in Delta Lake (Optimized for analytics).
5️⃣Use Power BI for Real-Time Dashboards
📌Example PySpark Code for IoT Data Processing in Databricks:
from pyspark.sql.functions import avg, col
df = spark.read.json("dbfs:/mnt/iot/raw/")
df_cleaned = df.filter(col("temperature").isNotNull()) \
.groupBy("device_id") \
.agg(avg("temperature").alias("avg_temperature"))
df_cleaned.write.format("delta").save("dbfs:/mnt/iot/processed/")
✔ Outcome: Real-time IoT data is processed and visualized in Power BI dashboards
4. Automating Data Ingestion from REST APIs
🔹 Scenario:
A healthcare company needs to fetch patient records from a third-party API, process them,
and store them in Azure SQL Database.
🔹 Solution Approach:
✅Extract Data from API using Web Activity in ADF.
✅Transform Data in Mapping Data Flow (clean, remove duplicates).
✅Load Data into Azure SQL Database for reporting.
🔹 Step-by-Step Implementation:
1️⃣Create a Web Activity in ADF to call REST API (GET request).
2️⃣Store JSON response in ADLS for staging.
3️⃣Use Mapping Data Flow to parse and clean API data.
4️⃣Use Copy Data Activity to store data in Azure SQL.
5️⃣Schedule Pipeline Execution using Triggers.
📌Example API Request in ADF Web Activity:
{
"url": "https://api.example.com/patients",
"method": "GET",
"headers": {
"Authorization": "Bearer XYZ1️2️3️"
}
}
✔ Outcome: API data is automatically fetched and stored in Azure SQL for further analysis
5. Processing and Storing Large CSV Files in ADLS
🔹 Scenario:
A logistics company receives large CSV files daily containing shipment data. The goal is to
store them in Azure Data Lake and optimize for fast querying.
🔹 Solution Approach:
✅Ingest CSV Files using Event-Based Triggers in ADF.
✅Convert CSV to Parquet Format for better performance.
✅Store in Azure Data Lake & Query with Synapse.
🔹 Step-by-Step Implementation:
1️⃣Use Event-Based Trigger to detect new CSV files in ADLS.
2️⃣Copy Data Activity to move raw CSV files to staging folder.
3️⃣Use Mapping Data Flow to convert CSV to Parquet format.
4️⃣Store Processed Data in Azure Data Lake (ADLS Gen2️).
5️⃣Query Data Using Azure Synapse Serverless SQL.
📌Example Query to Read Parquet Data in Synapse:
SELECT * FROM OPENROWSET(
BULK 'https://datalake.blob.core.windows.net/processed/shipments.parquet',
FORMAT='PARQUET'
)
AS Shipments
✔ Outcome: Optimized Parquet files allow faster queries and reduced storage costs.
Comments
Post a Comment