Calling Durable Function from ADF Pipeline with Monitoring and Email Notifications

Azure Durable Functions are an excellent way to manage stateful workflows in Azure. When combined with Azure Data Factory (ADF), you can orchestrate complex data processing pipelines with ease. In this article, we'll discuss how to trigger a Durable Function from an ADF pipeline, monitor its progress, and send an email notification upon successful completion.

1. Overview of the Process

The process involves several key steps:

1. Trigger the Durable Function using an HTTP start request from the ADF pipeline.
2. Wait for a few seconds to allow the webhook URL to be generated.
3. Call the webhook URL to monitor the status of the Durable Function.
4. Check the status of the Durable Function periodically.
5. Send a success email upon completion or a failure email if an error occurs.

The flowchart below illustrates the workflow:

2. Setting Up the Durable Function

First, we need to set up a Durable Function in Azure. Here's a simple example:

[FunctionName("DurableFunctionExample")]
public async Task<IActionResult> Run(
    [HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req,
    [DurableClient] IDurableOrchestrationClient starter,
    ILogger log)
{
    string instanceId = await starter.StartNewAsync("YourOrchestrationFunction", null);
    log.LogInformation($"Started orchestration with ID = '{instanceId}'.");
    return starter.CreateCheckStatusResponse(req, instanceId);
}

This function triggers an orchestration function and returns the status response, which includes a webhook URL that can be used to monitor the progress.

3. Configuring ADF to Call the Durable Function

In the ADF pipeline, add a Web Activity to call the Durable Function. Here’s how you can set it up:

{
    "name": "TriggerDurableFunction",
    "type": "WebActivity",
    "typeProperties": {
        "url": "https://<your-function-app-name>.azurewebsites.net/api/DurableFunctionExample",
        "method": "POST",
        "headers": {
            "Content-Type": "application/json"
        },
        "body": {
            "parameter1": "@pipeline().parameters.parameter1"
        }
    }
}

This activity triggers the Durable Function and retrieves the webhook URL.

4. Adding a Delay to Generate the Webhook URL

After triggering the Durable Function, add a Wait activity to introduce a small delay (e.g., 2 seconds) to ensure the webhook URL is generated:

{
    "name": "WaitForWebhookURL",
    "type": "WaitActivity",
    "typeProperties": {
        "waitTimeInSeconds": 2
    }
}

5. Calling the Webhook URL and Monitoring Status

Next, add another Web Activity to call the webhook URL and monitor the status of the Durable Function:

{
    "name": "MonitorDurableFunction",
    "type": "WebActivity",
    "typeProperties": {
        "url": "@{activity('TriggerDurableFunction').output.statusQueryGetUri}",
        "method": "GET"
    }
}

This will continuously check the status of the Durable Function until it completes.

6. Sending Email Notifications

Finally, based on the result of the Durable Function, configure conditional logic in the ADF pipeline to send either a success or failure email. Use Web Activities to send the email via an HTTP endpoint:

{
    "name": "SendSuccessEmail",
    "type": "WebActivity",
    "typeProperties": {
        "url": "https://<your-email-service-url>/send",
        "method": "POST",
        "headers": {
            "Content-Type": "application/json"
        },
        "body": {
            "to": "@pipeline().parameters.emailReceiver",
            "subject": "Pipeline Execution Successful",
            "message": "The Durable Function completed successfully."
        }
    }
},
{
    "name": "SendFailureEmail",
    "type": "WebActivity",
    "typeProperties": {
        "url": "https://<your-email-service-url>/send",
        "method": "POST",
        "headers": {
            "Content-Type": "application/json"
        },
        "body": {
            "to": "@pipeline().parameters.emailReceiver",
            "subject": "Pipeline Execution Failed",
            "message": "The Durable Function encountered an error."
        }
    }
}

7. Conclusion

Integrating Azure Durable Functions with ADF pipelines provides a powerful way to handle complex workflows. By following this guide, you can set up a robust system to monitor your Durable Functions and notify stakeholders of the outcomes, ensuring smooth and efficient data processing pipelines.

This approach not only improves the maintainability of your pipelines but also ensures that issues are promptly communicated, reducing downtime and enhancing overall productivity.