Integrating Datadog in .NET Projects Using Serilog: A Complete Guide

In modern cloud-native and microservices architectures, applications are highly distributed, making it challenging to detect failures and performance bottlenecks. Monitoring and analytics tools are essential to ensure the performance and reliability of web services and applications. Among these tools, Datadog stands out as a popular choice. It is a cloud-based monitoring and security platform that provides real-time observability for applications, infrastructure, logs, and security across distributed systems. Datadog integrates with cloud providers, containers, databases, and third-party services, allowing teams to track performance, troubleshoot issues, and ensure system reliability.

This guide provides a step-by-step approach to integrating Datadog with a .NET project using Serilog, a powerful logging library for .NET applications.

Required NuGet Packages
To get started, you will need to install the following NuGet packages in your .NET project:

Serilog.AspNetCore: This package provides Serilog logging for ASP.NET Core applications.

Serilog.Sinks.Datadog.Logs: This Serilog sink sends events and logs directly to Datadog. By default, the sink sends logs over HTTPS for secure transmission.

Step 1: Sign Up for a Datadog Account
Before integrating Datadog, you must have an active Datadog account. If you do not have one, sign up on the Datadog website. During the sign-up process, note the region of your Datadog site, as this will be required for configuration. For example, if your site is US5, the base URL will be us5.datadoghq.com.

Step 2: Obtain the API Key
After signing up, log into the Datadog portal and navigate to the API settings. Here, you can generate or retrieve your API key. This key is essential for authenticating your application with Datadog’s logging services. Keep this key secure, as it grants access to your Datadog account.

Step 3: Identify the Logging Endpoints
Datadog provides different endpoints for log collection based on your site region. Use the site selector dropdown on the Datadog documentation page to find the supported endpoints for your region. This information is crucial for configuring the URL in your appsettings.json file. For instance, if you are using the US5 region, the endpoint might be https://http-intake.logs.us5.datadoghq.com.

Step 4: Configure Serilog and Datadog in appsettings.json
In your appsettings.json file, add the necessary configuration for Serilog and Datadog. The configuration should specify sinks for both the console and Datadog, set minimum log levels, define service properties, and enrich logs with additional context.

Here is an example configuration:

{
“Serilog”: {
“Using”: [“Serilog.Sinks.Console”, “Serilog.Sinks.Datadog.Logs”],
“MinimumLevel”: {
“Default”: “Information”,
“Override”: {
“Microsoft”: “Warning”,
“System”: “Error”
}
},
“WriteTo”: [
{
“Name”: “Console”
},
{
“Name”: “DatadogLogs”,
“Args”: {
“apiKey”: “YOUR_API_KEY”,
“url”: “YOUR_DATADOG_ENDPOINT”,
“source”: “YOUR_SERVICE_NAME”,
“service”: “YOUR_SERVICE_NAME”,
“host”: “YOUR_HOST_NAME”,
“tags”: [“env:production”, “team:backend”]
}
}
],
“Enrich”: [“FromLogContext”, “WithMachineName”, “WithThreadId”]
}
}

Replace YOUR_API_KEY with your actual Datadog API key, YOUR_DATADOG_ENDPOINT with the appropriate endpoint URL, and customize the service, host, and tags as needed for your environment.

Step 5: Initialize Serilog in Program.cs
In your Program.cs file, initialize Serilog to read the configuration from appsettings.json and add it to the logging providers. This ensures that your application uses Serilog for logging throughout its lifecycle.

Here is an example of how to set up Serilog in Program.cs:

using Serilog;

var builder = WebApplication.CreateBuilder(args);

builder.Host.UseSerilog((context, services, configuration) => {
configuration.ReadFrom.Configuration(context.Configuration);
});

var app = builder.Build();

app.UseSerilogRequestLogging();

// Additional middleware and routing configuration

app.Run();

This code reads the Serilog configuration from appsettings.json and integrates Serilog with the ASP.NET Core logging pipeline. The UseSerilogRequestLogging middleware automatically logs HTTP request information, providing valuable insights into application performance and errors.

Benefits of Using Datadog with Serilog
Integrating Datadog with Serilog offers several advantages:

Centralized Log Management: All logs are sent to Datadog, providing a single pane of glass for monitoring and troubleshooting.

Real-Time Observability: Datadog’s dashboard allows you to visualize logs, metrics, and traces in real time, enabling quick detection of issues.

Enhanced Debugging: With enriched logs and contextual information, developers can quickly identify the root cause of problems.

Scalability: Datadog handles high volumes of logs efficiently, making it suitable for large-scale applications.

Security: Logs are transmitted over HTTPS, ensuring data integrity and confidentiality.

Best Practices for Configuration
When configuring Serilog and Datadog, consider the following best practices:

Use Environment-Specific Settings: Store sensitive information like API keys in environment variables or secure configuration stores rather than hardcoding them in appsettings.json.

Set Appropriate Log Levels: Configure minimum log levels to avoid overwhelming Datadog with verbose logs. Use overrides to reduce noise from framework libraries.

Enrich Logs: Add contextual information such as machine name, thread ID, and custom properties to make logs more informative.

Monitor Log Volume: Keep an eye on log volume to avoid exceeding Datadog’s quotas and incurring additional costs.

Test in Development: Always test the configuration in a development environment before deploying to production to ensure logs are being sent correctly.

Conclusion
Integrating Datadog with Serilog in a .NET project enhances observability and simplifies monitoring in distributed systems. By following the steps outlined in this guide, you can set up a robust logging infrastructure that provides real-time insights into your application’s performance and reliability. With centralized log management and powerful analytics, Datadog and Serilog together form a comprehensive solution for modern application monitoring.