Disclaimer
This post aims to be a quick guide on how to setup Grafana, Tempo and your C# application to be able to send traces into Tempo and inspect them via Grafana. It does not cover best practices, advanced scenarios or logs/tracing/metrics linking which will be covered in a future post.
Introduction
In the rapidly evolving landscape of software development, the ability to observe and understand the inner workings of applications is valuable. This is where tracing comes into play, offering a granular view of a request's journey through various components of a system. However, the challenge doesn't stop at collecting traces; it extends to analyzing and visualizing this data in a way that's insightful and actionable.
Enter Grafana and Tempo, a duo that brings clarity to the chaos. Grafana provides a versatile platform that can integrate with multiple data sources, including Tempo, a high-scale, distributed tracing backend. Together, they enable developers to not only collect and store traces efficiently but also to explore and analyze them through intuitive, customizable dashboards.
This blog post is tailored for developers who are eager to dive deep into the world of application tracing within the .NET ecosystem, specifically focusing on C# applications. Whether you're troubleshooting a complex issue, striving to enhance your application's performance, or simply curious about understanding your application's behavior in depth.
Infrastructure
Setup
Grafana and Tempo are required for this tutorial. The most common option for setting up infrastructure is containers.
Tempo Configuration File
Tempo requires a configuration file in order to run. The following commands assume that a file named tempo-config.yml
exists in the current directory. The file should contain the following:
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
http:
grpc:
storage:
trace:
backend: local
local:
path: ./data/tempo/blocks
wal:
path: ./data/wal/blocks
Setup Grafana & Tempo using Podman Desktop
# Cleanup
podman rm gdev-grafana -f
podman rm gdev-tempo -f
# Create network
podman network create gdev-net
# Run Grafana & Tempo
podman run --network gdev-net --name gdev-grafana -d -p 3000:3000 grafana/grafana
podman run --network gdev-net --name gdev-tempo -d -p 3200:3200 -p 4317:4317 -v ./tempo-config.yml:/etc/tempo-config.yml grafana/tempo "-config.file=/etc/tempo-config.yml"
# Note: Grafana default credentials are admin/admin
Setup Validation
- Grafana:
curl -i http://localhost:3000/api/health
until response code is 200 - Tempo:
curl -i http://localhost:3200/ready
until response code is 200
Configuration
The next step is to connect Grafana with Tempo:
- Login to Grafana (default credentials are admin/admin)
- Open the main menu from the 3-bars icon on top left
- Navigate to "Connections"
- Enter "Tempo" on the search bar and select the "Tempo" option from the results
- Click on the "Add new data source" button on top right
- Type
http://gdev-tempo:3200
in the "URL" field - Click the "Save & Test" button at the bottom of the page
Application Setup
Create a new .NET application:
dotnet new console --name GDEV.Tracing.ConsoleApp
Install the following NuGet packages (run inside the application folder):
dotnet add package OpenTelemetry
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol
Paste the following code in Program.cs
:
using System;
using System.Diagnostics;
using OpenTelemetry;
using OpenTelemetry.Trace;
using OpenTelemetry.Resources;
class Program
{
// Create a "facility" label to group all our traces under
const string facility = "gdt-logging-consoleapp";
// This is the top-level activity that represents our application and generates all other activities
private static ActivitySource _source = new ActivitySource(facility, "1.0.0");
static async Task Main(string[] args)
{
// Create a tracer provider (a destination for our traces to be sent for storage)
using var tracerProvider = Sdk.CreateTracerProviderBuilder()
.SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(facility))
.AddSource(facility)
.AddOtlpExporter(o => { o.Endpoint = new Uri("http://localhost:4317"); })
.Build();
// Create a new activity (since there is no other activity this is a top level activity)
using (var activity = _source.StartActivity("User.Login"))
{
Console.WriteLine($"[{activity?.TraceId.ToHexString()}]:{activity?.DisplayName}");
await Task.Delay(TimeSpan.FromMilliseconds(100));
using (_source.StartActivity("Login"))
{
await Task.Delay(TimeSpan.FromMilliseconds(100));
using (var localActivity = _source.StartActivity("usp_GetUserByUserName"))
{
// Invoke stored procedure
await Task.Delay(TimeSpan.FromMilliseconds(10));
localActivity?.SetStatus(ActivityStatusCode.Ok);
}
using (_source.StartActivity("ValidatePassword"))
{
// Validate password
await Task.Delay(TimeSpan.FromMilliseconds(10));
}
using (_source.StartActivity("Get User Permissions"))
{
await Task.Delay(TimeSpan.FromMilliseconds(100));
}
}
activity?.SetStatus(ActivityStatusCode.Ok);
}
}
}
Finally, run the application to generate traces:
dotnet run
Traces Visualization
Once your application runs successfully and generates traces, you can visualize them in Grafana:
- Navigate to Grafana at
http://localhost:3000
- Go to the "Explore" section
- Select "Tempo" as your data source
- Use the TraceID that was output by your console application to search for specific traces
- Explore the trace timeline, spans, and detailed information about each operation
The traces will show you the complete flow of your simulated user login process, including:
- The main
User.Login
activity - Nested
Login
operations - Database calls (
usp_GetUserByUserName
) - Password validation steps
- Permission retrieval operations
This visualization helps you understand performance bottlenecks, identify slow operations, and debug complex distributed systems.
Project Resources
You can find all the code for this example in the project's GitLab Repository.
For more infrastructure setup options and details, visit: Infrastructure: Grafana & Tempo.