Disclaimer

This post aims to be a quick guide on how to setup Grafana, Tempo and your C# application to be able to send traces into Tempo and inspect them via Grafana. It does not cover best practices, advanced scenarios or logs/tracing/metrics linking which will be covered in a future post.

Introduction

In the rapidly evolving landscape of software development, the ability to observe and understand the inner workings of applications is valuable. This is where tracing comes into play, offering a granular view of a request's journey through various components of a system. However, the challenge doesn't stop at collecting traces; it extends to analyzing and visualizing this data in a way that's insightful and actionable.

Enter Grafana and Tempo, a duo that brings clarity to the chaos. Grafana provides a versatile platform that can integrate with multiple data sources, including Tempo, a high-scale, distributed tracing backend. Together, they enable developers to not only collect and store traces efficiently but also to explore and analyze them through intuitive, customizable dashboards.

This blog post is tailored for developers who are eager to dive deep into the world of application tracing within the .NET ecosystem, specifically focusing on C# applications. Whether you're troubleshooting a complex issue, striving to enhance your application's performance, or simply curious about understanding your application's behavior in depth.

Infrastructure

Setup

Grafana and Tempo are required for this tutorial. The most common option for setting up infrastructure is containers.

Tempo Configuration File

Tempo requires a configuration file in order to run. The following commands assume that a file named tempo-config.yml exists in the current directory. The file should contain the following:

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        http:
        grpc:

storage:
  trace:
    backend: local
    local:
      path: ./data/tempo/blocks
    wal:
      path: ./data/wal/blocks

Setup Grafana & Tempo using Podman Desktop

# Cleanup
podman rm gdev-grafana -f
podman rm gdev-tempo -f

# Create network
podman network create gdev-net

# Run Grafana & Tempo
podman run --network gdev-net --name gdev-grafana -d -p 3000:3000 grafana/grafana
podman run --network gdev-net --name gdev-tempo -d -p 3200:3200 -p 4317:4317 -v ./tempo-config.yml:/etc/tempo-config.yml grafana/tempo "-config.file=/etc/tempo-config.yml"

# Note: Grafana default credentials are admin/admin

Setup Validation

  • Grafana: curl -i http://localhost:3000/api/health until response code is 200
  • Tempo: curl -i http://localhost:3200/ready until response code is 200

Configuration

The next step is to connect Grafana with Tempo:

  1. Login to Grafana (default credentials are admin/admin)
  2. Open the main menu from the 3-bars icon on top left
  3. Navigate to "Connections"
  4. Enter "Tempo" on the search bar and select the "Tempo" option from the results
  5. Click on the "Add new data source" button on top right
  6. Type http://gdev-tempo:3200 in the "URL" field
  7. Click the "Save & Test" button at the bottom of the page

Application Setup

Create a new .NET application:

dotnet new console --name GDEV.Tracing.ConsoleApp

Install the following NuGet packages (run inside the application folder):

dotnet add package OpenTelemetry
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol

Paste the following code in Program.cs:

using System;
using System.Diagnostics;
using OpenTelemetry;
using OpenTelemetry.Trace;
using OpenTelemetry.Resources;

class Program
{ 
    // Create a "facility" label to group all our traces under
    const string facility = "gdt-logging-consoleapp";
 
    // This is the top-level activity that represents our application and generates all other activities
    private static ActivitySource _source = new ActivitySource(facility, "1.0.0");
    
    static async Task Main(string[] args)
    {
        // Create a tracer provider (a destination for our traces to be sent for storage)
        using var tracerProvider = Sdk.CreateTracerProviderBuilder()
            .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(facility))      
            .AddSource(facility)      
            .AddOtlpExporter(o => { o.Endpoint = new Uri("http://localhost:4317"); })
            .Build();

        // Create a new activity (since there is no other activity this is a top level activity)
        using (var activity = _source.StartActivity("User.Login"))
        {        
            Console.WriteLine($"[{activity?.TraceId.ToHexString()}]:{activity?.DisplayName}");

            await Task.Delay(TimeSpan.FromMilliseconds(100));

            using (_source.StartActivity("Login"))
            {   
                await Task.Delay(TimeSpan.FromMilliseconds(100));
                
                using (var localActivity = _source.StartActivity("usp_GetUserByUserName"))
                {
                    // Invoke stored procedure
                    await Task.Delay(TimeSpan.FromMilliseconds(10));
                    localActivity?.SetStatus(ActivityStatusCode.Ok);
                }

                using (_source.StartActivity("ValidatePassword"))
                {
                    // Validate password
                    await Task.Delay(TimeSpan.FromMilliseconds(10));
                }

                using (_source.StartActivity("Get User Permissions"))
                {   
                    await Task.Delay(TimeSpan.FromMilliseconds(100));
                }
            }

            activity?.SetStatus(ActivityStatusCode.Ok);
        }
    }
}

Finally, run the application to generate traces:

dotnet run

Traces Visualization

Once your application runs successfully and generates traces, you can visualize them in Grafana:

  1. Navigate to Grafana at http://localhost:3000
  2. Go to the "Explore" section
  3. Select "Tempo" as your data source
  4. Use the TraceID that was output by your console application to search for specific traces
  5. Explore the trace timeline, spans, and detailed information about each operation

The traces will show you the complete flow of your simulated user login process, including:

  • The main User.Login activity
  • Nested Login operations
  • Database calls (usp_GetUserByUserName)
  • Password validation steps
  • Permission retrieval operations

This visualization helps you understand performance bottlenecks, identify slow operations, and debug complex distributed systems.

Project Resources

You can find all the code for this example in the project's GitLab Repository.

For more infrastructure setup options and details, visit: Infrastructure: Grafana & Tempo.