Disclaimer

This post aims to be a quick guide on how to setup Grafana, Tempo and your C# application to be able to send traces into Tempo and inspect them via Grafana. It does not cover best practices, advanced scenarios or logs/tracing/metrics linking which will be covered in a future post.

Introduction

In the rapidly evolving landscape of software development, the ability to observe and understand the inner workings of applications is valuable. This is where tracing comes into play, offering a granular view of a request's journey through various components of a system. However, the challenge doesn't stop at collecting traces; it extends to analyzing and visualizing this data in a way that's insightful and actionable. Enter Grafana and Tempo, a duo that brings clarity to the chaos. Grafana provides a versatile platform that can integrate with multiple data sources, including Tempo, a high-scale, distributed tracing backend. Together, they enable developers to not only collect and store traces efficiently but also to explore and analyze them through intuitive, customizable dashboards. This blog post is tailored for developers who are eager to dive deep into the world of application tracing within the .NET ecosystem, specifically focusing on C# applications. Whether you're troubleshooting a complex issue, striving to enhance your application's performance, or simply curious about.

Infrastructure

Setup

Grafana and Tempo are required for this tutorial, the most common option for setting up infrastructure is containers.

Configuration Files

Tempo requires a configuration file in order to run. The following commands assume that a file named tempo-config.yml exists on the current directory. The file should contain the following:

Tempo Configuration File

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        http:
        grpc:

storage:
  trace:
    backend: local
    local:
      path: ./data/tempo/blocks
    wal:
      path: ./data/wal/blocks

tempo-config.yml: Tempo configuration file

Setup Grafana & Tempo using Podman Desktop

# Cleanup
podman rm gdev-grafana -f
podman rm gdev-tempo -f

# Create network
podman network create gdev-net

# Run Grafana & Tempo

podman run --network gdev-net --name gdev-grafana -d -p 3000:3000 grafana/grafana
podman run --network gdev-net --name gdev-tempo -d -p 3200:3200 -p 4317:4317 -v ./tempo-config.yml:/etc/tempo-config.yml grafana/tempo "-config.file=/etc/tempo-config.yml"

#Note: Grafana default credentials are admin/admin

setup-with-podman.ps1 (run it on PowerShell)

Setup Validation

Grafana: curl -i http://localhost:3000/api/health until response code is 200
Tempo: curl -i http://localhost:3200/ready until response code is 200

For more options and more details on setting up the infrastructure, visit: Infrastructure: Grafana & Tempo

Configuration

The next step is to connect Grafana with Tempo: 1. Login to Grafana (default credentials are admin/admin) 2. Open the main menu from the 3-bars icon on top left 3. Navigate to "Connections" 4. Enter "Tempo" on the search bar and Select the "Tempo" options from the results 5. Click on the "Add new data source" button on top right 6. Type http://gdev-tempo:3200 on the "URL" field 7. Click the "Save & Test" button on the bottom of the page

Application Setup

Create a new dotnet application

dotnet new console --name GDEV.Tracing.ConsoleApp

Install the following nuget packages (run inside the application folder)

dotnet add package OpenTelemetry
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol

Paste the following code in program.cs:

using System;
using System.Diagnostics;
using OpenTelemetry;
using OpenTelemetry.Trace;
using OpenTelemetry.Resources;

class Program
{ 
		// Create a "facility" label to group all our traces under
    const string facility = "gdt-logging-consoleapp";
 
    // This is the top-level activity that represents our application and generates all other activities
    private static ActivitySource _source = new ActivitySource(facility, "1.0.0");
    
    static async Task Main(string[] args)
    {
        // create a tracer provider (a destination for our traces to be send for storage)
        using var tracerProvider = Sdk.CreateTracerProviderBuilder()
            .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService(facility))      
            .AddSource(facility)      
            .AddOtlpExporter(o => { o.Endpoint = new Uri("http://localhost:4317"); })
            .Build();

        // create a new activity (since there is no other activity this is a top level activity)
        using (var activity = _source.StartActivity("User.Login"))
        {        
            Console.WriteLine($"[{activity?.TraceId.ToHexString()}]:{activity?.DisplayName}");

            await Task.Delay(TimeSpan.FromMilliseconds(100));

            using (_source.StartActivity("Login"))
            {   
                await Task.Delay(TimeSpan.FromMilliseconds(100));
                
                using (var localActicity = _source.StartActivity("usp_GetUserByUserName"))
                {
                    // invoke stored procedure
                    await Task.Delay(TimeSpan.FromMilliseconds(10));
                    localActicity?.SetStatus(ActivityStatusCode.Ok);
                }

                using (_source.StartActivity("ValidatePassword"))
                {
                    // Validate password
                    await Task.Delay(TimeSpan.FromMilliseconds(10));
                }

                using (_source.StartActivity("Get User Permissions"))
                {   
                    await Task.Delay(TimeSpan.FromMilliseconds(100));
                }
            }

            activity?.SetStatus(ActivityStatusCode.Ok);
        }
    }
}

Program.cs

Finally, run the app traces

dotnet run

Traces Visualization

…

C#: Tracing with Grafana & Tempo