DEV Community: Adam Gardner

Semgrep Observability with OpenTelemetry

Adam Gardner — Tue, 31 Mar 2026 10:49:52 +0000

Semgrep is a great open source security and code validation tool. Semgrep revolves around rules like this:

rules:
  - id: print-to-logger
    pattern: print($VAR)
    message: Use logging.info() instead of print()
    language: python
    severity: MEDIUM
    fix: logger.info($MSG)

The rule above will raise a MEDIUM severity issue every time a use of print() is used in your Python code. It will also provide the recommended fix and even take the value inside the print statement and produce the fix content. Thus print("Hello world!") becomes logger.info("Hello world!")

The rules.yaml file is then used to validate one or more (in this case) Python files:

semgrep scan -f rules.yaml app.py

Capturing Semgrep Output using the OpenTelemetry Collector

Semgrep is capable of producing JSON output which means it's really easy to grab using the OpenTelemetry collector. Let's re-run the previous command with a few more flags to produce JSON:

semgrep scan -f rules.yaml --json -o out.json app.py

It produces single line JSON (JSONL) like this (which I've expanded here for readability):

{
    "version": "1.152.0",
    "results": [
       // Rule violations are listed here...
        {
            "check_id": "print-to-logger",
            "path": "app.py",
            "start": {
                "line": 4,
                "col": 5,
                "offset": 46
            },
            "end": {
                "line": 4,
                "col": 18,
                "offset": 59
            },
            "extra": {
                "message": "Use logging.info() instead of print()",
                "fix": "logger.info(\"blah\")",
                "metadata": {},
                "severity": "MEDIUM",
                "fingerprint": "requires login",
                "lines": "requires login",
                "validation_state": "NO_VALIDATOR",
                "engine_kind": "OSS"
            }
        },
        ...
      ],
        "rules": [],
        "rules_parse_time": 0.0018589496612548828,
        "profiling_times": {
            "config_time": 0.1376628875732422,
            "core_time": 0.2947859764099121,
            "ignores_time": 4.291534423828125e-05,
            "total_time": 0.43769073486328125
        },
        "parsing_time": {
            "total_time": 0.0,
            "per_file_time": {
                "mean": 0.0,
                "std_dev": 0.0
            },
            "very_slow_stats": {
                "time_ratio": 0.0,
                "count_ratio": 0.0
            },
            "very_slow_files": []
        },
        "scanning_time": {
            "total_time": 0.009443998336791992,
            "per_file_time": {
                "mean": 0.009443998336791992,
                "std_dev": 0.0
            },
            "very_slow_stats": {
                "time_ratio": 0.0,
                "count_ratio": 0.0
            },
            "very_slow_files": []
        },
        "matching_time": {
            "total_time": 0.0,
            "per_file_and_rule_time": {
                "mean": 0.0,
                "std_dev": 0.0
            },
            "very_slow_stats": {
                "time_ratio": 0.0,
                "count_ratio": 0.0
            },
            "very_slow_rules_on_files": []
        },
        "tainting_time": {
            "total_time": 0.0,
            "per_def_and_rule_time": {
                "mean": 0.0,
                "std_dev": 0.0
            },
            "very_slow_stats": {
                "time_ratio": 0.0,
                "count_ratio": 0.0
            },
            "very_slow_rules_on_defs": []
        },
        "fixpoint_timeouts": [],
        "prefiltering": {
            "project_level_time": 0.0,
            "file_level_time": 0.0,
            "rules_with_project_prefilters_ratio": 0.0,
            "rules_with_file_prefilters_ratio": 1.0,
            "rules_selected_ratio": 1.0,
            "rules_matched_ratio": 1.0
        },
        "targets": [],
        "total_bytes": 0,
        "max_memory_bytes": 120384832
    }

Configure the OpenTelemetry collector to:

Monitor out.json
Parse the body text as JSON

The transform processor can also be used to process the JSONL lines as they transit through the collector. In this case the rules:

Set both the time and observed_time to the current time (since the log line doesn't explicitly state a timestamp
Adds a new Key/Value attribute pair to each log record of tool: semgrep (this is useful when the log line hits your Observability backend for filtering)
Adds another new Key/Value attribute pair to each log record where the key == results_found and the value is the length of the results array (again useful for backend processing - your O11y system may be able to compute lengths from an input array, but you can add it here to offload the processing / cost)
The final two rules effectively overwrite the version key as semgrep_version.

Note: The collector cannot rename attribute keys so you actually take the current value of version (ie. "1.152.0"), create a new attribute called semgrep_version, set the value of the value using the existing value and finally delete the existing version attribute.

receivers:
  filelog:
    include: [out.json]
    start_at: beginning
    operators:
      - type: json_parser
        parse_from: body

processors:
  transform:
    error_mode: ignore
    log_statements:
      - statements:
        - set(log.time, Now()) 
        - set(log.observed_time, Now())
        - set(log.attributes["tool"], "semgrep")
        - set(log.attributes["results_found"], Len(log.attributes["results"]))
        - set(log.attributes["semgrep_version"], log.attributes["version"])
        - delete_key(log.attributes, "version")

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [transform]
      exporters: [debug]

Save Money by shrinking output on no violations

The output can be a bit wordy even when there are no violations. We can shrink this using the transform processor with an additional rule:

- set(log.body, "Semgrep scan finished. No issues found.") where Len(log.attributes["results"]) == 0

Create metrics from log content

Notice there are lots of metric fields in the JSON so use the signal_to_metrics connector to transform log content to real OpenTelemetry metrics.

Add this content to the collector YAML (connectors should be at the same level as receivers and processors.

Then add the signal_to_metrics as both an output of the logs pipeline and an input to a metrics pipeline (that you need to define).

The idea here is that logs flow into the connector, are transformed to metrics and into the metrics pipeline they go.

connectors:
  signal_to_metrics:
    logs:
      - name: max_memory_bytes
        description: Extract the first number from the string
        gauge:
          value: Double(log.attributes["time"]["max_memory_bytes"])
      - name: profiling_times.config_time
        description: Extract the first number from the string
        gauge:
          value: Double(log.attributes["time"]["profiling_times"]["config_time"])

...

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [transform]
      exporters: [debug, signal_to_metrics]
    metrics:
      receivers: [signal_to_metrics]
      processors: []
      exporters: [debug]

Summary

Semgrep is a great security and validation tool and the results are really easy to process using the OpenTelemetry collector.

Subscribe to me on YouTube for more Observability and OpenTelemetry content.

Trasform your logs to metrics (and do maths with metrics) using the OpenTelemetry Collector

Adam Gardner — Sat, 07 Mar 2026 03:45:15 +0000

Logs are everywhere... But they're expensive to store, retrieve and process. And sometimes you only need metrics.

Did you know the OpenTelemetry is perfectly suited to be the place that you transform your logs to metrics? Log in, metrics out. Plus, did you know you can do maths in the collector?

In this video I show you how to do it using the transform processor to extract the values from the log then the signal to metrics connector.

First I use something structured and easy: JSON.

{ "foo": 1234, "bar": "5432", "baz": "Here is the number: 57 status = complete" }

Next I extract and generate metrics from an unstructured textual log line:

Purchase complete. Product price=6. Quantity=3. Product ID=987. user_id=123

Finally I take the product price and quantity and multiply them (in realtime) to produce a third metric (order total).

Use LD_PRELOAD to inject OpenTelemetry into everything

Adam Gardner — Wed, 04 Mar 2026 09:15:42 +0000

In this video I demonstrate the new OpenTelemetry injector.

It's a mechanism to automatically inject OTEL into your code with zero code or startup script changes. It's potentially a great way to gain Observability of non Kubernetes workloads like VMs. It leverages LD_PRELOAD. You add the LD_PRELOAD instruction to your VM at startup and the rest happens automatically.

Do heed the warnings towards the end of the video though since it's still early days for this tool!

Why your AGENTS.md files are being ignored (and how to fix it)

Adam Gardner — Wed, 04 Mar 2026 09:11:11 +0000

So... Like (I assume) everyone, I was experimenting with AGENTS.md the other day (I know, we're already bored of that and on to skills), but I suddenly realised that I had no idea whether or not the content was actually being used / obeyed.

Turns out it wasn't most of the time!

In this video I discuss my own experience, research of 5 leading models + the other industry research in this area.

Kubernetes Dashboard is Dead!

Adam Gardner — Sun, 22 Feb 2026 09:13:58 +0000

The Kubernetes dashboard is now deprecated and unmaintained. It's time to move on.

In this video I cover the replacement - a project I've been meaning to cover for a while now since it's so good. The CNCF project Headlamp

Introducing "badtrace". Generate "bad" OpenTelemetry traces easily

Adam Gardner — Sat, 22 Nov 2025 08:05:50 +0000

TLDR: badtrace on GitHub

I needed a tool that would deliberately generate OpenTelemetry traces that were "bad". I needed this for training, enablement and demo purposes so that I could easily fire a trace with a known issue into my Observability system and thus show "why it was bad" and "how it looks".

Yes, tools like tracepusher and telemetrygen already exist, but they typically:

a) Generate "good" or "healthy" traces
b) Generate single traces

Hence, "badtrace" was born.

It works based on "scenarios" (you can implement your own in a few lines of Python).

python app.py \
  --endpoint=http://localhost:4318 \
  --service-name=badtrace \ # optional: defaults to "badtrace"
  --trace-count=1 \ # optional: defaults to 1
  --insecure=true \
  --scenario=scenario1

Scenarios

As I write this, there are currently 4 scenarios (please come and help implement more) which can be toggled by changing scenario1 above to scenario2, scenario3 or scenario4 etc.

Scenario 1 creates a short, small trace. This is quick (in end-to-end time) and short in terms of number of spans. In reality this would be a very low value trace (and thus a candidate to filter or heavily sample).

Scenario 2 creates a trace where some of the spans have errored. This models a trace where some of the operations error (imagine calling third party services and some of them fail).

Scenario 3 creates a "chatty" client to server trace with the client repeatedly calling the same endpoint over and over. This models things like the N+1 query problem where (for example) multiple SELECT statements occur to a database (instead of batching up the SELECTS into fewer statements).

Scenario 4 is similar to 3. It creates a "chatty" client but this time, the connections occur to many different servers.

Check out badtrace

If this sounds useful, go check out badtrace on GitHub!

osquery + OpenTelemetry = ❤️

Adam Gardner — Sun, 16 Nov 2025 07:35:15 +0000

As you probably know by now, osquery effectively turns your endpoints into SQL endpoints that you can query: SELECT * FROM processes or SELECT * FROM users etc.

But, that data is much more useful if it's tied to other telemetry data coming from your VMs, endpoints or Kubernetes clusters. This is typically the domain of APM tools.

Using OpenTelemetry (and specifically the OpenTelemetry collector) we can bring those two worlds together.

In this video I show you how that's done.

Build a Kubernetes Cluster map in real time using DNS

Adam Gardner — Sun, 16 Nov 2025 07:03:33 +0000

DNS is the way things find other things, but did you know that you can use that fact to build a real time service to service communication / traffic flow on a Kubernetes cluster? That's what the smart folks over at Otterize have done and in this video, I put their network mapper tool on the OpenTelemetry demo to see how well it performs (spoiler: really well!)

Map a Kubernetes cluster with one command

Adam Gardner — Mon, 10 Nov 2025 23:27:01 +0000

DNS is the way things discover other things on the internet, but did you know there's a cool little network mapping tool that will build a realtime map of your Kubernetes cluster using DNS?

It's called the network mapper from Otterize and in the video below I run it through its paces - then unleash it on the OpenTelemetry demo to see how it copes (spoiler: well!)

Here's the code for the little visualisation app I created too: https://github.com/agardnerIT/network-mapper-visualizer

Monitor Gemini CLI using OpenTelemetry for realtime usage statistics

Adam Gardner — Mon, 03 Nov 2025 11:50:05 +0000

In this video, I'll show you how easy it is to monitor your Google Gemini CLI usage in realtime so that you don't overspend.

Plus, you can show your boss how long you've wasted sitting waiting for AI to answer you!

Monitor Gemini CLI using OpenTelemetry for realtime usage statistics

Adam Gardner — Mon, 03 Nov 2025 10:47:54 +0000

In this video, I'll show you how easy it is to monitor your Google Gemini CLI usage in realtime so that you don't overspend.

Plus, you can show your boss how long you've wasted sitting waiting for AI to answer you!

How Distributed Tracing Really Works

Adam Gardner — Tue, 21 Oct 2025 20:31:32 +0000

It's relatively simple these days to spin up an OpenTelemetry demo and / or auto-instrument services with OpenTelemetry.

That's fine - when it works - but it's when something breaks that perhaps you realise you don't quite understand what's going on under the covers. I've lost count of the number of times customers have reported "broken traces" to me as they flow through an uninstrumented tier, a load balancer or a queue.

In the following video I show precisely how distributed tracing using OpenTelemetry is able to do what it does (for HTTP) by spinning up two Python FastAPI applications, instrumenting them using OpenTelemetry and demonstrating that the traces are "broken" in Jaeger. Then I solve the issue (hint: it's all about propagation).

The code for all of this can be found here: https://github.com/agardnerIT/python-fastapi-requests-traceparent