Forem: Dean Hamstead

Syslog to PostgreSQL via Rsyslog: A Production-Ready Setup

Dean Hamstead — Sat, 04 Apr 2026 13:08:49 +0000

Syslog is the backbone of infrastructure logging, but storing logs as flat files makes querying, retention management, and analysis painful. The industry's knee-jerk reaction is to stand up Elasticsearch. Having run Elasticsearch extensively in production—for both high-write log ingestion/charting and high-read complex catalog searches—I am frankly exhausted by it. I am tired of Java. I'm tired of babysitting the JVM, tweaking heap sizes, fighting garbage collection pauses, and dedicating massive amounts of RAM just to keep the cluster from going red. If you already run PostgreSQL, you might not need to inflict that on yourself. While Elasticsearch demands a dedicated cluster and specialized operational knowledge, this PostgreSQL approach elegantly bolts log storage onto your existing database infrastructure.

In this post, I'll walk through a production-ready setup that pipes syslog directly into PostgreSQL via rsyslog, using features like range partitioning and pg_cron to achieve many of the same benefits as Elasticsearch's time-based indices and Index Lifecycle Management (ILM) — without the Java Virtual Machine (JVM) overhead or the separate cluster.

The full SQL schema, rsyslog configs, and everything discussed here is available for you to use as-is.

Architecture Overview

The setup has two parts:

Client side — forwards logs via TCP using rsyslog's omfwd module
Server side — receives logs on UDP, TCP, and (optionally) Reliable Event Logging Protocol (RELP) ports and writes them to PostgreSQL using ompgsql

┌──────────────┐
│ Linux Server │ ─┐
└──────────────┘  │   TCP/UDP 514   ┌──────────────────┐
                  ├────────────────▶│   Intermediate   │
┌──────────────┐  │                 │  Rsyslog Server  │
│     NAS      │ ─┘                 └────────┬─────────┘
└──────────────┘                             │
                                             │ RELP 2514
┌──────────────┐                             │
│   OPNsense   │ ─┐                      ┌───▼──────────────┐
└──────────────┘  │   TCP/UDP 514        │   Main Rsyslog   │
                  ├─────────────────────▶│      Server      │
┌──────────────┐  │                      └────────┬─────────┘
│Managed Switch│ ─┘                               │
└──────────────┘                           ompgsql│
                                                  ▼
                                         ┌──────────────────┐
                                         │   PostgreSQL     │
                                         │   (syslog DB)    │
                                         └──────────────────┘

(Note: The Intermediate Rsyslog Server shown above is entirely optional. It is included here to illustrate how you can meet strict security requirements—such as aggregating logs from a DMZ or isolated network segment before securely forwarding them via RELP to your main logging infrastructure.)

Client Configuration

On each log-producing host, the forwarding config is minimal:

action(
    type="omfwd"
    target="192.0.2.1"
    port="514"
    protocol="tcp"
    template="RSYSLOG_SyslogProtocol23Format"
)

TCP is used instead of UDP for reliable delivery — no dropped log lines on a busy network. The RSYSLOG_SyslogProtocol23Format template ensures RFC 5424 compliance, giving us structured fields like msgid and programname on the receiving end.

Server Configuration

The server receives logs on three protocols and writes them to PostgreSQL:

module(load="imudp")
module(load="imtcp")
module(load="imrelp")

input(
    type="imudp"
    port="514"
    rateLimit.interval="60"
    rateLimit.burst="2000"
)

input(
    type="imtcp"
    port="514"
)

input(
    type="imrelp"
    port="2514"
    maxDataSize="10k"
)

Three protocols for three use cases:

UDP/514 — universal compatibility, with rate limiting (burst="2000" per 60 seconds) to prevent a noisy host from overwhelming the server
TCP/514 — reliable delivery for clients that support it
RELP/2514 (Optional) — transaction-based syslog protocol with guaranteed delivery and replay on failure. It's a nice-to-have for rsyslog-to-rsyslog forwarding on unstable links, but standard TCP is sufficient for most deployments.

The PostgreSQL Output Template

The heart of the pipeline is the ompgsql module with a list-type template:

template(
    name="LogToPgSQL" type="list" option.stdsql="on" ) {
    constant(value="INSERT INTO system_events (...) VALUES ('")
    property(name="timereported" dateFormat="pgsql" date.inUTC="on")
    constant(value="', '")
    property(name="msgid")
    constant(value="', 'syslog', '")
    property(name="programname")
    -- ... more fields ...
    property(name="msg" escape="sql")
    -- ... more fields ...
}

Two details worth highlighting:

dateFormat="pgsql" and date.inUTC="on" — Syslog timestamps are notoriously inconsistent across different devices. This combination forces rsyslog to standardize everything to UTC at the ingestion point and format it exactly as PostgreSQL's TIMESTAMP WITH TIME ZONE expects (YYYY-MM-DD HH:MM:SS.mmm+TZ). No string parsing on the database side, and no timezone nightmares when querying later.
escape="sql" — built-in SQL escaping on msg and rawmsg prevents injection from malicious or malformed log content.

(For a complete list of variables you can extract from syslog messages, check out the official Rsyslog properties documentation.)

Queue Configuration for Resilience

action(
    type="ompgsql" server="localhost"
    user="rsyslog" pass="..."
    db="syslog" template="LogToPgSQL"

    queue.type="linkedList"
    queue.size="20000"
    queue.workerThreads="2"
)

The linked-list queue with 20,000 message capacity and 2 worker threads provides a basic buffer during database hiccups. If PostgreSQL briefly stalls, rsyslog holds messages in memory rather than dropping them. Two worker threads keep the pipeline flowing even when one thread is blocked on a slow write.

A Note for Production Deployments:
If you are implementing this in a production environment, Rsyslog's queuing subsystem is something you should absolutely dig deeper into. It is a killer feature for log reliability. Beyond simple in-memory linked lists, Rsyslog supports Disk-Assisted Queues.

If PostgreSQL goes down for an extended maintenance window, or if a broadcast storm triggers a massive log spike that exceeds your database write speed, a disk-assisted queue will seamlessly spill the buffered logs onto the local disk. Once the database recovers, Rsyslog drains the disk queue into PostgreSQL. This guarantees zero log loss during database outages or extreme traffic spikes—a level of resilience that usually requires deploying a dedicated message broker like Kafka in other logging stacks.

The PostgreSQL Schema

Now for the centerpiece of this setup: the database schema. This is where PostgreSQL starts to look a lot like a log aggregation platform.

1. Range Partitioning — PostgreSQL's Answer to Time-Based Indices

Elasticsearch handles log retention by creating a new index per time period — logs-2025.01.01, logs-2025.01.02, and so on. When it's time to delete old data, it drops the entire index. Instant, no compaction, no garbage collection.

PostgreSQL range partitioning is the same concept, expressed in SQL:

CREATE TABLE system_events (
    ID SERIAL NOT NULL,
    ReceivedAt TIMESTAMP WITH TIME ZONE NOT NULL,
    DeviceReportedTime TIMESTAMP WITH TIME ZONE NULL,
    EventID VARCHAR(60) NULL,
    EventLogType VARCHAR(60),
    EventSource VARCHAR(60),
    Facility SMALLINT NULL,
    FromHost VARCHAR(255) NULL,
    FromPort INT NULL,
    FromIpAddress INET NULL,
    HostName VARCHAR(255) NULL,
    Message TEXT,
    Priority SMALLINT NULL,
    RawMessage TEXT,
    RFC5424AppName VARCHAR(255) NULL,
    RFC5424MsgID VARCHAR(255) NULL,
    RFC5424ProcID VARCHAR(255) NULL,
    RFC5424ProtocolVersion SMALLINT NULL,
    RFC5424StructuredData TEXT NULL,
    SysLogTag VARCHAR(60),
    message_tsv tsvector
        GENERATED ALWAYS AS (to_tsvector('english', Message)) STORED,
    PRIMARY KEY (ID, ReceivedAt)
) PARTITION BY RANGE (ReceivedAt);

The ReceivedAt column is the partition key — every row is routed to the correct monthly partition automatically. The primary key includes ReceivedAt because PostgreSQL requires the partition key to be part of any unique constraint on a partitioned table.

The message_tsv column is a generated column that automatically maintains a full-text search vector from the Message column. It's always in sync — no application logic, no triggers, no stale data.

Monthly partitions are created like this:

CREATE TABLE system_events_2026_03 PARTITION OF system_events
    FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');

CREATE TABLE system_events_2026_04 PARTITION OF system_events
    FOR VALUES FROM ('2026-04-01') TO ('2026-05-01');

Partition Pruning in Action

Here's where partitioning pays off. When you query a date range, PostgreSQL's planner knows exactly which partitions to scan — and skips the rest entirely. This is called partition pruning, and it's the same mechanism Elasticsearch uses to skip irrelevant shards.

Watch what happens when we query a date range that has no matching partition:

EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM system_events
WHERE ReceivedAt >= '2025-03-01' AND ReceivedAt < '2025-04-01';

                                     QUERY PLAN
------------------------------------------------------------------------------------
 Result  (cost=0.00..0.00 rows=0 width=0) (actual time=0.001..0.001 rows=0 loops=1)
   One-Time Filter: false
 Planning:
   Buffers: shared hit=6
 Planning Time: 0.053 ms
 Execution Time: 0.012 ms

0.012 milliseconds. The planner determined at plan time that no partition covers March 2025, so it didn't scan anything at all. One-Time Filter: false means the entire query was short-circuited. This is partition pruning at its most effective.

On a real query that hits a partition, PostgreSQL scans only the matching partition — not January, not February, not April. The other partitions are invisible to the query.

Dropping Partitions vs. DELETE

This is the killer feature of partitioning for log retention. When it's time to delete old data, you don't run:

DELETE FROM system_events WHERE ReceivedAt < '2026-01-01';

A DELETE on millions of rows means:

Scanning millions of rows to find the ones to delete
Generating massive WAL (write-ahead log)
Leaving dead tuples behind that require VACUUM
VACUUM competing with your ingestion workload
Table bloat until autovacuum catches up

Instead, you drop the partition:

DROP TABLE system_events_2025_12;

This is instant. No scanning, no WAL, no VACUUM, no bloat. The entire partition file is removed from disk in a single filesystem operation. It's the same advantage Elasticsearch gets from dropping an index.

Real-World Partition Stats

In this deployment, the partitioned table holds 2.12 million rows across seven monthly partitions:

Partition	Rows	Table Size	Index Size	Total
2026_01	—	248 MB	70 MB	331 MB
2026_02	—	269 MB	51 MB	334 MB
2026_03	43,310	493 MB	95 MB	608 MB
2026_04	134,439	70 MB	14 MB	86 MB
2026_05+	0	0 bytes	160 kB	168 kB

Note that pg_total_relation_size('system_events') on the parent table returns 0 bytes — this is expected. The parent table in a partitioned setup is just a logical container. All the data lives in the child partitions.

Index overhead sits at 19-28% of table size, which is very reasonable. The primary key index is the largest single index component — a consequence of including ReceivedAt in the composite key for partitioning compatibility.

2. Index Strategy — Designed for How Logs Are Actually Queried

The schema includes a deliberately chosen set of indexes, each serving a specific query pattern:

BRIN Indexes on Time Columns

CREATE INDEX idx_system_events_receivedat_brin
    ON system_events USING brin (ReceivedAt DESC);
CREATE INDEX idx_system_events_devicereportedtime_brin
    ON system_events USING brin (DeviceReportedTime DESC);

BRIN (Block Range Index) stores the minimum and maximum values for each block range in the table. For append-only, time-ordered data like syslog, this is near-perfect. A BRIN index on ReceivedAt is typically ~200 KB — compared to ~23 MB for an equivalent B-tree index. That's a 99% reduction in index size with nearly identical performance for range scans.

BRIN works best for naturally ordered data like timestamps. If your log ingestion has significant out-of-order delivery (hours/days delayed), B-tree might be preferable despite the size difference.

B-tree Indexes on Common Filters

CREATE INDEX idx_system_events_fromhost ON system_events (FromHost);
CREATE INDEX idx_system_events_eventsource ON system_events (EventSource);
CREATE INDEX idx_system_events_facility ON system_events (Facility);
CREATE INDEX idx_system_events_priority ON system_events (Priority);

These cover the most common queries: "show me all logs from this host," "show me all sshd logs," "show me all auth facility messages." Simple, effective, and small enough to stay in memory.

Partial Index for Errors

CREATE INDEX idx_system_events_errors ON system_events
    (FromHost, EventSource, ReceivedAt DESC)
    WHERE Priority <= 3;

This is a partial composite index — it only indexes rows where Priority <= 3 (emerg, alert, crit, err), which is about 9% of all log messages. The three-column structure covers the equality filters (FromHost, EventSource) and the sort (ReceivedAt DESC) in a single index. Error dashboards hit this index and never touch the other 91% of rows.

GIN Index for Full-Text Search

CREATE INDEX idx_system_events_message_fts
    ON system_events USING gin (message_tsv);

This Generalized Inverted Index (GIN) is what closes the biggest gap vs. Elasticsearch. It enables fast keyword search across all log messages:

SELECT ReceivedAt, FromHost, EventSource,
       ts_headline('english', Message, q) AS context
FROM system_events, to_tsquery('english', 'authentication & failure') q
WHERE message_tsv @@ q
ORDER BY ReceivedAt DESC LIMIT 20;

Tradeoff: the message_tsv column adds ~20-30% to storage per row. If you don't need keyword search, omit the column and its index.

In testing, searching for 'authentication failure' across 2M rows took ~15ms with the GIN index vs. several seconds with a sequential scan.

3. pg_cron — PostgreSQL's Answer to Index Lifecycle Management

Elasticsearch has Index Lifecycle Management (ILM) — a built-in system that automatically creates new indices, rolls over old ones, and deletes expired data. PostgreSQL doesn't have ILM baked in, but it has pg_cron, and with a few stored procedures, you get the same result.

pg_cron runs scheduled SQL jobs directly inside PostgreSQL — no external cron daemon, no shell scripts, no connection management. It's cron, but it speaks SQL.

The Stored Procedures

Three procedures handle the full partition lifecycle:

create_monthly_partition — creates a single partition for a given year and month, idempotent:

CREATE OR REPLACE PROCEDURE public.create_monthly_partition(
    p_table_name TEXT, year INT, month INT)
LANGUAGE plpgsql AS $$
DECLARE
    partition_name TEXT;
    start_date DATE;
    end_date DATE;
BEGIN
    start_date := make_date(year, month, 1);
    end_date := (start_date + INTERVAL '1 month')::DATE;
    partition_name := format('%s_%s_%s', p_table_name, year, lpad(month::TEXT, 2, '0'));

    IF NOT EXISTS (
        SELECT FROM pg_tables
        WHERE tablename = partition_name AND schemaname = 'public'
    ) THEN
        EXECUTE format(
            'CREATE TABLE public.%I PARTITION OF public.%I FOR VALUES FROM (%L) TO (%L)',
            partition_name, p_table_name, start_date, end_date
        );
        RAISE NOTICE 'Created partition public.%', partition_name;
    ELSE
        RAISE NOTICE 'Partition public.% already exists, skipping', partition_name;
    END IF;
END;
$$;

create_future_partitions — loops N months ahead and creates them all:

CREATE OR REPLACE PROCEDURE public.create_future_partitions(
    p_table_name TEXT, months_ahead INT)
LANGUAGE plpgsql AS $$
DECLARE
    current_date DATE := CURRENT_DATE;
    target_date DATE;
    i INT;
BEGIN
    FOR i IN 0..months_ahead LOOP
        target_date := current_date + (i * INTERVAL '1 month');
        CALL public.create_monthly_partition(
            p_table_name,
            EXTRACT(YEAR FROM target_date),
            EXTRACT(MONTH FROM target_date)
        );
    END LOOP;
END;
$$;

drop_old_partitions — drops partitions older than a retention interval:

CREATE OR REPLACE PROCEDURE public.drop_old_partitions(
    p_table_name TEXT, retention_interval INTERVAL)
LANGUAGE plpgsql AS $$
DECLARE
    partition_name TEXT;
BEGIN
    FOR partition_name IN (
        SELECT tables.table_name
        FROM information_schema.tables
        WHERE tables.table_name LIKE p_table_name || '_%'
        AND tables.table_name < p_table_name || '_'
            || to_char(CURRENT_DATE - retention_interval, 'YYYY_MM')
        AND tables.table_schema = 'public'
    )
    LOOP
        EXECUTE 'DROP TABLE IF EXISTS public.' || quote_ident(partition_name);
        RAISE NOTICE 'Dropped partition public.%', partition_name;
    END LOOP;
END;
$$;

The Cron Jobs

Two scheduled jobs keep the system self-managing:

SELECT cron.schedule_in_database(
    'drop_old_system_events_partitions',
    '0 14 1 * *',
    $$CALL public.drop_old_partitions('system_events', INTERVAL '3 months');$$,
    'syslog'
);

SELECT cron.schedule_in_database(
    'create_future_system_events_partitions',
    '5 14 1 * *',
    $$CALL public.create_future_partitions('system_events', 3);$$,
    'syslog'
);

On the first of every month at 14:00:

Old partitions are dropped — anything older than 3 months vanishes instantly
Future partitions are created — the next 3 months are pre-provisioned

Five minutes between jobs ensures the drop completes before new partitions are created. No external scripts, no SSH keys, no cron entry management. The entire lifecycle runs inside PostgreSQL, observable through standard system tables:

SELECT jobid, jobname, schedule, command FROM cron.job;
SELECT jobid, status, start_time, end_time
FROM cron.job_run_details ORDER BY start_time DESC;

In this deployment, the job history tells a nice story: the cron jobs were scheduled before the syslog database existed. They failed every month from December 2025 through February 2026 with "connection failed." Once the database came online in March, they just worked — no manual intervention needed. This self-healing behavior means you can safely deploy the cron jobs before the database is ready - they'll automatically start working when the database becomes available.

This is the log retention equivalent of Elasticsearch's ILM — automatic rollover, automatic deletion — but implemented in pure SQL with a scheduling extension.

4. The INET Data Type

FromIpAddress INET NULL,

PostgreSQL's native INET type stores IPv4 and IPv6 addresses with built-in validation and operators. You can query by subnet directly:

SELECT * FROM system_events WHERE FromIpAddress <<= '192.0.2.0/24';

No string parsing, no LIKE '192.0.2.%' hacks. The <<= operator means "is contained in subnet" — it's indexable and fast.

5. Reference Tables with Foreign Keys

CREATE TABLE facilities (facility_id SMALLINT PRIMARY KEY, facility_name VARCHAR(100));
CREATE TABLE priorities (priority_id SMALLINT PRIMARY KEY, priority_name VARCHAR(100));

ALTER TABLE system_events
    ADD CONSTRAINT fk_facility FOREIGN KEY (Facility) REFERENCES facilities (facility_id);
ALTER TABLE system_events
    ADD CONSTRAINT fk_priority FOREIGN KEY (Priority) REFERENCES priorities (priority_id);

Syslog facility and priority are integers — facility=3, priority=6. Without context, those numbers are meaningless. Lookup tables enforce data integrity and make queries self-documenting.

In this deployment, the facility breakdown tells a clear story about the environment:

Facility	Count	Percentage
daemon	622,291	29.3%
local7	493,636	23.3%
cron	325,109	15.3%
auth	284,703	13.4%
user	154,686	7.3%

The dominance of daemon and local7 (commonly used by firewalls) suggests this is a network-heavy environment, which I suppose it is—I configured just about everything I could find in my home lab to forward syslog, including several managed switches, OpenWrt WiFi access points, and an OPNsense firewall. The fact that auth makes up 13.4% of the traffic is an immediate action item for me to look into; that represents hundreds of thousands of SSH sessions, sudo commands, and login attempts across the network.

The priority distribution looks like this:

Priority	Count	Percentage
info	1,240,125	58.5%
notice	405,596	19.1%
err	191,932	9.0%
debug	179,518	8.5%
warning	103,583	4.9%

While only 9.0% of the 2.12 million messages are errors, that's still nearly 200,000 discrete error events I need to investigate. Additionally, the 8.5% debug volume indicates that some of these lab devices are running in highly verbose modes. Tracking down and dialing back those debug logs is another thing I'll need to look into to keep the ingestion volume lean.

6. A Convenience View

CREATE VIEW v_system_events AS
    SELECT se.*, f.facility_name, p.priority_name
    FROM system_events se
    LEFT JOIN facilities f ON se.Facility = f.facility_id
    LEFT JOIN priorities p ON se.Priority = p.priority_id;

A view that joins the lookup tables means analysts never need to remember that facility_id = 3 means "daemon" — it's right there in the result set. Query the view, not the table.

7. Least-Privilege Database User

CREATE ROLE rsyslog WITH LOGIN NOSUPERUSER INHERIT NOCREATEDB NOCREATEROLE NOREPLICATION NOBYPASSRLS
    ENCRYPTED PASSWORD 'SCRAM-SHA-256$...';

GRANT USAGE ON SCHEMA public TO rsyslog;
GRANT USAGE, SELECT ON SEQUENCE public.system_events_id_seq TO rsyslog;
GRANT INSERT ON TABLE public.system_events TO rsyslog;

The rsyslog database user gets exactly what it needs: INSERT on the events table and SELECT on the sequence for the serial ID. Nothing more. SCRAM-SHA-256 authentication ensures passwords aren't stored as the legacy MD5 format.

8. SQL_ASCII Encoding for Heterogeneous Log Sources

CREATE DATABASE "syslog" WITH ENCODING 'SQL_ASCII' TEMPLATE template0;

Syslog messages come from dozens of different hosts, each with their own locale and encoding settings. A UTF-8 database would reject any byte sequence that isn't valid UTF-8 — and syslog is full of them. SQL_ASCII tells PostgreSQL to pass bytes through without validation. No encoding errors from a misbehaving device, no dropped logs.

The tradeoff is that you need to handle encoding in your application layer. For a log ingestion system that's primarily queried by humans who can tolerate the occasional garbled character, that's a reasonable trade.

What the Logs Look Like

After running this setup, the data tells a story about the infrastructure. The top log sources:

Source	Count
dhcpd	370,683
bbstored	286,399
sshd	182,323
radvd	131,274
dhclient	102,213
configd.py	92,668
filterlog	62,534

Since we already know this is an OPNsense-heavy home lab, seeing configd.py, filterlog, and radvd bubble to the top makes perfect sense. But the data provides a great reminder of exactly which services are the chattiest. The heavy dhcpd and dhclient traffic, for instance, highlights just how many DHCP leases are constantly being renewed by various lab and IoT devices across the network.

The ingestion rate shows the system ramping up as more hosts were configured:

Date	Rows
2026-03-29	16,375
2026-03-30	11,998
2026-03-31	31,312
2026-04-01	38,699
2026-04-02	39,349
2026-04-03	37,899
2026-04-04	18,489

From 16,375 rows in the earlier days to nearly 40,000 per day once all hosts were forwarding. At that rate, each monthly partition accumulates roughly 1 million rows — well within PostgreSQL's comfort zone, especially with partitioning keeping individual table sizes manageable.

Useful Queries

Here are practical queries for working with your log data:

Error Rate by Host (Last 24 Hours)

SELECT FromHost, count(*) AS errors
FROM system_events
WHERE Priority <= 3 AND ReceivedAt > now() - INTERVAL '24 hours'
GROUP BY 1 ORDER BY 2 DESC;

Full-Text Search with Highlighted Context

SELECT ReceivedAt, FromHost, EventSource,
        ts_headline('english', Message, q) AS context
FROM system_events, to_tsquery('english', 'authentication & failure') q
WHERE message_tsv @@ q
ORDER BY ReceivedAt DESC LIMIT 20;

Subnet Scan Using INET Type

SELECT FromHost, count(*) AS log_count
FROM system_events
WHERE FromIpAddress <<= '192.0.2.0/24'
GROUP BY 1 ORDER BY 2 DESC;

Error Trend Over Time

SELECT date_trunc('day', ReceivedAt) AS day, count(*) AS errors
FROM system_events
WHERE Priority <= 3
GROUP BY 1 ORDER BY 1;

Top Error Sources This Week

SELECT EventSource, FromHost, count(*) AS cnt
FROM system_events
WHERE Priority <= 3 AND ReceivedAt > now() - INTERVAL '7 days'
GROUP BY 1, 2 ORDER BY 3 DESC LIMIT 20;

Facility Breakdown with Percentages

SELECT facility_name, count(*) AS cnt,
       round(100.0 * count(*) / sum(count(*)) OVER (), 1) AS pct
FROM v_system_events
GROUP BY 1 ORDER BY 2 DESC;

Recent Errors from Specific Subnet (Optimized by Partial Error Index)

SELECT ReceivedAt, FromHost, Message
FROM system_events
WHERE Priority <= 3 
  AND FromIpAddress <<= '192.0.2.0/24'
  AND ReceivedAt > now() - INTERVAL '1 hour'
ORDER BY ReceivedAt DESC;

Top Generalized Error Messages (De-duplicating PIDs/IPs)

Syslog messages often contain unique numbers (like PIDs, ports, or IP addresses) that make identical errors look like distinct strings. This query uses regexp_replace to mask numbers, allowing you to group and count the true underlying error patterns.

SELECT regexp_replace(Message, '[0-9]+', 'X', 'g') AS generalized_message, 
       count(*) AS error_count
FROM system_events
WHERE Priority <= 3 
  AND ReceivedAt >= now() - INTERVAL '7 days'
GROUP BY generalized_message
ORDER BY error_count DESC
LIMIT 10;

PostgreSQL Tuning for Log Ingestion

It's worth noting the hardware profile of the environment described in this post: it is running in a small LXC container backed by unremarkable SSDs, with just 4 CPU cores and 8GB of RAM. Despite processing millions of logs and maintaining continuous GIN indexes, the system hovers at around 12% utilization. You do not need massive hardware to make this work.

To put some real numbers behind that efficiency, here is the actual performance data from this 8GB container:

Index Hit Rate: 99.87% (357,517 index scans vs. just 476 sequential scans). This proves the indexing strategy perfectly matches the query patterns. The database planner almost never resorts to scanning full tables.
Cache Hit Ratio: 84.3%. For a write-heavy log database, keeping 84% of accessed blocks in memory is a healthy balance. The SSDs handle the continuous sequential writes, while RAM serves the repetitive index updates and active lookups.
Zero-Overhead Historical Data. PostgreSQL's internal stats show the active month's partition (2026_04) has been autovacuumed 15 times to keep up with the insert churn, while all historic partitions show exactly 0 runs. Maintenance work is isolated entirely to the active partition, meaning your retained data costs zero CPU cycles to keep around.

That said, the default PostgreSQL configuration isn't optimized for write-heavy log workloads. Here is the real postgresql.conf block used to hit those numbers on a 4-core, 8GB machine:

# Memory
shared_buffers = 2GB                     # 25% of 8GB RAM
maintenance_work_mem = 1GB               # faster index creation and partition operations
work_mem = 32MB                          # per-operation memory for sorts and joins

# Disk & Asynchronous Behavior
effective_io_concurrency = 256           # highly parallel I/O for SSDs
max_worker_processes = 23                # enough workers for background jobs
max_parallel_workers = 4                 # matches the 4 CPU cores
default_toast_compression = lz4          # faster text compression on Postgres 14+

# WAL and Checkpoints
max_wal_size = 2GB
min_wal_size = 512MB
checkpoint_completion_target = 0.9       # spread checkpoint I/O out smoothly

# Autovacuum
autovacuum_max_workers = 4               # keep up with partition churn

The Magic of LZ4 Compression

One setting in that config deserves a special shoutout: default_toast_compression = lz4.

PostgreSQL automatically compresses large text columns behind the scenes (a mechanism called TOAST, or The Oversized-Attribute Storage Technique). By default, it uses an older, slower algorithm called PGLZ. If you are running PostgreSQL 14 or newer, switching this default to LZ4 provides drastically faster compression during log ingestion and faster decompression during queries. For text-heavy workloads like syslog Message and RawMessage blobs, this is one of the easiest ways to drop your CPU utilization almost instantly.

Properly Sizing `work_mem`

Another critical adjustment is work_mem = 32MB. In a log database, your queries almost exclusively involve massive GROUP BY and ORDER BY operations (like aggregating errors or sorting chronologically).

By default, PostgreSQL's work_mem is extremely conservative (4MB). If an analytical query requires more memory than that to execute a sort, PostgreSQL will spill the workspace over to your SSD as a temporary file, causing a massive latency spike. Giving those analytical queries 32MB of dedicated RAM ensures that complex groupings (like the regex deduplication query above) execute entirely in memory. In my 8GB container, that regex query groups over 10,000 recent errors in just ~106 milliseconds without ever touching the disk.

Backup and Restore

Partitioned tables behave mostly like regular tables for backup purposes, with a few quirks worth knowing:

pg_dump dumps the parent table definition and all partitions. Restoring recreates the full partitioned structure.
pg_basebackup works normally — partitions are just tables on disk.
Restoring a single partition — you can pg_dump a specific partition table and restore it independently, useful for recovering from a data archive.
pg_restore --jobs — parallel restore works across partitions, speeding up large restores.

For compliance archival, you can pg_dump specific partitions: pg_dump -t system_events_2026_03 > march2026.sql

One gotcha: if you restore into a database that already has partitions, you'll get conflicts. Always restore into a fresh database or drop existing partitions first.

Troubleshooting

Queue Full

If rsyslog's queue fills up (20,000 messages), new messages are dropped. Check with:

# Check rsyslog queue status
journalctl -u rsyslog | grep -i "queue"

Mitigate by increasing queue.size or adding disk-backed queuing:

queue.type="disk"
queue.spoolDirectory="/var/spool/rsyslog"
queue.maxDiskSpace="1g"

pg_cron Job Failed

Check the job run history:

SELECT jobid, runid, status, return_message, start_time, end_time
FROM cron.job_run_details
ORDER BY start_time DESC LIMIT 10;

Common failures:

"connection failed" — the target database wasn't reachable. Check that the database exists and the role has permissions.
"no partition of relation found" — a partition is missing. Run CALL public.create_future_partitions('system_events', 3); manually.
Permission denied — ensure the rsyslog role has the necessary grants on new partitions.

Partition Missing on Insert

If rsyslog logs an error like "no partition of relation 'system_events' found for row," the partition for that date range doesn't exist yet. This happens if pg_cron hasn't run or failed. Fix it manually:

CALL public.create_monthly_partition('system_events', 2026, 5);

Slow Queries

Use EXPLAIN (ANALYZE, BUFFERS) to check if partition pruning is working:

EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM system_events
WHERE ReceivedAt >= '2026-03-01' AND ReceivedAt < '2026-04-01';

Look for the partitions being scanned in the plan. If all partitions appear, your query isn't filtering on the partition key.

Check for unused indexes:

SELECT indexrelname, idx_scan,
       pg_size_pretty(pg_relation_size(indexrelid)) AS index_size
FROM pg_stat_user_indexes
WHERE relname LIKE 'system_events_%'
ORDER BY idx_scan;

If idx_scan is 0 after weeks of operation, consider dropping that index.

High CPU During Partition Creation

Creating many partitions at once can be CPU-intensive. The create_future_partitions procedure creates them sequentially to spread the load. If you need to create many partitions manually, consider adding a small delay between calls or running during off-peak hours.

Comparison: PostgreSQL vs. Elasticsearch

Feature	PostgreSQL	Elasticsearch
Time-based data split	Range partitions	Time-based indices
Pruning	Partition pruning at plan time	Shard routing
Retention	Instant `DROP TABLE`	Delete index
Automation	`pg_cron` + stored procedures	Index Lifecycle Management (ILM)
IP queries	Native `INET` type with subnet operators	IP field type
Self-documenting queries	Lookup tables + foreign keys + views	Index templates + Kibana
Ingestion resilience	rsyslog queue buffering	Logstash queue
Data integrity	ACID, foreign keys	Eventual consistency
Query language	SQL	Query DSL
Full-text search	`tsvector` + GIN index	Native, highly optimized
Aggregations at scale	Good up to ~100M rows	Excellent at petabyte scale
Operational Complexity	Low (uses existing infrastructure, standard SQL)	High (separate cluster, specialized knowledge needed)

PostgreSQL wins at structured log analysis with relational context, ACID guarantees, and operational simplicity. Elasticsearch wins at full-text search and aggregations at massive scale.

Where to Go From Here

This setup is production-ready as-is, but there are natural enhancements you can add to suit your needs:

Materialized Views for Dashboards

A materialized view is a pre-computed query result stored on disk — like a cached query. Unlike a regular view that re-runs every time, a materialized view stores the results and you refresh it on a schedule.

CREATE MATERIALIZED VIEW mv_daily_error_counts AS
SELECT
    date_trunc('day', ReceivedAt) AS day,
    FromHost,
    EventSource,
    Priority,
    count(*) AS error_count
FROM system_events
WHERE Priority <= 3
GROUP BY 1, 2, 3, 4;

CREATE UNIQUE INDEX ON mv_daily_error_counts (day, FromHost, EventSource, Priority);

Refresh it on a schedule:

REFRESH MATERIALIZED VIEW CONCURRENTLY mv_daily_error_counts;

The CONCURRENTLY keyword means the view stays queryable during refresh — no downtime for dashboards. Schedule it via pg_cron every hour and you have a near-real-time error dashboard backed by a tiny table instead of scanning millions of rows.

Composite Indexes for Specific Query Patterns

A composite index covers multiple columns in a single structure. The column order matters because of the leftmost prefix rule — an index on (A, B, C) supports queries on A, A+B, and A+B+C, but not on B alone or C alone.

CREATE INDEX idx_system_events_host_time ON system_events
    (FromHost, ReceivedAt DESC);

CREATE INDEX idx_system_events_source_time ON system_events
    (EventSource, ReceivedAt DESC);

These support your most common queries efficiently:

-- "Show me all logs from this host, newest first"
SELECT * FROM system_events WHERE FromHost = 'firewall-primary' ORDER BY ReceivedAt DESC;

-- "Show me all sshd logs from today"
SELECT * FROM system_events
WHERE EventSource = 'sshd' AND ReceivedAt > now() - INTERVAL '1 day';

Horizontal Scale-Out

If your ingestion volume ever outgrows a single node, this architecture scales out cleanly at both layers:

Rsyslog Scale-Out: You can run multiple instances of Rsyslog behind a network load balancer (like HAProxy or F5) to distribute the incoming TCP/UDP traffic. Because the Rsyslog instances are completely stateless log routers, you can scale them horizontally to handle massive broadcast storms, with all nodes writing concurrently to the central database.
PostgreSQL Scale-Out: To handle high dashboard concurrency or complex analytical queries, you can easily spin up Read Replicas to offload read pressure from your primary database. If your write volume eventually exceeds a single node's capacity, you can shard your time-partitioned tables across a cluster of worker nodes using distributed SQL or time-series extensions, unlocking petabyte-scale retention.

Other Enhancements

JSONB structured parsing — Parse common log formats (nginx, auth, systemd) into structured JSONB columns for targeted queries instead of LIKE '%pattern%'.
Row-Level Security — Restrict log access by team, host group, or customer using PostgreSQL RLS policies.
Data Archiving — COPY TO old partitions to an external file before dropping them, for compliance retention without bloating the database.
Grafana dashboards — Connect PostgreSQL directly as a datasource for real-time log dashboards, or use Metabase for self-service exploration by non-SQL users.
TLS Encryption — Standard syslog over TCP/UDP is unencrypted plaintext. If you are forwarding logs across untrusted networks and your client devices support it, configure rsyslog to use the TLS module to encrypt your log traffic in transit.

The beauty of this approach is that each enhancement uses PostgreSQL features you're already running. No new infrastructure, no new operational burden — just SQL.

Running It

Prerequisites: By default, Linux distributions do not ship with the PostgreSQL or RELP modules. For Debian/Ubuntu-based systems, install them first (omit rsyslog-relp if you are sticking with TCP/UDP):

sudo apt install rsyslog-pgsql rsyslog-relp

Create the database and schema with the provided syslog.sql
Drop the rsyslog server config into /etc/rsyslog.conf (or /etc/rsyslog.d/)
Configure clients with the forwarding config
Restart rsyslog: systemctl restart rsyslog
Verify: SELECT count(*) FROM v_system_events;

The whole pipeline is self-maintaining: partitions roll forward and backward automatically, old data vanishes without DELETE overhead, the queue absorbs transient failures, and full-text search is always in sync. For a log ingestion system, that's exactly what you want.

Appendix

The Problem with Flat File Logs

Flat file syslog works fine until you need to answer questions like:

"Show me all authentication failures from the last 48 hours across all hosts"
"How many error-level messages did our firewall generate this week?"
"What's the trend in disk warnings over the past three months?"

With flat files, these require grep, awk, and a lot of patience. With a database, they're a single SQL query.

Why Not Elasticsearch?

Elasticsearch is purpose-built for log aggregation, and it excels at full-text search and aggregations at massive scale. But it comes with real costs:

A separate cluster to provision, monitor, and upgrade
For small-to-medium deployments, Elasticsearch's operational overhead can still be significant compared to adding tables to an existing PostgreSQL instance
Index lifecycle management (ILM) to configure and maintain
No ACID (Atomicity, Consistency, Isolation, Durability) guarantees — log ingestion can lose data during restarts
Query Domain-Specific Language (DSL) that's powerful but has its own learning curve

PostgreSQL, on the other hand, is probably already running in your stack. It gives you ACID guarantees, relational joins, foreign keys, and a query language your team already knows. With the right schema design, it handles log ingestion and retention surprisingly well.

This isn't to say PostgreSQL replaces Elasticsearch at petabyte scale. But for small-to-medium deployments — a few million rows, dozens of hosts — a well-designed PostgreSQL setup can cover 80% of the use cases with 20% of the operational overhead.

Full SQL Schema

Here's the complete schema for reference. Save this as syslog.sql and run it with psql -f syslog.sql.

CREATE DATABASE "syslog" WITH ENCODING 'SQL_ASCII' TEMPLATE template0;

\connect syslog;

CREATE TABLE system_events
(
    ID SERIAL NOT NULL,
    DeviceReportedTime TIMESTAMP WITH TIME ZONE NULL,
    EventID VARCHAR(60) NULL,
    EventLogType VARCHAR(60),
    EventSource VARCHAR(60),
    Facility SMALLINT NULL,
    FromHost VARCHAR(255) NULL,
    FromPort INT NULL,
    FromIpAddress INET NULL,
    HostName VARCHAR(255) NULL,
    Message TEXT,
    Priority SMALLINT NULL,
    RawMessage TEXT,
    ReceivedAt TIMESTAMP WITH TIME ZONE NOT NULL,
    RFC5424AppName VARCHAR(255) NULL,
    RFC5424MsgID VARCHAR(255) NULL,
    RFC5424ProcID VARCHAR(255) NULL,
    RFC5424ProtocolVersion SMALLINT NULL,
    RFC5424StructuredData TEXT NULL,
    SysLogTag VARCHAR(60),
    message_tsv tsvector GENERATED ALWAYS AS (to_tsvector('english', Message)) STORED,
    PRIMARY KEY (ID, ReceivedAt)
) PARTITION BY RANGE (ReceivedAt);

-- BRIN indexes for time-ordered columns (much smaller than B-tree for append-only data)
CREATE INDEX idx_system_events_receivedat_brin ON system_events USING brin (ReceivedAt DESC);
CREATE INDEX idx_system_events_devicereportedtime_brin ON system_events USING brin (DeviceReportedTime DESC);
-- B-tree indexes for common query filters
CREATE INDEX idx_system_events_facility ON system_events (Facility);
CREATE INDEX idx_system_events_priority ON system_events (Priority);
CREATE INDEX idx_system_events_fromhost ON system_events (FromHost);
CREATE INDEX idx_system_events_eventsource ON system_events (EventSource);
-- Partial index for error-level logs only (Priority 0-3: emerg, alert, crit, err)
CREATE INDEX idx_system_events_errors ON system_events (FromHost, EventSource, ReceivedAt DESC)
    WHERE Priority <= 3;
-- Full-text search index on Message
CREATE INDEX idx_system_events_message_fts ON system_events USING gin (message_tsv);

-- Create table for Facility mappings
CREATE TABLE facilities (
    facility_id SMALLINT PRIMARY KEY,
    facility_name VARCHAR(100) NOT NULL
);

-- Insert common syslog facility values
INSERT INTO facilities (facility_id, facility_name) VALUES
(0, 'kern'), (1, 'user'), (2, 'mail'), (3, 'daemon'),
(4, 'auth'), (5, 'syslog'), (6, 'lpr'), (7, 'news'),
(8, 'uucp'), (9, 'cron'), (10, 'authpriv'), (11, 'ftp'),
(12, 'ntp'), (13, 'security'), (14, 'console'), (15, 'solaris-cron'),
(16, 'local0'), (17, 'local1'), (18, 'local2'), (19, 'local3'),
(20, 'local4'), (21, 'local5'), (22, 'local6'), (23, 'local7');

-- Create table for Priority mappings
CREATE TABLE priorities (
    priority_id SMALLINT PRIMARY KEY,
    priority_name VARCHAR(100) NOT NULL
);

-- Insert common syslog priority values
INSERT INTO priorities (priority_id, priority_name) VALUES
(0, 'emerg'), (1, 'alert'), (2, 'crit'), (3, 'err'),
(4, 'warning'), (5, 'notice'), (6, 'info'), (7, 'debug');

-- Add foreign key constraints
ALTER TABLE system_events
    ADD CONSTRAINT fk_facility FOREIGN KEY (Facility) REFERENCES facilities (facility_id);
ALTER TABLE system_events
    ADD CONSTRAINT fk_priority FOREIGN KEY (Priority) REFERENCES priorities (priority_id);

-- A convenient view
CREATE VIEW v_system_events AS
    SELECT se.*, f.facility_name, p.priority_name
    FROM system_events se
    LEFT JOIN facilities f ON se.Facility = f.facility_id
    LEFT JOIN priorities p ON se.Priority = p.priority_id;

-- Create initial partitions
CREATE TABLE system_events_2026_03 PARTITION OF system_events
    FOR VALUES FROM ('2026-03-01') TO ('2026-04-01');
CREATE TABLE system_events_2026_04 PARTITION OF system_events
    FOR VALUES FROM ('2026-04-01') TO ('2026-05-01');
CREATE TABLE system_events_2026_05 PARTITION OF system_events
    FOR VALUES FROM ('2026-05-01') TO ('2026-06-01');

-- Procedure to create a new monthly partition
CREATE OR REPLACE PROCEDURE public.create_monthly_partition(p_table_name TEXT, year INT, month INT)
LANGUAGE plpgsql AS $$
DECLARE
    partition_name TEXT;
    start_date DATE;
    end_date DATE;
BEGIN
    start_date := make_date(year, month, 1);
    end_date := (start_date + INTERVAL '1 month')::DATE;
    partition_name := format('%s_%s_%s', p_table_name, year, lpad(month::TEXT, 2, '0'));

    IF NOT EXISTS (
        SELECT FROM pg_tables WHERE tablename = partition_name AND schemaname = 'public'
    ) THEN
        EXECUTE format(
            'CREATE TABLE public.%I PARTITION OF public.%I FOR VALUES FROM (%L) TO (%L)',
            partition_name, p_table_name, start_date, end_date
        );
        RAISE NOTICE 'Created partition public.%', partition_name;
    ELSE
        RAISE NOTICE 'Partition public.% already exists, skipping', partition_name;
    END IF;
EXCEPTION
    WHEN OTHERS THEN
        RAISE WARNING 'Error creating partition public.%: %', partition_name, SQLERRM;
        RAISE;
END;
$$;

-- Procedure to create partitions for the next N months
CREATE OR REPLACE PROCEDURE public.create_future_partitions(p_table_name TEXT, months_ahead INT)
LANGUAGE plpgsql AS $$
DECLARE
    current_date DATE := CURRENT_DATE;
    target_date DATE;
    target_year INT;
    target_month INT;
    i INT;
BEGIN
    FOR i IN 0..months_ahead LOOP
        target_date := current_date + (i * INTERVAL '1 month');
        target_year := EXTRACT(YEAR FROM target_date);
        target_month := EXTRACT(MONTH FROM target_date);
        CALL public.create_monthly_partition(p_table_name, target_year, target_month);
    END LOOP;
EXCEPTION
    WHEN OTHERS THEN
        RAISE WARNING 'Error in create_future_partitions for %: %', p_table_name, SQLERRM;
        RAISE;
END;
$$;

-- Procedure to drop partitions older than a specified interval
CREATE OR REPLACE PROCEDURE public.drop_old_partitions(p_table_name TEXT, retention_interval INTERVAL)
LANGUAGE plpgsql AS $$
DECLARE
    partition_name TEXT;
    dropped_count INT := 0;
BEGIN
    FOR partition_name IN (
        SELECT tables.table_name
        FROM information_schema.tables
        WHERE tables.table_name LIKE p_table_name || '_%'
        AND tables.table_name < p_table_name || '_' || to_char(CURRENT_DATE - retention_interval, 'YYYY_MM')
        AND tables.table_schema = 'public'
    )
    LOOP
        EXECUTE 'DROP TABLE IF EXISTS public.' || quote_ident(partition_name);
        RAISE NOTICE 'Dropped partition public.%', partition_name;
        dropped_count := dropped_count + 1;
    END LOOP;
    IF dropped_count = 0 THEN
        RAISE NOTICE 'No partitions dropped for % older than %', p_table_name, retention_interval;
    END IF;
EXCEPTION
    WHEN OTHERS THEN
        RAISE WARNING 'Error dropping partitions for %: %', p_table_name, SQLERRM;
        RAISE;
END;
$$;

CALL public.create_future_partitions('system_events', 4);
CALL public.drop_old_partitions('system_events', INTERVAL '3 months');

GRANT ALL ON SCHEMA public TO postgres;
GRANT ALL ON TABLE public.system_events TO postgres;
GRANT EXECUTE ON PROCEDURE public.create_monthly_partition(TEXT, INT, INT) TO postgres;
GRANT EXECUTE ON PROCEDURE public.create_future_partitions(TEXT, INT) TO postgres;
GRANT EXECUTE ON PROCEDURE public.drop_old_partitions(TEXT, INTERVAL) TO postgres;

GRANT USAGE ON SCHEMA public TO rsyslog;
GRANT USAGE, SELECT ON SEQUENCE public.system_events_id_seq TO rsyslog;
GRANT INSERT ON TABLE public.system_events TO rsyslog;

-- Run this in the postgres database where pg_cron is installed
\connect postgres

CREATE EXTENSION IF NOT EXISTS pg_cron WITH SCHEMA public;

-- pg_cron job to delete partitions older than 3 months
SELECT cron.unschedule('drop_old_system_events_partitions');
SELECT cron.schedule_in_database(
    'drop_old_system_events_partitions',
    '0 14 1 * *',
    $$CALL public.drop_old_partitions('system_events', INTERVAL '3 months');$$,
    'syslog'
);

-- pg_cron job to create partitions for the next 3 months
SELECT cron.unschedule('create_future_system_events_partitions');
SELECT cron.schedule_in_database(
    'create_future_system_events_partitions',
    '5 14 1 * *',
    $$CALL public.create_future_partitions('system_events', 3);$$,
    'syslog'
);

Full Rsyslog Server Configuration

Save this to /etc/rsyslog.conf (or inside /etc/rsyslog.d/ depending on your OS) on the server receiving logs.

# Load required modules
module(load="imudp") # UDP syslog reception
module(load="imtcp") # TCP syslog reception
module(load="imrelp") # RELP syslog reception

# Define inputs
input(
    type="imudp"
    port="514"
    rateLimit.interval="60"
    rateLimit.burst="2000"
)

input(
    type="imtcp"
    port="514"
)

input(
    type="imrelp"
    port="2514"
    maxDataSize="10k"
)

# Load PostgreSQL output module
module(load="ompgsql")

# Define the SQL template
template(
    name="LogToPgSQL" type="list" option.stdsql="on" ) {
    constant(value="INSERT INTO system_events (DeviceReportedTime, EventID, EventLogType, EventSource, Facility, FromHost, FromIpAddress, HostName, Message, Priority, RawMessage, ReceivedAt, SysLogTag) VALUES ('")
    property(name="timereported" dateFormat="pgsql" date.inUTC="on")
    constant(value="', '")
    property(name="msgid")
    constant(value="', 'syslog', '")
    property(name="programname")
    constant(value="', '")
    property(name="syslogfacility")
    constant(value="', '")
    property(name="fromhost")
    constant(value="', '")
    property(name="fromhost-ip")
    constant(value="', '")
    property(name="hostname")
    constant(value="', '")
    property(name="msg" escape="sql")
    constant(value="', ")
    property(name="syslogpriority")
    constant(value=", '")
    property(name="rawmsg" escape="sql")
    constant(value="', '")
    property(name="timegenerated" dateFormat="pgsql" date.inUTC="on")
    constant(value="', '")
    property(name="syslogtag")
    constant(value="')")
}

# Action to write to PostgreSQL
action(
    type="ompgsql" server="localhost"
    user="rsyslog" pass="YOUR_STRONG_PASSWORD_HERE"
    db="syslog" template="LogToPgSQL"

    queue.type="linkedList"
    queue.size="20000"
    queue.workerThreads="2"
)

Full Rsyslog Client Configuration

Save this to /etc/rsyslog.d/99-forward.conf on any Linux client you want to forward logs from. Be sure to replace 192.0.2.1 with the actual IP of your Rsyslog server.

# Forward all logs via TCP to the central server
*.* action(
    type="omfwd"
    target="192.0.2.1"
    port="514"
    protocol="tcp"
    template="RSYSLOG_SyslogProtocol23Format" # RFC 5424 format
)

Manage the health of your CLI tools at scale

Dean Hamstead — Wed, 01 Apr 2026 10:35:16 +0000

Your services have dashboards, tracing, and alerting. Your CLI tools print to STDOUT and exit. When something breaks, debugging starts at the API gateway -- everything upstream is a black box. This makes no sense.

If your CLI talks to an API, it's part of the request path. Instrument it like any other participant.

This post describes how we instrumented an internal Perl CLI -- the same mycli tool from our earlier post on fatpacking -- with syslog logging, StatsD metrics, and correlation IDs. The post is strongly biased towards tooling internal to an organisation, which has the luxury of being opinionated: you control the deployment targets, you know where syslog goes, and you can lean on solved infrastructure rather than building your own. The principles generalise to any language and any CLI that talks to an API.

Why observability matters in CLI tools

Web services get dashboards as a matter of course[1]. Error rates, latency percentiles, request counts -- these are table stakes for any production service. CLI tools rarely get the same treatment, even when they're used just as heavily.

Once your CLI emits metrics, you can build per-tool dashboards that show error rates broken down by command, by user cohort, by API version, by CLI version, by deployment target. This is the same dimensional analysis you'd do for a web service, applied to a tool that runs on someone's laptop.

This integrates naturally with operational practices you're probably already using:

Continuous deployment. When you ship a new CLI version, the dashboard shows whether error rates changed. If command.device_list.errors spikes after a release, you know immediately -- not when someone files a ticket three days later.
Rollback decisions. If error rates climb after a release, the dashboard tells you in minutes -- roll back now, debug later. Without metrics, you're guessing whether the new version is the cause or a coincidence.
Canary deployments. Roll the new version to 10% of jumpboxes. Compare http.timing and http.errors between the canary and the stable cohort. The same deployment strategy that works for services works for CLI tools, but only if you have the metrics to compare.
Feature flags. If a new feature is gated behind a flag, metrics tell you whether the flagged code path is slower, more error-prone, or unused. Without instrumentation, feature flag decisions are based on "nobody complained".
Incident management. During a site event, the CLI dashboard shows whether the tool is contributing to or affected by the problem. A spike in http.status.503 from the CLI tells the incident commander that the API is rejecting requests before users report it. Conversely, if the CLI error rate is flat during an incident, you can rule it out as a contributing factor.
Adoption and deprecation. Metrics answer "is anyone still using the v1 endpoint?" and "has the team migrated to the new auth flow?" without surveys or guesswork.

The point is not that CLI tools are special -- it's that they're not. They're participants in the same distributed system as your services, and they deserve the same observability treatment. The investment is small: a correlation ID, a handful of counters, and a logging lifecycle. The return is that your CLI becomes a first-class citizen in your operational tooling rather than a blind spot.

[1] Yours does, right?

The three layers

We instrument at three levels, each serving a different audience and persistence model:

 Layer           Audience              Persistence
 -----           --------              -----------
 Verbose mode    Developer at terminal Ephemeral (STDERR)
 Syslog          Ops / incident review Durable (centralised logs)
 StatsD          Dashboards / alerting Aggregated (time-series)

A developer debugging their own command uses --verbose. An on-call engineer investigating a reported issue searches syslog by invocation ID. A platform team monitors command usage and error rates on dashboards. Same underlying data, different consumers, different retention.

Each layer is controlled independently and opt-in:

# Syslog only
MYCLI_LOG=1 mycli device list

# Verbose only (no syslog, no metrics)
mycli device list --verbose

# Everything
MYCLI_LOG=1 mycli device list --verbose

StatsD metrics emit whenever a statsd_host is configured -- no-ops otherwise. Syslog requires MYCLI_LOG=1 -- deliberately opt-in, since CLI tools run on personal machines and writing to syslog on every invocation without consent would be surprising.

The verbose layer itself has depth. --verbose shows the shape of the HTTP conversation -- method, URL, status, timing -- but deliberately omits headers and bodies to keep the output scannable. When that isn't enough, plugging in LWP::ConsoleLogger::Everywhere via perl -M gives a full HTTP trace without the CLI needing to build one. More on this in the debugging spectrum section below.

Invocation ID: the correlation key

Every mycli invocation generates a random 8-character hex ID at startup:

my @chars = ('0' .. '9', 'a' .. 'f');
my $id = join '', map { $chars[ int(rand @chars) ] } 1 .. 8;

This ID appears in three places:

Every syslog message -- prefixed as [f7a3b1c2]
Every HTTP request -- sent as the X-Invocation-Id header
Verbose STDERR output -- printed at startup

The server-side API logs this header alongside its own request ID. To trace a failing command end-to-end:

# Find the CLI side
grep 'f7a3b1c2' /var/log/mycli.log

# Find the server side
grep 'f7a3b1c2' /var/log/api.log

One string, full picture. No timestamps to correlate, no guessing which request came from which terminal.

User-Agent

In addition to the invocation ID, set the User-Agent header to mycli/<version>. This is trivial and gives the server side a way to filter by CLI version without any custom header support -- useful for canary deployment analysis and for spotting users running outdated versions.

Two-way correlation

The API returns its own request ID in a response header (X-Request-Id). The CLI logs this too:

[f7a3b1c2] http: 200 OK (142ms, application/json, 8431 bytes) req=a1b2c3d4

This gives you a join key in both directions: from the CLI's invocation ID you can find the server's request ID, and vice versa. When a user reports "mycli gave me an error", the request ID in the error message leads straight to the server-side trace.

What the server needs to do

The correlation only works if the server participates. The requirements are minimal:

Log the X-Invocation-Id header from incoming requests. Most API frameworks can do this with a single middleware or access log configuration change.
Return a request ID in every response (e.g., X-Request-Id). Many frameworks generate this by default.
Propagate both IDs into the server's own tracing and logging. If the API uses structured logging or distributed tracing, attach the invocation ID as a field or span attribute so it appears in the same search results.

If the server doesn't log the invocation ID, the CLI-side correlation still works (you can grep your CLI logs by invocation ID), but you lose the end-to-end join. If the server doesn't return a request ID, the CLI can still log its own invocation ID, but the user can't hand a request ID to the API team and say "look this up".

The ideal state is both: the CLI sends its ID, the server sends its ID, and both sides log both. This is a two-line change on the server and it makes every future debugging session faster.

Structured syslog

Every invocation logs a structured lifecycle to syslog:

Startup

[f7a3b1c2] startup: cli: mycli device list --status Active
[f7a3b1c2] startup: perl: 5.36.0 on linux
[f7a3b1c2] startup: env: API_KEY=ab12****, SERVER_URL=https://api.internal
[f7a3b1c2] config: key source: file (~/.config/mycli/api-key)
[f7a3b1c2] config: format: table, fields: all, tty

The API key is masked -- first four characters, then ****. Enough to identify which key is in use without leaking it to logs.

HTTP requests

[f7a3b1c2] http: GET https://api.internal/v1/devices
[f7a3b1c2] http: 200 OK (142ms, application/json, 8431 bytes) req=a1b2c3d4

Every request/response pair is logged with method, URL, status, elapsed time, content type, response size, and the server's request ID.

Shutdown

[f7a3b1c2] device_list: done (387ms, 24 results, 2 requests, cache 3/1)

One line summarising the entire command: wall-clock time, result count, number of HTTP requests made, and resolve cache statistics (3 items cached across 1 resource type).

Always format, conditionally emit

A subtle design choice: the logger always formats every message, even when logging is disabled. Only the syslog() call is conditional:

sub _emit {
    my ($self, $priority, $context, $detail) = @_;
    my $msg = sprintf '[%s] %s: %s', $self->{_id}, $context, $detail;
    syslog($priority, '%s', $msg) if $self->{_enabled};
    return $msg;
}

This means formatting bugs surface during normal development, not only when someone enables logging in production. The cost is negligible -- sprintf is fast.

A note on philosophy: when syslog is enabled, all levels are transmitted -- info, debug, error. There is no runtime knob to suppress debug messages. The belief behind this is that logging should always be on in production, not enabled after a problem is suspected. The time you most need debug-level detail is exactly the time you can't reproduce the issue. You can never have too much log detail, with the obvious exception of user or employee personal data, which should never be logged at any level.

What not to log

The API key masking (ab12****) is one example of a broader principle: log enough to identify, not enough to exploit.

Credentials and secrets -- mask API keys, tokens, and passwords. Show enough characters to distinguish between keys (we show four), then mask the rest. Apply the same caution to environment variables and URL query parameters that may carry tokens.
Request and response bodies -- don't log them. They may contain customer data, PII, or sensitive business logic. Log metadata (status, timing, size) but never content. Body inspection is what LWP::ConsoleLogger is for -- interactive, ephemeral, on-demand.

StatsD metrics

Every command emits a standard set of metrics to StatsD:

Per-command metrics

Metric	Type	Description
`mycli.command.<cmd>.calls`	counter	Command invocations
`mycli.command.<cmd>.timing`	timing	Wall-clock duration (ms)
`mycli.command.<cmd>.results`	gauge	Items returned
`mycli.command.<cmd>.errors`	counter	Unhandled exceptions

The command name is derived from the class hierarchy: MyCLI::App::Command::device::list becomes device_list.

Per-HTTP metrics

Metric	Type	Description
`mycli.http.calls`	counter	Total HTTP requests
`mycli.http.timing`	timing	Per-request duration (ms)
`mycli.http.errors`	counter	Non-2xx responses
`mycli.http.status.<code>`	counter	Per-status-code breakdown

Operational metrics

Metric	Type	Description
`mycli.auth.key_source.<src>`	counter	Where the API key came from
`mycli.auth.url_source.<src>`	counter	Where the server URL came from
`mycli.config.file.found`	counter	Config file was loaded
`mycli.config.file.none`	counter	No config file found
`mycli.output.format.<name>`	counter	Output format selection

What this tells you

The metrics answer questions that logs can't:

What commands are people actually using? -- sort command.*.calls by count. If nobody uses crossconnect list, don't spend time improving it.
Is the API getting slower? -- http.timing percentiles over time. The CLI is seeing the same latency as your users, including TLS negotiation and DNS.
Are auth errors increasing? -- http.status.401 spike means keys are being rotated or revoked.
How are people authenticating? -- auth.key_source.env vs auth.key_source.file tells you whether your team has adopted the recommended credential flow.
What output formats matter? -- if 90% of usage is output.format.json, your table renderer is mostly aesthetic.

Metric naming conventions

Prefix every metric with the tool name (mycli.*) to avoid collisions in a shared StatsD instance. Use a consistent dot-separated hierarchy (mycli.command.<cmd>.calls) rather than flat names -- this makes metrics discoverable by browsing the tree. Watch cardinality: derive command names from a fixed set (like the class hierarchy) rather than user input, and keep dynamic segments like http.status.<code> to naturally bounded sets.

Verbose mode and the debugging spectrum

The three layers above cover durable observability -- data that outlives the terminal session. But the most common debugging scenario is someone at a keyboard wondering why their command isn't working. For this, the CLI has three levels of HTTP visibility:

Level 1: Silent (default)

No HTTP output. The user sees formatted results only. Syslog and metrics still capture everything in the background.

Level 2: `--verbose`

--> GET https://api.internal/v1/devices?status=Active
<-- 200 OK (142ms, application/json, 8431 bytes)

Printed to STDERR so it doesn't interfere with STDOUT piping. Shows method, URL, status, timing, and size. This is enough for "is my request hitting the right endpoint?" and "why is this slow?".

The design choice here is restraint. Verbose mode shows the shape of the conversation -- what was asked, what came back, how long it took. It deliberately omits headers and bodies. This keeps the output scannable when a command makes multiple requests.

Level 3: `LWP::ConsoleLogger::Everywhere`

When --verbose isn't enough -- when you need to see request headers, response headers, and full bodies -- plug in LWP::ConsoleLogger::Everywhere:

# From source
perl -MLWP::ConsoleLogger::Everywhere -Ilib bin/mycli device get 42

# Fatpacked binary (with API key redaction)
LWPCL_REDACT_HEADERS=Authorization \
  PERL5OPT="-MLWP::ConsoleLogger::Everywhere" \
  ./mycli-packed device get 42

This is a full HTTP trace: every header, every byte of the request and response body, formatted and syntax-highlighted. It's invaluable for debugging serialisation issues, unexpected headers, or auth failures.

The reason we don't build this into --verbose is that it's a different tool for a different job. Verbose mode is for operators; full HTTP tracing is for developers debugging the CLI itself. The -M flag means the capability is always available without cluttering the option namespace or adding a dependency that most users will never need.

Error reporting and surfacing correlation IDs

When the API returns an error, the CLI needs to show the user enough information to report the problem without overwhelming them with internals. Our error output includes the server's request ID:

Error: 403 Forbidden
  The API key does not have permission to access this resource.
  Request ID: a1b2c3d4

The request ID is the bridge between the user and the operations team. "It gave me a 403, request ID a1b2c3d4" is a complete bug report. The on-call engineer greps the server logs for a1b2c3d4, finds the full request context (authenticated user, requested resource, policy that denied access), and resolves the issue -- without asking the user to reproduce it, enable verbose mode, or paste terminal output.

The invocation ID doesn't appear in normal error output -- it's an internal correlation key for log analysis, not a user-facing artifact. If syslog is enabled, the invocation ID is already in the logs alongside the request ID, providing the join in both directions.

The execution wrapper

All of this comes together in the base command's execute() method, which wraps every leaf command:

sub execute {
    my ($self, $opt, $args) = @_;
    my $cmd     = $self->_metric_name;
    my $start   = Time::HiRes::time();

    $self->logger->info($cmd, 'start');
    $self->metrics->increment("command.$cmd.calls");

    eval { $self->_execute($opt, $args) };

    my $elapsed_ms   = int((Time::HiRes::time() - $start) * $MS_PER_SEC);
    my $requests     = $self->client->request_count;
    my $result_count = $self->{_result_count};

    $self->metrics->timing("command.$cmd.timing", $elapsed_ms);
    $self->metrics->gauge("command.$cmd.results", $result_count)
        if defined $result_count;

    if (my $err = $@) {
        $self->metrics->increment("command.$cmd.errors");
        $self->logger->error($cmd, $err);
        die $err;
    }

    $self->logger->info($cmd, sprintf 'done (%dms, %s results, %d requests)',
        $elapsed_ms, $result_count // 'n/a', $requests);
}

Leaf commands implement _execute() and don't think about observability at all. They call $self->client->get(...), render results, and return. The wrapper handles timing, logging, metrics, and error reporting. This is the single place where the observability contract is enforced -- no leaf command can accidentally skip it.

Design principles

A few principles that guided these choices:

Zero cost when off. Logging and metrics are lazy-initialised. If you never enable syslog or configure StatsD, the modules aren't even loaded.
Instrument the framework, not the features. Leaf commands don't contain observability code. The base command wrapper and HTTP client handle everything. New commands get full instrumentation for free.
Correlate by default. The invocation ID requires no opt-in. Every request carries it. The server just has to log it.
Separate concerns by audience. Verbose mode is for the person at the terminal. Syslog is for the person investigating after the fact. Metrics are for the person watching trends. Don't conflate them.
Don't build what you can plug in. Full HTTP tracing via LWP::ConsoleLogger is better than anything we'd build ourselves. Keep verbose mode lean and let the specialist tool handle the rest.

Testing observability

Instrumentation code is easy to write and easy to break silently. If nobody notices that the invocation ID stopped appearing in syslog, it might be months before an incident reveals the gap. A few testing strategies:

Unit test the logger's formatting. The _emit method returns the formatted message even when syslog is disabled. Assert that the invocation ID, context, and detail appear in the expected format.
Unit test metric emissions. Mock the StatsD client and assert that command.<cmd>.calls is incremented, command.<cmd>.timing receives a value, and command.<cmd>.errors fires on exception. These are contract tests -- they verify that the execution wrapper keeps its promises.
Assert the invocation ID propagates. Mock the HTTP client and verify that outgoing requests carry the X-Invocation-Id header with the same value the logger is using.
Integration test the full lifecycle. Run a command against a mocked API, capture STDERR with --verbose, and assert the --> / <-- lines appear with the expected method, URL, and status.

The "always format, conditionally emit" pattern helps here: the logger exercises all formatting code paths in every test run, even when syslog isn't available in the test environment.

Tracing an incident: a walkthrough

Here's how the instrumentation plays out during a real debugging scenario. This walkthrough exercises every layer described above: error output with a request ID, the metrics dashboard, syslog correlation, and two-way ID join.

A user reports: "mycli device list is failing intermittently." They include the error message:

Error: 503 Service Unavailable
  The API is temporarily unable to handle the request.
  Request ID: e4f5a6b7

Step 1: Find the server side. The on-call engineer greps the API logs for e4f5a6b7 and finds the request hit a backend that was in the middle of a deployment. The 503 was a transient error from a rolling restart.

Step 2: Assess the blast radius. But is it just this one user? The engineer checks the CLI dashboard: mycli.http.status.503 shows a spike over the last 20 minutes, coinciding with the deployment window. It's not one user -- it's everyone hitting that backend.

Step 3: Find the CLI side. The server log for e4f5a6b7 also contains the X-Invocation-Id: c8d9e0f1. Grepping the centralised CLI logs for c8d9e0f1 shows the full client-side context: which command was run, which user ran it, what arguments were passed, and that the request took 12 seconds before returning 503 (suggesting the backend was hanging, not failing fast).

Step 4: Verify the fix. After the deployment completes, the 503 counter drops to zero. The engineer confirms on the dashboard that error rates are back to baseline across all commands.

Total debugging time: minutes. Without instrumentation, this would have been a ticket saying "it's broken sometimes" followed by back-and-forth to reproduce, enable verbose mode, and collect output.

Summary

 +-------------------+    X-Invocation-Id    +-------------------+
 | mycli             |-----------------------| API               |
 |                   |    X-Request-Id       |                   |
 | - syslog [id]     |<----------------------| - access log [id] |
 | - StatsD metrics  |                       | - request trace   |
 | - verbose STDERR  |                       |                   |
 +-------------------+                       +-------------------+
         |                                           |
         v                                           v
 +-------------------+                       +-------------------+
 | Centralised logs  |<--- grep by ID ------>| Centralised logs  |
 | Metrics dashboard |                       | APM / tracing     |
 +-------------------+                       +-------------------+

Key takeaways:

Your CLI is part of the distributed system. If it talks to an API, it's a participant in the request path -- treat it like a service, not a script.
A correlation ID is the single most valuable thing you can add. One random string, sent as an HTTP header, ties client logs to server logs. Everything else builds on this.
Separate layers by audience. Verbose mode for the developer at the terminal, structured logs for the on-call engineer after the fact, metrics for dashboards and alerting. Same data, different consumers, different lifetimes.
Instrument the framework, not the features. A single execution wrapper gives every command logging, metrics, and error reporting for free. Leaf commands shouldn't contain observability code.
The server needs to participate. Log the client's invocation ID, return your own request ID. Without this, correlation is one-sided.
Log everything except secrets and personal data. Mask credentials, never log request bodies, and keep logging always on -- the time you need debug detail is the time you can't reproduce the issue.
Start simple, keep the door open. Wrap your logging backend so the rest of the codebase never touches it directly. Start with whatever works for your deployment targets today -- Sys::Syslog, Fluent::Logger, a file. When your infrastructure is ready for OpenTelemetry or wide events (see Appendix A), the swap is localised.

The investment is small: a correlation ID, a handful of counters, and a logging lifecycle. The return is that your CLI becomes a first-class citizen in your operational tooling rather than a blind spot.

References

LWP::ConsoleLogger::Everywhere -- drop-in full HTTP tracing for any LWP-based client
NO_COLOR -- convention for suppressing colour output, relevant to verbose/debug output formatting
OpenTelemetry -- industry-standard observability framework; Perl SDK on CPAN
StatsD -- the metrics aggregation protocol used for CLI instrumentation
Shipping a Perl CLI as a single file with App::FatPacker -- companion post on building and distributing mycli

Getting started

If you want to add observability to an existing CLI tool, here's a practical order of operations. Each step is independently useful -- you don't need to do all five before any of them pay off.

Generate a random invocation ID at startup. Eight hex characters is enough. Send it as an X-Invocation-Id header on every HTTP request. This single change makes every future debugging session easier.
Set User-Agent to <tool>/<version>. Trivial, and it lets the server side filter by CLI version without any custom header support.
Log three lifecycle events. Startup (command line, environment, config source), each HTTP request/response (method, URL, status, timing), and shutdown (duration, result count). Even logging to STDERR behind a --debug flag is better than nothing.
Emit one counter per command invocation. If you have StatsD or a metrics collector, mycli.command.<cmd>.calls is the single most useful metric -- it tells you what people are actually using. If you don't have a metrics pipeline, a cheap alternative is to emit key=value pairs in your log lines (e.g. command=device_list duration_ms=387 status=ok) -- most log aggregation tools, including Grafana itself, can extract fields from these lines and build charts and dashboards without a separate metrics stack.
Wrap your command entry point. Move timing, logging, and metric emission into a single wrapper around leaf command execution. New commands get instrumentation for free, and no leaf command can accidentally skip it.

Appendix A: Wide events

Our implementation uses separate syslog lines for each lifecycle phase (startup, HTTP, shutdown) and separate StatsD counters for aggregation. This works, but it means correlating data across multiple log lines at query time -- you need the invocation ID to join them together.

An increasingly popular alternative is the wide event (or what Stripe called a canonical log line in 2019): a single, information-dense structured record emitted once per unit of work, containing every attribute you collected along the way. Instead of five syslog lines and ten StatsD counters, you emit one event with fields like command=device_list duration_ms=387 results=24 http_requests=2 http_status=200 auth_source=file output_format=table cache_hits=3.

The advantages are significant:

Faster queries -- all the data is colocated in one record. No joins, no correlation by ID.
Ad hoc analysis -- during an incident you can group by any combination of fields without having pre-defined a metric for it.
Simpler pipeline -- one event replaces multiple log lines and multiple metric emissions. Less code, fewer failure modes.

We didn't take this approach because our logging infrastructure is syslog-based and doesn't support high-cardinality structured queries. If you have access to a columnar store (Honeycomb, ClickHouse, a data warehouse), wide events are the stronger choice. The execution wrapper already collects all the data in one place -- the change would be emitting it as a single structured record instead of spreading it across syslog and StatsD.

For more on wide events, see A Practitioner's Guide to Wide Events and All You Need Is Wide Events, Not Metrics.

Appendix B: Why Sys::Syslog and not a logging framework?

Perl has several mature logging frameworks -- Log::Any, Log::Dispatch, Log::Log4perl -- any of which would be a fine choice here. We went with Sys::Syslog directly. This is an opinionated trade-off worth explaining.

What Sys::Syslog gives you

Syslog is a solved problem on servers and jumpboxes. The local syslog daemon (rsyslog, syslog-ng, journald) handles buffering, rotation, compression, and forwarding to a central log aggregator. The CLI doesn't need to know where the logs go, how to authenticate to a remote endpoint, or what to do when the network is down. It calls syslog(), the daemon takes it from there. This is a clean separation of concerns: the application produces structured messages, the infrastructure handles transmission.

There are no extra dependencies beyond core Perl. No configuration files, no adapter registration, no output plugin selection. The logger module is ~50 lines. For a fatpacked binary where every dependency has a cost, this matters.

What a framework would give you

A framework like Log::Any or Log::Dispatch provides output abstraction: you write $log->info(...) and configure the destination at deployment time -- syslog, a file, STDERR, a network endpoint, or multiple at once. The application code doesn't change when the destination does. This is a genuine advantage when the tool runs in environments with different logging infrastructure, or when libraries you depend on already use Log::Any.

Where the trade-off bites

The opinionated choice of Sys::Syslog works well when every target machine runs a syslog daemon. It falls apart on developer laptops and desktops.

macOS ships with a syslog-compatible interface via Apple System Log, but the log viewer has moved to Console.app and the unified logging system. Messages from syslog() end up in a different place than most macOS users expect, and the retention policy may discard them quickly. On Windows, there is no syslog daemon at all.

You have two choices here:

Accept the gap. Detect the platform at startup and disable syslog on macOS and Windows. The CLI still has --verbose for interactive debugging, and StatsD metrics still flow if a collector is configured. You lose durable logging on developer machines, but you avoid adding complexity to the CLI itself. This is the approach we took -- the primary deployment targets are Linux servers and jumpboxes where syslog is reliable.

Solve logging everywhere. Use a framework like Log::Dispatch with pluggable outputs: syslog on Linux, a file on macOS, a network endpoint everywhere. This means the CLI now owns the full logging pipeline: transport selection, buffering when the destination is unavailable, possibly TLS for log data in transit, possibly client-side authentication to a log aggregator. Each of these is individually tractable, but collectively they add configuration surface, failure modes, and dependencies that the syslog approach avoids entirely.

There is a middle ground: In an organization with tight control of staff laptops and desktops (as is increasingly common), solving the logging problems in the CLI or having a local logging daemon is very feasible.

Another opinionated choice: Fluent::Logger

If your infrastructure runs Fluentd or Fluent Bit, Fluent::Logger is worth considering as an equally opinionated alternative to Sys::Syslog. It sends structured events directly to a Fluent collector over a local socket or TCP, which then handles routing, buffering, and delivery to whatever backend you use (Elasticsearch, S3, a data warehouse). Like Sys::Syslog, it delegates transport to purpose-built infrastructure. Unlike syslog, the events are natively structured -- key-value pairs rather than format strings -- which makes the path to wide events shorter.

The advantage of making an opinionated backend choice -- whether that's Sys::Syslog, Fluent::Logger, or something else entirely -- is that it removes abstraction layers that aren't adding value. If you know where your logs go, a framework like Log::Any is indirection without a benefit. You pay for adapter registration, output plugin configuration, and an extra dependency, but you only ever use one backend. An abstraction earns its keep when requirements are genuinely uncertain; when they're known, it's just ceremony.

The elephant in the room: OpenTelemetry

Of course, the industry is converging on OpenTelemetry as the standard answer to all of the above. Perl has solid support via the OpenTelemetry distribution on CPAN. If your organisation already runs an OTel collector, plumbing it into your CLI from the start is the right long-term bet.

Keeping the door open

The important thing is that the rest of the codebase never touches Sys::Syslog directly. Every module calls $self->logger->info(...), ->error(...), or ->debug(...). The actual syslog calls are isolated to two private methods in the logger class: _emit (which formats and transmits) and _open_syslog (which calls openlog). Swapping Sys::Syslog for Log::Dispatch, Fluent::Logger, or an OpenTelemetry log bridge would mean changing those two methods and nothing else.

This is the pragmatic middle path: start with the simplest backend that works for your deployment targets, but wrap it so the choice is easy to revisit. For a server-side CLI deployed to a controlled fleet, Sys::Syslog is a sensible default -- zero-config, zero-dependency, and delegates the hard problems to purpose-built infrastructure. If the tool later needs to run on developer laptops as a primary deployment target, the logging framework swap is a localised change rather than a rewrite.

Discussion

Have you plumbed observability into a CLI tool? I'd love to hear what worked and what didn't -- whether you went with OpenTelemetry traces, wide events from day one, or bolted logging on after the fact. What was the moment that made you invest in CLI instrumentation? Was it an incident that was hard to trace, a question about adoption you couldn't answer, or just good hygiene? And if you haven't done it yet -- what's holding you back?

Shipping a Perl CLI as a single file with App::FatPacker

Dean Hamstead — Tue, 31 Mar 2026 12:41:49 +0000

Modern software distribution has converged on a simple idea: ship a self-contained artifact. Whether that means a statically linked binary, a container image, or a snap/flatpak, the benefits are the same -- dependency management is solved at build time, platform differences are absorbed, and upgrades and rollbacks reduce to swapping a single file.

Perl's App::FatPacker applies the same principle to Perl scripts. It bundles every pure-Perl dependency into a single executable file. No cpanm, no local::lib, no Makefile on the target -- just copy the file and run it. The technique is well-established -- cpm (the CPAN installer we use in the build) is itself distributed as a fatpacked binary.

The distribution pipeline looks like this:

 Code repo --> CI --> fatpack --> deploy --> laptops / jumpboxes / servers
                       |
                 single file,
                 no dependencies

This post walks through how we fatpacked an internal CLI we'll call mycli, a ~90-module Perl app, into a single file. The approach generalises to any App::Cmd-based tool.

A good practice for internal tools is to provide all three interfaces: a web frontend, an API, and a CLI. The web frontend is the easiest to discover; the API enables automation and integration; the CLI is the fastest path for engineers who live in a terminal. FatPacker makes the CLI trivially deployable.

mycli is a thin client -- it talks to an internal REST API over HTTPS and renders the response locally. There is no local state beyond a config file and environment variables. You could build an equivalent tool against a binary RPC protocol such as gRPC or Thrift -- the fatpacking approach is the same.

 +--------------------+           +-------------------+
 | Workstation        |   HTTPS   | Server            |
 |                    |           |                   |
 |  $ mycli resource  |---------->|  REST API ---+    |
 |    list ...        |<----------|  (JSON)  DB  |    |
 +--------------------+           +-------------------+

Despite being a thin client, mycli is not trivial. It includes:

Pluggable output renderers (table, JSON, YAML, CSV, plain text)
Colour output with NO_COLOR support
Automatic pager integration (less -RFX) and pipe/TTY detection
Activity spinner
Multi-format ID resolution (numeric, UUID prefix, name lookup)
Command aliases (ls/list, get/show)
Config file discovery chain (env var, XDG path, dotfile)
Timezone-aware timestamp rendering
Structured syslog logging with per-invocation correlation IDs
StatsD metrics instrumentation
HTTP debugging hooks

All of this fatpacks cleanly because each feature is backed by pure-Perl modules.

This makes it an ideal fatpack candidate: the only XS dependency is Net::SSLeay for TLS, which is typically already present on the target system. Everything else is pure Perl.

Why FatPacker over PAR::Packer?

The other well-known option for single-file Perl distribution is PAR::Packer. PAR bundles everything -- including XS modules and even the perl interpreter itself -- into a self-extracting archive. At runtime it unpacks to a temp directory and executes from there.

FatPacker takes a different approach: modules are inlined as strings inside the script and served via a custom @INC hook. There is no extraction step, no temp directory, and no architecture coupling. The trade-off is that FatPacker only handles pure Perl -- XS modules must already be on the target.

For a thin REST client where the only XS dependency is Net::SSLeay, FatPacker wins on simplicity: the output is a plain Perl script, it starts instantly, and it runs on any architecture with a compatible perl. PAR is the better choice when you need to bundle XS-heavy dependencies or ship a binary to machines without Perl at all.

What fatpacking does

FatPacker prepends a BEGIN block to your script containing every dependency as a string literal, keyed by module path. A custom @INC hook serves these strings to require instead of reading from disk. The original script is appended unchanged.

$ wc -l bin/mycli mycli-packed
      13 bin/mycli
   48721 mycli-packed

That ~49k line file runs identically to the original, on any machine with Perl 5.24+.

The problem with naive fatpacking

The standard FatPacker workflow is:

fatpack trace bin/mycli
fatpack packlists-for $(cat fatpacker.trace) > packlists
fatpack tree $(cat packlists)
fatpack file bin/mycli > mycli-packed

This breaks for non-trivial apps because fatpack trace uses compile-time analysis (B::minus_c). It misses anything loaded at runtime via require:

App::Cmd discovers commands via Module::Pluggable at runtime
Text::ANSITable loads border styles and colour themes dynamically
LWP::UserAgent loads protocol handlers on first request
YAML::Any probes for available backends at runtime

If the trace misses a module, the packed binary dies with Can't locate Foo/Bar.pm in @INC at the worst possible moment.

The solution: a custom trace helper

Instead of relying on fatpack trace, we wrote a helper script that requires every module the app could ever load, then dumps %INC at exit. This captures the complete runtime dependency tree.

#!/usr/bin/env perl
# bin/trace-helper -- not shipped, build-time only
use strict;
use warnings;
use lib 'lib';

# Modules loaded lazily that fatpack misses
require Data::Unixish::Apply;
require Digest::SHA;
require HTTP::Request;
require LWP::UserAgent;
require String::RewritePrefix;

# Exercise objects to trigger deep runtime loads
{
    require Text::ANSITable;
    my $t = Text::ANSITable->new(use_color => 1, use_utf8 => 1);
    $t->border_style('UTF8::SingleLineBold');
    $t->color_theme('Text::ANSITable::Standard::NoGradation');
    $t->columns(['a']);
    $t->add_row(['1']);
    $t->draw;  # forces all rendering deps to load
}

# Every App::Cmd leaf command
require MyCLI::App;
require MyCLI::App::Command::device::list;
require MyCLI::App::Command::device::get;
# ... all 80+ command modules ...

END {
    open my $fh, '>', 'fatpacker.trace' or die $!;
    for my $inc (sort keys %INC) {
        next unless defined $INC{$inc};
        next if $inc =~ m{\AMyCLI/};  # our own modules come from lib/
        print $fh "$inc\n";
    }
}

Key points:

Don't call ->run -- App::Cmd subdispatch will die on duplicate command names across namespaces. Just require every leaf.
Exercise both code paths -- Text::ANSITable loads different modules for colour vs plain, UTF-8 vs ASCII. Instantiate both.
Exclude your own namespace -- FatPacker embeds modules from fatlib/; your lib/ modules are embedded separately. Including them in the trace causes duplicates.

Forcing pure-Perl backends

FatPacker can only bundle pure Perl. Many popular modules ship dual XS/pure-Perl backends and prefer XS at runtime. If XS is available during the trace, the pure-Perl fallback won't appear in %INC and won't get bundled.

Force pure-Perl mode during the build:

# In the fatpack build script
export B_HOOKS_ENDOFSCOPE_IMPLEMENTATION=PP
export LIST_MOREUTILS_PP=1
export MOO_XS_DISABLE=1
export PACKAGE_STASH_IMPLEMENTATION=PP
export PARAMS_VALIDATE_IMPLEMENTATION=PP
export PERL_JSON_BACKEND=JSON::PP
export PUREPERL_ONLY=1

PUREPERL_ONLY=1 is a convention respected by many dual XS/PP distributions at install time, preventing XS compilation entirely. The per-module variables above cover modules that don't check PUREPERL_ONLY.

Combine this with --pp at install time to avoid pulling in XS at all:

cpm install -L local --target-perl 5.24.0 --pp

Pinning the target Perl version

The --target-perl flag to cpm is critical and easy to overlook. Without it, cpm resolves dependency versions against your build machine's Perl. If you're building on 5.38 but deploying to a jumpbox running 5.24, you'll silently install module versions that use postfix dereferencing, subroutine signatures, or other features that don't exist on the target.
The packed binary will fail at runtime with a syntax error -- far from the build where you could catch it.

This tells cpm's resolver to only consider module versions whose metadata declares compatibility with 5.24.0. Combined with perl -c as a post-install sanity check, this catches version mismatches before the slow trace step.

The complete build script

Here is the full pipeline, wrapped in a shell script. It supports incremental builds (reuses local/ and trace cache) and --clean for full rebuilds.

#!/bin/sh
set -e

CLEAN=0
[ "$1" = "--clean" ] && CLEAN=1

# 0. Prerequisites
for cmd in cpm fatpack perl; do
    command -v "$cmd" >/dev/null 2>&1 || {
        echo "Error: '$cmd' is not installed." >&2; exit 1
    }
done

export PERL_USE_UNSAFE_INC=1  # Perl 5.26+ removed . from @INC

# 1. Install deps (pure-perl only)
if [ "$CLEAN" = 1 ] || [ ! -d local/ ]; then
    rm -rf local/
    cpm install -L local --target-perl 5.24.0 --pp
fi

# 2. Set up paths
export PERL5LIB=$PWD/lib:$PWD/local/lib/perl5
export PATH=$PWD/local/bin:$PATH

# 3. Force pure-perl backends
export B_HOOKS_ENDOFSCOPE_IMPLEMENTATION=PP
export LIST_MOREUTILS_PP=1
export MOO_XS_DISABLE=1
export PACKAGE_STASH_IMPLEMENTATION=PP
export PARAMS_VALIDATE_IMPLEMENTATION=PP
export PERL_JSON_BACKEND=JSON::PP
export PUREPERL_ONLY=1

# 4. Verify compilation
perl -c bin/mycli || exit 1

# 5. Trace
if [ "$CLEAN" = 1 ] || [ ! -f fatpacker.trace ]; then
    perl -Ilib bin/trace-helper
    echo "Trace: $(wc -l < fatpacker.trace) modules"
fi

# 6. Pack
fatpack packlists-for $(cat fatpacker.trace) > packlists
fatpack tree $(cat packlists)

# Strip arch-specific dirs and non-essential files
rm -rf fatlib/$(perl -MConfig -e 'print $Config{archname}')
find fatlib -name '*.pod' -delete
find fatlib -name '*.pl'  -delete

# Bundle
fatpack file bin/mycli > mycli-packed
chmod +x mycli-packed
echo "Built mycli-packed ($(wc -c < mycli-packed) bytes)"

Step by step: what happens

Prerequisites -- verify cpm, fatpack, and perl are available
Install -- cpm installs all dependencies into local/ as pure Perl, targeting 5.24.0
Paths and env -- set PERL5LIB, PATH, and pure-Perl overrides
Compile check -- perl -c bin/mycli catches syntax errors before the slow trace step
Trace -- the helper script loads everything and writes the module list to fatpacker.trace
Packlists and tree -- fatpack packlists-for maps module names to installed packlist files; fatpack tree copies the .pm files into fatlib/
Clean up -- remove .pod, .pl, and arch-specific directories to reduce size
Bundle -- fatpack file inlines everything from fatlib/ into the script

Makefile integration

For teams that prefer make, add targets that delegate to the shell script:

# In Makefile.PL, inside MY::postamble
.PHONY: pack clean_fatpack

pack:
    ./fatpack

clean :: clean_fatpack

clean_fatpack:
    rm -rf fatlib fatpacker.trace packlists mycli-packed local/

Then building is just:

perl Makefile.PL
make pack

Adding a new dependency

When someone adds use Some::New::Module to the codebase, the fatpacked binary will break with Can't locate Some/New/Module.pm in @INC unless the build picks it up. The workflow is:

Add the module to cpanfile
If the module is loaded at runtime (via require or a plugin mechanism), add a require Some::New::Module line to the trace helper
Rebuild with --clean

./fatpack --clean

The --clean flag is important. Without it, the build reuses the cached local/ directory and fatpacker.trace from the previous run. The new module won't appear in either, and the packed binary will silently ship without it.

A good safeguard is to run perl -c mycli-packed after every build -- this catches missing modules at build time rather than in production.

What about perlstrip?

Perl::Strip can reduce the packed file by ~30% by removing comments, POD, and whitespace from bundled modules. We deliberately left it off. For an internal tool, the size saving (~1.7 MB) is not worth the trade-off: stripped files are harder to debug with stack traces, and perlstrip has a known issue corrupting files that contain use utf8.

Gotchas and tips

XS modules cannot be fatpacked

Modules with C extensions (.so/.xs) cannot be inlined. They must already exist on the target system. If your app has many XS dependencies, consider PAR::Packer instead (see above).

PERL_USE_UNSAFE_INC

Perl 5.26 removed . from @INC. Some older CPAN modules assume it's there during install or test. Set PERL_USE_UNSAFE_INC=1 during the build to avoid spurious failures. This only affects the build environment, not the packed binary.

Pinto / private CPAN

If your organisation runs a private CPAN mirror (Pinto, OrePAN2, etc.), point cpm at it with --resolver:

cpm install -L local --resolver 02packages,$PINTO_REPO --pp

Docker builds

FatPacker and Docker are complementary. Use Docker for the build environment (consistent Perl version, cpm, fatpack installed), and ship either the container image or just the packed file:

COPY mycli-packed /usr/local/bin/mycli
RUN chmod +x /usr/local/bin/mycli

Summary

The core recipe is three pieces:

A trace helper that loads every module your app could use at runtime, capturing the full dependency tree via %INC
Pure-Perl enforcement via environment variables and cpm --pp
The standard fatpack pipeline: packlists, tree, clean up, bundle

The result is a single file you can scp to any box with Perl 5.24+ and run immediately. No CPAN, no Makefile, no containers required.

References

App::FatPacker on CPAN
FatPacking Perl applications -- talk by Andrew Rodland covering the core technique, pure-Perl enforcement, and cpm
arodland/swr fatpack script -- a clean, minimal reference implementation of the full pipeline
App::cpm -- fast CPAN installer (itself shipped as a fatpacked binary); --target-perl and --pp flags are essential for fatpack builds

Making Git tolerable

Dean Hamstead — Wed, 11 Mar 2026 09:59:41 +0000

Once you have seen better than git, you can never come back. But you can make it slightly more tolerable.

Extensions

These two extensions will substantially increase your velocity, help you flow, and keep you in the zone.

git-branchless

When forced to use Git I naturally use a stacked commit workflow to move much faster, cutting out the overhead of constant feature branching to keep my development process lean. Instead of wrestling with a web of branches, git-branchless enables a "stacked" approach where each atomic change is a single commit built directly on top of the last.

The tool acts as a high-powered coordinator for this linear progression; it provides a smartlog to visualize your local commit graph and navigation commands like next and prev to jump through your stack without manual checkouts. Most importantly, it handles the "restack" automatically—if you amend a commit in the middle of your stack, git-branchless instantly rebases all descendant commits to keep the entire chain in sync. If you’re curious about how to implement this without the usual Git friction, Ben Congdon’s article is a fantastic introduction to the philosophy and the tools that make it work.

See also the original epriestly workflow

git-autofixup

To move even faster and keep my history clean, I use git-autofixup to assist in organizing stacked commits. Instead of manually hunting for SHAs to fix up small edits or review feedback, this tool uses git blame to automatically assign unstaged changes to the correct commit in your stack. It essentially turns the tedious process of "tidying up" into a mechanical one, generating the necessary fixup! commits so that a simple git rebase --autosquash can perfectly integrate them. If you’re looking to streamline your workflow and avoid the friction of manual rebasing, Jordan Torbiak’s post on git-autofixup is a great intro to the tool.

My Opinionated .gitconfig, Explained

I've refined my ~/.gitconfig into something that is faster, safer, and produces cleaner output. Here's the whole thing, broken down section by section.

The Full Config

[advice]
    defaultBranchName = false
[blame]
    coloring = highlightRecent
[alias]
    checkout-remote = "!f() { git checkout -b \"$1\" \"origin/$1\"; }; f"
    ci = commit
    co = checkout
    lg = log --graph --oneline --decorate --all
    ls = show --pretty= --name-only HEAD
    patch = !git --no-pager diff --no-color
    resync = "!f() { r=${1:-origin}; git fetch \"$r\" && git reset --hard \"$r/$(git branch --show-current)\"; }; f"
    st = status
    stashed = stash list --pretty=format:'%gd: %Cred%h%Creset %Cgreen[%ar]%Creset %s'
    wip = commit -am 'WIP'
[branch]
    autosetuprebase = always
    sort = -committerdate
[color]
    ui = auto
[commit]
    cleanup = scissors
    verbose = true
[core]
    compression = 9
    fsmonitor = true
    pager = diff-so-fancy | less --tabs=4 -RFX
    preloadindex = true
    untrackedCache = true
[diff]
    algorithm = histogram
    colorMoved = dimmed_zebra
    colorMovedWS = allow-indentation-change
    context = 10
    mnemonicPrefix = true
    noprefix = true
    renames = copy
[fetch]
    fsckobjects = true
    parallel = 0
    prune = true
    pruneTags = true
[gc]
    writeCommitGraph = true
[grep]
    patternType = perl
[init]
    defaultBranch = master
[interactive]
    diffFilter = diff-so-fancy --patch
[log]
    date = iso
[merge]
    conflictStyle = zdiff3
    ff = only
    keepbackup = false
[pack]
    threads = 0
[pager]
    diff = diff-so-fancy | less --tabs=1,5 -RFX
    log = diff-so-fancy | less --tabs=1,5 -RFX
    show = diff-so-fancy | less --tabs=1,5 -RFX
[pull]
    ff = only
    rebase = true
[push]
    autoSetupRemote = true
    default = simple
    followtags = true
[rebase]
    autoStash = true
    autosquash = true
    missingCommitsCheck = error
    updateRefs = true
[receive]
    fsckObjects = true
[rerere]
    autoupdate = true
    enabled = true
[tag]
    sort = version:refname
[transfer]
    fsckobjects = true
[user]
    email = my@email
    name = Me

Now let's walk through it.

Silencing Noise

[advice]
    defaultBranchName = false

Every time you git init, Git helpfully suggests configuring a default branch name. Once you've set one, you don't need the reminder. This turns it off.

Better Blame

[blame]
    coloring = highlightRecent

Recent changes glow brighter, older ones fade into the background. When you git blame a file, your eye is immediately drawn to what changed recently — which is usually what you're investigating.

Aliases That Earn Their Keep

[alias]
    checkout-remote = "!f() { git checkout -b \"$1\" \"origin/$1\"; }; f"
    ci = commit
    co = checkout
    lg = log --graph --oneline --decorate --all
    ls = show --pretty= --name-only HEAD
    patch = !git --no-pager diff --no-color
    resync = "!f() { r=${1:-origin}; git fetch \"$r\" && git reset --hard \"$r/$(git branch --show-current)\"; }; f"
    st = status
    stashed = stash list --pretty=format:'%gd: %Cred%h%Creset %Cgreen[%ar]%Creset %s'
    wip = commit -am 'WIP'

The classics — ci, co, st — need no explanation. The interesting ones:

lg gives you a compact, visual branch graph of your entire repo. It's the first thing I run when switching context to a project.
checkout-remote creates a local tracking branch in one command: git checkout-remote feature-x.
resync is the nuclear option — it fetches and hard-resets your branch to match the remote. This is what you want with stacked diffs and all force pushes.
stashed lists your stashes with color-coded hashes, relative timestamps, and messages. Far more readable than the default.
wip is a quick checkpoint. Stage everything, commit with "WIP", keep moving. I autosquash these away later, cherry-pick or amend them.
patch outputs a clean diff with no pager or color, ready for piping to a file or another tool.

Branch Behaviour

[branch]
    autosetuprebase = always
    sort = -committerdate

Every new tracking branch is automatically configured for rebase on pull. Branch listings are sorted by most recently committed, not alphabetically — because "which branch did I touch last?" is almost always the question.

Commits

[commit]
    cleanup = scissors
    verbose = true

verbose = true shows the full diff in your commit message editor. You can review exactly what you're committing while writing the message.

cleanup = scissors is the real gem. Instead of stripping all lines starting with # (the default), Git uses a scissors marker (-- >8 --) to separate your message from the help text. This means you can freely use # in commit messages — Markdown headings, #1234 issue references at the start of a line — without Git eating them.

Core Performance

[core]
    compression = 9
    fsmonitor = true
    pager = diff-so-fancy | less --tabs=4 -RFX
    preloadindex = true
    untrackedCache = true

Maximum zlib compression for objects (smaller repos, slightly slower writes — worth it). The filesystem monitor daemon, preloaded index, and untracked file cache all make git status noticeably faster on large repos.

diff-so-fancy as the pager transforms Git's output from "functional" to "beautiful."

Diff Configuration

[diff]
    algorithm = histogram
    colorMoved = dimmed_zebra
    colorMovedWS = allow-indentation-change
    context = 10
    mnemonicPrefix = true
    noprefix = true
    renames = copy

This is where I've spent the most time tuning.

histogram produces cleaner, more readable diffs than the default Myers algorithm.
colorMoved = dimmed_zebra highlights lines that were moved (not added/removed) with alternating dimmed colors. Combined with colorMovedWS = allow-indentation-change, it still detects moved code even when you've re-indented it (e.g., wrapping in a new block). These two together make refactoring diffs dramatically easier to review.
context = 10 shows 10 lines of surrounding context instead of the default 3. More context means fewer trips back to the source while reviewing.
noprefix removes the a/ and b/ path prefixes so file paths are directly copy-pasteable. mnemonicPrefix replaces them with meaningful labels (i/ for index, w/ for working tree) when prefixes are shown.
renames = copy detects both file renames and copies.

Fetch: Integrity and Cleanliness

[fetch]
    fsckobjects = true
    parallel = 0
    prune = true
    pruneTags = true

Every fetched object is validated for integrity. Stale remote-tracking branches and tags are automatically cleaned up. Fetches from multiple remotes happen in parallel (0 = use all CPUs).

Keeping Things Fast

[gc]
    writeCommitGraph = true
[pack]
    threads = 0

The commit-graph file dramatically speeds up any operation that walks history — git log, git blame, git merge-base. Repacking uses all available CPU cores.

Grep and Interactive

[grep]
    patternType = perl
[interactive]
    diffFilter = diff-so-fancy --patch

Perl-compatible regexes for git grep — because life's too short for basic regex. Interactive operations like git add -p also get the diff-so-fancy treatment.

Merge Strategy

[merge]
    conflictStyle = zdiff3
    ff = only
    keepbackup = false

zdiff3 is the best conflict style available. It shows the common ancestor alongside both sides of the conflict and cleans up redundant lines that diff3 leaves behind. It makes conflicts significantly easier to resolve.

Fast-forward only means no surprise merge commits. If the merge can't fast-forward, I want to know about it and make an explicit decision. No .orig backup files cluttering the tree after conflict resolution.

Pull and Push

[pull]
    ff = only
    rebase = true
[push]
    autoSetupRemote = true
    default = simple
    followtags = true

Pulls rebase by default, and only fast-forward. Pushes automatically set up remote tracking (-u is no longer needed), only push the current branch, and automatically include relevant annotated tags.

Rebase: The Heart of the Workflow

[rebase]
    autoStash = true
    autosquash = true
    missingCommitsCheck = error
    updateRefs = true

This is a rebase-centric workflow, and these settings make it seamless:

autoStash stashes uncommitted changes before rebase and restores them after. No more "cannot rebase: you have unstaged changes."
autosquash automatically processes fixup! and squash! commits during interactive rebase. Combined with the wip alias, this is how I keep a clean history.
missingCommitsCheck = error is a safety net — if you accidentally delete a line during interactive rebase (dropping a commit), Git aborts instead of silently losing work.
updateRefs automatically updates stacked branch pointers during rebase. Essential if you work with stacked PRs.

Object Integrity Everywhere

[receive]
    fsckObjects = true
[transfer]
    fsckobjects = true

Combined with fetch.fsckobjects, this validates every object during every transfer — fetch, push, clone. Belt and suspenders for repository integrity.

Rerere: Reuse Recorded Resolution

[rerere]
    autoupdate = true
    enabled = true

One of Git's most underrated features. When you resolve a merge conflict, Git remembers the resolution. The next time the same conflict appears (common during repeated rebases), Git applies the same fix automatically and stages it. Once you enable this, you wonder how you ever lived without it.

Tags and Logs

[tag]
    sort = version:refname
[log]
    date = iso
[init]
    defaultBranch = master

Tags sort by semantic version (v2.10 comes after v2.9, not before v2.2). Dates display in ISO 8601 format. New repos default to master.

Pretty Pager Output

[pager]
    diff = diff-so-fancy | less --tabs=1,5 -RFX
    log = diff-so-fancy | less --tabs=1,5 -RFX
    show = diff-so-fancy | less --tabs=1,5 -RFX

diff-so-fancy for diff, log, and show with specific tab settings. Combined with the [color] and [interactive] settings, every piece of Git output that involves diffs looks great.

Final Thoughts

A good .gitconfig is a force multiplier. Most of these settings cost nothing in terms of workflow changes — they just make the defaults smarter. The ones that do change behaviour (rebase-by-default, fast-forward-only, scissors cleanup) are deliberate choices that enforce a cleaner Git history.

Start with the defaults you have, add what makes sense, and revisit every few months. Git keeps shipping useful features that most people never discover.

Honorable Mention: Game of Trees

Check out Game of Trees - the git clone with and ISC license with a well conceived cli - which might be what the doctor orders for many people.

One Diff, One Thesis

This is the key to velocity.

VSCode as a Perl IDE

Dean Hamstead — Sat, 05 Jun 2021 20:23:11 +0000

Overview

The VSCode IDE implements it's language support via the Language Server Protocol which is designed to cleanly separate language support from the editor. For Perl a fully functional language server is implemented in Perl::LanguageServer and will work with any LSP capable editor.

VSCode can edit both locally or in a remote environment. This brief guide will get you started editing locally (i.e. on your desktop or laptop), however the Perl VSCode extension and Perl's LanguageServer will work in either scenario.

Install a Perl interpreter for development

Although a Perl interpreter is likely already installed (i.e. system perl) on Linux , BSD, or MacOS the best practice is to leave if for the system to use and install your own interpreter for your development work. On Windows you likely will install Strawberry Perl.

Please follow this guide to installing a perl interpreter for your development.

Install Perl's LanguageServer

Assuming cpanm is installed, simply run:

cpanm Perl::LanguageServer

That's it. The Perl LanguageServer is now ready.

Install VSCode (actually VSCodium)

VSCodium is the fully-foss version of VSCode (think Chromium). I recommend you install VSCodium rather than VSCode.

Fedora, SuSE, and the Debian family of Linux distributions can install from this maintained repo.

For other OS including MacOS and Windows simply pick your favourite approach from the official install instructions

Install the Perl extension for VSCode

Run VSCode (VSCodium) and using the familiar "File -> Open" interface, open a Perl source file. Observe that whilst syntax highlighting is available already via the built-in "Perl Language Basics" plugin, actual IDE functions do not - i.e. try "Run -> Start Debugging".

We need to configure VSCode to use our Perl LanguageServer by installing the Perl extension. The good news is that the extension is in the official VS Marketplace making it very quick and easy.

In actual VSCode the quickest route is hit Ctrl-P then type:

ext install richterger.perl

You could also click "File -> Preferences -> Extensions" then search for "perl" and click install.

In VSCodium the Open VSX registry is used which doesn't yet include the perl extensions.

From the VS Marketplace plugin page download the extension with the "Download Extension" link on the right hand side.

On your local system's command line (there is no need to close VSCodium), change directory to your downloads and run:

codium --install-extension richterger.perl-*.vsix

Returning to VSCode or VSCodium, observe that the Perl extension is now installed in the extensions list ("File -> Preferences -> Extensions" or Ctrl-Shift-X)

Open a Perl source file, click "Run -> Start Debugging" or hit F5 and observe there is no error as before.

Now explore all VSCocde IDE functions working nicely with Perl!