DEV Community: Mecanik1337

COBOL Database Migration Tool - VSAM, DB2, CICS, IMS to 5 Modern SQL Databases With DAL Code in 6 Languages

Mecanik1337 — Sun, 29 Mar 2026 06:15:34 +0000

Good morning.

This is my next addition to the COBOL world.

A few weeks ago I posted about Easy COBOL Migrator, a desktop transpiler that converts COBOL to 6 modern languages. The most common question was: "What about the data layer?" Fair question. That's the part where migration projects actually stall and budgets explode. So I built the companion tool.

Easy COBOL DB Migrator analyzes COBOL data definitions and generates modern database schemas, data access layer code, and migration scripts. Desktop app, runs offline, no cloud, no AI.

What it parses

The tool detects 10 source system types automatically:

Flat files (sequential and line-sequential) from FD record layouts. All 5 VSAM types: KSDS (indexed with RECORD KEY), ESDS (entry-sequenced), RRDS (relative with RELATIVE KEY), VRRDS (variable-length relative), LDS (linear). DB2 embedded SQL with EXEC SQL blocks, host variable resolution against WORKING-STORAGE, cursor detection, DECLARE CURSOR parsing. CICS file operations: READ, WRITE, REWRITE, DELETE, STARTBR, READNEXT mapped to database equivalents. IMS/DL-I with CALL 'CBLTDLI' and EXEC DLI, function codes (GU, GN, GNP, ISRT, REPL, DLET), PCB mask structures and qualified SSAs.

What it generates

SQL DDL for PostgreSQL, MySQL, SQL Server, Oracle and SQLite with correct dialect. Each database gets proper auto-increment syntax (SERIAL vs AUTO_INCREMENT vs IDENTITY vs sequence), identifier quoting, type mappings and constraint definitions.

Data access layer code in Java, C#, Python, C++, Rust and Go. Each language gets entity/model classes and repository/DAO classes with full CRUD (create, findById, findAll, update, delete). Java gets BigDecimal and JDBC PreparedStatement. C# gets decimal and ADO.NET. Python gets dataclass with Decimal and DB-API 2.0. C++ gets struct with std::optional and ODBC. Rust gets struct with derive macros and sqlx patterns. Go gets struct with db tags and database/sql.

Migration/ETL scripts with database-specific bulk load commands: COPY for PostgreSQL, LOAD DATA for MySQL, BULK INSERT for SQL Server, SQL*Loader for Oracle, .import for SQLite. Plus validation queries for row counts and constraint verification.

HTML migration report with field-level type mapping rationale, migration issues by severity, and access pattern documentation.

CSV mapping exports and interactive ER diagrams exportable as PNG/SVG.

The type mapping engine

Every PIC/USAGE combination maps correctly per database:

PIC S9(7)V99 COMP-3 becomes NUMERIC(9,2) in PostgreSQL, DECIMAL(9,2) in MySQL/SQL Server, NUMBER(9,2) in Oracle.

PIC X(30) becomes VARCHAR(30). Short alphanumerics (10 chars or less) use CHAR to preserve COBOL fixed-width semantics. Longer fields use VARCHAR.

Level 88 conditions become CHECK constraints or BOOLEAN columns. Two values ("Y"/"N") generate BOOLEAN. Multiple enumerated values generate CHECK (or ENUM in MySQL).

OCCURS arrays normalize to child tables with foreign keys and a seq_num column. FILLER fields are skipped. REDEFINES uses the primary definition and documents the alternative.

IMS hierarchy flattening

IMS segments become relational tables with foreign keys:

IMS:                          SQL:
POLICY (root)            ->   policy (PK: policy_key)
  +-- CLAIMANT           ->   claimant (FK -> policy)
  |     +-- CLAIM        ->   claim (FK -> claimant)
  |           +-- PAYMENT ->  payment (FK -> claim)
  +-- COVERAGE           ->   coverage (FK -> policy)

DL/I function codes map to SQL: GU becomes SELECT WHERE, GN becomes cursor FETCH, ISRT becomes INSERT, REPL becomes UPDATE, DLET becomes DELETE.

Enterprise: actual data extraction

The Enterprise edition reads real COBOL data files and handles EBCDIC-to-ASCII conversion (CP037), COMP-3 packed decimal unpacking, binary field parsing, zoned decimal sign handling, and implied decimal insertion. Exports as batched SQL INSERT statements or CSV.

How both tools work together

Run Easy COBOL Migrator. Get converted code with TODO markers where EXEC SQL/CICS/DLI blocks were.
Run Easy COBOL DB Migrator on the same source. Get database schema, data access code, and migration scripts.
Replace the TODO markers with generated repository/DAO code.
Run validation queries to verify data migration.

Free demo: PostgreSQL + C++, up to 3 source files.

👉 Free demo download

Desktop app, Qt/C++, Windows/macOS/Linux. Everything offline. This is how I spend my free time apparently.

If you've got COBOL with VSAM files, DB2 queries or IMS databases, throw it at the demo. I want to know what breaks.

I Built a Desktop COBOL Migration Tool That Converts to 6 Modern Languages

Mecanik1337 — Mon, 16 Mar 2026 07:23:24 +0000

COBOL migration is the hottest topic in enterprise tech right now. Anthropic's Claude Code blog post knocked $40 billion off IBM's market cap in a single day. Banks are panicking about the "COBOL cliff" as 92% of COBOL developers approach retirement by 2030.

I spent the last six months building a COBOL migration tool that takes a different approach from the AI-powered solutions making headlines. Easy COBOL Migrator is a desktop app that transpiles COBOL to C++ 17, Java 17, C# 12, Python 3, Rust and Go. It uses a full compiler pipeline, not an LLM. Everything runs offline. Your source code never leaves your machine.

Why not use AI for COBOL migration?

For a 200-line program, pasting COBOL into ChatGPT works fine. For a 200,000-line codebase with 300 shared copybooks and COMP-3 packed decimal arithmetic, AI-based COBOL migration breaks down for three reasons.

The output is non-deterministic. Same COBOL in, different code out. Every time. Try reviewing or diffing that across a project with 500 files. A deterministic compiler gives you the same output for the same input, which means your migration results are auditable and reproducible.

Decimal precision gets destroyed. COBOL's PIC 9(7)V99 COMP-3 does exact fixed-point math using binary-coded decimal. LLMs routinely map this to double or float. That introduces rounding errors. A bank finding a 0.01 discrepancy in reconciliation doesn't file a bug report. They file a regulatory incident.

Copybook resolution doesn't exist. Real COBOL codebases use COPY ... REPLACING to share data definitions across hundreds of programs. The same copybook might be included 50 times with different substitutions. No LLM has a mechanism to resolve this across a whole project.

AI is great for understanding COBOL. It's not reliable for converting it at scale with deterministic results.

How the COBOL migration pipeline works

The architecture is a traditional multi-stage compiler:

COBOL Source
  > COPY Preprocessor (resolves copybooks, applies REPLACING)
  > Lexer (220+ keywords, fixed/free format auto-detection)
  > Parser (recursive descent, 36 statement types, full AST)
  > Semantic Analyzer (symbol tables, type checking, scope validation)
  > Code Generator (one per target language)
  > Generated Source + Migration Report

Each stage is a separate C++ class. The parser builds a complete AST with expression trees, condition nodes and hierarchical data structures. The semantic analyzer validates everything before code generation starts.

The hardest parts of building a COBOL migration tool

PIC clauses are their own language. PIC S9(7)V99 COMP-3 means signed, 7 integer digits, 2 decimal digits, stored as packed BCD. Mapping this correctly to BigDecimal in Java, decimal in C#, decimal.Decimal in Python, f64 in Rust, float64 in Go and a custom Decimal type in C++ took weeks of careful work. This is where most COBOL migration tools silently break.

PERFORM THRU is cursed. PERFORM PARA-A THRU PARA-C executes all paragraphs from A through C in source order, including whatever falls between them. It's implicit, position-dependent control flow with no equivalent in any modern language. I generate sequential paragraph calls, which is correct as long as paragraph order is preserved.

Level numbers create invisible nesting. COBOL doesn't use braces or indentation for data structures. 01 starts a record, then 05, 10, 15 underneath are children based purely on numeric hierarchy. Miss this relationship and your struct generation produces flat fields instead of nested structures.

EVALUATE TRUE is not a switch statement. Each WHEN clause is an independent boolean condition, not a value comparison. I spent a week generating switch statements before realizing the correct output is if/else if.

Fixed-format column rules are unforgiving. Columns 1-6 are sequence numbers. Column 7 is an indicator (* for comment, - for continuation, D for debug). Columns 8-72 are code. Columns 73-80 are identification that gets ignored. Accidentally reading column 73 as code corrupts your parse on every single line.

What the COBOL migration output looks like

This COBOL:

       IDENTIFICATION DIVISION.
       PROGRAM-ID. PAYROLL-CALC.

       DATA DIVISION.
       WORKING-STORAGE SECTION.

       01  WS-EMPLOYEE.
           05  WS-NAME            PIC X(25).
           05  WS-HOURS           PIC 9(3)V99.
           05  WS-RATE            PIC 9(3)V99.
           05  WS-GROSS-PAY       PIC 9(5)V99.
           05  WS-TAX             PIC 9(5)V99.
           05  WS-NET-PAY         PIC 9(5)V99.

       01  WS-TAX-RATE            PIC V99 VALUE 0.20.
       01  WS-OVERTIME-MULT       PIC 9V9 VALUE 1.5.
       01  WS-OVERTIME-THRESHOLD  PIC 9(3) VALUE 40.
       01  WS-OVERTIME-HOURS      PIC 9(3)V99 VALUE 0.
       01  WS-REGULAR-HOURS       PIC 9(3)V99 VALUE 0.
       01  WS-COUNTER             PIC 9(2) VALUE 0.

       01  WS-PAY-GRADE           PIC 9 VALUE 0.
           88  JUNIOR             VALUE 1.
           88  SENIOR             VALUE 2.
           88  MANAGER            VALUE 3.

       PROCEDURE DIVISION.
       MAIN-PROGRAM.
           MOVE "Alice Johnson" TO WS-NAME
           MOVE 45.00 TO WS-HOURS
           MOVE 32.50 TO WS-RATE
           MOVE 2 TO WS-PAY-GRADE

           PERFORM CALCULATE-PAY
           PERFORM DISPLAY-PAYSLIP
           PERFORM DISPLAY-GRADE

           STOP RUN.

       CALCULATE-PAY.
           IF WS-HOURS > WS-OVERTIME-THRESHOLD
               COMPUTE WS-REGULAR-HOURS =
                   WS-OVERTIME-THRESHOLD
               SUBTRACT WS-OVERTIME-THRESHOLD FROM WS-HOURS
                   GIVING WS-OVERTIME-HOURS
               COMPUTE WS-GROSS-PAY =
                   (WS-REGULAR-HOURS * WS-RATE) +
                   (WS-OVERTIME-HOURS * WS-RATE *
                    WS-OVERTIME-MULT)
           ELSE
               COMPUTE WS-GROSS-PAY =
                   WS-HOURS * WS-RATE
           END-IF

           MULTIPLY WS-GROSS-PAY BY WS-TAX-RATE
               GIVING WS-TAX ROUNDED

           SUBTRACT WS-TAX FROM WS-GROSS-PAY
               GIVING WS-NET-PAY.

       DISPLAY-PAYSLIP.
           DISPLAY "================================"
           DISPLAY "  PAYROLL SUMMARY"
           DISPLAY "================================"
           DISPLAY "Employee:  " WS-NAME
           DISPLAY "Hours:     " WS-HOURS
           DISPLAY "Rate:      " WS-RATE
           DISPLAY "Gross Pay: " WS-GROSS-PAY
           DISPLAY "Tax (20%): " WS-TAX
           DISPLAY "Net Pay:   " WS-NET-PAY
           DISPLAY "================================".

       DISPLAY-GRADE.
           EVALUATE TRUE
               WHEN JUNIOR
                   DISPLAY "Grade: Junior"
               WHEN SENIOR
                   DISPLAY "Grade: Senior"
               WHEN MANAGER
                   DISPLAY "Grade: Manager"
               WHEN OTHER
                   DISPLAY "Grade: Unknown"
           END-EVALUATE.

Becomes this C++:

// Transpiled from COBOL program: PAYROLL-CALC
// Generated by Easy COBOL Migrator on 2026-03-16 07:09:47

#include <iostream>
#include <string>
#include <cmath>
#include <cstdlib>
#include <cstdint>

// COBOL string move helper: truncate or right-pad to field width
inline std::string cobolMove(const std::string& src, int len) {
    std::string r = src; r.resize(len, ' '); return r;
}

// COBOL string comparison: pad shorter operand with spaces
inline int cobolCmp(const std::string& a, const std::string& b) {
    size_t len = std::max(a.size(), b.size());
    std::string la = a, lb = b;
    la.resize(len, ' '); lb.resize(len, ' ');
    return la.compare(lb);
}

void main_program();
void calculate_pay();
void display_payslip();
void display_grade();

// WORKING-STORAGE variables
struct {
    std::string ws_name = std::string(25, ' ');
    double ws_hours = 0.0;
    double ws_rate = 0.0;
    double ws_gross_pay = 0.0;
    double ws_tax = 0.0;
    double ws_net_pay = 0.0;
} ws_employee;
double ws_tax_rate = 0.20;
double ws_overtime_mult = 1.5;
int ws_overtime_threshold = 40;
double ws_overtime_hours = 0;
double ws_regular_hours = 0;
int ws_counter = 0;
int ws_pay_grade = 0;
inline bool junior() { return ws_pay_grade == 1; }
inline bool senior() { return ws_pay_grade == 2; }
inline bool manager() { return ws_pay_grade == 3; }

int64_t return_code = 0;

void main_program() {
    ws_employee.ws_name = cobolMove("Alice Johnson", 25);
    ws_employee.ws_hours = 45.00;
    ws_employee.ws_rate = 32.50;
    ws_pay_grade = 2;
    calculate_pay();
    display_payslip();
    display_grade();
    exit(0);
}

void calculate_pay() {
    if (ws_employee.ws_hours > ws_overtime_threshold) {
        ws_regular_hours = ws_overtime_threshold;
        ws_overtime_hours = ws_employee.ws_hours - (ws_overtime_threshold);
        ws_employee.ws_gross_pay = ((ws_regular_hours * ws_employee.ws_rate) + ((ws_overtime_hours * ws_employee.ws_rate) * ws_overtime_mult));
    } else {
        ws_employee.ws_gross_pay = (ws_employee.ws_hours * ws_employee.ws_rate);
    }
    ws_employee.ws_tax = std::round((ws_employee.ws_gross_pay * ws_tax_rate) * 100.0) / 100.0;
    ws_employee.ws_net_pay = ws_employee.ws_gross_pay - (ws_employee.ws_tax);
}

void display_payslip() {
    std::cout << "================================" << std::endl;
    std::cout << "  PAYROLL SUMMARY" << std::endl;
    std::cout << "================================" << std::endl;
    std::cout << "Employee:  " << ws_employee.ws_name << std::endl;
    std::cout << "Hours:     " << ws_employee.ws_hours << std::endl;
    std::cout << "Rate:      " << ws_employee.ws_rate << std::endl;
    std::cout << "Gross Pay: " << ws_employee.ws_gross_pay << std::endl;
    std::cout << "Tax (20%): " << ws_employee.ws_tax << std::endl;
    std::cout << "Net Pay:   " << ws_employee.ws_net_pay << std::endl;
    std::cout << "================================" << std::endl;
}

void display_grade() {
    if (junior()) {
        std::cout << "Grade: Junior" << std::endl;
    } else if (senior()) {
        std::cout << "Grade: Senior" << std::endl;
    } else if (manager()) {
        std::cout << "Grade: Manager" << std::endl;
    } else {
        std::cout << "Grade: Unknown" << std::endl;
    }
}

int main() {
    main_program();
    return 0;
}

Which compiles and runs without even polishing it:

The migration report flags any constructs that need manual attention, like EXEC SQL blocks or dynamic CALL targets.

COBOL migration coverage

The parser handles all four COBOL divisions:

36 statement types including MOVE, COMPUTE, IF/EVALUATE, all PERFORM variants including THRU and VARYING with AFTER, SORT/MERGE with key-field extraction, STRING/UNSTRING, INSPECT, SEARCH/SEARCH ALL, CALL with static and dynamic dispatch, GO TO DEPENDING ON, and RELEASE/RETURN for sort procedures
Full data description with levels 01-88, OCCURS DEPENDING ON, REDEFINES, RENAMES, FILLER, all PIC/USAGE variants, and MOVE/ADD/SUBTRACT CORRESPONDING
LINKAGE SECTION transpilation for subprogram interfaces with CALL BY REFERENCE, BY CONTENT and BY VALUE parameter passing
File I/O with OPEN/CLOSE/READ/WRITE/REWRITE/DELETE/START, record packing/unpacking, seek-based in-place update and FILE STATUS tracking
COPY preprocessor with nested copybooks up to 10 levels, REPLACING with pseudo-text substitution and circular-include detection
40+ intrinsic functions mapped to native equivalents in all 6 languages
EXEC SQL, EXEC CICS and EXEC DLI blocks preserved as comments with migration notes
Built-in analysis tools: dead code detection, cross-reference maps, complexity metrics and COBOL source utilities

Known limitations: indexed/relative file random/dynamic access converts to sequential with a migration note recommending database replacement. Report Writer and Screen Section are not supported.

Who needs a COBOL migration tool?

Migration consultants who want to accelerate their conversion projects
Development teams at banks, insurers or government agencies migrating off mainframes
Solo developers contracted for COBOL modernization work
Anyone curious about what a 1959 programming language looks like as Rust

Demo

There's a free demo that converts single files up to 500 lines to C++ output.

Product page

If your organisation needs help with the parts the tool can't convert, like EXEC SQL replacement, database re-platforming or end-to-end migration, I also offer hands-on COBOL migration consulting.

498 automated tests. 6 languages. One developer in London who now knows more about COBOL than he ever planned to.

I'd love to hear what breaks. Drop your weirdest COBOL in the demo and let me know.

I Built Free Desktop GUIs for Every Major Cloudflare Developer Service

Mecanik1337 — Fri, 13 Mar 2026 03:31:47 +0000

If you've worked with Cloudflare's developer platform - D1, R2, KV, or Images, you know the drill: you're either writing CLI commands or jumping between tabs in a Cloudflare dashboard that wasn't really designed for heavy developer use.

I got tired of it. So I built a suite of free desktop apps to fix that.

What I Built

Four cross-platform (Windows / macOS / Linux) Qt desktop apps, each targeting a different Cloudflare service:

App	Service	What it does
Easy Cloudflare D1	D1 (serverless SQL)	Browse, query, and manage your D1 databases with a full SQL editor
Easy Cloudflare R2	R2 (object storage)	Upload, browse, and manage R2 buckets like an S3 browser
Easy Cloudflare KV	Workers KV	Browse, create, edit, and monitor KV namespaces and key-value pairs
Easy Cloudflare Images	Cloudflare Images	Batch upload, variant preview, signed URL generation, bulk export

All four are completely free. No subscription, no SaaS, no telemetry.

Why Desktop and Not a Web App?

A few reasons:

Security first. Your Cloudflare API credentials never leave your machine. They're stored with strong local encryption (Windows DPAPI, macOS/Linux AES-256-GCM). No backend, no database of your tokens sitting somewhere you don't control. You can also restrict your API token by IP in the Cloudflare dashboard so a leaked token becomes useless.

Native feel. Browser-based tools for managing databases or object storage always feel slightly off. A native Qt app renders fast, handles large result sets well, and doesn't fight with your browser's tab limit.

The Pain Points These Solve

D1

Cloudflare's dashboard D1 editor is functional but limited, no query history, no result export, no multi-statement execution comfort. Easy Cloudflare D1 gives you a proper SQL editor with table browsing, so you can actually work with your data without reaching for wrangler.

R2

If you've managed S3 before, you've probably used Cyberduck, S3 Browser, or similar tools. R2 had nothing comparable. Easy Cloudflare R2 fills that gap; drag and drop files, browse buckets, manage objects visually.

KV

Workers KV is fantastic for edge config, feature flags, and caching; but inspecting what's actually stored is painful without a dedicated tool. Easy Cloudflare KV gives you a clean view of all your namespaces and lets you read, write, and delete keys without writing scripts.

Images

The Cloudflare Images dashboard handles basic uploads, but batch workflows, variant previewing side-by-side, and generating signed URLs require either the API or a lot of copy-pasting. Easy Cloudflare Images makes all of that a few clicks.

Technical Stack

Built with C++ and Qt6, which gives native performance and a consistent experience across all three platforms from a single codebase. The binary is protected and closed-source, but the tools themselves are free to use.

How to Get Them

Each app is available at mecanik.dev/products. You fill in a short form to get a free license key (3 activations per key) and download links for all platforms are emailed to you instantly.

Would love feedback from anyone already using Cloudflare's developer platform. What features would make these more useful for your workflow?

Build a Fast NLP Pipeline with Modern Text Tokenizer in C++

Mecanik1337 — Fri, 08 Aug 2025 09:14:58 +0000

As a C++ developer in natural language processing (NLP) or machine learning (ML), you've probably hit the tokenization wall.

Tokenization, more specifically splitting text into words, sub-words, or characters is critical for models like BERT or DistilBERT, but most C++ tokenizers are either slow, bloated with dependencies, or choke on Unicode.

That's why I built Modern Text Tokenizer, a header-only, UTF-8-aware C++ library that's fast, lightweight, and ready for ML pipelines.

In this tutorial, I'll show you how to set up Modern Text Tokenizer, integrate it into your C++ project, and use it to preprocess text for NLP tasks.

If you're building CLI tools, on-device NLP apps, or feeding data to transformer models, this tokenizer will save you time and headaches.

Why Modern Text Tokenizer?

Modern Text Tokenizer is designed for developers who need performance and simplicity:

Zero Dependencies: No Boost or ICU, just a single header file.
UTF-8 Safe: Handles emojis, multilingual text, and multibyte characters.
ML-Ready: Maps tokens to vocabulary IDs for BERT, DistilBERT, and more.
Fast: Processes 174,000 characters in ~4.5 ms (36.97 MB/s throughput on Ryzen 9 5900X).
Cross-Platform: Tested on Ubuntu, Windows, and GitHub Actions CI.

Want to see it in action? Let's walk through setting it up and tokenizing text for an NLP pipeline.

Setting Up Modern Text Tokenizer

Here's how to integrate it into your C++ project in minutes.

Clone the Repo

Grab the header file from the Modern Text Tokenizer GitHub repository:

git clone https://github.com/Mecanik/Modern-Text-Tokenizer.git

Copy Modern-Text-Tokenizer.hpp into your project directory.

Include the Header

Add the header to your C++ code:

#include "Modern-Text-Tokenizer.hpp"

Compile

Ensure you're using C++17 or later and optimize for performance:

g++ -std=c++17 -O3 -o tokenizer_demo main.cpp

Download a Vocabulary

For ML tasks, you'll need a vocabulary file (like from HuggingFace). For example, to use DistilBERT’s vocab:

curl -O https://huggingface.co/distilbert/distilbert-base-uncased/raw/main/vocab.txt

Tokenizing Text - A Quick Example

Let's tokenize some text and encode it for an ML model. Here's a complete example:

#include <iostream>
#include "Modern-Text-Tokenizer.hpp"

int main() {
    TextTokenizer tokenizer;

    // Configure the tokenizer
    tokenizer
        .set_lowercase(true)
        .set_split_on_punctuation(true)
        .set_keep_punctuation(true);

    // Load a vocabulary (optional for ML pipelines)
    tokenizer.load_vocab("vocab.txt");

    // Encode text to token IDs
    std::string text = "Hello, world 👋 Let's build NLP apps.";
    auto ids = tokenizer.encode(text);

    // Print token IDs
    std::cout << "Token IDs: ";
    for (const auto& id : ids) {
        std::cout << id << " ";
    }
    std::cout << std::endl;

    // Decode back to text
    std::string decoded = tokenizer.decode(ids);
    std::cout << "Decoded: " << decoded << std::endl;

    return 0;
}

Run this, and you'll see the token IDs and the decoded text. The tokenizer handles UTF-8 characters like emojis (👋) without breaking a sweat.

Performance Benchmarks

Modern Text Tokenizer is built for speed. Here's how it performs on a Ryzen 9 5900X with -O3 optimization, processing 174,000 characters:

Task	Time (μs)	Throughput
Tokenization	2159	36.97 MB/s
Encoding	1900	-
Decoding	430	-
Total	4489	-

This makes it ideal for real-time NLP applications or large-scale text processing.

Advanced Usage - Customizing for Your needs

The fluent API lets you tailor the tokenizer to your project. For example, to tokenize without lowercase and keep punctuation as separate tokens:

TextTokenizer tokenizer;
tokenizer
    .set_lowercase(false)
    .set_split_on_punctuation(true)
    .set_keep_punctuation(true);

auto ids = tokenizer.encode("Hello, World! C++ is awesome.");

Need to preprocess for BERT? Load the vocab and add special tokens:

tokenizer.load_vocab("vocab.txt");
auto ids = tokenizer.encode("[CLS] Hello, world! [SEP]");

This ensures compatibility with transformer models.

Use Cases for C++ Developers

Here are some practical scenarios where Modern Text Tokenizer shines:

ML Preprocessing: Prepare text for BERT or DistilBERT in C++ without Python.
CLI Tools: Build fast text-processing utilities for data pipelines.
Embedded NLP: Run tokenization on resource-constrained devices.
Multilingual Apps: Process text in any language with full UTF-8 support.

Why Not Use Python?

Python's tokenizers (like HuggingFace's transformers) are great but come with overhead. If you're working in C++ for performance, portability, or embedded systems, Modern Text Tokenizer lets you stay in C++ without sacrificing functionality.

Try It

Ready to turbocharge your C++ NLP projects? Clone the Modern Text Tokenizer repo, include the header, and start tokenizing. Got questions or cool use cases? Share them in the comments or open an issue on GitHub.

For a deeper dive into its design and performance, check out my blog post.

Let's make C++ NLP faster and easier!

How to use efficiently IPSET with CSF Firewall

Mecanik1337 — Sat, 02 Mar 2024 09:30:36 +0000

Hello,

This is my first ever post here 😀

In this post we will discuss how to use efficiently IPSET with CSF (ConfigServer Security & Firewall) Firewall. Unfortunately, I will not discuss what CSF is because it's not within the scope, but you can read about it here.

If you already used CSF or just getting started using it, you might have noticed that the DENY_IP_LIMIT and DENY_TEMP_IP_LIMIT is relatively small and fills up very fast. CSF advises you to use IPSET if you wish to increase this limit.

What it doesn't tell you clearly is that even if you install IPSET and enable it inside CSF, you still need to increase the values of DENY_IP_LIMIT and DENY_TEMP_IP_LIMIT otherwise it will have no effect. Trust me when I say that such a simple thing caused me quite some headaches...

What is IPSET?

IPSET is a Linux kernel feature that allows for efficient management of large sets of IP addresses. It's particularly useful for implementing firewalls and access control lists. CSF leverages IPSET for managing IP addresses efficiently.

Diagram:

      +------------+
      |    IPSET   |
      +------------+
            |
    +-------|-------+
    |       |       |
+-------+ +-------+ +-------+
| IP    | | IP    | | IP    |
| Address| | Address| | Address|
+-------+ +-------+ +-------+

The "IPSET" box represents the IPSET data structure managed by CSF.
Each IP address is stored within the IPSET, organized for efficient lookup.
CSF interacts with IPSET to add, remove, or query IP addresses based on firewall rules and security policies.

How is IPSET configured in CSF?

By default CSF provides you with following configuration:

# The following sets the hashsize for ipset sets, which must be a power of 2.
#
# Note: Increasing this value will consume more memory for all sets
# Default: "1024"
LF_IPSET_HASHSIZE = "1024"

# The following sets the maxelem for ipset sets.
#
# Note: Increasing this value will consume more memory for all sets
# Default: "65536"
LF_IPSET_MAXELEM = "65536"

So what does this actually mean for you? The default settings for LF_IPSET_HASHSIZE and LF_IPSET_MAXELEM in CSF are set conservatively to balance memory usage and performance on a typical system. With this configuration the IPSET can accommodate up to 67,108,864 IP addresses (theoretically 😅).

The configuration settings for LF_IPSET_HASHSIZE and LF_IPSET_MAXELEM affect the efficiency and capacity of the IPSET data structure within CSF. While it's true that the default configuration allows for a theoretical maximum of over 67 million IP addresses, it's essential to consider limitations such as memory usage and performance.

Generally, increasing LF_IPSET_MAXELEM will consume more memory as it allows for storing more IP addresses, while LF_IPSET_HASHSIZE primarily affects lookup performance but also has some memory overhead.

Keep in mind:

A larger hash table requires more memory to store IP addresses efficiently. By defaulting to a smaller hash size (LF_IPSET_HASHSIZE = "1024"), CSF ensures that it doesn't consume excessive memory resources on systems with limited RAM or where memory usage needs to be carefully managed.
While a larger hash table can improve lookup performance for IP addresses, it may also introduce overhead in terms of memory access and CPU usage. By keeping the default hash size relatively small, CSF aims to strike a balance between memory consumption and performance impact on the system.
The default settings are chosen to be compatible with a wide range of systems and use cases. They provide a reasonable starting point that works well for many installations without requiring fine-tuning.
CSF is designed to be scalable, allowing administrators to adjust settings like LF_IPSET_HASHSIZE and LF_IPSET_MAXELEM based on their specific requirements and available resources. While the default settings may be conservative, they can be modified to better suit the needs of individual environments.

In summary, the default hash size in CSF is set to a relatively small value to strike a balance between memory usage and performance impact. Administrators can adjust these settings based on their specific requirements, system resources, and performance considerations.

Increasing IPSET efficiently for performance

In order to store millions of IP addresses efficiently you need to configure the limits accordingly, based on your available resources. As explained earlier, IPSET is mainly about RAM. So with that being said, I will provide example configurations for various amounts of RAM with balanced IP set settings.

These configurations aim to strike a balance between memory usage, performance, and scalability. Keep in mind that these are example configurations, and you may need to adjust them based on your specific requirements and system characteristics.

System with 2 GB RAM

LF_IPSET_MAXELEM = "65536
LF_IPSET_HASHSIZE = "131072 (Approx. 131K IPs)

System with 4 GB RAM

LF_IPSET_MAXELEM = "131072"
LF_IPSET_HASHSIZE = "262144" (~262K IPs)

System with 6 GB RAM

LF_IPSET_MAXELEM = "196608"
LF_IPSET_HASHSIZE = "393216" (~393K IPs)

System with 8 GB RAM

LF_IPSET_MAXELEM = "262144"
LF_IPSET_HASHSIZE = "524288" (~524K IPs)

System with 16 GB RAM

LF_IPSET_MAXELEM = "524288"
LF_IPSET_HASHSIZE = "1048576" (~1M IPs)

System with 32 GB RAM

LF_IPSET_MAXELEM = "1048576"
LF_IPSET_HASHSIZE = "2097152" (~2M IPs)

System with 64 GB RAM

LF_IPSET_MAXELEM = "2097152"
LF_IPSET_HASHSIZE = " 4194304" (~4M IPs)

System with 128 GB RAM

LF_IPSET_MAXELEM = "4194304"
LF_IPSET_HASHSIZE = "8388608" (~8M IPs)

Adjusting LF_IPSET_HASHSIZE proportionally higher than LF_IPSET_MAXELEM helps maintain efficient lookup times, especially as the number of stored IP addresses increases. These configurations aim to balance memory usage with performance requirements, allowing for optimal operation of CSF IP sets.

You can monitor the current IPSET usage with the following command:

ipset list -t

This command will list detailed information about the IP sets, including statistics such as the number of elements and the size of the hash table.

Potential overhead

When increasing the limits for LF_IPSET_HASHSIZE and LF_IPSET_MAXELEM in CSF, there are several potential overheads to consider:

One of the most significant overheads is the increased memory usage. IPSET data structures consume memory, and larger hash tables or higher maximum element limits will require more RAM. This can impact overall system performance, especially on systems with limited memory resources.
While larger hash tables can improve lookup performance by reducing collisions and improving cache locality, excessively large hash tables may lead to slower lookup times due to increased memory access overhead.
Allocating more memory for IPSET may lead to resource contention with other processes running on the system. If the system is under heavy load, increased memory usage by IPSET could exacerbate resource contention issues.
In extreme cases, allocating too much memory for IPSET could potentially lead to system instability, especially if the system runs out of physical memory and starts swapping excessively.

So please keep these in mind and don't start increasing the IPSET limits in CSF on small/budget type VPS.

And that about concludes my first post here. Unfortunately I don't have more time right now to expand this article, but maybe in the coming weeks I will post more useful stuff about CSF 🙂

PS: I really don't like the default editor here. It's too basic and quite bugged.

Until we meet again!