Production query plans without production data

2026-03-08T22:05:00+00:00

In the previous article</a> we covered how the PostgreSQL planner reads pg_class</code> and pg_statistic</code> to estimate row counts, choose join strategies, and decide whether an index scan is worth it. The message was clear: when statistics are wrong, everything else goes with it.

Streaming replication provides bit-to-bit replication, so all replicas share the same statistics with primary server.</div> But there was one thing we didn't talk about. Statistics are specific to the database cluster that generated them. The primary way to populate them is `ANALYZE` which requires the actual data. PostgreSQL 18 changed that. Two new functions: pg_restore_relation_stats</code> and pg_restore_attribute_stats</code> write numbers directly into the catalog tables. Combined with pg_dump --statistics-only</code>, you can treat optimizer statistics as a deployable artifact. Compact, portable, plain SQL.

The feature was driven by the upgrade use case</a>. In the past, major version upgrades used to leave pg_statistic</code> empty, forcing you to run ANALYZE</code>. Which might take hours on large clusters. With PostgreSQL 18 upgrades now transfer statistics automatically. But that's just the beginning. The same logic lets you export statistics from production and inject them anywhere - test database, local debugging, or as part of CI pipelines.

The problem</a> </h2> Your CI database has 1,000 rows. Production has 50 million. The planner makes completely different decisions for each. Running EXPLAIN</code> in CI tells you nothing about the production plan. This is the core premise behind RegreSQL</a>. Catching query plan regressions in CI is far more reliable when the planner sees production-scale statistics. Same applies to debugging. A query is slow in production and you want to reproduce the plan locally, but your database has different statistics, and planner chooses the predictable path. Porting production stats can provide you that snapshot of thinking planner has to do in production, without actually going to production. pg_restore_relation_stats</a> </h2> The first of function behind portable PostgreSQL statistics is pg_restore_relation_stats</code>. It writes table-level data directly into pg_class</code> in form of variadic name/value pairs. SELECT pg_restore_relation_stats( 'schemaname', 'public', 'relname', 'orders', 'relpages', 123513::integer, 'reltuples', 50000000::real, 'relallvisible', 123513::integer, 'relallfrozen', 120000::integer );</code></pre> But that's just an example. Let's modify some real statistics to see the full value. We will create a small table, inject fake production-like statistics and watch the planner to change its mind. CREATE TABLE test_orders ( id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY, customer_id integer NOT NULL, amount numeric(10,2) NOT NULL, status text NOT NULL DEFAULT 'pending', created_at date NOT NULL DEFAULT CURRENT_DATE ); INSERT INTO test_orders (customer_id, amount, status, created_at) SELECT (random() * 9999 + 1)::int, (random() * 5000 + 5)::numeric(10,2), (ARRAY['pending','shipped','delivered','cancelled'])[floor(random()*4+1)::int], '2024-01-01'::date + (random() * 365)::int FROM generate_series(1, 10000); CREATE INDEX ON test_orders (created_at); CREATE INDEX ON test_orders (status); ANALYZE test_orders;</code></pre> When you check the current statistics, it has predictable data. SELECT relname, relpages, reltuples FROM pg_class WHERE relname = 'test_orders';</code></pre> relname | relpages | reltuples -------------+----------+----------- test_orders | 74 | 10000 (1 row)</code></pre> With 10,000 rows across 74 pages, the planner picks a sequential scan. EXPLAIN SELECT * FROM test_orders WHERE created_at > '2024-06-01';</code></pre> QUERY PLAN ----------------------------------------------------------------- Seq Scan on test_orders (cost=0.00..199.00 rows=5891 width=26) Filter: (created_at > '2024-06-01'::date) (2 rows)</code></pre> Now inject production-scale table stats: SELECT pg_restore_relation_stats( 'schemaname', 'public', 'relname', 'test_orders', 'relpages', 123513::integer, 'reltuples', 50000000::real, 'relallvisible', 123513::integer );</code></pre> And you might be surprised by the result. EXPLAIN SELECT * FROM test_orders WHERE created_at > '2024-06-01';</code></pre> QUERY PLAN ------------------------------------------------------------------ Seq Scan on test_orders (cost=0.00..448.45 rows=17649 width=26) Filter: (created_at > '2024-06-01'::date)</code></pre> The planner is still using the sequential plan. Only the estimated number of rows has changed. Why? If you remember from previous article, it's where column level statistics come into play. Histogram bounds still match the original 10,000 rows we inserted. pg_restore_attribute_stats</a> </h2> This function writes column-level statistics into pg_statistic</code> the same catalog that ANALYZE populates</a> with MCVs, histograms, and correlation</a>. In previous section, we left the planner stuck on a sequential scan despite believing the table has 50 million rows. The missing piece is column-level statistics. Let's pick up where we left off and inject histogram bounds for created_at</code>. SELECT pg_restore_attribute_stats( 'schemaname', 'public', 'relname', 'test_orders', 'attname', 'created_at', 'inherited', false::boolean, 'null_frac', 0.0::real, 'avg_width', 4::integer, 'n_distinct', -0.05::real, 'histogram_bounds', '{2019-01-01,2019-07-01,2020-01-01,2020-07-01,2021-01-01,2021-07-01,2022-01-01,2022-07-01,2023-01-01,2023-07-01,2024-01-01}'::text, 'correlation', 0.98::real );</code></pre> Now the planner knows the data spans 5 years. A query filtering on the last 6 months of 2024 covers a narrow slice. EXPLAIN SELECT * FROM test_orders WHERE created_at > '2024-06-01';</code></pre> QUERY PLAN ---------------------------------------------------------------------------------------------------- Index Scan using test_orders_created_at_idx on test_orders (cost=0.29..153.21 rows=6340 width=26) Index Cond: (created_at > '2024-06-01'::date)</code></pre> Histogram bounds divide the non-MCV portion of the data into equal-population buckets. If most_common_vals</code> accounts for most of the data, the histogram covers only the remaining tail. The number of buckets is controlled by default_statistics_target</code> (default 100, meaning 101 bounds). </div> And that's a plan flip! The histogram tells the planner the data spans 2019–2024, so > '2024-06-01'</code> matches a narrow tail. A small fraction of 50 million rows. The index scan that was ignored before is now the obvious choice. Table-level stats set the scale, column-level stats shaped the selectivity, and together they changed the plan. The correlation</code> statistic tells the planner how closely the physical row order matches the column's sort order. A value near 1.0 means sequential access patterns - making index scan cheaper</a> because the next row is likely on the same or adjacent page. For time-series data like created_at</code> where rows are inserted chronologically, correlation is typically very high. </div> Injecting a skewed distribution</a> </h2> The same function handles MCV lists</a>. In production, your status</code> column isn't uniform, 95% of orders are delivered, 1.5% are pending. SELECT pg_restore_attribute_stats( 'schemaname', 'public', 'relname', 'test_orders', 'attname', 'status', 'inherited', false::boolean, 'null_frac', 0.0::real, 'avg_width', 9::integer, 'n_distinct', 5::real, 'most_common_vals', '{delivered,shipped,cancelled,pending,returned}'::text, 'most_common_freqs', '{0.95,0.015,0.015,0.015,0.005}'::real[] );</code></pre> You can see EXPLAIN SELECT * FROM test_orders WHERE status = 'pending';</code></pre> QUERY PLAN --------------------------------------------------------------------------------------- Bitmap Heap Scan on test_orders (cost=8.93..90.42 rows=599 width=27) Recheck Cond: (status = 'pending'::text) -> Bitmap Index Scan on test_orders_status_idx (cost=0.00..8.78 rows=599 width=0) Index Cond: (status = 'pending'::text) (4 rows)</code></pre> and compare it with EXPLAIN SELECT * FROM test_orders WHERE status = 'delivered';</code></pre> QUERY PLAN ------------------------------------------------------------------ Seq Scan on test_orders (cost=0.00..448.45 rows=28458 width=27) Filter: (status = 'delivered'::text) (2 rows)</code></pre> Same column, same operator, different plans. The planner uses a bitmap index scan for pending</code> (1.5% rare enough to justify the index) and a sequential scan for delivered</code> (95% being most of the table). The selectivity ratios from the MCV list drive the plan choice. You might have noticed the row estimates (599 and 28,458) are lower than you'd expect for a 50-million-row table. The planner checks the actual physical file size. Our table is only 74 pages on disk, not the 123,513 we injected. Hence the planner scales reltuples</code> and relpages</code> down proportionally. The absolute numbers shrink, but the ratios between them stay correct, and it's the ratios that determine plan shape. When you use pg_dump --statistics-only</code> in practice, you're typically restoring into a database with comparable data volume, so the estimates align naturally. </div> </div> </div> </div> </div> pg_regresql extension The pg_regresql</code> extension fixes this scaling problem. It hooks into the planner to trust the injected relpages</code> value instead of reading the physical file size, so cost estimates match production even when your test database is tiny. Read more</a> </div> </div> pg_dump</a> </h2> The functions we covered are the mechanics. For operational use pg_dump</code> provides everything you need. PostgreSQL 18 added three flags. Flag</th> Effect</th></tr></thead> --statistics</code></td> dump the statistics (you have to request it explicitely)</td></tr> --statistics-only</code></td> dump only the statistics, not schema or data</td></tr> --no-statistics</code></td> do not dump statistics</td></tr> </tbody></table> When you export the statistics for your production database pg_dump --statistics-only -d production_db > stats.sql </code></pre> you will see the output is series of SELECT pg_restore_relation_stats(...)</code> and SELECT pg_restore_attribute_stats(...)</code> calls. Exactly as we explained above. The full workflow to turn your production data into testable plans might look like this: # 1. dump schema from production pg_dump --schema-only -d production_db > schema.sql # 2. dump statistics from production pg_dump --statistics-only -d production_db > stats.sql # 3. create test database with schema createdb test_db psql -d test_db -f schema.sql # 4. load fixture data (optional; masked, minimal) psql -d test_db -f fixtures.sql # 5. inject production statistics psql -d test_db -f stats.sql # 6. query plans now match production psql -d test_db -c "EXPLAIN SELECT * FROM test_orders WHERE status = 'pending'"</code></pre> Statistics dumps are tiny. A database with hundreds of tables and thousands of columns produces a statistics dump under 1MB. The production data might be hundreds of GB. The statistics that describe it fit in a text file. </div> Keeping injected statistics alive</a> </h2> Now you might ask yourself, where's the catch? And there's a big one, the autovacuum will eventually kick in and run ANALYZE</code>. Which will overwrite your injected statistics with real numbers and you are back where you started. To prevent this, disable autovacuum analyze on the tables you've injected. -- disable autovacuum ALTER TABLE test_orders SET (autovacuum_enabled = false); -- or set analyze threshold so high it nevers kicks-in ALTER TABLE test_orders SET (autovacuum_analyze_threshold = 2147483647);</code></pre> Be careful here. If you're also writing data to these tables in dev: running migrations, loading fixtures, testing inserts, the injected statistics will drift further from reality with every write. The planner will plan based on a production distribution that no longer reflects the local data. For read-only query plan testing this is exactly what you want. For integration tests that modify data, you may need to re-inject statistics after each test run. And please, never ever do this in production! </div> What's not covered?</a> </h2> As we have seen earlier, it's not worth trying to inject relpages</code> as the planner checks the actual file size and scales it proportationally. This limits the number of absolute rows planner might estimate. I.e. to get comparable numbers to production environment you still would have to create comparable data volume (which isn't a problem when talking about the primary use case of this feature - restoring backups). It's also worth to note that CREATE STATISTICS</code> used for multivariate correlations, distinct counts across column groups and MCV lists for column combinations</a> are not covered within PostgreSQL 18. Those still require ANALYZE</code> after restore. PostgreSQL 19 will close this gap with pg_restore_extended_stats()</code>. Security</a> </h2> The restore functions require the MAINTAIN</code> privilege on the target table. This is the same privilege needed for ANALYZE</code>, VACUUM</code>, REINDEX</code>, and CLUSTER</code> as it was introduced in PostgreSQL 17</a>. The easiest way to grant it for automation: GRANT pg_maintain TO ci_service_account;</code></pre> This grants MAINTAIN</code> on all tables in the database. Enough for a CI pipeline to inject statistics without needing superuser. PostgreSQL Statistics: Why queries run slow 2026-02-26T23:35:00+00:00 Every query starts with a plan. Every slow query probably starts with a bad one. And more often than not, the statistics are to blame. But how does it really work? PostgreSQL doesn't run the query to find out - it estimates the cost. It reads pre-computed data from pg_class</code> and pg_statistic</code> and does the maths to figure out the cheapest path to your data. In ideal scenario, the numbers read are accurate, and you get the plan you expect. But when they are stale, the situation gets out of control. Planner estimates 500 rows, plans a nested loop, and hits 25,000. What seemed as optimal plan turns into a cascading failure. How do statistics get stale? It can be either bulk load, a schema migration, faster-than-expected growth, or simply VACUUM</code> not keeping up. Whatever the cause, the result is the same. The planner is flying blind. Choosing paths based on reality that no longer exists. In this post we will go inside the two catalogs the planner depends on, understand what ANALYZE</code> actually gets for you from a 30,000-row table, and see how those numbers determine whether your query takes milliseconds or minutes. Sample schema</a> </h2> For demonstration purposes we will use the same schema as in the article Reading Buffer statistics in EXPLAIN output</a>. CREATE TABLE customers ( id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY, name text NOT NULL ); CREATE TABLE orders ( id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY, customer_id integer NOT NULL REFERENCES customers(id), amount numeric(10,2) NOT NULL, status text NOT NULL DEFAULT 'pending', note text, created_at date NOT NULL DEFAULT CURRENT_DATE ); INSERT INTO customers (name) SELECT 'Customer ' || i FROM generate_series(1, 2000) AS i; INSERT INTO orders (customer_id, amount, status, note, created_at) SELECT (random() * 1999 + 1)::int, (random() * 500 + 5)::numeric(10,2), (ARRAY['pending','shipped','delivered','cancelled'])[floor(random()*4+1)::int], CASE WHEN random() < 0.3 THEN 'Some note text here for padding' ELSE NULL END, '2022-01-01'::date + (random() * 1095)::int FROM generate_series(1, 100000); ANALYZE customers; ANALYZE orders;</code></pre>What the Planner Reads</a> </h2> As mentioned above, every decision the planner makes is based on two sources: table-level metadata from pg_class</code></li> column-level metadata from pg_statistic</code>.</li> </ul> pg_class - relational-level stats</a> </h3> It actually tracks all relations. Not just tables and indexes, but also partitions, TOAST tables, sequences, composite types, and views. </div>Every table, index and materialized view has a row in `pg_class`. Before it even looks at column-level statistics, the planner reads four values from it: Column</th> Meaning</th></tr></thead> relpages</code></td> Number of 8KB pages representing the table on the disk</td></tr> reltuples</code></td> Estimated number of live rows in the table</td></tr> relallvisible</code></td> Pages where all tuples are visible to all transactions</td></tr> </tbody></table> For our sample table it looks like this. SELECT relname, relpages, reltuples, relallvisible FROM pg_class WHERE relname = 'orders';</code></pre> relname | relpages | reltuples | relallvisible ---------+----------+-----------+--------------- orders | 856 | 100000 | 856</code></pre>Compared to reading page from disk, handling single tuple represents tiny overhead. Per-row overhead is configured using cpu_tuple_cost</code>, which defaults to 0.01</code></div> The planner sees 100,000 rows spread across 856 pages. Every cost estimate starts from those two numbers. relpages</code> drives sequential scan cost - each page is one unit of I/O work as configured via seq_page_cost</code>. reltuples</code> controls estimates for joins, aggregations, and pretty much everything else. The reltuples</code> value is only an estimate, not a live count. It's updated by ANALYZE</code> (and autovacuum), not by individual INSERTs or DELETEs. Between ANALYZE runs, PostgreSQL scales reltuples</code> proportionally when relpages</code> value changes - if the table grows by 10% in pages, the planner assumes 10% more rows too. This works well enough for normal growth, but breaks down with bloat. If dead tuples are inflating the number of pages used, without adding real rows, the planner overestimates the table size. </div> The other column mentioned above matters for specific operations. relallvisible</code> tells the planner how much of the table can be read with an index-only scan. Meaning an index-only scan can return results using the index without checking the heap for visibility. pg_statistic (via pg_stats) - column-level stats</a> </h3> Knowing the size of a table is only half the picture. To estimate how many rows might match, the planner needs to understand the data inside each column. PostgreSQL maintains statistics in pg_statistic</code> catalog. Which you're most likely never going to use directly. In practice you will use the view pg_stats</code> which presents the human-friendly data behind it. The most interesting values it exposes are: Statistic</th> What it tells the planner</th></tr></thead> null_frac</code></td> Fraction of entries that are NULL values</td></tr> avg_width</code></td> Average width in bytes</td></tr> n_distinct</code></td> Number of distinct values (negative means fraction of rows)</td></tr> most_common_vals</code></td> Most frequent values</td></tr> most_common_freqs</code></td> Frequencies of those values</td></tr> histogram_bounds</code></td> Values dividing the remaining data into equal-population buckets (most_common_vals are excluded)</td></tr> correlation</code></td> Statistical correlation between physical row ordering and logical ordering of the column values</td></tr> </tbody></table> SELECT attname, null_frac, avg_width, n_distinct, most_common_vals, most_common_freqs, histogram_bounds, correlation FROM pg_stats WHERE tablename = 'orders' AND attname = 'status';</code></pre>-[ RECORD 1 ]-----+-------------------------------------- attname | status null_frac | 0 avg_width | 8 n_distinct | 4 most_common_vals | {pending,shipped,delivered,cancelled} most_common_freqs | {0.25396666,0.25,0.24973333,0.2463} histogram_bounds | correlation | 0.2524199</code></pre> From this planner knows there are exactly 4 distinct values, more or less equally distributed, without any NULL values. When you write a predicate WHERE status = 'pending'</code> it will estimate ~25% of rows are going to match. All that by not running a query, but reading the catalog row. Different situation is when checking column note</code>. SELECT attname, null_frac, avg_width, n_distinct, array_length(most_common_vals, 1) AS mcv_count, array_length(histogram_bounds, 1) AS histogram_buckets FROM pg_stats WHERE tablename = 'orders' AND attname = 'note';</code></pre>-[ RECORD 1 ]-----+------- attname | note null_frac | 0.6982 avg_width | 32 n_distinct | 1 mcv_count | 1 histogram_buckets |</code></pre> For this column there's ~70% of NULLs, and only one distinct value. Therefore WHERE note IS NOT NULL</code> will estimate 30% of rows. Selectivity in Action</a> </h2> Now that we have covered what data the planner has available, let's have a look at how it's used to estimate how many rows a certain part of a query will read. This "guess" is called Selectivity. And it's defined as a floating-point number between 0 and 1. The formula is pretty simple: Estimated Rows = Total Rows * Selectivity</code></pre> But the way Selectivity is calculated depends entirely on the operator you use (=</code>, <</code>, ></code> or LIKE</code>). The equality (Most Common Values)</a> </h3> Equality is easiest to start with. When you use WHERE status = 'shipped'</code> the planner first checks most_common_vals</code> (MCV) list. If there's a match, the selectivity is same as the one provided by most_common_freqs</code>. If the value isn't in the list, the planner assumes the value is part of the "remaining" population. It subtracts all MCV frequencies from 1.0 and divides the remainder by the number of other distinct values. (1.0 - (SELECT sum(s) FROM unnest(most_common_freqs) s)) / (n_distinct - array_length(most_common_vals, 1))</code></pre>The Range lookup (The Histogram)</a> </h3> Life would be easy if we would be looking only for exact values. Most of the time we need to utilise range lookup. For example WHERE amount > 400</code>. MCVs are useless in this case as there might be thousands or millions of unique constants. This is where histogram_bounds</code> comes in. PostgreSQL divides the column values into a number of buckets, where each bucket contains equal number of rows (not values). The Selectivity in this case is determined by how many buckets your query covers. In our example, if we have bounds (100, 200, 300, 400, 500, 600)</code> the planner will establish it covers 2 full buckets. Since there are 5 buckets in total the Selectivity is going to be 0.4 (2/5). If you are now wondering where 2 and 5 come from? The histogram_bounds</code> array defines boundaries between buckets. With bounds (100, 200, 300, 400, 500, 600)</code> there are 5 buckets in total. (100-200), (200-300), (300-400), (400-500), (500-600)</code></pre> And the same logic applies to matching. The condition amount > 400</code> matches on (400-500)</code> and (500-600)</code> buckets (2 in total). The histogram's biggest weakness is linear interpolation. If your data has massive spikes in distribution the planner will always assume a perfect distribution.</div> Slightly more complex situation happens in the cases where your value falls inside a bucket. If your query has WHERE amount > 350</code> the planner locates the bucket containing value 350 (in our example (300-400)</code>), assumes the data is linearly distributed and calculates the ratio within that bucket. Search and pattern matching</a> </h3> This is probably the most treacherous territory for the planner. For substring matching patterns like WHERE note LIKE '%middle%'</code>, there's no histogram or list of values to rely on. The planner must fall back to "magic constants" hardcoded in the PostgreSQL source code. The default for generic patterns is 0.5% of the total rows, defined as #define DEFAULT_MATCH_SEL 0.005</code></pre> Slightly better situation comes for prefixed matches like WHERE note LIKE 'boringSQL%'</code> where PostgreSQL can fall back to range conditions and use histogram bounds. While this is a subtle difference, it makes a night and day difference. Correlation and index scan cost</a> </h3> Remember correlation</code> from the pg_stats</code>? It says how closely the physical order of rows on disk matches the logical order of column values. Values close to 1.0 mean high correlation, values near 0 mean data is laid out randomly across pages. If you recall how data is organized in 8KB page</a> the row locality matters. This matters because it determines whether an index scan is worth it. The planner assumes a random page read costs 4× more than a sequential one (random_page_cost = 4.0</code> vs seq_page_cost = 1.0</code>). When correlation is high, the rows an index points to are physically adjacent. The planner expects sequential I/O and costs the scan cheaply. When correlation is low, each lookup likely hits a different page, and the planner costs each of those reads at the higher random rate. That difference alone can be enough to make a sequential scan cheaper than an index scan. n_distinct and join estimation</a> </h3> Value n_distinct</code> plays an important role in joins. This further stresses the importance of running ANALYZE</code> after bulk data changes. Let's imagine this simplified logic for the equality join: In reality, Postgres also accounts for null_frac</code> (NULLs don't join) and MCVs on both sides. If both sides have MCVs, it calculates the "inner product" of the frequencies.</div> estimated_rows = (rows_left × rows_right) / max(n_distinct_left, n_distinct_right)</code></pre> Let's say you're trying to join two tables, both having roughly 2,000 distinct values for the join key. Outdated n_distinct</code> will cause significant estimation drifts and the planner may pick a wrong join strategy altogether. This is the heap we are talking about</a> </h3> Please, keep in mind this describes how the planner estimates rows for a basic heap scan. Indexes, join selectivity, and complex types like JSONB each come with their own estimation logic, operator handling and quirks. The fundamentals of estimation are the same nevertheless. What if there are no statistics?</a> </h2> So far we have touched on why up-to-date statistics are a must. But what if there are no statistics at all? For example for a new table or new column when ANALYZE</code> has not yet run. In those cases PostgreSQL falls back to hardcoded defaults. Condition type</th> Default selectivity</th> Constant</th></tr></thead> Equality (=</code>)</td> 0.5%</td> DEFAULT_EQ_SEL = 0.005</code></td></tr> Range (></code>, <</code>)</td> 33.3%</td> DEFAULT_INEQ_SEL = 0.3333</code></td></tr> Range (BETWEEN</code>)</td> 0.5%</td> DEFAULT_RANGE_INEQ_SEL = 0.005</code></td></tr> Pattern matching (LIKE</code>)</td> 0.5%</td> DEFAULT_MATCH_SEL = 0.005</code></td></tr> IS NULL</code></td> 0.5%</td> DEFAULT_UNK_SEL = 0.005</code></td></tr> IS NOT NULL</code></td> 99.5%</td> DEFAULT_NOT_UNK_SEL = 0.995</code></td></tr> </tbody></table> Nothing that a quick ANALYZE</code> can't fix, correct? Or maybe not. Where No Statistics Go While this article focuses on statistics and getting them right, there are situations where no statistics will be available (never or not predictably). If you believe it won't affect you, please, think twice. CTEs and subqueries when not inlined/materialized have no statistics. See Good CTE, Bad CTE</a> for when exactly this happens.</li> Temporary tables are not touched by autovacuum, so no automatic ANALYZE</code>.</li> Foreign tables do not guarantee stats are propagated.</li> And, to a big surprise, computed expressions in WHERE like WHERE amount * 1.1 > 500</code> or lower(email) = 'hello@example.com'</code>; unless you create an expression index or extended statistics.</li> </ul> And btw, do you know about TRUNCATE</code> - it's fast way to get rid of the data, but it leaves statistics behind... </div> How ANALYZE</code> Works</a> </h2> As we have already mentioned several times, ANALYZE</code> is the only mechanism that populates pg_class</code> and pg_statistic</code> with fresh data. Understanding what it samples, what it computes, and what it misses, is key to understanding why statistics are sometimes wrong. The process itself consists of 6 separate steps (as reported by pg_stat_progress_analyze</code>): initializing</li> acquiring sample rows</li> acquiring inherited sample rows (child/partitioned tables)</li> computing statistics (MCVs, histograms, correlation, etc.)</li> computing extended statistics (see below)</li> finalizing and writing to pg_statistic</code></li> </ol> For our purposes we will only cover sampling, computing statistics and writing to pg_statistic</code>. The sampling mechanism</a> </h3> ANALYZE actually doesn't read the entire table. It samples what is considered to be a statistically justified minimum sample size (defined as 300 for the reservoir sampling</a> algorithm). For PostgreSQL that means 300 × default_statistics_target</code> rows. SHOW default_statistics_target;</code></pre> default_statistics_target --------------------------- 100 (1 row)</code></pre> With the default target of 100, that's 30,000 rows. For our 100,000-row orders table, ANALYZE</code> reads about 30% of the data. For a 50-million-row table, it reads 0.06%. The same target also controls the size of the MCV list and histogram. Up to 100 entries each. With a target of 100, don't be surprised that histogram_bounds</code> contains 101 values. 100 buckets need 101 boundaries to close them.</div> The sampling is two-stage. First, ANALYZE</code> selects a random set of pages. Then it reads all live rows from those pages. This gives a representative cross-section without reading every page. Computing statistics</a> </h3> Once ANALYZE has its 30,000 sample rows (considering the default values), it processes each column independently. The pipeline for a single column looks roughly like this: First, it counts NULLs and calculates null_frac</code> and avg_width</code> - the cheapest statistics to compute. Then it sorts the non-null values and builds the MCV list by counting duplicates. Values that appear frequently enough make the cut; the rest are passed to the histogram builder, which divides them into equal-population buckets. Finally, because the values are already sorted, ANALYZE compares the logical sort order against the physical tuple positions (which page each row came from) to compute correlation</code>. The key detail here is that MCVs and histograms are not built from the same pool. Values that land in most_common_vals</code> are excluded from histogram_bounds</code>. This is why you'll sometimes see a column with MCVs but no histogram, or a histogram but no MCVs. They represent different slices of the same data. Writing to pg_statistic</a> </h3> Once all columns are processed, ANALYZE writes the results into pg_statistic</code> - one row per column. If a row for that column already exists, it's updated in place. This is a regular heap update, which means the old row becomes a dead tuple. On tables with many columns or frequent ANALYZE runs, this can cause pg_statistic</code> itself to bloat. After pg_statistic</code> is updated, ANALYZE refreshes relpages</code>, reltuples</code>, and relallvisible</code> in pg_class</code>. These values are recalculated from the sampling, not from a full table scan (they are estimates too). Controlling statistics quality</a> </h2> default_statistics_target</a> </h3> The default target of 100 works well for most columns. It means up to 100 MCVs, 101 histogram bounds, and a 30,000-row sample. Increasing it helps when: A column has many distinct values and the top 100 don't cover enough of the distribution</li> Range queries on skewed data produce bad estimates because histogram buckets are too coarse</li> Join estimates are off because n_distinct</code> is inaccurate</li> </ul> The cost scales linearly. Setting it to 1000 means 300,000 sampled rows, up to 1000 MCVs, more catalog storage, and slower planning from larger arrays to search. The maximum is 10,000. You don't have to raise it globally. For a single problematic column: ALTER TABLE orders ALTER COLUMN status SET STATISTICS 500; ANALYZE orders;</code></pre> Now status</code> gets up to 500 MCVs and 501 histogram bounds, while every other column stays at 100. Extended statistics</a> </h3> Standard statistics treat each column independently. This means the planner can't know that city = 'Edinburgh'</code> and country = 'UK'</code> are correlated. It multiplies their selectivities independently, potentially underestimating by orders of magnitude. Extended statistics solve this for specific column combinations: CREATE STATISTICS orders_status_date (dependencies, ndistinct, mcv) ON status, created_at FROM orders; ANALYZE orders;</code></pre> This tells ANALYZE to compute functional dependencies, combined distinct counts, and combined MCVs between the columns. The planner can then use these to avoid the independence assumption for queries filtering on both columns. The three types of extended statistics serve different purposes: dependencies capture functional dependencies between columns. Helps when knowing one column's value determines or narrows another's (e.g. zip_code</code> largely determines city</code>).</li> ndistinct tracks the number of distinct value combinations across columns. Helps with GROUP BY</code> on multiple columns where the planner would otherwise multiply distinct counts independently.</li> mcv builds a combined most-common-values list for column tuples. The most powerful but most expensive option. Helps with multi-column WHERE conditions on correlated values.</li> </ul> You can create extended statistics with any combination of these types. Start with dependencies</code> as it's cheapest, add mcv</code> when multi-column filter estimates are consistently wrong. Extended statistics are worth creating when you see EXPLAIN</code> estimates that are consistently wrong on multi-column filters, and the columns are logically correlated. They are computed during step 5 of the ANALYZE process and stored in pg_statistic_ext_data</code>. Diagnosing bad estimates</a> </h2> When a query is slow, the first question should always be: did the planner estimate correctly? Compare the estimate to reality with EXPLAIN ANALYZE</code>. Estimate off by handful of rows means statistics are fine. But when you see estimates off by 10x or more, that's where planning goes wrong. A nested loop that looks cheap for 100 rows becomes a disaster at 10,000. The statistics tell you what the planner believed, and comparing that with reality tells you what to do next. Either run ANALYZE</code>, consider tuning the statistics target for a specific column, or creating extended statistics if multiple columns are involved. The planner is only as good as what it reads from the catalog. When estimates go wrong, don't blame the planner. Check the data it's working with. Inside PostgreSQL's 8KB Page 2026-02-19T21:52:00+00:00 If you read previous post about buffers</a>, you already know PostgreSQL might not necessarily care about your rows. You might be inserting a user profile, or retrieving payment details, but all that Postgres works with are blocks of data. 8KB blocks, to be precise. You want to retrieve one tiny row? PostgreSQL hauls an entire 8,192-byte page off the disk just to give it to you. You update a single boolean flag? Same thing. The 8KB page is THE atomic unit of I/O. But knowing those pages exist isn't enough. To understand why the database behaves the way it does, you need to understand how it works. Every time you execute INSERT</code>, PostgreSQL needs to figure out how to fit it into one of those 8,192-byte pages. The buffer pool caches them, Write-Ahead Log (WAL) protects them, and VACUUM cleans them. The deep dive into the PostgreSQL storage internals starts by understanding what happens inside those 8KB pages. Pages that are used by PostgreSQL to organize all data - tables, indexes, sequences, TOAST relations. The 8KB</a> </h2> In case of Oracle, the default block size is set at database creation (DB_BLOCK_SIZE</code>), though tablespaces with non-standard block sizes can be created separately.</div>Before looking inside we actually need to discuss why 8KB in first place? The answer might be surprising - it's a setting that survived for over 40 years and nobody found a good reason to change it. Plus PostgreSQL isn't the only one who thinks that way. Oracle and SQL Server use the same exact number. The 8KB page size can be traced down to original Berkley POSTGRES project created in mid-1980s. In those times Unix systems typically used 4KB or 8KB virtual memory pages, and disk sectors were 512 bytes. Choosing 8KB meant a single database page mapped cleanly to OS memory pages and aligned well with filesystem I/O. And the math still works today. Modern Linux kernels manage memory in 4KB virtual memory pages. SSDs now use 4KB physical sectors instead of 512 bytes. The default filesystem block size on ext4 and XFS is 4KB. PostgreSQL's 8KB page still maps to two OS pages, two disk sectors, two filesystem blocks. The hardware changed underneath, but the alignment is still there. But the choice isn't just about hardware alignment. It's a tradeoff between two opposing forces. Make the page size too small and you going to increase the overhead (page metadata being good example). Make it too large and you waste space and increase I/O requirements when you need a single narrow row. The different situation is outside OLTP world. DuckDB, designed for analytical columnar workloads, uses 256KB blocks. When you're scanning millions of rows sequentially, the overhead-per-page penalty barely matters - you want big chunks to maximize throughput. Is PostgreSQL 8KB page size fixed? Technically no. PostgreSQL supports 1, 2, 4, 8, 16, or 32KB pages via --with-blocksize</code> at compile time. Some analytical workloads with wide rows benefit from 16KB or 32KB pages. But you'll need to rebuild everything from scratch, and unless you have a very specific reason and understand the downstream implications you almost certainly don't want to. The default works. </div> You can confirm the page size on any running instance. SHOW block_size; block_size ------------ 8192 (1 row)</code></pre>pageinspect</a> </h2> PostgreSQL comes with an extension called pageinspect</code> that lets you read raw page contents from SQL. It is part of the contrib modules and available on most installations. Let's enable it and create a small test table: CREATE EXTENSION IF NOT EXISTS pageinspect; CREATE TABLE page_demo ( id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY, title text NOT NULL, created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, value numeric(10,2) ); INSERT INTO page_demo (title, created_at, value) VALUES ('Ergonomic standing desk, oak finish', '2026-01-15 09:23:45+00', 549.99), ('Wireless mechanical keyboard', '2026-01-15 10:05:12+00', 189.00), ('Desk lamp', '2026-01-16 14:31:22+00', 45.00);</code></pre> We have three rows. PostgreSQL has written them into a single heap page. Let's look inside. Reading a raw page header</a> </h2> The page_header</code> function takes a raw page and returns the parsed header fields: SELECT * FROM page_header(get_raw_page('page_demo', 0));</code></pre>-[ RECORD 1 ]--------- lsn | F/1F9ED410 checksum | 0 flags | 0 lower | 36 upper | 7976 special | 8192 pagesize | 8192 version | 4 prune_xid | 0</code></pre> Every one of those fields lives in the first 24 bytes of the page. Together they form the page header, and they tell PostgreSQL everything it needs to know about the page before touching any actual data. The page header: 24 bytes of metadata</a> </h2> The first two fields are about safety, making sure the page survives crashes and silent corruption. pd_lsn (8 bytes) the Log Sequence Number of the last WAL record that modified this page. During crash recovery, PostgreSQL compares this LSN against the WAL stream. If the WAL record's LSN is less than or equal to the page's LSN, the page is already up to date and the record is skipped. If it is greater, the record must be replayed. This single field is what makes crash recovery work. pd_checksum (2 bytes) a checksum of the page contents. This is only active if the cluster was initialized with initdb --data-checksums</code> (or checksums were enabled later with pg_checksums</code>). When enabled, PostgreSQL verifies the checksum every time it reads a page from disk. A mismatch means silent data corruption, the kind that would otherwise go undetected until your data is wrong in ways nobody can explain. Are checksums enabled on your cluster? Before PostgreSQL 17, they were off by default. Check with SHOW data_checksums;</code>. If you see off</code> on a production database, that's worth fixing. You can enable them retroactively with pg_checksums</code>, though it requires a full shutdown. Magic of PD_ALL_VISIBLE</code> Indexes know what your data is, but not who is allowed to see it (in case a row was recently deleted or updated). Postgres has to do extra work to double-check the main table (the "heap") to ensure a row is visible. When a page is marked PD_ALL_VISIBLE, it guarantees every row on that page is old enough to be seen by everyone. This lets Postgres skip the expensive heap check entirely and serve data straight from the index. A speed boost you may know as an Index-Only Scan. </div> The next fields are the page's spatial ma. They tell PostgreSQL where things are and where there's room for more. pd_flags (2 bytes) bit flags that describe the page state. The important ones are: PD_HAS_FREE_LINES</code> (are there any unused line pointers?), PD_PAGE_FULL</code> (not enough free space for new tuple) and PD_ALL_VISIBLE</code> (all tuples on page are visible to everyone). pd_lower and pd_upper (2 bytes each) define the free space gap. pd_lower</code> marks where the line pointer array ends -- in our output it is 36, that is 24 bytes of header plus 3 line pointers at 4 bytes each: 24 + (3 x 4) = 36. pd_upper</code> marks where tuple data begins -- our value is 7976, meaning the three tuples together occupy bytes 7976 through 8191. Everything between these two offsets is free space. As you insert rows, these two values creep toward each other until there's no room left. pd_special (2 bytes) the byte offset to the "special space" at the end of the page. For heap (table) pages, this equals the page size (8192), meaning there is no special space. For index pages, this area contains index-specific metadata. pd_pagesize_version (2 bytes) encodes both the page size and the layout version number. The version is currently 4 for all modern PostgreSQL releases. Finally, one field dedicated to cleanup. The Mini-Cleanup (Page-Level Pruning) Instead of waiting for a heavy VACUUM, Postgres can do garbage collection on the fly. When a normal query reads a page, it checks `pd_prune_xid`. If that transaction ID is older than all currently running transactions, the query instantly reclaims the dead space itself. This is one of the cases where SELECT might modify data pages. </div> pd_prune_xid (4 bytes) the oldest transaction ID whose dead tuples have not yet been pruned from this page. When a new transaction accesses the page and finds that prune_xid</code> is older than the global horizon, it triggers page-level pruning. A lightweight cleanup that reclaims space without a full VACUUM. What is a line pointer?</a> </h2> If you're paying attention, we just mentioned line pointers in pd_lower</code> description. If the page header discussed above is the general metadata, the line pointers (internally represented as ItemIdData</code>) are the page table of contents. Every time you insert a row, PostgreSQL doesn't just drop the raw data into the page. It actually splits the job into two steps. It puts the bulky, unpredictable row data (the tuple) at the very bottom of the page (more on this in the next section).</li> It adds a tiny, fixed 4-byte line pointer right after the page header.</li> </ol> This pointer acts as a direct map. It holds exact byte offset and length of the tuple it points to. It allows PostgreSQL to only look at the offset and read the required number of bytes to get the tuple. The line pointer array starts immediately after the 24-byte header and grows downward. Let's look at ours. SELECT lp, lp_off, lp_flags, lp_len FROM heap_page_items(get_raw_page('page_demo', 0));</code></pre> lp | lp_off | lp_flags | lp_len ----+--------+----------+-------- 1 | 8112 | 1 | 79 2 | 8032 | 1 | 77 3 | 7976 | 1 | 53</code></pre> Each line pointer is 4 bytes and contains three pieces of information: lp the ordinal position in the array (1-based). This, combined with the page number, forms the ctid - the physical address of a tuple. Row 1 on page 0 has ctid (0,1)</code>.</li> lp_off the byte offset within the page where the actual tuple data begins. Notice the offsets decrease: 8112, 8032, 7976. Tuples are packed from the bottom up.</li> </ul> Line pointers are the reason PostgreSQL can move tuples around within a page (during defragmentation) or perform HOT (Heap-Only Tuple) updates without invalidating index entries. An index stores a ctid like (0, 2)</code>, which means "page 0, line pointer 2". Because the index points to the pointer rather than the physical byte offset, Postgres can shuffle data or redirect pointers under the hood while the index reference remains perfectly valid. </div> lp_flags the state of the pointer. The values are: 0 = LP_UNUSED</code> (available for reuse), 1 = LP_NORMAL</code> (points to a live tuple), 2 = LP_REDIRECT</code> (points to another line pointer, used after HOT pruning), 3 = LP_DEAD</code> (the tuple has been determined dead but the pointer persists until cleanup).</li> lp_len the total length of the tuple in bytes, including the 23-byte tuple header, null bitmap, alignment padding, and actual column data.</li> </ul> The anatomy of a page</a> </h2> Here is how all these pieces fit together inside the 8KB block. We already mentioned it above, but if you review the image carefully you can confirm the important insight. While line pointers are allocated downward from the header (pd_lower increases) the tuple data is stored upwards from the bottom of the page (pd_upper decreases). Free space is the gap between them. This is called Slotted page layout and it's easy way how to prevent unnecessary data fragmentation. By storing predictable line pointers at top, they don't mix together with bulky and unpredicable tuples. </div> </div> </div> </div> </div> Interactive Page Visualizer Watch the opposing growth directions in action. Insert rows and click regions for byte-level details. Open Visualizer</a> </div> </div> Free space: the gap in the middle</a> </h2> The free space on a page is the region between pd_lower and pd_upper. For our page: SELECT lower, upper, upper - lower AS free_space FROM page_header(get_raw_page('page_demo', 0));</code></pre> lower | upper | free_space -------+-------+------------ 36 | 7976 | 7940 (1 row)</code></pre> With only three small rows, we have 7,940 bytes of free space. Almost the entire page is still available. Let's see what happens when we add more data: INSERT INTO page_demo (title, created_at, value) SELECT 'Generated item ' || i, '2026-01-15 00:00:00+00'::timestamptz + (i || ' hours')::interval, (i * 11.11)::numeric(10,2) FROM generate_series(1, 10) AS i; SELECT lower, upper, upper - lower AS free_space FROM page_header(get_raw_page('page_demo', 0));</code></pre> lower | upper | free_space -------+-------+------------ 76 | 7336 | 7260 (1 row)</code></pre> You can observe pd_lower</code> moved from 36 to 76. Why? With 10 new line pointers at 4 bytes each we moved by 40 bytes.</li> pd_upper</code> dropped from 7,976 to 7,336. The 10 new tuples consumed 640 bytes of the data space.</li> And the free space shrank to 7,260 bytes.</li> </ul> How fast does a page fill up?</a> </h2> If we going to think about the page capacity, let's consider each row in this page costs: 4 bytes for the line pointer</li> 23 bytes for the tuple header (transaction metadata, null bitmap offset, info mask)</li> Alignment padding to the nearest 8-byte boundary after the header (1 byte of padding, bringing the header to 24 bytes)</li> Actual column data: 4 bytes for the integer, variable bytes for the text, variable bytes for the numeric</li> </ul> For our new schema, each tuple is a bit larger than before. We saw earlier that our first 3 tuples took up exactly 216 bytes, which averages exactly 72 bytes per tuple. Add the 4-byte line pointer, and each row costs about 76 bytes of page space. With 8,192 bytes in a page minus the 24-byte header, we have 8,168 bytes of usable space. At roughly 76 bytes per row, we can theoretically fit approximately 107 rows on a single page. Let's test it. We already have 13 rows (3 original plus 10 generated). Let's insert more and check: INSERT INTO page_demo (title, created_at, value) SELECT 'Generated item ' || i, '2026-01-15 00:00:00+00'::timestamptz + (i || ' hours')::interval, (i * 11.11)::numeric(10,2) FROM generate_series(11, 150) AS i; SELECT count(*) AS tuples_on_page_0 FROM heap_page_items(get_raw_page('page_demo', 0));</code></pre> Total of 119 tuples fit on page 0 before PostgreSQL had to start using page 1. A bit more than our estimate of 107 -- the shorter generated titles take less space than "Ergonomic standing desk, oak finish". The exact number always depends on column values, but this gives you a practical sense of capacity. Because this schema includes a variable-length text column, the row size flexes, but we still hit a very predictable ceiling right around 110-120 rows per 8KB block. If you want to estimate how many pages a table will need, the rough formula is: rows * average_row_size / 8192</code>. But remember that "average_row_size" includes the 23-byte tuple header, alignment padding, and the 4-byte line pointer. PostgreSQL's pg_column_size()</code> function can help you measure actual row sizes. </div> Spanning multiple pages</a> </h2> With 160 rows in our table, we have started storing the tuples into a second page. Let's verify using the system catalog: SELECT relpages, reltuples FROM pg_class WHERE relname = 'page_demo';</code></pre>The relpages</code> and reltuples</code> values are estimates updated by ANALYZE and autovacuum. After bulk inserts, run ANALYZE page_demo;</code> to refresh them.</div> relpages | reltuples ----------+----------- 2 | 160 (1 row)</code></pre> Two pages. Let's insert substantially more data to see a real multi-page table: INSERT INTO page_demo (title, created_at, value) SELECT 'Generated row ' || i, CURRENT_TIMESTAMP + (i || ' minutes')::interval, (random() * 1000)::numeric(10,2) FROM generate_series(1, 500) AS i; ANALYZE page_demo; SELECT relpages, reltuples FROM pg_class WHERE relname = 'page_demo';</code></pre> relpages | reltuples ----------+----------- 6 | 660 (1 row)</code></pre> Six pages now. Each page is an independent 8KB block with its own header. Let's confirm by reading the headers of the first two: SELECT 0 AS page, lower, upper, upper - lower AS free_space FROM page_header(get_raw_page('page_demo', 0)) UNION ALL SELECT 1, lower, upper, upper - lower FROM page_header(get_raw_page('page_demo', 1));</code></pre> page | lower | upper | free_space ------+-------+-------+------------ 0 | 500 | 552 | 52 1 | 504 | 512 | 8 (2 rows)</code></pre> Both pages are nearly full, with only 52/8 bytes of free space remaining. Which is too little for another row to fit there. The space that isn't there</a> </h2> You may have noticed that pd_special</code> equals 8192 for our heap pages. The same size as the entire page size. I.e. there's no special space allocated. This applies to the heap pages. But not all pages are heap pages. Index pages use the special space at the end of the page to store index-specific metadata. For a B-tree index, it includes pointers to sibling pages (for range scans), the tree level, and flags about the page type (leaf, internal, root, deleted). Let's peek at the primary key index that was automatically created for our table: SELECT type, live_items, dead_items, avg_item_size, page_size, free_size FROM bt_page_stats('page_demo_pkey', 1);</code></pre> type | live_items | dead_items | avg_item_size | page_size | free_size ------+------------+------------+---------------+-----------+----------- l | 367 | 0 | 16 | 8192 | 808 (1 row)</code></pre> The type</code> is l</code> for leaf page. And if we check its page header: SELECT special FROM page_header(get_raw_page('page_demo_pkey', 1));</code></pre> special --------- 8176 (1 row)</code></pre> The special space starts at byte 8176, giving us 16 bytes (8192 - 8176) of B-tree metadata at the end of the page. Heap pages have none; index pages rely on it. Putting it all together</a> </h2> Let's close with a complete view of our page. We know the header, the line pointers, the free space, and the tuple data regions. Here is a summary query that shows all of it at once: SELECT 'header' AS region, 0 AS start_byte, 23 AS end_byte, 24 AS size_bytes UNION ALL SELECT 'line pointers', 24, lower - 1, lower - 24 FROM page_header(get_raw_page('page_demo', 0)) UNION ALL SELECT 'free space', lower, upper - 1, upper - lower FROM page_header(get_raw_page('page_demo', 0)) UNION ALL SELECT 'tuple data', upper, special - 1, special - upper FROM page_header(get_raw_page('page_demo', 0)) UNION ALL SELECT 'special space', special, 8191, 8192 - special FROM page_header(get_raw_page('page_demo', 0));</code></pre> region | start_byte | end_byte | size_bytes ---------------+------------+----------+------------ header | 0 | 23 | 24 line pointers | 24 | 499 | 476 free space | 500 | 551 | 52 tuple data | 552 | 8191 | 7640 special space | 8192 | 8191 | 0 (5 rows)</code></pre> 476 bytes of line pointers means 119 entries (476 / 4). 7,640 bytes of tuple data. 52 bytes of free space. And zero bytes of special space (it's heap page). That is the entire PostgreSQL 8KB page. Twenty-four bytes of header tell PostgreSQL where everything is. Line pointers provide table of contents. Reading Buffer statistics in EXPLAIN output 2026-02-06T15:52:00+00:00 In the article about Buffers in PostgreSQL</a> we kept adding EXPLAIN (ANALYZE, BUFFERS)</code> to every query without giving much thought to the output. Time to fix that. PostgreSQL breaks down buffer usage for each plan node, and once you learn to read those numbers, you'll know exactly where your query spent time waiting for I/O - and where it didn't have to. That's about as fundamental as it gets when diagnosing performance problems. PostgreSQL 18: BUFFERS by Default Starting with PostgreSQL 18, EXPLAIN ANALYZE</code> automatically includes buffer statistics - you no longer need to explicitly add BUFFERS</code>. The examples below use the explicit syntax for compatibility with older versions, but on PG18+ a simple EXPLAIN ANALYZE</code> gives you the same information. </div> A complete example</a> </h2> For this article we will use following schema and seeded data. CREATE TABLE customers ( id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY, name text NOT NULL ); CREATE TABLE orders ( id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY, customer_id integer NOT NULL REFERENCES customers(id), amount numeric(10,2) NOT NULL, status text NOT NULL DEFAULT 'pending', note text, created_at date NOT NULL DEFAULT CURRENT_DATE ); INSERT INTO customers (name) SELECT 'Customer ' || i FROM generate_series(1, 2000) AS i; -- seed data: ~100,000 orders spread across 2022-2025 INSERT INTO orders (customer_id, amount, status, note, created_at) SELECT (random() * 1999 + 1)::int, (random() * 500 + 5)::numeric(10,2), (ARRAY['pending','shipped','delivered','cancelled'])[floor(random()*4+1)::int], CASE WHEN random() < 0.3 THEN 'Some note text here for padding' ELSE NULL END, '2022-01-01'::date + (random() * 1095)::int -- ~3 years of data FROM generate_series(1, 100000); -- make sure stats are up to date ANALYZE customers; ANALYZE orders; -- we are going to skip indexes on purpose -- and fire sample query select count(1) from customers;</code></pre> Let's start with a random query EXPLAIN (ANALYZE, BUFFERS) SELECT o.*, c.name FROM orders o JOIN customers c ON o.customer_id = c.id WHERE o.created_at > '2024-01-01';</code></pre> and its output. QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------- Hash Join (cost=58.00..2253.87 rows=33784 width=71) (actual time=0.835..26.695 rows=33239.00 loops=1) Hash Cond: (o.customer_id = c.id) Buffers: shared hit=13 read=857 -> Seq Scan on orders o (cost=0.00..2107.00 rows=33784 width=58) (actual time=0.108..18.106 rows=33239.00 loops=1) Filter: (created_at > '2024-01-01'::date) Rows Removed by Filter: 66761 Buffers: shared read=857 -> Hash (cost=33.00..33.00 rows=2000 width=17) (actual time=0.697..0.698 rows=2000.00 loops=1) Buckets: 2048 Batches: 1 Memory Usage: 118kB Buffers: shared hit=13 -> Seq Scan on customers c (cost=0.00..33.00 rows=2000 width=17) (actual time=0.007..0.231 rows=2000.00 loops=1) Buffers: shared hit=13 Planning: Buffers: shared hit=130 read=29 dirtied=3 Planning Time: 1.585 ms Execution Time: 28.067 ms (16 rows)</code></pre> As that's quite a bit of information, let's break it down by individual categories. Shared buffers: hit, read, dirtied & written</a> </h3> As described in previous article</a> these are most common buffers statistics you will see. shared hit is number of pages found in shared buffers (i.e. cached). This is fast path where no disk I/O is required. Higher is better for performance. shared read identifies number of pages not in shared buffers, fetched from disk (or OS cache), and each one of them adds potential I/O latency. If you see a SELECT that shows `dirtied` pages. It's not a bug. PostgreSQL sets hint bits and prunes HOT chains during reads - the first backend to read a page after writes will dirty it. Normal behavior, not a problem.</div>**shared dirtied** has number of pages modified by this query. The query changed the data that was already cached (in the buffer pool) and those pages will eventually need to be written to the disk. shared written is a number of pages written to disk during query execution. To remind us, this happens when query needs buffer space but needs to evict dirty pages synchronously. If you see this repeatedly during a SELECT it might be a warning sign - your background writer is not keeping up as it should. Let's have a look at our query's top-level buffer stats: Buffers: shared hit=13 read=857</code></pre> Only 13 pages were cached in shared buffers, while 857 had to be fetched from disk (or OS cache). No pages were dirtied or written - expected for a pure SELECT with no side effects. But where did those 13 hits come from? The breakdown by node tells us: -> Seq Scan on orders o Buffers: shared read=857 -> Seq Scan on customers c Buffers: shared hit=13</code></pre> The customers table (in this case small - 2,000 rows, 13 pages) was fully cached - likely accessed frequently or as it's in our case accessed recently. The orders table (100,000 rows, 857 pages) had zero hits - every single page required I/O. This is typical after a restart or when scanning a table that doesn't fit comfortably in shared buffers. Interpreting the ratio</a> </h3> In the context of this article we're going to consider ratio between shared hit and total buffers processed. Is there a perfect ratio you should strive for? As we will demonstrate there's no such universal value. Let's calculate it for our query: hit_ratio = shared hit / (shared hit + shared read) = 13 / (13 + 857) = 1.5%</code></pre>In an OLTP workload, the same small set of rows gets accessed over and over - fetch a customer by ID, look up an order by reference number, check inventory for a product. The working set is a small fraction of the total database. These queries touch a handful of pages each, and those pages stay hot in shared buffers because they keep getting requested. A well-tuned OLTP system naturally converges toward high hit ratios - not because someone set a target, but because the access pattern keeps the relevant data cached.</div>Honestly, that looks terrible. If this were the ratio of most of your OLTP queries you can easily say - there's a problem. But this was run on a freshly loaded dataset with cold caches - every page of the orders table had to be fetched for the first time. Run the same query again and you'll likely see most of those 857 reads become hits as shared buffers and the OS page cache warm up. On a test environment (where nothing else runs you will most likely hit the 100%). What matters is the hit ratio per query, tracked over time, compared to its own baseline: A reporting query scanning a large date range might consistently show 10-30% hit ratio. That's fine - it's expected to touch cold data.</li> A query serving your login page should be near 100%. If it drops to 80%, something changed - maybe the table grew, an index was rebuilt, or shared_buffers is under pressure from a new workload.</li> A query that ran at 95% hit ratio last week and now runs at 40% deserves investigation, regardless of whether 40% sounds "good" or "bad" in isolation.</li> </ul> The ratio is a diagnostic tool, not a scorecard. Use it to spot regressions, compare before-and-after when tuning, and understand where your query's time is actually going. A low ratio paired with high execution time points you toward I/O as the bottleneck. A high ratio with high execution time tells you to look elsewhere - maybe CPU, maybe row count, maybe a bad plan. Context matters more than absolute numbers. Compare similar queries over time, not arbitrary benchmarks. Local buffers</a> </h2> Local buffers track I/O for temporary tables. Unlike regular tables that live in shared buffers, temp tables use per-backend memory - each connection gets its own local buffer pool, controlled by the temp_buffers</code> setting. CREATE TEMP TABLE temp_large_orders AS SELECT o.id, o.amount, o.status, o.created_at, c.name AS customer_name FROM orders o JOIN customers c ON o.customer_id = c.id WHERE o.amount > 200; EXPLAIN (ANALYZE, BUFFERS) SELECT status, count(*), sum(amount) FROM temp_large_orders GROUP BY status;</code></pre> First thing you should notice is there's no shared buffers at all, at least in execution phase. The entire query ran against local buffers because temp tables are invisible to other backends. QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------- HashAggregate (cost=1281.60..1284.10 rows=200 width=72) (actual time=24.659..24.661 rows=4.00 loops=1) Group Key: status Batches: 1 Memory Usage: 32kB Buffers: local hit=576 -> Seq Scan on temp_large_orders (cost=0.00..979.20 rows=40320 width=48) (actual time=0.009..5.965 rows=60731.00 loops=1) Buffers: local hit=576 Planning: Buffers: shared hit=36 read=5 Planning Time: 0.294 ms Execution Time: 24.708 ms (10 rows)</code></pre> The individual values that might be reported are local hit/read with the same concept as shared, but for temp tables in the per-backend buffer pool. Another value is local dirtied/written representing temp table modifications. "Dirtied" means the query modified pages in the local buffer pool. "Written" means dirty pages had to be flushed to disk to make room for new ones - the same clock-sweep eviction mechanism as shared buffers, but against the local buffer pool. Unlike shared buffers, temp table writes don't generate WAL and aren't subject to checkpointing. In practice, local written</code> is rare to see - PostgreSQL handles temp table overflow efficiently enough that you're unlikely to encounter it unless your temp_buffers</code> is severely undersized relative to your temp table workload. Temp buffers: when work_mem isn't enough</a> </h2> While local buffers are not that often considered a problem, or visible, temp buffers track cases where operations spill from memory to disk for sorts, hashes, and other operations that exceed current work_mem</code> settings. SET work_mem = '256kB'; EXPLAIN (ANALYZE, BUFFERS) SELECT o.id, o.amount, o.status, o.created_at, c.name FROM orders o JOIN customers c ON o.customer_id = c.id ORDER BY o.amount DESC;</code></pre> We explictely forced lower work_mem</code> to see the impact. QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------------- Sort (cost=38374.70..38874.70 rows=200000 width=36) (actual time=109.345..120.574 rows=200000.00 loops=1) Sort Key: o.amount DESC Sort Method: external merge Disk: 9736kB Buffers: shared hit=1738, temp read=3636 written=3722 -> Hash Join (cost=116.00..4353.56 rows=200000 width=36) (actual time=1.597..34.857 rows=200000.00 loops=1) Hash Cond: (o.customer_id = c.id) Buffers: shared hit=1738 -> Seq Scan on orders o (cost=0.00..3712.00 rows=200000 width=27) (actual time=0.016..6.973 rows=200000.00 loops=1) Buffers: shared hit=1712 -> Hash (cost=66.00..66.00 rows=4000 width=17) (actual time=1.568..1.569 rows=4000.00 loops=1) Buckets: 4096 Batches: 1 Memory Usage: 235kB Buffers: shared hit=26 -> Seq Scan on customers c (cost=0.00..66.00 rows=4000 width=17) (actual time=0.012..0.629 rows=4000.00 loops=1) Buffers: shared hit=26 Planning: Buffers: shared hit=15 Planning Time: 1.184 ms Execution Time: 123.932 ms</code></pre> There you can see the temp read/written with number of pages read from and written to temporary files on disk. This indicates the operation couldn't fit in memory. Naming Confusion Alert temp read/written</code> in EXPLAIN has nothing to do with the temp_buffers</code> parameter. temp_buffers</code> = memory for temporary tables (CREATE TEMP TABLE</code>)</li> temp read/written</code> = disk spill from sorts/hashes (governed by work_mem</code>)</li> </ul> </div> The Sort Method: external merge Disk: 9736kB</code> confirms it - sorting 200,000 rows with only 256kB of work_mem</code> forced PostgreSQL to spill ~9.7MB to temporary files on disk. The temp written=3722</code> happened during the sort phase as pages were flushed out, and temp read=3636</code> happened during the merge phase as PostgreSQL read them back to produce the final sorted result. Notice something else: the Hash Join and everything below it shows only shared hit=1738</code> with no temp buffers at all. The hash table for 4,000 customers fit comfortably in 235kB of memory. The temp spill is isolated to the Sort node - buffer stats always attribute I/O to the node that caused it. Try bumping work_mem</code> to something reasonable and the spill disappears: SET work_mem = '16MB';</code></pre> You should see no temp</code> buffers at all. The sort completed in memory, execution time dropped, and the only I/O was reading the actual table data. To reduce temp file usage you can: Increase work_mem</code> (careful there, don't forget it's per-operation setting, not per-query, so a complex query with multiple sorts or hash joins allocates work_mem</code> for each one)</li> Optimize the query to process fewer rows before the sort</li> And probably most importantly, consider adding indexes to avoid sorts entirely - an index on orders(amount DESC)</code> would eliminate the sort node altogether</li> </ul> Planning buffers</a> </h2> Up until now we have avoided planning buffers completely. It's an addition that started with PostgreSQL 13, allowing you to see the buffer usage during the query planning, separate from the execution: Planning: Buffers: shared hit=36 read=5</code></pre> Why does planning need buffers? The planner reads system catalogs (pg_class</code>, pg_statistic</code>, pg_index</code>, etc.) to understand table structures and statistics. Complex queries touching many tables can have a non-trivial impact on planning-time I/O. High read</code> count in planning phase suggests either system catalogues aren't cached (cold start most likely), or your query is touching many tables or columns. If planning time is a problem, ensure system catalogs stay hot. On systems with many partitions, planning overhead can become significant - this is one reason partition pruning matters. The blurry line between planning and execution buffers</a> </h3> While writing the articles for this blog I often face the doubts whatever I have everything correct. Because with a complex system like PostgreSQL one is always learning. Recently learned more about planning buffers based on my somewhat technically imprecise assumption. PostgreSQL actually does not resolve all metadata during the planning phase. The planner does the minimum work needed to choose the best plan, but defers some catalog lookups to execution time. When a Sort node first runs, it looks up the comparison function from pg_amproc</code> via get_opfamily_proc()</code>. That lookup hits shared buffers and gets counted as execution buffers. On the second run in the same session, the syscache already has that information - no buffer access, fewer reported buffers. Putting it together</a> </h2> Here's a sample output of a query with problems across every buffer category: Buffers: shared hit=50 read=15000 written=847 temp read=2500 written=2500 Planning: Buffers: shared hit=12 read=156 Planning Time: 45.678 ms Execution Time: 12345.678 ms</code></pre> Reading this top to bottom: the hit ratio is abysmal (50 hits vs 15,000 reads), so the working set isn't cached. The written=847</code> means the query forced synchronous evictions - the background writer can't keep up. The temp spill points to an operation exceeding work_mem</code>. Even planning needed 156 reads, suggesting system catalogs got evicted from cache. Each number points to a specific tuning lever: shared_buffers</code>, bgwriter_lru_maxpages</code>, work_mem</code>, or query optimization to touch less data. Looking beyond single queries</a> </h2> Single query analysis is useful, but patterns across your workload matter more. pg_stat_statements</code> exposes the same buffer counters aggregated over time: SELECT substring(query, 1, 60) AS query, calls, shared_blks_hit, shared_blks_read, round(100.0 * shared_blks_hit / nullif(shared_blks_hit + shared_blks_read, 0), 2) AS hit_pct, temp_blks_written FROM pg_stat_statements WHERE calls > 100 ORDER BY shared_blks_read DESC LIMIT 10;</code></pre> This shows which queries are causing the most disk reads across your system - often more actionable than analyzing one query at a time. Conclusion</a> </h2> Buffer statistics transform EXPLAIN from "here's the plan" to "here's exactly where the time went." Every number points to a specific cause and a specific fix. Once you start reading them, you stop guessing and start tuning. If you need to get bigger picture on buffer management, check out Introduction to Buffers in PostgreSQL</a>. Introduction to Buffers in PostgreSQL 2026-01-24T16:57:00+00:00 The work around RegreSQL</a> led me to focus a lot on buffers. If you are a casual PostgreSQL user, you have probably heard about adjusting shared_buffers</code> and followed the good old advice to set it to 1/4 of available RAM. But after we went a little bit too enthusiastic about them on a recent Postgres FM episode</a> I've been asked what that's all about. Buffers are one of those topics that easily gets forgotten. And while they are a foundation block of PostgreSQL's performance architecture, most of us treat them as a black box. This article is going to attempt to change that. The 8KB page</a> </h2> There's one concept we need to cover before diving into the buffers. And that's the concept of the 8KB page. Everything in PostgreSQL is stored in blocks that are 8KB wide. When PostgreSQL reads the data, it does not read individual rows. It reads the entire page. When it writes something, same thing - same page. You want to retrieve one small row, you will always retrieve much more data to go along with it. And if you followed carefully, same applies to writes. -- you can check the block size (which should be almost always 8192 bytes) show block_size; block_size ------------ 8192 (1 row)</code></pre> Every table and index is a collection of these pages. A row might span multiple pages if it's large enough, but the page remains the atomic unit of I/O. Want to look inside? If you want to see exactly how PostgreSQL packs your data, headers, and line pointers into these 8,192 bytes, check out the byte-level tour in Inside PostgreSQL's 8KB Page</a>. </div> PostgreSQL vs OS</a> </h2> The interesting part is understanding why PostgreSQL needs to maintain its own infrastructure for its own buffer cache, when the operating system can already cache disk pages. The answer is quite simple. PostgreSQL understands the data it reads. Whilst the operating system only sees files and bytes. PostgreSQL sees tables, indexes, query plans and has semantic knowledge to cache things faster. Consider this example: a query needs to perform a sequential scan of a large table. The OS might happily cache all those pages, but PostgreSQL knows this is a one-time operation and uses a special strategy (ring buffers) to avoid eviction of the main cache. The second important aspect is ACID - or better, the guarantee that write-ahead log (WAL) reaches stable storage before the data page is modified. The OS does not differentiate and can't effectively guarantee this durability requirement (only at the cost of performance impact). shared_buffers</a> </h2> Now we can move to what we all know is the main PostgreSQL "cache". The shared_buffers</code> parameter controls the size of the shared memory, accessible to all backend processes. If any backend needs to retrieve a page, it first checks the shared buffers. If the page is present - it's a hit. No disk I/O is needed. Miss? Read it from disk (or OS cache) and store it in shared buffers for next time. WAL has its own buffer area (wal_buffers). This separation exists because WAL writes are sequential and must be persisted before the corresponding data change can be considered committed. The default is 3% of shared_buffers, capped at 16MB (one WAL segment).</div> show shared_buffers; shared_buffers ---------------- 128MB (1 row)</code></pre> The default value 128MB is very conservative and is part of making sure the PostgreSQL default installation will work pretty much on any system (including those with limited RAM). But in your regular environment this value should typically be much higher. The value of 128MB there refers to the actual buffered content. If you consider a single page being 8KB, you can imagine this as 16,384 individual slots for storing the data. </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> Interactive Visualizer Explore how PostgreSQL's buffer pool actually works. Watch the clock sweep algorithm evict cold pages, see buffers transition between clean, dirty, and pinned states, and understand how the hash table enables O(1) lookups. Open Visualizer</a> </div> </div> However, there's more to the buffer pool than just the pages themselves. PostgreSQL needs to track metadata and provide fast lookups, so the shared memory area is organized into three components: buffer blocks - the actual 8KB pages where the data lives</li> buffer descriptors - a parallel array of ~64-byte structures, one per slot.</li> hash table used for mapping page identifiers to individual buffer slots.</li> </ul> Each descriptor then tracks which page is cached in the slot (tag), flags about the state (dirty, valid and I/O in-progress), and pin/usage counters. O(1) = constant time, regardless of buffer pool size.</div>The hash table enables fast lookups. When a backend needs a specific page, it hashes the page identifier and jumps directly to the right bucket—no need to scan all 16,384 slots. This keeps buffer lookups at O(1) regardless of pool size. When a backend needs page N of table orders</code>, it hashes the identifier, looks up the hash table - which drives the hit/miss logic. Pin and usage counts</a> </h2> What happens to each buffer slot is ruled by the two counters mentioned above: pin and usage count. Pin count tracks active references. When a backend (for example a running query) is actively reading or modifying a page, it pins the buffer so it can't be evicted. When the backend finishes, it unpins the buffer. Usage count tracks how recently and frequently a buffer was accessed. Each access increments the count (capped at 5). During eviction, the clock sweep decrements this value—buffers with higher counts survive longer, while those at 0 get evicted. The usage counter is important to avoid behaviour when a single sequential scan would flush the entire buffer pool. Imagine reading a full 1GB table. Without this protection, it would evict everything in shared buffers, ignoring frequently accessed data. We will also touch on this specific behaviour later, in the ring buffers. PostgreSQL provides you functionality to examine what's happening in the shared buffer cache in real time using extension pg_buffercache</a>. The clock sweep algorithm</a> </h2> Now that we covered tracking of the usage, what happens when PostgreSQL needs to load a page and all slots are taken? It needs to try to make some space. A simple LRU check would represent much higher maintenance; updating the linked list on each buffer load would dramatically increase the complexity.</div>That's where the clock sweep algorithm comes in. Why clock? Imagine the buffer pool as a circular clock, and the algorithm always moves forward, sweeping along the way. As it passes each slot it Is the buffer pinned? Skips it as it's being used.</li> Is the usage counter > 0, reduce it by one and continue to next slot (i.e. it survives till next round).</li> If the usage counter hits 0 and it's not in use, it's going to be evicted.</li> </ol> The clock sweep is a simple way to ensure cold pages get evicted quickly, while hot pages tend to stick and survive multiple sweeps. Dirty buffers and the background writer</a> </h2> Don't forget that a change is always first written to WAL.</div>Up until now we have discussed that a buffer loaded from disk is the same one in the shared buffers cache. When a backend modifies the page, the buffer becomes "dirty" but it's not immediately written to storage. Dirty buffers represent I/O work that has not yet happened. As it would be inefficient to write it immediately. The same page can be modified multiple times in short bursts, making the previous I/O operations unnecessary. Instead dirty pages accumulate until one of the following events happen. PostgreSQL writes all dirty buffers to disk during checkpoint. This is a periodic process (which you can force using the CHECKPOINT</code> command). This is a point where data on disk becomes consistent. After its successful completion, crash recovery only needs to replay write-ahead logs from that moment forward. The second mechanism is the background writer. It continuously scans for dirty buffers and writes them before anyone needs to evict them. This way, when a backend runs clock sweep, it finds clean buffers ready to evict, without needing to wait for I/O. It also spreads writes over time instead of bursty spikes during eviction pressure. And as suggested the last chance to write the dirty pages to the disk is when the clock sweep finds a dirty buffer. It must write the data to disk so it can be reused for the new page. This is the worst case as it resorts to synchronous I/O operation. The ultimate goal is to balance the clean buffers available, and therefore preventing backends from being blocked by synchronous write during the eviction. If you followed carefully you can see how bad flow or settings can cause a problem. Eviction happens when new buffers are loaded (I/O operation), and if the dirty buffer is evicted you force another I/O operation. Ring buffers</a> </h2> As mentioned above there are also special types of buffers. We already touched the scenario when a query performs the scans of a large table. In a naive implementation, the sequential scan would load all data into shared buffers, evicting everything else along the way. Your warmed cache would vanish and subsequent queries would suffer for minutes to come. PostgreSQL's solution for this case is ring buffers. Small, private buffer pools for bulk operations. Instead of using the shared buffer pool, certain operations get their own limited ring. The individual cases are: Sequential scans on large tables over 1/4 of shared_buffers</code> use a dedicated 256 KB ring buffer. Pages cycle through this tiny ring and never touch the main cache. -- perform a sequential scan over a large table EXPLAIN (ANALYZE, BUFFERS) SELECT count(1) FROM ring_buffer_test;</code></pre> Which would show you Buffers: shared read=127285</code> instead of hit. I will follow with the separate article on how to read buffers in explain plans. Bulk writes (COPY, CREATE TABLE AS) use a 16MB capped ring buffer, large enough for efficient batching, yet small enough not to pollute the shared buffer pool. As VACUUM touches every page and shouldn't evict the hot data, it uses its own dedicated ring buffer. Historically it was set to 256KB, but since PostgreSQL 17 it can be configured using vacuum_buffer_usage_limit</code> (while previous two are given). Local buffers</a> </h2> The second kind of exception to shared buffer pool, is the session based temporary tables. In this case as the concurrency is out of question, each backend has its own local buffer pool, controlled by temp_buffers</code> (8MB default). Local buffers are faster than shared buffers because they have simpler locking. There is no need for the heavy cross-process coordination required in the main cache. While this might seem like a minor implementation detail, it can translate into a powerful optimisation strategy. Many developers tend to default to complex CTE logic</a> for intermediate data, but using temporary tables offers a distinct advantage in the form of lower I/O as the changes to temporary tables are not WAL logged and by their nature reduce pollution of shared buffer pools. If your workload involves large temporary tables, increasing temp_buffers can help keep those operations purely in RAM. However, remember that this memory is per-connection, so it multiplies across all backends. The OS Page Cache</a> </h2> PostgreSQL doesn't bypass the operating system. Every read and write goes through the kernel, which maintains its own page cache. This creates double buffering - the same 8KB page can exist in both PostgreSQL's shared buffers and the OS cache simultaneously. The OS cache acts as a "Level 2 cache" - when PostgreSQL evicts a page, it often still lives in OS memory.</div> This sounds wasteful, but it's actually a feature. When PostgreSQL evicts a clean page to make room, that page drops into the OS cache rather than disappearing. A moment later, if you need it back, the OS serves it from RAM - no disk I/O required. The OS also provides read-ahead - detecting sequential access patterns and pre-loading pages before PostgreSQL requests them. This relationship explains the classic advice: set shared_buffers</code> to 25% of RAM. You're deliberately leaving space for the OS to act as a safety net. -- on dedicated servers with large RAM, 40% can work well -- but always leave room for OS cache and other processes ALTER SYSTEM SET shared_buffers = '8GB';</code></pre>This parameter allocates no memory - it's purely a hint for cost estimation.</div> PostgreSQL needs to know about this combined cache for query planning. That's where effective_cache_size</code> comes in - it tells the planner how much total cache (shared buffers + OS) to assume when estimating costs. -- estimate of total cache available (shared + OS) SHOW effective_cache_size;</code></pre> A higher value encourages the planner to favor index scans, assuming data is likely cached somewhere even if not in shared buffers. Conclusion</a> </h2> Buffers are one of the keystones of PostgreSQL internals. They control whether the query will hit fast RAM or slow disk; and at the same time they play a crucial role in the fragile balance of dirty pages, WAL and durability. The shared buffer pool isn't just a cache - it's a sophisticated memory manager with clock sweep eviction, usage count decay, ring buffer isolation, and background maintenance. Understanding these mechanisms helps you tune effectively and diagnose problems when they start. The hidden cost of PostgreSQL arrays 2026-01-12T20:50:00+00:00 Starting with arrays in PostgreSQL is as simple as declaring a column as integer[]</code>, inserting some values, and you are done. Or building the array on the fly. SELECT '{1,2,3}'::int[]; SELECT array[1,2,3];</code></pre> int4 --------- {1,2,3} (1 row) array --------- {1,2,3} (1 row)</code></pre> The official documentation</a> provides a good introduction. But beneath this straightforward interface lies a set of more complex properties than most of us realise. Arrays in PostgreSQL are not just "lists" in a field. They have their own memory management strategy, their own index logic, and a lot of edge-case scenarios. As it goes with boringSQL deep-dives, this article will explore the corners of array functionality that might break your production. The document model temptation</a> </h2> Wait? Are we going to talk about JSONB arrays? Not at all. The whole concept of arrays in RDBMSs is actually document storage in disguise. In database design, locality ensures faster retrieval times by keeping related data close on physical storage.Whether you use a distinct integer[]</code> type or a JSON list [1, 2, 3]</code>, you are making the exact same architectural decision: you are prioritising locality over normalisation. When you store tag_ids</code> in an array, you are embedding related data directly into a row - just like a NoSQL database might embed subdocuments. This is not inherently wrong. Document databases exist for good reasons: they eliminate joins, simplify reads, and map naturally to application objects. But PostgreSQL is a relational database. It was designed around the relational model, where: foreign keys enforce referential integrity</a></li> joins connect normalised tables</li> updates modify individual rows, not entire lists</li> </ul> Arrays give you document-model convenience, but you lose relational promises. There are no foreign keys and no ON DELETE referential_action</code> (like CASCADE) for array elements. If you delete a tags</code> entry, the orphaned ID will remain in your array forever. The rule of thumb is that if you find yourself in need of referential integrity - you most likely want a link table, not an array. Arrays are for data that shares the same lifecycle as the parent row. Not for relationships spanning across different tables. A practical example is the author of a blog post (one author might write multiple other posts), whereas a whitelist of IP addresses for a service account is only applicable to the given entity. With JSONB being so flexible, you might wonder why we still bother with quirkier native types. The answer lies in the 'boring' part of the database: predictability and efficiency. An integer[]</code> column guarantees that every element is an integer — similar to how enums</a> enforce a fixed set of allowed values. Arrays are also more storage-efficient for primitives because they don't carry the metadata overhead of JSON objects. The syntax gotchas</a> </h2> This post assumes basic knowledge of arrays. We won't cover the basics. Arrays don't have to start at 1</a> </h3> By default, SQL arrays start at 1. And seemingly, there is nothing wrong with iterating through them in a fashion similar to: FOR i IN 1 .. array_length(fruits, 1) LOOP RAISE NOTICE 'Index % contains: %', i, fruits[i]; END LOOP;</code></pre> that is until you find an array with the arbitrary bounds. Which PostgreSQL allows. SELECT '[-5:-3]={10,20,30}'::int[];</code></pre> To make sure you iterate correctly through any given array, always use array_lower()</code> and array_upper()</code> in PL/pgSQL SELECT array_lower('[-5:-3]={10,20,30}'::int[], 1);</code></pre> array_lower ------------- -5 (1 row)</code></pre> or generate_subscripts()</code> in SQL. SELECT generate_subscripts('[-5:-3]={10,20,30}'::int[], 1);</code></pre> generate_subscripts --------------------- -5 -4 -3 (3 rows)</code></pre>Missing dimensions</a> </h3> When creating a table, you might expect strict typing. That's true for everything — except the array dimensions. You might think integer[][]</code> enforces a 2D matrix. Except it does not. The []</code> syntax is effectively syntactic sugar. PostgreSQL does not enforce the number of dimensions of sub-arrays at the schema level at all by default. CREATE TABLE dimension_test ( matrix integer[][] ); INSERT INTO dimension_test VALUES ('{{1,2}, {3,4}}'); -- this is not going to fail INSERT INTO dimension_test VALUES ('{1,2,3}'); -- 3D matrix works too INSERT INTO dimension_test VALUES ('{{{1,2},{3,4}}, {{5,6},{7,8}}}');</code></pre> If you want to enforce a specific array dimension, you cannot rely on the type definition. Instead, you must use a CHECK</code> constraint. CREATE TABLE strict_matrix ( -- dims: ensure it's 2-Dimensional -- array_length: make sure it's exactly 3x3 board integer[] CHECK (array_ndims(board) = 2 AND array_length(board, 1) = 3) );</code></pre>INSERT INTO strict_matrix VALUES ( ARRAY[ [0, 1, 0], [1, 1, 0], [0, 0, 1] ] );</code></pre> The only exception is that PostgreSQL enforces uniformity of arrays on every nesting level. This means it rejects sub-arrays with different sizes. INSERT INTO dimension_test VALUES ('{{1,2}, {3}}'); ERROR: malformed array literal: "{{1,2}, {3}}" LINE 1: INSERT INTO dimension_test VALUES ('{{1,2}, {3}}'); ^ DETAIL: Multidimensional arrays must have sub-arrays with matching dimensions.</code></pre>Slicing arrays</a> </h3> When accessing array values, it's important to consider that the syntax [1]</code> and [1:1]</code> are different. While the first one is an accessor, the second one acts like a constructor. select matrix[1][1] from dimension_test ; matrix -------- 1 (1 row)</code></pre> When slicing an array, even if the slice is a single-element, it will be returned as a single-element array, not a scalar value. select matrix[1:1][1:1], matrix[1][1:1], matrix[1:1][1] from dimension_test ; matrix | matrix | matrix --------+--------+-------- {{1}} | {{1}} | {{1}} (1 row)</code></pre>The ugly</a> </h3> Accessing array values has a forgiving behaviour, making it more difficult to find underlying bugs: -- out-of-bound access returns NULL SELECT (ARRAY[1,2,3])[10]; array -------- (null) (1 row)</code></pre>-- out of bounds slicing returns an empty array SELECT (ARRAY[1,2,3])[5:10]; array ------- {} (1 row)</code></pre> But the single most confusing aspect that might trip you up coming from other programming languages is the fact that PostgreSQL treats multi-dimensional arrays as a single matrix, not an array of arrays. -- wrong dimensionality SELECT (ARRAY[[1,2],[3,4]])[1]; array -------- (null) (1 row) </code></pre> While in other languages you expect {1,2}</code> as a result, PostgreSQL can't give it to you. It tries to return the first cell and fails (because the index is not complete). And to paraphrase Fletcher's "Double Bubble" analogy (which also went very wrong — extra points for getting the reference), you can't fix this by using slice notation. select ('{{1,2},{3,4}}'::int[])[1:1]; int4 --------- {{1,2}} (1 row)</code></pre> The only way to solve this puzzle is to unnest</code> the slice and re-aggregate the results. SELECT array_agg(val) FROM unnest(('{{1,2},{3,4}}'::int[])[1:1]) val; array_agg ----------- {1,2} (1 row)</code></pre> Just be aware that `array_agg` does not guarantee the order of aggregated elements unless you use an `ORDER BY` clause. While it usually works with simple `unnest` queries, relying on implicit ordering can be risky. </div> The other alternative is to cast it to JSONB and back. Not pretty no matter how you look at it. If you need to work with complex multi-dimensional structures where each sub-array has independent meaning, just use JSONB. It will do exactly what you expect. Indexing arrays</a> </h2> While you can use a B-tree index on an array column, it won't help you unless you are looking for whole-array equality and sorting an array by dictionary rules — and even on regular columns, PostgreSQL can ignore your indexes</a> when it decides a sequential scan is cheaper. -- which is bigger? B is correct -- A: {1, 1000, 1000} -- B: {2, 0}</code></pre> Rendering B-tree indexes useless for any real-world index operations. When working with arrays, you actually need GIN (Generalized Inverted Index). If a B-tree index is a phone book, GIN is an index at the back of the book. To query one or more elements, you need to find all possible locations and then intersect them to find the locations that match. CREATE INDEX posts_by_tags ON posts USING GIN (tags);</code></pre> GIN indexes are designed for set operations, making presence the key feature, while ignoring order. Here are operators that GIN provides for arrays: Containment @></code> - matches rows that include ALL of the selected items. -- match ALL tags tags @> '{urgent, bug}'</code></pre> Overlap &&</code> - does the row include ANY of the selected items. -- match bug or feature tags && '{bug, feature}'</code></pre> There's also <@</code> and =</code> (equality), but they should be easy to understand. Two sides of ANY</code></a> </h3> This is where it gets interesting. While you might often hear (myself included) that you shouldn't use dynamic SQL for IN</code> lists and should use ANY</code> instead, there are some dangers you need to be aware of. The ANY</code> operator behaves very differently depending on which side of the comparison the array sits. The advice to use ANY</code> holds true when you are passing lists into the database. Instead of generating a query with 100 distinct parameters (WHERE id IN ($1, $2, ... $100)</code>), which bloats your query cache and forces hard-parses, you should pass a single array parameter. -- good: one parameter, one query plan SELECT * FROM users WHERE id = ANY($1::int[]);</code></pre> The trap is assuming this syntax is equally efficient when querying array columns. It is not. If you use ANY</code> to check if a value exists inside a table column, you are effectively asking the database to loop. -- bad: GIN does not support ANY; turning this into a seq scan SELECT * FROM tickets WHERE 'feature' = ANY(tags);</code></pre> When you write WHERE 'feature' = ANY(tags)</code>, you are not actually using an array operator. What you wrote is a scalar integer equality operator =</code> applied inside a loop construct. Since the scalar operator =</code> is not part of array_ops</code>, the planner assumes the index cannot help and falls back to a sequential scan. The correct way to rewrite the query is: -- good again SELECT * FROM tickets WHERE tags @> ARRAY['feature'];</code></pre>Fast updates and the trade-off</a> </h3> Because a GIN index is built to work with sets, it is expensive to maintain. With a B-tree index, one row equals one index entry. In a GIN index, one row equals N index entries, where N is the number of elements in your array. This leads to write multiplication. To prevent this, PostgreSQL defaults to using a "fast update" mechanism. This is a strategy where new entries are added to a pending list (an unsorted temporary buffer) and only merged into the main index structure later (during VACUUM). Given the unsorted nature of a GIN index, it is an exception to VACUUM is a Lie</a>, as VACUUM actually does perform structural maintenance here.While this makes INSERT operations manageable, it can slow down SELECTs. Every time you query the index, PostgreSQL must scan the organised main index plus the entire messy pending list. If that list grows large, your query performance might degrade. If you operate a read-heavy workflow with infrequent writes, you should disable this to guarantee consistent reading performance. CREATE INDEX posts_by_tags ON posts USING GIN (tags) WITH (fastupdate = off);</code></pre>Storage and modification</a> </h2> Now it's time to return to the document model</a>. In PostgreSQL, rows are immutable (MVCC); there is no such thing as an "in-place update", and arrays are stored as atomic values. This results in a very uncomfortable truth — to modify a single element of an array, PostgreSQL must copy and rewrite the entire row. UPDATE user_activity SET event_ids = event_ids || 10001 WHERE user_id = 50;</code></pre> Every single append rewrites the entire array, which in effect results in a rewrite of the entire row. TOASTed</a> </h3> When any array grows large enough (> 2 KB — see below), PostgreSQL moves it automatically to a separate storage area using TOAST</a>. While this keeps the data row lean, it turns array updates into a severe performance bottleneck. The difference this introduces is subtle but comes with a big impact. While a standard MVCC update simply copies the row version on the main heap, updating a TOASTed array forces PostgreSQL to fetch all external chunks, decompress the entire object into memory, apply the change, and then recompress and write the new full-size blob back to the TOAST table. This turns a simple modification into a CPU and I/O-intensive operation that rewrites the entire dataset rather than just the delta. The threshold is derived from TOAST_TUPLES_PER_PAGE</code> (default: 4), ensuring 4 tuples fit on a page.Threshold: ~2 KBWhere does the 2 KB value come from? The TOAST threshold is calculated to ensure at least four tuples can fit on a single 8 KB heap page</a>. PostgreSQL uses this to balance efficiency against the overhead of TOAST indirection. The compressions</a> </h3> Before version 14, PostgreSQL relied on pglz</code> — an algorithm prioritising compression ratio over speed. This made the "decompress-modify-compress" cycle of TOAST painful. PostgreSQL 14 introduced LZ4 as an alternative: ALTER TABLE articles ALTER COLUMN tags SET COMPRESSION lz4;</code></pre> LZ4 is significantly faster for both compression and decompression, with only slightly lower compression ratios. If you are working with large arrays, switching to LZ4 is one of the easiest ways to reduce the CPU penalty of TOAST. When a large array might make sense</a> </h3> You might have gained the impression that arrays are bad. When evaluating the use of arrays, the real question isn't how big the array is but rather how often do you modify it? An array of 10,000 elements that you write once and is read-only for the rest of its lifecycle is a completely valid use case. An array of 50 elements that you append to on every incoming request is the real villain here. If you combine this with compression, you might get an interesting mix. DROP TABLE IF EXISTS compression_test; CREATE TABLE compression_test ( id serial PRIMARY KEY, compressed_floats float4[], raw_floats float4[] ); -- do not compress raw_floats ALTER TABLE compression_test ALTER COLUMN raw_floats SET STORAGE EXTERNAL; -- insert semi-random data with low cardinality INSERT INTO compression_test (compressed_floats, raw_floats) SELECT semi_random_arr, semi_random_arr FROM ( SELECT ARRAY( SELECT floor(random() * 50)::float4 FROM generate_series(1, 10000) ) as semi_random_arr ) as generator; SELECT 'Compressed (EXTENDED)' as strategy, pg_size_pretty(pg_column_size(compressed_floats)::bigint) as size_on_disk FROM compression_test UNION ALL SELECT 'Raw (EXTERNAL)', pg_size_pretty(pg_column_size(raw_floats)::bigint) FROM compression_test;</code></pre> strategy | size_on_disk -----------------------+-------------- Compressed (EXTENDED) | 15 kB Raw (EXTERNAL) | 39 kB (2 rows) </code></pre>Bulk loading with arrays</a> </h2> Up until now, it might seem that arrays don't actually bring many benefits. While they can have rough edges around storage, they are incredibly useful for transport. The fastest way to insert 5,000 rows isn't a loop in your application, and it's definitely not a massive VALUES (...), (...)</code> string. It's unnest</code>. INSERT INTO measurements (sensor_id, value, captured_at) SELECT * FROM unnest( $1::int[], -- array of sensor IDs $2::float[], -- array of values $3::timestamptz[] -- array of timestamps );</code></pre> All that is needed is one network round trip, and one query to parse and plan. PostgreSQL handles the arrays row by row internally for you. This works both for UPSERTs and MERGE</a>. The cases for special arrays</a> </h2> Standard PostgreSQL arrays are polymorphic types (anyarray</code>). This provides a powerful feature that allows a single function definition to operate on many different data types. They have to handle integers, strings, timestamps, and custom types equally well. But if you have specific data types, you can unlock significant performance gains by using specialised extensions. The intarray</code> extension</a> </h3> If you are dealing exclusively with 4-byte integers (int4</code>/integer</code>), the built-in array operations are leaving performance on the table. The intarray</a> extension provides specialised functions and index operators that are significantly faster than the generic implementation. To use it, you must explicitly enable it: CREATE EXTENSION IF NOT EXISTS intarray;</code></pre> The difference in developer ergonomics is immediate. To sort an array in standard SQL, you are forced to unnest, order, and re-aggregate. With intarray, you get native functions like sort() and uniq(). -- standard arrays SELECT array_agg(val ORDER BY val) FROM unnest('{3, 1, 2}'::int[]) val; -- intarray SELECT sort('{3, 1, 2}'::int[]);</code></pre> Beyond raw management functions, it introduces a specialised query syntax that simplifies complex boolean logic. Instead of chaining multiple overlap (&&</code>) and containment (@></code>) checks, you can express your requirements in a single "query string" using the @@</code> operator. -- standard arrays SELECT * FROM staff WHERE available_days @> '{1}' -- must include Mon AND available_days && '{6, 7}' -- must include Sat OR Sun AND NOT (available_days @> '{2}'); -- must NOT include Tue -- intarray SELECT * FROM staff WHERE available_days @@ '1 & (6 | 7) & !2';</code></pre> The only catch is the type restriction. intarray</code> is strictly limited to signed 32-bit integers. If your values exceed 2 billion, you are back where you started. AI with pgvector</code></a> </h3> You cannot talk about arrays in 2026 without mentioning pgvector</a>. While it markets itself as a "vector store", internally it is simply an array of floats with a different mathematical focus. Standard arrays are binary: they care about Exact Matches (Overlap &&</code>, Containment @></code>). Vectors are all about fuzzy distance (Cosine <=></code>, Euclidean <-></code>). If you are building search or recommendation features, pgvector allows you to treat your array column not as a list of "facts" (tag A, tag B), but as coordinates in semantic space. Nevertheless, the architectural decision is exactly the same as using a standard array: you are trading strict structure for convenience. Since there is no way to "join" two rows based on how similar they are, you store the vector directly on the row. You accept a larger table size in exchange for the ability to ask, "What is close to this?". PS: Thanks to Matthias Feist for inspiring this article. Instant database clones with PostgreSQL 18 2025-12-22T23:53:16+00:00 Have you ever watched a long running migration script</a>, wondering if it's about to wreck your data? Or wish you can "just" spin a fresh copy of database for each test run? Or wanted to have reproducible snapshots to reset between runs of your test suite, (and yes, because you are reading boringSQL) needed to reset the learning environment? When your database is a few megabytes, pg_dump</code> and restore works fine. But what happens when you're dealing with hundreds of megabytes/gigabytes - or more? Suddenly "just make a copy" becomes a burden. You've probably noticed that PostgreSQL connects to template1</code> by default. What you might have missed is that there's a whole templating system hiding in plain sight. Every time you run CREATE DATABASE dbname;</code></pre> PostgreSQL quietly clones standard system database template1</code> behind the scenes. Making it same as if you would use CREATE DATABASE dbname TEMPLATE template1;</code></pre> The real power comes from the fact that you can replace template1</code> with any database. You can find more at Template Database documentation</a>. In this article, we will cover a few tweaks that turn this templating system into an instant, zero-copy database cloning machine. CREATE DATABASE ... STRATEGY</a> </h2> Before PostgreSQL 15, when you created a new database from a template, it operated strictly on the file level. This was effective, but to make it reliable, Postgres had to flush all pending operations to disk (using CHECKPOINT</code>) before taking a consistent snapshot. This created a massive I/O spike - a "Checkpoint Storm" - that could stall your production traffic. Version 15 of PostgreSQL introduced new parameter CREATE DATABASE ... STRATEGY = [strategy]</code> and at the same time changed the default behaviour how the new databases are created from templates. The new default become WAL_LOG</code> which copies block-by-block via the Write-Ahead Log (WAL), making I/O sequential (and much smoother) — operations that also show up in EXPLAIN buffer statistics</a> — and support for concurrency without facing latency spike. This prevented the need to CHECKPOINT but made the database cloning operation potentially significantly slower. For an empty template1</code>, you won't notice the difference. But if you try to clone a 500GB database using WAL_LOG, you are going to be waiting a long time. The STRATEGY</code> parameter allows us to switch back to the original method FILE_COPY</code> to keep the behaviour, and speed. And since PostgreSQL 18, this opens the whole new set of options. FILE_COPY</a> </h2> Because the FILE_COPY</code> strategy is a proxy to operating system file operations, we can change how the OS handles those files. When using standard file system (like ext4</code>), PostgreSQL reads every byte of the source file and writes it to a new location. It's a physical copy. However starting with PostgreSQL 18 - file_copy_method</code> gives you options to switch that logic; while default option remains copy</code>. With modern filesystems (like ZFS, XFS with reflinks, APFS, etc.) you can switch it to clone</code> and leverage CLONE</code> (FICLONE</code> on Linux) operation for almost instant operation. And it won't take any additional space. Quick setup checklist: Linux with XFS or ZFS, macOS with APFS, or FreeBSD with ZFS</li> PostgreSQL 18+ cluster on that filesystem</li> Set file_copy_method = clone</code> in your config</li> Reload configuration</li> </ul> </div> The benchmark</a> </h2> We need some dummy data to copy. This is the only part of the tutorial where you have to wait. Let's generate a ~6GB database. CREATE DATABASE source_db; \c source_db CREATE TABLE boring_data ( id serial PRIMARY KEY, payload text ); -- generate 50m rows INSERT INTO boring_data (payload) SELECT md5(random()::text) || md5(random()::text) FROM generate_series(1, 50000000); -- force a checkpoint CHECKPOINT;</code></pre> You can verify the database now has roughly 6GB of data. Name | source_db Owner | postgres Encoding | UTF8 Locale Provider | libc Collate | en_US.UTF-8 Ctype | en_US.UTF-8 Locale | ICU Rules | Access privileges | Size | 6289 MB Tablespace | pg_default Description |</code></pre> While enabling \timing</code> you can test the default (WAL_LOG) strategy. And on my test volume (relatively slow storage) I get CREATE DATABASE slow_copy TEMPLATE source_db; CREATE DATABASE Time: 67000.615 ms (01:07.001)</code></pre> Now, let's verify our configuration is set for speed: show file_copy_method; file_copy_method ------------------ clone (1 row)</code></pre> Let's request the semi-instant clone of the same database, without taking extra disk space at the same time. CREATE DATABASE fast_clone TEMPLATE source_db STRATEGY=FILE_COPY; CREATE DATABASE Time: 212.053 ms</code></pre> That's a quite an improvement, isn't it? Working with cloned data</a> </h2> That was the simple part. But what is happening behind the scenes? When you clone a database with file_copy_method = clone</code>, PostgreSQL doesn't duplicate any data. The filesystem creates new metadata entries that point to the same physical 8KB pages</a>. Both databases share identical storage. This can create some initial confusion. If you ask PostgreSQL for the size: SELECT pg_database_size('source_db') as source, pg_database_size('fast_clone') as clone;</code></pre> PostgreSQL reports both as ~6GB because that's the logical size - how much data each database "contains" - i.e. logical size. -[ RECORD 1 ]------ source | 6594041535 clone | 6594041535</code></pre> The interesting part happens when you start writing. PostgreSQL doesn't update tuples in place. When you UPDATE a row, it writes a new tuple version somewhere (often a different page entirely) and marks the old one as dead. The filesystem doesn't care about PostgreSQL internals - it just sees writes to 8KB pages</a>. Any write to a shared page triggers a copy of that entire page. A single UPDATE will therefore trigger copy-on-write on multiple pages: the page holding the old tuple</li> the page receiving the new tuple</li> index pages if any indexed columns changed</li> FSM and visibility map pages as PostgreSQL tracks free space</li> </ul> And later, VACUUM touches even more pages while cleaning up dead tuples</a>. In this case diverging quickly from the linked storage. XFS proof</a> </h2> Using the database OID and relfilenode we can verify the both databases are now sharing physical blocks. root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16402/16404 Filesystem type is: 58465342 File size of /var/lib/postgresql/18/main/base/16402/16404 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 2031: 10471550.. 10473581: 2032: shared 1: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared 2: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared 3: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared 4: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared 5: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared 6: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof /var/lib/postgresql/18/main/base/16402/16404: 7 extents found root@clone-demo:/var/lib/postgresql# root@clone-demo:/var/lib/postgresql# root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16418/16404 Filesystem type is: 58465342 File size of /var/lib/postgresql/18/main/base/16418/16404 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 2031: 10471550.. 10473581: 2032: shared 1: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared 2: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared 3: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared 4: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared 5: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared 6: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof /var/lib/postgresql/18/main/base/16418/16404: 7 extents found</code></pre> All it takes is to update some rows using update boring_data set payload = 'new value' || id where id IN (select id from boring_data limit 20);</code></pre> and the situation will start to change. root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16402/16404 Filesystem type is: 58465342 File size of /var/lib/postgresql/18/main/base/16402/16404 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 39: 10471550.. 10471589: 40: 1: 40.. 2031: 10471590.. 10473581: 1992: shared 2: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared 3: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared 4: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared 5: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared 6: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared 7: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof /var/lib/postgresql/18/main/base/16402/16404: 7 extents found root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16418/16404 Filesystem type is: 58465342 File size of /var/lib/postgresql/18/main/base/16418/16404 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 39: 10297326.. 10297365: 40: 1: 40.. 2031: 10471590.. 10473581: 1992: 10297366: shared 2: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared 3: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared 4: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared 5: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared 6: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared 7: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof /var/lib/postgresql/18/main/base/16418/16404: 8 extents found root@clone-demo:/var/lib/postgresql#</code></pre> In this case extent 0 no longer has shared flag, first 40 blocks size (with default size 4KB) now diverge, making it total of 160KB. Each database now has its own copy at different physical address. The remaining extents are still shared. Things to be aware of</a> </h2> Cloning is tempting but there's one serious limitation you need to be aware if you ever attempt to do it in production. The source database can't have any active connections during cloning. This is a PostgreSQL limitation, not a filesystem one. For production use, this usually means you create a dedicated template database rather than cloning your live database directly. Or given the relatively short time the operation takes you have to schedule the cloning in times where you can temporary block/terminate all connections. Other limitation is that the cloning only works within a single filesystem. If your databases spans multiple table spaces on different mount points, cloning will fall back to regular physical copy. Finally, in most managed cloud environments (AWS RDS, Google Cloud SQL), you will not have access to the underlying filesystem to configure this. You are stuck with their proprietary (and often billed) functionality. But for your own VMs or bare metal? Go ahead and try it. VACUUM Is a Lie (About Your Indexes) 2025-12-11T23:34:00+00:00 There is common misconception that troubles most developers using PostgreSQL: tune VACUUM or run VACUUM, and your database will stay healthy. Dead tuples will get cleaned up. Transaction IDs recycled. Space reclaimed. Your database will live happily ever after. But there are couple of dirty "secrets" people are not aware of. First of them being VACUUM is lying to you about your indexes. The anatomy of storage</a> </h2> When you delete a row in PostgreSQL, it is just marked as a 'dead tuple'. Invisible for new transactions but still physically present. Only when all transactions referencing the row are finished, VACUUM can come along and actually remove them - reclamining the space in the heap (table) space. To understand why this matters differently for tables versus indexes, you need to picture how PostgreSQL actually stores your data. Your table data lives in the heap - a collection of 8 KB pages where rows are stored wherever they fit. There's no inherent order. When you INSERT a row, PostgreSQL finds a page with enough free space and slots the row in. Delete a row, and there's a gap. Insert another, and it might fill that gap - or not - they might fit somewhere else entirely. This is why SELECT * FROM users</code> without an ORDER BY can return rows in order initially, and after some updates in seemingly random order, and that order can change over time. The heap is like Tetris. Rows drop into whatever space is available, leaving gaps when deleted. When VACUUM runs, it removes those dead tuples and compacts the remaining rows within each page. If an entire page becomes empty, PostgreSQL can reclaim it entirely. And while indexes are on surface the same collection of 8KB pages, they are different. A B-tree index must maintain sorted order - that's the whole point of their existence and the reason why WHERE id = 12345</code> is so fast — though even indexes can be ignored</a> under certain conditions. PostgreSQL can binary-search down the tree instead of scanning every possible row. You can learn more about the fundamentals of B-Tree Indexes and what makes them fast</a>. But if the design of the indexes is what makes them fast, it's also their biggest responsibility. While PostgreSQL can fit rows into whatever space is available, it can't move the entries in index pages to fit as much as possible. VACUUM can remove dead index entries. But it doesn't restructure the B-tree. When VACUUM processes the heap, it can compact rows within a page and reclaim empty pages. The heap has no ordering constraint - rows can be anywhere. But B-tree pages? They're locked into a structure. VACUUM can remove dead index entries, yes. Many developers assume VACUUM treats all pages same. No matter whether they are heap or index pages. VACUUM is supposed to remove the dead entries, right? Yes. But here's what it doesn't do - it doesn't restructure the B-tree. What VACUUM actually does Removes dead tuple pointers from index pages</li> Marks completely empty pages as reusable</li> Updates the free space map</li> </ul> What VACUUM cannot do: Merge sparse pages together (can do it for empty pages)</li> Reduce tree depth</li> Deallocate empty-but-still-linked pages</li> Change the physical structure of the B-tree</li> </ul> Your heap is Tetris, gaps can get filled. Your B-tree is a sorted bookshelf. VACUUM can pull books out, but can't slide the remaining ones together. You're left walking past empty slots every time you scan. The experiment</a> </h2> Let's get hands-on and create a table, fill it, delete most of it and watch what happens. CREATE EXTENSION IF NOT EXISTS pgstattuple; CREATE TABLE demo (id integer PRIMARY KEY, data text); -- insert 100,000 rows INSERT INTO demo (id, data) SELECT g, 'Row number ' || g || ' with some extra data' FROM generate_series(1, 100000) g; ANALYZE demo;</code></pre> At this point, our index is healthy. Let's capture the baseline: SELECT relname, pg_size_pretty(pg_relation_size(oid)) as file_size, pg_size_pretty((pgstattuple(oid)).tuple_len) as actual_data FROM pg_class WHERE relname IN ('demo', 'demo_pkey');</code></pre>relname | file_size | actual_data -----------+-----------+------------- demo | 7472 kB | 6434 kB demo_pkey | 2208 kB | 1563 kB</code></pre> Now remove some data, 80% to be precise - somewhere in the middle: DELETE FROM demo WHERE id BETWEEN 10001 AND 90000;</code></pre> The goal is to simulate a common real-world pattern: data retention policies, bulk cleanup operations, or the aftermath of a data migration gone wrong</a>. VACUUM demo; SELECT relname, pg_size_pretty(pg_relation_size(oid)) as file_size, pg_size_pretty((pgstattuple(oid)).tuple_len) as actual_data FROM pg_class WHERE relname IN ('demo', 'demo_pkey');</code></pre>relname | file_size | actual_data -----------+-----------+------------- demo | 7472 kB | 1278 kB demo_pkey | 2208 kB | 313 kB</code></pre> The table shrunk significantly, while index remained unchanged You now have 20,000 rows indexed by a structure built to handle 100,000. But notice something important: even though actual_data</code> dropped to ~1.3 MB (reflecting our 20,000 remaining rows), the file_size</code> for the index is still 2208 kB. VACUUM cleaned out the dead tuple data, but it doesn't return space to the OS or compact index structures - it only marks pages as reusable within PostgreSQL. EDIT: the original table had a size without running VACCUM, which I must left out during what's sometime cumbersome process of authoring blog posts. Big thanks come to Creston Jamison of Scaling PostgreSQL</a> to notice it, and actually take what must have been a lot of time to analyze my posts. </blockquote> This experiment is really an extreme case, but demonstrates the problem. Understanding page states</a> </h2> Leaf pages have several states: Full page (>80% density), when the page contains many index entries, efficiently utilizing space. Each 8KB page</a> read returns substantial useful data. This is optimal state. Partial page (40-80% density) with some wasted space, but still reasonably efficient. Common at tree edges or after light churn. Nothing to be worried about. Sparse page (<40% density) is mostly empty. You're reading an 8KB page to find a handful of entries. The I/O cost is the same as a full page, but you get far less value. Empty page (0% density) with zero live entries, but the page still exists in the tree structure. Pure overhead. You might read this page during a range scan and find absolutely nothing useful. A note on fillfactor</a> </h3> You might be wondering how can fillfactor help with this? It's the setting you can apply both for heap and leaf pages, and controls how full PostgreSQL packs the pages during the data storage. The default value for B-tree indexes is 90%. This leaves 10% of free space on each leaf page for future insertions. CREATE INDEX demo_index ON demo(id) WITH (fillfactor = 70);</code></pre> A lower fillfactor (like 70%) leaves more room, which can reduce page splits when you're inserting into the middle of an index - useful for tables random index column inserts or those with heavily updated index columns. But if you followed carefully the anatomy of storage section, it doesn't help with the bloat problem. Quite the opposite. If you set lower fillfactor and then delete majority of your rows, you actually start with more pages, and bigger chance to end up with more sparse pages than partial pages. Leaf page fillfactor is about optimizing for updates and inserts. It's not a solution for deletion or index-column update bloat. Why the planner gets fooled</a> </h2> PostgreSQL's query planner estimates costs based on physical statistics</a>, including the number of pages in an index. EXPLAIN ANALYZE SELECT * FROM demo WHERE id BETWEEN 10001 AND 90000;</code></pre>QUERY PLAN -------------------------------------------------------------------------------------------------------------------- Index Scan using demo_pkey on demo (cost=0.29..29.29 rows=200 width=41) (actual time=0.111..0.112 rows=0 loops=1) Index Cond: ((id >= 10001) AND (id <= 90000)) Planning Time: 1.701 ms Execution Time: 0.240 ms (4 rows)</code></pre> While the execution is almost instant, you need to look behind the scenes. The planner estimated 200 rows and got zero. It traversed the B-tree structure expecting data that doesn't exist. On a single query with warm cache, this is trivial. Under production load with thousands of queries and cold pages, you're paying I/O cost for nothing. Again and again. If you dig further you discover much bigger problem. SELECT relname, reltuples::bigint as row_estimate, relpages as page_estimate FROM pg_class WHERE relname IN ('demo', 'demo_pkey');</code></pre>relname | row_estimate | page_estimate -----------+--------------+--------------- demo | 20000 | 934 demo_pkey | 20000 | 276</code></pre> The relpages</code> value comes from the physical file size divided by the 8 KB page size</a>. PostgreSQL updates it during VACUUM and ANALYZE, but it reflects the actual file on disk - not how much useful data is inside. Our index file is still 2.2 MB (276 pages × 8 KB), even though most pages are empty. The planner sees 276 pages for 20,000 rows and calculates a very low rows-per-page ratio. This is when planner can come to conclusion - this index is very sparse - let's do a sequential scan instead. Oops. "But wait," you say, "doesn't ANALYZE fix statistics?" Yes and no. ANALYZE</code> updates the row count estimate. It will no longer think you have 100,000 rows but 20,000. But it does not shrink relpages, because that reflects the physical file size on disk. ANALYZE</code> can't change that. The planner now has accurate row estimates but wildly inaccurate page estimates. The useful data is packed into just ~57 pages worth of entries, but the planner doesn't know that. cost = random_page_cost × pages + cpu_index_tuple_cost × tuples ([visible in EXPLAIN output](/posts/explain-buffers/))</code></pre> With a bloated index: pages are oversized (276 instead of ~57)</li> The per-page cost gets multiplied by empty pages</li> Total estimated cost is artificially high</li> </ul> The hollow index</a> </h2> We can dig even more into the index problem when we look at internal stats: SELECT * FROM pgstatindex('demo_pkey');</code></pre>-[ RECORD 1 ]------+-------- version | 4 tree_level | 1 index_size | 2260992 root_block_no | 3 internal_pages | 1 leaf_pages | 57 empty_pages | 0 deleted_pages | 217 avg_leaf_density | 86.37 leaf_fragmentation | 0</code></pre> Wait, what? The avg_leaf_density is 86% and it looks perfectly healthy. That's a trap. Due to the hollow index (we removed 80% right in the middle) we have 57 well-packed leaf pages, but the index still contains 217 deleted pages. This is why avg_leaf_density</code> alone is misleading. The density of used pages looks great, but 79% of your index file is dead weight. The simplest way to spot index bloat is comparing actual size to expected size. SELECT c.relname as index_name, pg_size_pretty(pg_relation_size(c.oid)) as actual_size, pg_size_pretty((c.reltuples * 40)::bigint) as expected_size, round((pg_relation_size(c.oid) / nullif(c.reltuples * 40, 0))::numeric, 1) as bloat_ratio FROM pg_class c JOIN pg_index i ON c.oid = i.indexrelid WHERE c.relkind = 'i' AND c.reltuples > 0 AND c.relname NOT LIKE 'pg_%' AND pg_relation_size(c.oid) > 1024 * 1024 -- only indexes > 1 MB ORDER BY bloat_ratio DESC NULLS LAST;</code></pre>index_name | actual_size | expected_size | bloat_ratio ------------+-------------+---------------+------------- demo_pkey | 2208 kB | 781 kB | 2.8</code></pre> A bloat_ratio</code> of 2.8 means the index is nearly 3x larger than expected. Anything above 1.8 - 2.0 deserves investigation. We filter to indexes over 1 MB - bloat on tiny indexes doesn't matter that much. Please, adjust the threshold based on your environment; for large databases, you might only care about indexes over 100 MB. But here comes BIG WARNING: pgstatindex() we used earlier physically reads the entire index. On a 10 GB index, that's 10 GB of I/O. Don't run it against all indexes on a production server - unless you know what you are doing! REINDEX</a> </h2> How to actually fix index bloat problem? REINDEX</code> is s straightforward solution as it rebuilds the index from scratch. REINDEX INDEX CONCURRENTLY demo_pkey ;</code></pre> After which we can check the index health: SELECT * FROM pgstatindex('demo_pkey');</code></pre>-[ RECORD 1 ]------+------- version | 4 tree_level | 1 index_size | 466944 root_block_no | 3 internal_pages | 1 leaf_pages | 55 empty_pages | 0 deleted_pages | 0 avg_leaf_density | 89.5 leaf_fragmentation | 0</code></pre> And SELECT relname, pg_size_pretty(pg_relation_size(oid)) as file_size, pg_size_pretty((pgstattuple(oid)).tuple_len) as actual_data FROM pg_class WHERE relname IN ('demo', 'demo_pkey');</code></pre>relname | file_size | actual_data -----------+-----------+------------- demo | 7472 kB | 1278 kB demo_pkey | 456 kB | 313 kB</code></pre> Our index shrunk from 2.2 MB to 456 KB - 79% reduction (not a big surprise though). As you might have noticed we have used CONCURRENTLY</code> to avoid using ACCESS EXCLUSIVE lock. This is available since PostgreSQL 12+, and while there's an option to omit it - the pretty much only reason to do so is during planned maintenance to speed up the index rebuild time. pg_squeeze</a> </h2> If you look above at the file_size of our relations, we have managed to reclaim the disk space for the affected index (it was REINDEX</code> after all), but the table space was not returned back to the operating system. That's where pg_squeeze</a> shines. Unlike trigger-based alternatives, pg_squeeze uses logical decoding, resulting in lower impact on your running system. It rebuilds both the table and all its indexes online, with minimal locking: CREATE EXTENSION pg_squeeze; SELECT squeeze.squeeze_table('public', 'demo');</code></pre> The exclusive lock is only needed during the final swap phase, and its duration can be configured. Even better, pg_squeeze is designed for regular automated processing - you can register tables and let it handle maintenance whenever bloat thresholds are met. pg_squeeze makes sense when both table and indexes are bloated, or when you want automated management. REINDEX CONCURRENTLY is simpler when only indexes need work. There's also older tool pg_repack - for a deeper comparison of bloat-busting tools, see article The Bloat Busters: pg_repack vs pg_squeeze</a>. VACUUM FULL (The nuclear option)</a> </h2> VACUUM FULL</code> rewrites the entire table and all indexes. While it fixes everything it comes with a big but - it requires an ACCESS EXCLUSIVE lock - completely blocking all reads and writes for the entire duration. For a large table, this could mean hours of downtime. Generally avoid this in production. Use pg_squeeze instead for the same result without the downtime. When to act, and when to chill</a> </h2> Before you now go and REINDEX</code> everything in sight, let's talk about when index bloat actually matters. B-trees expand and contract with your data. With random insertions affecting index columns - UUIDs, hash keys, etc. the page splits happen constantly. Index efficiency might get hit at occassion and also settle around 70 - 80% over different natural cycles of your system usage. That's not bloat. That's the tree finding its natural shape for your data. The bloat we demonstrated - 57 useful pages drowning in 217 deleted ones - is extreme. It came from deleting 80% of contiguous data. You won't see this from normal day to day operations. When do you need to act immediately: after a massive DELETE (retention policy, GDPR purge, failed migration cleanup)</li> bloat_ratio</code> exceeds 2.0 and keeps climbing</li> query plans suddenly prefer sequential scans on indexed columns</li> index size is wildly disproportionate to row count</li> </ul> But in most cases you don't have to panic. Monitor weekly and when indexes bloat ratio continously grow above warning levels, schedule a REINDEX CONCURRENTLY</code> during low traffic period. Index bloat isn't an emergency until it is. Know the signs, have the tools ready, and don't let VACUUM's silence fool you into thinking everything's fine. Conclusion</a> </h2> VACUUM is essential for PostgreSQL. Run it. Let autovacuum do its job. But understand its limitations: it cleans up dead tuples, not index structure. The truth about PostgreSQL maintenance is that VACUUM handles heap bloat reasonably well, but index bloat requires explicit intervention. Know when your indexes are actually sick versus just breathing normally - and when to reach for REINDEX. VACUUM handles heap bloat. Index bloat is your problem. Know the difference. RegreSQL: Regression Testing for PostgreSQL Queries 2025-11-13T22:47:00+00:00 TL;DR - RegreSQL brings PostgreSQL's regression testing methodology to your application queries, catching both correctness bugs and performance regressions before production. As puzzling as it might seem, the common problem with production changes is the ever-present "AHA" moment when things start slowing down or crashing straight away. Testing isn't easy as it is, but there's a widespread practice gap when it comes to testing SQL queries. Some might pretend to "fix it" by using ORMs to abstract away the problem. Others treat SQL as "just glue code" that doesn't deserve systematic testing. Most settle for integration tests that verify the application layer works, never actually testing whether their queries will survive the next schema change or index modification. For PostgreSQL development itself, the project has a robust regression test suite that has been preventing disasters in core development for decades. The database itself knows how to test SQL systematically - we just don't use those same techniques for our own queries. Enter RegreSQL</a>, a tool originally created by Dimitri Fontaine for The Art of PostgreSQL</a> book (which is excellent for understanding and mastering PostgreSQL as a database system), designed to bring the same regression testing framework to our application queries. I've been trying to use it for some time, but due to missing features and limitations gave up several times. Until now. I decided to fork the project and spend the time needed to take it to the next level. Introduction</a> </h2> The RegreSQL</a> promise starts with the biggest strength and perceived weakness of SQL queries. They are just strings. And unless you use something like sqlc</a> (for Go), PG'OCaml</a> or Rust's SQLx</a> toolkit giving you compile-time checking, your queries are validated only when they are executed. Which in better case mean either usually slow-ish test suite or integration tests, in worst scenario only when deployed. ORMs are another possibility - completely abstracting away SQL (but more on that later). But even with compile-time checking, you are only checking for one class of problems: schema mismatches. What about behavior changes after schema migration or performance regressions? What about understanding whether your optimization actually made things faster or just moved the problem elsewhere? This is where RegreSQL comes in. Rather than trying to turn SQL into something else, RegreSQL embraces "SQL as strings" reality and applies the same testing methodology PostgreSQL itself uses: regression testing. You write (or generate - continue reading) your SQL queries, provide input data, and RegreSQL verifies that future changes don't break those expectations. The features don't stop there though - it tracks performance baselines, detects common query plan regressions (like sequential scans), and gives you framework for systematic experimentation with the schema changes and query change management. Basic regression testing</a> </h2> Enough with theory. Let's jump in straight into the action and see what a sample run of RegreSQL looks like $ regresql text Connecting to 'postgres://radim:password123@192.168.139.28/cdstore_test'… ✓ Running regression tests... ✓ album-by-artist_list-albums-by-artist.1.json (0.00s) ✓ album-by-artist_list-albums-by-artist.2.json (0.00s) ✓ album-tracks_list-tracks-by-albumid.2.json (0.00s) ✓ album-tracks_list-tracks-by-albumid.1.json (0.00s) ✓ artist_top-artists-by-album.1.json (0.00s) ✓ genre-topn_genre-top-n.top-1.json (0.00s) ✓ genre-topn_genre-top-n.top-3.json (0.00s) ✓ genre-tracks_tracks-by-genre.json (0.00s) Results: 8 passed, 0 failed, 8 skipped (0.00s)</code></pre> In this example based on Chinook database</a> (as used originally in The Art of PostgreSQL book), RegreSQL scans the current directory (or one provided by -C /path/to/project</code>) for *.sql</code> files and attempts to run all queries against the configured PostgreSQL connection. The individual files can contain either single or multiple sql queries. Like following example -- name: top-artists-by-album -- Get the list of the N artists with the most albums SELECT artist.name, count(*) AS albums FROM artist LEFT JOIN album USING (artist_id) GROUP BY artist.name ORDER BY albums DESC LIMIT :n;</code></pre> The syntax for the queries supports both positional arguments (like $1</code> known from libpq library) or (preferred) psql</code> style variable (:varname</code>). The each identified query (not file) is then executed for 0..N times, based on number of predefined plans and verified to the expected results - validating the expected data matches the one returned. The support for SQL files handling is available separately with https://github.com/boringSQL/queries (Go version only for now). This gives you what original RegreSQL tool has introduced - change your schema, refactor a query, run regresql test</code> and see immediately what broke. The test suite now has ability to catch regressions before they are committed / shipped. The current version built on top of it, giving you better console formatter instead of TAP style output, as well as jUnit, JSON and GitHub actions formatters for better integration into your CI/CD pipelines. Performance regression testing</a> </h2> Basic regression testing catches correctness issues - wrong results, broken queries, schema mismatches. But there's another class of production issues it misses. Performance regressions. No matter how unbelievable it might sound but queries get deployed without appropriate indexes — which might then be ignored</a> entirely — or they change over time. Simple fix - both for handwritten SQL or ORM code - can switch from milliseconds to seconds. You add index that helps one query, but tanks another. You modify conditionals and accidently force a sequential scan of millions of rows</a>. This is where it hurts. RegreSQL addresses this by tracking performance baselines alongside correctness. Once baselines are generated $ regresql baseline Connecting to 'postgres://appuser:password123@192.168.139.28/cdstore_test'… ✓ Creating baselines directory: regresql/baselines Creating directory 'regresql/baselines' Creating baselines for queries: ./ Created baseline: album-by-artist_list-albums-by-artist.1.json Created baseline: album-by-artist_list-albums-by-artist.2.json Created baseline: album-tracks_list-tracks-by-albumid.1.json Created baseline: album-tracks_list-tracks-by-albumid.2.json Created baseline: artist_top-artists-by-album.1.json Created baseline: genre-topn_genre-top-n.top-1.json Created baseline: genre-topn_genre-top-n.top-3.json Created baseline: genre-tracks_tracks-by-genre.json Baselines have been created successfully! Baseline files are stored in: regresql/baselines</code></pre> the test command not only tests the regressions to the captured times, but also detects the common bad patterns in query execution plans</a>. For now it provides warnings for detection of sequential scans - both on their and/or with nested loops and multiple sort operations. I believe this alone might provide a valuable insights and reduce the mishaps in production. It's also a place where further development of RegreSQL will take place. To demonstrate this, let's review the test output with the baselines. Connecting to 'postgres://appuser:password123@192.168.139.28/cdstore_test'… ✓ Running regression tests... ✓ album-by-artist_list-albums-by-artist.1.json (0.00s) ✓ album-by-artist_list-albums-by-artist.2.json (0.00s) ✓ album-by-artist_list-albums-by-artist.1.cost (22.09 <= 22.09 * 110%) (0.00s) ⚠️ Sequential scan detected on table 'artist' Suggestion: Consider adding an index if this table is large or this query is frequently executed ⚠️ Nested loop join with sequential scan detected Suggestion: Add index on join column to avoid repeated sequential scans ✓ album-by-artist_list-albums-by-artist.2.cost (22.09 <= 22.09 * 110%) (0.00s) ⚠️ Sequential scan detected on table 'artist' Suggestion: Consider adding an index if this table is large or this query is frequently executed ⚠️ Nested loop join with sequential scan detected Suggestion: Add index on join column to avoid repeated sequential scans ✓ album-tracks_list-tracks-by-albumid.1.json (0.00s) ✓ album-tracks_list-tracks-by-albumid.2.json (0.00s) ✓ album-tracks_list-tracks-by-albumid.1.cost (8.23 <= 8.23 * 110%) (0.00s) ✓ album-tracks_list-tracks-by-albumid.2.cost (8.23 <= 8.23 * 110%) (0.00s) ✓ artist_top-artists-by-album.1.json (0.00s) ✓ artist_top-artists-by-album.1.cost (35.70 <= 35.70 * 110%) (0.00s) ⚠️ Multiple sequential scans detected on tables: album, artist Suggestion: Review query and consider adding indexes on filtered/joined columns ✓ genre-topn_genre-top-n.top-1.json (0.00s) ✓ genre-topn_genre-top-n.top-3.json (0.00s) ✓ genre-topn_genre-top-n.top-1.cost (6610.59 <= 6610.59 * 110%) (0.00s) ⚠️ Multiple sequential scans detected on tables: genre, artist Suggestion: Review query and consider adding indexes on filtered/joined columns ⚠️ Multiple sort operations detected (2 sorts) Suggestion: Consider composite indexes for ORDER BY clauses to avoid sorting ⚠️ Nested loop join with sequential scan detected Suggestion: Add index on join column to avoid repeated sequential scans ✓ genre-topn_genre-top-n.top-3.cost (6610.59 <= 6610.59 * 110%) (0.00s) ⚠️ Multiple sequential scans detected on tables: artist, genre Suggestion: Review query and consider adding indexes on filtered/joined columns ⚠️ Multiple sort operations detected (2 sorts) Suggestion: Consider composite indexes for ORDER BY clauses to avoid sorting ⚠️ Nested loop join with sequential scan detected Suggestion: Add index on join column to avoid repeated sequential scans ✓ genre-tracks_tracks-by-genre.json (0.00s) ✓ genre-tracks_tracks-by-genre.cost (37.99 <= 37.99 * 110%) (0.00s) ⚠️ Multiple sequential scans detected on tables: genre, track Suggestion: Review query and consider adding indexes on filtered/joined columns Results: 16 passed (0.00s)</code></pre> As you can see, despite from not having baseline, RegreSQL is able to detect the basic bad patterns that should be addressed before queries can be considered "production ready". In some cases, having the detection of sequential scans, or just tracking query costs baselines might be considered undesirable, which would lead to false positives. RegreSQL enables this to be addressed by query metadata as demonstrated below. -- name: query_name -- metadata: key1=value1, key2=value2 SELECT ...;</code></pre> At this point RegreSQL recognizes notest</code> to skip the query testing altogether (not just cost tracking)</li> nobaseline</code> to skip cost tracking</li> noseqscanwarn</code> to keep cost tracking but disable sequential scan warnings</li> and difffloattolerance</code> to cost failure threshold (default 10% at the moment).</li> </ul> -- name: query_name -- regresql: notest, nobaseline -- regresql: noseqscanwarn -- regresql: difffloattolerance:0.25 -- query that can vary in cost by 20% without being considered a failure SELECT ...;</code></pre>ORM enters the room</a> </h2> ORMs abstract away SQL, but they still generate it - much like view inlining</a> where the planner rewrites your SQL behind the scenes - and that generated SQL can have performance problems you won't catch until production. Consider this common scenario: you start with a simple SQLAlchemy query that works fine, then months later add eager loading for related data: orders = ( session.query(Order) .filter(Order.user_id == user_id) .options( joinedload(Order.user), joinedload(Order.shipping_address), selectinload(Order.items) # NEW: Load order items ) .all() )</code></pre> That innocent selectinload(Order.items)</code> generates a separate query - and without an index on order_items.order_id</code>, it performs a sequential scan. RegreSQL can catch this by intercepting ORM-generated SQL using SQLAlchemy's event system: @event.listens_for(engine, "before_cursor_execute") def capture_sql(conn, cursor, statement, *args): captured_queries.append(statement)</code></pre> Run your ORM code, capture the SQL, save it as a .sql file, and test it with RegreSQL. The performance baseline testing will flag the missing index before it hits production. This is currently experimental, but ORM integration is a key area for RegreSQL's future development. Test Data Management</a> </h2> Up until now we have covered how RegreSQL verifies query correctness and tracks performance regressions. But there's a critical prerequisite we've only skimmed through. Every regression test needs consistent, reproducible data. Change the data, change their cardinality, and your expected results become meaningless. Your performance baselines drift. Your tests become flaky. Traditional approach to create test data might involve Database dumps become unmanageable - 500MB files you can't review, can't understand, that break with every schema migration, and whose data becomes stale as production evolves. Which version of the dump are your tests even using?</li> SQL scripts might be better than dumps, but still imperative and hard to maintain. You end up with INSERT statements scattered across multiple files, managing foreign keys manually, and debugging constraint violations.</li> Factories in application code might work great for integration tests, but we're testing SQL directly. Do you really want to maintain parallel data generation in your application language just for SQL tests?</li> Shared test database is the synonym for classic "works on my machine" problem. State leaks between tests. Parallel execution becomes impossible. Debugging is a nightmare.</li> </ul> What we need is something that's declarative (what data, not how to insert it), reproducible (similar data every time), composable (build complex scenarios from simple pieces), and scalable (from 10 rows to 100,000). This is where next improvement in RegreSQL's fixture system comes in. Think of it as infrastructure-as-code for your test data. You describe the data you need in YAML files, and RegreSQL handles the rest - dependencies, cleanup, foreign keys, and even realistic data generation at scale. RegreSQL's fixture system lets you define test data in YAML files stored in regresql/fixtures/</code>. Here's a simple example fixture: basic_users description: a handful of test users cleanup: rollback data: - table: users rows: - id: 1 email: alice@example.com name: Alice Anderson created_at: 2024-01-15 - id: 2 email: bob@example.com name: Bob Builder created_at: 2024-02-20</code></pre> To use this fixture in your tests, reference it in the query's plan file (regresql/plans/get-user.yaml</code>) you can just reference the fixture fixtures: - basic_users "1": email: alice@example.com "2": email: bob@example.com</code></pre> And when you run regresql test</code>, the fixture is automatically loaded before the query executes, and cleaned up afterward. No manual setup scripts, no state leakage between tests. But it does not stop with static fixtures. When you want to test queries against realistic volumes you can use range of data generators including sequences, random integer, decimal, string, uuid, email and name generators</li> date_between for generating random timestamps within a range</li> foreign key references to be able to reuse data from other table's fixtures</li> range to select value from predefined sources</li> Go template support</li> </ul> fixture: realistic_orders generate: - table: customers count: 1000 columns: id: generator: sequence start: 1 email: generator: email domain: shop.example.com name: generator: name type: full created_at: generator: date_between start: "2023-01-01" end: "2024-12-31" - table: orders count: 5000 columns: id: generator: sequence start: 1 customer_id: generator: int min: 1 max: 1000 amount: generator: decimal min: 10.00 max: 999.99 precision: 2 order_date: generator: date_between start: "2023-01-01" end: "2024-12-31"</code></pre> This generates 1,000 customers and 5,000 orders with realistic-looking data - names, emails, dates, and amounts that feel production-like. The fixtures are also stackable and can be build on top of each other. For example if you need to make sure users fixtures are created before orders fixtures, just declare the dependency (the already planned improvement is to include the support automatic foreign-key detection to avoid ID hard-coding). RegreSQL loads fixtures in dependency order and handles cleanup in reverse. fixture: orders_with_shipping depends_on: - basic_users data: - table: orders rows: - id: 101 user_id: 1 # References Alice from basic_users total: 99.99 status: shipped</code></pre> Should the available options for fixtures (manual data or data generators) not be enough, you always have options to use good old SQL based data generation. fixture: mixed_setup description: Combine SQL with YAML and generated data cleanup: rollback # SQL executes first (either as file or inline) sql: - file: sql/setup_schema.sql - inline: "INSERT INTO config (key, value) VALUES ('version', '1.0');" # followed YAML data data: - table: users rows: - id: 1 email: admin@example.com # and finally generated data generate: - table: orders count: 100 columns: id: generator: sequence start: 1 user_id: generator: int min: 1 max: 1</code></pre> RegreSQL provides commands to inspect and validate your fixtures # List all available fixtures regresql fixtures list # Show fixture details and dependencies regresql fixtures show realistic_orders # Validate fixture definitions regresql fixtures validate # Show dependency graph regresql fixtures deps # Apply fixture manually (for debugging) regresql fixtures apply basic_users</code></pre> The fixture system has been design to transforms test data from a maintenance burden into a documented, version-controlled process. Your YAML files become the single source of truth for what data your tests need, making it easy to understand test scenarios and maintain test data as the application evolves. (EDIT 2025-11-20) The dynamically generated fixtures on each tests will break the correctness testing of RegreSQL. You have to use fixtures according to your use Fixed generated fixtures re-used between tests</li> Use RegreSQL to test fixtures before/after migration</li> </ul> I'm working on next release of RegreSQL to address DX of fixtures usage. RegreSQL future</a> </h2> Introducing a new open source project is an ambitious goal, and RegreSQL</a> is just starting up. Despite the fork being in works for almost 2 years. In coming weeks and months I plan further improvements, as well as better documentation and more tutorials. The project is maintained as part of my boringSQL brand, where it's vital component (together with pgTap) for building SQL Labs</a> which (as I sincerely hope) will provide a foundation for its further development. At the same time RegreSQL is an attempt to give back to welcoming PostgreSQL community, make developer user experience slightly better if possible and (just maybe) provide one more argument against the case that SQL queries are not testable. RegreSQL is available at boringsql.com/products/regresql</a> - feel free to open issue, or drop me email about the project at radim@boringsql.com</a> or connect on LinkedIn</a>. Beyond Start and End: PostgreSQL Range Types 2025-11-02T21:40:00+00:00 One of the most read articles at boringSQL is Time to Better Know The Time in PostgreSQL</a> where we dived into the complexities of storing and handling time operations in PostgreSQL. While the article introduced the range data types, there's so much more to them. And not only for handling time ranges. In this article we will cover why to consider range types and how to work with them. Bug Not Invented Here</a> </h2> But before we can talk about the range types, let's try to understand why we should look at them in the first place. Let's imagine a booking platform for large flash sales of the seats, that goes live at 10pm and will be taken by storm by thousands of people who want to get their tickets. CREATE TABLE seat_holds ( hold_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, seat_id INTEGER NOT NULL REFERENCES seats(id), user_session_id UUID NOT NULL, -- define the hold period explicitly hold_started_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP, hold_expires_at TIMESTAMPTZ NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP ); CREATE INDEX seat_holds_on_seat_id ON seat_holds(seat_id); CREATE INDEX seat_holds_on_expiration ON seat_holds(hold_expires_at);</code></pre> While the table design looks perfectly reasonable, it has one serious flaw - there's no database-level atomicity guarantee there to prevent two holds for the same seat_id</code> at the same time. The table design requires on application logic to check for the existing holds before inserting a new hold, and at the same time it does not provide any high-concurrency guarantee. If all you have in your toolbelt are those two columns you will end up increasing the complexity to make it work. You started with one problem and soon your application developers might want to ask you to add caching layer (most likely external K/V store) to place the holds there and very soon you have N-problems when you will resort to building a complex, custom application-side locking mechanism that is bug-prone and difficult to maintain. Other possibility is to bring all the operations on seat holds into more complex transaction management. Which is literally invitation for disaster in extreme contention situation like flash ticket sales. No matter which blocking strategy you use SERIALIZABLE</code> transaction isoliation or pessimistic locking using SELECT ... FOR UPDATE</code> will create a large overhead in application logic (retries, massive contention on database resources, etc.). And as we are going to talk about range data types, there's a first possible option to solve the problem directly on database level. -- needed to add GiST support to equality of integers (for seat_id) CREATE EXTENSION IF NOT EXISTS btree_gist; ALTER TABLE seat_holds ADD CONSTRAINT seat_holds_no_overlap EXCLUDE USING gist ( seat_id WITH =, tsrange(hold_started_at, hold_expires_at) WITH && );</code></pre> Which will add the constraint directly on the table level and enable atomic conflict detection for your seat holds with minimum locking overhead. This guarantees that the database will never allow two overlapping holds to exist for the same seat, irrespective of how many concurrent users are attempting to book. The real win here is data integrity - you're making it impossible to have invalid state in your database, not just unlikely. While you'll still need retry logic in your application when conflicts occur, you've moved the correctness guarantee from application code (where bugs hide) into the database schema (where it's enforced). The GiST (Generalized Search Tree) index is the crucial component which makes checking for overlapping time ranges effective even under extreme load. But really, if you look at the proposed fix, it's still a workaround - we're converting two separate TIMESTAMPTZ</code> columns into a range type on the fly, when range types already include native GiST support out of the box. Introducing the Data Range Types</a> </h2> You’ve seen the power of the EXCLUDE</code> constraint to solve the concurrency problem, but why to settle for workaround (unless it's temporary as part of the bigger refactoring) instead of going all the way in? This brings us to the core of the matter: PostgreSQL's native Range Types. PostgreSQL provides a set of built-in range types, all following the pattern of type and range: int4range</code> for integer</li> int8range</code> for bigint</li> numrange</code> for numeric</li> tsrange</code> for timestamp without time zone</li> tstzrange</code> for timestamp with time zone (which we briefly saw above)</li> daterange</code> for date</li> </ul> And it does not stop there. You can easily define your own custom range types over any basic data type. First win: cleaner schema</a> </h2> When using start</code> and end</code> columns you are not explicitely telling the database that these two columns are single concept representing time span. The logic to work with those two columns resides only in your queries and application code. When you refactor our sample table to embrace the native range type, it becomes more expressive and inherently correct. The application code no longer needs to manage two separate boundaries. CREATE TABLE seat_holds_native ( hold_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, seat_id INTEGER NOT NULL REFERENCES seats(id), user_session_id UUID NOT NULL, hold_period TSTZRANGE NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP );</code></pre> This is the power of a first-class data type. We've shifted the burden from the application logic to the database schema, making the table definition itself communicate its intent more clearly. Second win: Atomicity guaranteed</a> </h2> While the new schema is cleaner, the real database win comes from enforcing our concurrency guarantee - preventing two seats from being double-booked. To achieve this you can reuse the exclusion constraint as demonstrated previously. ALTER TABLE seat_holds_native ADD CONSTRAINT seat_holds_no_overlap EXCLUDE USING gist ( seat_id WITH =, hold_period WITH && );</code></pre> This time you can use hold_period</code> directly without need to explicitely convert it. This constraint enforces two rules at once: seat_id WITH =</code> ensures the constraint only applies to holds for the same seat.</li> hold_period WITH &&</code> checking the overlap of hold periods with the operator &&</code></li> </ol> Finally EXCLUDE USING gist</code> is the crucial technical detail, telling PostgreSQL to use GiST index to enforce the constraint. This is not specific to range types, as EXCLUDE</code> constraint can't exist without an index to enforce it (common use cases might include arrays, geometric data, etc.). Range boundaries: A quick math refresher</a> </h2> Before we dive into the operators, let's take a moment to understand how PostgreSQL represents range boundaries. If you remember your high school math, range types use the same notation as intervals in mathematics. PostgreSQL ranges can have four different boundary types: Inclusive boundaries - [</code> and ]</code> An inclusive boundary includes the endpoint value in the range. Think of it as "less than or equal to" or "greater than or equal to". -- [10, 20] includes both 10 and 20 -- represents 10 ≤ x ≤ 20 SELECT int4range(10, 20, '[]');</code></pre> Exclusive boundaries - (</code> and )</code> An exclusive boundary excludes the endpoint value from the range. This is "less than" or "greater than" without the equality. -- (10, 20) excludes both 10 and 20 -- represents 10 < x < 20 SELECT int4range(10, 20, '()'); </code></pre> Mixed boundaries - [)</code> or (]</code> You can mix and match. The most common and default pattern in PostgreSQL is [)</code> - inclusive lower bound, exclusive upper bound. This is particularly useful for timestamps and dates because it naturally represents "from the start of one period up to (but not including) the start of the next". As mentioned, the default boundary [)</code> eliminates the natural ambiguity when representing consecutive periods. -- these ranges are adjacent, not overlapping -- Week 1 SELECT '[2025-11-01, 2025-11-08)'::tstzrange; -- Week 2 SELECT '[2025-11-08, 2025-11-15)'::tstzrange;</code></pre> With this notation, the end of one period is exactly the start of the next, with no gaps or overlaps. This makes it perfect for time-based ranges, inventory availability windows, or any scenario where you're dividing a continuum into distinct segments. Canonicalization and Range Set Operations</a> </h2> Boundaries are not the only aspect of the ranges that behave like mathematical sets, allowing arithmetic operations and canonicalization of the discrete ranges. For discrete range types (int4range, int8range, daterange), multiple representations can actually mean the exact same set of values. For example, for integers, the range [10, 20] (inclusive on both ends) is the same set as (9, 21) (exclusive on both ends) or the default PostgreSQL canonical form [10, 21) (inclusive lower, exclusive upper). PostgreSQL uses a canonicalization function to convert all equivalent discrete ranges into a single, uniform representation (default [)</code> boundary mentioned above), which is essential for accurate equality checks and indexing. -- [10, 20] includes integers 10, 11, ..., 20. SELECT int4range(10, 20, '[]') AS original_range; -- Result: [10,21) -- (9, 21) includes integers 10, 11, ..., 20. SELECT int4range(9, 21, '()') AS original_range; -- Result: [10,21)</code></pre> Exception are continuous ranges (think floats and timestamps with fractional seconds) where PostgreSQL won't use canonicalization because a boundary change always means a change in the contained values as there's no easy to define "next value". I.e. there's no next value for 20.0 (i.e. not 20.0001, nor 20.000001, etc.) and changing boundary would change it's meaning. The operator toolkit</a> </h2> The type ranges and by their definition GiST (in this instance range_ops) and GIN (array_ops) indexes come with number of operator that makes your life easier. Overlap operator - &&</code> As mentioned already above the overlap operator is the most fundamental one. It simply checks whether two ranges share any common data points. -- find holds active at any point between 10:00 and 11:00 SELECT * FROM seat_holds_native WHERE hold_period && '[2025-12-25 10:00, 2025-12-25 11:00)'::tstzrange;</code></pre> Contains operator - @></code> For checking against specific moments in time, we might turn to the Contains operator. It verifies whatever the range on the left completely containts the element on the right (which might be both underlying data type or range type). -- find holds that are active at the specific momement SELECT * FROM seat_holds_native WHERE hold_period @> '2025-11-05 15:00'::timestamptz; -- find holds that are active at the specific time range SELECT * FROM seat_holds_native WHERE hold_period @> '[2025-12-25 10:00, 2025-12-25 10:15)'::tstzrange;</code></pre> Contained By operator - <@</code> In contrast, the Contained By operator checks the reverse relationship - whatever the range on the left is entirely contained by the range on the right. -- find holds that are within November '25 SELECT * FROM seat_holds_native WHERE hold_period <@ '[2025-11-01, 2025-12-01)'::tstzrange;</code></pre> Strictly Before/After operators - <<</code> and >></code> operators allow you to query ranges that are completely separated from the reference range (i.e. don't even touch the boundaries). -- find holds that finished strictly before 10 November SELECT * FROM seat_holds_native WHERE hold_period << '[2025-11-10, 2025-11-15)'::tstzrange;</code></pre> Boundary Extension operators - &<</code> and &></code> let you reason about range boundaries independently, checking whether one range extends beyond another's endpoints (i.e. it can start/end anywhere within the given range). -- find holds that end before or at the same time as reference range ends SELECT * FROM seat_holds_native WHERE hold_period &< '[2025-11-08 17:00, 2025-11-08 18:00)'::tstzrange; -- find holds that start at or after reference range starts SELECT * FROM seat_holds_native WHERE hold_period &>'[2025-11-08 09:00, 2025-11-08 18:00)'::tstzrange;</code></pre> Finally the Adjecent operator - -|-</code> checks if two ranges are perfectly contiguous - they MUST touch at exactly one boundary point, but do not overlap. This might be invaluable when checking if a customer can extend an existing hold without any gap or conflict. -- find holds that are immediately adjacent (touching) to given range SELECT * FROM seat_holds_native WHERE hold_period -|- '[2025-11-08 17:00, 2025-11-08 18:00)'::tstzrange;</code></pre>To Infinity and Beyond</a> </h2> Similar to base types ranges in PostgreSQL can handle NULL values, but it does not stop there. There are also special states specifically applicable to data type ranges: empty</code> and infinity</code>. Let's start with infinite bounds, the bound that shows the real power of the ranges. You can define range that extend infinitely in either direction (or both at the same time). -- range that never expires (upper bound is infinite) SELECT '[2025-11-01 10:00, infinity)'::tstzrange; -- range that has always been valid (lower bound is infinite) SELECT '[-infinity, 2025-11-01 10:00)'::tstzrange; -- range covering all time SELECT '[-infinity, infinity)'::tstzrange;</code></pre> This gives you ability to describe the "from this points forward" use cases. As we will cover later we can easily define lifetime subscription. -- lifetime subscription that never expires INSERT INTO subscriptions (user_id, plan, active_period) VALUES (42, 'lifetime', '[2025-11-01, infinity)'); -- all active subscriptions right now SELECT * FROM subscriptions WHERE active_period @> NOW();</code></pre> Using infinity</code> is far more elegant solution that using NULL values or "special" values like 2099-31-12</code> - it's explicit and clearly communicates the data intent. At any point you can validate whatever range has infinite bounds: SELECT lower_inf('[2025-11-01, infinity)'::tstzrange) AS lower_is_infinite, upper_inf('[2025-11-01, infinity)'::tstzrange) AS upper_is_infinite;</code></pre>Understanding NULL vs empty: Schrödinger's Range</a> </h2> Ranges can be NULL or empty, and these are completely different things. NULL is Schrödinger's range - you haven't looked in the box yet, so it could be anything or nothing. Empty is when you've opened the box and confirmed it's empty. Let's see this in practice: -- NULL range: we don't know what the period is INSERT INTO seat_holds_native (seat_id, user_session_id, hold_period) VALUES (42, 'abc-123', NULL); -- empty range: we know the period is explicitly "nothing" INSERT INTO seat_holds_native (seat_id, user_session_id, hold_period) VALUES (43, 'def-456', 'empty');</code></pre> The main difference between them is when it comes to handling. SELECT NULL::tstzrange && '[2025-11-01, 2025-11-08)'::tstzrange; -- Result: NULL (not true, not false—we don't know) SELECT 'empty'::tstzrange && '[2025-11-01, 2025-11-08)'::tstzrange; -- Result: false (we know it doesn't overlap)</code></pre> And you can check for them in your queries using built-in function isempty</code>. -- check for NULL (like any column) SELECT * FROM seat_holds_native WHERE hold_period IS NULL; -- check for empty (special function) SELECT * FROM seat_holds_native WHERE isempty(hold_period);</code></pre> In practice, you'll mostly use NOT NULL</code> constraints to prevent NULL ranges entirely. Empty ranges are useful but rare - usually for representing cancelled/void periods you need to keep for special purposes - like audit trail. Practical Integer ranges for Tiered pricing</a> </h2> While we introduced ranges we mostly paid attention to the date/time handling the usefulness of range types goes well beyond that. One of the practical applications where integer ranges provide real values can be demostrated on tiered pricing. CREATE TABLE quantity_discounts ( discount_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, product_id INTEGER NOT NULL, quantity_range INT4RANGE NOT NULL, discount_percentage NUMERIC(5,2) NOT NULL, -- no overlapping tiers EXCLUDE USING GIST ( product_id WITH =, quantity_range WITH && ) ); INSERT INTO quantity_discounts (product_id, quantity_range, discount_percentage) VALUES -- 1-9 units: no discount (1, '[1,10)', 0.00), -- 10-49 units: 5% off (1, '[10,50)', 5.00), -- 50-99 units: 10% off (1, '[50,100)', 10.00), -- 100+ units: 15% off (1, '[100,1000)', 15.00); -- verify what discount we offer for ordering 75 units? SELECT discount_percentage FROM quantity_discounts WHERE product_id = 1 AND quantity_range @> 75; -- Result: 10.00</code></pre>Making Bad Data Impossible</a> </h2> If the introduction of the range types provided the case for cleaner schema you can go ahead and make hard limits structurally impossible. While this is not advocating for the transition of the full business logic into database schema, you can eliminate the edge cases that should never make it to the database. CREATE TABLE promotional_campaigns ( campaign_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, name TEXT NOT NULL, active_period TSTZRANGE NOT NULL, budget_range NUMRANGE NOT NULL, discount_percentage NUMERIC(5,2) NOT NULL, -- campaigns must be at least 1 days long CONSTRAINT campaigns_minimum_duration CHECK (upper(active_period) - lower(active_period) >= INTERVAL '1 days'), -- budget must be between $1000 and $100000 CONSTRAINT campaigns_valid_budget CHECK (budget_range <@ numrange(1000, 100000, '[]')), -- active period must not be empty CONSTRAINT campaigns_valid_period CHECK (NOT isempty(active_period)) );</code></pre> While this example demonstrates hypothetical example within the schema definition, please remember they shouldn't be used to implement business process. The goal of the constraints is to enforce data integrity, i.e. structure requirements (minimum duration, non-empty data), physical or domain boundaries. Any other logic should make it's way either to application logic or parts that are easier to modify (think functions). Multiranges: When one range is not enough</a> </h2> Up until now, we've been working with single continuous ranges. But what happens when you need to represent fragmented ranges? In past you needed a separate table with a foreign key relationship. With multiranges, you can store multiple non-contiguous ranges in a single column. PostgreSQL 14 introduced multirange types for all the built-in range types: int4multirange</code>, int8multirange</code>, nummultirange</code></li> tsmultirange</code>, tstzmultirange</code>, datemultirange</code></li> </ul> The real power of multiranges lies in schema density and query efficiency. We can prove this by comparing the cost of storing and querying the exact same data using two different valid range schemas. Let's consider storing 20 fragmented and non-contiguous periods - a pattern common for historical data. CREATE TABLE user_periods_single_range ( period_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, user_id INTEGER NOT NULL, active_period TSTZRANGE NOT NULL ); CREATE INDEX user_periods_single_range_gist_idx ON user_periods_single_range USING gist (active_period); -- 20 fragmented periods for user_id 42 (20 rows total) INSERT INTO user_periods_single_range (user_id, active_period) SELECT 42, tstzrange( '2025-01-01'::timestamptz + (i * 2 || ' days')::interval, '2025-01-01'::timestamptz + (i * 2 + 1 || ' days')::interval ) FROM generate_series(1, 20) AS s(i);</code></pre> compared to 1 row for the same 20 periods aggregated using range_agg</code> function to consolidate data. CREATE TABLE user_periods_multirange ( user_id INTEGER PRIMARY KEY, active_periods TSTZMULTIRANGE NOT NULL ); CREATE INDEX user_periods_multirange_gist_idx ON user_periods_multirange USING gist (active_periods); -- consolidate the 20 TSTZRANGE rows into 1 TSTZMULTIRANGE row INSERT INTO user_periods_multirange (user_id, active_periods) SELECT user_id, -- range_agg function automatically handles merging adjacent ranges range_agg(active_period)::TSTZMULTIRANGE FROM user_periods_single_range WHERE user_id = 42 GROUP BY user_id;</code></pre> consider now the difference between EXPLAIN SELECT period_id, user_id FROM user_periods_single_range WHERE user_id = 42 AND active_period @> '2025-01-20 12:00:00+00'::timestamptz;</code></pre> QUERY PLAN ------------------------------------------------------------------------------------------------- Bitmap Heap Scan on user_periods_single_range (cost=4.19..13.67 rows=1 width=12) Recheck Cond: (active_period @> '2025-01-20 13:00:00+01'::timestamp with time zone) Filter: (user_id = 42) -> Bitmap Index Scan on user_periods_single_range_gist_idx (cost=0.00..4.19 rows=6 width=0) Index Cond: (active_period @> '2025-01-20 13:00:00+01'::timestamp with time zone)</code></pre> and same version of the consolidated data EXPLAIN ANALYZE SELECT user_id, active_periods FROM user_periods_multirange WHERE user_id = 42 AND active_periods @> '2025-01-20 12:00:00+00'::timestamptz;</code></pre> QUERY PLAN ------------------------------------------------------------------------------------------------------------- Index Scan using user_periods_multirange_pkey on user_periods_multirange (cost=0.15..8.17 rows=1 width=36) Index Cond: (user_id = 42) Filter: (active_periods @> '2025-01-20 13:00:00+01'::timestamp with time zone)</code></pre> Giving you significant reduction in the query cost. And while the example was simple enough for the demonstration purposes you can easily define the helper schema for better indexable data access and opportunities to reduce the storage requirements. One important note is that subtle change with the &&</code> operators. Whereas with single range &&</code> operator checks if two continues ranges overlap, for multiranges the operator checks if ANY range in multirange overlaps. Creating custom range types</a> </h2> While PostgreSQL provides built-in range types for common data types, you can create custom range types for any data type that has a meaningful ordering. Let's demostrate this with a type for IP address ranges. To create a custom range type, you need to provide a subtype difference function that tells PostgreSQL how to calculate the "distance" between two values: -- function to calculate difference between two IP addresses using bigint CREATE OR REPLACE FUNCTION inet_diff(x INET, y INET) RETURNS FLOAT8 AS $$ SELECT ( (host(x)::inet - '0.0.0.0'::inet) - (host(y)::inet - '0.0.0.0'::inet) )::float8; $$ LANGUAGE SQL IMMUTABLE STRICT; -- custom inetrange type CREATE TYPE inetrange AS RANGE ( subtype = inet, subtype_diff = inet_diff ); -- convert CIDR ranges to inetranges CREATE OR REPLACE FUNCTION cidr_to_inetrange(cidr CIDR) RETURNS inetrange AS $$ SELECT inetrange( host(network($1))::inet, host(broadcast($1))::inet, '[]' ); $$ LANGUAGE SQL IMMUTABLE STRICT;</code></pre> Now you can use these custom types just like the built-in ones: CREATE TABLE ip_blocklists ( blocklist_name TEXT PRIMARY KEY, blocked_range inetrange NOT NULL ); INSERT INTO ip_blocklists (blocklist_name, blocked_range) VALUES ('attack #434401', cidr_to_inetrange('192.168.1.0/24')), ('attack #434401 (1)', '[203.0.113.50,203.0.113.99]'), ('attack #434401 (2)', '[203.0.113.143,203.0.113.159]');</code></pre> and locate which attack has been assigned to particular malicious IP address. SELECT blocklist_name, blocked_range FROM ip_blocklists WHERE blocked_range @> '192.168.1.25'::INET;</code></pre> blocklist_name | blocked_range ----------------+----------------------------- attack #434401 | [192.168.1.0,192.168.1.255]</code></pre> But wait a second, wasn't the fragmented nature of the ranges used in case for multiranges? Building real-life production and auto adaptive block list would most likely soon create extremely fragmented set of targets to block. Can we defined it for our custom range types? Well no, because PostgreSQL is amazing! Since PostgreSQL 14 every time you define custom range type, it will automatically create corresponding multirange! Making it easy to consolidate the fragmented data corresponding to individual attack to multiranges. SELECT typname FROM pg_type WHERE typname LIKE 'inet%range';</code></pre> typname ---------------- inetmultirange inetrange</code></pre> Danger zone! When creating a custom range type the subtype_diff function is more than just simple helper. It plays important role in indexing and query performance. It really tells PostgreSQL planner how far apart the values in range are, which is crucial when building GiST indexes for ranges. In our example above, if inet_diff</code> returned 0</code> for every pair of IP addresses, PostgreSQL would think all ranges are "equally close". This would lead to un-balanced indexes, with large hotposts in the indexes. End result would be that range operators would effectively be almost as slow as sequantial scans. Performance deep dive: GiST vs GIN</a> </h2> Throughout this article, we've been using GiST indexes almost exclusively, particularly when enforcing exclusion constraints. But PostgreSQL also supports GIN indexes for range types, and understanding when to use each can make the difference between a query that completes in milliseconds versus one that grinds your database to a halt. Before we deep dive, let's recap what those two indexes do. GiST (Generalized Search Tree) is a balanced tree structure that organizes ranges by their bounding boxes, and grouping those that are "close together" in same tree nodes. While GIN (Generalized Inverted Index) would decompose ranges into their components and indexing those. Therefore GiST works for continous ranges (timestamps and numerical values), while GIN works for discrete ranges (as you can't generate unpredictable range of values of floats for example). Given this characteristics you can almost certainly say GIN indexes are almost always going to be bigger compared to the GiST ones, as they are always trying to index a continous space. The most important thing to know upfront? As we already used without explicitely mentioning it - you can't use GIN with EXCLUDE constraints. GiST is mandatory there. Therefore while GiST index is going to be preferred for cases like CREATE TABLE seat_holds ( hold_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, seat_id INTEGER NOT NULL, hold_period TSTZRANGE NOT NULL );</code></pre> the GIN index is actually preferred for CREATE TABLE venue_blackouts ( venue_id INTEGER, blocked_dates DATERANGE NOT NULL );</code></pre> The reason is simple - dates are discrete type. There's only 365 (or 366) combinations in a given year. While timestamps have microsecond precision - millions of possible values per day. The complexity of the index types is far beyond this article scope, so let's just iterate over basic rules what index type to use. Most applications should just use GiST and move on. The performance difference rarely matters until you're dealing with millions of rows and very specific query patterns. Don't prematurely optimize - GiST is the safe, versatile default that works well for almost everything. You can always add a GIN index later if profiling shows it would help. PostgreSQL's query planner is smart enough to pick the better index when both are available. Conclusion</a> </h2> From my experience range types represent one of PostgreSQL's most underutilized features, yet they offer immediate benefits: cleaner schemas, built-in data integrity, and query patterns that feel natural once you embrace them. What started as a solution to prevent double-booking seats evolved into a comprehensive look at how treating ranges as first-class citizens transforms your database design. But we've really just scratched the surface. Timestamp ranges in particular open an entire world of possibilities we haven't touched - temporal tables. The ability to maintain complete historical records with automatic versioning, query data "as of" any point in time, and track changes without cluttering your schema with audit columns deserves its own deep dive. That's a topic for a future article. For now, the next time you reach for separate start</code> and end</code> columns, stop and ask yourself: "Should this be a range?" More often than not, the answer is yes. Your future self - the one debugging a problems in least convenient time - will thank you. PostgreSQL maintenance without superuser 2025-09-13T20:55:00+00:00 How many people/services have superuser access to your PostgreSQL cluster(s)? Did you ever ask why your software engineers might need it? Or your BI team? Why those use cases require same privileges as someone who can drop your databases? The answer isn't because these operations are inherently dangerous - it's because PostgreSQL historically offered limited options for operational access or simply because not enough people are aware of the options. So the common practice is to either got basic permissions or handover the keys to the kingdom. PostgreSQL's built-in predefined roles solve this problem by providing purpose-built privileges for common maintenance tasks. Instead of granting superuser access for routine operations, you can delegate specific capabilities - monitoring teams get comprehensive observability access, backup services get data reading capabilities, and maintenance scripts get precisely the permissions they need, nothing more. What are Predefined Roles?</a> </h2> PostgreSQL's built-in administrative roles are purpose-built permission sets that solve the superuser dilemma for common maintenance tasks. Out of the box, there are 15 predefined roles that provide granular access to specific operational capabilities without requiring full superuser privileges. While you can view their list and description in official documentation</a>, in this article we will explore them bit more thoroughly and at the same time look into system catalogs to understand them better. The individual roles can be grouped by their functionality and most of them are quite easy to grasp, ranging from simple monitoring access to powerful filesystem operations that require careful consideration. Data Access Role pg_database_owner</code> - Database-specific ownership (special case)</li> pg_read_all_data</code> - Read access to all tables, views, sequences</li> pg_write_all_data</code> - Write access to all tables, views, sequences</li> </ul> Monitoring & Observability pg_monitor</code> - Which is actually monitoring meta role that contains 3 roles listed below</li> pg_read_all_settings</code> - Configuration access</li> pg_read_all_stats</code> - Statistics views</li> pg_stat_scan_tables</code> - Table scanning for stats</li> </ul> System Operations pg_signal_backend</code> - Cancel queries, terminate sessions</li> pg_checkpoint</code> - Run CHECKPOINT command</li> pg_maintain</code> - VACUUM, ANALYZE, REINDEX operations (PostgreSQL 17+)</li> pg_signal_autovacuum_worker</code> - Signal autovacuum worker (PostgreSQL 18+)</li> </ul> File System Access pg_read_server_files</code> - Read files from server filesystem</li> pg_write_server_files</code> - Write files to server filesystem</li> pg_execute_server_program</code> - Execute programs on server</li> </ul> And specialised use cases pg_create_subscription</code> - Logical replication management</li> pg_use_reserved_connections</code> - Connection reservation (PostgreSQL 16+)</li> </ul> Why Use Predefined Roles?</a> </h2> The primary benefit of predefined roles is expanding the pool of users who can safely manage PostgreSQL databases. Traditional PostgreSQL administration created an artificial binary choice: either users had basic access with limited capabilities, or they required full superuser privileges for operational tasks. This forced many organizations to grant excessive permissions simply to perform routine operations like monitoring, backups, or maintenance. Predefined roles break this limitation by allowing more granular control over operational tasks without distributing superuser privileges. Instead of having a small number of highly privileged superusers handling all administrative work, organizations can now delegate specific capabilities to appropriate operational roles - monitoring teams get monitoring access, backup services get data reading capabilities, and maintenance scripts get precisely the permissions they need. The deeper benefit is abstraction. Not only do you switch from manually managing permissions on individual system objects, you switch to managing logical capability sets. This creates a clear separation between what you want to allow and how it's actually implemented. The advantage is clearly demonstrated in following example -- granting select on each schema separately GRANT USAGE ON SCHEMA finance, hr, app, audit TO analytics_team; GRANT SELECT ON ALL TABLES IN SCHEMA finance TO analytics_team; GRANT SELECT ON ALL SEQUENCES IN SCHEMA finance TO analytics_team; -- versus one time setup GRANT pg_read_all_data TO analytics_team;</code></pre> The predefined roles approach automatically covers: All current tables, views and sequences</li> All schemas</li> Future objects created after the GRANT</code></li> </ul> And the good news is the benefits don't end here. The real thing on operational level comes with the scope of those predefined roles - they apply at cluster level (instead on database level). Only notable exception is role pg_database_owner</code> which we will cover next. Before going there, let's briefly mention second benefit, and that's the fact any new operational features in PostgreSQL will be covered by those predefined roles automatically. The Evolution of Predefined Roles</a> </h2> Understanding when and why predefined roles were introduced provides important insight into PostgreSQL's operational priorities and helps explain some of the design decisions. The journey from a single role in PostgreSQL 9.6 to today's comprehensive set reflects real-world pain points that PostgreSQL developers encountered in production environments. PostgreSQL 9.6 (2016) introduced first predefined role pg_signal_backend</code> to address common frustration - inability to cancel running query without giving away superuser rights. -- This became possible in 9.6 without superuser privileges SELECT pg_cancel_backend(12345); -- Cancel a query SELECT pg_terminate_backend(12345); -- Terminate a session</code></pre> PostgreSQL 10 (2017) brought biggest expansion of predefined roles with, adding four monitoring specific roles to improve database observability, and support growing importance of database monitoring in production environments. -- Create the monitoring user CREATE USER postgres_exporter WITH PASSWORD 'monitoring_password'; -- Grant comprehensive monitoring access GRANT pg_monitor TO postgres_exporter;</code></pre> Giving access to variety of metrics and settings. With single GRANT</code> you can access data: From pg_read_all_settings (inherited via pg_monitor) to SELECT * FROM pg_settings</code> - Configuration parameters for alerting on misconfigurations and configuration metrics like shared_buffers, work_mem, max_connections</li> </ul> From pg_read_all_stats (inherited via pg_monitor): -- Database-wide statistics (connections, transactions, blocks read/written) SELECT * FROM pg_stat_database -- Current query activity and connection states SELECT * FROM pg_stat_activity -- Replication lag and status SELECT * FROM pg_stat_replication -- Background writer performance SELECT * FROM pg_stat_bgwriter -- Lock contention monitoring SELECT * FROM pg_locks</code></pre> From pg_stat_scan_tables (inherited via pg_monitor): Enhanced table statistics collection with sequential scan information</li> More detailed I/O statistics for table access patterns</li> </ul> The pg_monitor</code> role is particularly clever in its design. Rather than being a single monolithic permission set, it's actually a composite of the other three roles. This allows for granular access when needed - you can grant pg_read_all_stats for basic monitoring without including configuration access, or grant the full pg_monitor bundle for comprehensive monitoring capabilities. PostgreSQL 11 (2018) laid the foundation for secure file system operations with the introduction of three powerful roles that fundamentally changed how PostgreSQL handles server-side file access: pg_read_server_files</code></li> pg_write_server_files</code></li> pg_execute_server_program</code></li> </ul> Addressing many long-standing issues with providing file system access, like ETL processes that needs to read CSV files and data processing pipelines often involving external programs. The file system roles represented a significant security advancement. Before PostgreSQL 11, operations like server-side COPY FROM file were superuser-only, forcing administrators to either grant excessive privileges or find workarounds. PostgreSQL 14 (2021) introduced three significant roles that addressed different data access patterns: pg_read_all_data</code></li> pg_write_all_data</code></li> pg_database_owner</code></li> </ul> The first two solved a common backup and analytics challenge. The special case is pg_database_owner</code> which we will cover later. -- create ETL extraction user CREATE USER etl_extractor WITH PASSWORD 'extract_password'; GRANT pg_read_all_data TO etl_extractor; -- create ETL loader with write access CREATE USER etl_loader WITH PASSWORD 'loader_password'; GRANT pg_write_all_data TO etl_loader;</code></pre> One big caveat that comes with those ALL data roles is that they still (despite what the name might suggest) respect Row Level Security policies (RLS). This is design choice of PostgreSQL which prioritises security by default - even for users with broad data access. The only way to avoid RLS is to explicitly grant BYPASSRLS</code> role attribute. PostgreSQL 15 (2022) expanded set with pg_checkpoint</code>, reflecting the reality that checkpoint operations are sometimes needed for maintenance but don't require full superuser privileges. PostgreSQL 16 (2023) expanded the list by introduction of pg_use_reserved_connections</code> - resolving a critical problem with high-traffic databases, where previously only superusers could access reserved connections. Introduction of this roles expanded the pool of potential users who might be able to resolve incidents or high-load situations. For example with pg_signal_backend</code> grant.</li> pg_create_subscription</code> - with ability to create Logical Replication</a> subscriptions and confirming the place of PostgreSQL in the distributed database scenarios.</li> </ul> PostgreSQL 17 (2024) finally saw pg_maintain</code>, role that had a turbulent development history (initially committed for PostgreSQL 16), allowing non-superusers to perform crucial maintenance tasks like VACUUM, ANALYZE, REINDEX, REFRESH MATERIALIZED VIEW, CLUSTER, and LOCK TABLE. The pg_maintain</code> role is particularly valuable for: Automated maintenance scripts that can now run with minimal privileges, reducing the security footprint of scheduled maintenance operations.</li> Operational teams managing multiple databases who need consistent maintenance capabilities without requiring superuser access to each database.</li> Cloud and managed environments where maintenance operations need to be delegated to operations staff without granting broader system access.</li> </ul> PostgreSQL 18 (2025) continues the trend with pg_signal_autovacuum_worker</code> expanding the ability for non-superusers to signal autovacuum workers specifically, enabling them to cancel vacuum operations on specific tables or terminate autovacuum sessions that may be causing performance issues during critical periods. The Magic of pg_database_owner</a> </h2> If you paid attention, there's one role that stands apart from all others - pg_database_owner</code>. While it does not provide any specific privileges that wouldn't be possible without superuser access, it serves as a marker for the owner of the database. This role is crucial for maintaining database ownership and ensuring that the database is managed by the correct user. Before PostgreSQL 15 you would have public</code> schema owned by postgres</code> user. demo=> \dn List of schemas Name | Owner --------+---------- public | postgres</code></pre> It came with the baggage of relaxed default permission, allowing every user (PUBLIC) to create objects. Making this a maintenance nightmare. Starting with PostgreSQL 15 and above, the public</code> schema is owned by pg_database_owner</code> role. demo=# \dn List of schemas Name | Owner --------+------------------- public | pg_database_owner</code></pre> Unlike any other roles, pg_database_owner</code> has a unique behaviour - it membership changes with the current database. As the code snippet above shows the "shape-shifting nature" of this role fixed the public</code> schema and complexity of the permissions associated with it. Another special behaviour is that the role itself does not come with any permissions. Let that sink in - zero permissions. Its power only comes from what you grant it or inherit. It opens up system for elaborate template inheritance. -- define functionality in template1 \c template1 postgres CREATE OR REPLACE FUNCTION db_owner_stats() RETURN ... SECURITY DEFINER; GRANT EXECUTE ON FUNCTION db_owner_stats() TO pg_database_owner; -- and let it automatically apply to all new databases CREATE DATABASE app_prod OWNER first_app; CREATE DATABASE demo_prod OWNER second_app; -- both first_app and second_app automatically inherit db_owner_stats function on their databases</code></pre> And when we are talking about magic, the special status of the role prevents this role to be granted to any other role. demo=# GRANT pg_database_owner TO labs_app; ERROR: role "pg_database_owner" cannot have explicit members</code></pre>Conclusion</a> </h2> Predefined roles transform PostgreSQL administration from ad-hoc permission management into systematic capability delegation. They solve the fundamental problem of needing operational access without operational trust. The evolution continues - each PostgreSQL release adds new predefined roles addressing real-world operational challenges. The pattern is established: identify common pain points, create focused roles, eliminate the need for superuser privileges in routine operations. The next time you reach for superuser access to solve an operational problem, check if PostgreSQL has already provided a predefined role for exactly that purpose. You'll likely find it already has one. Beyond the Basics of Logical Replication 2025-06-24T07:10:00+00:00 With First Steps with Logical Replication</a> we set up a basic working replication between a publisher and a subscriber and were introduced to the fundamental concepts. In this article, we're going to expand on the practical aspects of logical replication operational management, monitoring, and dive deep into the foundations of logical decoding. Initial Data Copy</a> </h2> As we demonstrated in the first part, when setting up the subscriber, you can choose (or not to) to rely on initial data copy using the option WITH (copy_data = false)</code>. While the default copy is incredibly useful behavior, this default has characteristics you should understand before using it in a production environment. The mechanism effectively asks the publisher to copy the table data by taking a snapshot (courtesy of MVCC), sending it to the subscriber, and thanks to the replication slot "bookmark," seamlessly continues streaming the changes from the point the snapshot was taken. Simplicity is the key feature here, as a single command handles the snapshot, transfer, and transition to ongoing streaming. The trade-off you're making is when it comes to performance, solely due to the fact that it's using a single process per table. Behind the scenes the initial synchronization is done using COPY</code> given the best possible performance. While it works almost instantly for majority of the tables, you will encounter notable delay and overhead when dealing with tables with gigabytes of data. Although parallelism can be controlled by the max_sync_workers_per_subscription</code> configuration parameter, it still might leave you waiting for hours (and days) for any real-life database to get replicated. You can monitor whether the tables have already been synchronized or are still waiting/in progress using the pg_subscription_rel</code> catalog. SELECT srrelid::regclass AS table_name, srsubstate FROM pg_subscription_rel;</code></pre> Where each table will have one of the following states: i</code> not yet started</li> d</code> copy is in progress</li> s</code> syncing (or waiting for confirmation)</li> r</code> done & replicating</li> </ul> Luckily, the state r</code> indicates that the streaming of changes can start even if not all tables are synchronized. Nevertheless, the replication slot retains the LSN position, which means that the publisher will retain WAL files until the subscriber has caught up. Manual Synchronization for Production Workloads</a> </h2> As mentioned above, the implicit copy might bring performance trade-offs that are simply too much for production use, unless you consider it while designing your logical replication topology. In all other scenarios, going with manual synchronization is the way forward. The entire process has one leading principle: the data you load manually must be consistent with the point in time from which the logical replication stream begins. This is achieved by creating a logical replication slot before (or synchronized with) the point at which you restore the data (PITR) and enabling it once the data is transferred. Only this will allow you to correctly apply all subsequent changes on the subscriber. There are several ways to achieve this, and you will have to evaluate the mechanism based on the available constraints in your particular use case: Use the backup & restore mechanism that supports Point-in-Time Recovery (PITR) if the amount of data to be synchronized approaches the total size of the backed-up data (i.e., most of the tables).</li> Orchestrate the data change ingestion on the publisher with the backup, by stopping the incoming changes (for example, during a planned maintenance window) and only restoring them when a consistent snapshot is available to a known LSN. This is for cases where you can control table ingestion and the amount of data being transferred fits the maintenance window.</li> If none of this is available, you might be able to manually advance replication slots to a predefined LSN. Please understand, this should be a last resort as it might be considered expert domain.</li> </ol> The recommended way for the most reliable backup and restore handling is the use of pgBackRest, as it allows you to restore a particular backup in advance and apply only changes needed later to advance to the selected Point-in-Time, hence significantly reducing the time required for the initial sync. On the other hand, if you are only using a small portion of the data within the backup on your subscriber, it might create resource constraints not acceptable for the setup. While pg_dump can't be considered a reliable backup tool, it might help you in case you are talking about making a consistent backup of a small subset of tables using the "synchronized snapshot" created using pg_export_snapshot</code>. pg_dump -d publisher_db --snapshot="000004A2-1" --data-only -Fc -f initial_data.dump</code></pre> Please note, the topic of consistent backups and Point-in-Time-Restore is way beyond this logical replication guide. Monitoring Logical Replication</a> </h2> Once you set up your publishers and subscribers and have the initial data in place, it's time to start thinking about how to keep that process running. For day-to-day operations, the basic building block for monitoring is the pg_replication_slots</code> catalog. Sample query: SELECT slot_name, plugin, slot_type, active, pg_size_pretty( pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) ) AS wal_retained_size FROM pg_replication_slots WHERE slot_type = 'logical';</code></pre> giving you result: -[ RECORD 1 ]-----+----------------------------- slot_name | my_sample_subscription plugin | pgoutput slot_type | logical active | t wal_retained_size | 49 MB</code></pre> You should look for two things: active</code> indicating whether the slot is being used by any connection. While this might be acceptable for shorter durations, it might be a sign that the subscriber is down or not configured properly.</li> The computed wal_retained_size</code> is a critical metric. An increasing value indicates problems consuming changes by the subscriber, and your cluster might be at risk of running out of disk space.</li> </ul> Both values are important as they might indicate problems you need to pay attention to. The active</code> flag might indicate dangling slots, where the subscriber has been destroyed (or simply wiped) without properly dropping the subscription—hence leaving the replication slot behind. In such cases, the only direct response is dropping the slot by its name: SELECT pg_drop_replication_slot('my_dangling_slot');</code></pre> If you want to dig deeper, you can also use pg_stat_replication</code> that provides more (albeit volatile) data about the replication status. It can give you other metrics, like replay_lag</code>. To wrap up the monitoring part, you can either design your custom checks or use a modified version of the query above: SELECT slot_name, CASE WHEN active THEN 1 ELSE 0 END as active_status, pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) as wal_bytes_retained, extract(epoch from now()) as time FROM pg_replication_slots WHERE slot_type = 'logical';</code></pre> as part of a Grafana alerting rule: Alert Rule: "Inactive Replication Slot" Condition: - Query: active_status - Reducer: last() - Evaluator: below 1 - For: 30m # if inactive for 30+ minutes Additional conditions (AND): - Query: wal_bytes_retained - Reducer: last() - Evaluator: above 1073741824 # 1GB limit </code></pre> Nevertheless, you need to evaluate the monitoring details applicable for your particular use case(s). There might also be other options, for example, if your architecture allows for it, to use max_slot_wal_keep_size</a> and mark the replication slot invalid and release the WAL files when a particular slot falls behind the configured value (for example, 100GB). While this might sound reasonable, you must considered whatever your application or use case is able to recover from such a data loss. Evolving Publications</a> </h2> Going back to the first part of our journey into logical replication, there's an important distinction to make. While we have described the initial use case, in real-life your publisher schema will change as time goes by. Tables will be created and included implicitly (via FOR ALL TABLES</code>) or explicitly added/removed. ALTER PUBLICATION my_publication ADD TABLE public.new_table; ALTER PUBLICATION my_publication DROP TABLE public.old_table;</code></pre> But what happens when you add a new table? For the existing subscription, the answer is not much at all. The only way for the subscriber to reflect the changes in the underlying configuration is to refresh the subscription. It's a process where PostgreSQL compares the current subscription table list with the publication table list, (if configured to do so) starts the initial synchronization process and once finished goes to apply streamed changes. As mentioned, you can refresh a subscription with or without copying the initial data. The copy_data</code> option controls this behavior and is currently the only supported setting. ALTER SUBSCRIPTION my_subscription REFRESH PUBLICATION; ALTER SUBSCRIPTION my_subscription REFRESH PUBLICATION WITH (copy_data = false);</code></pre> When you remove a table from the publisher, the situation is much simpler as the changes for that particular table are no longer considered for logical decoding. But what happens if you were to add a new table to the publication without refreshing the subscription? Despite the WAL files including changes for all tables, and the logical decoding process will process all published table changes, there will be no additional WAL retention to consider. The subscriber will advance confirmed_flush_lsn</code> as before, simply because the initial state for a newly added table has not yet been recorded and streamed changes will be (for that moment) ignored. Logical Decoding</a> </h2> If you paid attention throughout our journey or already worked a bit with logical replication, you might have come across something called the pgoutput</code> plugin. This is the built-in default mechanism responsible for logical decoding—a mechanism that goes through the WAL stream and transforms it into a higher-level format. An example is, instead of physical replication—byte X at offset Y, in page Z—the logical decoding translates the WAL stream to row-level changes. It uses the context awareness of the source database to understand the changes and change from object identifier to the schema, table, and column name, as well as the respective values (both old and new). It also provides a way to assemble the changes into the correct transaction flow. The plugin's job is to format that output in a particular way and by itself is not aware of: any transport layer details</li> replication slots</li> any retry logic</li> </ul> PostgreSQL offers a robust framework for logical decoding. The built-in plugins are just a start, and developers can create custom output plugins. But not only that—we can also look inside the logical decoded stream manually. Let's consider this example: -- create new publication CREATE PUBLICATION all_data FOR ALL TABLES; -- create new replication slot with pgoutput plugin SELECT pg_create_logical_replication_slot('plugin_demo', 'pgoutput'); -- sample table CREATE TABLE IF NOT EXISTS demo_table ( id INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, name TEXT, email TEXT ); -- generate some changes with insert/update/delete INSERT INTO demo_table (name, email) VALUES ('John Doe', 'john@example.com'); UPDATE demo_table SET email = 'doe@example.com' WHERE id = 1; DELETE FROM demo_table WHERE id = 1; -- get binary changes SELECT * FROM pg_logical_slot_get_binary_changes('plugin_demo', NULL, NULL, 'proto_version', '1', 'publication_names', 'all_data'); -- drop the replication slot and the publication SELECT pg_drop_replication_slot('plugin_demo'); DROP TABLE demo_table; DROP PUBLICATION all_data;</code></pre> Giving us a peek into the recently performed changes (output is shortened): lsn | xid | data ------------+------+--------------------------------- 1/D6997380 | 1038 | \x4200000001d69974180 1/D6997380 | 1038 | \x52000042057075626c696300646... 1/D6997380 | 1038 | \x49000042054e000374000... 1/D6997448 | 1038 | \x430000000001d6997... 1/D6997448 | 1039 | \x4200000001d6997... 1/D6997448 | 1039 | \x49000042054e0.... 1/D6997510 | 1039 | \x430000000001d6...</code></pre> Which is correct, but won't help us to demonstrate the flow. Don't forget pgoutput</code> is the actual plugin used for logical decoding, and is binary. But don't despair, PostgreSQL offers the test_decoding</code> plugin which decodes the changes into a human-readable text representation. -- setup the publication CREATE PUBLICATION all_data FOR ALL TABLES; -- create replication slot with test_decoding plugin SELECT pg_create_logical_replication_slot('plugin_demo', 'test_decoding'); CREATE TABLE IF NOT EXISTS demo_table ( id INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, name TEXT, email TEXT ); -- generate some changes with insert/update/delete INSERT INTO demo_table (name, email) VALUES ('John Doe', 'john@example.com'); UPDATE demo_table SET email = 'doe@example.com' WHERE id = 1; DELETE FROM demo_table WHERE id = 1; -- get human readable changes SELECT * FROM pg_logical_slot_get_changes('plugin_demo', NULL, NULL); -- drop the replication slot and the publication SELECT pg_drop_replication_slot('plugin_demo'); DROP TABLE demo_table; DROP PUBLICATION all_data;</code></pre> This time the output is actually going to be helpful. lsn | xid | data ------------+------+----------------------------------------------------------------------------------------------------- 1/D69CCC30 | 1048 | BEGIN 1048 1/D69CCC98 | 1048 | table public.demo_table: INSERT: id[integer]:1 name[text]:'John Doe' email[text]:'john@example.com' 1/D69CCDC0 | 1048 | COMMIT 1048 1/D69CCDC0 | 1049 | BEGIN 1049 1/D69CCDC0 | 1049 | table public.demo_table: UPDATE: id[integer]:1 name[text]:'John Doe' email[text]:'doe@example.com' 1/D69CCE50 | 1049 | COMMIT 1049 1/D69CCE50 | 1050 | BEGIN 1050 1/D69CCE50 | 1050 | table public.demo_table: DELETE: id[integer]:1 1/D69CCEC0 | 1050 | COMMIT 1050</code></pre> Just reading it, you can easily follow the sequence of the transaction (auto-commit in psql) and track individual changes. We can also reiterate the importance of REPLICA IDENTITY</code> with this example. If you double-check the DDL for demo_table</code>, you will see IDENTITY</code> used, which by its definition will create a primary key. With default replica identity, you are effectively relying on that as the primary source of identifying data. You can use the test_decoding</code> plugin to demonstrate the verbosity it will create. (In this example, leaving the setup and tear down.) ALTER TABLE demo_table REPLICA IDENTITY FULL; INSERT INTO demo_table (id, name, email) OVERRIDING SYSTEM VALUE VALUES (999, 'John Doe', 'john@example.com'); UPDATE demo_table SET email = 'doe@example.com' WHERE id = 999; DELETE FROM demo_table WHERE id = 999; SELECT * FROM pg_logical_slot_get_changes('plugin_demo', NULL, NULL);</code></pre> And you can observe the changes. While by its nature INSERT</code> gives you the same output (all values are being sent), UPDATE</code> and DELETE</code> are no longer relying on the primary key. Instead, they have to provide all values, for the UPDATE</code> case both old and new data: lsn | 1/D69F8770 xid | 1062 data | table public.demo_table: UPDATE: old-key: id[integer]:999 name[text]:'John Doe' email[text]:'john@example.com' new-tuple: id[integer]:999 name[text]:'John Doe' email[text]:'doe@exa mple.com'</code></pre> and for DELETE</code> the old data. lsn | 1/D69F8828 xid | 1063 data | table public.demo_table: DELETE: id[integer]:999 name[text]:'John Doe' email[text]:'doe@example.com'</code></pre> Making it obvious how misconfigured replica identity might increase the amount of data decoded and replicated. Fine-grained Publication Control</a> </h2> Before we introduce the advanced replication topologies in later parts of this series, let's have a look at how you can precisely control what data to replicate to the subscribers. Fine-grained control helps you to control replication overhead, reduce network traffic, enhance security, and ensure the subscribers only receive the data they need. The first option to filter the data is by explicit column_list</code>, allowing you to exclude sensitive (PII or similar) or unnecessary data. When selecting the columns, you should include the primary key or the columns behind replica identity; if not, PostgreSQL will add them automatically. CREATE PUBLICATION hr_analytics FOR TABLE hr.employees (employee_id, first_name, last_name, department, start_date);</code></pre> Similar to column selection, you can control what operations a publication replicates. The available options are 'insert', 'update', 'delete', and 'truncate'. CREATE PUBLICATION hr_analytics FOR TABLE hr.employees (employee_id, first_name, last_name, department, start_date) WITH (publish = 'insert,delete');</code></pre>Securing Logical Replication</a> </h2> In the previous section, we hit a crucial element of logical replication—limiting access to the publication. Otherwise, what purpose would it have to limit fields published if another publication would still offer them without restrictions? So far in this series, we have relied for simplicity on SUPERUSER</code> access to create and manage publications. To design logical replication for real-life production use, you need a least-privilege model. Luckily for us, the PostgreSQL security model for logical replication follows the same rules as for regular queries. If a user can't SELECT from a specific table, or specific columns or rows, they can't include them in a publication either. Therefore, this section assumes you are already familiar with the PostgreSQL permission model. The high-level overview of the permissions needed is that the replication user must have CONNECT</code> permission on the database, USAGE</code> on the schema, and SELECT</code> on specific tables (columns). You also need to ensure (if applicable) row-level security (RLS) is correctly managed. Using the fine-grained example used above, we can demonstrate the setup. First, creating a role with LOGIN</code> and REPLICATION</code> clauses and granting database and schema access: CREATE ROLE hr_analytics_role WITH LOGIN REPLICATION PASSWORD 'a_strong_password'; GRANT CONNECT ON DATABASE my_publisher_db TO hr_analytics_role; GRANT USAGE ON SCHEMA hr TO hr_analytics_role;</code></pre> The crucial step is to correctly define the GRANT</code> for selecting the data. GRANT SELECT (employee_id, first_name, last_name, department, hire_date) ON hr.employees TO hr_analytics_role;</code></pre> This approach ensures that even if someone gains access to the replication role, they cannot expose data beyond what was explicitly granted. The publication will respect these column-level restrictions, creating a secure boundary for your replicated data. Wrap Up</a> </h2> Today we've moved beyond the basics of logical replication to cover the essential practices for production environments and expanded the understanding of how it actually works. This article is second part of the upcoming guide Mastering Logical Replication in PostgreSQL</a>. If you are interested in the topic, please consider subscribing to get the latest articles as they are published. First steps with Logical Replication in PostgreSQL 2025-06-11T00:00:00+00:00 Most applications start with a single PostgreSQL database, but over time, the need to scale out, distribute the load, or integrate naturally arises. PostgreSQL's logical replication is one of the features that meets these demands by streaming row-level changes from one PostgreSQL instance to another, all using a publish-subscribe model. Logical replication is more than an advanced feature; it provides a flexible framework you can build on to further distribute and integrate PostgreSQL within your architecture. In this article, we will start with the foundation, explore the core ideas behind logical replication, and learn how to use it. Physical vs. Logical Replication</a> </h2> Before we can dive deeper, let's understand the role of replication in PostgreSQL and how it's built on top of the Write-Ahead Log (WAL). The WAL is a sequential, append-only log that records every change made to the cluster data. For durability purposes, all modifications are first written to the WAL and only then permanently written to disk. This allows PostgreSQL to recover from crashes by replaying logged changes. Versioned changes, necessitated by concurrent transactions, are managed through Multi-Version Concurrency Control (MVCC). Instead of overwriting data directly, MVCC creates multiple versions of rows, allowing each transaction to see a consistent snapshot of the database. It is the WAL that captures these versioned changes along with the transactional metadata to ensure data consistency at any given point in time. Physical replication is built directly on the Write-Ahead Log. It enables streaming of the binary WAL data from the primary server to one or more standby servers (replicas), effectively creating a byte-for-byte copy of the entire cluster. This requirement makes the replicas read-only, making them ideal candidates for failover or scaling purposes. Compared to this, Logical replication, while also being built on top of the WAL data, takes a fundamentally different approach. Instead of streaming raw change data, logical replication decodes the WAL into logical, row-level changes – such as INSERT</code>, UPDATE</code>, and DELETE</code> – and only then sends them to the subscribers using a Publish-Subscribe model. Compared to physical replication, this allows selective replication, while allowing writable subscribers which are not strictly tied to a single publisher. This might increase the flexibility of available setups, however logical replication does not replicate DDL changes. </th> Physical</th> Logical</th></tr></thead> Data Streamed</td> Binary WAL segments</td> Row-level SQL changes</td></tr> Scope</td> Byte-for-byte stream</td> Selected tables</td></tr> Node Type</td> Read-only standby</td> Fully writable instance</td></tr> PostgreSQL Version</td> All servers must match major version</td> Supports across versions</td></tr> Database Schema</td> Changes automatically replicated</td> Changes must be applied on subscriber(s) separately</td></tr> Use Case</td> Failover, high-availability, and read scaling</td> Integration, zero-downtime upgrades and schema migrations, complex topologies</td></tr> </tbody></table> Physical replication is your go-to for high availability and disaster recovery, where you want a fast, exact copy of the entire database cluster that can take over in case of failure. It’s simple to set up and very efficient but limited in flexibility. Logical replication shines when you need fine-grained control over what data is replicated, require writable replicas, or want to integrate PostgreSQL with other systems or versions. It’s ideal for zero-downtime upgrades, multi-region deployments, and building scalable, modular architectures. Setting up Logical Replication</a> </h2> Enough of boring theory for now. To get started, you will need two instances of PostgreSQL – it's up to you whether you provision two virtual machines or two clusters running on the same computer. We will call one publisher and the other subscriber. Preparing the Publisher</a> </h3> First, you need to prepare the publisher to emit the logical changes. You will need to modify postgresql.conf</code> (or add particular conf.d</code> configuration) with the following parameters: wal_level = logical max_replication_slots = 10 max_wal_senders = 10</code></pre> Your publisher also needs to be reachable by the subscriber, in most cases via a TCP socket. listen_addresses = '*'</code></pre> Here, the configuration is: wal_level</code></a> set to logical</code> is a crucial piece of the configuration. It tells PostgreSQL how much information to write to the WAL, in this case, to support logical decoding.</li> max_replication_slots</code></a> defines the maximum number of replication slots that can be created on the server. Each logical replication subscription needs its own slot.</li> max_wal_senders</code></a> should be set high enough to accommodate all expected concurrent replication connections from subscribers. Each active replication subscription consumes a wal_sender slot.</li> </ul> should match the maximum number of connections from the publisher (primary) to subscribers or replication clients. The number should be higher than or equal to the number of replication slots to avoid replication problems. You will also need to configure client authentication in pg_hba.conf</code> to allow the subscriber to connect for replication (for simplification, we allow all users): # TYPE DATABASE USER ADDRESS METHOD host replication all subscriber_ip/32 scram-sha-256</code></pre> While for the purposes of the article we will assume the use of a superuser, making it convenient: CREATE USER my_user_name SUPERUSER PASSWORD 'my_secure_password';</code></pre> It is recommended to use a dedicated replication user for real-life deployments: CREATE USER replication_user WITH REPLICATION ENCRYPTED PASSWORD 'my_secure_password';</code></pre> Once configured, restart PostgreSQL for the configuration changes to take effect. After that, connect to the server and prepare the environment. -- example for psql CREATE DATABASE logical_demo_publisher; \c logical_demo_publisher;</code></pre> Create a sample table and seed initial data: CREATE TABLE products ( id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, name TEXT NOT NULL, category TEXT, price DECIMAL(10, 2) NOT NULL, stock_quantity INT DEFAULT 0, created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP, updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP, description TEXT, is_active BOOLEAN DEFAULT TRUE ); -- seed some data (10 records) INSERT INTO products ( name, category, price, stock_quantity, description, is_active ) SELECT 'Product Batch ' || s.id AS name, CASE (s.id % 5) WHEN 0 THEN 'Electronics' WHEN 1 THEN 'Books' WHEN 2 THEN 'Home Goods' WHEN 3 THEN 'Apparel' ELSE 'Miscellaneous' END AS category, ROUND((RANDOM() * 500 + 10)::numeric, 2) AS price, FLOOR(RANDOM() * 200)::int AS stock_quantity, 'Auto-generated description for product ID ' || s.id || '. Lorem ipsum dolor sit amet, consectetur adipiscing elit.' AS description, (s.id % 10 <> 0) AS is_active FROM generate_series(1, 10) AS s(id);</code></pre> The final step for the publisher is to create a publication to define what data we want to publish. CREATE PUBLICATION my_publication FOR TABLE products;</code></pre>Preparing the Subscriber</a> </h3> Next, we will use our subscriber instance to receive the logical changes. There's no need to make any configuration changes just for subscribing, as it's not emitting changes itself (please note the "just"). And create the target database and schema. The schema creation is an important part as: -- example for psql CREATE DATABASE my_subscriber_db; \c my_subscriber_db; CREATE TABLE products ( id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, name TEXT NOT NULL, category TEXT, price DECIMAL(10, 2) NOT NULL, stock_quantity INT DEFAULT 0, created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP, updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP, description TEXT, is_active BOOLEAN DEFAULT TRUE );</code></pre> Now we have the foundation for testing logical replication. The simplest way is to create a subscription using connection details and the name of the publication to subscribe to. CREATE SUBSCRIPTION my_subscription CONNECTION 'host=publisher_ip_address port=5432 user=your_replication_user password=my_secure_password dbname=logical_demo_publisher' PUBLICATION my_publication;</code></pre> my_subscription</code> should be a descriptive name for your subscription.</li> CONNECTION</code> defines how to connect to the publisher (which is a regular connection string and could also be a service</a>).</li> PUBLICATION</code> specifies the name of the publication you created earlier on the publisher.</li> </ul> If your connection was correct, the subscription will by default start with the initial data sync and listen for incoming changes. If you have used the queries above, you can validate that the data is now on the subscriber. # SELECT count(1) FROM products; count --------- 10 (1 row)</code></pre> The number of records should match the number of records created using generate_series</code> above (10 in our example). You can go ahead and insert a single row again on your publisher instance and validate the data being replicated to the subscriber. Congratulations! You have set up your first logical replication in PostgreSQL! Core Concepts of Logical Replication</a> </h2> That was easy, right? And that's the goal. Now that you've seen logical replication in action, let's delve deeper into the core concepts that made this work. Publication</a> </h3> As the name implies, a publication is where it all starts. It's essentially a catalogue of data you offer from the publisher. You can publish</a> a number of objects: --- specific tables CREATE PUBLICATION my_publication FOR TABLE products, orders; --- everything in your database (make sure you really want to do this) CREATE PUBLICATION all_data FOR ALL TABLES; --- replicate specific columns only (PostgreSQL 15 and higher) CREATE PUBLICATION generic_data FOR TABLE products (id, name, price); --- filter published data (PostgreSQL 15 and higher) CREATE PUBLICATION active_products FOR TABLE products WHERE (is_active = true); --- filter different operations CREATE PUBLICATION my_publication FOR TABLE products, orders WITH (publish = 'insert'); --- or you can mix & match it to get exactly what you need CREATE PUBLICATION eu_customers FOR TABLE customers (id, email, country, created_at) WHERE (country IN ('DE', 'FR', 'IT')), TABLE orders (id, customer_id, total_amount) WHERE (total_amount > 100) WITH (publish = 'insert, update');</code></pre> As you can see, logical replication really gives you quite a lot of options and is far from the rigid, byte-for-byte copying of physical replication. This is exactly the characteristic that allows you to build complex topologies. Once you set up your publication(s), you can review it: --- in psql \dRp \dRp+ my_publication --- using pg_catalog SELECT * FROM pg_publication_tables WHERE pubname = 'my_publication';</code></pre> From these options, it's easy to think of a publication as a customisable data feed. Subscription</a> </h3> The other side of logical replication is a subscription. It defines what and how to consume events from a publisher. You subscribe to a publication using connection details and a number of options. --- basic subscription with full connection detail CREATE SUBSCRIPTION my_subscription CONNECTION 'host=publisher_host port=5432 user=repl_user password=secret dbname=source_db' PUBLICATION my_publication; -- subscription using service definition CREATE SUBSCRIPTION realtime_only CONNECTION 'service=my_source_db' PUBLICATION my_publication;</code></pre> The default behaviour of the subscription is to copy existing data, start immediately, and continue with streaming data changes. Once you start experimenting with logical replication, you can control that behaviour. --- subscribe without initial copy of the data CREATE SUBSCRIPTION streaming_only CONNECTION 'service=my_source_db' PUBLICATION my_publication WITH (copy_data = false); --- or defer the start for later CREATE SUBSCRIPTION manual_start CONNECTION 'service=my_source_db' PUBLICATION my_publication WITH (enabled = false);</code></pre> You can monitor your subscriptions: --- in psql \dRs \dRs+ my_subscription --- or their status using pg_catalog SELECT subname, received_lsn, latest_end_lsn, latest_end_time FROM pg_stat_subscription;</code></pre>Replication Slot</a> </h3> Publications and subscriptions establish the data flow, but there's a critical piece missing: how does the publisher keep track of multiple subscribers reading the WAL at different speeds? Replication slots are the answer. They act as a persistent booking in the WAL stream that tracks exactly where each subscriber is (or was last time), ensuring no changes are lost even if the connection stops. Let's have a look at what a replication slot can tell us about itself. # SELECT * FROM pg_replication_slots; -[ RECORD 1 ]-------+--------------- slot_name | my_subscription plugin | pgoutput slot_type | logical datoid | 16390 database | my_db_source temporary | f active | t active_pid | 3162 xmin | catalog_xmin | 777 restart_lsn | 0/1DEFBF0 confirmed_flush_lsn | 0/1DEFC28 wal_status | reserved safe_wal_size | two_phase | f inactive_since | conflicting | f invalidation_reason | failover | f synced | f</code></pre> The most interesting attributes are: slot_name</code></li> plugin</code> used for logical replication</li> slot_type</code> confirming it's for logical replication</li> active</code> indicates whether there's a subscriber reading from this slot</li> </ul> And most important of those being: restart_lsn</code> identifying the LSN where this slot "holds" WAL files from being released</li> confirmed_flush_lsn</code> being the last LSN position the subscriber confirmed it has successfully processed (i.e., applied).</li> </ul> While we won't go into details about WAL, it's enough to say LSN (Log Sequence Number) is a unique address or position in the WAL that identifies the position in the stream of database changes. In most cases, you don't have to create replication slots manually – subscriptions create and manage them. The persistent nature of the slots guarantees they survive the restart of both publisher and subscriber. Please note, if the subscription is not actively consuming the changes, the publisher PostgreSQL won't release WAL files that contain changes a slot has not consumed. This prevents data loss, but can easily fill up disk if any of the subscribers fall too far behind. You can monitor the WAL retention per each replication slot. # SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS wal_retained FROM pg_replication_slots; -[ RECORD 1 ]+-------------------- slot_name | my_subscription active | t wal_retained | 265 MB</code></pre>Replication Identity</a> </h3> As we have already covered, logical replication works with row-level changes, such as INSERT</code>, UPDATE</code>, and DELETE</code>. And not all operations are equal. While INSERT</code> is relatively straightforward (a new row is sent to the subscriber), for both UPDATE</code> and DELETE</code> operations, PostgreSQL needs to be able to uniquely identify the target row to modify on the subscriber. This is managed by the REPLICA IDENTITY</code> property of each published table. By default, if a table has a primary key (it has, right?), or you can specify it USING INDEX</code> for a unique index, or as an alternative, specify FULL</code> (should be generally avoided) to all old column's row values to find the target row to update. Please note, PostgreSQL might default to REPLICA IDENTITY FULL</code> if no primary key or suitable key index exists. Whenever possible, please always: Choose REPLICA IDENTITY DEFAULT</code> (for primary key), giving you the most efficient choice. If you don't have a primary key, please consider using it (a surrogate index is always an option).</li> REPLICA IDENTITY USING INDEX index_name</code> if there would be a better (or smaller) unique index matching the business logic or architecture of your database model.</li> </ul> There are special cases of what not to do when it comes to replication identity, but we will touch on those cases in the advanced section of this guide. Schema Changes</a> </h3> Although it was already mentioned, it's vital to reiterate that logical replication, as it streams only row-level changes rather than full WAL segments, does not automatically replicate Data Definition Language (DDL) changes. Any schema changes made on the publisher must be manually applied to all subscribers before data reflecting the change is replicated. We will cover schema specifically in later parts of this guide, but to sum it up, this is the recommended order of operations when it comes to schema changes: (Optional) but recommended for breaking changes, pause replication where applicable.</li> Apply and verify DDL changes to the subscribers.</li> Apply and verify DDL changes on the publisher.</li> Resume the replication if applicable.</li> </ol> While this might seem like a major drawback compared to physical replication, it's the characteristic that gives us the most flexibility. Other Considerations</a> </h3> As logical replication reliably handles row-level changes, it's crucial to understand other specific limitations and behaviours. Specifically: It's important to understand that sequences are not replicated. While they are used to generate the value on the publisher, and their value advances there, the corresponding sequence (if it exists) on the subscriber won't be updated.</li> Other database objects won't be replicated. While it might be understandable for views, stored procedures, triggers, and rules, you also can't rely on it for materialized views.</li> Special consideration (similar to schema changes) must be paid to user-defined data types. If a replicated table uses a user-defined type (e.g., a column using an ENUM</code>), the type must already be available on the subscriber(s) and have exactly the same name and structure. As ENUM</code>s are represented as ordered sets, even the order of the values matters!</li> </ul> Living with Logical Replication</a> </h2> In our previous example, we set up the publisher and subscriber, relied on the initial copy of the data, and let things run. Both of them will survive a restart, and thanks to the replication slot, they will keep going as soon as they come online. But what if you need to perform maintenance or do regular things like a schema update on the subscriber or the publisher? Sometimes you might need to temporarily stop the replication. PostgreSQL makes this straightforward with subscription management, allowing you to stop consuming the changes. ALTER SUBSCRIPTION my_subscription DISABLE;</code></pre> As we hinted earlier, when disabled, the subscription stops consuming changes from the publisher, while keeping the replication slot. This means: No new changes are applied on the subscriber.</li> WAL files will start to accumulate on the publisher (since the replication slot holds its position).</li> </ul> While disabled, you can't really check the status from the subscriber, as only the replication slot has the data. You can use the query we mentioned above on the publisher to check the status. SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS wal_retained FROM pg_replication_slots;</code></pre> When ready, you can resume the subscription. ALTER SUBSCRIPTION my_subscription ENABLE;</code></pre>Coordinating Schema Changes</a> </h2> As we mentioned earlier, since logical replication does not automatically replicate DDL changes, you need to coordinate schema updates manually. Here's a basic overview of how to perform such a change, step by step. First, disable the replication on the subscriber and apply your schema changes. --- disable replication ALTER SUBSCRIPTION my_subscription DISABLE; --- apply changes on the subscriber ALTER TABLE products ADD COLUMN category_id INTEGER; CREATE INDEX products_by_category ON products(category_id);</code></pre> Only then can you apply the changes to the publisher. ALTER TABLE products ADD COLUMN category_id INTEGER; CREATE INDEX products_by_category ON products(category_id);</code></pre> And resume the replication. ALTER SUBSCRIPTION my_subscription ENABLE;</code></pre> If you applied the schema change to the publisher first, any new data using the new column would fail to replicate to the subscriber that doesn't have the column yet. Handling Incompatible Changes</a> </h2> PostgreSQL's default conflict resolution is very simple. When a conflict occurs, logical replication stops, and the subscription enters an error state. For example, if you consider a product row already present on the subscriber, you will see something like this in the log files: ERROR: duplicate key value violates unique constraint "products_pkey" DETAIL: Key (id)=(452) already exists. CONTEXT: processing remote data for replication origin "pg_16444" during message type "INSERT" for replication target relation "public.products" in transaction 783, finished at 0/1E020E8 LOG: background worker "logical replication apply worker" (PID 457) exited with exit code 1</code></pre> As you can see, logical replication stopped with an error code and won't continue until you manually resolve the conflict. To do so, you need to identify the conflicting data and decide what to do with it. -- option 1: remove the conflicting data DELETE FROM products WHERE id = 452; -- option 2: update the data to match the expected state UPDATE products SET name = 'New Product', price = 29.99 WHERE id = 452;</code></pre> Only after the conflict is resolved can the logical replication continue. This has been only a very simple example of a conflict that might arise during logical replication. Other examples might involve: Data problems, constraint violations, or type mismatch</li> Schema conflicts (missing or renamed table/column)</li> Permissions issues or row-level security</li> </ul> In all those cases, you still need to go and fix the problem before logical replication can resume. Wrap Up</a> </h2> Logical replication in PostgreSQL opens up powerful possibilities beyond traditional physical replication. We have covered the foundations that can get you started with setting up your own logical replication environment, including understanding the differences between physical and logical replication, configuring publishers and subscribers, and managing core components like publications, subscriptions, and replication slots. This article is part of the upcoming guide Mastering Logical Replication in PostgreSQL</a>. If you are interested in the topic, please consider subscribing to get the latest articles as they are published. Or you can continue to Part2: Beyond The Basics of Logical Replication</a> available now. PostgreSQL Service Connections 2025-05-15T00:00:00+00:00 There are so many ways to connect to PostgreSQL. One of my favourite, yet underutilised is using service definition, the feature of any application using the libpq</code> library. In this article we will explore what service definition is, where and how it can make your life with PostgreSQL much easier. PostgreSQL connection methods</a> </h3> PostgreSQL offers several methods to establish connections to databases, each with its own benefits and use cases: Providing individual parameters like host, port, database, username, and password directly</li> Using a URI format like postgresql://username:password@hostname:port/dbname</li> Using environment variables PGHOST, PGPORT, PGDATABASE, etc.</li> Command line arguments</li> And using pre-defined connection profiles stored in the service connection file(s)</li> </ul> While the first four methods are widely known and used, service definitions remain somewhat of a hidden gem despite offering significant advantages in terms of security, maintainability, and convenience. Service definition and format</a> </h3> Connection service file</a> allows defining either per-user or system-wide connections (if both exist, the user one will take precedence). The service file uses INI format and employs the full list of libpq</code> parameters to configure the service. A sample file looks like: [mydb] host=localhost port=5432 user=some_admin password=password123 dbname=my_db</code></pre> You can consult the list of parameters, together with their description, directly using the documentation</a>. All you need to do is place the file in the correct location, which is: - either user-specific ~/.pg_service.conf</code> - or system-wide pg_service.conf in the PostgreSQL's configuration directory and try it directly using: psql service=mydb</code></pre>Why use service definitions?</a> </h2> With all other methods available to connect to PostgreSQL, why even bother with service definition? The primary benefit is the level of separation. While creating connection strings/DNS and environment variables is easy, they all suffer from the same fundamental problem: tight coupling between your application and database connection details. Service definitions break this coupling by extracting connection parameters into a standardised configuration format that any libpq-based application can reference. Imagine using a single service definition across development, test, and production environments. All it takes is to define: service=my_app_db</code></pre> and supply the correct service definition for the target environment. The application remains unaware of the underlying connection details, creating a clean abstraction layer with clear service name identification. Another aspect is re-usability. While your application might rely on an easy to change configuration mechanism, you can easily spot the difference between these two commands: pg_dump --schema-only -h my-app-db.internal -p 5432 -U some_user -d application > schema.dump pg_dump service=my_app_db > schema.dump</code></pre> The same applies to even internal mechanisms inside the running PostgreSQL (dblink</code> being a prime example - see below). This simplification extends to database migration, replication setup, and monitoring configuration. The consistency of service-based connections reduces errors and streamlines operational procedures. The benefits increase with improved security. Having a service definition file stored outside your project GIT repository significantly reduces accidental credentials sneaking inside the repository. Separation (which you can further increase by use of passfile</code> instead of password</code>) extends flexibility, while improving the security baseline. Using Service Definitions</a> </h2> Here are some ways to use service definitions: With DSN keywords: service=my_app_db service=my_app_db application_name=myapp</code></pre> In connection URLs: postgresql:///?service=my_app_db postgresql:///?service=my_app_db&application_name=myapp</code></pre> Via environment variables: PGSERVICE=my_app_db</code></pre> Given that libpq</code> is the foundation of most programming language drivers, the same is applicable almost across the board: In Go: // Go db, err := sql.Open("postgres", "service=myservice") // Go (pgx) conn, err := pgx.Connect(context.Background(), "service=my_app_db")</code></pre> Same with Python: # Python (with psycopg2) conn = psycopg2.connect("service=my_app_db")</code></pre> Java: // Java System.setProperty( "org.postgresql.pgservicefile", "/path/to/pg_service.conf"); Connection conn = DriverManager.getConnection("jdbc:postgresql:///?service=my_app_db"</code></pre> Not forgetting PHP: // PHP $conn = pg_connect("service=my_app_db"); // PHP PDO $pdo = new PDO("pgsql:service=my_app_db");</code></pre> Or Rust: // Rust let (client, connection) = tokio_postgres::connect("service=my_app_db", NoTls).await?;</code></pre> Unfortunately, the services definitions are not available in drivers/modules not based on libpq</code>, which includes, for example, Node's pg module. Using service definition inside PostgreSQL</a> </h2> A notable use case for service definition is within PostgreSQL itself. Hardcoding configuration details - either directly or using idempotent schema files - is not particularly fun. Service definition significantly improves configuration management in these cases. This is the case for dblink</code>: SELECT * FROM dblink('service=my_app_db', 'SELECT user_id, email FROM users');</code></pre> As well as FDW configuration: CREATE SERVER foreign_server FOREIGN DATA WRAPPER postgres_fdw OPTIONS (service 'my_app_db');</code></pre> While postgres_fdw</code> naturally supports service definition, being based on libpq</code>, please note this is an exception and you can't expect it for other foreign data wrappers. Similar to that, you need to consider the validity of the credentials, as libpq</code> reads the service definition at connection time, not at definition time. I.e., for Postgres FDW: Service name is stored on CREATE SERVER</code> execution</li> Service definition is read when connection is established (foreign table accessed for the first time or service connection is validated)</li> </ul> Similar for dblink</code> where the definition is read each time it establishes a new connection. Don't forget the existing connections are not affected. Partial Service Definitions</a> </h2> The flexibility offered by service definition does not end there. The libpq</code> library follows a specific order when determining connection parameters: Command-line parameters take highest precedence</li> Environment variables come next</li> Service definitions provide the next level</li> Default values are used as a last resort</li> </ol> This hierarchy enables you to define only some parameters in the service definition: [mydb] host=localhost port=5432 dbname=my_db</code></pre> With no credentials defined, you can later present the specific keywords (like credentials in this example) using other means: PGUSER=user1 PGPASSWORD=pass1 psql service=my_db</code></pre> The same applies to connection strings/DSNs, etc. This allows you to use a "best of both worlds" approach, combining the consistency of service definition with the flexibility to override it. Conclusion</a> </h2> Service definitions represent a powerful but often overlooked feature of PostgreSQL's connection system. They provide a clean separation of connection details from application code, enhance security by centralising credential management, and simplify connection management across different environments and applications. Time to Better Know The Time in PostgreSQL 2025-04-06T00:00:00+00:00 To honor the name of the site (boringSQL) let's deep dive into a topic which might sound obvious, but it might be never ending source of surprises and misunderstanding. Simple things</a> </h2> We can start with the simple statement like SELECT '2025-03-30 00:30' as t;</code></pre> Result: t ------------------ 2025-03-30 00:30</code></pre> which gives you (and I do hope that's not a big surprise) a simple text literal, rather than anything to do with date and time. As soon as you use the string literal in queries similar to SELECT * FROM events WHERE start_at > '2025-03-30 00:30';</code></pre> PostgreSQL will implicitly convert the string into timestamp without time zone. The reason why it can happen is the fact start_at</code> is most likely timestamp based field, allowing automatic cast to match the column's data type. Which is unlike the query SELECT '2025-03-30 01:00' - interval '15 minutes';</code></pre> Result: ERROR: invalid input syntax for type interval: "2025-03-30 01:00"</code></pre> which will simply not work, as PostgreSQL can't perform automatic cast (which as we will cover later might be surprising behavior, but that's how things are). The correct way to get this example query to work is to use either one of two ways (both functionally equivalent but different notation). -- PostgreSQL cast notation SELECT '2025-03-30 01:00'::timestamp - interval '15 minutes'; -- SQL standard explicit type notation SELECT timestamp '2025-03-03 01:00' - interval '15 minutes';</code></pre>PostgreSQL timestamp vs timestamptz</a> </h2> The next possible source of confusion when working with time in PostgreSQL is presence of two distinct data types: timestamp</code> (or timestamp without time zone</code>)</li> timestamptz</code> (or timestamp with time zone</code>) Despite what the names suggest, the key difference isn't whether they store timezone information, but rather how they handle it during storage and retrieval.</li> </ul> IMPORTANT: before we cover the details, remember to always use timestamptz</code>. As official Don't Do This</a> page shows, using timestamp</code> is like storing picture of the clock, instead of point in time. And while this article might use timestamp</code> it's purely for demonstration purposes. </blockquote> All you need to remember is the fact timestamp</code> stores the exact datetime value as entered without any timezone context or adjustments. It's essentially a snapshot of a calendar date and wall clock time, disconnected from any particular geographic location. -- this stores what you provided SELECT '2025-03-30 01:30'::timestamp;</code></pre> Result: timestamp --------------------- 2025-03-30 01:30:00</code></pre> On the other hand timestamptz</code> normalizes all input to UTC internally and then converts values to the session's timezone for display. This provides geographic context to your datetime values. -- this converts your input to UTC based on your session timezone SELECT '2025-03-30 01:30'::timestamptz;</code></pre> Result: timestamptz ----------------------- 2025-03-30 01:30:00+01</code></pre> And this is where potential confusion might start. Let's take this example -- With UTC timezone SET timezone = 'UTC'; SELECT '2025-03-30 01:30'::timestamptz;</code></pre> Result: timestamptz ----------------------- 2025-03-30 01:30:00+00</code></pre>-- With Tokyo timezone SET timezone = 'Asia/Tokyo'; SELECT '2025-03-30 01:30'::timestamptz;</code></pre> Result: timestamptz ----------------------- 2025-03-30 01:30:00+09</code></pre> which gives you correct representation based on the session timezone. This comes in handy when we add the actual storage. -- Create table and view with Berlin timezone SET timezone = 'Europe/Berlin'; CREATE TABLE time_flies(moment timestamp with time zone); INSERT INTO time_flies VALUES ('2025-03-30 00:30'); SELECT moment FROM time_flies;</code></pre> Result: moment ------------------------ 2025-03-30 00:30:00+01</code></pre>-- View the same data with New York timezone SET timezone = 'America/New_York'; SELECT moment FROM time_flies;</code></pre> Result: moment ------------------------ 2025-03-29 19:30:00-04</code></pre> This demonstrates the advantage of using automatic time zone conversion, allowing you to avoid not only time-zone bugs, but also DST related issues. While most of us might be fast asleep during our local DST changes, it's no fun to account for the time difference in a wrong. Using timestamptz</code> is sure way how to avoid it. Let's take an example of recent (at the time of writing the article) DST change. -- Set Berlin timezone (during DST change) SET timezone = 'Europe/Berlin'; -- Using timestamp (without timezone awareness) SELECT '2025-03-30 01:30'::timestamp + interval '45 minutes' as end_time;</code></pre> Result: end_time --------------------- 2025-03-30 02:15:00</code></pre>-- Using timestamptz (with timezone awareness) SELECT '2025-03-30 01:30'::timestamptz + interval '45 minutes' as end_time;</code></pre> Result: end_time ------------------------ 2025-03-30 03:15:00+02</code></pre>AT TIME ZONE</a> </h2> Another powerful but potentially confusing operator in PostgreSQL is AT TIME ZONE</code> which allows you to convert timestamps between different time zones, but its behavior differs depending on the input data type. When applied to timestamp (without time zone) When you apply AT TIME ZONE to a timestamp, PostgreSQL interprets the input as being in the specified time zone and converts it to a timestamptz: -- First, set UTC timezone SET timezone='UTC'; -- Interpret '2025-03-30 01:30' as if it were in the 'Europe/Paris' time zone SELECT '2025-03-30 01:30'::timestamp AT TIME ZONE 'Europe/Paris';</code></pre> Result: timezone ------------------------ 2025-03-30 00:30:00+00</code></pre> What it does is effectively "Take this wall clock time, consider it as being in the specified time zone, and give me the corresponding moment in time (as a timestamptz)". The representation here is based on session time zone settings. When applied to timestamptz (with time zone) Conversely, when you apply AT TIME ZONE to a timestamptz, PostgreSQL converts the timestamp to the specified time zone and returns a timestamp without time zone: -- Set timezone SET timezone='America/Port_of_Spain'; -- Convert the timestamptz to how it would appear on wall clocks in Tokyo SELECT '2025-03-30 01:30+01:00'::timestamptz AT TIME ZONE 'Asia/Tokyo';</code></pre> Result: timezone --------------------- 2025-03-30 09:30:00</code></pre> This operation says: "Take this moment in time and show me what wall clock time it would be in the specified time zone." Practical usage Probably the only use when correctly using timestamps</code> is to provide "wall clock" representation of the times at various time zones at once for (view) representational purposes. -- What time is the company-wide meeting in various offices? SELECT '2025-03-30 15:00'::timestamptz AS meeting_time_utc, '2025-03-30 15:00'::timestamptz AT TIME ZONE 'Europe/London' AS london_office_time, '2025-03-30 15:00'::timestamptz AT TIME ZONE 'Asia/Tokyo' AS tokyo_office_time;</code></pre> Result: meeting_time_utc | london_office_time | tokyo_office_time ------------------------+---------------------+--------------------- 2025-03-30 15:00:00+02 | 2025-03-30 14:00:00 | 2025-03-30 22:00:00</code></pre> It's important to note that in API based clients, it's always better to represent full time notation and leave the clients with the representational layer. Similar to that, unless you need to work with multiple time zones, and you depend on the correct local time representation it's always better to use local session timezone</code> setting instead of using AT TIME ZONE</code> operator. This applies for both time input and output. Common mistake Let's consider the example where you might consider "magically casting" already stored time using AT TIME ZONE</code>. -- Setup example SET timezone='Europe/Berlin'; CREATE TABLE different_moments(moment1 timestamptz, moment2 timestamptz); INSERT INTO different_moments (moment1) values ('2025-03-30 01:30'); -- Apply double time zone conversion UPDATE different_moments SET moment2 = moment1 AT TIME ZONE 'Europe/Berlin' AT TIME ZONE 'America/New_York'; -- View the results SELECT * FROM different_moments;</code></pre> Result: moment1 | moment2 ------------------------+------------------------ 2025-03-30 01:30:00+01 | 2025-03-30 07:30:00+02</code></pre> Unless you are aware of the implicit conversion between timestamps</code> and timestamp</code> and vice-versa you might be set for a lot of surprises. In this particular case the first conversion performed redundant conversion and removed the time zone offset. While second "forced" it to NYC time, while the subsequent select rendered it in local session timezone. Timestamps and the storage</a> </h2> While you will use timestamp with time zone</code> from now on, PostgreSQL still comes with several nuances related to the way it can store the data. First lesser known feature might be timestamp precision. Let's consider following table. -- Table with various timestamp precision specifications CREATE TABLE precision ( t1 timestamp with time zone, -- default precision (6) t2 timestamp(0) with time zone, -- seconds precision t3 timestamp(2) with time zone, -- centiseconds precision t4 timestamp(4) with time zone -- 10 microseconds precision );</code></pre> Effectively the timestamp</code> comes with default precision of 6 digits (microseconds), but you can specify anywhere from 0 to 6 for both timestamp</code> and timestamptz</code> types. At the same time it does not have any impact on the storage (8 bytes) regardless the precision specified. -- Insert the same timestamp into all columns INSERT INTO precision VALUES (current_timestamp, current_timestamp, current_timestamp, current_timestamp); SELECT * FROM precision;</code></pre> Result: -[ RECORD 1 ]--------------------- t1 | 2025-04-03 21:19:20.952354+02 t2 | 2025-04-04 21:19:21+02 t3 | 2025-04-04 21:19:20.95+02 t4 | 2025-04-04 21:19:20.9524+02</code></pre> Other than precision there are some other interesting aspects of the timestamps storage in PostgreSQL: It has finite range of timestamptz '4713-01-01 BC'</code> and timestamptz '294276-12-31'</code></li> and at the same time supports positive and negative infinity with 'infinity'::timestamptz</code> and '-infinity'::timestamptz</code> that might provide alternative to NULL</code> for open ended intervals. If you choose to use infinity instead of NULL</code> values, you can test whatever the provided date/timestamp is finite value or not using isfinite(timestamp)</code>.</li> </ul> Getting the current time</a> </h2> Not many developers are aware that there are different ways to get the current datetime in SQL. These methods not only differ in syntax but also in their underlying logic. CURRENT_TIMESTAMP vs NOW()</a> </h3> CURRENT_TIMESTAMP is part of the SQL Standard and interestingly, it's defined as both a "time-varying variable" and a "datetime value function". In PostgreSQL, you can use it with or without parentheses, making both these forms valid: SELECT CURRENT_TIMESTAMP; SELECT CURRENT_TIMESTAMP(2); -- (2) specifies precision</code></pre> Along with CURRENT_TIMESTAMP, the SQL Standard also defines CURRENT_TIME</code> and CURRENT_DATE</code> for working with individual time and date components respectively. NOW()</code>, while not part of the SQL Standard, is a widely used alternative across many database systems. Unlike CURRENT_TIMESTAMP, it's strictly a function and therefore always requires parentheses when called. Its straightforward syntax - NOW()</code> - makes it a popular choice among developers. Transaction Behavior</a> </h3> Both NOW()</code> and CURRENT_TIMESTAMP</code> share the same transaction behavior: they return the timestamp from the start of the current transaction, not the moment of execution: BEGIN; SELECT NOW(); -- Returns transaction start time SELECT pg_sleep(5); -- Wait 5 seconds SELECT NOW(); -- Still returns the same time COMMIT;</code></pre> This behavior ensures data consistency within transactions but might be unexpected when measuring elapsed time. Alternative Time Functions</a> </h3> When you need the actual current time regardless of transaction status, CLOCK_TIMESTAMP()</code> is your best choice. It returns the precise moment of execution, making it particularly useful for measuring real elapsed time. Here's how it differs from NOW()</code>: BEGIN; SELECT pg_sleep(1); SELECT NOW() AS transaction_time, CLOCK_TIMESTAMP() AS actual_time; COMMIT;</code></pre> Result: transaction_time | actual_time ------------------------------+------------------------------- 2025-04-03 19:28:25.310254+00 | 2025-04-03 19:28:26.311917+00</code></pre> Another useful function is STATEMENT_TIMESTAMP()</code>, which captures the time when the current statement began executing. This differs subtly from CLOCK_TIMESTAMP()</code>, which gives you the exact moment of function execution. The difference becomes clear in this example: SELECT pg_sleep(2), STATEMENT_TIMESTAMP() AS statement, CLOCK_TIMESTAMP() AS wall_time;</code></pre> Result: pg_sleep | statement | wall_time ----------+-------------------------------+------------------------------ | 2025-04-03 19:32:15.984459+00 | 2025-04-03 19:32:17.98661+00</code></pre> These timestamp functions serve different purposes and are invaluable when you need to measure transaction timing or analyze statement-level performance. Each provides a different perspective on time within your database operations. Intervals</a> </h2> Guess if you have worked with time in PostgreSQL you have come across with intervals. You probably just entered the it using something like interval '1 year 2 months 3 days 4 hours'</code>. Technical that's representation driven by setting intervalstyle</code> which (as in this case) is set to postgres_verbose</code>. But you have much more options. -- PostgreSQL default style SET intervalstyle = 'postgres'; SELECT interval '1 year 2 months 3 days 4 hours';</code></pre> Result: interval ------------------------------- 1 year 2 mons 3 days 04:00:00</code></pre>-- SQL standard style SET intervalstyle = 'sql_standard'; SELECT interval '1 year 2 months 3 days 4 hours';</code></pre> Result: interval ------------------ +1-2 +3 +4:00:00</code></pre>-- ISO 8601 style SET intervalstyle = 'iso_8601'; SELECT interval '1 year 2 months 3 days 4 hours';</code></pre> Result: interval ------------ P1Y2M3DT4H</code></pre> With the session intervalstyle</code> you only change the interval format representation, not the actual value. Nevertheless the interval representation you might run into an interval definition that might not longer make it obvious (both in input or output) to interpret. That's where function justify_interval</code> comes in play - helping you to normalize time expression into conventional formats. -- Normalize complex interval SELECT justify_interval(interval '15 months 40 days 30 hours');</code></pre> Result: justify_interval --------------------------------- @ 1 year 4 mons 11 days 6 hours</code></pre>-- Practical example: normalize project durations SELECT project_name, total_hours || ' hours' AS raw_duration, justify_interval(interval '1 hour' * total_hours) AS normalized_duration FROM (VALUES ('Database Migration', 1850), ('API Development', 720), ('UI Redesign', 340)) AS projects(project_name, total_hours);</code></pre> Result: project_name | raw_duration | normalized_duration --------------------+--------------+-------------------------- Database Migration | 1850 hours | @ 2 mons 17 days 2 hours API Development | 720 hours | @ 1 mon UI Redesign | 340 hours | @ 14 days 4 hours</code></pre> One of the important specific with internal normalization is the fact it uses 30-days time periods as a month definition. -- Month definition in normalize intervals SELECT justify_interval(interval '30 days');</code></pre> Result: justify_interval ------------------ @ 1 mon</code></pre> Which might be understandable, but you need to beware of the edge cases when using days and months intervals. -- Edge case comparison: month vs 30 days SELECT date '2025-01-31' + interval '1 month' as example1, date '2025-01-31' + interval '30 days' as example2;</code></pre> Result: -[ RECORD 1 ]----------------- example1 | 2025-02-28 00:00:00 example2 | 2025-03-02 00:00:00</code></pre> And one last thing, before wrapping up the interval - you can (same as with dates/timestamp) get specific parts using extract</code> function. -- Extract specific parts from intervals SELECT extract(days from interval '1 year 3 months 21 days') AS days, extract(hours from interval '25:30:45') AS hours;</code></pre> Result: days | hours ------+------- 21 | 25</code></pre>Time ranges</a> </h2> While working with timestamp comes natural, it's time ranges which often feel like ugly ducklings. And quite unfairly so - timestamp ranges in PostgreSQL are powerful yet underutilized features that elegantly solve common time-based challenges. PostgreSQL offers dedicated range types that are perfect for modeling time periods, reservations, schedules, and any situation where you need to track intervals with defined start and end points: tsrange</code> - range of timestamp without time zone</li> tstzrange</code> range of timestamp with time zone And to follow the earlier advice - just as you should default to timestamps</code>, always use tstzrange</code> for time ranges unless you have VERY specific reason for it. Timestamp ranges aren't merely convenient syntax - they provide a complete set of operations for determining relationships between time periods and enable powerful constraints that handle complex business rules without reinventing the wheel. Timestamp ranges can be created using the range constructor syntax or string syntax.</li> </ul> -- Range constructor syntax SELECT tstzrange( '2025-04-01 09:00:00+02'::timestamptz, '2025-04-01 17:30:00+02'::timestamptz, '[)' ) AS workday; -- String syntax SELECT '[2025-04-01 09:00:00+02, 2025-04-01 17:30:00+02)'::tstzrange AS workday;</code></pre> Result: workday ["2025-04-01 07:00:00+00","2025-04-01 15:30:00+00")</code></pre> The default boundary notation for timestamp ranges is [)</code> - lower bound inclusive (includes the start time) and upper bound exclusive (excludes the end time). For time-based applications, this convention is particularly valuable. Consider scheduling consecutive one-hour meetings from 9:00 to 10:00 and 10:00 to 11:00. With [)</code> notation, these are represented as [09:00, 10:00)</code> and [10:00, 11:00)</code>, creating perfect adjacency without overlap or gaps. If you used inclusive bounds [09:00, 10:00]</code> and [10:00, 11:00]</code>, the moment 10:00 would technically belong to both ranges - a logical impossibility for scheduling. This natural alignment with how we conceptualize time slots makes [)</code> notation the default and recommended choice for most of the application use cases. Timestamp ranges support numerous operations for checking relationships: -- 1. Check if a timestamp is within a range SELECT '[2025-04-01 09:00, 2025-04-01 17:00)'::tstzrange @> '2025-04-01 12:30'::timestamptz AS is_during_workday;</code></pre> Result: is_during_workday ------------------ t</code></pre>-- 2. Check if two ranges overlap SELECT '[2025-04-01 09:00, 2025-04-01 12:00)'::tstzrange && '[2025-04-01 11:00, 2025-04-01 14:00)'::tstzrange AS meetings_overlap;</code></pre> Result: meetings_overlap ----------------- t</code></pre>-- 3. Extract time range bounds SELECT lower('[2025-04-01 09:00, 2025-04-01 17:00)'::tstzrange) AS start_time, upper('[2025-04-01 09:00, 2025-04-01 17:00)'::tstzrange) AS end_time;</code></pre> Result: start_time | end_time --------------------------+-------------------------- 2025-04-01 09:00:00+00 | 2025-04-01 17:00:00+00</code></pre> As well as obvious manipulations: -- 1. Merge adjacent ranges SELECT '[2025-04-01 09:00, 2025-04-01 12:00)'::tstzrange + '[2025-04-01 12:00, 2025-04-01 17:00)'::tstzrange AS full_day;</code></pre> Result: full_day ------------------------------------------------------------- ["2025-04-01 09:00:00+00","2025-04-01 17:00:00+00")</code></pre>-- 2. Find intersection between overlapping ranges SELECT '[2025-04-01 09:00, 2025-04-01 14:00)'::tstzrange * '[2025-04-01 12:00, 2025-04-01 17:00)'::tstzrange AS overlap_period;</code></pre> Result: overlap_period ------------------------------------------------------------- ["2025-04-01 12:00:00+00","2025-04-01 14:00:00+00")</code></pre>-- 3. Find gap between non-overlapping ranges SELECT '[2025-04-01 09:00, 2025-04-01 11:00)'::tstzrange - '[2025-04-01 14:00, 2025-04-01 17:00)'::tstzrange AS gap;</code></pre> Result: gap ------------------------------------------------------------- ["2025-04-01 11:00:00+00","2025-04-01 14:00:00+00")</code></pre> And if you followed the previously mentioned logic, you wouldn't double guess tstzrange</code> handling of the DST transitions. -- Testing DST transition handling SET timezone = 'Europe/Berlin'; SELECT '[2025-03-30 01:00:00, 2025-03-30 03:00:00)'::tstzrange AS dst_transition_range, upper('[2025-03-30 01:00:00, 2025-03-30 03:00:00)'::tstzrange) - lower('[2025-03-30 01:00:00, 2025-03-30 03:00:00)'::tstzrange) AS actual_duration;</code></pre> Result: dst_transition_range | actual_duration -----------------------------------------------------+----------------- ["2025-03-30 01:00:00+01","2025-03-30 03:00:00+02") | @ 1 hour</code></pre> The area where time ranges win in almost all scenarios is indexing. If you consider a manual implementation of the range using start_at</code> and end_at</code> columns you typically rely on B-tree indexes. While they might work for simple queries on just one of the values, they fall short for time ranges. Consider this example: -- Traditional approach with two columns and B-tree indexes SELECT * FROM events_columns WHERE start_at <= '2025-04-01 17:00' AND end_at >= '2025-04-01 09:00';</code></pre> For this query, PostgreSQL might use one of the indexes if it's selective enough, but the fundamental problem remains: each B-tree index can only efficiently filter on its own column. The planner might use the start_at index to find events starting before 17:00, or the end_at index to find events ending after 09:00, but it can't use both indexes simultaneously to find the intersection. With range types, we can use GiST (Generalized Search Tree) indexes that are specifically designed for range operations: -- Create specialized GiST index for time range operations CREATE INDEX idx_events_range ON events_range USING GIST(time_period);</code></pre> This allows you to transform the sample query into: -- Range-based approach using the && (overlap) operator SELECT * FROM events_range WHERE time_period && tstzrange('2025-04-01 09:00', '2025-04-01 17:00');</code></pre> The performance gap between these approaches is substantial – range queries with GiST indexes typically outperform the traditional approach by 2-10x on large datasets. This difference becomes even more pronounced as your data grows. Time to wrap up</a> </h2> Working with time in PostgreSQL might seem straightforward on the surface, but as we've explored, it's filled with nuances that can make or break your application's reliability. The key takeaways from this deep dive should be: Always use timestamptz instead of timestamp</li> Be intentional about time zone handling</li> Leverage PostgreSQL's specialized time tools</li> Mind the edge cases</li> </ul> Time handling remains one of the most deceptively complex aspects of software engineering, but PostgreSQL offers a robust toolkit for managing it effectively. By understanding these "boring" but essential concepts, you'll avoid common pitfalls and build applications that handle time with confidence and precision. VIEW inlining in PostgreSQL 2025-02-08T00:00:00+00:00 Database VIEWs are powerful tools that often don't get the attention they deserve when building database-driven applications. They make our database work easier in several ways: They let us reuse common query patterns instead of writing them over and over</li> They give us a place to define business rules once and use them everywhere</li> They help us write cleaner, more organized queries</li> </ul> Let's see how this works with a practical example. Imagine we want to work with active users - users who have used the application within the last 7 days. Instead of writing this condition in every query, we can define a view: CREATE VIEW active_users AS SELECT * FROM users; WHERE last_login > current_date - INTERVAL '7 days';</code></pre> Now we can easily use this view whenever we need active users. For example, if we want to find active users from Germany: SELECT user_id FROM active_users WHERE country = 'Germany';</code></pre> While this simple example shows how views can make developers' lives easier by organizing and reusing common logic, there's more to the story. In this article, we'll explore something even more interesting: how PostgreSQL can optimize these views through a process called "inlining" - making them not just convenient, but fast too. What is VIEW inlining</a> </h2> When you use a view, it acts like a building block in SQL queries. Taking our previous example, PostgreSQL effectively transforms the query by replacing the view with its underlying subquery. SELECT user_id FROM ( SELECT * FROM users WHERE last_login > current_date - INTERVAL '7 days' ) active users WHERE country = 'Germany';</code></pre> While we write the query using active_users</code> VIEW, PostgreSQL query planner will see it as an opportunity to optimize it further. Instead of treating the sub-query as separate step (and retrieving large subset of the users), it effectively transforms the query and inlines the view into the query itself. Behind the scenes, PostgreSQL will execute the query similar to: SELECT user_id FROM users WHERE last_login > current_date - INTERVAL '7 days' AND country = 'Germany';</code></pre> The inlining process allows the PostgreSQL query planner to optimize the entire query as a single unit. This allows it to select the best indexes, possible join strategies and other aspects together, rather than having to execute the individual parts separately and then filtering the data that won't be otherwise used. This is exactly what VIEWs are for - we get the best of both worlds - clean, modular code for developers and optimal performance for the database. Inlining in action</a> </h2> The easiest way how to preview the VIEW inlining is to run EXPLAIN Seq Scan on users (cost=0.00..19.20 rows=1 width=4) Filter: ((country = 'Germany'::text) AND (last_login > (CURRENT_DATE - '7 days'::interval)))</code></pre> The Filter part here is the one showing how the filtering conditions from both the query and the view got merged together by the query planner. This works especially well with complex queries involving joins, aggregations and filters. PostgreSQL can optimize entire query chain as one unit instead of executing individual pieces separately. CREATE VIEW order_analytics AS SELECT o.customer_id, c.country, DATE_TRUNC('month', o.created_at) as month, COUNT(*) as orders, SUM(o.total_amount) as revenue FROM orders o JOIN customers c ON c.id = o.customer_id GROUP BY 1, 2, 3;</code></pre> When you use this VIEW in the query SELECT customer_id, revenue FROM order_analytics WHERE country = 'Germany' AND month >= '2024-01-01' AND revenue > 1000;</code></pre> you will see how PostgreSQL effectively inlines the filter conditions where they belong. HashAggregate (cost=205.97..207.97 rows=200 width=16) Group Key: o.customer_id, c.country, (date_trunc('month'::text, o.created_at)) Filter: (sum(o.total_amount) > 1000) -> Hash Join (cost=16.12..185.25 rows=1235 width=20) Hash Cond: (o.customer_id = c.id) -> Seq Scan on orders o Filter: (created_at >= '2024-01-01'::date) -> Hash -> Seq Scan on customers c Filter: (country = 'Germany'::text)</code></pre> This works especially well with increasing query complexity. Particulary involving joins and further filters. PostgreSQL can optimize entire query chain as one unit instead of executing individual pieces separately. CREATE VIEW completed_orders AS SELECT o.id, o.customer_id, o.total_amount, o.created_at, o.status, p.name as product_name, c.email, c.country FROM orders o JOIN customers c ON c.id = o.customer_id JOIN order_items oi ON oi.order_id = o.id JOIN products p ON p.id = oi.product_id WHERE o.status = 'completed'; SELECT * FROM completed_orders WHERE country = 'Germany'; CREATE INDEX idx_orders_customer_id ON orders(customer_id); CREATE INDEX idx_order_items_order_id ON order_items(order_id); CREATE INDEX idx_order_items_product_id ON order_items(product_id);</code></pre> as demonstrated in the query plan Nested Loop (cost=0.30..19.70 rows=1 width=176) -> Nested Loop (cost=0.15..13.52 rows=1 width=148) Join Filter: (o.id = oi.order_id) -> Nested Loop (cost=0.15..12.43 rows=1 width=144) -> Seq Scan on orders o (cost=0.00..1.05 rows=1 width=80) Filter: (status = 'completed'::text) -> Index Scan using customers_pkey on customers c (cost=0.15..8.17 rows=1 width=68) Index Cond: (id = o.customer_id) Filter: (country = 'Germany'::text) -> Seq Scan on order_items oi (cost=0.00..1.04 rows=4 width=8) -> Index Scan using products_pkey on products p (cost=0.15..6.17 rows=1 width=36) Index Cond: (id = oi.product_id)</code></pre>Planner barriers</a> </h2> While PostgreSQL is very good at inlining views, certain operations create "planner barrier" that prevent the optimization. In case of views some of the conditions that will cause it are DISTINCT ON operations</li> Window functions (OVER clauses)</li> Set operations (UNION/INTERSECT/EXCEPT)</li> More complex aggregations</li> Common Table Expressions that are materialized (either with MATERIALIZED hint or by the decision of the query planner</a>)</li> Use of VOLATILE functions</li> And in some cases complex subqueries</li> </ul> When planner is not able to inline VIEWs it might lead to the unnecessary performance and resource impact. I.e. each sub query might materialize results separately, leading to the already retrieved rows to be discarded by another part of the query as one of the most common examples. When using EXPLAIN this is demostated by one of the following options: Subquery scan on...</code> nodes</li> Materialization nodes</li> Separate aggregation/sorting steps before the main execution</li> </ul> As mentioned above the window function acts as such a planner barrier. CREATE VIEW order_insights AS SELECT o.customer_id, o.total_amount, o.created_at, ROW_NUMBER() OVER (PARTITION BY o.customer_id ORDER BY o.created_at) as order_sequence, SUM(o.total_amount) OVER (PARTITION BY o.customer_id ORDER BY o.created_at) as running_total FROM orders o; EXPLAIN SELECT * FROM order_insights WHERE customer_id = 1 AND total_amount > 500;</code></pre> Giving us easy to spot Subquery scan</code> node. Subquery Scan on order_insights (cost=1.06..1.10 rows=1 width=84) Filter: (order_insights.total_amount > '500'::numeric) -> WindowAgg (cost=1.06..1.08 rows=1 width=84) -> Sort (cost=1.06..1.06 rows=1 width=44) Sort Key: o.created_at -> Seq Scan on orders o (cost=0.00..1.05 rows=1 width=44) Filter: (customer_id = 1)</code></pre>Runtime Materialization (Not MATERIALIZED VIEWs)</a> </h2> Let's clear up a possible confusion - we're not talking about CREATE MATERIALIZED VIEW here. This is about PostgreSQL's runtime decision to cache view results in memory during query execution. Query planner might use explicit materialization when it needs to reference view results multiple times or prevent repeated expensive computations. The query planner shows this through a Materialize node: CREATE VIEW customer_summary AS SELECT customer_id, COUNT(*) as orders, SUM(total_amount) as total_spent FROM orders GROUP BY 1; EXPLAIN SELECT a.customer_id, a.total_spent, b.total_spent as other_customer_spent FROM customer_summary a JOIN customer_summary b ON b.total_spent > a.total_spent;</code></pre> Giving the perfect example of materialization. Nested Loop (cost=46.00..200.75 rows=3333 width=68) Join Filter: (b.total_spent > (sum(orders.total_amount))) -> HashAggregate (cost=23.00..24.25 rows=100 width=44) Group Key: orders.customer_id -> Seq Scan on orders (cost=0.00..18.00 rows=1000 width=15) -> Materialize (cost=23.00..25.75 rows=100 width=32) -> Subquery Scan on b (cost=23.00..25.25 rows=100 width=32) -> HashAggregate (cost=23.00..24.25 rows=100 width=44) Group Key: orders_1.customer_id -> Seq Scan on orders orders_1 (cost=0.00..18.00 rows=1000 width=15)</code></pre> Next to the materialization - the materialized VIEWs (CREATE MATERIALIZED VIEW</code>) are for practical purposes "just another table" they present an ultimate planner barrier. PostgreSQL must scan the materialized data directly, trading query flexibility for performance Tips for writing inlining friendly VIEWs</a> </h2> Well, there's not a single "right" way how to write views. And don't forget - VIEWs should make your database easier to work with, not harder. When designing views, focus on single responsibility - each view should handle one aspect of business logic rather than trying to optimize everything at once. Simple views are more likely to get inlined by PostgreSQL's query planner. There are several common patterns that prevent inlining Avoid window functions, distinct and set operations completely</li> Split complex logic into multiple views if possible</li> Minimize subquery usage</li> Consider materialized views for aggregations</li> </ol> View dependencies are a critical consideration. Even minor schema changes can trigger cascading view modifications across your database. Instead of creating deep view hierarchies, split complex logic into smaller, composable views. For critical views that many applications depend on, consider versioning (v1, v2) rather than modifying existing ones. When in doubt, write the plain SQL first, then extract common patterns into views. This helps avoid overengineering and keeps your database schema maintainable. DELETEs are difficult 2024-11-23T00:00:00+00:00 Your database is ticking along nicely - until a simple DELETE brings it to its knees. What went wrong? While we tend to focus on optimizing SELECT and INSERT operations, we often overlook the hidden complexities of DELETE. Yet, removing unnecessary data is just as critical. Outdated or irrelevant data can bloat your database, degrade performance, and make maintenance a nightmare. Worse, retaining some types of data without valid justification might even lead to compliance issues. At first glance, the DELETE command seems straightforward. Even the PostgreSQL documentation</a> provides simple examples like: DELETE FROM films WHERE kind <> 'Musical'; DELETE FROM films;</code></pre> These queries might work effortlessly on your development machine, where only a few hundred records exist. But what happens when you try running a similar DELETE in production, where datasets are orders of magnitude larger? In this article, we’ll uncover why DELETE operations demand careful consideration and explore how to handle them effectively. What really happens when you DELETE the data?</a> </h2> At first glance, a DELETE query might seem straightforward. However, once the query is executed, a series of intricate steps occur: Row Identification: Similar to a SELECT operation, the query identifies rows visible to the current transaction (considering MVCC) and checks for locks.</li> Lock Acquisition: The database acquires row-level exclusive locks to prevent other operations on the targeted rows.</li> BEFORE DELETE trigger: If a BEFORE DELETE trigger is defined, it is executed at this point.</li> Marking Rows as Deleted: Instead of being physically removed, the rows are marked as deleted in the current transaction, rendering them invisible to future queries (depending on transaction isolation). If table has large data objects, TOAST table will have to be involved too.</li> Index Updates: The corresponding index entries are also marked for deletion (if applicable).</li> Cascaded Actions: Cascading operations, such as ON DELETE CASCADE, are performed on related tables.</li> AFTER DELETE Trigger: If an AFTER DELETE trigger is defined, it is executed.</li> Write-Ahead Log (WAL): Changes are recorded in the WAL-first at the row level, followed by index-level updates.</li> </ol> Only when the transaction is committed do these changes become permanent and visible to transactions starting afterward. However, even at this point, the data is not physically removed. This is how bloat is created. Until the autovacuum process or a manual VACUUM operation reclaims the space, the “deleted” data remains. This leftover data contributes to bloat, which can degrade query performance over time. The key question now is whether DELETEs are truly the hardest operation we can subject our database to. The answer? Quite possibly. While UPDATEs come close in complexity, they’re typically designed in ways that make them less challenging: UPDATEs usually modify only a limited number of columns, reducing the potential number of index updates lower.</li> Not all UPDATEs trigger a full row (COLD) update, where the old row is marked as dead and a new row is created. With careful table and query design, you can minimise these cases. HOT (Heap-Only Tuple) updates, for instance, are easier to achieve with fixed-length columns.</li> Unlike DELETEs, UPDATEs don’t trigger cascaded actions - they only involve triggers that are explicitly defined.</li> </ul> And then comes AUTOVACUUM</a> </h2> When autovacuum kicks in (and you really want it to kick in) - typically triggered by thresholds for the number of dead tuples or changes to the table - a significant amount of work is required to clean them up. Let’s break it down step by step: The process begins by scanning the table. While it’s not always a full table scan, the autovacuum checks the visibility map and pages where dead tuples might exist. This can happen incrementally, allowing even large tables to be processed - provided your autovacuum settings allow it to run frequently enough.</li> Each tuple is checked to ensure it’s no longer visible to any active or pending transactions.</li> Dead tuples that pass the visibility check are physically removed from the table.</li> Corresponding index entries for the removed tuples are updated.</li> The now-empty space is marked for reuse in future INSERT or UPDATE operations.</li> Table statistics are updated to reflect the current state, helping the query planner make better decisions.</li> The changes, including tuple removals and index updates, are logged in the Write-Ahead Log (WAL) for durability and replication.</li> If the table has TOASTed data (large objects), the associated TOAST tables are processed.</li> The visibility map is updated to mark cleaned pages as fully visible again.</li> Autovacuum resets thresholds to determine when the next vacuum operation should occur.</li> </ul> This process continues until it reaches the configured vacuum cost limit (in the case of autovacuum), at which point it pauses or stops. While autovacuum helps keep your database in check, it’s clear that reclaiming dead tuples is no small task - and it underscores why DELETE operations can have lasting effects on database performance. While the AUTOVACUUM</code> might sounds as a bad news, without it, your database would quickly become bloated with dead tuples, leading to degraded performance, slower queries, increased storage usage, and even the potential for out-of-disk errors as unused space cannot be reclaimed. Further considerations</a> </h2> For what seems like a simple DELETE, a surprising amount of work has already taken place, but the complexity doesn’t stop there. DELETE operations can introduce additional challenges, particularly when replication, resource contention, or the size of the operation comes into play. In environments with replication to hot standby or replicas, DELETEs become more time-sensitive. The transaction cannot complete until the corresponding WAL (Write-Ahead Log) records have been written to disk on the standby. This is a fundamental requirement for maintaining data consistency in high-availability setups, where at least one standby server is typically involved. Additionally, if the standby is actively serving read operations, it must account for DELETEs before confirming the changes, potentially introducing further delays. The size of the DELETE operation also plays a critical role. Small DELETEs, such as removing a single row, tend to have minimal impact. However, as the size of the operation grows, so does the volume of WAL records generated. Large DELETEs can overwhelm the system, slowing down transactions and straining the replication process. Standby servers must work harder to process the incoming WAL stream, which can bottleneck performance if their throughput is insufficient. Resource contention adds yet another layer of complexity, particularly for large DELETEs. Generating WAL records, handling regular transactional workloads, and running background processes can collectively push the system towards I/O saturation. This creates competition for CPU and memory resources, leading to slower operations across the board. Finally, once the data is marked for deletion, the autovacuum process must eventually step in to physically remove it. This introduces its own set of challenges, as autovacuum must deal with the same resource contention and I/O demands, compounding the overall impact of the initial DELETE operation. Soft-deletes are not the solution</a> </h2> Soft-deletes might seem like an easy way to sidestep the complexities of traditional DELETE operations. After all, updating a deleted_at</code> field is straightforward and unlikely to trigger a COLD update. However, soft-deletes are not a true mechanism for data removal, and they come with their own set of complications. While soft-deletes can provide a simple way to implement “undo” functionality, they raise serious questions about data consistency. For instance, do you only mark the main entity as deleted, or do you also cascade the status to all related records in referenced tables? Failing to cascade properly can leave your database in an inconsistent state, making it difficult to maintain data integrity. Soft-deletes also require consideration in your application logic. Every query must include appropriate filters to exclude “deleted” rows, which can complicate query design and increase the risk of oversights. One missed filter could expose data that should no longer be visible, leading to potential security or business logic issues. Finally, soft-deletes don’t solve the problem - they do merely postpone it. The data is still in your database, consuming storage and potentially contributing to performance degradation over time. Sooner or later, you’ll need to deal with the actual removal of this data, bringing you back to the same challenges that DELETEs pose in the first place. At the time of writing this article we can only speculate how much will support of temporal PRIMARY KEY and UNIQUE constriants in PostgreSQL 18 will transform the balances in future. But given the complexity of the feature I wouldn't bet on it just yet. Batching is the answer</a> </h2> Giving PostgreSQL time to process and catch up with large-scale changes is critical when dealing with operations like DELETEs. The core issue here is the duration and magnitude of the transaction. The shorter the transaction and the fewer changes made, the better PostgreSQL can manage and reconcile those changes. This principle is universal across all database operations and underscores the importance of minimizing the impact of individual transactions. While you can optimize certain aspects, like row identification (using indexes, clustering, or similar techniques), larger datasets demand a more strategic approach - batching. For example, deleting 1 million rows in a single transaction is a textbook case of what not to do. Instead, splitting the operation into smaller batches, such as deleting 10,000 rows across 100 iterations, is far more effective. Will this method be faster than performing one massive DELETE? Likely not, especially if you include a wait time between batches to allow PostgreSQL to handle other workloads. However, the trade-off is worthwhile. By batching, you give PostgreSQL more breathing room to manage changes without overwhelming regular transactional workloads - unless, of course, you’ve scheduled dedicated maintenance time for the operation. How to Batch DELETEs</a> </h3> The easiest way to batch DELETEs is to use a subquery or a Common Table Expression (CTE)</a> to limit the number of rows affected in each iteration. For example, instead of executing a bulk DELETE like this: DELETE FROM films WHERE kind <> 'Musical';</code></pre> You can break the operation into smaller chunks. Using a query like the following, you can repeatedly delete rows in manageable batches (for instance, using \watch in psql to automate iterations): DELETE FROM films WHERE ctid IN ( SELECT ctid FROM films WHERE kind <> 'Musical' LIMIT 250 );</code></pre> The use of ctid</code> in this example is PostgreSQL system column which provides a unique identifier for each row. By selecting ctid</code> values in the subquery, you can limit the number of rows affected in each iteration. This approach is more efficient than using LIMIT</code> directly in the main query, as it avoids the need to re-scan the table for each batch. If you don't feel comfortable with ctid</code> (which might deserve article on its own) you can use regular lookup by primary key and LIMIT</code>. Planning for Autovacuum</a> </h3> Batching alone doesn’t directly solve the issue of autovacuum catching up with the changes. You’ll need to plan for that separately. Adjusting autovacuum settings or triggering manual VACUUM</code> and VACUUM ANALYZE</code> runs can help manage bloat created during the DELETE process. However, disabling autovacuum is rarely advisable unless you’ve carefully planned for manual maintenance throughout the batch operations. Skipping this step risks leaving behind performance-impacting bloat that will require even more effort to address later. Drop whole range of data with partitioning</a> </h2> Data that is naturally segmented - for example by time of the creation - makes it an excellent candidate for removal through partitioning. Partitioning allows you to bypass DELETE operations altogether by simply dropping or truncating the relevant partitions. This approach is far more efficient and avoids the overhead of scanning, locking, and marking rows as deleted, effectively eliminating the problem with the bloat. While partitioning adds some complexity to schema design and query planning, it can provide significant performance benefits for DELETE-heavy workloads, especially when combined with automated partition management. Conclusion</a> </h2> DELETE operations are often a source of unpleasant surprises - not just by affecting performance and creating bloat, but by striking back at the times we least expect. To handle them effectively, focus on strategies such as batching, monitoring autovacuum, or leveraging partitioning for large datasets. By considering DELETE operations during schema design, you can maintain an efficient database, reduce maintenance headaches, and ensure it continues to run smoothly as your data grows. Text identifiers in PostgreSQL database design 2024-11-09T00:00:00+00:00 Whether you are designing a standalone application or a microservice, you will inevitably encounter the topic of sharing identifiers. Whether it’s URLs of web pages, RESTful API resources, JSON documents, CSV exports, or something else, the identifier of specific resources will be exposed. /orders/123 /products/345/variants/1</code></pre> While an identifier is just a number and does not carry any negative connotations, there are valid reasons why you might want to avoid exposing them. These reasons include: Security and Data Exposure: Numerical identifiers are sequential and predictable, which can expose information about the underlying data source (e.g., the volume of the data) and provide a basis for ID enumeration.</li> Privacy and Confidentiality: Concerns may arise about concealing the volume of referenced data. For example, the number of customers, clients, or orders might be information a business prefers to keep private.</li> Non-descriptive Nature: Integers as identifiers can lead to confusion. An ID like 123 does not convey any additional information, making debugging edge cases more challenging.</li> </ol> These and other reasons (like SEO optimisation) have led to the increased use of text-based identifiers. Their readability and versatility make them ideal for external data sharing. However, in database (or data model) design, the advantages of text identifiers are often overshadowed by the problems they introduce. While text identifiers improve interoperability, they frequently come with performance and storage trade-offs. In contrast, integers are naturally faster and more efficient to process, resulting in lower storage requirements and faster indexing, sorting, and searching-tasks that computers are optimised for. In this article, we will explore scenarios where using text identifiers directly in the database design might seem natural and discuss strategies for using them effectively. What Makes Text Identifiers Appealing?</a> </h3> Let’s be honest-text identifiers are popular for a reason. For humans, they are much more readable and, in some cases, add extra context. (Raise your hand if you don’t appreciate Heroku’s quirky and memorable names!) If chosen carefully, they can also be easier to recall. Text identifiers can embed additional context. Consider the order number APAC-20241103-8237, which encodes both the region and the order date. Another popular reason for using text identifiers is their uniqueness in distributed environments. They’re especially handy when people need to interact with them directly. For example, customers copying an order number from an email or support teams discussing an issue benefit from a readable, meaningful identifier. It’s simpler, more intuitive, and less likely to cause headaches when someone tries to recall or share it. When an Identifier Isn’t Just an Identifier</a> </h3> Problems with text identifiers arise when they are used as natural keys in your data or database model. Despite their benefits, text identifiers often make poor primary keys for several reasons: Context Changes: The additional context provided by text identifiers will likely change, necessitating updates. Despite assurances to the contrary, changes are inevitable.</li> Sorting Issues: Sorting text identifiers can be tricky, particularly with locale-based sorting or when numbers are embedded within the identifier (e.g., order1434 vs. order349).</li> </ul> The ultimate issue is efficiency: Text identifiers typically require more storage space than numeric ones. Each character in a text field occupies more bytes than a simple integer, leading to larger database sizes and slower performance, especially during indexing or handling large datasets.</li> Databases are optimised for numeric operations, making searches, joins, and indexing on text fields inherently slower. This performance gap can significantly affect large datasets, impacting application efficiency.</li> Text identifiers complicate managing relationships between tables. The additional storage requirements affect not only the source table but also all referencing entities. Multiply this increased storage by the number of referencing tables, and you’ll get a sense of the overall impact.</li> </ul> As databases grow, these issues become more pronounced. The combination of increased storage needs and slower operations contributes to database bloat, which can degrade system performance. Additionally, bloated indexes can mislead the query planner into making suboptimal choices, such as favouring sequential scans over index scans, complicating regular operations further. Aren't UUIDs the Solution to All This?</a> </h3> It depends. While UUIDs offer numerical representation and are excellent for unique key generation across distributed systems, they are not always the best choice: In comparison to BIGINT, there are few real-world scenarios that necessitate the full range of UUIDs. Premature optimisation most often isn’t worthwhile for most solutions.</li> UUIDs’ 16-byte (128-bit) storage size can be less efficient than many text identifiers. Even BIGINT, at 8 bytes (64 bits), is more storage-efficient. This inefficiency extends to indexes, joins, and other operations.</li> </ul> A matter of opinion: UUIDs are plain ugly.</li> </ul> Moreover, different versions of UUIDs offer varying benefits. For example, UUIDv1 includes a timestamp component, making it somewhat sortable, while UUIDv4 is entirely random. Even sortable UUIDs may not offer significant advantages over more streamlined options like BIGINT or carefully structured text identifiers. In most cases, the trade-offs in performance and readability make UUIDs less appealing unless their globally unique nature is essential across distributed systems. Real-Life Examples</a> </h3> Let’s move beyond theory and explore practical examples: CREATE TABLE sessions ( token TEXT PRIMARY KEY, user_id INT NOT NULL REFERENCES users(user_id), ... ); CREATE TABLE products ( sku TEXT PRIMARY KEY, label TEXT NOT NULL, ... ); CREATE TABLE documents ( document_id TEXT PRIMARY KEY, ... );</code></pre> In these cases, using text identifiers as primary keys might seem logical because: They serve as natural keys for the entity.</li> </ul> They are likely propagated outside the service scope, hence needing storage anyway.</li> They are not expected to change.</li> The storage impact for a single table seems negligible.</li> </ul> However, this logic falters under scrutiny. Text identifiers often propagate beyond the service scope, leading to pressure to update them. Updating primary keys is no trivial matter. Consider: While SKUs ideally never change, real-world scenarios like rebranding, product consolidation, or supplier changes can necessitate updates. Despite their appeal, SKUs are poor primary key candidates.</li> Randomly generated text identifiers (like session tokens) will… TBD.</li> </ul> The real issue arises when referencing these identifiers across tables: CREATE TABLE session_logs ( log_id BIGINT GENERATED ALWAYS AS IDENTITY, token TEXT NOT NULL REFERENCES sessions(token), ... ); CREATE TABLE product_reviews ( review_id INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY, product_sku TEXT NOT NULL REFERENCES products(sku), ... ); CREATE TABLE customer_orders ( order_id TEXT PRIMARY KEY, customer_id TEXT NOT NULL REFERENCES customers(customer_id), ... ); CREATE TABLE document_revisions ( revision_id INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY, document_id TEXT NOT NULL REFERENCES documents(document_id), ... );</code></pre> In such scenarios, text identifiers can become problematic. Firstly, you lose the ability to modify them. Secondly, the increased storage requirements become evident. For instance, an additional 100 bytes per reference across a million records results in an extra 100 MB of storage for just one reference. However, storage isn’t the biggest concern; indexing is. In PostgreSQL, indexing text fields - whether as primary or foreign keys - requires significantly more space than indexing numeric fields. This leads to bloated indexes, slower lookups, and more fragmented operations, particularly in queries relying on foreign key relationships. As a result, the query planner might resort to full table scans instead of index scans, further degrading performance. These performance issues often remain undetected in development or testing environments but can cause significant disruptions in production. Reintroducing Text Identifiers Effectively</a> </h3> The goal of this post is not to discourage the use of text identifiers entirely but to highlight why they are unsuitable as primary keys. Here are some strategies to handle them effectively: 1. Introduce a Surrogate Key</a> </h4> One straightforward solution is to introduce a surrogate primary key, replacing references to text identifiers: CREATE TABLE products ( product_id INT GENERATED BY DEFAULT AS IDENTITY, sku TEXT NOT NULL, label TEXT NOT NULL, ... ); CREATE INDEX products_by_sku ON products(sku); CREATE TABLE product_reviews ( review_id SERIAL PRIMARY KEY, product_id INT REFERENCES products(product_id), ... );</code></pre> This approach maintains efficient retrieval by SKU via an index. 2. Use Mapping Tables for Greater Flexibility</a> </h4> Mapping tables allow you to: * Update identifiers without affecting the parent entity. * Maintain a history of text identifiers linked to a specific entity.</code></pre>CREATE TABLE products ( product_id INT GENERATED BY DEFAULT AS IDENTITY, label TEXT NOT NULL, ... ); CREATE TABLE product_skus ( product_sku_id INT GENERATED BY DEFAULT AS IDENTITY, product_id INT REFERENCES products(product_id), sku TEXT NOT NULL, created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, deleted_at TIMESTAMPTZ ); CREATE UNIQUE INDEX unique_product_skus ON product_skus (product_id, sku) WHERE deleted_at IS NULL;</code></pre> This approach accommodates changing SKUs without compromising data integrity. A related use case could be linking a mapping table to a product variant: CREATE TABLE product_variants ( variant_id INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, product_id INT NOT NULL REFERENCES products(product_id), sku TEXT NOT NULL, name TEXT NOT NULL, ... );</code></pre>3. Decompose the Text Identifier’s Meaning</a> </h4> Text identifiers often embed meaningful information, such as regions, dates, or categories. By decomposing the identifier, you can store this contextual data separately, improving both flexibility and performance. For example, instead of storing an order identifier like ORD-EMEA-00789 directly, you can design a more robust schema: CREATE TABLE orders ( order_id INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, region_id INT NOT NULL REFERENCES regions(region_id), order_date DATE NOT NULL, ... ); CREATE TABLE regions ( region_id INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, name TEXT NOT NULL, code TEXT NOT NULL, ... ); INSERT INTO regions (name, code) VALUES ('Europe, the Middle East, and Africa', 'EMEA'), ('Asia Pacific', 'APAC'); To generate a user-friendly order number, you can create a function: CREATE OR REPLACE FUNCTION get_order_number(p_order_id INT) RETURNS TEXT AS $$ DECLARE v_region_code TEXT; v_formatted_order_id TEXT; v_order_number TEXT; BEGIN SELECT r.code INTO v_region_code FROM orders o JOIN regions r ON o.region_id = r.region_id WHERE o.order_id = p_order_id; IF NOT FOUND THEN RETURN NULL; END IF; v_formatted_order_id := TO_CHAR(p_order_id, 'FM00000'); v_order_number := 'ORD-' || v_region_code || '-' || v_formatted_order_id; RETURN v_order_number; END; $$ LANGUAGE plpgsql;</code></pre> This method enables you to store and retrieve structured data efficiently while still providing a readable and meaningful identifier for external use. 4. Reversible Text IDs</a> </h4> For scenarios requiring enhanced security or privacy, where identifiers should not be easily enumerable or predictable, you can use reversible text-based IDs. These allow you to present users with seemingly random text representations while maintaining efficient numerical storage internally. Sqids</a> is a project that can help achieve this. It generates URL-friendly unique identifiers from numbers and can encode multiple numerical identifiers into a single string. Here’s an example using the Sqids project: [42] -> JgaEBgznCpUZo3Kk [42, 430004] -> lTiYlvsGkh59m1PQ</code></pre> The generated identifiers are reversible with knowledge of the shared alphabet, allowing you to decode requests without hitting the database, which can be beneficial in high-throughput environments. This technique is particularly useful for user, account, or session identifiers, balancing the need for security with operational efficiency. However, it’s crucial to remember that this is not a substitute for robust security practices. Proper authentication and authorisation mechanisms are still necessary to secure your application. By thoughtfully integrating these strategies, you can leverage the benefits of text identifiers where appropriate while avoiding common pitfalls in database design. This balance ensures efficient, maintainable systems that meet both technical and business needs. 5. Leverage Generate columns</a> </h4> In edge case scenarios, where keeping text identifiers with embedded context is necessary, generated columns in PostgreSQL can be a valuable feature. Since version 12, PostgreSQL allows defining columns whose values are automatically computed from other columns in the table. This ensures consistency without manual intervention. For example, you can define a function to handle the formatting logic: CREATE OR REPLACE FUNCTION get_formatted_order_id(p_region_id INT, p_order_id INT) RETURNS TEXT AS $$ DECLARE v_region_code TEXT; BEGIN v_region_code := CASE p_region_id WHEN 1 THEN 'EMEA' WHEN 2 THEN 'APAC' ELSE 'OTHER' END; RETURN 'ORD-' || v_region_code || '-' || LPAD(p_order_id::TEXT, 5, '0'); END; $$ LANGUAGE plpgsql IMMUTABLE;</code></pre> The function is marked IMMUTABLE because PostgreSQL requires generated columns to use only immutable functions-those guaranteed to return the same result for the same input every time. CREATE TABLE orders ( order_id INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, region_id INT NOT NULL, order_date DATE NOT NULL, formatted_order_id TEXT GENERATED ALWAYS AS ( get_formatted_order_id(region_id, order_id) ) STORED );</code></pre> This setup ensures that formatted_order_id</code> is automatically updated whenever region_id</code> or order_id</code> changes. The column is physically stored, enhancing read performance for frequently queried data. Using generated columns can simplify maintaining consistency in derived values like formatted text identifiers. However, be mindful of their characteristics, such as: Automated Updates: The system automatically recomputes the column’s value when the referenced columns change.</li> Immutability Requirements: Only immutable functions can be used, ensuring reliable and consistent computation.</li> </ul> Wrapping up</a> </h3> Text identifiers will stay with us, and that's great. They’re human-readable, memorable, and can pack a lot of meaningful context into a simple string. They make external interactions smoother, whether it’s a customer referencing an order number or a support team tracing an issue. They even add a bit of charm and personality to otherwise sterile identifiers. However it's important to keep their usage in control. The nice rule I've heard is Use text identifiers when communicating externally - for example, in URLs or API responses</li> Internally, always rely on numerical IDs - surrogate keys like INT or BIGINT (even UUID if you go for planet dominanance) where necessary, to maintain database efficiency and integrity.</li> </ul> This approach allows you to leverage the strengths of text identifiers for external communication while keeping your database optimised for performance and scalability. By storing text identifiers in dedicated fields and using numeric primary keys for internal operations, you achieve a the right balance and do best to maintain the system performance. We need to talk about ENUMs 2024-09-04T00:00:00+00:00 Designing a database schema, whether for a new application or a new feature, always raises a lot of questions. The choices you make can have a big impact on how well your database performs and how easy it is to maintain and scale. Whether you’re just getting started with PostgreSQL or consider yourself a seasoned pro, it’s easy to rely on old habits or outdated advice. In this article, I want to take a fresh look at one of those topics that often sparks debate: the use of ENUMs in PostgreSQL. I have to admit, not so long ago, I would advise "don't use ENUMs" without thinking about it too much. Relying only on random articles and some personal but outdated experience, this had been my go-to answer for some years. And while there were several limitations of ENUMs in PostgreSQL in the (distant) past, the support has improved a long time ago. The improvements are so long-standing that: PostgreSQL 9.1 (released in 2011) introduced the option to add new values to ENUMs without a table rewrite. From what I can say this fact alone remained the source of biggest misconception when thinking about ENUMs.</li> Renaming of values was added in version 10 (2017).</li> The ability to ALTER TYPE ... ADD VALUE</code> in a transaction block was introduced in PostgreSQL 12 (2019).</li> </ul> And new features coming (at the time of writing this article) with PostgreSQL 17 allow the use of newly added values within the same transaction block (previously not possible without an explicit commit). Therefore, I want to correct even myself and say—let's give ENUMs another chance. This article will go into detail and help us correctly decide when it makes sense to use them or not. How are ENUMs Implemented?</a> </h2> Every stored ENUM value occupies 4 bytes on disk. This is the size of the OID, representing the actual ENUM value, stored as a row within the pg_enum</code> table. You can see this yourself by querying the table directly, but it will come by default with no data defined. # select * from pg_catalog.pg_enum; oid | enumtypid | enumsortorder | enumlabel -----+-----------+---------------+----------- (0 rows)</code></pre> But once you define a new ENUM data type, using: CREATE TYPE order_status AS ENUM ('new', 'pending', 'processing', 'shipped', 'delivered');</code></pre> You will get the actual OIDs. # select * from pg_catalog.pg_enum; oid | enumtypid | enumsortorder | enumlabel --------+-----------+---------------+------------ 211018 | 211016 | 1 | new 211020 | 211016 | 2 | pending 211022 | 211016 | 3 | processing 211024 | 211016 | 4 | shipped 211026 | 211016 | 5 | delivered</code></pre> The table straightforwardly identifies both the type and individual values, but also the enumsortorder</code>, which is the foundation for BEFORE/AFTER</code> new value(s) definition. ALTER TYPE order_status ADD VALUE 'cancelled' BEFORE 'shipped';</code></pre> Resulting in: oid | enumtypid | enumsortorder | enumlabel --------+-----------+---------------+------------ 211018 | 211016 | 1 | new 211020 | 211016 | 2 | pending 211022 | 211016 | 3 | processing 211024 | 211016 | 4 | shipped 211026 | 211016 | 5 | delivered 211027 | 211016 | 3.5 | cancelled</code></pre> To confirm the actual storage size of the ENUM value, you can use a sample table, the previously defined type order_status</code>, a single data row, and the pg_column_size</code> function. CREATE TABLE orders ( id SERIAL PRIMARY KEY, status order_status NOT NULL, order_date DATE NOT NULL DEFAULT CURRENT_DATE ); INSERT INTO orders (status, order_date) VALUES ('new', '2024-09-01');</code></pre> This gives you confirmation of the actual 4-byte storage. # SELECT id, status, pg_column_size(status) AS status_size id | status | status_size ----+--------+------------- 1 | new | 4</code></pre> Returning to the pg_enum</code> table, the structure of the table provides other important clues: enumlabel</code> is of type name</code>, which is effectively a 63-byte varchar for storing system identifiers.</li> The UNIQUE CONSTRAINT pg_enum_typid_label_index</code> prevents us from creating two values with the same name.</li> However, given the nature of VARCHAR</code>, it is case-sensitive, allowing you to define new</code> and NEW</code> as two distinct values.</li> </ul> Life with ENUMs</a> </h2> Now that we understand how ENUMs are implemented, let's review the lifecycle of such a data type in a typical database. As demonstrated above, creation of new ENUMs is simple, and since version 12 (and expanded in 17), it's possible even in transaction blocks, making it easier to work with database migration tools without needing to explicitly worry about transaction management. Adding new values does not require a table rewrite and can be done without further hesitation. You can also safely rename existing values. Similarly, you can use ENUMs in arrays, indexes, or compare them as needed (based on their order). ALTER TYPE name ADD VALUE [ IF NOT EXISTS ] new_enum_value [ { BEFORE | AFTER } neighbor_enum_value ] ALTER TYPE name RENAME VALUE existing_enum_value TO new_enum_value</code></pre> It's also important to mention that ENUMs allow you to use a default value, like this: status order_status NOT NULL DEFAULT 'new'</code></pre> Unfortunately, this is where the flexibility ends at the moment. There's no easy way to remove or re-order existing values (please, don't modify pg_enum</code>—or any system tables, for that matter). The topic of removing existing ENUM value(s) can easily open up a whole discussion about the best way to 'deprecate' and drop a value over time. The only suitable way to perform these operations is by creating a new type and altering the column. Please be aware of the ways described in How not to change PostgreSQL column type</a> will apply in this scenario. Using CHECK Constraints as an Alternative to ENUMs</a> </h2> While ENUMs are a powerful tool in PostgreSQL for enforcing a fixed set of values in a column, they come with the limitations mentioned above. An alternative approach to achieving similar functionality is to use a CHECK constraint with a TEXT column. This method offers more flexibility when managing the allowed values, especially in scenarios where the set of valid values might change frequently. A CHECK constraint can be applied to a column to enforce that its value must be one of a predefined set of values. Here’s how you can define a table using a CHECK constraint as an alternative to an ENUM: CREATE TABLE orders ( id SERIAL PRIMARY KEY, status TEXT NOT NULL, order_date DATE NOT NULL DEFAULT CURRENT_DATE, CHECK (status IN ('new', 'pending', 'processing', 'shipped', 'delivered')) );</code></pre> Modification of the constraints is possible, but you will always have to DROP CONSTRAINT</code> and then ADD CONSTRAINT</code>. ALTER TABLE orders DROP CONSTRAINT orders_status_check; ALTER TABLE orders ADD CONSTRAINT orders_status_check CHECK (status IN ('new', 'pending', 'processing', 'shipped', 'delivered', 'returned'));</code></pre> While this operation is not as 'heavyweight' as changing the data type, please be aware that an ACCESS EXCLUSIVE lock will be acquired for the entire table—meaning no other operations can be performed on the table. With a growing table size and increasing concurrency, this might still lead to application-level downtime. Another limitation is the lack of direct support for sorting, meaning you will have to implement it manually in every query or abstract the sorting detail using a view. SELECT id, status, order_date FROM orders ORDER BY ARRAY_POSITION( ARRAY['new', 'pending', 'processing', 'shipped', 'delivered', 'returned'], status );</code></pre>Reference Tables as the Real Alternative</a> </h2> It might be surprising, but one of the motivations for using either ENUMs or CHECK constraints is to avoid additional JOINs. It's no surprise that JOINs are a source of pain for many people who struggle with SQL, and therefore they try to limit their use. While there might be negligible impact on performing such an operation, there are no other reasons not to use reference tables—it’s not storage (unless you over-optimise with smallint</code> and extremely large datasets) and definitely not operational considerations. In fact, reference tables are often better in most cases. They not only address all the use cases supported by both ENUMs and CHECK constraints, but they can also resolve the limitations of these methods. Creating and Using a Reference Table</a> </h3> Let’s create a reference table similar to the ENUM data type we’ve discussed above: CREATE TABLE order_statuses ( status_id INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, status_name TEXT NOT NULL ); CREATE UNIQUE INDEX unique_status_name ON order_statuses (LOWER(status_name)); INSERT INTO order_statuses (status_name) VALUES ('new'), ('pending'), ('processing'), ('shipped'), ('delivered'), ('cancelled');</code></pre>Updating the Orders Table to Use the Reference Table</a> </h3> To use the reference table, you’ll need to alter your existing orders</code> table. This involves dropping the current status</code> column if it’s using an ENUM or CHECK constraint, and replacing it with a foreign key that references the order_statuses</code> table: ALTER TABLE orders DROP COLUMN IF EXISTS status; ALTER TABLE orders ADD COLUMN status_id INT, ADD CONSTRAINT fk_order_status FOREIGN KEY (status_id) REFERENCES order_statuses(status_id);</code></pre>Impact of Using Reference Tables</a> </h3> While reference tables do not provide anything for free, such as ENUM ordering or simple CHECK constraints, they offer far greater flexibility. Here are some examples of what you can achieve with reference tables: Custom Sorting Logic: You can implement specific sorting by adding an additional sort_order</code> column to the order_statuses</code> table. This allows you to easily change the order of statuses without altering the schema.</li> Deprecating Values: By adding a deleted_at</code> column (or a similar field), you can deprecate certain status values. Combined with a trigger, you can prevent these deprecated values from being used in new data while maintaining historical records.</li> Renaming Values: Unlike ENUMs, renaming a status is straightforward - simply update the value in the order_statuses</code> table.</li> Internationalisation: If your application needs to support multiple languages, you can extend the order_statuses</code> table with additional columns for different languages or even create a separate lookup table that stores translations.</li> Additional Metadata: Reference tables can store extra information about each status, such as descriptions, colours (for UI purposes), or associated icons, which can be particularly useful in applications with rich user interfaces.</li> </ul> Having mentioned all advantages of using Reference Tables, it's also important to mention the fact it's not straightforward to set the default values for such a references. Definitely not in obvious way which ENUMs provide. The real alternative is to create table/column without default value and alter it specifically for given database using ALTER COLUMN ... SET DEFAULT</code> ALTER TABLE orders ALTER COLUMN status_id SET DEFAULT (SELECT status_id FROM order_statuses WHERE status_name = 'new');</code></pre>Performance Considerations: ENUMs, CHECK Constraints, and Reference Tables</a> </h2> Thanks to their internal representation as OIDs, ENUMs offer predictable performance and indexing strategies, often outperforming text-based CHECK constraints. Since ENUM comparisons rely on integers, they tend to be faster than string comparisons required for CHECK constraints. Reference tables, on the other hand, inherently require a JOIN, which introduces the overhead of accessing two tables and potentially working with multiple indexes. While I initially planned to publish benchmarks comparing the performance of all three approaches, I decided not to include specific numbers. It’s true that ENUMs consistently performed the fastest, and reference tables were somewhat slower due to the necessary JOIN. However, as with most benchmarks, the scenarios were artificial. The actual performance difference, though measurable, is unlikely to be significant in practical applications. In any reasonably complex query, other factors are far more likely to impact performance than whether you use ENUMs or not. Conclusion: Use ENUMs Where They Make Sense</a> </h2> As with many decisions in software development, the use of ENUMs comes down to "it depends." This article has provided clarity on how ENUMs work, where their limitations lie, and how to make an informed decision about when to use them. To recap: ENUMs are ideal for small, static sets of values that require strict type enforcement and minimal changes, preferably data types that are not directly tied to the business logic.</li> CHECK constraints offer more flexibility than ENUMs, making them a good choice when the allowed values might change, but the data type remains simple, and there's no</li> Reference tables provide the most flexibility and scalability, making them the go-to choice for dynamic or complex value sets and when you want to avoid the limitations of ENUMs and CHECK constraints.</li> </ul> But in the end, no plan ever survives first contact with reality. It’s essential to revisit and reconsider your decisions as your application evolves, ensuring that your database schema continues to serve the needs of your project as effectively as possible. And while it's not meant as advice - personally I'm still going to gravitate towards the advice "don't use ENUMs" though. I'm yet to see value-based field that does not change over time. Beyond Simple Upserts with MERGE in PostgreSQL 2024-08-25T00:00:00+00:00 Understanding how comfortable someone is with databases and SQL often comes down to the features they use. In PostgreSQL, one such feature that distinguishes more advanced users is the MERGE</code> command, introduced in version 15 and expanded in version 17 (in beta at the time of writing this article). Before MERGE</code>, developers typically relied on INSERT ... ON CONFLICT DO UPDATE</code> for upserts—a method introduced in PostgreSQL 9.5 that has since become a staple in many developers' toolkits. While ON CONFLICT</code> offers a straightforward solution for simple upsert scenarios, it can quickly become limiting as business logic grows in complexity. This is where the MERGE</code> command excels. Introduced in the SQL:2003 standard, MERGE</code> allows for more sophisticated data synchronisation tasks by combining multiple operations—such as conditional inserts, updates, and deletes—into a single, atomic statement. In this article, we’ll explore the capabilities of the MERGE</code> command, comparing it with traditional upsert methods and examining how it can streamline database operations. Through practical examples, we'll illustrate how MERGE</code> can simplify even the most complex workflows, making it an indispensable tool for developers working with PostgreSQL. This introduction sets the stage by highlighting the evolution from ON CONFLICT</code> to MERGE</code>, explains the context in which MERGE</code> becomes valuable, and clearly outlines what the reader will gain from the article. Hands-on Example: Managing a Score Points System for a Mobile Game</a> </h2> Imagine you’re managing a tournament score system for an online game, where players earn special status by participating in weekly tournaments. The simple business logic would be: Players can earn score points by participating in the tournament.</li> Players must participate in all tournaments to maintain their score (i.e., maintain the streak).</li> Players who participate in a second consecutive tournament can achieve the status of veteran</code> (if they reach 1,000 score points) or gain the status of star</code> (if they reach 100 or more score points).</li> </ul> The score points are updated weekly, based on the outcome of the previous week's participation. We will start with a simple database schema consisting of a table tournament_scores</code> that tracks (as the name suggests) only active users. CREATE TABLE tournament_scores ( player_id INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, player_name TEXT UNIQUE NOT NULL, score INT NOT NULL DEFAULT 0, status TEXT NOT NULL DEFAULT 'newbie' CHECK (status IN ('newbie', 'veteran', 'star')) );</code></pre> And populate it with some sample data: INSERT INTO tournament_scores (player_name, score, status) VALUES ('PlayerOne', 900, 'newbie'), -- Regular player, close to Veteran promotion ('PlayerTwo', 1200, 'veteran'), -- Already a Veteran player ('PlayerThree', 300, 'newbie'); -- Regular player with a lower score</code></pre>Upsert using ON CONFLICT</code></a> </h2> When activity data is received, there are multiple ways to process it. ON CONFLICT</code> is used here for demonstration purposes. INSERT INTO tournament_scores (player_name, score) VALUES ('PlayerOne', 50), -- Add points for PlayerOne ('PlayerTwo', 120), -- Add points for PlayerTwo ('PlayerFour', 70) -- New player without an account, PlayerFour ON CONFLICT (player_name) DO UPDATE SET score = tournament_scores.score + EXCLUDED.score, status = CASE WHEN (tournament_scores.score + EXCLUDED.score) > 1000 THEN 'veteran' ELSE 'newbie' END;</code></pre> This performs the basic requirements, updating the scores and possibly evaluating the status for existing tournament users. To remove users who no longer participated (and hence broke the streak) and apply other conditional logic, you would need to use separate statements. Handling Upserts with MERGE</code></a> </h2> Now, let’s introduce the MERGE</code> command and implement the same logic as above. MERGE INTO tournament_scores ts USING ( VALUES ('PlayerOne', 50), -- Add points for PlayerOne ('PlayerTwo', 120), -- Add points for PlayerTwo ('PlayerFour', 70) -- New player without an account, PlayerFour ) AS v(player_name, score_added) ON ts.player_name = v.player_name WHEN MATCHED THEN UPDATE SET score = ts.score + v.score_added, status = CASE WHEN (ts.score + v.score_added) > 1000 THEN 'veteran' ELSE 'newbie' END WHEN NOT MATCHED THEN INSERT (player_name, score) VALUES (v.player_name, v.score_added);</code></pre> Here, we define the table tournament_scores</code> as the target and the list of VALUES as the source. Using a conditional clause, we define the logic for both matched and unmatched entries, giving us both INSERT and UPDATE paths. The evaluation paths in this case are called when_clauses</code>. The MERGE</code> statement allows you to specify multiple clauses with different conditions, for example, allowing you to award users star</code> status if they gain more than 100 score points within a given tournament. MERGE INTO tournament_scores ts USING ( VALUES ('PlayerOne', 50), -- Add points for PlayerOne ('PlayerTwo', 120), -- Add points for PlayerTwo ('PlayerFour', 70) -- New player without an account, PlayerFour ) AS v(player_name, score_added) ON ts.player_name = v.player_name WHEN MATCHED AND v.score_added > 100 THEN UPDATE SET score = ts.score + v.score_added, status = 'star' WHEN MATCHED THEN UPDATE SET score = ts.score + v.score_added, status = CASE WHEN (ts.score + v.score_added) > 1000 THEN 'veteran' ELSE 'newbie' END WHEN NOT MATCHED THEN INSERT (player_name, score) VALUES (v.player_name, v.score_added);</code></pre> Here’s a summary of how it works: The MERGE</code> command evaluates the when_clauses</code> in the order they are written.</li> If a row matches the condition specified in a clause, the action defined in that clause is performed, and the row is no longer eligible to be matched against subsequent when_clauses</code>.</li> </ol> While we introduced multiple evaluation paths, the MERGE</code> command in PostgreSQL goes beyond that and expands the WHEN NOT MATCHED</code> clause, which effectively becomes WHEN NOT MATCHED [BY TARGET]</code>. PostgreSQL allows you to specify WHEN NOT MATCHED BY SOURCE</code> to perform the necessary merge statement for the data not present in the source table. Handling DELETEs with MERGE</code></a> </h2> The functionality completely missing from regular upserts with ON CONFLICT</code> is the ability to delete missing entries. In our sample scenario, we want to penalise players who broke their streak and did not participate in last week's tournament by effectively deleting their entries from the target table. MERGE INTO tournament_scores ts USING ( VALUES ('PlayerOne', 50), -- Add points for PlayerOne ('PlayerTwo', 120), -- Add points for PlayerTwo ('PlayerFour', 70) -- New player without an account, PlayerFour ) AS v(player_name, score_added) ON ts.player_name = v.player_name WHEN MATCHED AND v.score_added > 100 THEN UPDATE SET score = ts.score + v.score_added, status = 'star' WHEN MATCHED THEN UPDATE SET score = ts.score + v.score_added WHEN NOT MATCHED THEN INSERT (player_name, score) VALUES (v.player_name, v.score_added) WHEN NOT MATCHED BY SOURCE THEN DELETE;</code></pre> The introduction of the DELETE</code> merge operation complements all possible MERGE</code> outcomes: The first is merge_update</code>, applicable to WHEN MATCHED</code> and WHEN NOT MATCHED</code> clauses, consisting of regular UPDATE</code> operations.</li> The second is merge_insert</code> for the WHEN NOT MATCHED</code> clause, allowing (as the name implies) data to be inserted.</li> The DELETE</code> we introduce is part of merge_delete</code>.</li> </ul> Technically, there's also the ability to specify DO NOTHING</code> for all available when_clauses</code>, similar to upserts using ON CONFLICT</code>. Using MERGE</code> Output</a> </h2> Starting from PostgreSQL 17, the MERGE</code> command has been extended to support RETURNING</code> clauses to process merged data further. When an INSERT</code> or UPDATE</code> action is performed, the new values of the target table's columns are used. When a DELETE</code> is performed, the old values of the target table's columns are used. Conclusion</a> </h2> The MERGE</code> command significantly enhances the ability to handle complex INSERT</code> and UPDATE</code> logic by allowing multiple operations within a single query while also capturing even the most intricate scenarios. In this article, we have explored the syntax and common use cases of MERGE</code>. For further exploration, note that the source</code> dataset is where you can perform any data transformations required, making MERGE</code> particularly well-suited for ETL operations, data archival, cleanup tasks, and more. Incorporating MERGE</code> into your PostgreSQL toolkit can simplify database operations, reduce the risk of data inconsistencies, and streamline your Gentle Introduction to Window Functions in PostgreSQL 2024-07-07T00:00:00+00:00 Understanding the relationship between data points is crucial. For instance, you might need to identify the most recent orders for each customer or track changes in sensor readings over time. Unlike aggregate functions, which summarise data into a single row, it is window functions that allow you to analyse data while preserving each row’s details. This is the core of the logic, but don’t worry if you struggle to imagine the difference, as we will cover all of it in this article. PostgreSQL supports SQL window functions, facilitating complex calculations across related rows within a table. These functions are particularly useful for tasks such as ranking entries, calculating running totals, finding moving averages, and comparing individual entries. Mastering window functions can significantly enhance your data analysis capabilities. You can easily use similar window functions as you would with aggregation. Let’s take a simple example of sensor readings: CREATE TABLE sensor_readings ( sensor_id bigint, reading_value decimal, reading_time timestamp with time zone default current_timestamp ); INSERT INTO sensor_readings (sensor_id, reading_value, reading_time) VALUES (1, 32.7, '2024-07-01 11:24:34'), (1, 33.1, '2024-07-02 11:29:01'), (1, 33.1, '2024-07-02 12:03:59'), (1, 33.0, '2024-07-03 10:12:15'), (2, 32.8, '2024-07-01 13:17:01'), (2, 35.8, '2024-07-02 09:18:11'), (3, 29.1, '2024-07-01 13:54:03'), (3, 30.3, '2024-07-01 14:12:09'), (3, 31.5, '2024-07-02 16:07:43');</code></pre> When you start with the aggregation functions, it is straightforward for anybody familiar with SQL: SELECT sensor_id, AVG(reading_value) FROM sensor_readings GROUP BY sensor_id;</code></pre> with the output sensor_id | avg -----------+--------------------- 3 | 30.3000000000000000 2 | 34.3000000000000000 1 | 32.9750000000000000</code></pre> While similar, using the window function AVG we can get almost same result: SELECT sensor_id, reading_value, AVG(reading_value) OVER (PARTITION BY sensor_id) AS avg_reading_value FROM sensor_readings;</code></pre> such as sensor_id | reading_value | avg_reading_value -----------+---------------+--------------------- 1 | 32.7 | 32.9750000000000000 1 | 33.1 | 32.9750000000000000 1 | 33.1 | 32.9750000000000000 1 | 33.0 | 32.9750000000000000 2 | 32.8 | 34.3000000000000000 2 | 35.8 | 34.3000000000000000 3 | 29.1 | 30.3000000000000000 3 | 30.3 | 30.3000000000000000 3 | 31.5 | 30.3000000000000000</code></pre> This reiterates the fundamental difference between the two sets of functions. As mentioned earlier, while the results for sensor_id</code> are the same in both cases, the aggregate function summarised it into a single row (grouped by sensor_id</code>), whereas the window function provides the value for the set of rows in the partition defined by sensor_id</code>. After showing the average in a window function, it’s important to note that traditional aggregation functions might not be the most helpful in the context of window function logic. Despite functions like AVG</code>, COUNT</code>, and SUM</code>—which are the most used aggregation functions—being available as window functions, we used the above only to demonstrate the difference. OVER clause</a> </h2> Before diving into the individual functions, let’s cover the syntax first. From the sample query above, you already get the basic syntax of the window functions, with the OVER</code> clause being a primary identification of windowing functionality. window_function ([expression...]) OVER window_definition</code></pre> In our example, the basic window functions can be similar to the aggregate ones— like AVG</code>, SUM</code> and COUNT</code> - and a number of the functions we will cover shortly. The window definition part specifies how the window function will see the data it works over. It can include: Partitioning to divide the result sets into separate partitions (think groups in aggregate functions).</li> Definition of the ordering of the rows within each partition.</li> Specification of how to apply framing of the subset of rows for each row’s calculation.</li> </ol> From all the components of the window syntax, only partitioning is mandatory. If we revisit our first window function example above, you can identify the partitioning part (PARTITION BY sensor_id</code>). As already mentioned, you can easily compare the partitioning logic to the grouping used in aggregate functions, with the same properties—like partitioning data segments by multiple fields, using functions and other logic. With partitioning comes hand in hand ordering of the data. When you start thinking of row properties, instead of grouping data into a single row, order starts to make a difference. Let’s take the following example: window_function ([expression...]) OVER (PARTITION BY sensor_id ORDER BY reading_time)</code></pre> This will provide different data compared to unsorted ones. If you struggle to find the application for this, think of the moving average or row numbering. The last component of the window definition is framing, allowing you to further limit the data over which the window function is calculated. There are two framing expressions: ROWS BETWEEN</code> and RANGE BETWEEN</code>. Using the framing, we can turn AVG from the above example to provide our first real use case when you might want to use window functions. SELECT reading_time, sensor_id, reading_value, AVG(reading_value) OVER ( PARTITION BY sensor_id ORDER BY reading_time ROWS BETWEEN 1 PRECEDING AND 1 FOLLOWING ) AS moving_avg_reading_value FROM sensor_readings;</code></pre> Implementing the moving average of the individual readings, taking into account a maximum of 3 rows including the current one. reading_time | sensor_id | reading_value | moving_avg_reading_value ------------------------+-----------+---------------+-------------------------- 2024-07-01 11:24:34+02 | 1 | 32.7 | 32.9000000000000000 2024-07-02 11:29:01+02 | 1 | 33.1 | 32.9666666666666667 2024-07-02 12:03:59+02 | 1 | 33.1 | 33.0666666666666667 2024-07-03 10:12:15+02 | 1 | 33.0 | 33.0500000000000000 2024-07-01 13:17:01+02 | 2 | 32.8 | 34.3000000000000000 2024-07-02 09:18:11+02 | 2 | 35.8 | 34.3000000000000000 2024-07-01 13:54:03+02 | 3 | 29.1 | 29.7000000000000000 2024-07-01 14:12:09+02 | 3 | 30.3 | 30.3000000000000000 2024-07-02 16:07:43+02 | 3 | 31.5 | 30.9000000000000000</code></pre>Exploring Window functions</a> </h2> Now that we have a foundational understanding of window functions and their components, let’s dive into specific window functions. We’ll cover some of the most commonly used functions, such as ROW_NUMBER()</code>, RANK()</code>, DENSE_RANK()</code>, LAG()</code>, LEAD()</code>, FIRST_VALUE()</code>, LAST_VALUE()</code> and more. For each function, we’ll provide examples to illustrate their practical applications. ROW_NUMBER</a> </h3> The ROW_NUMBER()</code> function assigns a unique sequential integer to rows within a partition of a result set, starting with one for the first row in each partition. SELECT sensor_id, reading_time, reading_value, ROW_NUMBER() OVER (PARTITION BY sensor_id ORDER BY reading_time) AS row_num FROM sensor_readings;</code></pre> This makes it easy to find the first/last entries for a specified window. As an example, you can experiment with getting only the last reading for each hour. SELECT sensor_id, reading_time, reading_value FROM ( SELECT sensor_id, reading_time, reading_value, ROW_NUMBER() OVER ( PARTITION BY sensor_id, date_trunc('hour', reading_time) ORDER BY reading_time DESC ) AS row_num FROM sensor_readings ) AS ranked_readings WHERE row_num = 1;</code></pre>RANK and DENSE_RANK</a> </h3> The RANK()</code> and DENSE_RANK()</code> functions are used to assign a rank to each row within a partition, based on the order of one or more values. These functions are useful when scoring the values and handling ties. The main difference between RANK</code> and DENSE_RANK</code> is how they deal with gaps. Example use to find the highest reading_value per sensor: SELECT sensor_id, reading_time, reading_value, RANK() OVER (PARTITION BY sensor_id ORDER BY reading_value DESC) AS rank FROM sensor_readings;</code></pre> If you consider the sorting for the sensor_id</code> 1 in our sample seed the difference between RANK</code> and DENSE_RANK</code> is easy to demonstrate. sensor_id | reading_time | reading_value | rank -----------+------------------------+---------------+------ 1 | 2024-07-02 11:29:01+02 | 33.1 | 1 1 | 2024-07-02 12:03:59+02 | 33.1 | 1 1 | 2024-07-03 10:12:15+02 | 33.0 | 3 1 | 2024-07-01 11:24:34+02 | 32.7 | 4</code></pre> Giving a natural ranking with tie on the reading value 33.1 and leaving a gap on 2nd rank, whereas DENSE_RANK</code> wouldn't include the gap. sensor_id | reading_time | reading_value | rank -----------+------------------------+---------------+------ 1 | 2024-07-02 11:29:01+02 | 33.1 | 1 1 | 2024-07-02 12:03:59+02 | 33.1 | 1 1 | 2024-07-03 10:12:15+02 | 33.0 | 2 1 | 2024-07-01 11:24:34+02 | 32.7 | 3</code></pre>LAG</code> and LEAD</code></a> </h3> To evaluate previous or subsequent rows without the need to self-join the dataset, you can utilise the power of the window functions LAG</code> and LEAD</code>. These functions are particularly useful for calculating differences between rows, comparing current and previous values, or fetching future values for comparison. The LAG</code> function provides access to a value in a previous row within the partition, while LEAD</code> provides access to a subsequent row. SELECT sensor_id, reading_time, reading_value, LAG(reading_value, 1) OVER (PARTITION BY sensor_id ORDER BY reading_time) AS previous_value, LEAD(reading_value, 1) OVER (PARTITION BY sensor_id ORDER BY reading_time) AS next_value FROM sensor_readings;</code></pre> In this query, we are using a value of 1, but you can choose any position necessary. By using LAG()</code> and LEAD()</code>, you can perform advanced analyses that require looking backward or forward within your dataset, making it easier to derive meaningful insights and trends. FIRST_value</code> and LAST_VALUES</code></a> </h3> The functions FIRST_value</code> and LAST_VALUES</code> are similar, except they (as the name says) give the first/last value of the partition. The FIRST_VALUE</code> is useful when comparing the partition values to the first value (for example opening price for a day), and LAST_VALUE</code> to identity the closing values. Other Window functions</a> </h3> The complete list of the window functions is available in the documentation</a>. Re-using the window definition</a> </h3> As you might have noticed, the query we used to demonstrate LAG</code> and LEAD</code> was rather verbose. This was due to the repeated definition of the window for both columns. Luckily, the syntax of the window functions allows you to define and re-use the window definition. SELECT sensor_id, reading_time, reading_value, LAG(reading_value, 1) OVER readings_window AS previous_value, LEAD(reading_value, 1) OVER readings_window AS next_value FROM sensor_readings WINDOW readings_window AS (PARTITION BY sensor_id ORDER BY reading_time);</code></pre> This way you can ensure the consistent window definition and consistency in complex queries. The time keepers: pg_cron and pg_timetable 2024-06-15T00:00:00+00:00 Working with PostgreSQL, and virtually any database system, extends far beyond merely inserting and retrieving data. Many application and business processes, maintenance tasks, reporting, and orchestration tasks require the integration of a job scheduler. While third-party tools can drive automation, you can also automate the execution of predefined tasks directly within the database environment. Although system-level cron might be a starting point, the power of the database system lies in its ability to store all the necessary information alongside your data/schema. In this article, we will explore pg_cron</code> and pg_timetable</code> as two distinct PostgreSQL-specific tools for scheduled task automation. The Many Roles of Job Scheduling</a> </h2> Usually, the first requirement to automate job execution is the optimisation of the PostgreSQL cluster and databases. Except in very low usage scenarios, routine maintenance is the foundation of all database deployments. Whether it is VACUUMing, index rebuilding, or updating statistics, these are tasks that you will eventually need to automate to maintain operational efficiency. Job scheduling is also indispensable for maintaining data quality, particularly through operations like refreshing materialised views or batch removal of outdated data from the system. From there, it is just a step to reporting, which can involve the automation of generating operational and business reports. I also believe that databases are an excellent place to coordinate business processes. Automating data flows between components, teams, and departments helps create agile systems. As mentioned above, there is no better place to store the description of such automations than alongside your data. pg_cron: Automation's First Gear</a> </h2> Traditionally, the role of automation started with operating system tools like cron or Task Scheduler. That's where pg_cron</code></a> comes in. It's a PostgreSQL extension that provides the simplicity and familiarity of cron's scheduling directly within the database environment. Due to its dependency on a shared library (because of the use of the background worker), it requires a full cluster restart. Nevertheless, due to its popularity, it is available within most managed cloud environments. # requires configuration update in postgresql.conf (or conf.d) shared_preload_libraries = 'pg_cron' # add extension as superuser CREATE EXTENSION pg_cron;</code></pre> As long as you are familiar with cron-like syntax, you can get started immediately by using commands like: SELECT cron.schedule('30 3 * * 6', $$DELETE FROM events WHERE event_time < now() - interval '1 week'$$);</code></pre> By default, pg_cron</code> exposes its functionality in the postgres</code> database (configurable), where it expects its metadata tables. Personally, I consider this a drawback, as it makes it less obvious to the casual DBA who might not be aware of the scheduling logic present. In the same database, you can perform basic monitoring, for example, getting details of running and recently completed jobs: SELECT * FROM cron.job_run_details ORDER BY start_time DESC LIMIT 5;</code></pre> There is no direct support to talk to other PostgreSQL clusters (you would have to facilitate this, for example, using Foreign Data Wrappers). While cron-like syntax is beneficial for adoption, it is where pg_cron</code> falls short, as it does not support advanced cases like task chaining, dependencies, and triggers. Full Speed with pg_timetable</a> </h2> If pg_cron</code> offers automation in first gear, pg_timetable</code> is where you can go full speed ahead. It not only provides cron-like syntax but elevates it to a whole new level with task chaining, parameter support, multiple execution clients, enhanced scheduling, and much more. The first difference you might notice is in its distribution. Timetable is not a PostgreSQL extension but a standalone binary (or available as a Docker image) that you have to configure and run. This immediately raises the bar in terms of the infrastructure it requires. On the other hand, not being distributed as an extension makes it compatible with all managed services by default (if you can get the process up and running). While the basic syntax might be similar to pg_cron</code>: SELECT timetable.add_job('execute-func', '5 0 * 8 *', 'SELECT public.my_func()');</code></pre> this is not anywhere near the limits of pg_timetable</code>. add_job</code> is a helper function that creates a simple one-task chain. For task execution, it directly supports more options, including built-in tasks (like sending emails if properly configured, downloading files, sleeping, copying files, etc.), external commands, the ability to choose which client should execute the job, concurrency, and more. In its full definition, add_job</code> is quite powerful (see documentation</a> for details): SELECT timetable.add_job( job_name => 'notify every minute', job_schedule => '* * * * *', job_command => 'SELECT pg_notify($1, $2)', job_parameters => '["TT_CHANNEL", "Hello World!"]'::jsonb, job_kind => 'SQL'::timetable.command_kind, job_client_name => NULL, job_max_instances => 1, job_live => TRUE, job_self_destruct => FALSE, job_ignore_errors => TRUE ) AS chain_id;</code></pre> The components of pg_timetable</code> consist of the command (SQL/program or built-in), task controlling the configuration for the command execution (error handling, timeout, or database connection to use), and chain which can contain a number of tasks chained together. The big difference comes in the scheduling options. While pg_cron</code> is limited (as its name suggests) to cron-like syntax, pg_timetable</code> uses it for simple use cases but offers much more. It supports schedules @every</code> and @after</code>, allowing repeated execution and breaking away from the limitations of cron notation as it can go down to custom intervals (including second intervals). Another case is @reboot</code> for instances when the pg_timetable</code> controller reconnects to the database. The timetable setup manages own schema migrations and stores all the configuration in schema timetable</code> (by default), where you can find the definitions of chains/tasks and relevant auditing information (logs). And yes, if you have been paying attention, pg_timetable</code> architecture allows for running tasks across multiple PostgreSQL clusters, making it an advanced orchestration tool. The database connection, which can be configured in-place or via a drop-in connection service file</a>, can be set on a per-task basis. With chains, the setup can be much more complex: DO $$ DECLARE v_chain_id BIGINT; v_notify_task_id BIGINT; BEGIN INSERT INTO timetable.chain ( chain_name, run_at, max_instances, live) VALUES ( 'Generate Weekly Report', '5 4 * * 1', 1, TRUE ) RETURNING chain_id INTO v_chain_id; INSERT INTO timetable.task ( chain_id, task_order, task_name, command, database_connection ) VALUES ( v_chain_id, 1, -- task_order 'generate_report', -- task_name 'SELECT generate_weekly_report();', -- command NULL -- database_connection ); INSERT INTO timetable.task ( chain_id, task_order, task_name, command, database_connection ) VALUES ( v_chain_id, 2, -- task_order (second task in the chain) 'notify_management', -- task_name 'SELECT pg_notify($1, $2)', -- command 'service=service_db' -- database_connection ) RETURNING task_id INTO v_notify_task_id; INSERT INTO timetable.task_parameter (task_id, order_id, value) VALUES (v_notify_task_id, 1, 'management_notifications'), (v_notify_task_id, 2, 'Weekly report'); END; $$ LANGUAGE plpgsql;</code></pre>The Comparison</a> </h2> When deciding between pg_cron</code> and pg_timetable</code>, it's essential to consider the specific needs of your use case, as both tools target different scenarios and scales of deployment. Here’s a comparison to help you decide which tool is more suitable for your requirements: Feature</th> pg_cron</th> pg_timetable</th></tr></thead> Distribution</td> PostgreSQL extension</td> Standalone executable</td></tr> Scheduling</td> Limited (cron-like)</td> Extensive (intervals, calendar, custom)</td></tr> Job Types</td> SQL</td> SQL, Built-in, Shell</td></tr> Job Chaining</td> No</td> Yes</td></tr> Error Handling</td> Basic</td> Intermediate</td></tr> Logging</td> Basic</td> Intermediate</td></tr> Dependencies</td> No</td> Yes</td></tr> Ease of Use</td> Easy</td> Moderate</td></tr> Configuration</td> Easy</td> Moderate</td></tr> Orchestration</td> Single Cluster</td> Advanced (multiple clusters)</td></tr> </tbody></table> When to Use pg_cron</code></a> </h3> If your requirements are straightforward, such as running maintenance tasks, refreshing materialised views, or generating periodic reports using simple SQL commands, pg_cron</code> is an excellent choice. Its cron-like syntax is familiar and easy to use.</li> pg_cron</code> is a PostgreSQL extension, making it easier to install and configure within your existing PostgreSQL environment. This makes it ideal for users who prefer minimal setup effort.</li> When your environment is limited to a single PostgreSQL cluster, pg_cron</code> is well-suited for the job. It does not support orchestration across multiple clusters, so it’s best used in environments where all operations are confined to one database cluster.</li> If Basic Error Handling and Logging is sufficient, pg_cron</code> provides the necessary functionalities without additional complexity.</li> </ol> When to Use pg_timetable</code></a> </h3> If you need more advanced scheduling capabilities and task depedencies and chaining, such as task chaining, complex intervals, and custom execution times, pg_timetable</code> is the better choice. It supports extensive scheduling options beyond the typical cron syntax.</li> pg_timetable</code> supports a variety of job types, including SQL commands, built-in tasks (like sending emails or downloading files), and shell commands. This makes it ideal for more complex automation needs that go beyond simple SQL execution.</li> If your environment spans multiple PostgreSQL clusters, pg_timetable</code> can handle advanced orchestration, allowing tasks to be executed across different clusters seamlessly.</li> Intermediate Error Handling and Logging provides better insights and control over job executions.</li> Although bit more complex to setup, pg_timetable</code> works as standalone Service, which makes it suitable for environments where extensions cannot be installed directly.</li> </ol> Choosing between pg_cron</code> and pg_timetable</code> depends on your specific needs. For simpler, single-cluster tasks, pg_cron</code> offers ease of use and straightforward setup. For more complex requirements involving advanced scheduling, task dependencies, and multi-cluster orchestration, pg_timetable</code> provides a more powerful and flexible solution. By understanding the strengths and limitations of each tool, you can make an informed decision that best suits your database automation needs. PS: there's also pgAgent</a>, but it is generally not used and does not seem to be actively maintained. Deep Dive into PostgREST - Time Off Manager (Part 3) 2024-06-06T00:00:00+00:00 This is the third and final instalment of "Deep Dive into PostgREST". In the first part</a>, we explored basic CRUD functionalities. In the second part</a>, we moved forward with abstraction and used the acquired knowledge to create a simple request/approval workflow. In Part 3, we will explore authentication and authorisation options to finish something that might resemble a real-world application. Authentication with pgcrypto</a> </h2> There's no authorisation without knowing user identity. So let's start there. Our users table from the first part had an email</code> to establish the identity, but no way to verify it. We will address this by adding a password column. Of course, nobody in their right mind would store passwords in plain text. To securely store users' passwords, we are going to utilise PostgreSQL pgcrypto</code> extension (documentation</a>). This built-in extension provides a suite of cryptographic functions for hashing, encryption, and more. In our case, we will leverage crypt</code> with the gen_salt</code> function to generate password hash using bcrypt algorithm. Let's start with loading the extension and adding password column: CREATE EXTENSION IF NOT EXISTS pgcrypto; ALTER TABLE users ADD COLUMN password_hash TEXT NOT NULL DEFAULT gen_salt('bf'); </code></pre> The new column password_hash</code> will accommodate the variable-length bcrypt hash, while default function gen_salt('bf')</code> creates a unique bcrypt salt for every user. Now that our table structure is set, let's see how we can securely set and verify passwords using pgcrypto. When a user sets or changes their password, we'll hash it using crypt before storing it in the password_hash column.Here's how: UPDATE users SET password_hash = crypt('new_password', gen_salt('bf')) WHERE email = 'user@example.com';</code></pre> and during login, we will verify the hash of the password the user enters with the stored password_hash: SELECT user_id FROM users WHERE email = 'user@example.com' AND password_hash = crypt('entered_password', password_hash);</code></pre>JWT Authentication</a> </h2> With the ability to securely verify users' identity, let's move to the next step and build stateless authentication. The de-facto standard is JSON Web Tokens (JWTs). Represented by digitally signed information, they contain claims about user identity. The digital signature ensures the token's integrity and verifies that it hasn't been tampered with. You can find more in official Introduction to JSON Web Tokens</a>. While PostgreSQL doesn't have built-in JWT support, we will have to either rely on pgjwt</code> extension (GitHub repo</a>) CREATE EXTENSION IF NOT EXISTS pgjwt;</code></pre> or you can re-create the logic by including PL/pgSQL that comes with it</a> (please, make sure you replace/remove the @extschema@</code> to match the schema you are using). In this article we will use schema jwt</code>. To make the token signature, we need to re-configure PostgREST to decode JWT tokens and configure the secret inside the database (to be able to use it in our code). For security reasons, the key must be at least 32 characters long. You can either use your own method to generate (for example, your password manager) or add it using the following CLI commands export LC_CTYPE=C echo "jwt-secret = \"$(LC_ALL=C tr -dc 'A-Za-z0-9' </dev/urandom | head -c32)\"" >> postgrest.conf</code></pre> And use the generated secret for the database level configuration parameter, using ALTER DATABASE time_off_manager SET "pgrst.jwt_secret" to "your-jwt-secret-generated-above1";</code></pre> Please, make sure the database name is updated accordingly to set it correctly. And, I cannot stress it enough, make sure the secret is really 32 characters. The web user role</a> </h2> In previous parts of this guide, we have worked with two user roles. The first, in PostgREST terminology, called authenticator, is the role used in the db-uri</code> parameter. It's the role used to access the database and its job is to impersonate other users based on the authentication (or lack thereof) of the HTTP requests. In a real production application, this role should be configured to have limited access. The second role, called anonymous, is used in db-anon-role</code> parameter. This is the role impersonated for all unauthenticated HTTP requests. The third role, or roles, representing authenticated web users. In our JWT tokens, we will use one specifically designed for PostgREST. { "role": "time_off_user" }</code></pre> When JWT is successfully validated, with a role claim, PostgREST will switch to the database role with the provided name for the duration of the HTTP request. While PostgREST is quite flexible, we will limit the impersonated role for authenticated requests to a single hard-coded role. For our application, it's going to be called time_off_user</code>. Let's generate some JWTs</a> </h2> The first step is to implement logic to create a JWT token asserting all claims we need for our application - in the case of Time Off Manager, we will rely on user_id and role. As discussed in the previous section, we will use the hard-coded role time_off_user</code>. Let's create the role and grant our authenticator the permissions. CREATE ROLE time_off_user NOLOGIN; GRANT time_off_user TO time_off_manager;</code></pre> Now create a function, users can call directly to verify the identity and generate the JWT token. CREATE OR REPLACE FUNCTION api.login(email text, password text) RETURNS text LANGUAGE plpgsql SECURITY DEFINER AS $function$ DECLARE user_record public.users; BEGIN SELECT * INTO user_record FROM public.users WHERE users.email = login.email; IF user_record.password_hash = crypt(password, user_record.password_hash) THEN RETURN create_jwt(user_record.user_id, 'user_id'); ELSE RAISE EXCEPTION 'Invalid email or password'; END IF; END; $function$;</code></pre> Missing there is helper create_jwt</code> CREATE OR REPLACE FUNCTION public.create_jwt(user_id integer, role text) RETURNS text LANGUAGE plpgsql AS $function$ DECLARE payload JSON; BEGIN payload := json_build_object( 'user_id', user_id, 'role', 'time_off_user', 'exp', extract(epoch from now()) + 3600 -- 1-hour expiration ); RETURN jwt.sign(payload, current_setting('pgrst.jwt_secret')); -- Use configuration value END; $function$;</code></pre> Restart the PostgREST instance to apply the changes, and you can try to generate a token using cURL (assuming you have changed the password as demonstrated in the first section ): curl -X POST "http://localhost:3000/rpc/login" \ -d '{"email":"manager2@example.com", "password": "new_password"}' \ -H "Content-Type: application/json"</code></pre> If you set everything as expected you will get a base64 encoded token. To verify/debug it, you can try to use JWT.io</a>. Prepare the permissions</a> </h2> The next step is where things get really interesting. First, we need to clean up excessive permissions. Some obvious, some less so. First let's start with the revoking all the permissions for time_off_anonymous</code> we provided in previous part. REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA api FROM time_off_anonymous; REVOKE EXECUTE ON ALL FUNCTIONS IN SCHEMA api FROM time_off_anonymous;</code></pre> and same we need to adjust the default privileges ALTER DEFAULT PRIVILEGES IN SCHEMA api REVOKE SELECT ON TABLES FROM time_off_anonymous; ALTER DEFAULT PRIVILEGES IN SCHEMA api REVOKE EXECUTE ON FUNCTIONS FROM time_off_anonymous; </code></pre> Which is the obvious part. The most likely surprising part is the need to remove EXECUTE</code> permissions for PUBLIC</code>. In PostgreSQL, granting usage on a schema also grants the ability to execute functions to PUBLIC</code>. Unless you revoke these privileges, all users will be able to execute the functions. For our purposes we want to REVOKE the access for PUBLIC</code> and only grant EXECUTE</code> on function api.login()</code> GRANT USAGE ON SCHEMA api TO time_off_user; -- REVOKE EXECUTE ALTER DEFAULT PRIVILEGES IN SCHEMA api REVOKE EXECUTE ON FUNCTIONS FROM PUBLIC; REVOKE EXECUTE ON ALL FUNCTIONS IN SCHEMA api FROM PUBLIC; -- GRANT EXECUTE GRANT EXECUTE ON FUNCTION api.login to time_off_anonymous;</code></pre> And finally, we need to establish the necessary permissions for time_off_user</code>, the role which will be used for authenticated requests. GRANT SELECT ON ALL TABLES IN SCHEMA public TO time_off_user; GRANT SELECT ON ALL TABLES IN SCHEMA api TO time_off_user; GRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA api TO time_off_user; ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO time_off_user; ALTER DEFAULT PRIVILEGES IN SCHEMA api GRANT SELECT ON TABLES TO time_off_user; ALTER DEFAULT PRIVILEGES IN SCHEMA api GRANT EXECUTE ON FUNCTIONS TO time_off_user;</code></pre> We will now use this to get an overview of only relevant vacation balances for the authenticated user. Authorization with PostgREST</a> </h2> With authentication working, let's start with the implementation of fine-grained authorization in PostgREST. As a first step, the goal is to ensure that users only see the vacation balances they are supposed to. I.e. either their own (for the regular employees) or their own and the people the user manages (for the managers). For this we will need to access JWT token claims, specifically user_id</code>. To avoid relatively complex notation, let's setup a helper CREATE OR REPLACE FUNCTION public.current_user_id() RETURNS integer LANGUAGE plpgsql SECURITY DEFINER AS $function$ DECLARE user_id INTEGER; BEGIN user_id := current_setting('request.jwt.claims', true)::json->>'user_id'; IF user_id IS NULL THEN RAISE EXCEPTION 'User ID not found in JWT claims'; END IF; RETURN user_id::integer; END; $function$;</code></pre> PostgREST seamlessly integrates with PostgreSQL's Row Level Security (RLS) to filter data based on user permissions. RLS policies are rules applied at the row level to determine if a user can access a particular row in a table. ALTER TABLE public.time_off_transactions ENABLE ROW LEVEL SECURITY; CREATE POLICY select_own_balance ON public.time_off_transactions FOR SELECT USING (public.current_user_id() = user_id); CREATE POLICY select_supervised_balance ON public.time_off_transactions FOR SELECT USING (EXISTS (SELECT 1 FROM api.users WHERE users.user_id = time_off_transactions.user_id AND users.manager_user_id = public.current_user_id()));</code></pre> If you are (and I do hope so) using PostgreSQL 15 and higher, you need to switch the view from the default security definer to invoker. ALTER VIEW api.vacation_balances SET (security_invoker = true);</code></pre> This way the view, and the underlying tables will be evaluated using the permissions of the user querying the view, not the view owner. This is the behaviour we want. For versions beyond PostgreSQL 15 and more complex use cases, you might need to implement function-based security instead. But without further delay, let's test "the magic" and (assuming you have authenticated as a manager) you might try to retrieve the vacation balances using cURL curl "http://localhost:3000/vacation_balances" \ -H "Authorization: Bearer ${JWT_TOKEN}"</code></pre> to get a result similar to [{"year":2024,"user_id":8,"total_amount":25}, {"year":2024,"user_id":9,"total_amount":25}, {"year":2024,"user_id":10,"total_amount":25}, {"year":2024,"user_id":11,"total_amount":25}, {"year":2024,"user_id":12,"total_amount":25}, {"year":2024,"user_id":13,"total_amount":25}]</code></pre> Guess, it's important to mention the Row Security Policies which are used are much more complex, and the whole topic would deserve another article. For now, I do recommend you to consult the documentation</a></del>, to get more understanding. With the multi-role access you can implement with PostgREST, it might be interesting for you to focus on BYPASSRLS</code> which might be applied to certain role(s). As mentioned before, Row Level Security is just one way to solve this. The traditional approach would be to use functions to hide the separation logic. Wrapping up the Time Off Manager</a> </h2> Before we wrap up our tutorial, let's finish the core functionality of the Time Off Manager - the approval workflow. Given the basic concepts covered above, this is going to be a good exercise to use all of it. Similar how we updated time_off_transactions</code>, we will apply Row Level Security to time_off_requests</code>. ALTER TABLE public.time_off_requests ENABLE ROW LEVEL SECURITY; CREATE POLICY select_own_requests ON public.time_off_requests FOR SELECT USING (public.current_user_id() = user_id); CREATE POLICY select_subordinate_requests ON public.time_off_requests FOR SELECT USING (EXISTS (SELECT 1 FROM public.users WHERE users.user_id = time_off_requests.user_id AND users.manager_user_id = public.current_user_id()));</code></pre> Don't forget to switch the view to security_invoker</code> model. ALTER VIEW api.pending_requests SET (security_invoker = true);</code></pre> And we wouldn't be done with requests creation, if we wouldn't update function api.request_time_off</code> to take the advantage of the newly propagated user_id</code> from authentication. CREATE OR REPLACE FUNCTION api.request_time_off(leave_type text, period daterange) RETURNS integer LANGUAGE plpgsql SECURITY DEFINER AS $function$ DECLARE v_leave_type_id INT; v_request_id INT; BEGIN -- original validation logic -- Ensure the user is requesting for themselves IF public.current_user_id() != request_time_off.user_id THEN RAISE EXCEPTION 'User can only request time off for themselves'; END IF; -- the rest of the function END; $function$;</code></pre> The similar update then applies to the api.update_request</code> function to ensure only managers can approve/reject time off requests. CREATE OR REPLACE FUNCTION api.update_request(request_id integer, new_status text) RETURNS void LANGUAGE plpgsql SECURITY DEFINER AS $function$ DECLARE v_requested_user_id INT; v_request_manager_id INT; BEGIN -- the original code up to the retrival of the requests IF public.current_user_id() != v_request_manager_id THEN RAISE EXCEPTION 'Only the manager can approve or reject this request'; END IF; -- update the request status END; $function$;</code></pre> And that's about it! Obviously, we can't pretend Time Off Manager is anywhere complete, and the real-life application would require much more than what we have covered. But it should provide a good training platform to allow you to write a real API-based backends using just PostgREST. Correction 2024-06-11: during the final edits the function public.current_user_id</code> somehow got missing from the Markdown [FIXED]. Custom PostgreSQL extensions with Rust 2024-05-24T00:00:00+00:00 This article explores the pgrx framework, which simplifies the creation of custom PostgreSQL extensions to bring more logic closer to your database. Traditionally, writing such extensions required familiarity with C and a deep understanding of PostgreSQL internals, which could be quite challenging. pgrx</code> lowers the barrier and allows developers to use Rust, known for its safety and performance, making the process of creating efficient and safe database extensions much more accessible. pg_sysload</a> </h2> When working with large datasets and migrations (as discussed in How Not to Change PostgreSQL Column Type</a>), or during resource-intensive maintenance tasks, you'll want to optimise speed and minimise disruption to other processes. One way to control the pace of batch operations is to consider the load on the underlying system. Many Unix-based systems (we will focus on Linux) provide a valuable metric called the system load average. This average consists of three values: the 1-minute, 5-minute, and 15-minute load averages. The load average is not normalised for the number of CPU cores, so a load average of 1 on a single-core system means full utilisation, while on a quad-core system, it indicates 25% utilisation. In many cases, the system load average is also an excellent indicator of how ongoing operations are impacting a busy database cluster. In this article, we will create a PostgreSQL extension with a function called sys_loadavg()</code> that retrieves this load information. We will use the /proc/loadavg</code> file (part of the proc filesystem), which exposes underlying system details. Getting Started with pgrx</code></a> </h2> Before we start, ensure you have: Rust installed (Get Started with Rust</a>)</li> System dependencies</a> installed for pgrx</code></li> </ul> With these prerequisites in place, you can install pgrx</code> itself and create a new extension skeleton: cargo install --locked cargo-pgrx cargo pgrx new pg_sysload cd pg_sysload</code></pre> This gives you a complete environment for developing your own PostgreSQL extensions in Rust. While Rust might seem daunting at first (especially with its async and other features), the language itself is quite powerful. Its ease of extension creation makes it worth a second look. Setting Up the Extension</a> </h2> While the pgrx new</code> command sets up a basic template, we will create a more sophisticated extension. Its primary logic involves parsing the /proc/loadavg</code> file, extracting its values, and returning them to the user. The logic itself is straightforward: #[pg_extern] fn sys_loadavg() -> Option<Vec<f64>> { // Read the contents of the /proc/loadavg let mut file = match fs::File::open("/proc/loadavg") { Ok(file) => file, Err(err) => { pgrx::error!("Error reading /proc/loadavg: {}", err); } }; let mut contents = String::new(); if let Err(err) = file.read_to_string(&mut contents) { pgrx::error!("Error reading /proc/loadavg: {}", err); } // Extract the load average fields let fields = contents.split_whitespace().collect::<Vec<_>>(); if fields.len() >= 3 { Some( fields[..3] .iter() .filter_map(|s| f64::from_str(s).ok()) .collect(), ) } else { pgrx::error!("Invalid format in /proc/loadavg"); } }</code></pre> To limit usage to systems supporting the proc filesystem, we check for the presence of the file when the extension is loaded: #[pg_guard] fn _PG_init() { INIT.call_once(|| { let loadavg_available = fs::metadata("/proc/loadavg").is_ok(); if !loadavg_available { pgrx::error!("/proc/loadavg not found. Extension cannot load."); } }); }</code></pre> This basic check prevents users from loading the extension on incompatible systems. With minor details omitted, that's almost all there is to it. You can find the complete lib.rs</code> file in the accompanying GitHub repository</a>. Running the Extension</a> </h2> Where pgrx</code> truly shines is in all the heavy lifting it does for you. It leverages the cargo</code> command. To initialise the extension, run: cargo pgrx init</code></pre> This downloads and prepares PostgreSQL source code for versions 12 to 16 (at the time of writing). After a bit of waiting (hopefully successfully, if you have all the system dependencies), run the extension: cargo pgrx run</code></pre> You'll get a psql</code> prompt for a dedicated PostgreSQL instance. Create and use the extension: CREATE EXTENSION pg_sysload;</code></pre> Now you can try the newly exposed function: SELECT sys_loadavg(); sys_loadavg ------------------ {1.17, 0.57, 0.32} (1 row)</code></pre> And that's all. Your very first PostgreSQL extension is working. Throttling Batch Processing</a> </h2> With the extension ready, let's revisit our original goal: throttling long-running batch data processing. The data from sys_loadavg</code> can be used in various ways: Dynamic Sleep Times: Insert calculated sleep intervals based on the system load.</li> Dynamic Batch Sizes: Adjust the number of rows processed per batch based on available resources.</li> </ul> Here's an example: -- Sleep for 5 seconds multiplied by the 1-minute load of the system SELECT pg_sleep((sys_loadavg())[1] * 5); -- Assume 20 cores and for each one available, add 100 rows to process SELECT ... LIMIT (SELECT (20 - (sys_loadavg())[1])::int * 100);</code></pre> This allows you to fine-tune processing for unsupervised operation, automatically adapting to the current system load. Conclusion</a> </h2> And there you have it! You've just built your first PostgreSQL extension using Rust and the pgrx</code> framework. I have always found the prospect of writing an extension quite daunting (probably because I'm long gone from the C-ecosystem), but this wasn't so bad. You've now got a handy tool for monitoring system load, perfect for keeping tabs on how your database processes long-running migrations. This is just scratching the surface. We haven't even touched on the really fun stuff – extending PostgreSQL's internals with custom data types, operators, or indexes. You could build extensions that transform or aggregate data, hook into external APIs, or create background workers, bringing various business logic directly into the database engine. And yes, tapping into the database's core can be a bit risky, and it is always important to assess risks and recovery options. But whatever you choose to create, remember, with pgrx</code> at your side, you've got the power of Rust to keep things safe and sound. Deep Dive into PostgREST - Time Off Manager (Part 2) 2024-05-18T00:00:00+00:00 Let's recap the first part</a> of "Deep Dive into PostgREST," where we explored the basic functionality to expose and query any table using an API, demonstrated using cURL</code>. All it took was to set up a db-schema</code> and give the db-anon-role</code> some permissions. But unless you are creating the simplest of CRUD applications, this only scratches the surface. In Part 2, we will expand APIs, provide better abstraction, and implement the foundation of what can be considered business logic, all while extending the sample "Time Off Manager" application. While the previous instalment being introductiory only, make sure you don't miss the important details in this one. Before we move on, let's do a bit of housekeeping and clean up the permissions setup from the first part. This way, we can start with a clean slate (when it comes to the permissions) and avoid any lingering rights that could confuse us later. REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA public FROM time_off_anonymous;</code></pre>Dedicated schema for the API</a> </h2> Using the public</code> schema (or any schema(s) where your core data model resides) is a fast way to get started. However, using a dedicated schema for the API is beneficial for several reasons: It provides options for better abstraction, which you might appreciate later when refactoring the original data model.</li> Data customisation is also a requirement unless you prefer building a "fat" client and thus transferring the majority of the business logic there. Combining data from multiple tables helps shield the consumer from complex queries and relations.</li> While we won't explore it as a security feature, it can also provide relevant boundaries.</li> </ul> Let's get started with the creation of the schema itself, and expose the users using a view and setting basic permissions. We will do this by setting default permissions, so we don't have to repeat the same for all objects as we create them. Please note that default privileges are applied only to new objects created. CREATE SCHEMA api; GRANT USAGE ON SCHEMA api TO time_off_anonymous; ALTER DEFAULT PRIVILEGES IN SCHEMA api GRANT SELECT ON TABLES TO time_off_anonymous; ALTER DEFAULT PRIVILEGES IN SCHEMA api GRANT EXECUTE ON FUNCTIONS TO time_off_anonymous;</code></pre> Ok, let's create our first view: CREATE VIEW api.users AS SELECT u.user_id, u.email, m.user_id as manager_user_id, m.email AS manager_email, u.created_at FROM public.users AS u LEFT JOIN public.users AS m ON u.manager_id = m.user_id WHERE u.deleted_at is null;</code></pre> This way, we established the foundation for listing users while providing much richer information about a person's manager and preventing potential additional roundtrips between client and API. We also included simple business logic—by omitting the soft-deleted employees. After creating the views, it's important to update the postgrest.conf</code> file to use the new api</code> schema: db-uri = "postgres://username:password@localhost/time_off_manager" db-schema = "api" db-anon-role = "time_off_anonymous"</code></pre> This change tells PostgREST to treat the api</code> schema as the primary interface for API requests, rather than the public schema. Once done, feel free to restart the PostgREST server and test the new setup. $ curl "http://localhost:3000/users"</code></pre> We can further extend the example by provisioning a view with a summary of the vacation days available per calendar year (of course, for simplicity we won't consider transfers between years, etc.). CREATE VIEW api.vacation_balances AS SELECT EXTRACT(YEAR FROM transaction_date) AS year, user_id, SUM(amount) AS total_amount FROM public.time_off_transactions JOIN api.users USING (user_id) WHERE leave_type_id = (SELECT leave_type_id FROM public.leave_types WHERE label = 'vacation') GROUP BY EXTRACT(YEAR FROM transaction_date), user_id;</code></pre> While this is nothing groundbreaking, we could (and should) have certainly used views in the first part. So what makes this important now? As mentioned above, the main use cases are abstraction, data access, and security. Additional benefits might include other concerns like derived columns (e.g., full name if we were using first and last name columns), enriching data (e.g., building full URLs from fragments), and other logic specific to the presentation layer. For practice, we can expose possible time off types via a simple view: CREATE VIEW api.leave_types AS SELECT label FROM public.leave_types WHERE deleted_at is null;</code></pre>Let's modify the data</a> </h2> While the simplicity of performing CRUD operations on tables in the public schema was straightforward in the first part of the tutorial, using VIEWs introduces certain complexities. One key limitation of VIEWs is their restricted capability for data modification. Let's look at how to encapsulate the logic. You might have guessed it—the most obvious way forward is to introduce a database stored procedure for any business logic we want to expose. CREATE OR REPLACE FUNCTION api.add_user( email text, manager_id integer ) RETURNS integer AS $$ DECLARE v_new_user_id integer; BEGIN -- check if email already exists PERFORM 1 FROM api.users WHERE users.email = add_user.email; IF FOUND THEN RAISE EXCEPTION 'The email address % is already in use', add_user.email USING ERRCODE = 'unique_violation'; END IF; -- sample business logic: all employees must have manager IF manager_id IS NULL THEN RAISE EXCEPTION 'manager_id must be provided and cannot be null'; END IF; INSERT INTO public.users (email, manager_id) VALUES (add_user.email, add_user.manager_id) RETURNING user_id INTO v_new_user_id; RETURN v_new_user_id; END; $$ LANGUAGE plpgsql SECURITY DEFINER;</code></pre> Ok, we have introduced a rather complex piece of logic there. While it might look simple (if you've ever seen a PL/pgSQL function), it comes with a couple of important details. First and foremost, the function api.add_user</code> presents an example of how to implement business logic. In this case, it's providing a custom error message should the employee's email already exist and enforcing the logic that manager_id</code> must be provided. You can probably think of more cases to enrich data. Now for the more difficult part—you might not be familiar with the SECURITY</code> attribute. Every function created can have two modes, INVOKER</code> (which is the default) and DEFINER</code>. If you look above, when we removed all the permissions for our db-anon-role</code> from schema public</code>, the user lost privileges to perform any operations there. Without modifying the security attribute, the function would result in an error message permission denied for table users</code>. Using SECURITY DEFINER</code> guarantees your application user's (the one used for the creating the view) permissions will be used when calling the function. This is the opposite of how VIEWs work in this example, as they come with the default option security_invoker</code> disabled by default. You can refer to the documentation</a> to learn more. If you are not familiar with PL/pgSQL, I recommend you check the basic statements</a>. With the function in place, let's call it using cURL: $ curl "http://localhost:3000/rpc/add_user" -X POST \ -d '{ "email": "admin2@example.com", "manager_id": 2 }' \ -H "Content-Type: application/json"</code></pre> This is the second time we need to deconstruct the seemingly simple logic. Let's start with the URL. The /rpc/</code> prefix is important. Every stored procedure in the schema defined using db-schema</code> is accessible under this prefix. The prefix helps to differentiate object types, as in PostgreSQL you are allowed to have the same name for a table/view and a function. The second part is the request method, in this case POST</code>. While you can use both GET</code> and POST</code>, it's important to understand the implications. Using the GET</code> method sets the PostgREST transaction to read-only mode and is a powerful security measure—explicitly preventing the operation from performing any modification of the data (even indirectly via functions or triggers). Should you perform the function using GET</code>, you would get a cannot execute INSERT in a read-only transaction</code> error message to confirm it. The third part of the call is the way the data is presented. In this example, it's using JSON (declared as Content-Type: application/json</code> header) and passed within the request body. If required, you can choose other formats—like application/x-www-form-urlencoded</code>, text/xml</code>, application/octet-stream</code> for bytea</code>, and more using custom media type handlers</a>. Therefore, the same can be achieved using: $ curl "http://localhost:3000/rpc/add_user" -X POST \ -d "email=admin2%40example.com&manager_id=2" \ -H "Content-Type: application/x-www-form-urlencoded"</code></pre> Let's omit the XML example completely [sic]. As we have introduced the function that modifies the data, we should explore the option to do the same with read-only functions. CREATE OR REPLACE FUNCTION api.get_max_vacation_days() RETURNS INTEGER STABLE AS $$ SELECT max_days FROM public.leave_types WHERE label = 'vacation'; $$ LANGUAGE sql SECURITY DEFINER;</code></pre> Here, we defined the function get_max_vacation_days</code>, which enables retrieving the current number of vacation days for all employees. To call the function, we can simply run: $ curl "http://localhost:3000/rpc/get_max_vacation_days"</code></pre> This summarises the basics of using a dedicated schema and VIEWs. While using stored procedures is the most obvious way, you can also use updatable views and INSTEAD OF TRIGGERS. This approach allows you to hide the possible transition from table to views and implement the insert logic using the trigger function. While this is a powerful technique, I recommend using direct function calls for clarity (at least when you are getting started). Simple workflow management</a> </h2> As we have introduced quite a lot of (potentially new) concepts, let's practice more and get our hands dirty with the implementation of a simple workflow management system. In our case, it's functionality for employees to request time off and manage approval flow. The premise is simple: Employee can request a time off of a certain type</li> Employee can approve/reject the time off request for their reports</li> The boss can do whatever they want</li> </ul> Let's start with defining the table for time off requests. Please notice we are going to create it again in schema public</code> and not directly expose it via the API. CREATE TABLE public.time_off_requests ( request_id SERIAL PRIMARY KEY, user_id INT REFERENCES public.users(user_id), leave_type_id INT REFERENCES public.leave_types(leave_type_id), requested_date DATE, period DATERANGE, status TEXT CHECK (status IN ('pending', 'approved', 'rejected')), created_at TIMESTAMP WITH TIME ZONE DEFAULT current_timestamp );</code></pre> We will add functionality to request time off via a function: CREATE OR REPLACE FUNCTION api.request_time_off( user_id INT, leave_type TEXT, period DATERANGE ) RETURNS INTEGER AS $$ DECLARE v_leave_type_id INT; v_request_id INT; BEGIN -- validate the leave type SELECT leave_type_id INTO v_leave_type_id FROM public.leave_types WHERE label = request_time_off.leave_type; IF v_leave_type_id IS NULL THEN RAISE EXCEPTION 'Invalid leave type: %', request_time_off.leave_type; END IF; -- check if the user ID is valid PERFORM 1 FROM public.users WHERE users.user_id = request_time_off.user_id; IF NOT FOUND THEN RAISE EXCEPTION 'Invalid user ID: %', request_time_off.user_id; END IF; -- insert the new time off request INSERT INTO public.time_off_requests ( user_id, leave_type_id, requested_date, period, status ) VALUES ( request_time_off.user_id, v_leave_type_id, CURRENT_DATE, request_time_off.period, 'pending' ) RETURNING request_id INTO v_request_id; RETURN v_request_id; END; $$ LANGUAGE plpgsql SECURITY DEFINER;</code></pre> For the purposes of this guide, let's leave details out, and we would expect the period</code> to be the applicable working days. To reiterate, the function above: Provides basic application logic, setting the status to 'pending'</li> Explicitly declares SECURITY DEFINER</code> to facilitate permissions to access the newly created table.</li> </ul> We can call it using: $ curl -X POST "http://localhost:3000/rpc/request_time_off" \ -d '{"user_id": 6, "leave_type": "vacation", "period": "[2024-05-20,2024-05-21]"} ' \ -H "Content-Type: application/json"</code></pre> For the client to be able to display the pending requests, let's introduce the view using the request fields and adding the manager ID of the employee, which we will utilise in the next step: CREATE VIEW api.pending_requests AS SELECT r.request_id, r.user_id, r.leave_type_id, r.requested_date, r.period, r.status, r.created_at, u.manager_id FROM public.time_off_requests r JOIN public.users u ON r.user_id = u.user_id WHERE r.status = 'pending' ORDER BY r.created_at;</code></pre> With this in place, we can move forward with implementing the function api.update_request</code> to provide the necessary business logic to approve/reject the requests. CREATE OR REPLACE FUNCTION api.update_request( request_id INT, user_id INT, new_status TEXT ) RETURNS VOID AS $$ DECLARE v_requested_user_id INT; v_request_manager_id INT; BEGIN -- validate the new status IF new_status NOT IN ('approved', 'rejected') THEN RAISE EXCEPTION 'Invalid status: %. Only "approved" or "rejected" are allowed', new_status; END IF; -- retrieve the request together with the users associated with it SELECT pr.user_id, pr.manager_id INTO v_requested_user_id, v_request_manager_id FROM api.pending_requests pr WHERE pr.request_id = update_request.request_id; IF NOT FOUND THEN RAISE EXCEPTION 'There''s no pending Time off request ID %', request_id; END IF; -- prevent users from self-approving their requests IF user_id = v_requested_user_id THEN RAISE EXCEPTION 'User cannot approve or reject their own request'; END IF; -- check if the user is either the requester’s manager or the boss IF (v_request_manager_id IS NOT NULL AND user_id <> v_request_manager_id) THEN RAISE EXCEPTION 'Only the manager or The Boss can approve or reject the request'; END IF; -- update the request status UPDATE public.time_off_requests SET status = new_status WHERE time_off_requests.request_id = update_request.request_id; END; $$ LANGUAGE plpgsql SECURITY DEFINER;</code></pre> This time, the function shouldn't have any surprises—just ensure you follow all the points addressed previously. With the functionality to approve/reject the requests, we are missing the last crucial step: deducting the balance upon approval. There are two ways to go about this: either add the logic to the function above or implement triggers. While a full discussion goes beyond this guide, let's provide a brief summary of when to choose which solution. Choose a function if you have complex logic reusable by many functions (which does not prevent re-use in triggers), and use triggers if you prefer the automatic business rule enforcement and consistency on the database level. For this tutorial let's do triggers to expand further the use of different SQL techniques (plus I would personally choose this solution anyway). Before jumping in, let's create a helper function that will calculate the number of days to deduct from the balance. CREATE OR REPLACE FUNCTION public.days_in_daterange(period daterange) RETURNS INT AS $$ SELECT upper(period) - lower(period) $$ LANGUAGE sql;</code></pre> With it in place, we can create the function: CREATE OR REPLACE FUNCTION public.create_transaction_on_approval() RETURNS TRIGGER AS $$ BEGIN IF NEW.status = 'approved' THEN INSERT INTO time_off_transactions ( user_id, leave_type_id, transaction_date, time_off_period, amount ) VALUES ( NEW.user_id, NEW.leave_type_id, CURRENT_DATE, NEW.period, -public.days_in_daterange(NEW.period) ); END IF; RETURN NEW; END; $$ LANGUAGE plpgsql;</code></pre> And set up the trigger itself: CREATE TRIGGER create_transaction_on_approval AFTER UPDATE ON time_off_requests FOR EACH ROW WHEN (NEW.status = 'approved' AND OLD.status = 'pending') EXECUTE FUNCTION create_transaction_on_approval();</code></pre> With the approval logic in place, we can try the original function via API: $ curl -X POST "http://localhost:3000/rpc/update_request" \ -d '{"request_id": 1, "user_id": 2, "new_status": "approved"}' \ -H "Content-Type: application/json"</code></pre> If you have used the correct IDs (both for request ID and user ID), you can now verify the balances using the pre-defined view vacation_balances</code>. $ curl "http://localhost:3000/vacation_balances"</code></pre> This demonstrates the basic workflow and how it can be exposed using PostgREST as an API. If you want to continue practising more database logic, you can add the following logic: Prevent negative vacation balances (intermediate)</li> Prevent employees of a single manager from requesting overlapping vacations (advanced)</li> </ul> End of Part 2</a> </h2> By introducing a dedicated API schema, we have added an abstraction layer that provides an effective boundary between the model/logic itself and the API specifics. While this is a good practice, it's up to you how to expose functionality. The main benefit of a dedicated schema is simplicity, allowing you to grant privileges on the schema in bulk rather than maintaining granular permissions. It also provides an easy way to expose existing databases with PostgREST. While the current state of the sample API for Time Off Manager would work for small teams, in the next part, we will move towards authentication for added security and privacy. And don't forget you can find the source code in the GitHub repository</a>. Deep Dive into PostgREST - Time Off Manager (Part 1) 2024-05-11T00:00:00+00:00 The primary motivation behind boringSQL is to explore the robust world of SQL and the PostgreSQL ecosystem, demonstrating how these "boring" tools can cut through the ever-increasing noise and complexity of modern software development. In this series, I'll guide you through building a simple yet fully functional application—a Time Off Manager. The goal of this project is not only to demonstrate practical database/SQL approaches but also to provide a complete, extendable solution that you can immediately build upon. Each part of this series will deliver a self-contained application, setting the stage for introducing more complex functionalities and practices. The first part of this guide focuses mainly on the application logic and exposing raw data using postgREST, allowing you to grasp the concept. Requirements</a> </h2> This tutorial assumes you can install PostgreSQL, connect to it using your preferred DB client, have permissions to create a new database, and understand the basics of schema operations and SQL. Similarly, you should be able to follow the installation instructions for postgREST</a>. The complete source code for the guide is available at GitHub</a>. The Time Off Manager</a> </h2> When choosing a sample application for this tutorial, I was torn between the popular but simple TodoMVC and a slightly more complex application. In the end, a sample like Time Off Manager provides a much richer database scheme, going beyond the basics and offering a solution closer to real-life systems. Time Off Manager is also an excellent example that combines simple business logic, workflows, and practicality. The main requirements are: Maintaining the list of users (employees) with their respective managers</li> Allowing the maintenance of the time off balance, with an audit history</li> Introducing an approval workflow and automation</li> </ul> While the application is intended for learning purposes, the database and API can be easily exposed and built upon. The Database</a> </h2> To get started, you will need to create a database. You can do this in two ways, either using the CREATE DATABASE statement: CREATE DATABASE time_off_manager;</code></pre>Users model</a> </h2> The foundation and first sample functionality exposed by postgREST is the users themselves. The table schema behind it represents the employees and manages hierarchical relations, which will become important in the second part when we set up the approval workflow. CREATE TABLE users ( user_id int GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, email text NOT NULL UNIQUE, manager_id int REFERENCES users(user_id), created_at timestamp with time zone DEFAULT current_timestamp, deleted_at timestamp with time zone );</code></pre> For testing purposes, let's populate the data using a seed of 3 managers (one being "the boss" and hence not having a manager) and 10 employees. DO $$ DECLARE v_boss_id int; v_manager_id int; i int; BEGIN -- create "the boss" INSERT INTO users (email, manager_id) VALUES ('owner@example.com', NULL) RETURNING user_id INTO v_boss_id; -- setup 2 managers FOR i IN 1..2 LOOP INSERT INTO users (email, manager_id) VALUES ('manager' || i || '@example.com', v_boss_id) RETURNING user_id INTO v_manager_id; -- and 5 employees for each one of them FOR j IN 1..5 LOOP INSERT INTO users (email, manager_id) VALUES ('employee' || (5 * (i - 1) + j) || '@example.com', v_manager_id); END LOOP; END LOOP; END $$;</code></pre> In case you are not aware, the seed data is produced by an anonymous block</a>. This way, we executed the logic to create the relationship required without the need to create (and later drop) the PL/pgSQL function. Expose users using postgREST</a> </h2> Having the users table in place, together with sample data, we are ready to expose the data using the REST API. To start, you only need to create a very simple configuration for postgREST using the file postgrest.conf (you can place the file in the folder created for this sample project). Sample postgrest.conf</code>: db-uri = "postgres://username:password@localhost/time_off_manager" db-schema = "public" db-anon-role = "time_off_anonymous"</code></pre> The most important part is the db-uri, which follows the standard PostgreSQL connection string format: postgres://username:password@host/database_name</code></pre> You need to replace: username</code> - with your actual database username. Ideally, you will use a separate user (for example, time_off_manager) for each application, rather than your personal account.</li> password</code> - the password for the user</li> host</code> - either localhost (if running locally) or a remote server name or IP address</li> database_name</code> - in this case, time_off_manager we created earlier.</li> </ul> NOTE: We are using hardcoded data (which can potentially get stored in your source code repository), and this is mainly for learning purposes. Any deployment resembling a production environment should follow best practices to reduce security risks. The other configuration options are db-schema</code>, which we will leave pointing to the public schema for this part, and db-anon-role</code>, configuring the role postgREST should use when executing requests on behalf of unauthenticated clients (effectively all requests in Part 1). For PostgreSQL 15 and higher, please make sure your username</code> is the owner of the database created (should you create it alternatively) in order to be able to create objects in public</code> schema. To set up the database role, you need to run the following commands: CREATE ROLE time_off_anonymous NOLOGIN; GRANT SELECT, INSERT, UPDATE ON users TO time_off_anonymous; GRANT time_off_anonymous TO CURRENT_USER;</code></pre> Please note you this example assumes CURRENT_USER</code> is same as the username used in the db-uri</code> configured above. The latter GRANT statement assigns the role time_off_anonymous to the configured user to execute the anonymous commands. Once you have the configuration file ready (assuming it's in the current directory), you can start the postgREST server and expose it locally on port 3000 (by default): $ postgrest postgrest.con 10/May/2024:22:06:12 +0200: Starting PostgREST 12.0.3... 10/May/2024:22:06:12 +0200: Attempting to connect to the database... 10/May/2024:22:06:12 +0200: Connection successful 10/May/2024:22:06:12 +0200: Listening on port 3000 10/May/2024:22:06:12 +0200: Listening for notifications on the pgrst channel 10/May/2024:22:06:12 +0200: Config reloaded 10/May/2024:22:06:12 +0200: Schema cache loaded</code></pre>Exploring the Users Model Using cURL</a> </h2> When you have the server successfully running, you can try to access the data using: $ curl "http://localhost:3000/users"</code></pre> and get the list of all the sample data we created above. The API for reading data is described in detail</a> in the documentation, but you can try different alternatives. # get one specific user $ curl http://localhost:3000/users?user_id=eq.10 # query only active (i.e. non-deleted users) curl "http://localhost:3000/users?deleted_at=is.null" # or display users without any manager (i.e. boss) curl "http://localhost:3000/users?manager_id=is.null"</code></pre> Similarly, you can create, update, and delete users as required—giving you full CRUD functionality with rich filtering options. Create user: $ curl -X POST "http://localhost:3000/users" \ -H "Content-Type: application/json" \ -d '{"email": "admin1@example.com", "manager_id": 1}'</code></pre> Update user: $ curl -X PATCH "http://localhost:3000/users?user_id=eq.10" \ -H "Content-Type: application/json" \ -d '{"email": "updateduser@example.com"}'</code></pre> or, delete one: $ curl -X DELETE "http://localhost:3000/users?user_id=eq.10"</code></pre> I haven't mentioned CRUD by accident; that's the primary use-case of postgREST—exposing CRUD operations on the tables within the configured schema (public</code> in this part). You can delegate authorisation at the database level. Please note, the IDs are hardcoded, and you might need to check the output of the commands to get the correct ones (should you have truncated the table before or otherwise manipulated the data). Basic models for managing time off</a> </h2> With the User model set up and tested via the API, it's time to return to the core of the Time Off Manager functionality: absence tracking itself. For these purposes, we will define two tables—leave types (identifying the reason or type of the leave) and the time off transactions (or updates) themselves. CREATE TABLE leave_types ( leave_type_id int GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, label text NOT NULL, description text, max_days int, created_at timestamp with time zone DEFAULT current_timestamp, deleted_at timestamp with time zone ); CREATE TABLE time_off_transactions ( transaction_id int GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, created_at timestamp with time zone DEFAULT current_timestamp, user_id int NOT NULL REFERENCES users(user_id), leave_type_id int NOT NULL REFERENCES leave_types(leave_type_id), transaction_date date, time_off_period daterange, amount int NOT NULL, description text ); CREATE INDEX transaction_for_user ON time_off_transactions(user_id);</code></pre> The tables should be self-explanatory, but for clarity, let's delve into the detail of the transaction tracking. Time off is tracked for individual users, the time off transaction type to identify the reason for the change, the period during which the absence is recorded, and the amount of effective days (simplified logic to account for weekends, public holidays, and similar). To get started we also need to seed the data: INSERT INTO leave_types (label, description, max_days) VALUES ('vacation', 'Annual vacation leave', 25), ('sick-leave', 'Leave for health reasons', 10), ('unpaid-leave', 'Leave without pay', NULL), ('sabbatical', 'Extended leave for study or travel', NULL);</code></pre> and create the initial time off balances (this is a simplified version, assuming the employee always has the full balance, ignoring possible mid-year joiners, etc.): DO $$ DECLARE v_leave_type record; BEGIN FOR v_leave_type IN SELECT * FROM leave_types WHERE label = 'vacation' LOOP INSERT INTO time_off_transactions (user_id, leave_type_id, transaction_date, amount, description) SELECT user_id, v_leave_type.leave_type_id, '2024-01-01', v_leave_type.max_days, 'Initial balance for year 2024' FROM users; END LOOP; END $$; </code></pre> This now gives us the ability to start tracking the leave of absence for users and keep a full audit log of it. Before we can start using the table via the API, we need to complete the last important step: giving access to the configured db-anon-role for the tables. We can do this by assigning the grant to individual tables using: GRANT SELECT, INSERT, UPDATE ON leave_types TO time_off_anonymous; GRANT SELECT, INSERT, UPDATE ON time_off_transactions TO time_off_anonymous;</code></pre> or modify the default privileges. Spoiler alert—we will move away from using the public schema in Part 2, so let's not do more than we need at the moment :) Working with absence</a> </h2> Similar to the users example, we can now track absence directly using the API. Before firing off cURL commands, there's one last thing to remember. postgREST requires metadata about the database schema itself, and it might be expensive to perform on the fly, hence to avoid this, the server caches the schema. To reload the schema (after making schema changes), it is required to reload the cache. This can be done via several basic options: restarting the server</li> issuing a (SIGUSR1) signal to the server process</li> </ul> and more advanced ways</a> (outside Part 1 of this tutorial). Once reloaded, you can start exploring more: # getting a time off transaction for particular user $ curl "http://localhost:3000/time_off_transactions?user_id=eq.1" # submitting new leave of absence $ curl -X POST http://localhost:3000/time_off_transactions \ -H "Content-Type: application/json" \ -d '{"user_id": 5, "leave_type_id": 1, "transaction_date": "2024-02-26", "time_off_period": "[2024-02-28,2024-03-04]", "amount": -4, "description": "Vacation to avoid panic over leap year"}'</code></pre> All these commands work as expected. Should you forget to reload the schema, you might face an HTTP Status 404 (Not Found) on the newly created objects. End of Part 1</a> </h2> While not being very sophisticated, I hope this first part has demonstrated the CRUD capabilities postgREST can offer with minimal effort. In the next part, we will move away from accessing the database tables directly and provide more functionality via a new schema api, providing a per-user time off balance view and introducing workflow management. How not to change PostgreSQL column type 2024-05-04T00:00:00+00:00 One of the surprises that comes with developing applications and operating a database cluster behind them is the discrepancy between practice and theory, development environment and the production. A perfect example of such a mismatch is changing a column type. The conventional knowledge on how to change a column type in PostgreSQL (and other systems compliant with the SQL standard) is to: ALTER TABLE table_name ALTER COLUMN column_name [SET DATA] TYPE new_data_type</code></pre> which is obviously the semantically correct way, but given the right circumstances, you might be set for a rather unpleasant surprise. The Problem</a> </h2> Let's create a sample table and demonstrate the isolated behaviour that you might observe. Let's start with 10 million rows (which really is just a drop in the whole world of data). -- create very simple table CREATE TABLE sample_table ( id INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, label TEXT, created_at TIMESTAMP WITHOUT TIME ZONE DEFAULT NOW(), updated_at TIMESTAMP WITHOUT TIME ZONE DEFAULT NOW() ); -- populate with 10m records INSERT INTO sample_table (label) SELECT 'hash: ' || md5(random()::text) FROM generate_series(1, 7000000);</code></pre> and let's change the id type from INT to BIGINT. alter_type_demo=# ALTER TABLE sample_table ALTER COLUMN id TYPE bigint; ALTER TABLE Time: 21592.190 ms (00:21.592)</code></pre> And ... 21 seconds later, you have your change. Mind you, this is a small table with roughly 600 MB of data in it. What if you going to face 100x that amount? Let's have a look what went on behind the scene. The things PostgreSQL must do</a> </h2> Changing a data type (and many other operations you might experience) is no simple task, and the PostgreSQL engine has to perform several tasks: Rewrite the table is the most obvious culprit. Changing a column from INT to BIGINT requires 4 additional bytes to be allocated for every single tuple (ahem, think row). As the original table schema required a fixed amount of bytes, the cluster stored them in the most efficient way. i.e., every single row, in our example 10 million of them, must be read and re-written using the correct tuple size.</li> Locks might not be a problem in our synthetic example, but should you execute the ALTER command in production with hundreds or thousands of concurrent queries, you will have to wait for all of them to release the locks.</li> Indexes and constraints - if the column being altered is indexed or has constraints, they need to be rebuilt/revalidated. This is additional overhead.</li> Transactions & Write-Ahead Log is another big part of the problem. To guarantee durability (the 'D' in ACID), PostgreSQL has to record each change in WAL files. This way, if the database crashes, the system can replay the WAL files to reconstruct the lost modifications since the last checkpoint.</li> </ul> As you can see, there's quite a bit involved in performing something that might be understood as routine table maintenance. The size of the data being modified, disk I/O and capacity, and general system congestion come into play. But the real problem does not end here. If we are talking about any sort of serious production deployment, you must consider more things: Real-time replication, both physical and logical: This adds an additional layer of complexity. For read replicas, the default behaviour ensures that a synchronous commit is maintained to achieve consistency across the database cluster. This setup guarantees that a transaction is only finalised once all standby replicas have confirmed receipt of the changes. However, this introduces new challenges, as the performance now also depends on network throughput—including potential congestion—and the latency and I/O performance of the standby nodes.</li> Recovery and backups are another important area to consider. While the regular backups' size might not be affected that much, you have to consider everything that happens between the last backup before the change and the next one and ensure point-in-time consistency.</li> Less common but not unheard of might be asynchronous replicas or reserved slots for logical replication. Generating a large volume of changes (and therefore WAL files) can leave the less performant (or infrequent) replication systems behind for a considerable amount of time. While this might be acceptable, you need to make sure the source system has enough disk space to hold the WAL files for a long enough time</li> </ul> As you can see, altering the column data type is not as straightforward as it might seem. Current CI/CD practices often make it very easy for software developers to commit and roll out database migrations to a production environment, only to find themselves in the middle of a production incident minutes later. While a staging deployment might help, it's not guaranteed to share the same characteristics as production (either due to the level of load or monetary constraints). The problem is, therefore (and I will repeat myself), the scale of the amount of data being modified, overall congestion of the system, I/O capacity, and the target table's importance in the application design. At the end of the day, it translates to the total time needed for the migration to finish, and the unique constraints your business can or maybe cannot afford. The simplest solution to the problem is to schedule the planned maintenance during low traffic periods and get it done. How to Safely Change a PostgreSQL Column Type</a> </h2> What if you need to rewrite hundreds of gigabytes or even terabytes of data and can't afford anything more than minimal downtime? Let's explore how to change a column type properly. Let's start with The Bad News - you cannot avoid rewriting the entire table, which will generate a significant amount of WAL files in the process. This is a given, and you must plan how to manage it. The Good News: You can spread the potential downtime over a much longer period than it might take to process the data. The specific requirements and constraints will vary based on individual business needs, so careful planning is essential. The full migration can be summarised as series of following steps: Add a new column to the target table with the correct type. Ensure the column is NULLable and does not have a default value to avoid forcing a full table rewrite[^1]. For example, should you need to increase ID of order_id</code> you will end up with new column new_order_id</code>.</li> Setup a trigger to update the new column as new data comes in. This ensures that all new data during the migration will have the new column populated.</li> Implement a function or logic to batch migrate the values from old to new column over time. The size of the batch and the timing should align with the operational constraints of your business/environment.</li> Migrate Old Values: Depending on your constraints, data size, and I/O capabilities, this process could take anywhere from hours to several weeks, or possibly longer. While a SQL or PL/pgSQL function running in a terminal session (consider using tmux) might suffice for shorter migrations, more extended migrations may require a more sophisticated approach. This topic alone could be a good subject for a separate blog post or guide.</li> Once the migration is complete, create constraints and indexes that reflect the new column. Be aware of potential locking issues, especially if the field is part of any foreign keys.</li> </ul> At this point you are ready to perform the switch itself. If you can verify all rows have correctly populated new column, it's time to embrace the most difficult part. If possible in one transaction and or smaller scheduled downtime Drop the legacy column. This operation usually only locks the table for a short duration.</li> After dropping the old column, rename the new column. This step completes most of the migration process.</li> </ul> It's good practise to consider the restart of all the application relying on the changed table, as some tools (ORMs... I'm looking at you) might cache the OIDs and not handle the change gracefully. And that would be it - except not really. Dropping the column only removes the reference and the data itself will remain physically on the disk. This is the scenario where you will might need to perform VACUUM FULL</code> - which could lock the table and rewrite it completely—potentially defeating the purpose of a concurrent migration. This brings us back to the original article which motivated me to write this guide - [[The Bloat Busters: pg_repack vs pg_squeeze]] is the way to go. Preparation and familiarity with these tools in advance are highly recommended. Conclusion</a> </h2> While changing the column type in PostgreSQL can be as simple as issuing an ALTER TABLE command, it is important for everyone involved to understand the complexities attached to it. Whether you are the software developer requesting the change, someone reviewing it, or the individual tasked with resolving incidents when such changes are deployed to the production environment without careful planning, a deep understanding of this process is crucial. Moreover, grasping this particular change enables you to easily project insights onto other potentially costly operations. [^1] Correction: it's actually done without full table rewrite since PostgreSQL 11 (Fast Column Creation with Defaults</a>) The Bloat Busters: pg_repack vs pg_squeeze 2024-04-27T00:00:00+00:00 As the database size increases and the number of transactions per second rise, you'll inevitably face the challenge of the table bloat. Although PostgreSQL assists as much as possible with its auto-vacuum feature</a>, there will come a time when you will compel whether to run VACUUM FULL</code>. Unless you have option of longish downtime windows, this is not an easy decision. Thankfully, the rich ecosystem of PostgreSQL offers more than one solution how to make it simpler. Ignoring older tools like pg_reorg</code>, two contenders worth considering are pg_repack and pg_squeeze. This article dives into their strengths and weaknesses to help you decide which one is better for your specific use case. Cases of Heavy Duty Maintenance</a> </h2> In scenarios with a continuous and predictable transaction pattern, you can usually rely on auto-vacuum. However, there are use cases where this won't be sufficient. Such examples typically involve "bulk" operations—whether it's bulk imports, deletions, or a combination of both. Imagine scenarios where: You migrate one or more column data types</a> over a longer period (and no, using ALTER COLUMN name TYPE new_type</code> is not the best option).</li> You drop a column that has been moved to a different table.</li> You need to modify a large amount of data, using soft-deletes and later actual DELETEs</a>.</li> Due to changes in compliance requirements, you need to DELETE a massive amount of data.</li> </ul> In all these cases, you might end up with a huge table and the traditional solution would be VACUUM FULL</code>, which is associated with significant downtime due to table locking. Alternatives to VACUUM FULL</a> </h2> Due to the limitations of VACUUM FULL</code>, several alternatives have emerged. The first contender in this space was pg_reorg</code> (which this comparison will not cover), later superseded by pg_repack</code>. The relatively new kid on the block is pg_squeeze</code>. Each solution comes with different architectures, deployment methods, and sets of pros and cons. As an added benefit, both tools can effectively serve as alternatives to CLUSTER</code> (which also requires the exclusive table lock similar to full vacuum), making them effective in optimising data storage based on the given order. Both tools also share common behaviour. They won’t magically remove the bloat—you need to have at least the same amount of space available as the new table will require. Therefore, deploying both tools needs to be considered well before you reach critical levels of available disk space. Both will also generate a significant volume of WAL files — affecting every 8KB page</a> along the way — which need to be stored, processed, backed up, etc. While not directly comparable to these tools, another method that can be used manually is table partitioning. Although it won’t be included in this comparison, if planned in advance, it can simplify certain maintenance tasks, improve performance, and make routine vacuuming faster and more efficient. It's only fair to say - it all depends on the specific scenarios and usage. pg_repack</a> </h2> At the time of writing this article, pg_repack</code> can be considered the most well-known solution for combating table bloat in the PostgreSQL ecosystem. Built as an extension, it's easy to install (either from package repositories or via a self-compiled artifact) and its setup does not require a cluster restart. pg_repack</code> works by creating a new copy of the table being processed, setting up triggers to replicate new data while it fills up with the existing data. An exclusive full table lock is required both at the start and finish of the process when swapping the old and new tables. The maintenance is initiated from the CLI and allows you to specify a number of arguments to fine-tune the repacking of individual tables to match the requirements. pg_repack</code> allows re-clustering of data based on columns only, i.e., it does not explicitly require an index and can therefore overcome some limitations of pg_squeeze</code> in this regard. pg_repack</code> is very effective in reclaiming space and can work with all types of bloat. It's available out of the box both on Amazon RDS and Google Cloud SQL. The drawback you might experience when terminating the process (which may be necessary for various reasons, such as impact on the running environment) is that it won’t clean up all the fragments as it won't remain connected to the target database. pg_squeeze</a> </h2> Compared to the previous solution, pg_squeeze</code> is built differently, relying on logical decoding instead of triggers. Its main benefit is a lower impact on the host system during table rebuilding, improving availability and stability. Like pg_repack</code>, pg_squeeze</code> creates a new table and copies the existing data from the bloated table. Logical replication is involved in streaming changes from the original table to the newly created one in real-time. This allows the new table to stay up-to-date during the process without unnecessary impact on the regular operations performed on the bloated table. Thus, pg_squeeze</code> significantly reduces the need for locking. The exclusive lock is needed only during the final phase of the operation, when the old table is swapped out for the new, optimized table. The duration of the exclusive lock can also be configured. While pg_squeeze</code> is also available as an extension, its deployment necessitates configuration changes involving wal_level</code>, max_replication_slots</code>, and shared_preload_libraries</code> — the same settings used for logical replication</a>. Due to this, a restart of the cluster is required. On the other hand, pg_squeeze</code> is designed for regular, rather than ad-hoc processing only. You can register a table for regular processing, and whenever the table meets the criteria to be "squeezed," a task will be added to a queue, where it will be sequentially processed in the order they were created. The automated processing offers basic options usable in most scenarios, but you might find it limited if you need to work around other operational constraints of the cluster/environment. Having said that, the maintenance of the table can also be triggered manually. While pg_squeeze</code> might be considered superior to pg_repack</code> in terms of maintenance operations and impact, it comes with a significant caveat when reclaiming space—compared to pg_repack</code>, it copies the full rows as they are. This behaviour renders it ineffective at removing bloat created due to dropped columns (behaviour still present at the time of the writing of this article at the end of April 2024). As already mentioned, pg_squeeze</code> can use a specified index for clustering when needed. The limitation you might find is that clustering cannot be performed on a partial index. Compared to pg_repack</code>, it always seems to clean up all the artefacts accordingly (thanks to the always-running worker process). Unfortunately at the moment of writing the article pg_squeeze</code> is not available for Amazon RDS, only Google Cloud SQL. Conclusion</a> </h2> It's remarkable to have two mature and production-tested tools at your disposal. Deciding between them comes down to the specific requirements, the use cases, and the operational specifics of the business services. The very opinionated difference between the tools can be made, using pg_squeeze</code> for automated, continuous cleaning of specific tables, and pg_repack</code> as the heavyweight champion of controlled setups. (Edit) Added availability on Amazon RDS and Google Cloud SQL. Are SQL & Databases Boring? Absolutely - and That's a Good Thing! 2024-04-21T00:00:00+00:00 Twenty years ago, I would have laughed if you had told me I'd be promoting databases as the technology to turn to. Back then, databases were just a 'dummy' storage for me—a necessary evil, a sentiment shared by many developers. At that point, I felt so strongly about it that I was a trainer for Hibernate, a tool which helped me keep those pesky databases at arm's length. Fast forward, and here I am, starting boringSQL, a site dedicated to helping demystify the depths of SQL and databases in general. Whether you've just started or want to gain expert knowledge. What changed, you might ask? Nothing actually. Well, a few things did. The frameworks changed. I moved from the Java world to Rails, jumped to Python and then to Go, worked with countless frontend paradigms, and oversaw the implementation of a number of technologies. The same experiments were tried again and again. Only one part of the tech stood the test of time. Yes, it's the database. And out of many, one in particular—PostgreSQL. And while alternatives promised, and still promise, that the grass would be greener—I can honestly say that whatever revolution comes in the technology landscape, databases will hold the same place in the next twenty years. This is not the first boring site, and rest assured, the concept of celebrating the mundane in technology is far from new. The tech landscape is paved with many "boring" things, from frameworks to methodologies to complete stacks. Businesses depend on these technologies to run their operations day in, day out. And while it might be fun to try new things every month, it's not the headline-driven development that is the way forward. Hence, if you ask—is SQL boring? Are databases boring? I would also say YES. They are among the finalists of the least popular conversation starters. But it's the quiet power of reliability that has a healing effect in a world where technology changes at the speed of light. When and Why PostgreSQL Indexes Are Ignored 2024-04-14T00:00:00+00:00 While it's true the most problems can be solved by the appropriate use of the index, there are cases where you will just waste resources doing so. For casual developer it might seems like PostgreSQL decided to do its own thing, but when you look behind the scenes it all makes perfect sense. Here's a quick run down of the some reasons why planner might pass on index and rather do things without it. The case of missing condition</a> </h2> In the evolution of the software developer journey into the realms of databases, the partial indexes are the next best thing. Why to create the full index where you can easily restrict it to the sub-set of the records? The trick is ensure every query aligns with the condition given during the index creation. Think of an order management system, where most active orders are going to those in 'open' status. Most of the day-to-day operational tasks will resolve around those entries, a partial index like CREATE INDEX open_orders ON orders(order_id) WHERE state = 'open';</code></pre> provides a perfect match. It will allow to iterate only on a relatively small subset of the data, hence saving the disc space. As long as the application developers are aware of the partial condition, and how to use it. The column order matters</a> </h2> The second common case is just a step away from the partial index. The compound indexes, or multi-column indexes, are versatile and powerful - that is when used correctly. The crucial element is the order in which the columns are indexed. The order impacts whatever it gets used or not at all. The key is to match the left-most columns of the index in your queries. Once these are aligned, you can optionally skip or include additional columns in the filter, but the initial columns must be used to leverage the index effectively. While it might be easy not to miss the condition of the country, when filtering for cities CREATE INDEX contact_location ON contact_details (country_id, city);</code></pre> consider the scenario which might not be so obvious. CREATE INDEX brand_products_per_category ON products (category_id, brand_id);</code></pre> In this case the index will only get used if either both category_id</code> and brand_id</code> are used, or at least category_id</code> - but not for brand_id</code> alone. In latter case PostgreSQL planner might (unless another index is available) to opt in for scan of the entire table. In this case alternative strategy would be to switch the column order (should the application logic support it). Low Selectivity</a> </h2> If we use the example of the partial index from the first section, we can easily demonstrate another case when index just might get ignored forever. It's the case for the columns with low selectivity, where a predominant value overshadows others, making an index less useful. Take example of the ordering system. While having index for open</code> orders is helpful, in most scenarios majority of the records will end up in delivered</code> state. Hence CREATE INDEX delivered_orders ON orders(order_id) WHERE state = 'delivered';</code></pre> might seem beneficial, but in reality PostgreSQL will most likely chose to skip it and instead to scan the table. The reason? Should you consider the regular business operations, you might find vast majority (let's say 95%) or orders shipped and considered finished. With such a high percentage of the records orders, the index will do little to narrow the search for the records. Planner hence will perform the sequential scan directly. Why? It's all to do with the statistics that are available on the table. table_name | orders column_name | state n_distinct | 4 most_common_vals | {delivered,cancelled,pending,open} most_common_freqs | {0.95023334,0.030166665,0.013,0.0096}</code></pre> Those are fictional statistics matching the order distribution described above. From there you can see that delivered</code> orders dominate the dataset, compromising approximately out of 95.02% of the records. The other statuses make up just for a small fraction. The above data sample comes from the query similar to SELECT tablename AS table_name, attname AS column_name, n_distinct, most_common_vals, most_common_freqs FROM pg_stats WHERE tablename = 'orders' AND schemaname = 'public';</code></pre>Outdated Statistics</a> </h2> As demonstrated low selectivity to addressed by statistics, instead of index, it's not the only case where stale data can lead to inefficient query planning. Keeping database statistics current is crucial for PostgreSQL to make informed decisions about whether to use an index or opt for a sequential scan. PostgreSQL heavily relies on statistics to estimate costs and make the right decisions across different query execution plans. The statistics cover various data points, like total number of rows in table(s), distinct values, their distribution, histogram bounds, correlation between physical row ordering and column values, and much more. The trouble starts when statistics get out-of-date. It's similar as navigating using old maps. It just might you send you going in circles. How might the statistics in PostgreSQL get dated? Most cases involve high volume of data modifications or bulk operations, but will be affected also by schema changes. For all those operations it's always good idea to ANALYZE</code> the table to keep the DB up-to-date. The prevent the statistics outdated, PostgreSQL alone either manual ANALYZE</code> or periodically tries to trigger it as part of autovacuum daemon. The key factor is whatever it can run fast and frequently enough to keep up with the changes. You can adjust the autovaccuum settings globally or per-table. Keeping statistics up-to-date is crucial for maintaining good query performance, as it ensures that the query planner has accurate information to base its decisions on. Summary</a> </h2> As demonstrated creating index might not be always the best course of action. Understanding those edge-cases is crucial not only for DBAs but also for developers. By keeping those considerations you can better design the schema and indexing strategy. Indexes are still the next best thing in database world, but they require thoughtful implementation and maintenance. The goal is not only to create indexes, but to create the right indexes based on accurate, up-to-date data and aligned with your specific query patterns.

Condition</th>	PG ≤ 11</th>	PG 12–16</th>	PG 17–18</th></tr></thead>
Single ref, pure SELECT</td>	Materialized</td>	Inlined</td>	Inlined</td></tr>
Multiple refs, pure SELECT</td>	Materialized</td>	Materialized</td>	Materialized (better stats)</td></tr>
VOLATILE function</td>	Materialized</td>	Materialized</td>	Materialized</td></tr>
STABLE function</td>	Materialized</td>	Inlined</td>	Inlined</td></tr>
Data-modifying (DML)</td>	Materialized</td>	Materialized</td>	Materialized</td></tr>
FOR UPDATE / FOR SHARE</td>	Materialized</td>	Materialized</td>	Materialized</td></tr>
Recursive</td>	Materialized</td>	Materialized</td>	Materialized</td></tr>
Explicit `MATERIALIZED</code></td>`	-</td>	Materialized</td>	Materialized</td></tr>
Explicit `NOT MATERIALIZED</code></td>`	-</td>	Inlined</td>	Inlined</td></tr> </tbody></table> The statistics black hole</a> </h2> As mentioned in PostgreSQL Statistics: Why queries run slow</a>, materialized CTEs are one of the places "where no statistics go." This is arguably the biggest practical problem with CTE materialization.</p> When the planner materializes a CTE, the result set is stored in a temporary tuplestore. This tuplestore has no `pg_statistic</code> entries - no histograms, no MCVs, no correlation data. The planner has to estimate row counts and value distributions using hardcoded defaults.</p>` `Let's see this in action. Here's a CTE over 10,000 rows:</p>` `EXPLAIN </span>WITH</span> all_orders </span>AS</span> MATERIALIZED (</span></span> SELECT * FROM</span> orders</span></span> )</span></span> SELECT * FROM</span> all_orders </span>WHERE status =</span> 'pending'</span> AND</span> amount </span>></span> 400</span>;</span></span></code></pre>` QUERY PLAN</span></span> -----------------------------------------------------------------------</span></span> CTE Scan on all_orders (cost=1855.00..4355.00 rows=5290 width=92)</span></span> Filter: ((amount > '400'::numeric) AND (status = 'pending'::text))</span></span> CTE all_orders</span></span> -> Seq Scan on orders (cost=0.00..1855.00 rows=100000 width=59)</span></span> (4 rows)</span></span></code></pre> The planner estimated 5,290 rows. Where does that number come from? The planner has no MCV list for status</code> inside the CTE, no histogram for amount</code>. It falls back to default selectivities of 0.3333</code> for the range comparison on amount</code> and a rough guess for the equality on status</code>, and multiplies them against the 100,000 input rows.</p> `If this CTE were inlined, the planner would read the actual statistics from pg_statistic</code> for the orders</code> table and produce estimates based on real data distribution, not defaults.</div> In a simple query this might not matter much. But when a materialized CTE feeds into a join, default-based estimates can cascade. The planner might choose a nested loop where a hash join would be better, or vice versa. It might underestimate memory needs and spill to disk unexpectedly.</p>`PG 17: Statistics propagation</a> </h3> PostgreSQL 17 brought two significant improvements to materialized CTEs:</p> Column statistics propagation.</strong> When the planner creates a CTE Scan</code> node, it now propagates column statistics from the underlying query into the scan node. This means n_distinct</code>, MCV lists, and histograms from the source table can inform estimates on the CTE scan.</p> Path key propagation.</strong> Materialized CTEs now preserve sort order information. If the CTE's subquery produces sorted output, the planner knows about it and can skip redundant sorts downstream.</p> These improvements significantly reduce the estimation gap, but they don't eliminate it. Inlined CTEs are still strictly better for planning accuracy, because the planner works directly with the base table statistics rather than propagated copies. If your CTE doesn't need to be materialized, don't force it.</p> When materialization helps</a> </h2> Materialization isn't always bad.</p> Multiple references.</strong> If the CTE result is used in multiple places, materialization computes it once. Without it, the subquery runs once per reference.</p> EXPLAIN </span>WITH</span> monthly_totals </span>AS</span> (</span></span> SELECT</span> date_trunc(</span>'month'</span>, created_at) </span>AS month</span>,</span></span> status</span>,</span></span> sum</span>(amount) </span>AS</span> total</span></span> FROM</span> orders</span></span> GROUP BY</span> 1</span>, </span>2</span></span> )</span></span> SELECT</span> cur</span>.</span>month</span>, </span>cur</span>.</span>status</span>, </span>cur</span>.</span>total</span>,</span></span> prev</span>.</span>total</span> AS</span> prev_month_total,</span></span> cur</span>.</span>total</span> -</span> prev</span>.</span>total</span> AS</span> delta</span></span> FROM</span> monthly_totals cur</span></span> LEFT JOIN</span> monthly_totals prev</span></span> ON</span> cur</span>.</span>month</span> =</span> prev</span>.</span>month</span> +</span> interval </span>'1 month'</span></span> AND</span> cur</span>.</span>status</span> =</span> prev</span>.</span>status</span>; </span></span></code></pre> QUERY PLAN</span></span> -------------------------------------------------------------------------------</span></span> Merge Left Join (cost=3887.46..3990.90 rows=4384 width=136)</span></span> Merge Cond: ((cur.month = ((prev.month + '1 mon'::interval))) AND (cur.status = prev.status))</span></span> CTE monthly_totals</span></span> -> HashAggregate (cost=3105.00..3181.72 rows=4384 width=49)</span></span> Group Key: date_trunc('month'::text, (orders.created_at)::timestamp with time zone), orders.status</span></span> -> Seq Scan on orders (cost=0.00..2355.00 rows=100000 width=23)</span></span> -> Sort (cost=352.87..363.83 rows=4384 width=72)</span></span> Sort Key: cur.month, cur.status</span></span> -> CTE Scan on monthly_totals cur (cost=0.00..87.68 rows=4384 width=72)</span></span> -> Sort (cost=352.87..363.83 rows=4384 width=72)</span></span> Sort Key: ((prev.month + '1 mon'::interval)), prev.status</span></span> -> CTE Scan on monthly_totals prev (cost=0.00..87.68 rows=4384 width=72)</span></span> (12 rows)</span></span></code></pre> The aggregation runs once. Both cur</code> and prev</code> read from the materialized result. Without materialization, the entire aggregation would run twice.</p> Expensive VOLATILE expressions.</strong> If a CTE contains calls to volatile functions or expensive computations, materialization ensures they execute exactly once.</p> Data-modifying operations.</strong> The whole point of writable CTEs is that the side effects happen once and the RETURNING</code> data is available downstream. Materialization is not optional here.</p> When inlining isn't enough</a> </h2> The imperative mindset from the introduction, "first do this, then do that", doesn't go away just because the planner inlines your CTEs. And this one is my personal favourite, and source that never stop delivering query refactoring.</p> Developers still structure queries as sequential pipelines, and that structure itself can create performance problems that have nothing to do with materialization.</p> A common pattern is building queries as an assembly line: one CTE filters rows, the next LEFT JOINs related tables and aggregates metadata with GROUP BY</code>, the next filters on the aggregated results. It reads like a clean pipeline, but the GROUP BY</code> in the middle creates a wall the planner can't optimize past.</p> WITH</span> recent_orders </span>AS</span> (</span></span> SELECT * FROM</span> orders </span>WHERE</span> created_at </span>></span> '2024-01-01'</span></span> ),</span></span> order_metadata </span>AS</span> (</span></span> SELECT</span></span> o</span>.</span>id</span>,</span></span> bool_or(</span>oa</span>.</span>id</span> IS NOT NULL</span>) </span>AS</span> was_archived,</span></span> count</span>(</span>o2</span>.</span>id</span>) </span>AS</span> related_count</span></span> FROM</span> recent_orders o</span></span> LEFT JOIN</span> orders_archive oa </span>ON</span> o</span>.</span>id</span> =</span> oa</span>.</span>id</span></span> LEFT JOIN</span> orders o2 </span>ON</span> o</span>.</span>customer_id</span> =</span> o2</span>.</span>customer_id</span> AND</span> o2</span>.</span>id</span> !=</span> o</span>.</span>id</span></span> GROUP BY</span> o</span>.</span>id</span></span> )</span></span> SELECT</span> o.</span></span>, </span>m</span>.</span>was_archived</span>, </span>m</span>.</span>related_count</span></span> FROM</span> recent_orders o</span></span> JOIN</span> order_metadata m </span>ON</span> o</span>.</span>id</span> =</span> m</span>.</span>id</span></span> WHERE</span> m</span>.</span>was_archived</span> =</span> false</span></span> AND</span> m</span>.</span>related_count</span> ></span> 0</span>;</span></span></code></pre> Each CTE here is referenced once, so they all get inlined. No materialization, no optimization fence. The planner sees the full query. So what's the problem?</p> The GROUP BY</code> in order_metadata</code>. Even after inlining, the planner cannot push the was_archived = false</code> predicate past the aggregation. It must first LEFT JOIN every filtered order against orders_archive</code> and self-join orders</code>, compute the aggregates for all of them, and only then discard the rows that don't match. If recent_orders</code> returns 50,000 rows but only 200 were ever archived, you're joining and aggregating 49,800 rows for nothing.</p> The fix is to replace the aggregation-then-filter pattern with correlated EXISTS</code> subqueries:</p> SELECT</span> o.</span></span></span> FROM</span> orders o</span></span> WHERE</span> o</span>.</span>created_at</span> ></span> '2024-01-01'</span></span> AND NOT EXISTS</span> (</span></span> SELECT</span> 1</span> FROM</span> orders_archive oa </span>WHERE</span> oa</span>.</span>id</span> =</span> o</span>.</span>id</span></span> )</span></span> AND EXISTS</span> (</span></span> SELECT</span> 1</span> FROM</span> orders o2</span></span> WHERE</span> o2</span>.</span>customer_id</span> =</span> o</span>.</span>customer_id</span> AND</span> o2</span>.</span>id</span> !=</span> o</span>.</span>id</span></span> );</span></span></code></pre> EXISTS</code> short-circuits after finding the first matching row. The planner can push created_at > '2024-01-01'</code> all the way down to an index scan on orders</code>, then probe each related table per result. No aggregation, no wasted work.</p> The rule of thumb:</strong> if your CTE contains a GROUP BY</code> or a LEFT JOIN</code> just to compute a boolean ("does this row have related data?"), you've built a wall the planner can't see past. A correlated EXISTS</code> lets the planner push filters down and stop scanning early. This applies whether the CTE is materialized or inlined.</p> </div> Writable CTEs (the power and the traps)</a> </h2> Data-modifying CTEs let you INSERT</code>, UPDATE</code>, or DELETE</code> inside a WITH</code> clause and use the RETURNING</code> data in subsequent CTEs or the main query.</p> EXPLAIN </span>WITH</span> deleted </span>AS</span> (</span></span> DELETE FROM</span> orders</span></span> WHERE status =</span> 'cancelled'</span></span> AND</span> created_at </span><</span> '2023-01-01'</span></span> RETURNING </span></span></span> ),</span></span> archived </span>AS</span> (</span></span> INSERT INTO</span> orders_archive</span></span> SELECT FROM</span> deleted</span></span> RETURNING id</span></span> )</span></span> SELECT</span> count</span>(</span></span>) </span>FROM</span> archived;</span></span></code></pre> QUERY PLAN</span></span> ----------------------------------------------------------------------------------------------</span></span> Aggregate (cost=2709.19..2709.20 rows=1 width=8)</span></span> CTE deleted</span></span> -> Delete on orders (cost=0.00..2355.00 rows=8334 width=6)</span></span> -> Seq Scan on orders (cost=0.00..2355.00 rows=8334 width=6)</span></span> Filter: ((created_at < '2023-01-01'::date) AND (status = 'cancelled'::text))</span></span> CTE archived</span></span> -> Insert on orders_archive (cost=0.00..166.68 rows=8334 width=92)</span></span> -> CTE Scan on deleted (cost=0.00..166.68 rows=8334 width=92)</span></span> -> CTE Scan on archived (cost=0.00..166.68 rows=8334 width=0)</span></span> (9 rows) </span></span></code></pre> This deletes old cancelled orders, moves them to an archive table, and counts how many were archived - all in a single atomic statement, no application-level coordination needed.</p> But there are sharp edges.</p> You cannot read what you just wrote</a> </h3> All sub-statements in a data-modifying CTE see the same snapshot. This means the effects of one CTE are not visible</strong> to other CTEs or the main query when they read the target table</em>. Only the RETURNING</code> clause communicates data between CTE steps.</p> SELECT</span> count</span>(</span>1</span>) </span>FROM</span> orders </span>WHERE</span> customer_id </span>=</span> 1</span>;</span></span></code></pre> count</span></span> -------</span></span> 31</span></span> (1 row)</span></span></code></pre>WITH</span> ins </span>AS</span> (</span></span> INSERT INTO</span> orders (customer_id, amount, </span>status</span>, created_at)</span></span> VALUES</span> (</span>1</span>, </span>100</span>.</span>00</span>, </span>'pending'</span>, CURRENT_DATE)</span></span> RETURNING id</span></span> )</span></span> -- this does NOT see the row we just inserted</span></span> SELECT</span> count</span>(</span>1</span>) </span>FROM</span> orders </span>WHERE</span> customer_id </span>=</span> 1</span>;</span></span></code></pre> count</span></span> -------</span></span> 31</span></span> (1 row)</span></span></code></pre> The count(1)</code> query sees the pre-insert snapshot. If you need the inserted data, you must use the RETURNING</code> clause from the ins</code> CTE, not re-read the table.</p> Tuple shuffling</a> </h3> A common pattern is using writable CTEs to move rows between tables atomically:</p> WITH</span> moved </span>AS</span> (</span></span> DELETE FROM</span> orders_staging</span></span> RETURNING </span></span></span> )</span></span> INSERT INTO</span> orders</span></span> SELECT * FROM</span> moved;</span></span></code></pre> This deletes all rows from the staging table and inserts them into the production table in a single atomic operation. No window where data exists in both or neither table.</p> Data-modifying CTEs disable parallel query for the entire statement. If you have a complex query that mixes reads and writes, the write CTE prevents parallelism even for the read-only parts.</div> Recursive CTEs are always materialized</a> </h2> Recursive CTEs use an iterative working-table mechanism. Despite the name, they aren't truly recursive. PostgreSQL doesn't "call itself" by creating a nested stack of unfinished queries. Instead, it operates in loops.</p> Execute the non-recursive term (the "seed"). Put results in the working table.</li> Execute the recursive term using the working table as input. New rows become the next</em> working table.</li> Repeat until the recursive term returns no new rows.</li> Return the union of all iterations.</li> </ol> WITH RECURSIVE</span> org_chart </span>AS</span> (</span></span> -- Seed: start from the CEO</span></span> SELECT</span> id, </span>name</span>, manager_id, </span>1</span> AS</span> depth</span></span> FROM</span> employees</span></span> WHERE</span> manager_id </span>IS NULL</span></span> </span> UNION ALL</span></span> </span> -- Recursive term: find direct reports</span></span> SELECT</span> e</span>.</span>id</span>, </span>e</span>.</span>name</span>, </span>e</span>.</span>manager_id</span>, </span>oc</span>.</span>depth</span> +</span> 1</span></span> FROM</span> employees e</span></span> JOIN</span> org_chart oc </span>ON</span> e</span>.</span>manager_id</span> =</span> oc</span>.</span>id</span></span> )</span></span> SELECT * FROM</span> org_chart </span>ORDER BY</span> depth, </span>name</span>;</span></span></code></pre> id \| name \| manager_id \| depth</span></span> ----+---------+------------+-------</span></span> 1 \| Alice \| \| 1</span></span> 2 \| Bob \| 1 \| 2</span></span> 3 \| Charlie \| 1 \| 2</span></span> 4 \| Diana \| 2 \| 3</span></span> 5 \| Eve \| 2 \| 3</span></span> 6 \| Frank \| 3 \| 3</span></span> 7 \| Grace \| 3 \| 3</span></span> 8 \| Hank \| 6 \| 4</span></span> 9 \| Ivy \| 6 \| 4</span></span> (9 rows)</span></span></code></pre>UNION vs UNION ALL</a> </h3> The choice between UNION</code> and UNION ALL</code> in recursive CTEs matters more than in regular queries.</p> UNION ALL</code> keeps all rows, including duplicates. This is faster but dangerous: if your graph has cycles, the recursion never terminates. PostgreSQL will run until you cancel the query or memory is exhausted.</p> UNION</code> deduplicates at each iteration. This prevents infinite loops in cyclic graphs but adds the cost of hashing and comparing rows at every step.</p> PostgreSQL 14 added SQL-standard SEARCH</code> and CYCLE</code> clauses that replace the manual patterns for controlling traversal order (breadth-first vs. depth-first) and detecting cycles. SEARCH BREADTH FIRST BY</code> / SEARCH DEPTH FIRST BY</code> control the order, while CYCLE</code> automatically detects and marks cycles, which is far cleaner than the old pattern of accumulating an array of visited IDs.</p> For pure hierarchical data (trees without cycles), consider the ltree</code> extension as an alternative to recursive CTEs. It stores the full path as a label tree and supports efficient ancestor/descendant queries with GiST indexes. The trade-off is denormalized storage vs. on-the-fly recursion.</p> </div> The exotic edge cases</a> </h2> Partition pruning lost</a> </h3> When you materialize a CTE over a partitioned table, partition pruning cannot happen on the CTE scan side. The materialized result is a flat tuplestore disconnected from the partition metadata.</p> -- assume orders is range-partitioned by created_at</span></span> WITH</span> recent </span>AS</span> MATERIALIZED (</span></span> SELECT * FROM</span> orders</span></span> )</span></span> SELECT * FROM</span> recent </span>WHERE</span> created_at </span>></span> '2025-06-01'</span>;</span></span></code></pre> The created_at > '2025-06-01'</code> predicate is applied after</em> materialization. All partitions are scanned to build the CTE, even though only one or two would be needed. Use NOT MATERIALIZED</code> (or simply let the planner inline it) to preserve partition pruning.</p> Prepared statements and plan caching</a> </h3> PostgreSQL generates custom plans for the first 5 executions of a prepared statement. After that, it may switch to a generic plan. CTE inlining decisions can differ between custom and generic plans, because generic plans don't know the actual parameter values.</p> This means a CTE that gets inlined during your first 5 calls might start materializing on the 6th or vice versa. If you see sudden plan changes with prepared statements, check whether the CTE inlining behaviour has shifted.</p> work_mem spilling</a> </h3> Materialized CTEs store their results in memory, bounded by work_mem</code>. When the result set exceeds this limit, it silently spills to disk as a temporary file. This is not an error - it just gets slower.</p> Monitor with log_temp_files = 0</code> (logs all temp files) or check EXPLAIN (ANALYZE, BUFFERS)</code> for temp read/write counts.</p> PostgreSQL 18: EXPLAIN now shows memory/disk usage</strong> Starting with PostgreSQL 18, EXPLAIN ANALYZE</code> reports memory and disk usage for Material nodes, including CTE materialization. You can see exactly how much memory a materialized CTE consumed and whether it spilled to disk.</p> </div> CTE and security barrier views</a> </h3> A `security_barrier` view is a "black box" that forces the database to fully resolve the view's internal logic before any outer filters are applied.</div> Security-barrier views already prevent subquery flattening as a security measure (to stop user-defined functions from seeing rows they shouldn't). When you combine a security-barrier view with a CTE, you compound the optimisation barriers. The planner can neither inline the view nor the CTE. If performance matters in this scenario, consider materializing the security-sensitive filtering into a temporary table first. CTE vs. subquery vs. temporary table</a> </h2> CTE, referenced once</strong> the planner inlines it on PG 12+, so it doesn't affect the execution plan. This is the default choice for breaking up complex queries.</p> CTE, referenced multiple times (small result)</strong> represents acceptable cost. Materialization means the subquery runs once. For aggregations or filtered subsets that produce a few hundred rows, the overhead is minimal.</p> As Henrietta Dombrovskaya</a> emphasizes, "The best temporary table is the one you didn't create". Always exhaust your indexing and query-rewriting options before reaching for a temp table, as the DDL overhead often outweighs the execution gains.</div> CTE, referenced multiple times (large result) ss case when you might consider a temporary table instead. A materialized CTE has no indexes and no statistics. A temporary table can have both, and as covered in [Introduction to Buffers](/posts/introduction-to-buffers/), temporary tables use local buffers with simpler locking and no WAL overhead. If you're joining against 100k+ rows from a CTE, create a temp table, add an index, and `ANALYZE` it. Data-modifying operations</strong> with writable CTE. No alternative gives you the atomic, single-statement behavior.</p> Recursive CTEs</strong>. There's no alternative in pure SQL.</p> Large intermediate results needing indexes/statistics</strong>. If you really need them use temporary table. As discussed in Reading Buffer statistics</a>, temporary tables offer full planner support - indexes, statistics, and buffer management - that materialized CTEs simply don't have.</p> Scenario</th> Recommendation</th></tr></thead> Readability, single reference</td> CTE (inlined, free)</td></tr> Compute once, use many times (small)</td> CTE (materialized)</td></tr> Compute once, use many times (large)</td> Temporary table</td></tr> Atomic data modification</td> Writable CTE</td></tr> Hierarchy / graph traversal</td> Recursive CTE</td></tr> Need indexes on intermediate data</td> Temporary table</td></tr> </tbody></table> The PG 18 state of affairs</a> </h2> PostgreSQL 18 continues to refine CTE handling without any revolutionary changes:</p> EXPLAIN shows memory/disk usage</strong> for CTE materialization nodes. You can finally see whether your CTE fit in work_mem</code> or spilled to disk.</li> Better query plans for CTEs on the same table.</strong> The planner is smarter about eliminating redundant scans when multiple CTEs reference the same underlying table.</li> Data-modifying CTE fix for updatable views with rules.</strong> An edge case where writable CTEs interacted incorrectly with views defined using rules has been resolved.</li> CTE inlining is mature.</strong> The core inlining logic hasn't changed since PG 12. What has improved is everything around it - better statistics propagation (PG 17), better materialisation diagnostics (PG 18), better cost estimation.</li> </ul> CTEs are a good tool. Just know when you're holding the sharp end.</p> pg_regresql: truly portable PostgreSQL statistics 2026-03-21T14:32:00+00:00 The previous article</a> showed that PostgreSQL 18 makes optimizer statistics portable, but left one gap open:</p> It's not worth trying to inject relpages</code> as the planner checks the actual file size and scales it proportionally.</p> </blockquote> The planner doesn't trust pg_class.relpages</code>. It calls smgrnblocks()</code> to read the actual number of 8KB pages from disk. Your table is 74 pages on disk but pg_class.relpages</code> says 123,513? The planner uses the ratio to scale reltuples</code> down to match the actual file size. The selectivity ratios stay correct, plan shapes mostly survive, but the absolute cost estimates are off.</p> For debugging a single query, that's usually fine. For automated regression testing where you compare EXPLAIN</code> costs across runs, it breaks things. A cost threshold of 2× means something different when the baseline was computed from fake-scaled numbers.</p> As part of my work on RegreSQL</a> I'm happy to announce pg_regresql</a> extension which fixes this by hooking directly into the planner.</p> Why the planner ignores relpages</a> </h2> When PostgreSQL's planner calls get_relation_info()</code> in plancat.c</code>, it delegates to estimate_rel_size()</code> which ends up in table_block_relation_estimate_size()</code> in tableam.c</code>. There, the actual page count comes from the storage manager:</p> curpages </span>=</span> RelationGetNumberOfBlocks</span>(rel);</span></span></code></pre> The function then computes a tuple density from pg_class</code> (reltuples / relpages</code>) and multiplies it by curpages</code> to estimate tuples. So pg_class.reltuples</code> isn't ignored, it's scaled to match the real file size. The reasoning is sound for normal operation: the catalog might be stale, but the file system is always current.</p> The same applies to indexes. The planner reads their actual sizes from disk too.</p> What pg_regresql does</a> </h2> The extension hooks into get_relation_info_hook</code>, a planner callback that runs after PostgreSQL reads the physical file stats. The hook replaces the file-based numbers with the values stored in pg_class</code>:</p> rel->pages</code> ← pg_class.relpages</code></li> rel->tuples</code> ← pg_class.reltuples</code></li> rel->allvisfrac</code> ← pg_class.relallvisible / pg_class.relpages</code></li> </ul> It does the same for every index in rel->indexlist</code>. Pages and tuples for each index are overridden from the index's own pg_class</code> entry.</p> The guard conditions are simple: skip the override if relpages == 0</code> (empty or never analyzed) or reltuples == -1</code> (never analyzed). The hook only activates for tables that have been ANALYZE</code>d or had statistics injected.</p> Installation</a> </h2> Build from source using PGXS:</p> cd</span> pg_ext</span></span> make</span></span> make</span> install</span></span></code></pre> It requires no GUCs, background workers, or shared memory.</p> Usage</a> </h2> Load the extension in your session:</p> LOAD</span> 'pg_regresql'</span>;</span></span></code></pre> That's it. Every EXPLAIN</code> in this session will now use catalog statistics instead of file sizes. There are no functions to call, no tables to configure.</p> You can also load it per-database by adding it to session_preload_libraries</code> in postgresql.conf</code> or via ALTER DATABASE</code>:</p> ALTER DATABASE</span> test_db </span>SET</span> session_preload_libraries </span>=</span> 'pg_regresql'</span>;</span></span></code></pre>The difference it makes</a> </h2> Using the same test_orders</code> example from the previous article</a>: 10,000 actual rows, injected with production statistics claiming 50 million rows across 123,513 pages.</p> Without pg_regresql</code>:</p> EXPLAIN </span>SELECT * FROM</span> test_orders </span>WHERE</span> created_at </span>></span> '2024-06-01'</span>;</span></span></code></pre> QUERY PLAN</span></span> ----------------------------------------------------------------------------------------------------</span></span> Index Scan using test_orders_created_at_idx on test_orders (cost=0.29..153.21 rows=6340 width=26)</span></span> Index Cond: (created_at > '2024-06-01'::date)</span></span></code></pre> The plan shape is correct (index scan thanks to the histogram), but the row estimate is 6,340. For a 50-million-row table where the filter covers roughly 10% of the histogram range, the expected estimate should be in the millions. The planner saw 74 real pages on disk, scaled reltuples</code> down to ~30,000, then applied selectivity. The ratio is preserved but the absolute number is wrong.</p> With pg_regresql</code>:</p> LOAD</span> 'pg_regresql'</span>;</span></span> EXPLAIN </span>SELECT * FROM</span> test_orders </span>WHERE</span> created_at </span>></span> '2024-06-01'</span>;</span></span></code></pre> QUERY PLAN</span></span> -----------------------------------------------------------------------------------------------------------</span></span> Index Scan using test_orders_created_at_idx on test_orders (cost=0.29..153212.27 rows=10791836 width=27)</span></span> Index Cond: (created_at > '2024-06-01'::date)</span></span> (2 rows)</span></span></code></pre> Cost numbers now reflect the full 50 million rows.</p> Where this matters</a> </h2> Cost-based regression testing.</strong> If you're comparing EXPLAIN</code> costs between schema versions (which is what RegreSQL</a> does), you need the absolute numbers to be stable and realistic. With the scaling behavior, your baseline costs are proportional to your test database size, not production. A migration that doubles a cost in production might show a 1.3× increase in CI because the scaled-down numbers compress the range.</p> Reproducing production plans on a laptop.</strong> Sometimes the plan shape itself changes depending on the absolute numbers. A query with multiple joins might get a different join order when the planner sees 50 million rows vs. 30,000 rows, because the cost crossover between hash join and nested loop depends on the absolute row count, not just the ratio.</p> Index-only scans.</strong> The allvisfrac</code> (fraction of all-visible pages) matters for index-only scan costing. Without the hook, allvisfrac</code> is computed from the real relallvisible</code> catalog value divided by the real page count. With injected stats, relallvisible</code> might be 120,000 but the real page count is 74, so the fraction clamps to 1.0 and the planner overestimates how cheap index-only scans are. The hook fixes this by using the injected relpages</code> as the denominator.</p> What it doesn't do</a> </h2> Column-level statistics and ANALYZE</code> behavior are unchanged. The extension only affects how the planner reads table and index sizes. One thing worth noting: EXPLAIN ANALYZE</code> will still show actual row counts from the real (small) data. The extension changes the planner's cost estimates, not query execution.</p> The full workflow</a> </h2> Combining PostgreSQL 18's portable statistics with pg_regresql</code>, the full workflow looks like this:</p> # 1. dump schema and statistics from production</span></span> pg_dump</span> --schema-only -d</span> production_db</span> ></span> schema.sql</span></span> pg_dump</span> --statistics-only -d</span> production_db</span> ></span> stats.sql</span></span> </span> # 2. create test database</span></span> createdb</span> test_db</span></span> psql</span> -d</span> test_db</span> -f</span> schema.sql</span></span> </span> # 3. load minimal fixture data (optional)</span></span> psql</span> -d</span> test_db</span> -f</span> fixtures.sql</span></span> </span> # 4. inject production statistics</span></span> psql</span> -d</span> test_db</span> -f</span> stats.sql</span></span> </span> # 5. install pg_regresql and prevent stats from being overwritten</span></span> psql</span> -d</span> test_db</span> <<</span>SQL</span></span> ALTER DATABASE test_db SET session_preload_libraries = 'pg_regresql';</span></span> ALTER TABLE orders SET (autovacuum_enabled = false);</span></span> -- repeat for other tables</span></span> SQL</span></span> </span> # 6. reconnect and verify (plans now match production)</span></span> psql</span> -d</span> test_db</span> -c</span> "EXPLAIN SELECT * FROM orders WHERE status = 'pending'"</span></span></code></pre>Compatibility</a> </h2> The extension works with PostgreSQL 13 through 18. The portable statistics functions (pg_restore_relation_stats</code>, pg_restore_attribute_stats</code>) require PostgreSQL 18, but pg_regresql</code> works with any method of writing to pg_class</code>, including direct catalog updates on older versions.</p> PostgreSQL 19 will need a small update: the get_relation_info_hook</code> used by pg_regresql has been replaced</a> with build_simple_rel_hook</code>. The new hook runs slightly later with different arguments, but the override logic stays the same.</p> Don't use this in production.</strong> The extension makes the planner ignore reality. That's exactly what you want for testing with injected statistics. In production, you want the planner to see the actual file sizes so it can adapt to data growth, bloat, and vacuum activity. Keep pg_regresql</code> in your dev/test and CI databases. </div> Production query plans without production data 2026-03-08T22:05:00+00:00 In the previous article</a> we covered how the PostgreSQL planner reads pg_class</code> and pg_statistic</code> to estimate row counts, choose join strategies, and decide whether an index scan is worth it. The message was clear: when statistics are wrong, everything else goes with it.</p> Streaming replication provides bit-to-bit replication, so all replicas share the same statistics with primary server.</div> But there was one thing we didn't talk about. Statistics are specific to the database cluster that generated them. The primary way to populate them is `ANALYZE` which requires the actual data. PostgreSQL 18 changed that. Two new functions: pg_restore_relation_stats</code> and pg_restore_attribute_stats</code> write numbers directly into the catalog tables. Combined with pg_dump --statistics-only</code>, you can treat optimizer statistics as a deployable artifact. Compact, portable, plain SQL.</p> The feature was driven by the upgrade use case</a>. In the past, major version upgrades used to leave pg_statistic</code> empty, forcing you to run ANALYZE</code>. Which might take hours on large clusters. With PostgreSQL 18 upgrades now transfer statistics automatically. But that's just the beginning. The same logic lets you export statistics from production and inject them anywhere - test database, local debugging, or as part of CI pipelines.</p> The problem</a> </h2> Your CI database has 1,000 rows. Production has 50 million. The planner makes completely different decisions for each. Running EXPLAIN</code> in CI tells you nothing about the production plan. This is the core premise behind RegreSQL</a>. Catching query plan regressions in CI is far more reliable when the planner sees production-scale statistics.</p> Same applies to debugging</strong>. A query is slow in production and you want to reproduce the plan locally, but your database has different statistics, and planner chooses the predictable path. Porting production stats can provide you that snapshot of thinking planner has to do in production, without actually going to production.</p> pg_restore_relation_stats</a> </h2> The first of function behind portable PostgreSQL statistics is pg_restore_relation_stats</code>. It writes table-level data directly into pg_class</code> in form of variadic name/value pairs.</p> SELECT</span> pg_restore_relation_stats(</span></span> 'schemaname'</span>, </span>'public'</span>,</span></span> 'relname'</span>, </span>'orders'</span>,</span></span> 'relpages'</span>, </span>123513</span>::</span>integer</span>,</span></span> 'reltuples'</span>, </span>50000000</span>::</span>real</span>,</span></span> 'relallvisible'</span>, </span>123513</span>::</span>integer</span>,</span></span> 'relallfrozen'</span>, </span>120000</span>::</span>integer</span></span> );</span></span></code></pre> But that's just an example. Let's modify some real statistics to see the full value. We will create a small table, inject fake production-like statistics and watch the planner to change its mind.</p> CREATE TABLE</span> test_orders</span> (</span></span> id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> customer_id </span>integer NOT NULL</span>,</span></span> amount </span>numeric</span>(</span>10</span>,</span>2</span>)</span> NOT NULL</span>,</span></span> status text NOT NULL DEFAULT</span> 'pending'</span>,</span></span> created_at </span>date NOT NULL DEFAULT</span> CURRENT_DATE</span></span> );</span></span> </span> INSERT INTO</span> test_orders (customer_id, amount, </span>status</span>, created_at)</span></span> SELECT</span></span> (random()</span> </span> 9999</span> +</span> 1</span>)::</span>int</span>,</span></span> (random()</span> </span> 5000</span> +</span> 5</span>)::</span>numeric</span>(</span>10</span>,</span>2</span>),</span></span> (</span>ARRAY</span>['pending','shipped','delivered','cancelled'])[floor(random()4+1)::int],</span></span> '2024-01-01'</span>::</span>date +</span> (random()</span> </span> 365</span>)::</span>int</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>10000</span>);</span></span> </span> CREATE INDEX</span> ON</span> test_orders (created_at);</span></span> CREATE INDEX</span> ON</span> test_orders (</span>status</span>);</span></span> ANALYZE test_orders;</span></span></code></pre> When you check the current statistics, it has predictable data.</p> SELECT</span> relname, relpages, reltuples</span></span> FROM</span> pg_class </span>WHERE</span> relname </span>=</span> 'test_orders'</span>;</span></span></code></pre> relname \| relpages \| reltuples</span></span> -------------+----------+-----------</span></span> test_orders \| 74 \| 10000</span></span> (1 row)</span></span></code></pre> With 10,000 rows across 74 pages, the planner picks a sequential scan.</p> EXPLAIN </span>SELECT * FROM</span> test_orders </span>WHERE</span> created_at </span>></span> '2024-06-01'</span>;</span></span></code></pre> QUERY PLAN</span></span> -----------------------------------------------------------------</span></span> Seq Scan on test_orders (cost=0.00..199.00 rows=5891 width=26)</span></span> Filter: (created_at > '2024-06-01'::date)</span></span> (2 rows)</span></span></code></pre> Now inject production-scale table stats:</p> SELECT</span> pg_restore_relation_stats(</span></span> 'schemaname'</span>, </span>'public'</span>,</span></span> 'relname'</span>, </span>'test_orders'</span>,</span></span> 'relpages'</span>, </span>123513</span>::</span>integer</span>,</span></span> 'reltuples'</span>, </span>50000000</span>::</span>real</span>,</span></span> 'relallvisible'</span>, </span>123513</span>::</span>integer</span></span> );</span></span></code></pre> And you might be surprised by the result.</p> EXPLAIN </span>SELECT * FROM</span> test_orders </span>WHERE</span> created_at </span>></span> '2024-06-01'</span>;</span></span></code></pre> QUERY PLAN</span></span> ------------------------------------------------------------------</span></span> Seq Scan on test_orders (cost=0.00..448.45 rows=17649 width=26)</span></span> Filter: (created_at > '2024-06-01'::date)</span></span></code></pre> The planner is still using the sequential plan. Only the estimated number of rows has changed. Why? If you remember from previous article, it's where column level statistics come into play. Histogram bounds still match the original 10,000 rows we inserted.</p> pg_restore_attribute_stats</a> </h2> This function writes column-level statistics into pg_statistic</code> the same catalog that ANALYZE populates</a> with MCVs, histograms, and correlation</a>.</p> In previous section, we left the planner stuck on a sequential scan despite believing the table has 50 million rows. The missing piece is column-level statistics. Let's pick up where we left off and inject histogram bounds for created_at</code>.</p> SELECT</span> pg_restore_attribute_stats(</span></span> 'schemaname'</span>, </span>'public'</span>,</span></span> 'relname'</span>, </span>'test_orders'</span>,</span></span> 'attname'</span>, </span>'created_at'</span>,</span></span> 'inherited'</span>, false::</span>boolean</span>,</span></span> 'null_frac'</span>, </span>0</span>.</span>0</span>::</span>real</span>,</span></span> 'avg_width'</span>, </span>4</span>::</span>integer</span>,</span></span> 'n_distinct'</span>, </span>-</span>0</span>.</span>05</span>::</span>real</span>,</span></span> 'histogram_bounds'</span>, </span>'{2019-01-01,2019-07-01,2020-01-01,2020-07-01,2021-01-01,2021-07-01,2022-01-01,2022-07-01,2023-01-01,2023-07-01,2024-01-01}'</span>::</span>text</span>,</span></span> 'correlation'</span>, </span>0</span>.</span>98</span>::</span>real</span></span> );</span></span></code></pre> Now the planner knows the data spans 5 years. A query filtering on the last 6 months of 2024 covers a narrow slice.</p> EXPLAIN SELECT * FROM test_orders WHERE created_at > '2024-06-01';</span></span></code></pre> QUERY PLAN</span></span> ----------------------------------------------------------------------------------------------------</span></span> Index Scan using test_orders_created_at_idx on test_orders (cost=0.29..153.21 rows=6340 width=26)</span></span> Index Cond: (created_at > '2024-06-01'::date)</span></span></code></pre> Histogram bounds divide the non-MCV portion of the data into equal-population buckets. If most_common_vals</code> accounts for most of the data, the histogram covers only the remaining tail. The number of buckets is controlled by default_statistics_target</code> (default 100, meaning 101 bounds). </div> And that's a plan flip! The histogram tells the planner the data spans 2019–2024, so > '2024-06-01'</code> matches a narrow tail. A small fraction of 50 million rows. The index scan that was ignored before is now the obvious choice. Table-level stats set the scale, column-level stats shaped the selectivity, and together they changed the plan.</p> The correlation</code> statistic tells the planner how closely the physical row order matches the column's sort order. A value near 1.0 means sequential access patterns - making index scan cheaper</a> because the next row is likely on the same or adjacent page. For time-series data like created_at</code> where rows are inserted chronologically, correlation is typically very high. </div> Injecting a skewed distribution</a> </h2> The same function handles MCV lists</a>. In production, your status</code> column isn't uniform, 95% of orders are delivered, 1.5% are pending.</p> SELECT</span> pg_restore_attribute_stats(</span></span> 'schemaname'</span>, </span>'public'</span>,</span></span> 'relname'</span>, </span>'test_orders'</span>,</span></span> 'attname'</span>, </span>'status'</span>,</span></span> 'inherited'</span>, false::</span>boolean</span>,</span></span> 'null_frac'</span>, </span>0</span>.</span>0</span>::</span>real</span>,</span></span> 'avg_width'</span>, </span>9</span>::</span>integer</span>,</span></span> 'n_distinct'</span>, </span>5</span>::</span>real</span>,</span></span> 'most_common_vals'</span>, </span>'{delivered,shipped,cancelled,pending,returned}'</span>::</span>text</span>,</span></span> 'most_common_freqs'</span>, </span>'{0.95,0.015,0.015,0.015,0.005}'</span>::</span>real</span>[]</span></span> );</span></span></code></pre> You can see</p> EXPLAIN </span>SELECT * FROM</span> test_orders </span>WHERE status =</span> 'pending'</span>;</span></span></code></pre> QUERY PLAN</span></span> ---------------------------------------------------------------------------------------</span></span> Bitmap Heap Scan on test_orders (cost=8.93..90.42 rows=599 width=27)</span></span> Recheck Cond: (status = 'pending'::text)</span></span> -> Bitmap Index Scan on test_orders_status_idx (cost=0.00..8.78 rows=599 width=0)</span></span> Index Cond: (status = 'pending'::text)</span></span> (4 rows)</span></span></code></pre> and compare it with</p> EXPLAIN </span>SELECT * FROM</span> test_orders </span>WHERE status =</span> 'delivered'</span>;</span></span></code></pre> QUERY PLAN</span></span> ------------------------------------------------------------------</span></span> Seq Scan on test_orders (cost=0.00..448.45 rows=28458 width=27)</span></span> Filter: (status = 'delivered'::text)</span></span> (2 rows)</span></span></code></pre> Same column, same operator, different plans. The planner uses a bitmap index scan for pending</code> (1.5% rare enough to justify the index) and a sequential scan for delivered</code> (95% being most of the table). The selectivity ratios from the MCV list drive the plan choice.</p> You might have noticed the row estimates (599 and 28,458) are lower than you'd expect for a 50-million-row table. The planner checks the actual physical file size. Our table is only 74 pages on disk, not the 123,513 we injected. Hence the planner scales reltuples</code> and relpages</code> down proportionally. The absolute numbers shrink, but the ratios</i> between them stay correct, and it's the ratios that determine plan shape. When you use pg_dump --statistics-only</code> in practice, you're typically restoring into a database with comparable data volume, so the estimates align naturally. </div> </div> </div> </div> </div> pg_regresql extension</strong> The pg_regresql</code> extension fixes this scaling problem. It hooks into the planner to trust the injected relpages</code> value instead of reading the physical file size, so cost estimates match production even when your test database is tiny.</p> Read more</a> </div> </div> pg_dump</a> </h2> The functions we covered are the mechanics. For operational use pg_dump</code> provides everything you need. PostgreSQL 18 added three flags.</p> Flag</th> Effect</th></tr></thead> --statistics</code></td> dump the statistics (you have to request it explicitely)</td></tr> --statistics-only</code></td> dump only the statistics, not schema or data</td></tr> --no-statistics</code></td> do not dump statistics</td></tr> </tbody></table> When you export the statistics for your production database</p> pg_dump --statistics-only -d production_db > stats.sql</span></span> </span></code></pre> you will see the output is series of SELECT pg_restore_relation_stats(...)</code> and SELECT pg_restore_attribute_stats(...)</code> calls. Exactly as we explained above.</p> The full workflow to turn your production data into testable plans might look like this:</p> # 1. dump schema from production</span></span> pg_dump</span> --schema-only -d</span> production_db</span> ></span> schema.sql</span></span> </span> # 2. dump statistics from production</span></span> pg_dump</span> --statistics-only -d</span> production_db</span> ></span> stats.sql</span></span> </span> # 3. create test database with schema</span></span> createdb</span> test_db</span></span> psql</span> -d</span> test_db</span> -f</span> schema.sql</span></span> </span> # 4. load fixture data (optional; masked, minimal)</span></span> psql</span> -d</span> test_db</span> -f</span> fixtures.sql</span></span> </span> # 5. inject production statistics</span></span> psql</span> -d</span> test_db</span> -f</span> stats.sql</span></span> </span> # 6. query plans now match production</span></span> psql</span> -d</span> test_db</span> -c</span> "EXPLAIN SELECT * FROM test_orders WHERE status = 'pending'"</span></span></code></pre> Statistics dumps are tiny. A database with hundreds of tables and thousands of columns produces a statistics dump under 1MB. The production data might be hundreds of GB. The statistics that describe it fit in a text file. </div> Keeping injected statistics alive</a> </h2> Now you might ask yourself, where's the catch? And there's a big one, the autovacuum will eventually kick in and run ANALYZE</code>. Which will overwrite your injected statistics with real numbers and you are back where you started.</p> To prevent this, disable autovacuum analyze on the tables you've injected.</p> -- disable autovacuum</span></span> ALTER TABLE</span> test_orders </span>SET</span> (autovacuum_enabled </span>=</span> false);</span></span> </span> -- or set analyze threshold so high it nevers kicks-in</span></span> ALTER TABLE</span> test_orders </span>SET</span> (autovacuum_analyze_threshold </span>=</span> 2147483647</span>);</span></span></code></pre> Be careful here.</strong> If you're also writing data to these tables in dev: running migrations, loading fixtures, testing inserts, the injected statistics will drift further from reality with every write. The planner will plan based on a production distribution that no longer reflects the local data. </p> For read-only query plan testing this is exactly what you want. For integration tests that modify data, you may need to re-inject statistics after each test run.</p> And please, never ever do this in production!</p> </div> What's not covered?</a> </h2> As we have seen earlier, it's not worth trying to inject relpages</code> as the planner checks the actual file size and scales it proportationally. This limits the number of absolute rows planner might estimate. I.e. to get comparable numbers to production environment you still would have to create comparable data volume (which isn't a problem when talking about the primary use case of this feature - restoring backups).</p> It's also worth to note that CREATE STATISTICS</code> used for multivariate correlations, distinct counts across column groups and MCV lists for column combinations</a> are not covered within PostgreSQL 18. Those still require ANALYZE</code> after restore. PostgreSQL 19 will close this gap with pg_restore_extended_stats()</code>.</p> Security</a> </h2> The restore functions require the MAINTAIN</code> privilege on the target table. This is the same privilege needed for ANALYZE</code>, VACUUM</code>, REINDEX</code>, and CLUSTER</code> as it was introduced in PostgreSQL 17</a>.</p> The easiest way to grant it for automation:</p> GRANT</span> pg_maintain </span>TO</span> ci_service_account;</span></span></code></pre> This grants MAINTAIN</code> on all tables in the database. Enough for a CI pipeline to inject statistics without needing superuser.</p> PostgreSQL Statistics: Why queries run slow 2026-02-26T23:35:00+00:00 Every query starts with a plan. Every slow query probably starts with a bad one. And more often than not, the statistics are to blame. But how does it really work? PostgreSQL doesn't run the query to find out - it estimates the cost. It reads pre-computed data from pg_class</code> and pg_statistic</code> and does the maths to figure out the cheapest path to your data.</p> In ideal scenario, the numbers read are accurate, and you get the plan you expect. But when they are stale, the situation gets out of control. Planner estimates 500 rows, plans a nested loop, and hits 25,000. What seemed as optimal plan turns into a cascading failure.</p> How do statistics get stale? It can be either bulk load, a schema migration, faster-than-expected growth, or simply VACUUM</code> not keeping up. Whatever the cause, the result is the same. The planner is flying blind. Choosing paths based on reality that no longer exists.</p> In this post we will go inside the two catalogs the planner depends on, understand what ANALYZE</code> actually gets for you from a 30,000-row table, and see how those numbers determine whether your query takes milliseconds or minutes.</p> Sample schema</a> </h2> For demonstration purposes we will use the same schema as in the article Reading Buffer statistics in EXPLAIN output</a>.</p> CREATE TABLE</span> customers</span> (</span></span> id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name text NOT NULL</span></span> );</span></span> </span> CREATE TABLE</span> orders</span> (</span></span> id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> customer_id </span>integer NOT NULL REFERENCES</span> customers(id),</span></span> amount </span>numeric</span>(</span>10</span>,</span>2</span>)</span> NOT NULL</span>,</span></span> status text NOT NULL DEFAULT</span> 'pending'</span>,</span></span> note </span>text</span>,</span></span> created_at </span>date NOT NULL DEFAULT</span> CURRENT_DATE</span></span> );</span></span> </span> INSERT INTO</span> customers (</span>name</span>)</span></span> SELECT</span> 'Customer '</span> \|\|</span> i</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>2000</span>) </span>AS</span> i;</span></span> </span> INSERT INTO</span> orders (customer_id, amount, </span>status</span>, note, created_at)</span></span> SELECT</span></span> (random()</span> </span> 1999</span> +</span> 1</span>)::</span>int</span>,</span></span> (random()</span> </span> 500</span> +</span> 5</span>)::</span>numeric</span>(</span>10</span>,</span>2</span>),</span></span> (</span>ARRAY</span>['pending','shipped','delivered','cancelled'])[floor(random()4+1)::int],</span></span> CASE WHEN</span> random()</span> <</span> 0</span>.</span>3</span> THEN</span> 'Some note text here for padding'</span> ELSE NULL END</span>,</span></span> '2022-01-01'</span>::</span>date +</span> (random()</span> </span> 1095</span>)::</span>int</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>100000</span>);</span></span> </span> ANALYZE customers;</span></span> ANALYZE orders;</span></span></code></pre>What the Planner Reads</a> </h2> As mentioned above, every decision the planner makes is based on two sources:</p> table-level metadata from pg_class</code></li> column-level metadata from pg_statistic</code>.</li> </ul> pg_class - relational-level stats</a> </h3> It actually tracks all relations</i>. Not just tables and indexes, but also partitions, TOAST tables, sequences, composite types, and views. </div>Every table, index and materialized view has a row in `pg_class`. Before it even looks at column-level statistics, the planner reads four values from it: Column</th> Meaning</th></tr></thead> relpages</code></td> Number of 8KB pages representing the table on the disk</td></tr> reltuples</code></td> Estimated number of live rows in the table</td></tr> relallvisible</code></td> Pages where all tuples are visible to all transactions</td></tr> </tbody></table> For our sample table it looks like this.</p> SELECT</span> relname, relpages, reltuples, relallvisible</span></span> FROM</span> pg_class</span></span> WHERE</span> relname </span>=</span> 'orders'</span>;</span></span></code></pre> relname \| relpages \| reltuples \| relallvisible</span></span> ---------+----------+-----------+---------------</span></span> orders \| 856 \| 100000 \| 856</span></span></code></pre>Compared to reading page from disk, handling single tuple represents tiny overhead. Per-row overhead is configured using cpu_tuple_cost</code>, which defaults to 0.01</code></div> The planner sees 100,000 rows spread across 856 pages. Every cost estimate starts from those two numbers. relpages</code> drives sequential scan cost - each page is one unit of I/O work as configured via seq_page_cost</code>. reltuples</code> controls estimates for joins, aggregations, and pretty much everything else.</p> The reltuples</code> value is only an estimate, not a live count. It's updated by ANALYZE</code> (and autovacuum), not by individual INSERTs or DELETEs. Between ANALYZE runs, PostgreSQL scales reltuples</code> proportionally when relpages</code> value changes - if the table grows by 10% in pages, the planner assumes 10% more rows too.</p> This works well enough for normal growth, but breaks down with bloat. If dead tuples are inflating the number of pages used, without adding real rows, the planner overestimates the table size.</p> </div> The other column mentioned above matters for specific operations. relallvisible</code> tells the planner how much of the table can be read with an index-only scan. Meaning an index-only scan can return results using the index without checking the heap for visibility.</p> pg_statistic (via pg_stats) - column-level stats</a> </h3> Knowing the size of a table is only half the picture. To estimate how many rows might match, the planner needs to understand the data inside each column. PostgreSQL maintains statistics in pg_statistic</code> catalog. Which you're most likely never going to use directly. In practice you will use the view pg_stats</code> which presents the human-friendly data behind it.</p> The most interesting values it exposes are:</p> Statistic</th> What it tells the planner</th></tr></thead> null_frac</code></td> Fraction of entries that are NULL values</td></tr> avg_width</code></td> Average width in bytes</td></tr> n_distinct</code></td> Number of distinct values (negative means fraction of rows)</td></tr> most_common_vals</code></td> Most frequent values</td></tr> most_common_freqs</code></td> Frequencies of those values</td></tr> histogram_bounds</code></td> Values dividing the remaining data into equal-population buckets (most_common_vals are excluded)</td></tr> correlation</code></td> Statistical correlation between physical row ordering and logical ordering of the column values</td></tr> </tbody></table> SELECT</span> attname, null_frac, avg_width, n_distinct,</span></span> most_common_vals, most_common_freqs, histogram_bounds, correlation</span></span> FROM</span> pg_stats</span></span> WHERE</span> tablename </span>=</span> 'orders'</span> AND</span> attname </span>=</span> 'status'</span>;</span></span></code></pre>-[ RECORD 1 ]-----+--------------------------------------</span></span> attname \| status</span></span> null_frac \| 0</span></span> avg_width \| 8</span></span> n_distinct \| 4</span></span> most_common_vals \| {pending,shipped,delivered,cancelled}</span></span> most_common_freqs \| {0.25396666,0.25,0.24973333,0.2463}</span></span> histogram_bounds \|</span></span> correlation \| 0.2524199</span></span></code></pre> From this planner knows there are exactly 4 distinct values, more or less equally distributed, without any NULL values. When you write a predicate WHERE status = 'pending'</code> it will estimate ~25% of rows are going to match. All that by not running a query, but reading the catalog row.</p> Different situation is when checking column note</code>.</p> SELECT</span> attname, null_frac, avg_width, n_distinct,</span></span> array_length(most_common_vals, </span>1</span>) </span>AS</span> mcv_count,</span></span> array_length(histogram_bounds, </span>1</span>) </span>AS</span> histogram_buckets</span></span> FROM</span> pg_stats</span></span> WHERE</span> tablename </span>=</span> 'orders'</span> AND</span> attname </span>=</span> 'note'</span>;</span></span></code></pre>-[ RECORD 1 ]-----+-------</span></span> attname \| note</span></span> null_frac \| 0.6982</span></span> avg_width \| 32</span></span> n_distinct \| 1</span></span> mcv_count \| 1</span></span> histogram_buckets \|</span></span></code></pre> For this column there's ~70% of NULLs, and only one distinct value. Therefore WHERE note IS NOT NULL</code> will estimate 30% of rows.</p> Selectivity in Action</a> </h2> Now that we have covered what data the planner has available, let's have a look at how it's used to estimate how many rows a certain part of a query will read. This "guess" is called Selectivity</strong>. And it's defined as a floating-point number between 0 and 1.</p> The formula is pretty simple:</p> Estimated Rows = Total Rows * Selectivity</span></span></code></pre> But the way Selectivity is calculated depends entirely on the operator you use (=</code>, <</code>, ></code> or LIKE</code>).</p> The equality (Most Common Values)</a> </h3> Equality is easiest to start with. When you use WHERE status = 'shipped'</code> the planner first checks most_common_vals</code> (MCV) list. If there's a match, the selectivity is same as the one provided by most_common_freqs</code>.</p> If the value isn't in the list, the planner assumes the value is part of the "remaining" population. It subtracts all MCV frequencies from 1.0 and divides the remainder by the number of other distinct values.</p> (1.0 - (SELECT sum(s) FROM unnest(most_common_freqs) s))</span></span> /</span></span> (n_distinct - array_length(most_common_vals, 1))</span></span></code></pre>The Range lookup (The Histogram)</a> </h3> Life would be easy if we would be looking only for exact values. Most of the time we need to utilise range lookup. For example WHERE amount > 400</code>.</p> MCVs are useless in this case as there might be thousands or millions of unique constants. This is where histogram_bounds</code> comes in. PostgreSQL divides the column values into a number of buckets, where each bucket contains equal number of rows (not values).</p> The Selectivity in this case is determined by how many buckets your query covers. In our example, if we have bounds (100, 200, 300, 400, 500, 600)</code> the planner will establish it covers 2 full buckets. Since there are 5 buckets in total the Selectivity is going to be 0.4 (2/5).</p> If you are now wondering where 2 and 5 come from? The histogram_bounds</code> array defines boundaries between buckets. With bounds (100, 200, 300, 400, 500, 600)</code> there are 5 buckets in total.</p> (100-200), (200-300), (300-400), (400-500), (500-600)</span></span></code></pre> And the same logic applies to matching. The condition amount > 400</code> matches on (400-500)</code> and (500-600)</code> buckets (2 in total).</p> The histogram's biggest weakness is linear interpolation. If your data has massive spikes in distribution the planner will always assume a perfect distribution.</div> Slightly more complex situation happens in the cases where your value falls inside a bucket. If your query has WHERE amount > 350</code> the planner locates the bucket containing value 350 (in our example (300-400)</code>), assumes the data is linearly distributed and calculates the ratio within that bucket.</p> Search and pattern matching</a> </h3> This is probably the most treacherous territory for the planner. For substring matching patterns like WHERE note LIKE '%middle%'</code>, there's no histogram or list of values to rely on. The planner must fall back to "magic constants" hardcoded in the PostgreSQL source code.</p> The default for generic patterns is 0.5% of the total rows, defined as</p> #define DEFAULT_MATCH_SEL 0.005</span></span></code></pre> Slightly better situation comes for prefixed matches like WHERE note LIKE 'boringSQL%'</code> where PostgreSQL can fall back to range conditions and use histogram bounds. While this is a subtle difference, it makes a night and day difference.</p> Correlation and index scan cost</a> </h3> Remember correlation</code> from the pg_stats</code>? It says how closely the physical order of rows on disk matches the logical order of column values. Values close to 1.0 mean high correlation, values near 0 mean data is laid out randomly across pages.</p> If you recall how data is organized in 8KB page</a> the row locality matters. This matters because it determines whether an index scan is worth it. The planner assumes a random page read costs 4× more than a sequential one (random_page_cost = 4.0</code> vs seq_page_cost = 1.0</code>). When correlation is high, the rows an index points to are physically adjacent. The planner expects sequential I/O and costs the scan cheaply. When correlation is low, each lookup likely hits a different page, and the planner costs each of those reads at the higher random rate. That difference alone can be enough to make a sequential scan cheaper than an index scan.</p> n_distinct and join estimation</a> </h3> Value n_distinct</code> plays an important role in joins. This further stresses the importance of running ANALYZE</code> after bulk data changes.</p> Let's imagine this simplified logic for the equality join:</p> In reality, Postgres also accounts for null_frac</code> (NULLs don't join) and MCVs on both sides. If both sides have MCVs, it calculates the "inner product" of the frequencies.</div> estimated_rows = (rows_left × rows_right) / max(n_distinct_left, n_distinct_right)</span></span></code></pre> Let's say you're trying to join two tables, both having roughly 2,000 distinct values for the join key. Outdated n_distinct</code> will cause significant estimation drifts and the planner may pick a wrong join strategy altogether.</p> This is the heap we are talking about</a> </h3> Please, keep in mind this describes how the planner estimates rows for a basic heap scan. Indexes, join selectivity, and complex types like JSONB each come with their own estimation logic, operator handling and quirks. The fundamentals of estimation are the same nevertheless.</p> What if there are no statistics?</a> </h2> So far we have touched on why up-to-date statistics are a must. But what if there are no statistics at all? For example for a new table or new column when ANALYZE</code> has not yet run.</p> In those cases PostgreSQL falls back to hardcoded defaults.</p> Condition type</th> Default selectivity</th> Constant</th></tr></thead> Equality (=</code>)</td> 0.5%</td> DEFAULT_EQ_SEL = 0.005</code></td></tr> Range (></code>, <</code>)</td> 33.3%</td> DEFAULT_INEQ_SEL = 0.3333</code></td></tr> Range (BETWEEN</code>)</td> 0.5%</td> DEFAULT_RANGE_INEQ_SEL = 0.005</code></td></tr> Pattern matching (LIKE</code>)</td> 0.5%</td> DEFAULT_MATCH_SEL = 0.005</code></td></tr> IS NULL</code></td> 0.5%</td> DEFAULT_UNK_SEL = 0.005</code></td></tr> IS NOT NULL</code></td> 99.5%</td> DEFAULT_NOT_UNK_SEL = 0.995</code></td></tr> </tbody></table> Nothing that a quick ANALYZE</code> can't fix, correct? Or maybe not.</p> Where No Statistics Go</strong> While this article focuses on statistics and getting them right, there are situations where no statistics will be available (never or not predictably). If you believe it won't affect you, please, think twice. CTEs and subqueries</strong> when not inlined/materialized have no statistics. See Good CTE, Bad CTE</a> for when exactly this happens.</li> Temporary tables</strong> are not touched by autovacuum, so no automatic ANALYZE</code>.</li> Foreign tables</strong> do not guarantee stats are propagated.</li> And, to a big surprise, computed expressions in WHERE</strong> like WHERE amount * 1.1 > 500</code> or lower(email) = 'hello@example.com'</code>; unless you create an expression index or extended statistics.</li> </ul> And btw, do you know about TRUNCATE</code> - it's fast way to get rid of the data, but it leaves statistics behind...</p> </div> How ANALYZE</code> Works</a> </h2> As we have already mentioned several times, ANALYZE</code> is the only mechanism that populates pg_class</code> and pg_statistic</code> with fresh data. Understanding what it samples, what it computes, and what it misses, is key to understanding why statistics are sometimes wrong.</p> The process itself consists of 6 separate steps (as reported by pg_stat_progress_analyze</code>):</p> initializing</li> acquiring sample rows</li> acquiring inherited sample rows (child/partitioned tables)</li> computing statistics (MCVs, histograms, correlation, etc.)</li> computing extended statistics (see below)</li> finalizing and writing to pg_statistic</code></li> </ol> For our purposes we will only cover sampling, computing statistics and writing to pg_statistic</code>.</p> The sampling mechanism</a> </h3> ANALYZE actually doesn't read the entire table. It samples what is considered to be a statistically justified minimum sample size (defined as 300 for the reservoir sampling</a> algorithm). For PostgreSQL that means 300 × default_statistics_target</code> rows.</p> SHOW default_statistics_target;</span></span></code></pre> default_statistics_target</span></span> ---------------------------</span></span> 100</span></span> (1 row)</span></span></code></pre> With the default target of 100, that's 30,000 rows. For our 100,000-row orders table, ANALYZE</code> reads about 30% of the data. For a 50-million-row table, it reads 0.06%. The same target also controls the size of the MCV list and histogram. Up to 100 entries each.</p> With a target of 100, don't be surprised that histogram_bounds</code> contains 101 values. 100 buckets need 101 boundaries to close them.</div> The sampling is two-stage. First, ANALYZE</code> selects a random set of pages. Then it reads all live rows from those pages. This gives a representative cross-section without reading every page.</p> Computing statistics</a> </h3> Once ANALYZE has its 30,000 sample rows (considering the default values), it processes each column independently. The pipeline for a single column looks roughly like this:</p> First, it counts NULLs and calculates null_frac</code> and avg_width</code> - the cheapest statistics to compute. Then it sorts the non-null values and builds the MCV list by counting duplicates. Values that appear frequently enough make the cut; the rest are passed to the histogram builder, which divides them into equal-population buckets. Finally, because the values are already sorted, ANALYZE compares the logical sort order against the physical tuple positions (which page each row came from) to compute correlation</code>.</p> The key detail here is that MCVs and histograms are not built from the same pool. Values that land in most_common_vals</code> are excluded from histogram_bounds</code>. This is why you'll sometimes see a column with MCVs but no histogram, or a histogram but no MCVs. They represent different slices of the same data.</p> Writing to pg_statistic</a> </h3> Once all columns are processed, ANALYZE writes the results into pg_statistic</code> - one row per column. If a row for that column already exists, it's updated in place. This is a regular heap update, which means the old row becomes a dead tuple. On tables with many columns or frequent ANALYZE runs, this can cause pg_statistic</code> itself to bloat.</p> After pg_statistic</code> is updated, ANALYZE refreshes relpages</code>, reltuples</code>, and relallvisible</code> in pg_class</code>. These values are recalculated from the sampling, not from a full table scan (they are estimates too).</p> Controlling statistics quality</a> </h2> default_statistics_target</a> </h3> The default target of 100 works well for most columns. It means up to 100 MCVs, 101 histogram bounds, and a 30,000-row sample. Increasing it helps when:</p> A column has many distinct values and the top 100 don't cover enough of the distribution</li> Range queries on skewed data produce bad estimates because histogram buckets are too coarse</li> Join estimates are off because n_distinct</code> is inaccurate</li> </ul> The cost scales linearly. Setting it to 1000 means 300,000 sampled rows, up to 1000 MCVs, more catalog storage, and slower planning from larger arrays to search. The maximum is 10,000.</p> You don't have to raise it globally. For a single problematic column:</p> ALTER TABLE</span> orders </span>ALTER</span> COLUMN </span>status SET STATISTICS</span> 500</span>;</span></span> ANALYZE orders;</span></span></code></pre> Now status</code> gets up to 500 MCVs and 501 histogram bounds, while every other column stays at 100.</p> Extended statistics</a> </h3> Standard statistics treat each column independently. This means the planner can't know that city = 'Edinburgh'</code> and country = 'UK'</code> are correlated. It multiplies their selectivities independently, potentially underestimating by orders of magnitude.</p> Extended statistics solve this for specific column combinations:</p> CREATE STATISTICS</span> orders_status_date (dependencies, ndistinct, mcv)</span></span> ON status</span>, created_at </span>FROM</span> orders;</span></span> ANALYZE orders;</span></span></code></pre> This tells ANALYZE to compute functional dependencies, combined distinct counts, and combined MCVs between the columns. The planner can then use these to avoid the independence assumption for queries filtering on both columns.</p> The three types of extended statistics serve different purposes:</p> dependencies</strong> capture functional dependencies between columns. Helps when knowing one column's value determines or narrows another's (e.g. zip_code</code> largely determines city</code>).</li> ndistinct</strong> tracks the number of distinct value combinations across columns. Helps with GROUP BY</code> on multiple columns where the planner would otherwise multiply distinct counts independently.</li> mcv</strong> builds a combined most-common-values list for column tuples. The most powerful but most expensive option. Helps with multi-column WHERE conditions on correlated values.</li> </ul> You can create extended statistics with any combination of these types. Start with dependencies</code> as it's cheapest, add mcv</code> when multi-column filter estimates are consistently wrong.</p> Extended statistics are worth creating when you see EXPLAIN</code> estimates that are consistently wrong on multi-column filters, and the columns are logically correlated. They are computed during step 5 of the ANALYZE process and stored in pg_statistic_ext_data</code>.</p> Diagnosing bad estimates</a> </h2> When a query is slow, the first question should always be: did the planner estimate correctly? Compare the estimate to reality with EXPLAIN ANALYZE</code>.</p> Estimate off by handful of rows means statistics are fine. But when you see estimates off by 10x or more, that's where planning goes wrong. A nested loop that looks cheap for 100 rows becomes a disaster at 10,000.</p> The statistics tell you what the planner believed, and comparing that with reality tells you what to do next. Either run ANALYZE</code>, consider tuning the statistics target for a specific column, or creating extended statistics if multiple columns are involved.</p> The planner is only as good as what it reads from the catalog. When estimates go wrong, don't blame the planner. Check the data it's working with.</p> Inside PostgreSQL's 8KB Page 2026-02-19T21:52:00+00:00 If you read previous post about buffers</a>, you already know PostgreSQL might not necessarily care about your rows. You might be inserting a user profile, or retrieving payment details, but all that Postgres works with are blocks of data. 8KB blocks, to be precise. You want to retrieve one tiny row? PostgreSQL hauls an entire 8,192-byte page off the disk just to give it to you. You update a single boolean flag? Same thing. The 8KB page is THE atomic unit of I/O.</p> But knowing those pages exist isn't enough. To understand why the database behaves the way it does, you need to understand how it works. Every time you execute INSERT</code>, PostgreSQL needs to figure out how to fit it into one of those 8,192-byte pages.</p> The buffer pool caches them, Write-Ahead Log (WAL) protects them, and VACUUM cleans them. The deep dive into the PostgreSQL storage internals starts by understanding what happens inside those 8KB pages. Pages that are used by PostgreSQL to organize all data - tables, indexes, sequences, TOAST relations.</p> The 8KB</a> </h2> In case of Oracle, the default block size is set at database creation (DB_BLOCK_SIZE</code>), though tablespaces with non-standard block sizes can be created separately.</div>Before looking inside we actually need to discuss why 8KB in first place? The answer might be surprising - it's a setting that survived for over 40 years and nobody found a good reason to change it. Plus PostgreSQL isn't the only one who thinks that way. Oracle and SQL Server use the same exact number. The 8KB page size can be traced down to original Berkley POSTGRES project created in mid-1980s. In those times Unix systems typically used 4KB or 8KB virtual memory pages, and disk sectors were 512 bytes. Choosing 8KB meant a single database page mapped cleanly to OS memory pages and aligned well with filesystem I/O.</p> And the math still works today. Modern Linux kernels manage memory in 4KB virtual memory pages. SSDs now use 4KB physical sectors instead of 512 bytes. The default filesystem block size on ext4 and XFS is 4KB. PostgreSQL's 8KB page still maps to two OS pages, two disk sectors, two filesystem blocks. The hardware changed underneath, but the alignment is still there.</p> But the choice isn't just about hardware alignment. It's a tradeoff between two opposing forces. Make the page size too small</strong> and you going to increase the overhead (page metadata being good example). Make it too large</strong> and you waste space and increase I/O requirements when you need a single narrow row.</p> The different situation is outside OLTP world. DuckDB, designed for analytical columnar workloads, uses 256KB blocks. When you're scanning millions of rows sequentially, the overhead-per-page penalty barely matters - you want big chunks to maximize throughput.</p> Is PostgreSQL 8KB page size fixed? Technically no. PostgreSQL supports 1, 2, 4, 8, 16, or 32KB pages via --with-blocksize</code> at compile time. Some analytical workloads with wide rows benefit from 16KB or 32KB pages. But you'll need to rebuild everything from scratch, and unless you have a very specific reason and understand the downstream implications you almost certainly don't want to. The default works.</strong> </div> You can confirm the page size on any running instance.</p> SHOW block_size;</span></span> </span> block_size</span></span> ------------</span></span> 8192</span></span> (</span>1</span> row</span>)</span></span></code></pre>pageinspect</a> </h2> PostgreSQL comes with an extension called pageinspect</code> that lets you read raw page contents from SQL. It is part of the contrib modules and available on most installations. Let's enable it and create a small test table:</p> CREATE</span> EXTENSION </span>IF NOT EXISTS</span> pageinspect;</span></span> </span> CREATE TABLE</span> page_demo</span> (</span></span> id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> title </span>text NOT NULL</span>,</span></span> created_at </span>TIMESTAMPTZ DEFAULT</span> CURRENT_TIMESTAMP,</span></span> value numeric</span>(</span>10</span>,</span>2</span>)</span></span> );</span></span> </span> INSERT INTO</span> page_demo (title, created_at, </span>value</span>)</span></span> VALUES</span></span> (</span>'Ergonomic standing desk, oak finish'</span>, </span>'2026-01-15 09:23:45+00'</span>, </span>549</span>.</span>99</span>),</span></span> (</span>'Wireless mechanical keyboard'</span>, </span>'2026-01-15 10:05:12+00'</span>, </span>189</span>.</span>00</span>),</span></span> (</span>'Desk lamp'</span>, </span>'2026-01-16 14:31:22+00'</span>, </span>45</span>.</span>00</span>);</span></span></code></pre> We have three rows. PostgreSQL has written them into a single heap page. Let's look inside.</p> Reading a raw page header</a> </h2> The page_header</code> function takes a raw page and returns the parsed header fields:</p> SELECT * FROM</span> page_header(get_raw_page(</span>'page_demo'</span>, </span>0</span>));</span></span></code></pre>-[ RECORD 1 ]---------</span></span> lsn \| F/1F9ED410</span></span> checksum \| 0</span></span> flags \| 0</span></span> lower \| 36</span></span> upper \| 7976</span></span> special \| 8192</span></span> pagesize \| 8192</span></span> version \| 4</span></span> prune_xid \| 0</span></span></code></pre> Every one of those fields lives in the first 24 bytes of the page. Together they form the page header</strong>, and they tell PostgreSQL everything it needs to know about the page before touching any actual data.</p> The page header: 24 bytes of metadata</a> </h2> The first two fields are about safety, making sure the page survives crashes and silent corruption.</p> pd_lsn (8 bytes)</strong> the Log Sequence Number of the last WAL record that modified this page. During crash recovery, PostgreSQL compares this LSN against the WAL stream. If the WAL record's LSN is less than or equal to the page's LSN, the page is already up to date and the record is skipped. If it is greater, the record must be replayed. This single field is what makes crash recovery work.</p> pd_checksum (2 bytes)</strong> a checksum of the page contents. This is only active if the cluster was initialized with initdb --data-checksums</code> (or checksums were enabled later with pg_checksums</code>). When enabled, PostgreSQL verifies the checksum every time it reads a page from disk. A mismatch means silent data corruption, the kind that would otherwise go undetected until your data is wrong in ways nobody can explain.</p> Are checksums enabled on your cluster? Before PostgreSQL 17, they were off by default. Check with SHOW data_checksums;</code>. If you see off</code> on a production database, that's worth fixing. You can enable them retroactively with pg_checksums</code>, though it requires a full shutdown.</p> Magic of PD_ALL_VISIBLE</code></strong> Indexes know what your data is, but not who is allowed to see it (in case a row was recently deleted or updated). Postgres has to do extra work to double-check the main table (the "heap") to ensure a row is visible.</p> When a page is marked PD_ALL_VISIBLE, it guarantees every row on that page is old enough to be seen by everyone. This lets Postgres skip the expensive heap check entirely and serve data straight from the index. A speed boost you may know as an Index-Only Scan. </p> </strong></div> The next fields are the page's spatial ma. They tell PostgreSQL where things are and where there's room for more.</p> pd_flags (2 bytes)</strong> bit flags that describe the page state. The important ones are: PD_HAS_FREE_LINES</code> (are there any unused line pointers?), PD_PAGE_FULL</code> (not enough free space for new tuple) and PD_ALL_VISIBLE</code> (all tuples on page are visible to everyone).</p> pd_lower</strong> and pd_upper</strong> (2 bytes each) define the free space gap. pd_lower</code> marks where the line pointer array ends -- in our output it is 36, that is 24 bytes of header plus 3 line pointers at 4 bytes each: 24 + (3 x 4) = 36. pd_upper</code> marks where tuple data begins -- our value is 7976, meaning the three tuples together occupy bytes 7976 through 8191. Everything between these two offsets is free space. As you insert rows, these two values creep toward each other until there's no room left.</p> pd_special (2 bytes)</strong> the byte offset to the "special space" at the end of the page. For heap (table) pages, this equals the page size (8192), meaning there is no special space. For index pages, this area contains index-specific metadata.</p> pd_pagesize_version (2 bytes)</strong> encodes both the page size and the layout version number. The version is currently 4 for all modern PostgreSQL releases.</p> Finally, one field dedicated to cleanup.</p> The Mini-Cleanup (Page-Level Pruning)</strong> Instead of waiting for a heavy VACUUM, Postgres can do garbage collection on the fly. When a normal query reads a page, it checks `pd_prune_xid`. If that transaction ID is older than all currently running transactions, the query instantly reclaims the dead space itself.</p> This is one of the cases where SELECT might modify data pages.</p> </div> pd_prune_xid (4 bytes)</strong> the oldest transaction ID whose dead tuples have not yet been pruned from this page. When a new transaction accesses the page and finds that prune_xid</code> is older than the global horizon, it triggers page-level pruning. A lightweight cleanup that reclaims space without a full VACUUM.</p> What is a line pointer?</a> </h2> If you're paying attention, we just mentioned line pointers in pd_lower</code> description. If the page header discussed above is the general metadata, the line pointers</strong> (internally represented as ItemIdData</code>) are the page table of contents.</p> Every time you insert a row, PostgreSQL doesn't just drop the raw data into the page. It actually splits the job into two steps.</p> It puts the bulky, unpredictable row data (the tuple) at the very bottom of the page (more on this in the next section).</li> It adds a tiny, fixed 4-byte line pointer right after the page header.</li> </ol> This pointer acts as a direct map. It holds exact byte offset and length of the tuple it points to. It allows PostgreSQL to only look at the offset and read the required number of bytes to get the tuple.</p> The line pointer array starts immediately after the 24-byte header and grows downward. Let's look at ours.</p> SELECT</span> lp, lp_off, lp_flags, lp_len</span></span> FROM</span> heap_page_items(get_raw_page(</span>'page_demo'</span>, </span>0</span>));</span></span></code></pre> lp \| lp_off \| lp_flags \| lp_len</span></span> ----+--------+----------+--------</span></span> 1 \| 8112 \| 1 \| 79</span></span> 2 \| 8032 \| 1 \| 77</span></span> 3 \| 7976 \| 1 \| 53</span></span></code></pre> Each line pointer is 4 bytes and contains three pieces of information:</p> lp</strong> the ordinal position in the array (1-based). This, combined with the page number, forms the ctid</strong> - the physical address of a tuple. Row 1 on page 0 has ctid (0,1)</code>.</li> lp_off</strong> the byte offset within the page where the actual tuple data begins. Notice the offsets decrease: 8112, 8032, 7976. Tuples are packed from the bottom up.</li> </ul> Line pointers are the reason PostgreSQL can move tuples around within a page (during defragmentation) or perform HOT (Heap-Only Tuple) updates</strong> without invalidating index entries. An index stores a ctid like (0, 2)</code>, which means "page 0, line pointer 2". Because the index points to the pointer rather than the physical byte offset, Postgres can shuffle data or redirect pointers under the hood while the index reference remains perfectly valid. </div> lp_flags</strong> the state of the pointer. The values are: 0 = LP_UNUSED</code> (available for reuse), 1 = LP_NORMAL</code> (points to a live tuple), 2 = LP_REDIRECT</code> (points to another line pointer, used after HOT pruning), 3 = LP_DEAD</code> (the tuple has been determined dead but the pointer persists until cleanup).</li> lp_len</strong> the total length of the tuple in bytes, including the 23-byte tuple header, null bitmap, alignment padding, and actual column data.</li> </ul> The anatomy of a page</a> </h2> Here is how all these pieces fit together inside the 8KB block.</p> We already mentioned it above, but if you review the image carefully you can confirm the important insight. While line pointers are allocated downward from the header (pd_lower increases) the tuple data is stored upwards from the bottom of the page (pd_upper decreases). Free space is the gap between them.</p> This is called Slotted page layout</strong> and it's easy way how to prevent unnecessary data fragmentation. By storing predictable line pointers at top, they don't mix together with bulky and unpredicable tuples.</p> </div> </div> </div> </div> </div> Interactive Page Visualizer</strong> Watch the opposing growth directions in action. Insert rows and click regions for byte-level details.</p> Open Visualizer</a> </div> </div> Free space: the gap in the middle</a> </h2> The free space on a page is the region between pd_lower and pd_upper. For our page:</p> SELECT</span> lower, upper, upper </span>-</span> lower </span>AS</span> free_space</span></span> FROM</span> page_header(get_raw_page(</span>'page_demo'</span>, </span>0</span>));</span></span></code></pre> lower \| upper \| free_space</span></span> -------+-------+------------</span></span> 36 \| 7976 \| 7940</span></span> (1 row)</span></span></code></pre> With only three small rows, we have 7,940 bytes of free space. Almost the entire page is still available. Let's see what happens when we add more data:</p> INSERT INTO</span> page_demo (title, created_at, </span>value</span>)</span></span> SELECT</span> </span></span> 'Generated item '</span> \|\|</span> i, </span></span> '2026-01-15 00:00:00+00'</span>::</span>timestamptz +</span> (i </span>\|\|</span> ' hours'</span>)::interval, </span></span> (i </span></span> 11</span>.</span>11</span>)::</span>numeric</span>(</span>10</span>,</span>2</span>)</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>10</span>) </span>AS</span> i;</span></span> </span> SELECT</span> lower, upper, upper </span>-</span> lower </span>AS</span> free_space</span></span> FROM</span> page_header(get_raw_page(</span>'page_demo'</span>, </span>0</span>));</span></span></code></pre> lower \| upper \| free_space</span></span> -------+-------+------------</span></span> 76 \| 7336 \| 7260</span></span> (1 row)</span></span></code></pre> You can observe</p> pd_lower</code> moved from 36 to 76. Why? With 10 new line pointers at 4 bytes each we moved by 40 bytes.</li> pd_upper</code> dropped from 7,976 to 7,336. The 10 new tuples consumed 640 bytes of the data space.</li> And the free space shrank to 7,260 bytes.</li> </ul> How fast does a page fill up?</a> </h2> If we going to think about the page capacity, let's consider each row in this page costs:</p> 4 bytes</strong> for the line pointer</li> 23 bytes</strong> for the tuple header (transaction metadata, null bitmap offset, info mask)</li> Alignment padding</strong> to the nearest 8-byte boundary after the header (1 byte of padding, bringing the header to 24 bytes)</li> Actual column data</strong>: 4 bytes for the integer, variable bytes for the text, variable bytes for the numeric</li> </ul> For our new schema, each tuple is a bit larger than before. We saw earlier that our first 3 tuples took up exactly 216 bytes, which averages exactly 72 bytes per tuple. Add the 4-byte line pointer, and each row costs about 76 bytes of page space.</p> With 8,192 bytes in a page minus the 24-byte header, we have 8,168 bytes of usable space. At roughly 76 bytes per row, we can theoretically fit approximately 107 rows on a single page.</p> Let's test it. We already have 13 rows (3 original plus 10 generated). Let's insert more and check:</p> INSERT INTO</span> page_demo (title, created_at, </span>value</span>)</span></span> SELECT</span></span> 'Generated item '</span> \|\|</span> i,</span></span> '2026-01-15 00:00:00+00'</span>::</span>timestamptz +</span> (i </span>\|\|</span> ' hours'</span>)::interval,</span></span> (i </span></span> 11</span>.</span>11</span>)::</span>numeric</span>(</span>10</span>,</span>2</span>)</span></span> FROM</span> generate_series</span>(</span>11</span>, </span>150</span>) </span>AS</span> i;</span></span> </span> SELECT</span> count</span>(</span></span>) </span>AS</span> tuples_on_page_0</span></span> FROM</span> heap_page_items(get_raw_page(</span>'page_demo'</span>, </span>0</span>));</span></span></code></pre> Total of 119 tuples fit on page 0 before PostgreSQL had to start using page 1. A bit more than our estimate of 107 -- the shorter generated titles take less space than "Ergonomic standing desk, oak finish".</p> The exact number always depends on column values, but this gives you a practical sense of capacity. Because this schema includes a variable-length text column, the row size flexes, but we still hit a very predictable ceiling right around 110-120 rows per 8KB block.</p> If you want to estimate how many pages a table will need, the rough formula is: rows average_row_size / 8192</code>. But remember that "average_row_size" includes the 23-byte tuple header, alignment padding, and the 4-byte line pointer. PostgreSQL's pg_column_size()</code> function can help you measure actual row sizes.</p> </div> Spanning multiple pages</a> </h2> With 160 rows in our table, we have started storing the tuples into a second page. Let's verify using the system catalog:</p> SELECT</span> relpages, reltuples</span></span> FROM</span> pg_class</span></span> WHERE</span> relname </span>=</span> 'page_demo'</span>;</span></span></code></pre>The relpages</code> and reltuples</code> values are estimates updated by ANALYZE and autovacuum. After bulk inserts, run ANALYZE page_demo;</code> to refresh them.</div> relpages \| reltuples</span></span> ----------+-----------</span></span> 2 \| 160</span></span> (1 row)</span></span></code></pre> Two pages. Let's insert substantially more data to see a real multi-page table:</p> INSERT INTO</span> page_demo (title, created_at, </span>value</span>)</span></span> SELECT</span> </span></span> 'Generated row '</span> \|\|</span> i, </span></span> CURRENT_TIMESTAMP </span>+</span> (i </span>\|\|</span> ' minutes'</span>)::interval,</span></span> (random()</span> </span> 1000</span>)::</span>numeric</span>(</span>10</span>,</span>2</span>)</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>500</span>) </span>AS</span> i;</span></span> </span> ANALYZE page_demo;</span></span> </span> SELECT</span> relpages, reltuples</span></span> FROM</span> pg_class</span></span> WHERE</span> relname </span>=</span> 'page_demo'</span>;</span></span></code></pre> relpages \| reltuples</span></span> ----------+-----------</span></span> 6 \| 660</span></span> (1 row)</span></span></code></pre> Six pages now. Each page is an independent 8KB block with its own header. Let's confirm by reading the headers of the first two:</p> SELECT</span> 0</span> AS page</span>, lower, upper, upper </span>-</span> lower </span>AS</span> free_space</span></span> FROM</span> page_header(get_raw_page(</span>'page_demo'</span>, </span>0</span>))</span></span> UNION ALL</span></span> SELECT</span> 1</span>, lower, upper, upper </span>-</span> lower</span></span> FROM</span> page_header(get_raw_page(</span>'page_demo'</span>, </span>1</span>));</span></span></code></pre> page \| lower \| upper \| free_space</span></span> ------+-------+-------+------------</span></span> 0 \| 500 \| 552 \| 52</span></span> 1 \| 504 \| 512 \| 8</span></span> (2 rows)</span></span></code></pre> Both pages are nearly full, with only 52/8 bytes of free space remaining. Which is too little for another row to fit there.</p> The space that isn't there</a> </h2> You may have noticed that pd_special</code> equals 8192 for our heap pages. The same size as the entire page size. I.e. there's no special space allocated.</p> This applies to the heap pages. But not all pages are heap pages. Index pages use the special space at the end of the page to store index-specific metadata. For a B-tree index, it includes pointers to sibling pages (for range scans), the tree level, and flags about the page type (leaf, internal, root, deleted).</p> Let's peek at the primary key index that was automatically created for our table:</p> SELECT type</span>, live_items, dead_items, avg_item_size,</span></span> page_size, free_size</span></span> FROM</span> bt_page_stats(</span>'page_demo_pkey'</span>, </span>1</span>);</span></span></code></pre> type \| live_items \| dead_items \| avg_item_size \| page_size \| free_size</span></span> ------+------------+------------+---------------+-----------+-----------</span></span> l \| 367 \| 0 \| 16 \| 8192 \| 808</span></span> (1 row)</span></span></code></pre> The type</code> is l</code> for leaf page. And if we check its page header:</p> SELECT</span> special </span>FROM</span> page_header(get_raw_page(</span>'page_demo_pkey'</span>, </span>1</span>));</span></span></code></pre> special</span></span> ---------</span></span> 8176</span></span> (1 row)</span></span></code></pre> The special space starts at byte 8176, giving us 16 bytes (8192 - 8176) of B-tree metadata at the end of the page. Heap pages have none; index pages rely on it.</p> Putting it all together</a> </h2> Let's close with a complete view of our page. We know the header, the line pointers, the free space, and the tuple data regions. Here is a summary query that shows all of it at once:</p> SELECT</span></span> 'header'</span> AS</span> region,</span></span> 0</span> AS</span> start_byte,</span></span> 23</span> AS</span> end_byte,</span></span> 24</span> AS</span> size_bytes</span></span> UNION ALL</span></span> SELECT</span></span> 'line pointers'</span>,</span></span> 24</span>,</span></span> lower </span>-</span> 1</span>,</span></span> lower </span>-</span> 24</span></span> FROM</span> page_header(get_raw_page(</span>'page_demo'</span>, </span>0</span>))</span></span> UNION ALL</span></span> SELECT</span></span> 'free space'</span>,</span></span> lower,</span></span> upper </span>-</span> 1</span>,</span></span> upper </span>-</span> lower</span></span> FROM</span> page_header(get_raw_page(</span>'page_demo'</span>, </span>0</span>))</span></span> UNION ALL</span></span> SELECT</span></span> 'tuple data'</span>,</span></span> upper,</span></span> special </span>-</span> 1</span>,</span></span> special </span>-</span> upper</span></span> FROM</span> page_header(get_raw_page(</span>'page_demo'</span>, </span>0</span>))</span></span> UNION ALL</span></span> SELECT</span></span> 'special space'</span>,</span></span> special,</span></span> 8191</span>,</span></span> 8192</span> -</span> special</span></span> FROM</span> page_header(get_raw_page(</span>'page_demo'</span>, </span>0</span>));</span></span></code></pre> region \| start_byte \| end_byte \| size_bytes</span></span> ---------------+------------+----------+------------</span></span> header \| 0 \| 23 \| 24</span></span> line pointers \| 24 \| 499 \| 476</span></span> free space \| 500 \| 551 \| 52</span></span> tuple data \| 552 \| 8191 \| 7640</span></span> special space \| 8192 \| 8191 \| 0</span></span> (5 rows)</span></span></code></pre> 476 bytes of line pointers means 119 entries (476 / 4). 7,640 bytes of tuple data. 52 bytes of free space. And zero bytes of special space (it's heap page).</p> That is the entire PostgreSQL 8KB page. Twenty-four bytes of header tell PostgreSQL where everything is. Line pointers provide table of contents.</p> Reading Buffer statistics in EXPLAIN output 2026-02-06T15:52:00+00:00 In the article about Buffers in PostgreSQL</a> we kept adding EXPLAIN (ANALYZE, BUFFERS)</code> to every query without giving much thought to the output. Time to fix that. PostgreSQL breaks down buffer usage for each plan node, and once you learn to read those numbers, you'll know exactly where your query spent time waiting for I/O - and where it didn't have to. That's about as fundamental as it gets when diagnosing performance problems.</p> PostgreSQL 18: BUFFERS by Default</strong> Starting with PostgreSQL 18, EXPLAIN ANALYZE</code> automatically includes buffer statistics - you no longer need to explicitly add BUFFERS</code>. The examples below use the explicit syntax for compatibility with older versions, but on PG18+ a simple EXPLAIN ANALYZE</code> gives you the same information. </div> A complete example</a> </h2> For this article we will use following schema and seeded data.</p> CREATE TABLE</span> customers</span> (</span></span> id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name text NOT NULL</span></span> );</span></span> </span> CREATE TABLE</span> orders</span> (</span></span> id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> customer_id </span>integer NOT NULL REFERENCES</span> customers(id),</span></span> amount </span>numeric</span>(</span>10</span>,</span>2</span>)</span> NOT NULL</span>,</span></span> status text NOT NULL DEFAULT</span> 'pending'</span>,</span></span> note </span>text</span>,</span></span> created_at </span>date NOT NULL DEFAULT</span> CURRENT_DATE</span></span> );</span></span> </span> INSERT INTO</span> customers (</span>name</span>)</span></span> SELECT</span> 'Customer '</span> \|\|</span> i</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>2000</span>) </span>AS</span> i;</span></span> </span> -- seed data: ~100,000 orders spread across 2022-2025</span></span> INSERT INTO</span> orders (customer_id, amount, </span>status</span>, note, created_at)</span></span> SELECT</span></span> (random()</span> </span> 1999</span> +</span> 1</span>)::</span>int</span>,</span></span> (random()</span> </span> 500</span> +</span> 5</span>)::</span>numeric</span>(</span>10</span>,</span>2</span>),</span></span> (</span>ARRAY</span>['pending','shipped','delivered','cancelled'])[floor(random()4+1)::int],</span></span> CASE WHEN</span> random()</span> <</span> 0</span>.</span>3</span> THEN</span> 'Some note text here for padding'</span> ELSE NULL END</span>,</span></span> '2022-01-01'</span>::</span>date +</span> (random()</span> </span> 1095</span>)::</span>int</span> -- ~3 years of data</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>100000</span>);</span></span> </span> -- make sure stats are up to date</span></span> ANALYZE customers;</span></span> ANALYZE orders;</span></span> </span> -- we are going to skip indexes on purpose</span></span> </span> -- and fire sample query</span></span> select</span> count</span>(</span>1</span>) </span>from</span> customers;</span></span></code></pre> Let's start with a random query</p> EXPLAIN (ANALYZE, BUFFERS)</span></span> SELECT</span> o.</span></span>, </span>c</span>.</span>name</span></span> FROM</span> orders o</span></span> JOIN</span> customers c </span>ON</span> o</span>.</span>customer_id</span> =</span> c</span>.</span>id</span></span> WHERE</span> o</span>.</span>created_at</span> ></span> '2024-01-01'</span>;</span></span></code></pre> and its output.</p> QUERY PLAN</span></span> ----------------------------------------------------------------------------------------------------------------------------</span></span> Hash Join (cost=58.00..2253.87 rows=33784 width=71) (actual time=0.835..26.695 rows=33239.00 loops=1)</span></span> Hash Cond: (o.customer_id = c.id)</span></span> Buffers: shared hit=13 read=857</span></span> -> Seq Scan on orders o (cost=0.00..2107.00 rows=33784 width=58) (actual time=0.108..18.106 rows=33239.00 loops=1)</span></span> Filter: (created_at > '2024-01-01'::date)</span></span> Rows Removed by Filter: 66761</span></span> Buffers: shared read=857</span></span> -> Hash (cost=33.00..33.00 rows=2000 width=17) (actual time=0.697..0.698 rows=2000.00 loops=1)</span></span> Buckets: 2048 Batches: 1 Memory Usage: 118kB</span></span> Buffers: shared hit=13</span></span> -> Seq Scan on customers c (cost=0.00..33.00 rows=2000 width=17) (actual time=0.007..0.231 rows=2000.00 loops=1)</span></span> Buffers: shared hit=13</span></span> Planning:</span></span> Buffers: shared hit=130 read=29 dirtied=3</span></span> Planning Time: 1.585 ms</span></span> Execution Time: 28.067 ms</span></span> (16 rows)</span></span></code></pre> As that's quite a bit of information, let's break it down by individual categories.</p> Shared buffers: hit, read, dirtied & written</a> </h3> As described in previous article</a> these are most common buffers statistics you will see.</p> shared hit</strong> is number of pages found in shared buffers (i.e. cached). This is fast path where no disk I/O is required. Higher is better for performance.</p> shared read</strong> identifies number of pages not in shared buffers, fetched from disk (or OS cache), and each one of them adds potential I/O latency.</p> If you see a SELECT that shows `dirtied` pages. It's not a bug. PostgreSQL sets hint bits and prunes HOT chains during reads - the first backend to read a page after writes will dirty it. Normal behavior, not a problem.</div>shared dirtied has number of pages modified by this query. The query changed the data that was already cached (in the buffer pool) and those pages will eventually need to be written to the disk. shared written</strong> is a number of pages written to disk during query execution. To remind us, this happens when query needs buffer space but needs to evict dirty pages synchronously. If you see this repeatedly during a SELECT it might be a warning sign - your background writer is not keeping up as it should.</p> Let's have a look at our query's top-level buffer stats:</p> Buffers: shared hit=13 read=857</span></span></code></pre> Only 13 pages were cached in shared buffers, while 857 had to be fetched from disk (or OS cache). No pages were dirtied or written - expected for a pure SELECT with no side effects.</p> But where did those 13 hits come from? The breakdown by node tells us:</p> -> Seq Scan on orders o</span></span> Buffers: shared read=857</span></span> -> Seq Scan on customers c</span></span> Buffers: shared hit=13</span></span></code></pre> The customers table (in this case small - 2,000 rows, 13 pages) was fully cached - likely accessed frequently or as it's in our case accessed recently. The orders table (100,000 rows, 857 pages) had zero hits - every single page required I/O. This is typical after a restart or when scanning a table that doesn't fit comfortably in shared buffers.</p> Interpreting the ratio</a> </h3> In the context of this article we're going to consider ratio between shared hit and total buffers processed. Is there a perfect ratio you should strive for? As we will demonstrate there's no such universal value.</p> Let's calculate it for our query:</p> hit_ratio = shared hit / (shared hit + shared read)</span></span> = 13 / (13 + 857)</span></span> = 1.5%</span></span></code></pre>In an OLTP workload, the same small set of rows gets accessed over and over - fetch a customer by ID, look up an order by reference number, check inventory for a product. The working set is a small fraction of the total database. These queries touch a handful of pages each, and those pages stay hot in shared buffers because they keep getting requested. A well-tuned OLTP system naturally converges toward high hit ratios - not because someone set a target, but because the access pattern keeps the relevant data cached.</div>Honestly, that looks terrible. If this were the ratio of most of your OLTP queries you can easily say - there's a problem. But this was run on a freshly loaded dataset with cold caches - every page of the orders table had to be fetched for the first time. Run the same query again and you'll likely see most of those 857 reads become hits as shared buffers and the OS page cache warm up. On a test environment (where nothing else runs you will most likely hit the 100%). What matters is the hit ratio per query</em>, tracked over time</em>, compared to its own baseline</em>:</p> A reporting query scanning a large date range might consistently show 10-30% hit ratio. That's fine - it's expected to touch cold data.</li> A query serving your login page should be near 100%. If it drops to 80%, something changed - maybe the table grew, an index was rebuilt, or shared_buffers is under pressure from a new workload.</li> A query that ran at 95% hit ratio last week and now runs at 40% deserves investigation, regardless of whether 40% sounds "good" or "bad" in isolation.</li> </ul> The ratio is a diagnostic tool, not a scorecard. Use it to spot regressions, compare before-and-after when tuning, and understand where your query's time is actually going. A low ratio paired with high execution time points you toward I/O as the bottleneck. A high ratio with high execution time tells you to look elsewhere - maybe CPU, maybe row count, maybe a bad plan.</p> Context matters more than absolute numbers. Compare similar queries over time, not arbitrary benchmarks.</p> Local buffers</a> </h2> Local buffers track I/O for temporary tables. Unlike regular tables that live in shared buffers, temp tables use per-backend memory - each connection gets its own local buffer pool, controlled by the temp_buffers</code> setting.</p> CREATE</span> TEMP </span>TABLE</span> temp_large_orders </span>AS</span></span> SELECT</span> o</span>.</span>id</span>, </span>o</span>.</span>amount</span>, </span>o</span>.</span>status</span>, </span>o</span>.</span>created_at</span>, </span>c</span>.</span>name</span> AS</span> customer_name</span></span> FROM</span> orders o</span></span> JOIN</span> customers c </span>ON</span> o</span>.</span>customer_id</span> =</span> c</span>.</span>id</span></span> WHERE</span> o</span>.</span>amount</span> ></span> 200</span>;</span></span> </span> EXPLAIN (ANALYZE, BUFFERS)</span></span> SELECT status</span>, </span>count</span>(</span></span>), </span>sum</span>(amount)</span></span> FROM</span> temp_large_orders</span></span> GROUP BY status</span>;</span></span></code></pre> First thing you should notice is there's no shared buffers at all, at least in execution phase. The entire query ran against local buffers because temp tables are invisible to other backends.</p> QUERY PLAN</span></span> -------------------------------------------------------------------------------------------------------------------------------</span></span> HashAggregate (cost=1281.60..1284.10 rows=200 width=72) (actual time=24.659..24.661 rows=4.00 loops=1)</span></span> Group Key: status</span></span> Batches: 1 Memory Usage: 32kB</span></span> Buffers: local hit=576</span></span> -> Seq Scan on temp_large_orders (cost=0.00..979.20 rows=40320 width=48) (actual time=0.009..5.965 rows=60731.00 loops=1)</span></span> Buffers: local hit=576</span></span> Planning:</span></span> Buffers: shared hit=36 read=5</span></span> Planning Time: 0.294 ms</span></span> Execution Time: 24.708 ms</span></span> (10 rows)</span></span></code></pre> The individual values that might be reported are local hit/read</strong> with the same concept as shared, but for temp tables in the per-backend buffer pool.</p> Another value is local dirtied/written</strong> representing temp table modifications. "Dirtied" means the query modified pages in the local buffer pool. "Written" means dirty pages had to be flushed to disk to make room for new ones - the same clock-sweep eviction mechanism as shared buffers, but against the local buffer pool. Unlike shared buffers, temp table writes don't generate WAL and aren't subject to checkpointing.</p> In practice, local written</code> is rare to see - PostgreSQL handles temp table overflow efficiently enough that you're unlikely to encounter it unless your temp_buffers</code> is severely undersized relative to your temp table workload.</p> Temp buffers: when work_mem isn't enough</a> </h2> While local buffers are not that often considered a problem, or visible, temp buffers track cases where operations spill from memory to disk for sorts, hashes, and other operations that exceed current work_mem</code> settings.</p> SET</span> work_mem </span>=</span> '256kB'</span>;</span></span> </span> EXPLAIN (ANALYZE, BUFFERS)</span></span> SELECT</span> o</span>.</span>id</span>, </span>o</span>.</span>amount</span>, </span>o</span>.</span>status</span>, </span>o</span>.</span>created_at</span>, </span>c</span>.</span>name</span></span> FROM</span> orders o</span></span> JOIN</span> customers c </span>ON</span> o</span>.</span>customer_id</span> =</span> c</span>.</span>id</span></span> ORDER BY</span> o</span>.</span>amount</span> DESC</span>;</span></span></code></pre> We explictely forced lower work_mem</code> to see the impact.</p> QUERY PLAN</span></span> ----------------------------------------------------------------------------------------------------------------------------------</span></span> Sort (cost=38374.70..38874.70 rows=200000 width=36) (actual time=109.345..120.574 rows=200000.00 loops=1)</span></span> Sort Key: o.amount DESC</span></span> Sort Method: external merge Disk: 9736kB</span></span> Buffers: shared hit=1738, temp read=3636 written=3722</span></span> -> Hash Join (cost=116.00..4353.56 rows=200000 width=36) (actual time=1.597..34.857 rows=200000.00 loops=1)</span></span> Hash Cond: (o.customer_id = c.id)</span></span> Buffers: shared hit=1738</span></span> -> Seq Scan on orders o (cost=0.00..3712.00 rows=200000 width=27) (actual time=0.016..6.973 rows=200000.00 loops=1)</span></span> Buffers: shared hit=1712</span></span> -> Hash (cost=66.00..66.00 rows=4000 width=17) (actual time=1.568..1.569 rows=4000.00 loops=1)</span></span> Buckets: 4096 Batches: 1 Memory Usage: 235kB</span></span> Buffers: shared hit=26</span></span> -> Seq Scan on customers c (cost=0.00..66.00 rows=4000 width=17) (actual time=0.012..0.629 rows=4000.00 loops=1)</span></span> Buffers: shared hit=26</span></span> Planning:</span></span> Buffers: shared hit=15</span></span> Planning Time: 1.184 ms</span></span> Execution Time: 123.932 ms</span></span></code></pre> There you can see the temp read/written</strong> with number of pages read from and written to temporary files on disk. This indicates the operation couldn't fit in memory.</p> Naming Confusion Alert</strong></br> temp read/written</code> in EXPLAIN has nothing</strong> to do with the temp_buffers</code> parameter.</p> temp_buffers</code> = memory for temporary tables (CREATE TEMP TABLE</code>)</li> temp read/written</code> = disk spill from sorts/hashes (governed by work_mem</code>)</li> </ul> </div> The Sort Method: external merge Disk: 9736kB</code> confirms it - sorting 200,000 rows with only 256kB of work_mem</code> forced PostgreSQL to spill ~9.7MB to temporary files on disk. The temp written=3722</code> happened during the sort phase as pages were flushed out, and temp read=3636</code> happened during the merge phase as PostgreSQL read them back to produce the final sorted result.</p> Notice something else: the Hash Join and everything below it shows only shared hit=1738</code> with no temp buffers at all. The hash table for 4,000 customers fit comfortably in 235kB of memory. The temp spill is isolated to the Sort node - buffer stats always attribute I/O to the node that caused it.</p> Try bumping work_mem</code> to something reasonable and the spill disappears:</p> SET</span> work_mem </span>=</span> '16MB'</span>;</span></span></code></pre> You should see no temp</code> buffers at all. The sort completed in memory, execution time dropped, and the only I/O was reading the actual table data.</p> To reduce temp file usage you can:</p> Increase work_mem</code> (careful there, don't forget it's per-operation setting, not per-query, so a complex query with multiple sorts or hash joins allocates work_mem</code> for each one)</li> Optimize the query to process fewer rows before the sort</li> And probably most importantly, consider adding indexes to avoid sorts entirely - an index on orders(amount DESC)</code> would eliminate the sort node altogether</li> </ul> Planning buffers</a> </h2> Up until now we have avoided planning buffers completely. It's an addition that started with PostgreSQL 13, allowing you to see the buffer usage during the query planning, separate from the execution:</p> Planning:</span></span> Buffers: shared hit=36 read=5</span></span></code></pre> Why does planning need buffers? The planner reads system catalogs (pg_class</code>, pg_statistic</code>, pg_index</code>, etc.) to understand table structures and statistics. Complex queries touching many tables can have a non-trivial impact on planning-time I/O.</p> High read</code> count in planning phase suggests either system catalogues aren't cached (cold start most likely), or your query is touching many tables or columns.</p> If planning time is a problem, ensure system catalogs stay hot. On systems with many partitions, planning overhead can become significant - this is one reason partition pruning matters.</p> The blurry line between planning and execution buffers</a> </h3> While writing the articles for this blog I often face the doubts whatever I have everything correct. Because with a complex system like PostgreSQL one is always learning. Recently learned more about planning buffers based on my somewhat technically imprecise assumption.</p> PostgreSQL actually does not resolve all metadata during the planning phase. The planner does the minimum work needed to choose the best plan, but defers some catalog lookups to execution time. When a Sort node first runs, it looks up the comparison function from pg_amproc</code> via get_opfamily_proc()</code>. That lookup hits shared buffers and gets counted as execution</em> buffers. On the second run in the same session, the syscache already has that information - no buffer access, fewer reported buffers.</p> Putting it together</a> </h2> Here's a sample output of a query with problems across every buffer category:</p> Buffers: shared hit=50 read=15000 written=847</span></span> temp read=2500 written=2500</span></span> Planning:</span></span> Buffers: shared hit=12 read=156</span></span> Planning Time: 45.678 ms</span></span> Execution Time: 12345.678 ms</span></span></code></pre> Reading this top to bottom: the hit ratio is abysmal (50 hits vs 15,000 reads), so the working set isn't cached. The written=847</code> means the query forced synchronous evictions - the background writer can't keep up. The temp spill points to an operation exceeding work_mem</code>. Even planning needed 156 reads, suggesting system catalogs got evicted from cache.</p> Each number points to a specific tuning lever: shared_buffers</code>, bgwriter_lru_maxpages</code>, work_mem</code>, or query optimization to touch less data.</p> Looking beyond single queries</a> </h2> Single query analysis is useful, but patterns across your workload matter more. pg_stat_statements</code> exposes the same buffer counters aggregated over time:</p> SELECT</span></span> substring</span>(query, </span>1</span>, </span>60</span>) </span>AS</span> query,</span></span> calls,</span></span> shared_blks_hit,</span></span> shared_blks_read,</span></span> round</span>(</span>100</span>.</span>0</span> </span> shared_blks_hit </span>/</span></span> nullif</span>(shared_blks_hit </span>+</span> shared_blks_read, </span>0</span>), </span>2</span>) </span>AS</span> hit_pct,</span></span> temp_blks_written</span></span> FROM</span> pg_stat_statements</span></span> WHERE</span> calls </span>></span> 100</span></span> ORDER BY</span> shared_blks_read </span>DESC</span></span> LIMIT</span> 10</span>;</span></span></code></pre> This shows which queries are causing the most disk reads across your system - often more actionable than analyzing one query at a time.</p> Conclusion</a> </h2> Buffer statistics transform EXPLAIN from "here's the plan" to "here's exactly where the time went." Every number points to a specific cause and a specific fix. Once you start reading them, you stop guessing and start tuning.</p> If you need to get bigger picture on buffer management, check out Introduction to Buffers in PostgreSQL</a>.</p> Introduction to Buffers in PostgreSQL 2026-01-24T16:57:00+00:00 The work around RegreSQL</a> led me to focus a lot on buffers</strong>. If you are a casual PostgreSQL user, you have probably heard about adjusting shared_buffers</code> and followed the good old advice to set it to 1/4 of available RAM. But after we went a little bit too enthusiastic about them on a recent Postgres FM episode</a> I've been asked what that's all about.</p> Buffers are one of those topics that easily gets forgotten. And while they are a foundation block of PostgreSQL's performance architecture, most of us treat them as a black box. This article is going to attempt to change that.</p> The 8KB page</a> </h2> There's one concept we need to cover before diving into the buffers. And that's the concept of the 8KB page. Everything in PostgreSQL is stored in blocks that are 8KB wide.</p> When PostgreSQL reads the data, it does not read individual rows. It reads the entire page. When it writes something, same thing - same page. You want to retrieve one small row, you will always retrieve much more data to go along with it. And if you followed carefully, same applies to writes.</p> -- you can check the block size (which should be almost always 8192 bytes)</span></span> show block_size;</span></span> </span> block_size</span></span> ------------</span></span> 8192</span></span> (</span>1</span> row</span>)</span></span></code></pre> Every table and index is a collection of these pages. A row might span multiple pages if it's large enough, but the page remains the atomic unit of I/O.</p> Want to look inside?</strong> If you want to see exactly how PostgreSQL packs your data, headers, and line pointers into these 8,192 bytes, check out the byte-level tour in Inside PostgreSQL's 8KB Page</a>. </div> PostgreSQL vs OS</a> </h2> The interesting part is understanding why PostgreSQL needs to maintain its own infrastructure for its own buffer cache, when the operating system can already cache disk pages.</p> The answer is quite simple. PostgreSQL understands the data it reads. Whilst the operating system only sees files and bytes. PostgreSQL sees tables, indexes, query plans and has semantic knowledge to cache things faster.</p> Consider this example: a query needs to perform a sequential scan of a large table. The OS might happily cache all those pages, but PostgreSQL knows this is a one-time operation and uses a special strategy (ring buffers) to avoid eviction of the main cache.</p> The second important aspect is ACID - or better, the guarantee that write-ahead log (WAL) reaches stable storage before the data page is modified. The OS does not differentiate and can't effectively guarantee this durability requirement (only at the cost of performance impact).</p> shared_buffers</a> </h2> Now we can move to what we all know is the main PostgreSQL "cache". The shared_buffers</code> parameter controls the size of the shared memory, accessible to all backend processes. If any backend needs to retrieve a page, it first checks the shared buffers. If the page is present - it's a hit. No disk I/O is needed. Miss? Read it from disk (or OS cache) and store it in shared buffers for next time.</p> WAL has its own buffer area (wal_buffers</strong>). This separation exists because WAL writes are sequential and must be persisted before the corresponding data change can be considered committed. The default is 3% of shared_buffers, capped at 16MB (one WAL segment).</div> show shared_buffers;</span></span> </span> shared_buffers</span></span> ----------------</span></span> 128MB</span></span> (</span>1</span> row</span>)</span></span></code></pre> The default value 128MB is very conservative and is part of making sure the PostgreSQL default installation will work pretty much on any system (including those with limited RAM). But in your regular environment this value should typically be much higher.</p> The value of 128MB there refers to the actual buffered content. If you consider a single page being 8KB, you can imagine this as 16,384 individual slots for storing the data.</p> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> Interactive Visualizer</strong> Explore how PostgreSQL's buffer pool actually works. Watch the clock sweep algorithm evict cold pages, see buffers transition between clean, dirty, and pinned states, and understand how the hash table enables O(1) lookups.</p> Open Visualizer</a> </div> </div> However, there's more to the buffer pool than just the pages themselves. PostgreSQL needs to track metadata and provide fast lookups, so the shared memory area is organized into three components:</p> buffer blocks</strong> - the actual 8KB pages where the data lives</li> buffer descriptors</strong> - a parallel array of ~64-byte structures, one per slot.</li> hash table</strong> used for mapping page identifiers to individual buffer slots.</li> </ul> Each descriptor then tracks which page is cached in the slot (tag), flags about the state (dirty, valid and I/O in-progress), and pin/usage counters.</p> O(1) = constant time, regardless of buffer pool size.</div>The hash table enables fast lookups. When a backend needs a specific page, it hashes the page identifier and jumps directly to the right bucket—no need to scan all 16,384 slots. This keeps buffer lookups at O(1) regardless of pool size. When a backend needs page N of table orders</code>, it hashes the identifier, looks up the hash table - which drives the hit/miss logic.</p> Pin and usage counts</a> </h2> What happens to each buffer slot is ruled by the two counters mentioned above: pin and usage count.</p> Pin count tracks active references. When a backend (for example a running query) is actively reading or modifying a page, it pins the buffer so it can't be evicted. When the backend finishes, it unpins the buffer.</p> Usage count tracks how recently and frequently a buffer was accessed. Each access increments the count (capped at 5). During eviction, the clock sweep decrements this value—buffers with higher counts survive longer, while those at 0 get evicted.</p> The usage counter is important to avoid behaviour when a single sequential scan would flush the entire buffer pool. Imagine reading a full 1GB table. Without this protection, it would evict everything in shared buffers, ignoring frequently accessed data. We will also touch on this specific behaviour later, in the ring buffers.</p> PostgreSQL provides you functionality to examine what's happening in the shared buffer cache in real time using extension pg_buffercache</a>.</p> The clock sweep algorithm</a> </h2> Now that we covered tracking of the usage, what happens when PostgreSQL needs to load a page and all slots are taken? It needs to try to make some space.</p> A simple LRU check would represent much higher maintenance; updating the linked list on each buffer load would dramatically increase the complexity.</div>That's where the clock sweep algorithm comes in. Why clock? Imagine the buffer pool as a circular clock, and the algorithm always moves forward, sweeping along the way. As it passes each slot it Is the buffer pinned? Skips it as it's being used.</li> Is the usage counter > 0, reduce it by one and continue to next slot (i.e. it survives till next round).</li> If the usage counter hits 0 and it's not in use, it's going to be evicted.</li> </ol> The clock sweep is a simple way to ensure cold pages get evicted quickly, while hot pages tend to stick and survive multiple sweeps.</p> Dirty buffers and the background writer</a> </h2> Don't forget that a change is always first written to WAL.</div>Up until now we have discussed that a buffer loaded from disk is the same one in the shared buffers cache. When a backend modifies the page, the buffer becomes "dirty" but it's not immediately written to storage. Dirty buffers represent I/O work that has not yet happened. As it would be inefficient to write it immediately. The same page can be modified multiple times in short bursts, making the previous I/O operations unnecessary. Instead dirty pages accumulate until one of the following events happen.</p> PostgreSQL writes all dirty buffers to disk during checkpoint</strong>. This is a periodic process (which you can force using the CHECKPOINT</code> command). This is a point where data on disk becomes consistent. After its successful completion, crash recovery only needs to replay write-ahead logs from that moment forward.</p> The second mechanism is the background writer</strong>. It continuously scans for dirty buffers and writes them before anyone needs to evict them. This way, when a backend runs clock sweep, it finds clean buffers ready to evict, without needing to wait for I/O. It also spreads writes over time instead of bursty spikes during eviction pressure.</p> And as suggested the last chance to write the dirty pages to the disk is when the clock sweep finds a dirty buffer. It must write the data to disk so it can be reused for the new page. This is the worst case as it resorts to synchronous I/O operation.</p> The ultimate goal is to balance the clean buffers available, and therefore preventing backends from being blocked by synchronous write during the eviction. If you followed carefully you can see how bad flow or settings can cause a problem. Eviction happens when new buffers are loaded (I/O operation), and if the dirty buffer is evicted you force another I/O operation.</p> Ring buffers</a> </h2> As mentioned above there are also special types of buffers. We already touched the scenario when a query performs the scans of a large table.</p> In a naive implementation, the sequential scan would load all data into shared buffers, evicting everything else along the way. Your warmed cache would vanish and subsequent queries would suffer for minutes to come.</p> PostgreSQL's solution for this case is ring buffers</strong>. Small, private buffer pools for bulk operations. Instead of using the shared buffer pool, certain operations get their own limited ring.</p> The individual cases are:</p> Sequential scans on large tables</strong> over 1/4 of shared_buffers</code> use a dedicated 256 KB ring buffer. Pages cycle through this tiny ring and never touch the main cache.</p> -- perform a sequential scan over a large table</span></span> EXPLAIN (ANALYZE, BUFFERS) </span>SELECT</span> count</span>(</span>1</span>) </span>FROM</span> ring_buffer_test;</span></span></code></pre> Which would show you Buffers: shared read=127285</code> instead of hit</strong>. I will follow with the separate article on how to read buffers in explain plans.</p> Bulk writes</strong> (COPY, CREATE TABLE AS) use a 16MB capped ring buffer, large enough for efficient batching, yet small enough not to pollute the shared buffer pool.</p> As VACUUM</strong> touches every page and shouldn't evict the hot data, it uses its own dedicated ring buffer. Historically it was set to 256KB, but since PostgreSQL 17 it can be configured using vacuum_buffer_usage_limit</code> (while previous two are given).</p> Local buffers</a> </h2> The second kind of exception to shared buffer pool, is the session based temporary tables</strong>. In this case as the concurrency is out of question, each backend has its own local buffer pool, controlled by temp_buffers</code> (8MB default).</p> Local buffers are faster than shared buffers because they have simpler locking. There is no need for the heavy cross-process coordination required in the main cache.</p> While this might seem like a minor implementation detail, it can translate into a powerful optimisation strategy. Many developers tend to default to complex CTE logic</a> for intermediate data, but using temporary tables offers a distinct advantage in the form of lower I/O</strong> as the changes to temporary tables are not WAL logged and by their nature reduce pollution of shared buffer pools.</p> If your workload involves large temporary tables, increasing temp_buffers can help keep those operations purely in RAM. However, remember that this memory is per-connection, so it multiplies across all backends.</p> The OS Page Cache</a> </h2> PostgreSQL doesn't bypass the operating system. Every read and write goes through the kernel, which maintains its own page cache. This creates double buffering</strong> - the same 8KB page can exist in both PostgreSQL's shared buffers and the OS cache simultaneously.</p> The OS cache acts as a "Level 2 cache" - when PostgreSQL evicts a page, it often still lives in OS memory.</div> This sounds wasteful, but it's actually a feature. When PostgreSQL evicts a clean page to make room, that page drops into the OS cache rather than disappearing. A moment later, if you need it back, the OS serves it from RAM - no disk I/O required.</p> The OS also provides read-ahead</strong> - detecting sequential access patterns and pre-loading pages before PostgreSQL requests them.</p> This relationship explains the classic advice: set shared_buffers</code> to 25% of RAM. You're deliberately leaving space for the OS to act as a safety net.</p> -- on dedicated servers with large RAM, 40% can work well</span></span> -- but always leave room for OS cache and other processes</span></span> ALTER SYSTEM SET</span> shared_buffers </span>=</span> '8GB'</span>;</span></span></code></pre>This parameter allocates no memory - it's purely a hint for cost estimation.</div> PostgreSQL needs to know about this combined cache for query planning. That's where effective_cache_size</code> comes in - it tells the planner how much total cache (shared buffers + OS) to assume when estimating costs.</p> -- estimate of total cache available (shared + OS)</span></span> SHOW effective_cache_size;</span></span></code></pre> A higher value encourages the planner to favor index scans, assuming data is likely cached somewhere even if not in shared buffers.</p> Conclusion</a> </h2> Buffers are one of the keystones of PostgreSQL internals. They control whether the query will hit fast RAM or slow disk; and at the same time they play a crucial role in the fragile balance of dirty pages, WAL and durability.</p> The shared buffer pool isn't just a cache - it's a sophisticated memory manager with clock sweep eviction, usage count decay, ring buffer isolation, and background maintenance. Understanding these mechanisms helps you tune effectively and diagnose problems when they start.</p> The hidden cost of PostgreSQL arrays 2026-01-12T20:50:00+00:00 Starting with arrays in PostgreSQL is as simple as declaring a column as integer[]</code>, inserting some values, and you are done.</p> Or building the array on the fly.</p> SELECT</span> '{1,2,3}'</span>::</span>int</span>[];</span></span> SELECT array</span>[1,2,3];</span></span></code></pre> int4</span></span> ---------</span></span> {1,2,3}</span></span> (1 row)</span></span> </span> array</span></span> ---------</span></span> {1,2,3}</span></span> (1 row)</span></span></code></pre> The official documentation</a> provides a good introduction. But beneath this straightforward interface lies a set of more complex properties than most of us realise. Arrays in PostgreSQL are not just "lists" in a field. They have their own memory management strategy, their own index logic, and a lot of edge-case scenarios.</p> As it goes with boringSQL</strong> deep-dives, this article will explore the corners of array functionality that might break your production.</p> The document model temptation</a> </h2> Wait? Are we going to talk about JSONB arrays? Not at all. The whole concept of arrays in RDBMSs is actually document storage in disguise</strong>.</p> In database design, locality ensures faster retrieval times by keeping related data close on physical storage.</span>Whether you use a distinct integer[]</code> type or a JSON list [1, 2, 3]</code>, you are making the exact same architectural decision: you are prioritising locality over normalisation</strong>.</p> When you store tag_ids</code> in an array, you are embedding related data directly into a row - just like a NoSQL database might embed subdocuments. This is not inherently wrong. Document databases exist for good reasons: they eliminate joins, simplify reads, and map naturally to application objects.</p> But PostgreSQL is a relational database. It was designed around the relational model, where:</p> foreign keys</strong> enforce referential integrity</a></li> joins</strong> connect normalised tables</li> updates</strong> modify individual rows, not entire lists</li> </ul> Arrays give you document-model convenience, but you lose relational promises. There are no foreign keys and no ON DELETE referential_action</code> (like CASCADE) for array elements. If you delete a tags</code> entry, the orphaned ID will remain in your array forever.</p> The rule of thumb</strong> is that if you find yourself in need of referential integrity - you most likely want a link table, not an array. Arrays are for data that shares the same lifecycle as the parent row. Not for relationships spanning across different tables.</p> A practical example is the author of a blog post (one author might write multiple other posts), whereas a whitelist of IP addresses for a service account is only applicable to the given entity.</p> With JSONB being so flexible, you might wonder why we still bother with quirkier native types. The answer lies in the 'boring' part of the database: predictability and efficiency. An integer[]</code> column guarantees that every element is an integer — similar to how enums</a> enforce a fixed set of allowed values.</p> Arrays are also more storage-efficient for primitives because they don't carry the metadata overhead of JSON objects.</p> The syntax gotchas</a> </h2> This post assumes basic knowledge of arrays. We won't cover the basics.</p> Arrays don't have to start at 1</a> </h3> By default, SQL arrays start at 1. And seemingly, there is nothing wrong with iterating through them in a fashion similar to:</p> FOR</span> i </span>IN</span> 1</span> .. array_length(fruits, </span>1</span>) </span>LOOP</span></span> RAISE NOTICE </span>'Index % contains: %'</span>, i, fruits[i];</span></span> END LOOP</span>;</span></span></code></pre> that is until you find an array with the arbitrary bounds. Which PostgreSQL allows.</p> SELECT</span> '[-5:-3]={10,20,30}'</span>::</span>int</span>[];</span></span></code></pre> To make sure you iterate correctly through any given array, always use array_lower()</code> and array_upper()</code> in PL/pgSQL</p> SELECT</span> array_lower(</span>'[-5:-3]={10,20,30}'</span>::</span>int</span>[], </span>1</span>);</span></span></code></pre> array_lower</span></span> -------------</span></span> -5</span></span> (1 row)</span></span></code></pre> or generate_subscripts()</code> in SQL.</p> SELECT</span> generate_subscripts(</span>'[-5:-3]={10,20,30}'</span>::</span>int</span>[], </span>1</span>);</span></span></code></pre> generate_subscripts</span></span> ---------------------</span></span> -5</span></span> -4</span></span> -3</span></span> (3 rows)</span></span></code></pre>Missing dimensions</a> </h3> When creating a table, you might expect strict typing. That's true for everything — except the array dimensions. You might think integer[][]</code> enforces a 2D matrix. Except it does not. The []</code> syntax is effectively syntactic sugar. PostgreSQL does not enforce the number of dimensions of sub-arrays at the schema level at all by default.</p> CREATE TABLE</span> dimension_test</span> (</span></span> matrix </span>integer</span>[][] </span></span> );</span></span> </span> INSERT INTO</span> dimension_test </span>VALUES</span> (</span>'{{1,2}, {3,4}}'</span>);</span></span> </span> -- this is not going to fail</span></span> INSERT INTO</span> dimension_test </span>VALUES</span> (</span>'{1,2,3}'</span>);</span></span> </span> -- 3D matrix works too</span></span> INSERT INTO</span> dimension_test </span>VALUES</span> (</span>'{{{1,2},{3,4}}, {{5,6},{7,8}}}'</span>);</span></span></code></pre> If you want to enforce a specific array dimension, you cannot rely on the type definition. Instead, you must use a CHECK</code> constraint.</p> CREATE TABLE</span> strict_matrix</span> (</span></span> -- dims: ensure it's 2-Dimensional</span></span> -- array_length: make sure it's exactly 3x3</span></span> board </span>integer</span>[]</span> CHECK</span> (array_ndims(board) </span>=</span> 2</span> AND</span> array_length(board, </span>1</span>) </span>=</span> 3</span>)</span></span> );</span></span></code></pre>INSERT INTO strict_matrix VALUES (</span></span> ARRAY[</span></span> [0, 1, 0],</span></span> [1, 1, 0],</span></span> [0, 0, 1]</span></span> ]</span></span> );</span></span></code></pre> The only exception is that PostgreSQL enforces uniformity of arrays on every nesting level. This means it rejects sub-arrays with different sizes.</p> INSERT INTO</span> dimension_test </span>VALUES</span> (</span>'{{1,2}, {3}}'</span>);</span></span> ERROR: malformed </span>array</span> literal: </span>"{{1,2}, {3}}"</span></span> LINE</span> 1</span>: </span>INSERT INTO</span> dimension_test </span>VALUES</span> (</span>'{{1,2}, {3}}'</span>);</span></span> ^</span></span> DETAIL: Multidimensional arrays must have sub</span>-</span>arrays </span>with</span> matching dimensions.</span></span></code></pre>Slicing arrays</a> </h3> When accessing array values, it's important to consider that the syntax [1]</code> and [1:1]</code> are different. While the first one is an accessor, the second one acts like a constructor.</p> select</span> matrix[1][1]</span> from</span> dimension_test ;</span></span> matrix</span></span> --------</span></span> 1</span></span> (</span>1</span> row</span>)</span></span></code></pre> When slicing an array, even if the slice is a single-element, it will be returned as a single-element array, not a scalar value.</p> select</span> matrix[1:1][1:1], matrix[1][1:1], matrix[1:1][1]</span> from</span> dimension_test ;</span></span> matrix \| matrix \| matrix</span></span> --------+--------+--------</span></span> {{</span>1</span>}} \| {{</span>1</span>}} \| {{</span>1</span>}}</span></span> (</span>1</span> row</span>)</span></span></code></pre>The ugly</a> </h3> Accessing array values has a forgiving behaviour, making it more difficult to find underlying bugs:</p> -- out-of-bound access returns NULL</span></span> SELECT</span> (</span>ARRAY</span>[1,2,3])[10];</span></span> array</span></span> --------</span></span> (</span>null</span>)</span></span> (</span>1</span> row</span>)</span></span></code></pre>-- out of bounds slicing returns an empty array</span></span> SELECT</span> (</span>ARRAY</span>[1,2,3])[5:10];</span></span> array</span></span> -------</span></span> {}</span></span> (</span>1</span> row</span>)</span></span></code></pre> But the single most confusing aspect that might trip you up coming from other programming languages is the fact that PostgreSQL treats multi-dimensional arrays as a single matrix, not an array of arrays.</p> -- wrong dimensionality </span></span> SELECT</span> (</span>ARRAY</span>[[1,2],[3,4]])[1];</span></span> array</span></span> --------</span></span> (</span>null</span>)</span></span> (</span>1</span> row</span>) </span></span></code></pre> While in other languages you expect {1,2}</code> as a result, PostgreSQL can't give it to you. It tries to return the first cell and fails (because the index is not complete).</p> And to paraphrase Fletcher's "Double Bubble" analogy (which also went very wrong — extra points for getting the reference), you can't fix this by using slice notation.</p> select</span> (</span>'{{1,2},{3,4}}'</span>::</span>int</span>[])[1:1];</span></span> int4</span></span> ---------</span></span> {{</span>1</span>,</span>2</span>}}</span></span> (</span>1</span> row</span>)</span></span></code></pre> The only way to solve this puzzle is to unnest</code> the slice and re-aggregate the results.</p> SELECT</span> array_agg(val) </span>FROM</span> unnest((</span>'{{1,2},{3,4}}'</span>::</span>int</span>[])[1:1]) val;</span></span> array_agg</span></span> -----------</span></span> {</span>1</span>,</span>2</span>}</span></span> (</span>1</span> row</span>)</span></span></code></pre> Just be aware that `array_agg` does not guarantee the order of aggregated elements unless you use an `ORDER BY` clause. While it usually works with simple `unnest` queries, relying on implicit ordering can be risky. </div> The other alternative is to cast it to JSONB and back. Not pretty no matter how you look at it. If you need to work with complex multi-dimensional structures where each sub-array has independent meaning, just use JSONB. It will do exactly what you expect.</p> Indexing arrays</a> </h2> While you can use a B-tree index on an array column, it won't help you unless you are looking for whole-array equality and sorting an array by dictionary rules — and even on regular columns, PostgreSQL can ignore your indexes</a> when it decides a sequential scan is cheaper.</p> -- which is bigger? B is correct</span></span> -- A: {1, 1000, 1000}</span></span> -- B: {2, 0}</span></span></code></pre> Rendering B-tree indexes useless for any real-world index operations.</p> When working with arrays, you actually need GIN</strong> (Generalized Inverted Index). If a B-tree index is a phone book, GIN is an index at the back of the book. To query one or more elements, you need to find all possible locations and then intersect them to find the locations that match.</p> CREATE INDEX</span> posts_by_tags</span> ON</span> posts </span>USING</span> GIN (tags);</span></span></code></pre> GIN indexes are designed for set operations, making presence the key feature, while ignoring order. Here are operators that GIN provides for arrays:</p> Containment</strong> @></code> - matches rows that include ALL of the selected items.</p> -- match ALL tags</span></span> tags @</span>></span> '{urgent, bug}'</span></span></code></pre> Overlap</strong> &&</code> - does the row include ANY of the selected items.</p> -- match bug or feature</span></span> tags && </span>'{bug, feature}'</span></span></code></pre> There's also <@</code> and =</code> (equality), but they should be easy to understand.</p> Two sides of ANY</code></a> </h3> This is where it gets interesting. While you might often hear (myself included) that you shouldn't use dynamic SQL for IN</code> lists and should use ANY</code> instead, there are some dangers you need to be aware of.</p> The ANY</code> operator behaves very differently depending on which side of the comparison the array sits.</p> The advice to use ANY</code> holds true when you are passing lists into the database</strong>. Instead of generating a query with 100 distinct parameters (WHERE id IN ($1, $2, ... $100)</code>), which bloats your query cache and forces hard-parses, you should pass a single array parameter.</p> -- good: one parameter, one query plan</span></span> SELECT * FROM</span> users </span>WHERE</span> id </span>=</span> ANY($</span>1</span>::</span>int</span>[]);</span></span></code></pre> The trap is assuming this syntax is equally efficient when querying array columns. It is not. If you use ANY</code> to check if a value exists inside a table column, you are effectively asking the database to loop.</p> -- bad: GIN does not support ANY; turning this into a seq scan</span></span> SELECT * FROM</span> tickets </span>WHERE</span> 'feature'</span> =</span> ANY(tags);</span></span></code></pre> When you write WHERE 'feature' = ANY(tags)</code>, you are not actually using an array operator. What you wrote is a scalar integer equality operator =</code> applied inside a loop construct. Since the scalar operator =</code> is not part of array_ops</code>, the planner assumes the index cannot help and falls back to a sequential scan.</p> The correct way to rewrite the query is:</p> -- good again</span></span> SELECT * FROM</span> tickets </span>WHERE</span> tags @</span>> ARRAY</span>['feature'];</span></span></code></pre>Fast updates and the trade-off</a> </h3> Because a GIN index is built to work with sets, it is expensive to maintain. With a B-tree index, one row equals one index entry. In a GIN index, one row equals N index entries, where N is the number of elements in your array.</p> This leads to write multiplication. To prevent this, PostgreSQL defaults to using a "fast update" mechanism. This is a strategy where new entries are added to a pending list (an unsorted temporary buffer) and only merged into the main index structure later (during VACUUM).</p> Given the unsorted nature of a GIN index, it is an exception to VACUUM is a Lie</a>, as VACUUM actually does perform structural maintenance here.</span>While this makes INSERT operations manageable, it can slow down SELECTs. Every time you query the index, PostgreSQL must scan the organised main index plus the entire messy pending list. If that list grows large, your query performance might degrade.</p> If you operate a read-heavy workflow with infrequent writes, you should disable this to guarantee consistent reading performance.</p> CREATE INDEX</span> posts_by_tags</span> ON</span> posts </span>USING</span> GIN (tags) </span>WITH</span> (fastupdate </span>= off</span>);</span></span></code></pre>Storage and modification</a> </h2> Now it's time to return to the document model</a>. In PostgreSQL, rows are immutable (MVCC); there is no such thing as an "in-place update", and arrays are stored as atomic values. This results in a very uncomfortable truth — to modify a single element of an array, PostgreSQL must copy and rewrite the entire row</strong>.</p> UPDATE</span> user_activity</span></span> SET</span> event_ids </span>=</span> event_ids </span>\|\|</span> 10001</span></span> WHERE</span> user_id </span>=</span> 50</span>;</span></span></code></pre> Every single append rewrites the entire array, which in effect results in a rewrite of the entire row.</p> TOASTed</a> </h3> When any array grows large enough (> 2 KB — see below), PostgreSQL moves it automatically to a separate storage area using TOAST</a>. While this keeps the data row lean, it turns array updates into a severe performance bottleneck.</p> The difference this introduces is subtle but comes with a big impact. While a standard MVCC update simply copies the row version on the main heap, updating a TOASTed array forces PostgreSQL to fetch all external chunks, decompress the entire object into memory, apply the change, and then recompress and write the new full-size blob back to the TOAST table. This turns a simple modification into a CPU and I/O-intensive operation that rewrites the entire dataset rather than just the delta.</p> The threshold is derived from TOAST_TUPLES_PER_PAGE</code> (default: 4), ensuring 4 tuples fit on a page.</br></br>Threshold: ~2 KB</span>Where does the 2 KB value come from? The TOAST threshold is calculated to ensure at least four tuples can fit on a single 8 KB heap page</a>. PostgreSQL uses this to balance efficiency against the overhead of TOAST indirection.</p> The compressions</a> </h3> Before version 14, PostgreSQL relied on pglz</code> — an algorithm prioritising compression ratio over speed. This made the "decompress-modify-compress" cycle of TOAST painful.</p> PostgreSQL 14 introduced LZ4 as an alternative:</p> ALTER TABLE</span> articles </span>ALTER</span> COLUMN tags </span>SET COMPRESSION</span> lz4;</span></span></code></pre> LZ4 is significantly faster for both compression and decompression, with only slightly lower compression ratios. If you are working with large arrays, switching to LZ4 is one of the easiest ways to reduce the CPU penalty of TOAST.</p> When a large array might make sense</a> </h3> You might have gained the impression that arrays are bad. When evaluating the use of arrays, the real question isn't how big the array is</strong> but rather how often do you modify it</strong>? An array of 10,000 elements that you write once and is read-only for the rest of its lifecycle is a completely valid use case. An array of 50 elements that you append to on every incoming request is the real villain here.</p> If you combine this with compression, you might get an interesting mix.</p> DROP TABLE IF EXISTS</span> compression_test;</span></span> CREATE TABLE</span> compression_test</span> (</span></span> id </span>serial PRIMARY KEY</span>,</span></span> compressed_floats float4[], </span></span> raw_floats float4[]</span></span> );</span></span> </span> -- do not compress raw_floats</span></span> ALTER TABLE</span> compression_test </span>ALTER</span> COLUMN raw_floats </span>SET</span> STORAGE </span>EXTERNAL</span>;</span></span> </span> -- insert semi-random data with low cardinality</span></span> INSERT INTO</span> compression_test (compressed_floats, raw_floats)</span></span> SELECT</span> semi_random_arr, semi_random_arr</span></span> FROM</span> (</span></span> SELECT ARRAY</span>(</span></span> SELECT</span> floor</span>(random()</span> </span> 50</span>)::float4 </span></span> FROM</span> generate_series</span>(</span>1</span>, </span>10000</span>)</span></span> ) </span>as</span> semi_random_arr</span></span> ) </span>as</span> generator;</span></span> </span> SELECT</span> </span></span> 'Compressed (EXTENDED)'</span> as</span> strategy,</span></span> pg_size_pretty(pg_column_size(compressed_floats)::</span>bigint</span>) </span>as</span> size_on_disk</span></span> FROM</span> compression_test</span></span> UNION ALL</span></span> SELECT</span> </span></span> 'Raw (EXTERNAL)'</span>,</span></span> pg_size_pretty(pg_column_size(raw_floats)::</span>bigint</span>)</span></span> FROM</span> compression_test;</span></span></code></pre> strategy \| size_on_disk</span></span> -----------------------+--------------</span></span> Compressed (EXTENDED) \| 15 kB</span></span> Raw (EXTERNAL) \| 39 kB</span></span> (2 rows) </span></span></code></pre>Bulk loading with arrays</a> </h2> Up until now, it might seem that arrays don't actually bring many benefits. While they can have rough edges around storage, they are incredibly useful for transport.</p> The fastest way to insert 5,000 rows isn't a loop in your application, and it's definitely not a massive VALUES (...), (...)</code> string. It's unnest</code>.</p> INSERT INTO</span> measurements (sensor_id, </span>value</span>, captured_at)</span></span> SELECT FROM</span> unnest(</span></span> $</span>1</span>::</span>int</span>[], </span>-- array of sensor IDs</span></span> $</span>2</span>::</span>float</span>[], </span>-- array of values</span></span> $</span>3</span>::</span>timestamptz</span>[]</span> -- array of timestamps</span></span> );</span></span></code></pre> All that is needed is one network round trip, and one query to parse and plan. PostgreSQL handles the arrays row by row internally for you. This works both for UPSERTs and MERGE</a>.</p> The cases for special arrays</a> </h2> Standard PostgreSQL arrays are polymorphic types (anyarray</code>). This provides a powerful feature that allows a single function definition to operate on many different data types. They have to handle integers, strings, timestamps, and custom types equally well. But if you have specific data types, you can unlock significant performance gains by using specialised extensions.</p> The intarray</code> extension</a> </h3> If you are dealing exclusively with 4-byte integers (int4</code>/integer</code>), the built-in array operations are leaving performance on the table. The intarray</a> extension provides specialised functions and index operators that are significantly faster than the generic implementation.</p> To use it, you must explicitly enable it:</p> CREATE</span> EXTENSION </span>IF NOT EXISTS</span> intarray;</span></span></code></pre> The difference in developer ergonomics is immediate. To sort an array in standard SQL, you are forced to unnest, order, and re-aggregate. With intarray, you get native functions like sort() and uniq().</p> -- standard arrays</span></span> SELECT</span> array_agg(val </span>ORDER BY</span> val)</span></span> FROM</span> unnest(</span>'{3, 1, 2}'</span>::</span>int</span>[]) val;</span></span> </span> -- intarray</span></span> SELECT</span> sort(</span>'{3, 1, 2}'</span>::</span>int</span>[]);</span></span></code></pre> Beyond raw management functions, it introduces a specialised query syntax that simplifies complex boolean logic. Instead of chaining multiple overlap (&&</code>) and containment (@></code>) checks, you can express your requirements in a single "query string" using the @@</code> operator.</p> -- standard arrays</span></span> SELECT * FROM</span> staff</span></span> WHERE</span> available_days @</span>></span> '{1}'</span> -- must include Mon</span></span> AND</span> available_days && </span>'{6, 7}'</span> -- must include Sat OR Sun</span></span> AND NOT</span> (available_days @</span>></span> '{2}'</span>); </span>-- must NOT include Tue</span></span> </span> -- intarray</span></span> SELECT * FROM</span> staff</span></span> WHERE</span> available_days @@ </span>'1 & (6 \| 7) & !2'</span>;</span></span></code></pre> The only catch is the type restriction. intarray</code> is strictly limited to signed 32-bit integers. If your values exceed 2 billion, you are back where you started.</p> AI with pgvector</code></a> </h3> You cannot talk about arrays in 2026 without mentioning pgvector</a>. While it markets itself as a "vector store", internally it is simply an array of floats with a different mathematical focus.</p> Standard arrays are binary</strong>: they care about Exact Matches (Overlap &&</code>, Containment @></code>). Vectors are all about fuzzy distance</strong> (Cosine <=></code>, Euclidean <-></code>).</p> If you are building search or recommendation features, pgvector allows you to treat your array column not as a list of "facts" (tag A, tag B), but as coordinates in semantic space.</p> Nevertheless, the architectural decision is exactly the same as using a standard array: you are trading strict structure for convenience. Since there is no way to "join" two rows based on how similar they are, you store the vector directly on the row. You accept a larger table size in exchange for the ability to ask, "What is close to this?".</p> PS: Thanks to Matthias Feist for inspiring this article.</p> Instant database clones with PostgreSQL 18 2025-12-22T23:53:16+00:00 Have you ever watched a long running migration script</a>, wondering if it's about to wreck your data? Or wish you can "just" spin a fresh copy of database for each test run? Or wanted to have reproducible snapshots to reset between runs of your test suite, (and yes, because you are reading boringSQL) needed to reset the learning environment?</p> When your database is a few megabytes, pg_dump</code> and restore works fine. But what happens when you're dealing with hundreds of megabytes/gigabytes - or more? Suddenly "just make a copy" becomes a burden.</p> You've probably noticed that PostgreSQL connects to template1</code> by default. What you might have missed is that there's a whole templating system hiding in plain sight. Every time you run</p> CREATE DATABASE dbname;</span></span></code></pre> PostgreSQL quietly clones standard system database template1</code> behind the scenes. Making it same as if you would use</p> CREATE DATABASE dbname TEMPLATE template1;</span></span></code></pre> The real power comes from the fact that you can replace template1</code> with any database. You can find more at Template Database documentation</a>.</p> In this article, we will cover a few tweaks that turn this templating system into an instant, zero-copy database cloning machine.</p> CREATE DATABASE ... STRATEGY</a> </h2> Before PostgreSQL 15, when you created a new database from a template, it operated strictly on the file level. This was effective, but to make it reliable, Postgres had to flush all pending operations to disk (using CHECKPOINT</code>) before taking a consistent snapshot. This created a massive I/O spike - a "Checkpoint Storm" - that could stall your production traffic.</p> Version 15 of PostgreSQL introduced new parameter CREATE DATABASE ... STRATEGY = [strategy]</code> and at the same time changed the default behaviour how the new databases are created from templates. The new default become WAL_LOG</code> which copies block-by-block via the Write-Ahead Log (WAL), making I/O sequential (and much smoother) — operations that also show up in EXPLAIN buffer statistics</a> — and support for concurrency without facing latency spike. This prevented the need to CHECKPOINT but made the database cloning operation potentially significantly slower. For an empty template1</code>, you won't notice the difference. But if you try to clone a 500GB database using WAL_LOG, you are going to be waiting a long time.</p> The STRATEGY</code> parameter allows us to switch back to the original method FILE_COPY</code> to keep the behaviour, and speed. And since PostgreSQL 18, this opens the whole new set of options.</p> FILE_COPY</a> </h2> Because the FILE_COPY</code> strategy is a proxy to operating system file operations, we can change how the OS handles those files.</p> When using standard file system (like ext4</code>), PostgreSQL reads every byte of the source file and writes it to a new location. It's a physical copy. However starting with PostgreSQL 18 - file_copy_method</code> gives you options to switch that logic; while default option remains copy</code>.</p> With modern filesystems (like ZFS, XFS with reflinks, APFS, etc.) you can switch it to clone</code> and leverage CLONE</code> (FICLONE</code> on Linux) operation for almost instant operation. And it won't take any additional space.</p> Quick setup checklist:</strong></p> Linux with XFS or ZFS, macOS with APFS, or FreeBSD with ZFS</li> PostgreSQL 18+ cluster on that filesystem</li> Set file_copy_method = clone</code> in your config</li> Reload configuration</li> </ul> </div> The benchmark</a> </h2> We need some dummy data to copy. This is the only part of the tutorial where you have to wait. Let's generate a ~6GB database.</p> CREATE DATABASE</span> source_db</span>;</span></span> \c source_db</span></span> </span> CREATE TABLE</span> boring_data</span> (</span></span> id </span>serial PRIMARY KEY</span>,</span></span> payload </span>text</span></span> );</span></span> </span> -- generate 50m rows</span></span> INSERT INTO</span> boring_data (payload)</span></span> SELECT</span> md5(random()::</span>text</span>) </span>\|\|</span> md5(random()::</span>text</span>)</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>50000000</span>);</span></span> </span> -- force a checkpoint</span></span> CHECKPOINT</span>;</span></span></code></pre> You can verify the database now has roughly 6GB of data.</p> Name \| source_db</span></span> Owner \| postgres</span></span> Encoding \| UTF8</span></span> Locale Provider \| libc</span></span> Collate \| en_US.UTF-8</span></span> Ctype \| en_US.UTF-8</span></span> Locale \|</span></span> ICU Rules \|</span></span> Access privileges \|</span></span> Size \| 6289 MB</span></span> Tablespace \| pg_default</span></span> Description \|</span></span></code></pre> While enabling \timing</code> you can test the default (WAL_LOG) strategy. And on my test volume (relatively slow storage) I get</p> CREATE DATABASE slow_copy TEMPLATE source_db;</span></span> CREATE DATABASE</span></span> Time: 67000.615 ms (01:07.001)</span></span></code></pre> Now, let's verify our configuration is set for speed:</p> show file_copy_method;</span></span> file_copy_method</span></span> ------------------</span></span> clone</span></span> (1 row)</span></span></code></pre> Let's request the semi-instant clone of the same database, without taking extra disk space at the same time.</p> CREATE DATABASE fast_clone TEMPLATE source_db STRATEGY=FILE_COPY;</span></span> CREATE DATABASE</span></span> Time: 212.053 ms</span></span></code></pre> That's a quite an improvement, isn't it?</p> Working with cloned data</a> </h2> That was the simple part. But what is happening behind the scenes?</p> When you clone a database with file_copy_method = clone</code>, PostgreSQL doesn't duplicate any data. The filesystem creates new metadata entries that point to the same physical 8KB pages</a>. Both databases share identical storage.</p> This can create some initial confusion. If you ask PostgreSQL for the size:</p> SELECT</span> pg_database_size(</span>'source_db'</span>) </span>as</span> source,</span></span> pg_database_size(</span>'fast_clone'</span>) </span>as</span> clone;</span></span></code></pre> PostgreSQL reports both as ~6GB because that's the logical size - how much data each database "contains" - i.e. logical size.</p> -[ RECORD 1 ]------</span></span> source \| 6594041535</span></span> clone \| 6594041535</span></span></code></pre> The interesting part happens when you start writing. PostgreSQL doesn't update tuples in place. When you UPDATE a row, it writes a new tuple version somewhere (often a different page entirely) and marks the old one as dead. The filesystem doesn't care about PostgreSQL internals - it just sees writes to 8KB pages</a>. Any write to a shared page triggers a copy of that entire page.</p> A single UPDATE will therefore trigger copy-on-write on multiple pages:</p> the page holding the old tuple</li> the page receiving the new tuple</li> index pages if any indexed columns changed</li> FSM and visibility map pages as PostgreSQL tracks free space</li> </ul> And later, VACUUM touches even more pages while cleaning up dead tuples</a>. In this case diverging quickly from the linked storage.</p> XFS proof</a> </h2> Using the database OID and relfilenode we can verify the both databases are now sharing physical blocks.</p> root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16402/16404</span></span> Filesystem type is: 58465342</span></span> File size of /var/lib/postgresql/18/main/base/16402/16404 is 1073741824 (262144 blocks of 4096 bytes)</span></span> ext: logical_offset: physical_offset: length: expected: flags:</span></span> 0: 0.. 2031: 10471550.. 10473581: 2032: shared</span></span> 1: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared</span></span> 2: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared</span></span> 3: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared</span></span> 4: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared</span></span> 5: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared</span></span> 6: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof</span></span> /var/lib/postgresql/18/main/base/16402/16404: 7 extents found</span></span> root@clone-demo:/var/lib/postgresql#</span></span> root@clone-demo:/var/lib/postgresql#</span></span> root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16418/16404</span></span> Filesystem type is: 58465342</span></span> File size of /var/lib/postgresql/18/main/base/16418/16404 is 1073741824 (262144 blocks of 4096 bytes)</span></span> ext: logical_offset: physical_offset: length: expected: flags:</span></span> 0: 0.. 2031: 10471550.. 10473581: 2032: shared</span></span> 1: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared</span></span> 2: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared</span></span> 3: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared</span></span> 4: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared</span></span> 5: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared</span></span> 6: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof</span></span> /var/lib/postgresql/18/main/base/16418/16404: 7 extents found</span></span></code></pre> All it takes is to update some rows using</p> update</span> boring_data </span>set</span> payload </span>=</span> 'new value'</span> \|\|</span> id </span>where</span> id </span>IN</span> (</span>select</span> id </span>from</span> boring_data </span>limit</span> 20</span>);</span></span></code></pre> and the situation will start to change.</p> root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16402/16404</span></span> Filesystem type is: 58465342</span></span> File size of /var/lib/postgresql/18/main/base/16402/16404 is 1073741824 (262144 blocks of 4096 bytes)</span></span> ext: logical_offset: physical_offset: length: expected: flags:</span></span> 0: 0.. 39: 10471550.. 10471589: 40:</span></span> 1: 40.. 2031: 10471590.. 10473581: 1992: shared</span></span> 2: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared</span></span> 3: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared</span></span> 4: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared</span></span> 5: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared</span></span> 6: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared</span></span> 7: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof</span></span> /var/lib/postgresql/18/main/base/16402/16404: 7 extents found</span></span> root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16418/16404</span></span> Filesystem type is: 58465342</span></span> File size of /var/lib/postgresql/18/main/base/16418/16404 is 1073741824 (262144 blocks of 4096 bytes)</span></span> ext: logical_offset: physical_offset: length: expected: flags:</span></span> 0: 0.. 39: 10297326.. 10297365: 40:</span></span> 1: 40.. 2031: 10471590.. 10473581: 1992: 10297366: shared</span></span> 2: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared</span></span> 3: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared</span></span> 4: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared</span></span> 5: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared</span></span> 6: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared</span></span> 7: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof</span></span> /var/lib/postgresql/18/main/base/16418/16404: 8 extents found</span></span> root@clone-demo:/var/lib/postgresql#</span></span></code></pre> In this case extent 0 no longer has shared flag, first 40 blocks size (with default size 4KB) now diverge, making it total of 160KB. Each database now has its own copy at different physical address. The remaining extents are still shared.</p> Things to be aware of</a> </h2> Cloning is tempting but there's one serious limitation you need to be aware if you ever attempt to do it in production. The source database can't have any active connections during cloning. This is a PostgreSQL limitation, not a filesystem one. For production use, this usually means you create a dedicated template database rather than cloning your live database directly. Or given the relatively short time the operation takes you have to schedule the cloning in times where you can temporary block/terminate all connections.</p> Other limitation is that the cloning only works within a single filesystem. If your databases spans multiple table spaces on different mount points, cloning will fall back to regular physical copy.</p> Finally, in most managed cloud environments (AWS RDS, Google Cloud SQL), you will not have access to the underlying filesystem to configure this. You are stuck with their proprietary (and often billed) functionality. But for your own VMs or bare metal? Go ahead and try it.</p> VACUUM Is a Lie (About Your Indexes) 2025-12-11T23:34:00+00:00 There is common misconception that troubles most developers using PostgreSQL: tune VACUUM or run VACUUM, and your database will stay healthy. Dead tuples will get cleaned up. Transaction IDs recycled. Space reclaimed. Your database will live happily ever after.</p> But there are couple of dirty "secrets" people are not aware of. First of them being VACUUM is lying to you about your indexes</strong>.</p> The anatomy of storage</a> </h2> When you delete a row in PostgreSQL, it is just marked as a 'dead tuple'. Invisible for new transactions but still physically present. Only when all transactions referencing the row are finished, VACUUM can come along and actually remove them - reclamining the space in the heap (table) space.</p> To understand why this matters differently for tables versus indexes, you need to picture how PostgreSQL actually stores your data.</p> Your table data lives in the heap - a collection of 8 KB pages where rows are stored wherever they fit. There's no inherent order. When you INSERT a row, PostgreSQL finds a page with enough free space and slots the row in. Delete a row, and there's a gap. Insert another, and it might fill that gap - or not - they might fit somewhere else entirely.</p> This is why SELECT * FROM users</code> without an ORDER BY can return rows in order initially, and after some updates in seemingly random order, and that order can change over time. The heap is like Tetris. Rows drop into whatever space is available, leaving gaps when deleted.</p> </p> When VACUUM runs, it removes those dead tuples and compacts the remaining rows within each page. If an entire page becomes empty, PostgreSQL can reclaim it entirely.</p> And while indexes are on surface the same collection of 8KB pages, they are different. A B-tree index must maintain sorted order - that's the whole point of their existence and the reason why WHERE id = 12345</code> is so fast — though even indexes can be ignored</a> under certain conditions. PostgreSQL can binary-search down the tree instead of scanning every possible row. You can learn more about the fundamentals of B-Tree Indexes and what makes them fast</a>.</p> But if the design of the indexes is what makes them fast, it's also their biggest responsibility. While PostgreSQL can fit rows into whatever space is available, it can't move the entries in index pages to fit as much as possible.</p> </p> VACUUM can remove dead index entries. But it doesn't restructure the B-tree. When VACUUM processes the heap, it can compact rows within a page and reclaim empty pages. The heap has no ordering constraint - rows can be anywhere. But B-tree pages? They're locked into a structure. VACUUM can remove dead index entries, yes.</p> Many developers assume VACUUM treats all pages same. No matter whether they are heap or index pages. VACUUM is supposed to remove the dead entries, right?</p> Yes. But here's what it doesn't do - it doesn't restructure the B-tree</strong>.</p> What VACUUM actually does</strong></p> Removes dead tuple pointers from index pages</li> Marks completely empty pages as reusable</li> Updates the free space map</li> </ul> What VACUUM cannot do</strong>:</p> Merge sparse pages together (can do it for empty pages)</li> Reduce tree depth</li> Deallocate empty-but-still-linked pages</li> Change the physical structure of the B-tree</li> </ul> Your heap is Tetris, gaps can get filled. Your B-tree is a sorted bookshelf. VACUUM can pull books out, but can't slide the remaining ones together. You're left walking past empty slots every time you scan.</p> The experiment</a> </h2> Let's get hands-on and create a table, fill it, delete most of it and watch what happens.</p> CREATE</span> EXTENSION </span>IF NOT EXISTS</span> pgstattuple;</span></span> CREATE TABLE</span> demo</span> (id </span>integer PRIMARY KEY</span>, </span>data text</span>);</span></span> </span> -- insert 100,000 rows</span></span> INSERT INTO</span> demo (id, </span>data</span>)</span></span> SELECT</span> g, </span>'Row number '</span> \|\|</span> g </span>\|\|</span> ' with some extra data'</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>100000</span>) g;</span></span> </span> ANALYZE demo;</span></span></code></pre> At this point, our index is healthy. Let's capture the baseline:</p> SELECT</span></span> relname,</span></span> pg_size_pretty(pg_relation_size(</span>oid</span>)) </span>as</span> file_size,</span></span> pg_size_pretty((pgstattuple(</span>oid</span>)).tuple_len) </span>as</span> actual_data</span></span> FROM</span> pg_class</span></span> WHERE</span> relname </span>IN</span> (</span>'demo'</span>, </span>'demo_pkey'</span>);</span></span></code></pre>relname \| file_size \| actual_data</span></span> -----------+-----------+-------------</span></span> demo \| 7472 kB \| 6434 kB</span></span> demo_pkey \| 2208 kB \| 1563 kB</span></span></code></pre> Now remove some data, 80% to be precise - somewhere in the middle:</p> DELETE FROM</span> demo </span>WHERE</span> id </span>BETWEEN</span> 10001</span> AND</span> 90000</span>;</span></span></code></pre> The goal is to simulate a common real-world pattern: data retention policies, bulk cleanup operations, or the aftermath of a data migration gone wrong</a>.</p> VACUUM demo;</span></span> </span> SELECT</span></span> relname,</span></span> pg_size_pretty(pg_relation_size(oid)) as file_size,</span></span> pg_size_pretty((pgstattuple(oid)).tuple_len) as actual_data</span></span> FROM pg_class</span></span> WHERE relname IN ('demo', 'demo_pkey');</span></span></code></pre>relname \| file_size \| actual_data</span></span> -----------+-----------+-------------</span></span> demo \| 7472 kB \| 1278 kB</span></span> demo_pkey \| 2208 kB \| 313 kB</span></span></code></pre> The table shrunk significantly, while index remained unchanged You now have 20,000 rows indexed by a structure built to handle 100,000.</p> But notice something important: even though actual_data</code> dropped to ~1.3 MB (reflecting our 20,000 remaining rows), the file_size</code> for the index is still 2208 kB. VACUUM cleaned out the dead tuple data, but it doesn't return space to the OS or compact index structures - it only marks pages as reusable within PostgreSQL.</p> EDIT: the original table had a size without running VACCUM, which I must left out during what's sometime cumbersome process of authoring blog posts. Big thanks come to Creston Jamison of Scaling PostgreSQL</a> to notice it, and actually take what must have been a lot of time to analyze my posts.</p> </blockquote> This experiment is really an extreme case, but demonstrates the problem.</p> Understanding page states</a> </h2> Leaf pages have several states:</p> Full page (>80% density)</strong>, when the page contains many index entries, efficiently utilizing space. Each 8KB page</a> read returns substantial useful data. This is optimal state.</p> Partial page (40-80% density)</strong> with some wasted space, but still reasonably efficient. Common at tree edges or after light churn. Nothing to be worried about.</p> Sparse page (<40% density)</strong> is mostly empty. You're reading an 8KB page to find a handful of entries. The I/O cost is the same as a full page, but you get far less value.</p> Empty page (0% density)</strong> with zero live entries, but the page still exists in the tree structure. Pure overhead. You might read this page during a range scan and find absolutely nothing useful.</p> A note on fillfactor</a> </h3> You might be wondering how can fillfactor help with this? It's the setting you can apply both for heap and leaf pages, and controls how full PostgreSQL packs the pages during the data storage. The default value for B-tree indexes is 90%</strong>. This leaves 10% of free space on each leaf page for future insertions.</p> CREATE INDEX</span> demo_index</span> ON</span> demo(id) </span>WITH</span> (fillfactor </span>=</span> 70</span>);</span></span></code></pre> A lower fillfactor (like 70%) leaves more room, which can reduce page splits when you're inserting into the middle of an index - useful for tables random index column inserts or those with heavily updated index columns.</p> But if you followed carefully the anatomy of storage section, it doesn't help with the bloat problem. Quite the opposite. If you set lower fillfactor and then delete majority of your rows, you actually start with more pages, and bigger chance to end up with more sparse pages than partial pages.</p> Leaf page fillfactor is about optimizing for updates and inserts. It's not a solution for deletion or index-column update bloat.</p> Why the planner gets fooled</a> </h2> PostgreSQL's query planner estimates costs based on physical statistics</a>, including the number of pages in an index.</p> EXPLAIN ANALYZE </span>SELECT * FROM</span> demo </span>WHERE</span> id </span>BETWEEN</span> 10001</span> AND</span> 90000</span>;</span></span></code></pre>QUERY PLAN</span></span> --------------------------------------------------------------------------------------------------------------------</span></span> Index Scan using demo_pkey on demo (cost=0.29..29.29 rows=200 width=41) (actual time=0.111..0.112 rows=0 loops=1)</span></span> Index Cond: ((id >= 10001) AND (id <= 90000))</span></span> Planning Time: 1.701 ms</span></span> Execution Time: 0.240 ms</span></span> (4 rows)</span></span></code></pre> While the execution is almost instant, you need to look behind the scenes. The planner estimated 200 rows and got zero. It traversed the B-tree structure expecting data that doesn't exist. On a single query with warm cache, this is trivial. Under production load with thousands of queries and cold pages, you're paying I/O cost for nothing. Again and again.</p> If you dig further you discover much bigger problem.</p> SELECT</span> relname, reltuples::</span>bigint as</span> row_estimate, relpages </span>as</span> page_estimate</span></span> FROM</span> pg_class </span></span> WHERE</span> relname </span>IN</span> (</span>'demo'</span>, </span>'demo_pkey'</span>);</span></span></code></pre>relname \| row_estimate \| page_estimate</span></span> -----------+--------------+---------------</span></span> demo \| 20000 \| 934</span></span> demo_pkey \| 20000 \| 276</span></span></code></pre> The relpages</code> value comes from the physical file size divided by the 8 KB page size</a>. PostgreSQL updates it during VACUUM and ANALYZE, but it reflects the actual file on disk - not how much useful data is inside. Our index file is still 2.2 MB (276 pages × 8 KB), even though most pages are empty.</p> The planner sees 276 pages for 20,000 rows and calculates a very low rows-per-page ratio. This is when planner can come to conclusion - this index is very sparse - let's do a sequential scan instead</em>. Oops.</p> "But wait," you say, "doesn't ANALYZE fix statistics?"</p> Yes and no. ANALYZE</code> updates the row count estimate. It will no longer think you have 100,000 rows but 20,000. But it does not shrink relpages, because that reflects the physical file size on disk. ANALYZE</code> can't change that.</p> The planner now has accurate row estimates but wildly inaccurate page estimates. The useful data is packed into just ~57 pages worth of entries, but the planner doesn't know that.</p> cost = random_page_cost × pages + cpu_index_tuple_cost × tuples ([visible in EXPLAIN output](/posts/explain-buffers/))</span></span></code></pre> With a bloated index:</p> pages are oversized (276 instead of ~57)</li> The per-page cost gets multiplied by empty pages</li> Total estimated cost is artificially high</li> </ul> The hollow index</a> </h2> We can dig even more into the index problem when we look at internal stats:</p> SELECT * FROM</span> pgstatindex(</span>'demo_pkey'</span>);</span></span></code></pre>-[ RECORD 1 ]------+--------</span></span> version \| 4</span></span> tree_level \| 1</span></span> index_size \| 2260992</span></span> root_block_no \| 3</span></span> internal_pages \| 1</span></span> leaf_pages \| 57</span></span> empty_pages \| 0</span></span> deleted_pages \| 217</span></span> avg_leaf_density \| 86.37</span></span> leaf_fragmentation \| 0</span></span></code></pre> Wait, what? The avg_leaf_density is 86% and it looks perfectly healthy. That's a trap. Due to the hollow index (we removed 80% right in the middle) we have 57 well-packed leaf pages, but the index still contains 217 deleted pages.</p> This is why avg_leaf_density</code> alone is misleading. The density of used pages looks great, but 79% of your index file is dead weight.</p> The simplest way to spot index bloat is comparing actual size to expected size.</p> SELECT</span></span> c</span>.</span>relname</span> as</span> index_name,</span></span> pg_size_pretty(pg_relation_size(</span>c</span>.</span>oid</span>)) </span>as</span> actual_size,</span></span> pg_size_pretty((</span>c</span>.</span>reltuples</span> </span> 40</span>)::</span>bigint</span>) </span>as</span> expected_size,</span></span> round</span>((pg_relation_size(</span>c</span>.</span>oid</span>) </span>/</span> nullif</span>(</span>c</span>.</span>reltuples</span> </span> 40</span>, </span>0</span>))::</span>numeric</span>, </span>1</span>) </span>as</span> bloat_ratio</span></span> FROM</span> pg_class c</span></span> JOIN</span> pg_index i </span>ON</span> c</span>.</span>oid</span> =</span> i</span>.</span>indexrelid</span></span> WHERE</span> c</span>.</span>relkind</span> =</span> 'i'</span> </span></span> AND</span> c</span>.</span>reltuples</span> ></span> 0</span></span> AND</span> c</span>.</span>relname</span> NOT LIKE</span> 'pg_%'</span></span> AND</span> pg_relation_size(</span>c</span>.</span>oid</span>) </span>></span> 1024</span> </span> 1024</span> -- only indexes > 1 MB</span></span> ORDER BY</span> bloat_ratio </span>DESC NULLS LAST</span>;</span></span></code></pre>index_name \| actual_size \| expected_size \| bloat_ratio</span></span> ------------+-------------+---------------+-------------</span></span> demo_pkey \| 2208 kB \| 781 kB \| 2.8</span></span></code></pre> A bloat_ratio</code> of 2.8 means the index is nearly 3x larger than expected. Anything above 1.8 - 2.0 deserves investigation.</p> We filter to indexes over 1 MB - bloat on tiny indexes doesn't matter that much. Please, adjust the threshold based on your environment; for large databases, you might only care about indexes over 100 MB.</p> But here comes BIG WARNING</strong>: pgstatindex() we used earlier physically reads the entire index. On a 10 GB index, that's 10 GB of I/O. Don't run it against all indexes on a production server - unless you know what you are doing!</p> REINDEX</a> </h2> How to actually fix index bloat problem? REINDEX</code> is s straightforward solution as it rebuilds the index from scratch.</p> REINDEX </span>INDEX</span> CONCURRENTLY demo_pkey ;</span></span></code></pre> After which we can check the index health:</p> SELECT FROM</span> pgstatindex(</span>'demo_pkey'</span>);</span></span></code></pre>-[ RECORD 1 ]------+-------</span></span> version \| 4</span></span> tree_level \| 1</span></span> index_size \| 466944</span></span> root_block_no \| 3</span></span> internal_pages \| 1</span></span> leaf_pages \| 55</span></span> empty_pages \| 0</span></span> deleted_pages \| 0</span></span> avg_leaf_density \| 89.5</span></span> leaf_fragmentation \| 0</span></span></code></pre> And</p> SELECT</span></span> relname,</span></span> pg_size_pretty(pg_relation_size(</span>oid</span>)) </span>as</span> file_size,</span></span> pg_size_pretty((pgstattuple(</span>oid</span>)).tuple_len) </span>as</span> actual_data</span></span> FROM</span> pg_class</span></span> WHERE</span> relname </span>IN</span> (</span>'demo'</span>, </span>'demo_pkey'</span>);</span></span></code></pre>relname \| file_size \| actual_data</span></span> -----------+-----------+-------------</span></span> demo \| 7472 kB \| 1278 kB</span></span> demo_pkey \| 456 kB \| 313 kB</span></span></code></pre> Our index shrunk from 2.2 MB to 456 KB - 79% reduction (not a big surprise though).</p> As you might have noticed we have used CONCURRENTLY</code> to avoid using ACCESS EXCLUSIVE lock. This is available since PostgreSQL 12+, and while there's an option to omit it - the pretty much only reason to do so is during planned maintenance to speed up the index rebuild time.</p> pg_squeeze</a> </h2> If you look above at the file_size of our relations, we have managed to reclaim the disk space for the affected index (it was REINDEX</code> after all), but the table space was not returned back to the operating system.</p> That's where pg_squeeze</a> shines. Unlike trigger-based alternatives, pg_squeeze uses logical decoding, resulting in lower impact on your running system. It rebuilds both the table and all its indexes online, with minimal locking:</p> CREATE</span> EXTENSION pg_squeeze;</span></span> </span> SELECT</span> squeeze</span>.</span>squeeze_table</span>(</span>'public'</span>, </span>'demo'</span>);</span></span></code></pre> The exclusive lock is only needed during the final swap phase, and its duration can be configured. Even better, pg_squeeze is designed for regular automated processing - you can register tables and let it handle maintenance whenever bloat thresholds are met.</p> pg_squeeze makes sense when both table and indexes are bloated, or when you want automated management. REINDEX CONCURRENTLY is simpler when only indexes need work.</p> There's also older tool pg_repack - for a deeper comparison of bloat-busting tools, see article The Bloat Busters: pg_repack vs pg_squeeze</a>.</p> VACUUM FULL (The nuclear option)</a> </h2> VACUUM FULL</code> rewrites the entire table and all indexes. While it fixes everything it comes with a big but - it requires an ACCESS EXCLUSIVE lock - completely blocking all reads and writes for the entire duration. For a large table, this could mean hours of downtime.</p> Generally avoid this in production</strong>. Use pg_squeeze instead for the same result without the downtime.</p> When to act, and when to chill</a> </h2> Before you now go and REINDEX</code> everything in sight, let's talk about when index bloat actually matters.</p> B-trees expand and contract with your data</strong>. With random insertions affecting index columns - UUIDs, hash keys, etc. the page splits happen constantly. Index efficiency might get hit at occassion and also settle around 70 - 80% over different natural cycles of your system usage. That's not bloat. That's the tree finding its natural shape for your data.</p> The bloat we demonstrated - 57 useful pages drowning in 217 deleted ones - is extreme. It came from deleting 80% of contiguous data. You won't see this from normal day to day operations.</p> When do you need to act immediately:</p> after a massive DELETE (retention policy, GDPR purge, failed migration cleanup)</li> bloat_ratio</code> exceeds 2.0 and keeps climbing</li> query plans suddenly prefer sequential scans on indexed columns</li> index size is wildly disproportionate to row count</li> </ul> But in most cases you don't have to panic. Monitor weekly and when indexes bloat ratio continously grow above warning levels, schedule a REINDEX CONCURRENTLY</code> during low traffic period.</p> Index bloat isn't an emergency until it is. Know the signs, have the tools ready, and don't let VACUUM's silence fool you into thinking everything's fine.</p> Conclusion</a> </h2> VACUUM is essential for PostgreSQL. Run it. Let autovacuum do its job. But understand its limitations: it cleans up dead tuples, not index structure.</p> The truth about PostgreSQL maintenance is that VACUUM handles heap bloat reasonably well, but index bloat requires explicit intervention. Know when your indexes are actually sick versus just breathing normally - and when to reach for REINDEX.</p> VACUUM handles heap bloat. Index bloat is your problem. Know the difference.</p> RegreSQL: Regression Testing for PostgreSQL Queries 2025-11-13T22:47:00+00:00 TL;DR</strong> - RegreSQL brings PostgreSQL's regression testing methodology to your application queries, catching both correctness bugs and performance regressions before production.</em></p> As puzzling as it might seem, the common problem with production changes is the ever-present "AHA" moment when things start slowing down or crashing straight away. Testing isn't easy as it is, but there's a widespread practice gap when it comes to testing SQL queries. Some might pretend to "fix it" by using ORMs to abstract away the problem. Others treat SQL as "just glue code" that doesn't deserve systematic testing. Most settle for integration tests that verify the application layer works, never actually testing whether their queries will survive the next schema change or index modification.</p> For PostgreSQL development itself, the project has a robust regression test suite that has been preventing disasters in core development for decades. The database itself knows how to test SQL systematically - we just don't use those same techniques for our own queries. Enter RegreSQL</a>, a tool originally created by Dimitri Fontaine for The Art of PostgreSQL</em></a> book (which is excellent for understanding and mastering PostgreSQL as a database system), designed to bring the same regression testing framework to our application queries.</p> I've been trying to use it for some time, but due to missing features and limitations gave up several times. Until now. I decided to fork the project and spend the time needed to take it to the next level.</p> Introduction</a> </h2> The RegreSQL</a></strong> promise starts with the biggest strength and perceived weakness of SQL queries. They are just strings. And unless you use something like sqlc</a> (for Go), PG'OCaml</a> or Rust's SQLx</a> toolkit giving you compile-time checking, your queries are validated only when they are executed. Which in better case mean either usually slow-ish test suite or integration tests, in worst scenario only when deployed. ORMs are another possibility - completely abstracting away SQL (but more on that later).</p> But even with compile-time checking, you are only checking for one class of problems: schema mismatches. What about behavior changes after schema migration or performance regressions? What about understanding whether your optimization actually made things faster or just moved the problem elsewhere?</p> This is where RegreSQL comes in. Rather than trying to turn SQL into something else, RegreSQL embraces "SQL as strings" reality and applies the same testing methodology PostgreSQL itself uses: regression testing. You write (or generate - continue reading) your SQL queries, provide input data, and RegreSQL verifies that future changes don't break those expectations.</p> The features don't stop there though - it tracks performance baselines, detects common query plan regressions (like sequential scans), and gives you framework for systematic experimentation with the schema changes and query change management.</p> Basic regression testing</a> </h2> Enough with theory. Let's jump in straight into the action and see what a sample run of RegreSQL looks like</p> $ regresql text</span></span> Connecting to 'postgres://radim:password123@192.168.139.28/cdstore_test'… ✓</span></span> </span> Running regression tests...</span></span> </span> ✓ album-by-artist_list-albums-by-artist.1.json (0.00s)</span></span> ✓ album-by-artist_list-albums-by-artist.2.json (0.00s)</span></span> </span> ✓ album-tracks_list-tracks-by-albumid.2.json (0.00s)</span></span> ✓ album-tracks_list-tracks-by-albumid.1.json (0.00s)</span></span> </span> ✓ artist_top-artists-by-album.1.json (0.00s)</span></span> </span> ✓ genre-topn_genre-top-n.top-1.json (0.00s)</span></span> ✓ genre-topn_genre-top-n.top-3.json (0.00s)</span></span> </span> ✓ genre-tracks_tracks-by-genre.json (0.00s)</span></span> </span> Results: 8 passed, 0 failed, 8 skipped (0.00s)</span></span></code></pre> In this example based on Chinook database</a> (as used originally in The Art of PostgreSQL book), RegreSQL scans the current directory (or one provided by -C /path/to/project</code>) for .sql</code> files and attempts to run all queries against the configured PostgreSQL connection.</p> The individual files can contain either single or multiple sql queries. Like following example</p> -- name: top-artists-by-album</span></span> -- Get the list of the N artists with the most albums</span></span> SELECT</span></span> artist</span>.</span>name</span>,</span></span> count</span>(</span></span>) </span>AS</span> albums</span></span> FROM</span></span> artist</span></span> LEFT JOIN</span> album </span>USING</span> (artist_id)</span></span> GROUP BY</span></span> artist</span>.</span>name</span></span> ORDER BY</span></span> albums </span>DESC</span></span> LIMIT</span> :n;</span></span></code></pre> The syntax for the queries supports both positional arguments (like $1</code> known from libpq library) or (preferred) psql</code> style variable (:varname</code>). The each identified query (not file) is then executed for 0..N times, based on number of predefined plans and verified to the expected results - validating the expected data matches the one returned. The support for SQL files handling is available separately with https://github.com/boringSQL/queries (Go version only for now).</p> This gives you what original RegreSQL tool has introduced - change your schema, refactor a query, run regresql test</code> and see immediately what broke. The test suite now has ability to catch regressions before they are committed / shipped. The current version built on top of it, giving you better console formatter instead of TAP style output, as well as jUnit, JSON and GitHub actions formatters for better integration into your CI/CD pipelines.</p> Performance regression testing</a> </h2> Basic regression testing catches correctness issues - wrong results, broken queries, schema mismatches. But there's another class of production issues it misses. Performance regressions. No matter how unbelievable it might sound but queries get deployed without appropriate indexes — which might then be ignored</a> entirely — or they change over time. Simple fix - both for handwritten SQL or ORM code - can switch from milliseconds to seconds. You add index that helps one query, but tanks another. You modify conditionals and accidently force a sequential scan of millions of rows</a>. This is where it hurts.</p> RegreSQL addresses this by tracking performance baselines alongside correctness. Once baselines are generated</p> $ regresql baseline</span></span> Connecting to 'postgres://appuser:password123@192.168.139.28/cdstore_test'… ✓</span></span> Creating baselines directory: regresql/baselines</span></span> Creating directory 'regresql/baselines'</span></span> </span> Creating baselines for queries:</span></span> </span> ./</span></span> Created baseline: album-by-artist_list-albums-by-artist.1.json</span></span> Created baseline: album-by-artist_list-albums-by-artist.2.json</span></span> Created baseline: album-tracks_list-tracks-by-albumid.1.json</span></span> Created baseline: album-tracks_list-tracks-by-albumid.2.json</span></span> Created baseline: artist_top-artists-by-album.1.json</span></span> Created baseline: genre-topn_genre-top-n.top-1.json</span></span> Created baseline: genre-topn_genre-top-n.top-3.json</span></span> Created baseline: genre-tracks_tracks-by-genre.json</span></span> </span> Baselines have been created successfully!</span></span> Baseline files are stored in: regresql/baselines</span></span></code></pre> the test command not only tests the regressions to the captured times, but also detects the common bad patterns in query execution plans</a>. For now it provides warnings for detection of sequential scans - both on their and/or with nested loops and multiple sort operations. I believe this alone might provide a valuable insights and reduce the mishaps in production. It's also a place where further development of RegreSQL will take place.</p> To demonstrate this, let's review the test output with the baselines.</p> Connecting to 'postgres://appuser:password123@192.168.139.28/cdstore_test'… ✓</span></span> </span> Running regression tests...</span></span> </span> ✓ album-by-artist_list-albums-by-artist.1.json (0.00s)</span></span> ✓ album-by-artist_list-albums-by-artist.2.json (0.00s)</span></span> ✓ album-by-artist_list-albums-by-artist.1.cost (22.09 <= 22.09 * 110%) (0.00s)</span></span> ⚠️ Sequential scan detected on table 'artist'</span></span> Suggestion: Consider adding an index if this table is large or this query is frequently executed</span></span> ⚠️ Nested loop join with sequential scan detected</span></span> Suggestion: Add index on join column to avoid repeated sequential scans</span></span> ✓ album-by-artist_list-albums-by-artist.2.cost (22.09 <= 22.09 * 110%) (0.00s)</span></span> ⚠️ Sequential scan detected on table 'artist'</span></span> Suggestion: Consider adding an index if this table is large or this query is frequently executed</span></span> ⚠️ Nested loop join with sequential scan detected</span></span> Suggestion: Add index on join column to avoid repeated sequential scans</span></span> </span> ✓ album-tracks_list-tracks-by-albumid.1.json (0.00s)</span></span> ✓ album-tracks_list-tracks-by-albumid.2.json (0.00s)</span></span> ✓ album-tracks_list-tracks-by-albumid.1.cost (8.23 <= 8.23 * 110%) (0.00s)</span></span> ✓ album-tracks_list-tracks-by-albumid.2.cost (8.23 <= 8.23 * 110%) (0.00s)</span></span> </span> ✓ artist_top-artists-by-album.1.json (0.00s)</span></span> ✓ artist_top-artists-by-album.1.cost (35.70 <= 35.70 * 110%) (0.00s)</span></span> ⚠️ Multiple sequential scans detected on tables: album, artist</span></span> Suggestion: Review query and consider adding indexes on filtered/joined columns</span></span> </span> ✓ genre-topn_genre-top-n.top-1.json (0.00s)</span></span> ✓ genre-topn_genre-top-n.top-3.json (0.00s)</span></span> ✓ genre-topn_genre-top-n.top-1.cost (6610.59 <= 6610.59 * 110%) (0.00s)</span></span> ⚠️ Multiple sequential scans detected on tables: genre, artist</span></span> Suggestion: Review query and consider adding indexes on filtered/joined columns</span></span> ⚠️ Multiple sort operations detected (2 sorts)</span></span> Suggestion: Consider composite indexes for ORDER BY clauses to avoid sorting</span></span> ⚠️ Nested loop join with sequential scan detected</span></span> Suggestion: Add index on join column to avoid repeated sequential scans</span></span> ✓ genre-topn_genre-top-n.top-3.cost (6610.59 <= 6610.59 * 110%) (0.00s)</span></span> ⚠️ Multiple sequential scans detected on tables: artist, genre</span></span> Suggestion: Review query and consider adding indexes on filtered/joined columns</span></span> ⚠️ Multiple sort operations detected (2 sorts)</span></span> Suggestion: Consider composite indexes for ORDER BY clauses to avoid sorting</span></span> ⚠️ Nested loop join with sequential scan detected</span></span> Suggestion: Add index on join column to avoid repeated sequential scans</span></span> </span> ✓ genre-tracks_tracks-by-genre.json (0.00s)</span></span> ✓ genre-tracks_tracks-by-genre.cost (37.99 <= 37.99 * 110%) (0.00s)</span></span> ⚠️ Multiple sequential scans detected on tables: genre, track</span></span> Suggestion: Review query and consider adding indexes on filtered/joined columns</span></span> </span> Results: 16 passed (0.00s)</span></span></code></pre> As you can see, despite from not having baseline, RegreSQL is able to detect the basic bad patterns that should be addressed before queries can be considered "production ready".</p> In some cases, having the detection of sequential scans, or just tracking query costs baselines might be considered undesirable, which would lead to false positives. RegreSQL enables this to be addressed by query metadata as demonstrated below.</p> -- name: query_name</span></span> -- metadata: key1=value1, key2=value2</span></span> SELECT</span> ...;</span></span></code></pre> At this point RegreSQL recognizes</p> notest</code> to skip the query testing altogether (not just cost tracking)</li> nobaseline</code> to skip cost tracking</li> noseqscanwarn</code> to keep cost tracking but disable sequential scan warnings</li> and difffloattolerance</code> to cost failure threshold (default 10% at the moment).</li> </ul> -- name: query_name</span></span> -- regresql: notest, nobaseline</span></span> -- regresql: noseqscanwarn</span></span> -- regresql: difffloattolerance:0.25</span></span> -- query that can vary in cost by 20% without being considered a failure</span></span> SELECT</span> ...;</span></span></code></pre>ORM enters the room</a> </h2> ORMs abstract away SQL, but they still generate it - much like view inlining</a> where the planner rewrites your SQL behind the scenes - and that generated SQL can have performance problems you won't catch until production. Consider this common scenario: you start with a simple SQLAlchemy query that works fine, then months later add eager loading for related data:</p> orders</span> =</span> (</span></span> session.query(Order)</span></span> .filter(Order.user_id</span> ==</span> user_id)</span></span> .options(</span></span> joinedload(Order.user),</span></span> joinedload(Order.shipping_address),</span></span> selectinload(Order.items)</span> # NEW: Load order items</span></span> )</span></span> .all()</span></span> )</span></span></code></pre> That innocent selectinload(Order.items)</code> generates a separate query - and without an index on order_items.order_id</code>, it performs a sequential scan.</p> RegreSQL can catch this by intercepting ORM-generated SQL using SQLAlchemy's event system:</p> @event.listens_for</span>(engine,</span> "before_cursor_execute"</span>)</span></span> def</span> capture_sql</span>(conn, cursor, statement,</span> </span>args):</span></span> captured_queries.append(statement)</span></span></code></pre> Run your ORM code, capture the SQL, save it as a .sql file, and test it with RegreSQL. The performance baseline testing will flag the missing index before it hits production. This is currently experimental, but ORM integration is a key area for RegreSQL's future development.</p> Test Data Management</a> </h2> Up until now we have covered how RegreSQL verifies query correctness and tracks performance regressions. But there's a critical prerequisite we've only skimmed through. Every regression test needs consistent, reproducible data. Change the data, change their cardinality, and your expected results become meaningless. Your performance baselines drift. Your tests become flaky.</p> Traditional approach to create test data might involve</p> Database dumps</strong> become unmanageable - 500MB files you can't review, can't understand, that break with every schema migration, and whose data becomes stale as production evolves. Which version of the dump are your tests even using?</li> SQL scripts</strong> might be better than dumps, but still imperative and hard to maintain. You end up with INSERT statements scattered across multiple files, managing foreign keys manually, and debugging constraint violations.</li> Factories in application code</strong> might work great for integration tests, but we're testing SQL directly. Do you really want to maintain parallel data generation in your application language just for SQL tests?</li> Shared test database</strong> is the synonym for classic "works on my machine" problem. State leaks between tests. Parallel execution becomes impossible. Debugging is a nightmare.</li> </ul> What we need is something that's declarative (what data, not how to insert it), reproducible (similar data every time), composable (build complex scenarios from simple pieces), and scalable (from 10 rows to 100,000).</p> This is where next improvement in RegreSQL's fixture system comes in. Think of it as infrastructure-as-code for your test data. You describe the data you need in YAML files, and RegreSQL handles the rest - dependencies, cleanup, foreign keys, and even realistic data generation at scale.</p> RegreSQL's fixture system lets you define test data in YAML files stored in regresql/fixtures/</code>. Here's a simple example</p> fixture</span>:</span> basic_users</span></span> description</span>:</span> a handful of test users</span></span> cleanup</span>:</span> rollback</span></span> </span> data</span>:</span></span> -</span> table</span>:</span> users</span></span> rows</span>:</span></span> -</span> id</span>:</span> 1</span></span> email</span>:</span> alice@example.com</span></span> name</span>:</span> Alice Anderson</span></span> created_at</span>:</span> 2024-01-15</span></span> -</span> id</span>:</span> 2</span></span> email</span>:</span> bob@example.com</span></span> name</span>:</span> Bob Builder</span></span> created_at</span>:</span> 2024-02-20</span></span></code></pre> To use this fixture in your tests, reference it in the query's plan file (regresql/plans/get-user.yaml</code>) you can just reference the fixture</p> fixtures</span>:</span></span> -</span> basic_users</span></span> </span> "1"</span>:</span></span> email</span>:</span> alice@example.com</span></span> </span> "2"</span>:</span></span> email</span>:</span> bob@example.com</span></span></code></pre> And when you run regresql test</code>, the fixture is automatically loaded before the query executes, and cleaned up afterward. No manual setup scripts, no state leakage between tests. But it does not stop with static fixtures. When you want to test queries against realistic volumes you can use range of data generators</strong> including</p> sequences, random integer, decimal, string, uuid, email and name generators</li> date_between for generating random timestamps within a range</li> foreign key references to be able to reuse data from other table's fixtures</li> range to select value from predefined sources</li> Go template support</li> </ul> fixture</span>:</span> realistic_orders</span></span> generate</span>:</span></span> -</span> table</span>:</span> customers</span></span> count</span>:</span> 1000</span></span> columns</span>:</span></span> id</span>:</span></span> generator</span>:</span> sequence</span></span> start</span>:</span> 1</span></span> email</span>:</span></span> generator</span>:</span> email</span></span> domain</span>:</span> shop.example.com</span></span> name</span>:</span></span> generator</span>:</span> name</span></span> type</span>:</span> full</span></span> created_at</span>:</span></span> generator</span>:</span> date_between</span></span> start</span>:</span> "2023-01-01"</span></span> end</span>:</span> "2024-12-31"</span></span> </span> -</span> table</span>:</span> orders</span></span> count</span>:</span> 5000</span></span> columns</span>:</span></span> id</span>:</span></span> generator</span>:</span> sequence</span></span> start</span>:</span> 1</span></span> customer_id</span>:</span></span> generator</span>:</span> int</span></span> min</span>:</span> 1</span></span> max</span>:</span> 1000</span></span> amount</span>:</span></span> generator</span>:</span> decimal</span></span> min</span>:</span> 10.00</span></span> max</span>:</span> 999.99</span></span> precision</span>:</span> 2</span></span> order_date</span>:</span></span> generator</span>:</span> date_between</span></span> start</span>:</span> "2023-01-01"</span></span> end</span>:</span> "2024-12-31"</span></span></code></pre> This generates 1,000 customers and 5,000 orders with realistic-looking data - names, emails, dates, and amounts that feel production-like.</p> The fixtures are also stackable</strong> and can be build on top of each other. For example if you need to make sure users fixtures are created before orders fixtures, just declare the dependency (the already planned improvement is to include the support automatic foreign-key detection to avoid ID hard-coding). RegreSQL loads fixtures in dependency order and handles cleanup in reverse.</p> fixture</span>:</span> orders_with_shipping</span></span> depends_on</span>:</span></span> -</span> basic_users</span></span> </span> data</span>:</span></span> -</span> table</span>:</span> orders</span></span> rows</span>:</span></span> -</span> id</span>:</span> 101</span></span> user_id</span>:</span> 1</span> # References Alice from basic_users</span></span> total</span>:</span> 99.99</span></span> status</span>:</span> shipped</span></span></code></pre> Should the available options for fixtures (manual data or data generators) not be enough, you always have options to use good old SQL based data generation.</p> fixture</span>:</span> mixed_setup</span></span> description</span>:</span> Combine SQL with YAML and generated data</span></span> cleanup</span>:</span> rollback</span></span> </span> # SQL executes first (either as file or inline)</span></span> sql</span>:</span></span> -</span> file</span>:</span> sql/setup_schema.sql</span></span> -</span> inline</span>:</span> "INSERT INTO config (key, value) VALUES ('version', '1.0');"</span></span> </span> # followed YAML data</span></span> data</span>:</span></span> -</span> table</span>:</span> users</span></span> rows</span>:</span></span> -</span> id</span>:</span> 1</span></span> email</span>:</span> admin@example.com</span></span> </span> # and finally generated data</span></span> generate</span>:</span></span> -</span> table</span>:</span> orders</span></span> count</span>:</span> 100</span></span> columns</span>:</span></span> id</span>:</span></span> generator</span>:</span> sequence</span></span> start</span>:</span> 1</span></span> user_id</span>:</span></span> generator</span>:</span> int</span></span> min</span>:</span> 1</span></span> max</span>:</span> 1</span></span></code></pre> RegreSQL provides commands to inspect and validate your fixtures</p> # List all available fixtures</span></span> regresql</span> fixtures list</span></span> </span> # Show fixture details and dependencies</span></span> regresql</span> fixtures show realistic_orders</span></span> </span> # Validate fixture definitions</span></span> regresql</span> fixtures validate</span></span> </span> # Show dependency graph</span></span> regresql</span> fixtures deps</span></span> </span> # Apply fixture manually (for debugging)</span></span> regresql</span> fixtures apply basic_users</span></span></code></pre> The fixture system has been design to transforms test data from a maintenance burden into a documented, version-controlled process. Your YAML files become the single source of truth for what data your tests need, making it easy to understand test scenarios and maintain test data as the application evolves.</p> (EDIT 2025-11-20)</strong> The dynamically generated fixtures on each tests will break the correctness testing of RegreSQL. You have to use fixtures according to your use</p> Fixed generated fixtures re-used between tests</li> Use RegreSQL to test fixtures before/after migration</li> </ul> I'm working on next release of RegreSQL to address DX of fixtures usage.</p> RegreSQL future</a> </h2> Introducing a new open source project is an ambitious goal, and RegreSQL</a> is just starting up. Despite the fork being in works for almost 2 years. In coming weeks and months I plan further improvements, as well as better documentation and more tutorials. The project is maintained as part of my boringSQL</strong> brand, where it's vital component (together with pgTap) for building SQL Labs</a> which (as I sincerely hope) will provide a foundation for its further development.</p> At the same time RegreSQL</strong> is an attempt to give back to welcoming PostgreSQL community, make developer user experience slightly better if possible and (just maybe) provide one more argument against the case that SQL queries are not testable.</p> RegreSQL is available at boringsql.com/products/regresql</a> - feel free to open issue, or drop me email about the project at radim@boringsql.com</a> or connect on LinkedIn</a>.</p> Beyond Start and End: PostgreSQL Range Types 2025-11-02T21:40:00+00:00 One of the most read articles at boringSQL is Time to Better Know The Time in PostgreSQL</a> where we dived into the complexities of storing and handling time operations in PostgreSQL. While the article introduced the range data types, there's so much more to them. And not only for handling time ranges. In this article we will cover why to consider range types and how to work with them.</p> Bug Not Invented Here</a> </h2> But before we can talk about the range types, let's try to understand why we should look at them in the first place. Let's imagine a booking platform for large flash sales of the seats, that goes live at 10pm and will be taken by storm by thousands of people who want to get their tickets.</p> CREATE TABLE</span> seat_holds</span> (</span></span> hold_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> </span> seat_id </span>INTEGER NOT NULL REFERENCES</span> seats(id),</span></span> user_session_id UUID </span>NOT NULL</span>,</span></span> -- define the hold period explicitly</span></span> hold_started_at </span>TIMESTAMPTZ NOT NULL DEFAULT</span> CURRENT_TIMESTAMP,</span></span> hold_expires_at </span>TIMESTAMPTZ NOT NULL</span>,</span></span> created_at </span>TIMESTAMPTZ NOT NULL DEFAULT</span> CURRENT_TIMESTAMP</span></span> );</span></span> </span> CREATE INDEX</span> seat_holds_on_seat_id</span> ON</span> seat_holds(seat_id);</span></span> CREATE INDEX</span> seat_holds_on_expiration</span> ON</span> seat_holds(hold_expires_at);</span></span></code></pre> While the table design looks perfectly reasonable, it has one serious flaw</strong> - there's no database-level atomicity guarantee there to prevent two holds for the same seat_id</code> at the same time. The table design requires on application logic to check for the existing holds before inserting a new hold, and at the same time it does not provide any high-concurrency guarantee.</p> If all you have in your toolbelt are those two columns you will end up increasing the complexity to make it work. You started with one problem and soon your application developers might want to ask you to add caching layer (most likely external K/V store) to place the holds there and very soon you have N-problems when you will resort to building a complex, custom application-side locking mechanism that is bug-prone and difficult to maintain.</p> Other possibility is to bring all the operations on seat holds into more complex transaction management. Which is literally invitation for disaster in extreme contention</strong> situation like flash ticket sales. No matter which blocking strategy you use SERIALIZABLE</code> transaction isoliation or pessimistic locking using SELECT ... FOR UPDATE</code> will create a large overhead in application logic (retries, massive contention on database resources, etc.).</p> And as we are going to talk about range data types, there's a first possible option to solve the problem directly on database level.</p> -- needed to add GiST support to equality of integers (for seat_id)</span></span> CREATE</span> EXTENSION </span>IF NOT EXISTS</span> btree_gist;</span></span> </span> ALTER TABLE</span> seat_holds </span>ADD CONSTRAINT</span> seat_holds_no_overlap</span></span> EXCLUDE </span>USING</span> gist (</span></span> seat_id </span>WITH =</span>,</span></span> tsrange(hold_started_at, hold_expires_at) </span>WITH</span> &&</span></span> );</span></span></code></pre> Which will add the constraint directly on the table level and enable atomic conflict detection for your seat holds with minimum locking overhead. This guarantees that the database will never allow two overlapping holds to exist for the same seat, irrespective of how many concurrent users are attempting to book. The real win here is data integrity - you're making it impossible to have invalid state in your database, not just unlikely. While you'll still need retry logic in your application when conflicts occur, you've moved the correctness guarantee from application code (where bugs hide) into the database schema (where it's enforced). The GiST (Generalized Search Tree) index is the crucial component which makes checking for overlapping time ranges effective even under extreme load.</p> But really, if you look at the proposed fix, it's still a workaround - we're converting two separate TIMESTAMPTZ</code> columns into a range type on the fly, when range types already include native GiST support out of the box.</p> Introducing the Data Range Types</a> </h2> You’ve seen the power of the EXCLUDE</code> constraint to solve the concurrency problem, but why to settle for workaround (unless it's temporary as part of the bigger refactoring) instead of going all the way in?</p> This brings us to the core of the matter: PostgreSQL's native Range Types</strong>.</p> PostgreSQL provides a set of built-in range types, all following the pattern of type and range:</p> int4range</code> for integer</li> int8range</code> for bigint</li> numrange</code> for numeric</li> tsrange</code> for timestamp without time zone</li> tstzrange</code> for timestamp with time zone (which we briefly saw above)</li> daterange</code> for date</li> </ul> And it does not stop there. You can easily define your own custom range types</strong> over any basic data type.</p> First win: cleaner schema</a> </h2> When using start</code> and end</code> columns you are not explicitely telling the database that these two columns are single concept representing time span. The logic to work with those two columns resides only in your queries and application code.</p> When you refactor our sample table to embrace the native range type, it becomes more expressive and inherently correct. The application code no longer needs to manage two separate boundaries.</p> CREATE TABLE</span> seat_holds_native</span> (</span></span> hold_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> seat_id </span>INTEGER NOT NULL REFERENCES</span> seats(id),</span></span> user_session_id UUID </span>NOT NULL</span>,</span></span> hold_period TSTZRANGE </span>NOT NULL</span>,</span></span> created_at </span>TIMESTAMPTZ NOT NULL DEFAULT</span> CURRENT_TIMESTAMP</span></span> );</span></span></code></pre> This is the power of a first-class data type</strong>. We've shifted the burden from the application logic to the database schema, making the table definition itself communicate its intent more clearly.</p> Second win: Atomicity guaranteed</a> </h2> While the new schema is cleaner, the real database win comes from enforcing our concurrency guarantee - preventing two seats from being double-booked. To achieve this you can reuse the exclusion constraint</strong> as demonstrated previously.</p> ALTER TABLE</span> seat_holds_native </span>ADD CONSTRAINT</span> seat_holds_no_overlap</span></span> EXCLUDE </span>USING</span> gist (</span></span> seat_id </span>WITH =</span>,</span></span> hold_period </span>WITH</span> &&</span></span> );</span></span></code></pre> This time you can use hold_period</code> directly without need to explicitely convert it. This constraint enforces two rules at once:</p> seat_id WITH =</code> ensures the constraint only applies to holds for the same seat.</li> hold_period WITH &&</code> checking the overlap of hold periods with the operator &&</code></li> </ol> Finally EXCLUDE USING gist</code> is the crucial technical detail, telling PostgreSQL to use GiST index to enforce the constraint. This is not specific to range types, as EXCLUDE</code> constraint can't exist without an index to enforce it (common use cases might include arrays, geometric data, etc.).</p> Range boundaries: A quick math refresher</a> </h2> Before we dive into the operators, let's take a moment to understand how PostgreSQL represents range boundaries. If you remember your high school math, range types use the same notation as intervals in mathematics.</p> PostgreSQL ranges can have four different boundary types:</p> Inclusive boundaries -</strong> [</code> and ]</code> An inclusive boundary includes the endpoint value in the range. Think of it as "less than or equal to" or "greater than or equal to".</p> -- [10, 20] includes both 10 and 20</span></span> -- represents 10 ≤ x ≤ 20</span></span> SELECT</span> int4range(</span>10</span>, </span>20</span>, </span>'[]'</span>);</span></span></code></pre> Exclusive boundaries -</strong> (</code> and )</code> An exclusive boundary excludes the endpoint value from the range. This is "less than" or "greater than" without the equality.</p> -- (10, 20) excludes both 10 and 20</span></span> -- represents 10 < x < 20</span></span> SELECT</span> int4range(</span>10</span>, </span>20</span>, </span>'()'</span>);</span></span> </span></code></pre> Mixed boundaries -</strong> [)</code> or (]</code> You can mix and match. The most common and default pattern</strong> in PostgreSQL is [)</code> - inclusive lower bound, exclusive upper bound. This is particularly useful for timestamps and dates because it naturally represents "from the start of one period up to (but not including) the start of the next".</p> As mentioned, the default boundary [)</code> eliminates the natural ambiguity when representing consecutive periods.</p> -- these ranges are adjacent, not overlapping</span></span> -- Week 1</span></span> SELECT</span> '[2025-11-01, 2025-11-08)'</span>::tstzrange;</span></span> -- Week 2</span></span> SELECT</span> '[2025-11-08, 2025-11-15)'</span>::tstzrange;</span></span></code></pre> With this notation, the end of one period is exactly the start of the next, with no gaps or overlaps. This makes it perfect for time-based ranges, inventory availability windows, or any scenario where you're dividing a continuum into distinct segments.</p> Canonicalization and Range Set Operations</a> </h2> Boundaries are not the only aspect of the ranges that behave like mathematical sets, allowing arithmetic operations and canonicalization of the discrete ranges.</p> For discrete</strong> range types (int4range, int8range, daterange), multiple representations can actually mean the exact same set of values. For example, for integers, the range [10, 20] (inclusive on both ends) is the same set as (9, 21) (exclusive on both ends) or the default PostgreSQL canonical form [10, 21) (inclusive lower, exclusive upper).</p> PostgreSQL uses a canonicalization function</strong> to convert all equivalent discrete ranges into a single, uniform representation (default [)</code> boundary mentioned above), which is essential for accurate equality checks and indexing.</p> -- [10, 20] includes integers 10, 11, ..., 20.</span></span> SELECT</span> int4range(</span>10</span>, </span>20</span>, </span>'[]'</span>) </span>AS</span> original_range;</span></span> -- Result: [10,21)</span></span> </span> -- (9, 21) includes integers 10, 11, ..., 20.</span></span> SELECT</span> int4range(</span>9</span>, </span>21</span>, </span>'()'</span>) </span>AS</span> original_range;</span></span> -- Result: [10,21)</span></span></code></pre> Exception are continuous ranges</strong> (think floats and timestamps with fractional seconds) where PostgreSQL won't use canonicalization because a boundary change always means a change in the contained values as there's no easy to define "next value". I.e. there's no next value for 20.0 (i.e. not 20.0001, nor 20.000001, etc.) and changing boundary would change it's meaning.</p> The operator toolkit</a> </h2> The type ranges and by their definition GiST (in this instance range_ops) and GIN (array_ops) indexes come with number of operator that makes your life easier.</p> Overlap operator - &&</code></strong> As mentioned already above the overlap operator is the most fundamental one. It simply checks whether two ranges share any common data points.</p> -- find holds active at any point between 10:00 and 11:00</span></span> SELECT FROM</span> seat_holds_native </span>WHERE</span> hold_period && </span>'[2025-12-25 10:00, 2025-12-25 11:00)'</span>::tstzrange;</span></span></code></pre> Contains operator - @></code></strong> For checking against specific moments in time, we might turn to the Contains operator. It verifies whatever the range on the left completely containts the element on the right (which might be both underlying data type or range type).</p> -- find holds that are active at the specific momement</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period @</span>></span> '2025-11-05 15:00'</span>::</span>timestamptz</span>;</span></span> </span> -- find holds that are active at the specific time range</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period @</span>></span> '[2025-12-25 10:00, 2025-12-25 10:15)'</span>::tstzrange;</span></span></code></pre> Contained By operator - <@</code></strong> In contrast, the Contained By operator checks the reverse relationship - whatever the range on the left is entirely contained by the range on the right.</p> -- find holds that are within November '25</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period </span><</span>@ </span>'[2025-11-01, 2025-12-01)'</span>::tstzrange;</span></span></code></pre> Strictly Before/After operators - <<</code> and >></code></strong> operators allow you to query ranges that are completely separated from the reference range (i.e. don't even touch the boundaries).</p> -- find holds that finished strictly before 10 November</span></span> SELECT * FROM seat_holds_native WHERE hold_period << '[2025-11-10, 2025-11-15)'::tstzrange;</span></span></code></pre> Boundary Extension operators - &<</code> and &></code></strong> let you reason about range boundaries independently, checking whether one range extends beyond another's endpoints (i.e. it can start/end anywhere within the given range).</p> -- find holds that end before or at the same time as reference range ends</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period &</span><</span> '[2025-11-08 17:00, 2025-11-08 18:00)'</span>::tstzrange;</span></span> </span> -- find holds that start at or after reference range starts</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period &</span>></span>'[2025-11-08 09:00, 2025-11-08 18:00)'</span>::tstzrange;</span></span></code></pre> Finally the Adjecent operator - -\|-</code></strong> checks if two ranges are perfectly contiguous - they MUST touch at exactly one boundary point, but do not overlap. This might be invaluable when checking if a customer can extend an existing hold without any gap or conflict.</p> -- find holds that are immediately adjacent (touching) to given range</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period </span>-</span>\|</span>-</span> '[2025-11-08 17:00, 2025-11-08 18:00)'</span>::tstzrange;</span></span></code></pre>To Infinity and Beyond</a> </h2> Similar to base types ranges in PostgreSQL can handle NULL values, but it does not stop there. There are also special states specifically applicable to data type ranges: empty</code> and infinity</code>.</p> Let's start with infinite bounds</strong>, the bound that shows the real power of the ranges. You can define range that extend infinitely in either direction (or both at the same time).</p> -- range that never expires (upper bound is infinite)</span></span> SELECT</span> '[2025-11-01 10:00, infinity)'</span>::tstzrange;</span></span> </span> -- range that has always been valid (lower bound is infinite)</span></span> SELECT</span> '[-infinity, 2025-11-01 10:00)'</span>::tstzrange;</span></span> </span> -- range covering all time</span></span> SELECT</span> '[-infinity, infinity)'</span>::tstzrange;</span></span></code></pre> This gives you ability to describe the "from this points forward" use cases. As we will cover later we can easily define lifetime subscription.</p> -- lifetime subscription that never expires</span></span> INSERT INTO</span> subscriptions (user_id, plan, active_period)</span></span> VALUES</span> (</span>42</span>, </span>'lifetime'</span>, </span>'[2025-11-01, infinity)'</span>);</span></span> </span> -- all active subscriptions right now</span></span> SELECT * FROM</span> subscriptions</span></span> WHERE</span> active_period @</span>> NOW</span>();</span></span></code></pre> Using infinity</code> is far more elegant solution that using NULL values or "special" values like 2099-31-12</code> - it's explicit and clearly communicates the data intent.</p> At any point you can validate whatever range has infinite bounds:</p> SELECT</span></span> lower_inf(</span>'[2025-11-01, infinity)'</span>::tstzrange) </span>AS</span> lower_is_infinite,</span></span> upper_inf(</span>'[2025-11-01, infinity)'</span>::tstzrange) </span>AS</span> upper_is_infinite;</span></span></code></pre>Understanding NULL vs empty: Schrödinger's Range</a> </h2> Ranges can be NULL or empty, and these are completely different things. NULL is Schrödinger's range - you haven't looked in the box yet, so it could be anything or nothing. Empty is when you've opened the box and confirmed it's empty.</p> Let's see this in practice:</p> -- NULL range: we don't know what the period is</span></span> INSERT INTO</span> seat_holds_native (seat_id, user_session_id, hold_period)</span></span> VALUES</span> (</span>42</span>, </span>'abc-123'</span>, </span>NULL</span>);</span></span> </span> -- empty range: we know the period is explicitly "nothing"</span></span> INSERT INTO</span> seat_holds_native (seat_id, user_session_id, hold_period)</span></span> VALUES</span> (</span>43</span>, </span>'def-456'</span>, </span>'empty'</span>);</span></span></code></pre> The main difference between them is when it comes to handling.</p> SELECT NULL</span>::tstzrange && </span>'[2025-11-01, 2025-11-08)'</span>::tstzrange;</span></span> -- Result: NULL (not true, not false—we don't know)</span></span> </span> SELECT</span> 'empty'</span>::tstzrange && </span>'[2025-11-01, 2025-11-08)'</span>::tstzrange;</span></span> -- Result: false (we know it doesn't overlap)</span></span></code></pre> And you can check for them in your queries using built-in function isempty</code>.</p> -- check for NULL (like any column)</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period </span>IS NULL</span>;</span></span> </span> -- check for empty (special function)</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> isempty(hold_period);</span></span></code></pre> In practice, you'll mostly use NOT NULL</code> constraints to prevent NULL ranges entirely. Empty ranges are useful but rare - usually for representing cancelled/void periods you need to keep for special purposes - like audit trail.</p> Practical Integer ranges for Tiered pricing</a> </h2> While we introduced ranges we mostly paid attention to the date/time handling the usefulness of range types goes well beyond that. One of the practical applications where integer ranges provide real values can be demostrated on tiered pricing.</p> CREATE TABLE</span> quantity_discounts</span> (</span></span> discount_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> product_id </span>INTEGER NOT NULL</span>,</span></span> quantity_range INT4RANGE </span>NOT NULL</span>,</span></span> discount_percentage </span>NUMERIC</span>(</span>5</span>,</span>2</span>)</span> NOT NULL</span>,</span></span> </span> -- no overlapping tiers</span></span> EXCLUDE </span>USING</span> GIST (</span></span> product_id </span>WITH =</span>,</span></span> quantity_range </span>WITH</span> &&</span></span> )</span></span> );</span></span> </span> INSERT INTO</span> quantity_discounts (product_id, quantity_range, discount_percentage) </span>VALUES</span></span> -- 1-9 units: no discount</span></span> (</span>1</span>, </span>'[1,10)'</span>, </span>0</span>.</span>00</span>),</span></span> -- 10-49 units: 5% off</span></span> (</span>1</span>, </span>'[10,50)'</span>, </span>5</span>.</span>00</span>),</span></span> -- 50-99 units: 10% off</span></span> (</span>1</span>, </span>'[50,100)'</span>, </span>10</span>.</span>00</span>),</span></span> -- 100+ units: 15% off</span></span> (</span>1</span>, </span>'[100,1000)'</span>, </span>15</span>.</span>00</span>);</span></span> </span> -- verify what discount we offer for ordering 75 units?</span></span> SELECT</span> discount_percentage</span></span> FROM</span> quantity_discounts</span></span> WHERE</span> product_id </span>=</span> 1</span></span> AND</span> quantity_range @</span>></span> 75</span>;</span></span> -- Result: 10.00</span></span></code></pre>Making Bad Data Impossible</a> </h2> If the introduction of the range types provided the case for cleaner schema you can go ahead and make hard limits structurally impossible. While this is not advocating for the transition of the full business logic into database schema, you can eliminate the edge cases that should never make it to the database.</p> CREATE TABLE</span> promotional_campaigns</span> (</span></span> campaign_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name TEXT NOT NULL</span>,</span></span> active_period TSTZRANGE </span>NOT NULL</span>,</span></span> budget_range NUMRANGE </span>NOT NULL</span>,</span></span> discount_percentage </span>NUMERIC</span>(</span>5</span>,</span>2</span>)</span> NOT NULL</span>,</span></span> </span> -- campaigns must be at least 1 days long</span></span> CONSTRAINT</span> campaigns_minimum_duration</span></span> CHECK</span> (</span>upper</span>(active_period) </span>-</span> lower</span>(active_period) </span>>=</span> INTERVAL </span>'1 days'</span>),</span></span> </span> -- budget must be between $1000 and $100000</span></span> CONSTRAINT</span> campaigns_valid_budget</span></span> CHECK</span> (budget_range </span><</span>@ numrange(</span>1000</span>, </span>100000</span>, </span>'[]'</span>)),</span></span> </span> -- active period must not be empty</span></span> CONSTRAINT</span> campaigns_valid_period</span></span> CHECK</span> (</span>NOT</span> isempty(active_period))</span></span> );</span></span></code></pre> While this example demonstrates hypothetical example within the schema definition, please remember they shouldn't be used to implement business process. The goal of the constraints is to enforce data integrity, i.e. structure requirements (minimum duration, non-empty data), physical or domain boundaries. Any other logic should make it's way either to application logic or parts that are easier to modify (think functions).</p> Multiranges: When one range is not enough</a> </h2> Up until now, we've been working with single continuous ranges. But what happens when you need to represent fragmented ranges? In past you needed a separate table with a foreign key relationship. With multiranges, you can store multiple non-contiguous ranges in a single column.</p> PostgreSQL 14 introduced multirange types for all the built-in range types:</p> int4multirange</code>, int8multirange</code>, nummultirange</code></li> tsmultirange</code>, tstzmultirange</code>, datemultirange</code></li> </ul> The real power of multiranges lies in schema density</strong> and query efficiency</strong>. We can prove this by comparing the cost of storing and querying the exact same data using two different valid range schemas.</p> Let's consider storing 20 fragmented and non-contiguous periods - a pattern common for historical data.</p> CREATE TABLE</span> user_periods_single_range</span> (</span></span> period_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> user_id </span>INTEGER NOT NULL</span>,</span></span> active_period TSTZRANGE </span>NOT NULL</span></span> );</span></span> </span> CREATE INDEX</span> user_periods_single_range_gist_idx</span></span> ON</span> user_periods_single_range</span></span> USING</span> gist (active_period);</span></span> </span> -- 20 fragmented periods for user_id 42 (20 rows total)</span></span> INSERT INTO</span> user_periods_single_range (user_id, active_period)</span></span> SELECT</span></span> 42</span>,</span></span> tstzrange(</span></span> '2025-01-01'</span>::</span>timestamptz +</span> (i </span></span> 2</span> \|\|</span> ' days'</span>)::interval,</span></span> '2025-01-01'</span>::</span>timestamptz +</span> (i </span></span> 2</span> +</span> 1</span> \|\|</span> ' days'</span>)::interval</span></span> )</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>20</span>) </span>AS</span> s(i);</span></span></code></pre> compared to 1 row for the same 20 periods aggregated using range_agg</code> function to consolidate data.</p> CREATE TABLE</span> user_periods_multirange</span> (</span></span> user_id </span>INTEGER PRIMARY KEY</span>,</span></span> active_periods TSTZMULTIRANGE </span>NOT NULL</span></span> );</span></span> </span> CREATE INDEX</span> user_periods_multirange_gist_idx</span></span> ON</span> user_periods_multirange</span></span> USING</span> gist (active_periods);</span></span> </span> -- consolidate the 20 TSTZRANGE rows into 1 TSTZMULTIRANGE row</span></span> INSERT INTO</span> user_periods_multirange (user_id, active_periods)</span></span> SELECT</span></span> user_id,</span></span> -- range_agg function automatically handles merging adjacent ranges</span></span> range_agg(active_period)::TSTZMULTIRANGE</span></span> FROM</span> user_periods_single_range</span></span> WHERE</span> user_id </span>=</span> 42</span></span> GROUP BY</span> user_id;</span></span></code></pre> consider now the difference between</p> EXPLAIN</span></span> SELECT</span> period_id, user_id</span></span> FROM</span> user_periods_single_range</span></span> WHERE</span></span> user_id </span>=</span> 42</span></span> AND</span> active_period @</span>></span> '2025-01-20 12:00:00+00'</span>::</span>timestamptz</span>;</span></span></code></pre> QUERY PLAN</span></span> -------------------------------------------------------------------------------------------------</span></span> Bitmap Heap Scan on user_periods_single_range (cost=4.19..13.67 rows=1 width=12)</span></span> Recheck Cond: (active_period @> '2025-01-20 13:00:00+01'::timestamp with time zone)</span></span> Filter: (user_id = 42)</span></span> -> Bitmap Index Scan on user_periods_single_range_gist_idx (cost=0.00..4.19 rows=6 width=0)</span></span> Index Cond: (active_period @> '2025-01-20 13:00:00+01'::timestamp with time zone)</span></span></code></pre> and same version of the consolidated data</p> EXPLAIN ANALYZE</span></span> SELECT</span> user_id, active_periods</span></span> FROM</span> user_periods_multirange</span></span> WHERE</span></span> user_id </span>=</span> 42</span></span> AND</span> active_periods @</span>></span> '2025-01-20 12:00:00+00'</span>::</span>timestamptz</span>;</span></span></code></pre> QUERY PLAN</span></span> -------------------------------------------------------------------------------------------------------------</span></span> Index Scan using user_periods_multirange_pkey on user_periods_multirange (cost=0.15..8.17 rows=1 width=36)</span></span> Index Cond: (user_id = 42)</span></span> Filter: (active_periods @> '2025-01-20 13:00:00+01'::timestamp with time zone)</span></span></code></pre> Giving you significant reduction in the query cost. And while the example was simple enough for the demonstration purposes you can easily define the helper schema for better indexable data access and opportunities to reduce the storage requirements.</p> One important note is that subtle change with the &&</code> operators. Whereas with single range &&</code> operator checks if two continues ranges overlap, for multiranges the operator checks if ANY range in multirange overlaps.</p> Creating custom range types</a> </h2> While PostgreSQL provides built-in range types for common data types, you can create custom range types for any data type that has a meaningful ordering. Let's demostrate this with a type for IP address ranges.</p> To create a custom range type, you need to provide a subtype difference function that tells PostgreSQL how to calculate the "distance" between two values:</p> -- function to calculate difference between two IP addresses using bigint</span></span> CREATE OR REPLACE FUNCTION</span> inet_diff</span>(x </span>INET</span>, y </span>INET</span>)</span></span> RETURNS</span> FLOAT8 </span>AS</span> $$</span></span> SELECT</span> (</span></span> (host(x)::</span>inet -</span> '0.0.0.0'</span>::</span>inet</span>) </span>-</span></span> (host(y)::</span>inet -</span> '0.0.0.0'</span>::</span>inet</span>)</span></span> )::float8;</span></span> $$ </span>LANGUAGE SQL</span> IMMUTABLE STRICT;</span></span> </span> -- custom inetrange type</span></span> CREATE TYPE</span> inetrange</span> AS RANGE</span> (</span></span> subtype </span>= inet</span>,</span></span> subtype_diff </span>=</span> inet_diff</span></span> );</span></span> </span> -- convert CIDR ranges to inetranges</span></span> CREATE OR REPLACE FUNCTION</span> cidr_to_inetrange</span>(</span>cidr CIDR</span>)</span></span> RETURNS</span> inetrange </span>AS</span> $$</span></span> SELECT</span> inetrange(</span></span> host(network($</span>1</span>))::</span>inet</span>,</span></span> host(broadcast($</span>1</span>))::</span>inet</span>,</span></span> '[]'</span></span> );</span></span> $$ </span>LANGUAGE SQL</span> IMMUTABLE STRICT;</span></span></code></pre> Now you can use these custom types just like the built-in ones:</p> CREATE TABLE</span> ip_blocklists</span> (</span></span> blocklist_name </span>TEXT PRIMARY KEY</span>,</span></span> blocked_range inetrange </span>NOT NULL</span></span> );</span></span> </span> INSERT INTO</span> ip_blocklists (blocklist_name, blocked_range) </span>VALUES</span></span> (</span>'attack #434401'</span>, cidr_to_inetrange(</span>'192.168.1.0/24'</span>)),</span></span> (</span>'attack #434401 (1)'</span>, </span>'[203.0.113.50,203.0.113.99]'</span>),</span></span> (</span>'attack #434401 (2)'</span>, </span>'[203.0.113.143,203.0.113.159]'</span>);</span></span></code></pre> and locate which attack has been assigned to particular malicious IP address.</p> SELECT</span> blocklist_name, blocked_range</span></span> FROM</span> ip_blocklists</span></span> WHERE</span> blocked_range @</span>></span> '192.168.1.25'</span>::</span>INET</span>;</span></span></code></pre> blocklist_name \| blocked_range</span></span> ----------------+-----------------------------</span></span> attack #434401 \| [192.168.1.0,192.168.1.255]</span></span></code></pre> But wait a second, wasn't the fragmented nature of the ranges used in case for multiranges? Building real-life production and auto adaptive block list would most likely soon create extremely fragmented set of targets to block.</p> Can we defined it for our custom range types? Well no, because PostgreSQL is amazing! Since PostgreSQL 14 every time you define custom range type, it will automatically create corresponding multirange! Making it easy to consolidate the fragmented data corresponding to individual attack to multiranges.</p> SELECT</span> typname </span>FROM</span> pg_type </span>WHERE</span> typname </span>LIKE</span> 'inet%range'</span>;</span></span></code></pre> typname</span></span> ----------------</span></span> inetmultirange</span></span> inetrange</span></span></code></pre> Danger zone! When creating a custom range type the subtype_diff</strong> function is more than just simple helper. It plays important role in indexing and query performance</strong>. It really tells PostgreSQL planner how far apart the values in range are, which is crucial when building GiST indexes for ranges.</p> In our example above, if inet_diff</code> returned 0</code> for every pair of IP addresses, PostgreSQL would think all ranges are "equally close". This would lead to un-balanced indexes, with large hotposts in the indexes. End result would be that range operators would effectively be almost as slow as sequantial scans.</p> Performance deep dive: GiST vs GIN</a> </h2> Throughout this article, we've been using GiST indexes almost exclusively, particularly when enforcing exclusion constraints. But PostgreSQL also supports GIN indexes for range types, and understanding when to use each can make the difference between a query that completes in milliseconds versus one that grinds your database to a halt.</p> Before we deep dive, let's recap what those two indexes do. GiST (Generalized Search Tree)</strong> is a balanced tree structure that organizes ranges by their bounding boxes, and grouping those that are "close together" in same tree nodes. While GIN (Generalized Inverted Index)</strong> would decompose ranges into their components and indexing those. Therefore GiST works for continous ranges (timestamps and numerical values), while GIN works for discrete ranges (as you can't generate unpredictable range of values of floats for example). Given this characteristics you can almost certainly say GIN indexes are almost always going to be bigger compared to the GiST ones, as they are always trying to index a continous space.</p> The most important thing to know upfront? As we already used without explicitely mentioning it - you can't use GIN with</strong> EXCLUDE constraints.</strong> GiST is mandatory there.</p> Therefore while GiST index is going to be preferred for cases like</p> CREATE TABLE</span> seat_holds</span> (</span></span> hold_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> seat_id </span>INTEGER NOT NULL</span>,</span></span> hold_period TSTZRANGE </span>NOT NULL</span></span> );</span></span></code></pre> the GIN index is actually preferred for</p> CREATE TABLE</span> venue_blackouts</span> (</span></span> venue_id </span>INTEGER</span>,</span></span> blocked_dates DATERANGE </span>NOT NULL</span></span> );</span></span></code></pre> The reason is simple - dates are discrete type. There's only 365 (or 366) combinations in a given year. While timestamps have microsecond precision - millions of possible values per day.</p> The complexity of the index types is far beyond this article scope, so let's just iterate over basic rules what index type to use. Most applications should just use GiST and move on. The performance difference rarely matters until you're dealing with millions of rows and very specific query patterns. Don't prematurely optimize - GiST is the safe, versatile default that works well for almost everything. You can always add a GIN index later if profiling shows it would help. PostgreSQL's query planner is smart enough to pick the better index when both are available.</p> Conclusion</a> </h2> From my experience range types represent one of PostgreSQL's most underutilized features, yet they offer immediate benefits: cleaner schemas, built-in data integrity, and query patterns that feel natural once you embrace them. What started as a solution to prevent double-booking seats evolved into a comprehensive look at how treating ranges as first-class citizens transforms your database design.</p> But we've really just scratched the surface. Timestamp ranges in particular open an entire world of possibilities we haven't touched - temporal tables</strong>. The ability to maintain complete historical records with automatic versioning, query data "as of" any point in time, and track changes without cluttering your schema with audit columns deserves its own deep dive. That's a topic for a future article.</p> For now, the next time you reach for separate start</code> and end</code> columns, stop and ask yourself: "Should this be a range?" More often than not, the answer is yes. Your future self - the one debugging a problems in least convenient time - will thank you.</p> PostgreSQL maintenance without superuser 2025-09-13T20:55:00+00:00 How many people/services have superuser access to your PostgreSQL cluster(s)? Did you ever ask why your software engineers might need it? Or your BI team? Why those use cases require same privileges as someone who can drop your databases?</p> The answer isn't because these operations are inherently dangerous - it's because PostgreSQL historically offered limited options for operational access or simply because not enough people are aware of the options. So the common practice is to either got basic permissions or handover the keys to the kingdom.</p> PostgreSQL's built-in predefined roles</strong> solve this problem by providing purpose-built privileges for common maintenance tasks. Instead of granting superuser access for routine operations, you can delegate specific capabilities - monitoring teams get comprehensive observability access, backup services get data reading capabilities, and maintenance scripts get precisely the permissions they need, nothing more.</p> What are Predefined Roles?</a> </h2> PostgreSQL's built-in administrative roles are purpose-built permission sets that solve the superuser dilemma for common maintenance tasks. Out of the box, there are 15 predefined roles that provide granular access to specific operational capabilities without requiring full superuser privileges.</p> While you can view their list and description in official documentation</a>, in this article we will explore them bit more thoroughly and at the same time look into system catalogs to understand them better. The individual roles can be grouped by their functionality and most of them are quite easy to grasp, ranging from simple monitoring access to powerful filesystem operations that require careful consideration.</p> Data Access Role</strong></p> pg_database_owner</code> - Database-specific ownership (special case)</li> pg_read_all_data</code> - Read access to all tables, views, sequences</li> pg_write_all_data</code> - Write access to all tables, views, sequences</li> </ul> Monitoring & Observability</strong></p> pg_monitor</code> - Which is actually monitoring meta role that contains 3 roles listed below</li> pg_read_all_settings</code> - Configuration access</li> pg_read_all_stats</code> - Statistics views</li> pg_stat_scan_tables</code> - Table scanning for stats</li> </ul> System Operations</strong></p> pg_signal_backend</code> - Cancel queries, terminate sessions</li> pg_checkpoint</code> - Run CHECKPOINT command</li> pg_maintain</code> - VACUUM, ANALYZE, REINDEX operations (PostgreSQL 17+)</li> pg_signal_autovacuum_worker</code> - Signal autovacuum worker (PostgreSQL 18+)</li> </ul> File System Access</strong></p> pg_read_server_files</code> - Read files from server filesystem</li> pg_write_server_files</code> - Write files to server filesystem</li> pg_execute_server_program</code> - Execute programs on server</li> </ul> And specialised use cases</strong></p> pg_create_subscription</code> - Logical replication management</li> pg_use_reserved_connections</code> - Connection reservation (PostgreSQL 16+)</li> </ul> Why Use Predefined Roles?</a> </h2> The primary benefit of predefined roles is expanding the pool of users who can safely manage PostgreSQL databases</strong>. Traditional PostgreSQL administration created an artificial binary choice: either users had basic access with limited capabilities, or they required full superuser privileges for operational tasks. This forced many organizations to grant excessive permissions simply to perform routine operations like monitoring, backups, or maintenance.</p> Predefined roles break this limitation by allowing more granular control over operational tasks without distributing superuser privileges. Instead of having a small number of highly privileged superusers handling all administrative work, organizations can now delegate specific capabilities to appropriate operational roles - monitoring teams get monitoring access, backup services get data reading capabilities, and maintenance scripts get precisely the permissions they need.</p> The deeper benefit is abstraction</strong>. Not only do you switch from manually managing permissions on individual system objects, you switch to managing logical capability sets. This creates a clear separation between what</em> you want to allow and how</em> it's actually implemented.</p> The advantage is clearly demonstrated in following example</p> -- granting select on each schema separately</span></span> GRANT</span> USAGE </span>ON SCHEMA</span> finance, hr, app, </span>audit TO</span> analytics_team;</span></span> GRANT SELECT ON</span> ALL TABLES </span>IN SCHEMA</span> finance </span>TO</span> analytics_team;</span></span> GRANT SELECT ON</span> ALL SEQUENCES </span>IN SCHEMA</span> finance </span>TO</span> analytics_team;</span></span> </span> -- versus one time setup</span></span> GRANT</span> pg_read_all_data </span>TO</span> analytics_team;</span></span></code></pre> The predefined roles approach automatically covers:</p> All current tables, views and sequences</li> All schemas</li> Future objects created after the GRANT</code></li> </ul> And the good news is the benefits don't end here. The real thing on operational level comes with the scope of those predefined roles - they apply at cluster level</strong> (instead on database level). Only notable exception is role pg_database_owner</code> which we will cover next.</p> Before going there, let's briefly mention second benefit, and that's the fact any new operational features in PostgreSQL will be covered by those predefined roles automatically.</p> The Evolution of Predefined Roles</a> </h2> Understanding when and why predefined roles were introduced provides important insight into PostgreSQL's operational priorities and helps explain some of the design decisions. The journey from a single role in PostgreSQL 9.6 to today's comprehensive set reflects real-world pain points that PostgreSQL developers encountered in production environments.</p> PostgreSQL 9.6 (2016)</strong> introduced first predefined role pg_signal_backend</code> to address common frustration - inability to cancel running query without giving away superuser rights.</p> -- This became possible in 9.6 without superuser privileges</span></span> SELECT</span> pg_cancel_backend(</span>12345</span>); </span>-- Cancel a query</span></span> SELECT</span> pg_terminate_backend(</span>12345</span>); </span>-- Terminate a session</span></span></code></pre> PostgreSQL 10 (2017)</strong> brought biggest expansion of predefined roles with, adding four monitoring specific roles to improve database observability, and support growing importance of database monitoring in production environments.</p> -- Create the monitoring user</span></span> CREATE USER</span> postgres_exporter</span> WITH PASSWORD</span> 'monitoring_password'</span>;</span></span> </span> -- Grant comprehensive monitoring access</span></span> GRANT</span> pg_monitor </span>TO</span> postgres_exporter;</span></span></code></pre> Giving access to variety of metrics and settings. With single GRANT</code> you can access data:</p> From pg_read_all_settings (inherited via pg_monitor) to SELECT * FROM pg_settings</code> - Configuration parameters for alerting on misconfigurations and configuration metrics like shared_buffers, work_mem, max_connections</li> </ul> From pg_read_all_stats (inherited via pg_monitor):</p> -- Database-wide statistics (connections, transactions, blocks read/written)</span></span> SELECT * FROM</span> pg_stat_database</span></span> -- Current query activity and connection states</span></span> SELECT * FROM</span> pg_stat_activity</span></span> -- Replication lag and status</span></span> SELECT * FROM</span> pg_stat_replication</span></span> -- Background writer performance</span></span> SELECT * FROM</span> pg_stat_bgwriter</span></span> -- Lock contention monitoring</span></span> SELECT * FROM</span> pg_locks</span></span></code></pre> From pg_stat_scan_tables (inherited via pg_monitor):</p> Enhanced table statistics collection with sequential scan information</li> More detailed I/O statistics for table access patterns</li> </ul> The pg_monitor</code> role is particularly clever in its design. Rather than being a single monolithic permission set, it's actually a composite of the other three roles. This allows for granular access when needed - you can grant pg_read_all_stats for basic monitoring without including configuration access, or grant the full pg_monitor bundle for comprehensive monitoring capabilities.</p> PostgreSQL 11 (2018)</strong> laid the foundation for secure file system operations with the introduction of three powerful roles that fundamentally changed how PostgreSQL handles server-side file access:</p> pg_read_server_files</code></li> pg_write_server_files</code></li> pg_execute_server_program</code></li> </ul> Addressing many long-standing issues with providing file system access, like ETL processes that needs to read CSV files and data processing pipelines often involving external programs.</p> The file system roles represented a significant security advancement. Before PostgreSQL 11, operations like server-side COPY FROM file were superuser-only, forcing administrators to either grant excessive privileges or find workarounds.</p> PostgreSQL 14 (2021)</strong> introduced three significant roles that addressed different data access patterns:</p> pg_read_all_data</code></li> pg_write_all_data</code></li> pg_database_owner</code></li> </ul> The first two solved a common backup and analytics challenge. The special case is pg_database_owner</code> which we will cover later.</p> -- create ETL extraction user</span></span> CREATE USER</span> etl_extractor</span> WITH PASSWORD</span> 'extract_password'</span>;</span></span> GRANT</span> pg_read_all_data </span>TO</span> etl_extractor;</span></span> </span> -- create ETL loader with write access</span></span> CREATE USER</span> etl_loader</span> WITH PASSWORD</span> 'loader_password'</span>;</span></span> GRANT</span> pg_write_all_data </span>TO</span> etl_loader;</span></span></code></pre> One big caveat that comes with those ALL data roles is that they still (despite what the name might suggest) respect Row Level Security policies (RLS). This is design choice of PostgreSQL which prioritises security by default - even for users with broad data access. The only way to avoid RLS is to explicitly grant BYPASSRLS</code> role attribute.</p> PostgreSQL 15 (2022)</strong> expanded set with pg_checkpoint</code>, reflecting the reality that checkpoint operations are sometimes needed for maintenance but don't require full superuser privileges.</p> PostgreSQL 16 (2023)</strong> expanded the list by introduction of</p> pg_use_reserved_connections</code> - resolving a critical problem with high-traffic databases, where previously only superusers could access reserved connections. Introduction of this roles expanded the pool of potential users who might be able to resolve incidents or high-load situations. For example with pg_signal_backend</code> grant.</li> pg_create_subscription</code> - with ability to create Logical Replication</a> subscriptions and confirming the place of PostgreSQL in the distributed database scenarios.</li> </ul> PostgreSQL 17 (2024)</strong> finally saw pg_maintain</code>, role that had a turbulent development history (initially committed for PostgreSQL 16), allowing non-superusers to perform crucial maintenance tasks like VACUUM, ANALYZE, REINDEX, REFRESH MATERIALIZED VIEW, CLUSTER, and LOCK TABLE.</p> The pg_maintain</code> role is particularly valuable for:</p> Automated maintenance scripts that can now run with minimal privileges, reducing the security footprint of scheduled maintenance operations.</li> Operational teams managing multiple databases who need consistent maintenance capabilities without requiring superuser access to each database.</li> Cloud and managed environments where maintenance operations need to be delegated to operations staff without granting broader system access.</li> </ul> PostgreSQL 18 (2025)</strong> continues the trend with pg_signal_autovacuum_worker</code> expanding the ability for non-superusers to signal autovacuum workers specifically, enabling them to cancel vacuum operations on specific tables or terminate autovacuum sessions that may be causing performance issues during critical periods.</p> The Magic of pg_database_owner</a> </h2> If you paid attention, there's one role that stands apart from all others - pg_database_owner</code>. While it does not provide any specific privileges that wouldn't be possible without superuser access, it serves as a marker for the owner of the database. This role is crucial for maintaining database ownership and ensuring that the database is managed by the correct user.</p> Before PostgreSQL 15 you would have public</code> schema owned by postgres</code> user.</p> demo=> \dn</span></span> List of schemas</span></span> Name \| Owner</span></span> --------+----------</span></span> public \| postgres</span></span></code></pre> It came with the baggage of relaxed default permission, allowing every user (PUBLIC) to create objects. Making this a maintenance nightmare.</p> Starting with PostgreSQL 15 and above, the public</code> schema is owned by pg_database_owner</code> role.</p> demo=# \dn</span></span> List of schemas</span></span> Name \| Owner</span></span> --------+-------------------</span></span> public \| pg_database_owner</span></span></code></pre> Unlike any other roles, pg_database_owner</code> has a unique behaviour - it membership changes with the current database. As the code snippet above shows the "shape-shifting nature" of this role fixed the public</code> schema and complexity of the permissions associated with it.</p> Another special behaviour is that the role itself does not come with any permissions. Let that sink in - zero permissions. Its power only comes from what you grant it or inherit. It opens up system for elaborate template inheritance</em>.</p> -- define functionality in template1</span></span> \c template1 postgres</span></span> CREATE OR REPLACE FUNCTION</span> db_owner_stats</span>()</span> RETURN</span> ... </span>SECURITY</span> DEFINER;</span></span> GRANT EXECUTE ON FUNCTION</span> db_owner_stats()</span> TO</span> pg_database_owner;</span></span> </span> -- and let it automatically apply to all new databases</span></span> CREATE DATABASE</span> app_prod</span> OWNER</span> first_app;</span></span> CREATE DATABASE</span> demo_prod</span> OWNER</span> second_app;</span></span> </span> -- both first_app and second_app automatically inherit db_owner_stats function on their databases</span></span></code></pre> And when we are talking about magic, the special status of the role prevents this role to be granted to any other role.</p> demo</span>=</span>#</span> GRANT</span> pg_database_owner </span>TO</span> labs_app;</span></span> ERROR: </span>role</span> "pg_database_owner"</span> cannot have </span>explicit</span> members</span></span></code></pre>Conclusion</a> </h2> Predefined roles transform PostgreSQL administration from ad-hoc permission management into systematic capability delegation. They solve the fundamental problem of needing operational access without operational trust.</p> The evolution continues - each PostgreSQL release adds new predefined roles addressing real-world operational challenges. The pattern is established: identify common pain points, create focused roles, eliminate the need for superuser privileges in routine operations.</p> The next time you reach for superuser access to solve an operational problem, check if PostgreSQL has already provided a predefined role for exactly that purpose. You'll likely find it already has one.</p> Beyond the Basics of Logical Replication 2025-06-24T07:10:00+00:00 With First Steps with Logical Replication</a> we set up a basic working replication between a publisher and a subscriber and were introduced to the fundamental concepts. In this article, we're going to expand on the practical aspects of logical replication operational management, monitoring, and dive deep into the foundations of logical decoding.</p> Initial Data Copy</a> </h2> As we demonstrated in the first part, when setting up the subscriber, you can choose (or not to) to rely on initial data copy using the option WITH (copy_data = false)</code>. While the default copy is incredibly useful behavior, this default has characteristics you should understand before using it in a production environment.</p> The mechanism effectively asks the publisher to copy the table data by taking a snapshot (courtesy of MVCC), sending it to the subscriber, and thanks to the replication slot "bookmark," seamlessly continues streaming the changes from the point the snapshot was taken.</p> Simplicity</strong> is the key feature here, as a single command handles the snapshot, transfer, and transition to ongoing streaming.</p> The trade-off you're making is when it comes to performance</strong>, solely due to the fact that it's using a single process per table. Behind the scenes the initial synchronization is done using COPY</code> given the best possible performance. While it works almost instantly for majority of the tables, you will encounter notable delay and overhead when dealing with tables with gigabytes of data.</p> Although parallelism can be controlled by the max_sync_workers_per_subscription</code> configuration parameter, it still might leave you waiting for hours (and days) for any real-life database to get replicated. You can monitor whether the tables have already been synchronized or are still waiting/in progress using the pg_subscription_rel</code> catalog.</p> SELECT</span> srrelid::regclass </span>AS</span> table_name, srsubstate</span></span> FROM</span> pg_subscription_rel;</span></span></code></pre> Where each table will have one of the following states:</p> i</code> not yet started</li> d</code> copy is in progress</li> s</code> syncing (or waiting for confirmation)</li> r</code> done & replicating</li> </ul> Luckily, the state r</code> indicates that the streaming of changes can start even if not all tables are synchronized. Nevertheless, the replication slot retains the LSN position</strong>, which means that the publisher will retain WAL files</strong> until the subscriber has caught up.</p> Manual Synchronization for Production Workloads</a> </h2> As mentioned above, the implicit copy might bring performance trade-offs that are simply too much for production use, unless you consider it while designing your logical replication topology. In all other scenarios, going with manual synchronization is the way forward.</p> The entire process has one leading principle: the data you load manually must be consistent with the point in time from which the logical replication stream begins. This is achieved by creating a logical replication slot before (or synchronized with) the point at which you restore the data (PITR)</strong> and enabling it once the data is transferred. Only this will allow you to correctly apply all subsequent changes on the subscriber.</p> There are several ways to achieve this, and you will have to evaluate the mechanism based on the available constraints in your particular use case:</p> Use the backup & restore mechanism that supports Point-in-Time Recovery (PITR) if the amount of data to be synchronized approaches the total size of the backed-up data (i.e., most of the tables).</li> Orchestrate the data change ingestion on the publisher with the backup, by stopping the incoming changes (for example, during a planned maintenance window) and only restoring them when a consistent snapshot is available to a known LSN. This is for cases where you can control table ingestion and the amount of data being transferred fits the maintenance window.</li> If none of this is available, you might be able to manually advance replication slots to a predefined LSN. Please understand, this should be a last resort as it might be considered expert domain.</li> </ol> The recommended way for the most reliable backup and restore handling is the use of pgBackRest</strong>, as it allows you to restore a particular backup in advance and apply only changes needed later to advance to the selected Point-in-Time, hence significantly reducing the time required for the initial sync. On the other hand, if you are only using a small portion of the data within the backup on your subscriber, it might create resource constraints not acceptable for the setup.</p> While pg_dump can't be considered a reliable backup tool</strong>, it might help you in case you are talking about making a consistent backup of a small subset of tables using the "synchronized snapshot" created using pg_export_snapshot</code>.</p> pg_dump -d publisher_db --snapshot="000004A2-1" --data-only -Fc -f initial_data.dump</span></span></code></pre> Please note, the topic of consistent backups and Point-in-Time-Restore is way beyond this logical replication guide.</p> Monitoring Logical Replication</a> </h2> Once you set up your publishers and subscribers and have the initial data in place, it's time to start thinking about how to keep that process running.</p> For day-to-day operations, the basic building block for monitoring is the pg_replication_slots</code> catalog. Sample query:</p> SELECT</span></span> slot_name,</span></span> plugin,</span></span> slot_type,</span></span> active,</span></span> pg_size_pretty(</span></span> pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)</span></span> ) </span>AS</span> wal_retained_size</span></span> FROM</span> pg_replication_slots</span></span> WHERE</span></span> slot_type </span>=</span> 'logical'</span>;</span></span></code></pre> giving you result:</p> -[ RECORD 1 ]-----+-----------------------------</span></span> slot_name \| my_sample_subscription</span></span> plugin \| pgoutput</span></span> slot_type \| logical</span></span> active \| t</span></span> wal_retained_size \| 49 MB</span></span></code></pre> You should look for two things:</p> active</code> indicating whether the slot is being used by any connection. While this might be acceptable for shorter durations, it might be a sign that the subscriber is down or not configured properly.</li> The computed wal_retained_size</code> is a critical metric. An increasing value indicates problems consuming changes by the subscriber, and your cluster might be at risk of running out of disk space.</li> </ul> Both values are important as they might indicate problems you need to pay attention to. The active</code> flag might indicate dangling slots, where the subscriber has been destroyed (or simply wiped) without properly dropping the subscription—hence leaving the replication slot behind. In such cases, the only direct response is dropping the slot by its name:</p> SELECT</span> pg_drop_replication_slot(</span>'my_dangling_slot'</span>);</span></span></code></pre> If you want to dig deeper, you can also use pg_stat_replication</code> that provides more (albeit volatile) data about the replication status. It can give you other metrics, like replay_lag</code>.</p> To wrap up the monitoring part, you can either design your custom checks or use a modified version of the query above:</p> SELECT</span></span> slot_name,</span></span> CASE WHEN</span> active</span></span> THEN</span> 1</span></span> ELSE</span> 0</span></span> END as</span> active_status,</span></span> pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) </span>as</span> wal_bytes_retained,</span></span> extract(epoch </span>from now</span>()) </span>as time</span></span> FROM</span> pg_replication_slots</span></span> WHERE</span> slot_type </span>=</span> 'logical'</span>;</span></span></code></pre> as part of a Grafana alerting rule:</p> Alert Rule</span>:</span> "Inactive Replication Slot"</span></span> Condition</span>:</span></span> -</span> Query</span>:</span> active_status</span></span> -</span> Reducer</span>:</span> last()</span></span> -</span> Evaluator</span>:</span> below 1</span></span> -</span> For</span>:</span> 30m</span> # if inactive for 30+ minutes</span></span> </span> Additional conditions (AND)</span>:</span></span> -</span> Query</span>:</span> wal_bytes_retained</span></span> -</span> Reducer</span>:</span> last()</span></span> -</span> Evaluator</span>:</span> above 1073741824</span> # 1GB limit</span></span> </span></code></pre> Nevertheless, you need to evaluate the monitoring details applicable for your particular use case(s). There might also be other options, for example, if your architecture allows for it, to use max_slot_wal_keep_size</a> and mark the replication slot invalid and release the WAL files when a particular slot falls behind the configured value (for example, 100GB). While this might sound reasonable, you must considered whatever your application or use case is able to recover from such a data loss.</p> Evolving Publications</a> </h2> Going back to the first part of our journey into logical replication, there's an important distinction to make. While we have described the initial use case, in real-life your publisher schema will change as time goes by. Tables will be created and included implicitly (via FOR ALL TABLES</code>) or explicitly added/removed.</p> ALTER</span> PUBLICATION my_publication </span>ADD TABLE</span> public</span>.</span>new_table</span>;</span></span> ALTER</span> PUBLICATION my_publication</span> DROP TABLE</span> public</span>.old_table;</span></span></code></pre> But what happens when you add a new table? For the existing subscription, the answer is not much at all. The only way for the subscriber to reflect the changes in the underlying configuration is to refresh the subscription</strong>. It's a process where PostgreSQL compares the current subscription table list with the publication table list, (if configured to do so) starts the initial synchronization process and once finished goes to apply streamed changes.</p> As mentioned, you can refresh a subscription with or without copying the initial data. The copy_data</code> option controls this behavior and is currently the only supported setting.</p> ALTER</span> SUBSCRIPTION my_subscription</span></span> REFRESH PUBLICATION;</span></span> </span> ALTER</span> SUBSCRIPTION my_subscription</span></span> REFRESH PUBLICATION </span>WITH</span> (copy_data </span>=</span> false);</span></span></code></pre> When you remove a table from the publisher, the situation is much simpler as the changes for that particular table are no longer considered for logical decoding.</p> But what happens if you were to add a new table to the publication without refreshing the subscription? Despite the WAL files including changes for all tables, and the logical decoding process will process all published table changes</strong>, there will be no additional WAL retention to consider. The subscriber will advance confirmed_flush_lsn</code> as before, simply because the initial state for a newly added table has not yet been recorded and streamed changes will be (for that moment) ignored.</p> Logical Decoding</a> </h2> If you paid attention throughout our journey or already worked a bit with logical replication, you might have come across something called the pgoutput</code> plugin. This is the built-in default mechanism responsible for logical decoding—a mechanism that goes through the WAL stream and transforms it into a higher-level format.</p> An example is, instead of physical replication—byte X at offset Y, in page Z—the logical decoding translates the WAL stream to row-level changes. It uses the context awareness of the source database to understand the changes and change from object identifier to the schema, table, and column name, as well as the respective values (both old and new). It also provides a way to assemble the changes into the correct transaction flow.</p> The plugin's job is to format that output in a particular way and by itself is not aware of:</p> any transport layer details</li> replication slots</li> any retry logic</li> </ul> PostgreSQL offers a robust framework for logical decoding. The built-in plugins are just a start, and developers can create custom output plugins. But not only that—we can also look inside the logical decoded stream manually.</p> Let's consider this example:</p> -- create new publication</span></span> CREATE</span> PUBLICATION all_data </span>FOR</span> ALL TABLES;</span></span> </span> -- create new replication slot with pgoutput plugin</span></span> SELECT</span> pg_create_logical_replication_slot(</span>'plugin_demo'</span>, </span>'pgoutput'</span>);</span></span> </span> -- sample table</span></span> CREATE TABLE</span> IF</span> NOT EXISTS</span> demo_table (</span></span> id </span>INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name TEXT</span>,</span></span> email </span>TEXT</span></span> );</span></span> </span> -- generate some changes with insert/update/delete</span></span> INSERT INTO</span> demo_table (</span>name</span>, email) </span>VALUES</span> (</span>'John Doe'</span>, </span>'john@example.com'</span>);</span></span> UPDATE</span> demo_table </span>SET</span> email </span>=</span> 'doe@example.com'</span> WHERE</span> id </span>=</span> 1</span>;</span></span> DELETE FROM</span> demo_table </span>WHERE</span> id </span>=</span> 1</span>;</span></span> </span> -- get binary changes</span></span> SELECT * FROM</span> pg_logical_slot_get_binary_changes(</span>'plugin_demo'</span>, </span>NULL</span>, </span>NULL</span>, </span>'proto_version'</span>, </span>'1'</span>, </span>'publication_names'</span>, </span>'all_data'</span>);</span></span> </span> -- drop the replication slot and the publication</span></span> SELECT</span> pg_drop_replication_slot(</span>'plugin_demo'</span>);</span></span> DROP TABLE</span> demo_table;</span></span> DROP</span> PUBLICATION all_data;</span></span></code></pre> Giving us a peek into the recently performed changes (output is shortened):</p> lsn \| xid \| data</span></span> ------------+------+---------------------------------</span></span> 1/D6997380 \| 1038 \| \x4200000001d69974180</span></span> 1/D6997380 \| 1038 \| \x52000042057075626c696300646...</span></span> 1/D6997380 \| 1038 \| \x49000042054e000374000...</span></span> 1/D6997448 \| 1038 \| \x430000000001d6997...</span></span> 1/D6997448 \| 1039 \| \x4200000001d6997...</span></span> 1/D6997448 \| 1039 \| \x49000042054e0....</span></span> 1/D6997510 \| 1039 \| \x430000000001d6...</span></span></code></pre> Which is correct, but won't help us to demonstrate the flow. Don't forget pgoutput</code> is the actual plugin used for logical decoding, and is binary. But don't despair, PostgreSQL offers the test_decoding</code> plugin which decodes the changes into a human-readable text representation.</p> -- setup the publication</span></span> CREATE</span> PUBLICATION all_data </span>FOR</span> ALL TABLES;</span></span> </span> -- create replication slot with test_decoding plugin</span></span> SELECT</span> pg_create_logical_replication_slot(</span>'plugin_demo'</span>, </span>'test_decoding'</span>);</span></span> </span> CREATE TABLE</span> IF</span> NOT EXISTS</span> demo_table (</span></span> id </span>INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name TEXT</span>,</span></span> email </span>TEXT</span></span> );</span></span> </span> -- generate some changes with insert/update/delete</span></span> INSERT INTO</span> demo_table (</span>name</span>, email) </span>VALUES</span> (</span>'John Doe'</span>, </span>'john@example.com'</span>);</span></span> UPDATE</span> demo_table </span>SET</span> email </span>=</span> 'doe@example.com'</span> WHERE</span> id </span>=</span> 1</span>;</span></span> DELETE FROM</span> demo_table </span>WHERE</span> id </span>=</span> 1</span>;</span></span> </span> -- get human readable changes</span></span> SELECT * FROM</span> pg_logical_slot_get_changes(</span>'plugin_demo'</span>, </span>NULL</span>, </span>NULL</span>);</span></span> </span> -- drop the replication slot and the publication</span></span> SELECT</span> pg_drop_replication_slot(</span>'plugin_demo'</span>);</span></span> DROP TABLE</span> demo_table;</span></span> DROP</span> PUBLICATION all_data;</span></span></code></pre> This time the output is actually going to be helpful.</p> lsn \| xid \| data</span></span> ------------+------+-----------------------------------------------------------------------------------------------------</span></span> 1/D69CCC30 \| 1048 \| BEGIN 1048</span></span> 1/D69CCC98 \| 1048 \| table public.demo_table: INSERT: id[integer]:1 name[text]:'John Doe' email[text]:'john@example.com'</span></span> 1/D69CCDC0 \| 1048 \| COMMIT 1048</span></span> 1/D69CCDC0 \| 1049 \| BEGIN 1049</span></span> 1/D69CCDC0 \| 1049 \| table public.demo_table: UPDATE: id[integer]:1 name[text]:'John Doe' email[text]:'doe@example.com'</span></span> 1/D69CCE50 \| 1049 \| COMMIT 1049</span></span> 1/D69CCE50 \| 1050 \| BEGIN 1050</span></span> 1/D69CCE50 \| 1050 \| table public.demo_table: DELETE: id[integer]:1</span></span> 1/D69CCEC0 \| 1050 \| COMMIT 1050</span></span></code></pre> Just reading it, you can easily follow the sequence of the transaction (auto-commit in psql) and track individual changes.</p> We can also reiterate the importance of REPLICA IDENTITY</code> with this example. If you double-check the DDL for demo_table</code>, you will see IDENTITY</code> used, which by its definition will create a primary key. With default replica identity, you are effectively relying on that as the primary source of identifying data. You can use the test_decoding</code> plugin to demonstrate the verbosity it will create.</p> (In this example, leaving the setup and tear down.)</p> ALTER TABLE</span> demo_table </span>REPLICA IDENTITY</span> FULL;</span></span> </span> INSERT INTO</span> demo_table (id, </span>name</span>, email) OVERRIDING </span>SYSTEM VALUE VALUES</span> (</span>999</span>, </span>'John Doe'</span>, </span>'john@example.com'</span>);</span></span> UPDATE</span> demo_table </span>SET</span> email </span>=</span> 'doe@example.com'</span> WHERE</span> id </span>=</span> 999</span>;</span></span> DELETE FROM</span> demo_table </span>WHERE</span> id </span>=</span> 999</span>;</span></span> </span> SELECT * FROM</span> pg_logical_slot_get_changes(</span>'plugin_demo'</span>, </span>NULL</span>, </span>NULL</span>);</span></span></code></pre> And you can observe the changes. While by its nature INSERT</code> gives you the same output (all values are being sent), UPDATE</code> and DELETE</code> are no longer relying on the primary key. Instead, they have to provide all values, for the UPDATE</code> case both old and new data:</p> lsn \| 1/D69F8770</span></span> xid \| 1062</span></span> data \| table public.demo_table: UPDATE: old-key: id[integer]:999 name[text]:'John Doe' email[text]:'john@example.com' new-tuple: id[integer]:999 name[text]:'John Doe' email[text]:'doe@exa</span></span> mple.com'</span></span></code></pre> and for DELETE</code> the old data.</p> lsn \| 1/D69F8828</span></span> xid \| 1063</span></span> data \| table public.demo_table: DELETE: id[integer]:999 name[text]:'John Doe' email[text]:'doe@example.com'</span></span></code></pre> Making it obvious how misconfigured replica identity might increase the amount of data decoded and replicated.</p> Fine-grained Publication Control</a> </h2> Before we introduce the advanced replication topologies in later parts of this series, let's have a look at how you can precisely control what data to replicate to the subscribers. Fine-grained control helps you to control replication overhead, reduce network traffic, enhance security, and ensure the subscribers only receive the data they need.</p> The first option to filter the data is by explicit column_list</code>, allowing you to exclude sensitive (PII or similar) or unnecessary data. When selecting the columns, you should include the primary key or the columns behind replica identity; if not, PostgreSQL will add them automatically.</p> CREATE</span> PUBLICATION hr_analytics</span></span> FOR TABLE</span> hr</span>.</span>employees</span> (employee_id, first_name, last_name, department, </span>start_date</span>);</span></span></code></pre> Similar to column selection, you can control what operations a publication replicates. The available options are 'insert', 'update', 'delete', and 'truncate'.</p> CREATE</span> PUBLICATION hr_analytics</span></span> FOR TABLE</span> hr</span>.</span>employees</span> (employee_id, first_name, last_name, department, </span>start_date</span>)</span></span> WITH</span> (publish </span>=</span> 'insert,delete'</span>);</span></span></code></pre>Securing Logical Replication</a> </h2> In the previous section, we hit a crucial element of logical replication—limiting access to the publication. Otherwise, what purpose would it have to limit fields published if another publication would still offer them without restrictions? So far in this series, we have relied for simplicity on SUPERUSER</code> access to create and manage publications. To design logical replication for real-life production use, you need a least-privilege model.</p> Luckily for us, the PostgreSQL security model for logical replication follows the same rules as for regular queries. If a user can't SELECT from a specific table, or specific columns or rows, they can't include them in a publication either.</strong> Therefore, this section assumes you are already familiar with the PostgreSQL permission model.</p> The high-level overview of the permissions needed is that the replication user must have CONNECT</code> permission on the database, USAGE</code> on the schema, and SELECT</code> on specific tables (columns). You also need to ensure (if applicable) row-level security (RLS) is correctly managed.</p> Using the fine-grained example used above, we can demonstrate the setup. First, creating a role with LOGIN</code> and REPLICATION</code> clauses and granting database and schema access:</p> CREATE ROLE</span> hr_analytics_role </span>WITH LOGIN</span> REPLICATION </span>PASSWORD</span> 'a_strong_password'</span>;</span></span> </span> GRANT CONNECT ON DATABASE</span> my_publisher_db </span>TO</span> hr_analytics_role;</span></span> GRANT</span> USAGE </span>ON SCHEMA</span> hr </span>TO</span> hr_analytics_role;</span></span></code></pre> The crucial step is to correctly define the GRANT</code> for selecting the data.</p> GRANT SELECT</span></span> (employee_id, first_name, last_name, department, hire_date)</span></span> ON</span> hr</span>.</span>employees</span> TO</span> hr_analytics_role;</span></span></code></pre> This approach ensures that even if someone gains access to the replication role, they cannot expose data beyond what was explicitly granted. The publication will respect these column-level restrictions, creating a secure boundary for your replicated data.</p> Wrap Up</a> </h2> Today we've moved beyond the basics of logical replication to cover the essential practices for production environments and expanded the understanding of how it actually works.</p> This article is second part of the upcoming guide Mastering Logical Replication in PostgreSQL</a>. If you are interested in the topic, please consider subscribing to get the latest articles as they are published.</p> First steps with Logical Replication in PostgreSQL 2025-06-11T00:00:00+00:00 Most applications start with a single PostgreSQL database, but over time, the need to scale out, distribute the load, or integrate naturally arises. PostgreSQL's logical replication is one of the features that meets these demands by streaming row-level changes</strong> from one PostgreSQL instance to another, all using a publish-subscribe</strong> model. Logical replication is more than an advanced feature; it provides a flexible framework you can build on to further distribute and integrate PostgreSQL within your architecture.</p> In this article, we will start with the foundation, explore the core ideas behind logical replication, and learn how to use it.</p> Physical vs. Logical Replication</a> </h2> Before we can dive deeper, let's understand the role of replication in PostgreSQL and how it's built on top of the Write-Ahead Log (WAL)</strong>.</p> The WAL is a sequential, append-only log that records every change made to the cluster data. For durability purposes, all modifications are first written to the WAL and only then permanently written to disk. This allows PostgreSQL to recover from crashes by replaying logged changes.</p> Versioned changes, necessitated by concurrent transactions, are managed through Multi-Version Concurrency Control (MVCC)</strong>. Instead of overwriting data directly, MVCC creates multiple versions of rows, allowing each transaction to see a consistent snapshot of the database. It is the WAL that captures these versioned changes along with the transactional metadata to ensure data consistency at any given point in time.</p> Physical replication</strong> is built directly on the Write-Ahead Log. It enables streaming of the binary WAL data from the primary server to one or more standby servers (replicas), effectively creating a byte-for-byte copy of the entire cluster. This requirement makes the replicas read-only, making them ideal candidates for failover or scaling purposes.</p> Compared to this, Logical replication</strong>, while also being built on top of the WAL data, takes a fundamentally different approach. Instead of streaming raw change data, logical replication decodes the WAL into logical, row-level changes</strong> – such as INSERT</code>, UPDATE</code>, and DELETE</code> – and only then sends them to the subscribers using a Publish-Subscribe model. Compared to physical replication, this allows selective replication, while allowing writable subscribers which are not strictly tied to a single publisher. This might increase the flexibility of available setups, however logical replication does not replicate DDL changes</strong>.</p> </th> Physical</th> Logical</th></tr></thead> Data Streamed</strong></td> Binary WAL segments</td> Row-level SQL changes</td></tr> Scope</strong></td> Byte-for-byte stream</td> Selected tables</td></tr> Node Type</strong></td> Read-only standby</td> Fully writable instance</td></tr> PostgreSQL Version</strong></td> All servers must match major version</td> Supports across versions</td></tr> Database Schema</strong></td> Changes automatically replicated</td> Changes must be applied on subscriber(s) separately</td></tr> Use Case</strong></td> Failover, high-availability, and read scaling</td> Integration, zero-downtime upgrades and schema migrations, complex topologies</td></tr> </tbody></table> Physical replication is your go-to for high availability</strong> and disaster recovery</strong>, where you want a fast, exact copy of the entire database cluster that can take over in case of failure. It’s simple to set up and very efficient but limited in flexibility.</p> Logical replication shines when you need fine-grained control</strong> over what data is replicated, require writable replicas</strong>, or want to integrate</strong> PostgreSQL with other systems or versions. It’s ideal for zero-downtime upgrades</strong>, multi-region deployments</strong>, and building scalable, modular architectures</strong>.</p> Setting up Logical Replication</a> </h2> Enough of boring theory for now. To get started, you will need two instances of PostgreSQL – it's up to you whether you provision two virtual machines or two clusters running on the same computer. We will call one publisher</strong> and the other subscriber</strong>.</p> Preparing the Publisher</a> </h3> First, you need to prepare the publisher to emit the logical changes. You will need to modify postgresql.conf</code> (or add particular conf.d</code> configuration) with the following parameters:</p> wal_level = logical</span></span> max_replication_slots = 10</span></span> max_wal_senders = 10</span></span></code></pre> Your publisher also needs to be reachable by the subscriber, in most cases via a TCP socket.</p> listen_addresses = ''</span></span></code></pre> Here, the configuration is:</p> wal_level</code></a> set to logical</code> is a crucial piece of the configuration. It tells PostgreSQL how much information to write to the WAL, in this case, to support logical decoding.</li> max_replication_slots</code></a> defines the maximum number of replication slots that can be created on the server. Each logical replication subscription needs its own slot.</li> max_wal_senders</code></a> should be set high enough to accommodate all expected concurrent replication connections from subscribers. Each active replication subscription consumes a wal_sender slot.</li> </ul> should match the maximum number of connections from the publisher (primary) to subscribers or replication clients. The number should be higher than or equal to the number of replication slots to avoid replication problems.</p> You will also need to configure client authentication in pg_hba.conf</code> to allow the subscriber to connect for replication (for simplification, we allow all users):</p> # TYPE DATABASE USER ADDRESS METHOD</span></span> host replication all subscriber_ip/32 scram-sha-256</span></span></code></pre> While for the purposes of the article we will assume the use of a superuser, making it convenient:</p> CREATE USER</span> my_user_name</span> SUPERUSER </span>PASSWORD</span> 'my_secure_password'</span>;</span></span></code></pre> It is recommended to use a dedicated replication user for real-life deployments:</p> CREATE USER</span> replication_user</span> WITH</span> REPLICATION </span>ENCRYPTED PASSWORD</span> 'my_secure_password'</span>;</span></span></code></pre> Once configured, restart PostgreSQL for the configuration changes to take effect. After that, connect to the server and prepare the environment.</p> -- example for psql</span></span> </span> CREATE DATABASE</span> logical_demo_publisher</span>;</span></span> \c logical_demo_publisher;</span></span></code></pre> Create a sample table and seed initial data:</p> CREATE TABLE</span> products</span> (</span></span> id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name TEXT NOT NULL</span>,</span></span> category </span>TEXT</span>,</span></span> price </span>DECIMAL</span>(</span>10</span>, </span>2</span>) </span>NOT NULL</span>,</span></span> stock_quantity </span>INT DEFAULT</span> 0</span>,</span></span> created_at </span>TIMESTAMP WITH TIME ZONE DEFAULT</span> CURRENT_TIMESTAMP,</span></span> updated_at </span>TIMESTAMP WITH TIME ZONE DEFAULT</span> CURRENT_TIMESTAMP,</span></span> description TEXT</span>,</span></span> is_active </span>BOOLEAN DEFAULT</span> TRUE</span></span> );</span></span> </span> -- seed some data (10 records)</span></span> INSERT INTO</span> products (</span></span> name</span>,</span></span> category,</span></span> price,</span></span> stock_quantity,</span></span> description</span>,</span></span> is_active</span></span> )</span></span> SELECT</span></span> 'Product Batch '</span> \|\|</span> s</span>.</span>id</span> AS name</span>,</span></span> CASE</span> (</span>s</span>.</span>id</span> % </span>5</span>)</span></span> WHEN</span> 0</span> THEN</span> 'Electronics'</span></span> WHEN</span> 1</span> THEN</span> 'Books'</span></span> WHEN</span> 2</span> THEN</span> 'Home Goods'</span></span> WHEN</span> 3</span> THEN</span> 'Apparel'</span></span> ELSE</span> 'Miscellaneous'</span></span> END AS</span> category,</span></span> ROUND</span>((RANDOM()</span> </span> 500</span> +</span> 10</span>)::</span>numeric</span>, </span>2</span>) </span>AS</span> price,</span></span> FLOOR</span>(RANDOM()</span> </span> 200</span>)::</span>int AS</span> stock_quantity,</span></span> 'Auto-generated description for product ID '</span> \|\|</span> s</span>.</span>id</span> \|\|</span> '. Lorem ipsum dolor sit amet, consectetur adipiscing elit.'</span> AS description</span>,</span></span> (</span>s</span>.</span>id</span> % </span>10</span> <></span> 0</span>) </span>AS</span> is_active</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>10</span>) </span>AS</span> s(id);</span></span></code></pre> The final step for the publisher</strong> is to create a publication to define what data we want to publish.</p> CREATE</span> PUBLICATION my_publication </span>FOR TABLE</span> products;</span></span></code></pre>Preparing the Subscriber</a> </h3> Next, we will use our subscriber</strong> instance to receive the logical changes. There's no need to make any configuration changes just</em> for subscribing, as it's not emitting changes itself (please note the "just").</p> And create the target database and schema. The schema creation is an important part as:</p> -- example for psql</span></span> </span> CREATE DATABASE</span> my_subscriber_db</span>;</span></span> \c my_subscriber_db;</span></span> </span> CREATE TABLE</span> products</span> (</span></span> id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name TEXT NOT NULL</span>,</span></span> category </span>TEXT</span>,</span></span> price </span>DECIMAL</span>(</span>10</span>, </span>2</span>) </span>NOT NULL</span>,</span></span> stock_quantity </span>INT DEFAULT</span> 0</span>,</span></span> created_at </span>TIMESTAMP WITH TIME ZONE DEFAULT</span> CURRENT_TIMESTAMP,</span></span> updated_at </span>TIMESTAMP WITH TIME ZONE DEFAULT</span> CURRENT_TIMESTAMP,</span></span> description TEXT</span>,</span></span> is_active </span>BOOLEAN DEFAULT</span> TRUE</span></span> );</span></span></code></pre> Now we have the foundation for testing logical replication. The simplest way is to create a subscription using connection details and the name of the publication to subscribe to.</p> CREATE</span> SUBSCRIPTION my_subscription</span></span> CONNECTION</span> 'host=publisher_ip_address port=5432 user=your_replication_user password=my_secure_password dbname=logical_demo_publisher'</span></span> PUBLICATION my_publication;</span></span></code></pre> my_subscription</code> should be a descriptive name for your subscription.</li> CONNECTION</code> defines how to connect to the publisher (which is a regular connection string and could also be a service</a>).</li> PUBLICATION</code> specifies the name of the publication you created earlier on the publisher.</li> </ul> If your connection was correct, the subscription will by default start with the initial data sync and listen for incoming changes. If you have used the queries above, you can validate that the data is now on the subscriber.</p> #</span> SELECT</span> count</span>(</span>1</span>) </span>FROM</span> products;</span></span> count</span></span> ---------</span></span> 10</span></span> (</span>1</span> row</span>)</span></span></code></pre> The number of records should match the number of records created using generate_series</code> above (10 in our example). You can go ahead and insert a single row again on your publisher instance and validate the data being replicated to the subscriber.</p> Congratulations! You have set up your first logical replication in PostgreSQL!</p> Core Concepts of Logical Replication</a> </h2> That was easy, right? And that's the goal. Now that you've seen logical replication in action, let's delve deeper into the core concepts that made this work.</p> Publication</a> </h3> As the name implies, a publication</strong> is where it all starts. It's essentially a catalogue of data you offer from the publisher. You can publish</a> a number of objects:</p> --- specific tables</span></span> CREATE</span> PUBLICATION my_publication </span>FOR TABLE</span> products, orders;</span></span> </span> --- everything in your database (make sure you really want to do this)</span></span> CREATE</span> PUBLICATION all_data </span>FOR</span> ALL TABLES;</span></span> </span> --- replicate specific columns only (PostgreSQL 15 and higher)</span></span> CREATE</span> PUBLICATION generic_data </span>FOR TABLE</span> products (id, </span>name</span>, price);</span></span> </span> --- filter published data (PostgreSQL 15 and higher)</span></span> CREATE</span> PUBLICATION active_products </span>FOR TABLE</span> products </span>WHERE</span> (is_active </span>=</span> true);</span></span> </span> --- filter different operations</span></span> CREATE</span> PUBLICATION my_publication </span>FOR TABLE</span> products, orders</span></span> WITH</span> (publish </span>=</span> 'insert'</span>);</span></span> </span> --- or you can mix & match it to get exactly what you need</span></span> CREATE</span> PUBLICATION eu_customers </span>FOR</span></span> TABLE</span> customers (id, email, country, created_at)</span></span> WHERE</span> (country </span>IN</span> (</span>'DE'</span>, </span>'FR'</span>, </span>'IT'</span>)),</span></span> TABLE</span> orders (id, customer_id, total_amount)</span></span> WHERE</span> (total_amount </span>></span> 100</span>)</span></span> WITH</span> (publish </span>=</span> 'insert, update'</span>);</span></span></code></pre> As you can see, logical replication really gives you quite a lot of options and is far from the rigid, byte-for-byte copying</strong> of physical replication. This is exactly the characteristic that allows you to build complex topologies.</p> Once you set up your publication(s), you can review it:</p> --- in psql</span></span> \dRp</span></span> \dRp+ my_publication</span></span> </span> --- using pg_catalog</span></span> SELECT FROM pg_publication_tables WHERE pubname = 'my_publication';</span></span></code></pre> From these options, it's easy to think of a publication as a customisable data feed.</p> Subscription</a> </h3> The other side of logical replication is a subscription</strong>. It defines what and how to consume events from a publisher. You subscribe to a publication using connection details and a number of options.</p> --- basic subscription with full connection detail</span></span> CREATE</span> SUBSCRIPTION my_subscription</span></span> CONNECTION</span> 'host=publisher_host port=5432 user=repl_user password=secret dbname=source_db'</span></span> PUBLICATION my_publication;</span></span> </span> -- subscription using service definition</span></span> CREATE</span> SUBSCRIPTION realtime_only</span></span> CONNECTION</span> 'service=my_source_db'</span></span> PUBLICATION my_publication;</span></span></code></pre> The default behaviour of the subscription is to copy existing data</strong>, start immediately</strong>, and continue with streaming data changes. Once you start experimenting with logical replication, you can control that behaviour.</p> --- subscribe without initial copy of the data</span></span> CREATE</span> SUBSCRIPTION streaming_only</span></span> CONNECTION</span> 'service=my_source_db'</span></span> PUBLICATION my_publication</span></span> WITH</span> (copy_data </span>=</span> false);</span></span> </span> --- or defer the start for later</span></span> CREATE</span> SUBSCRIPTION manual_start</span></span> CONNECTION</span> 'service=my_source_db'</span></span> PUBLICATION my_publication</span></span> WITH</span> (</span>enabled =</span> false);</span></span></code></pre> You can monitor your subscriptions:</p> --- in psql</span></span> \dRs</span></span> \dRs+ my_subscription</span></span> </span> --- or their status using pg_catalog</span></span> SELECT subname, received_lsn, latest_end_lsn, latest_end_time</span></span> FROM pg_stat_subscription;</span></span></code></pre>Replication Slot</a> </h3> Publications and subscriptions establish the data flow, but there's a critical piece missing: how does the publisher keep track of multiple subscribers reading the WAL at different speeds? Replication slots</strong> are the answer. They act as a persistent booking in the WAL stream that tracks exactly where each subscriber is (or was last time), ensuring no changes are lost</strong> even if the connection stops.</p> Let's have a look at what a replication slot can tell us about itself.</p> #</span> SELECT * FROM</span> pg_replication_slots;</span></span> -</span>[ RECORD 1 ]</span>-------+---------------</span></span> slot_name \| my_subscription</span></span> plugin \| pgoutput</span></span> slot_type \| logical</span></span> datoid \| </span>16390</span></span> database</span> \| my_db_source</span></span> temporary \| f</span></span> active \| t</span></span> active_pid \| </span>3162</span></span> xmin \|</span></span> catalog_xmin \| </span>777</span></span> restart_lsn \| </span>0</span>/</span>1DEFBF0</span></span> confirmed_flush_lsn \| </span>0</span>/</span>1DEFC28</span></span> wal_status \| reserved</span></span> safe_wal_size \|</span></span> two_phase \| f</span></span> inactive_since \|</span></span> conflicting \| f</span></span> invalidation_reason \|</span></span> failover</span> \| f</span></span> synced \| f</span></span></code></pre> The most interesting attributes are:</p> slot_name</code></li> plugin</code> used for logical replication</li> slot_type</code> confirming it's for logical replication</li> active</code> indicates whether there's a subscriber reading from this slot</li> </ul> And most important of those being:</p> restart_lsn</code> identifying the LSN where this slot "holds" WAL files from being released</li> confirmed_flush_lsn</code> being the last LSN position the subscriber confirmed it has successfully processed (i.e., applied).</li> </ul> While we won't go into details about WAL, it's enough to say LSN (Log Sequence Number) is a unique address or position</strong> in the WAL that identifies the position in the stream of database changes.</p> In most cases, you don't have to create replication slots manually</strong> – subscriptions create and manage them. The persistent nature of the slots guarantees they survive the restart of both publisher and subscriber.</p> Please note</em>, if the subscription is not actively consuming the changes, the publisher PostgreSQL won't release WAL files that contain changes a slot has not consumed. This prevents data loss, but can easily fill up disk if any of the subscribers fall too far behind.</strong></p> You can monitor the WAL retention per each replication slot.</p> #</span> SELECT</span></span> slot_name,</span></span> active,</span></span> pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) </span>AS</span> wal_retained</span></span> FROM</span> pg_replication_slots;</span></span> </span> -</span>[ RECORD 1 ]</span>+</span>--------------------</span></span> slot_name \| my_subscription</span></span> active \| t</span></span> wal_retained \| </span>265</span> MB</span></span></code></pre>Replication Identity</a> </h3> As we have already covered, logical replication works with row-level changes, such as INSERT</code>, UPDATE</code>, and DELETE</code>. And not all operations are equal. While INSERT</code> is relatively straightforward (a new row is sent to the subscriber), for both UPDATE</code> and DELETE</code> operations, PostgreSQL needs to be able to uniquely identify the target row to modify on the subscriber.</p> This is managed by the REPLICA IDENTITY</code> property of each published table. By default, if a table has a primary key (it has, right?), or you can specify it USING INDEX</code> for a unique index, or as an alternative, specify FULL</code> (should be generally avoided) to all old column's row values to find the target row to update. Please note, PostgreSQL might default to REPLICA IDENTITY FULL</code> if no primary key or suitable key index exists.</p> Whenever possible, please always:</p> Choose REPLICA IDENTITY DEFAULT</code> (for primary key), giving you the most efficient choice. If you don't have a primary key, please consider using it (a surrogate index is always an option).</li> REPLICA IDENTITY USING INDEX index_name</code> if there would be a better (or smaller) unique index matching the business logic or architecture of your database model.</li> </ul> There are special cases of what not to do when it comes to replication identity, but we will touch on those cases in the advanced section of this guide.</p> Schema Changes</a> </h3> Although it was already mentioned, it's vital to reiterate that logical replication, as it streams only row-level changes rather than full WAL segments, does not automatically replicate Data Definition Language (DDL) changes</strong>.</p> Any schema changes made on the publisher must be manually applied to all subscribers</strong> before data reflecting the change is replicated.</p> We will cover schema specifically in later parts of this guide, but to sum it up, this is the recommended order of operations when it comes to schema changes:</p> (Optional) but recommended for breaking changes, pause replication where applicable.</li> Apply and verify DDL changes to the subscribers.</li> Apply and verify DDL changes on the publisher.</li> Resume the replication if applicable.</li> </ol> While this might seem like a major drawback compared to physical replication, it's the characteristic that gives us the most flexibility.</p> Other Considerations</a> </h3> As logical replication reliably handles row-level changes, it's crucial to understand other specific limitations and behaviours. Specifically:</p> It's important to understand that sequences</strong> are not replicated. While they are used to generate the value on the publisher, and their value advances there, the corresponding sequence (if it exists) on the subscriber won't be updated.</li> Other database objects won't be replicated. While it might be understandable for views, stored procedures, triggers, and rules, you also can't rely on it for materialized views.</li> Special consideration (similar to schema changes) must be paid to user-defined data types. If a replicated table uses a user-defined type (e.g., a column using an ENUM</code>), the type must already be available on the subscriber(s) and have exactly the same name and structure. As ENUM</code>s are represented as ordered sets, even the order of the values matters!</li> </ul> Living with Logical Replication</a> </h2> In our previous example, we set up the publisher and subscriber, relied on the initial copy of the data, and let things run. Both of them will survive a restart, and thanks to the replication slot, they will keep going as soon as they come online.</p> But what if you need to perform maintenance or do regular things like a schema update on the subscriber or the publisher? Sometimes you might need to temporarily stop the replication</strong>. PostgreSQL makes this straightforward with subscription management, allowing you to stop consuming the changes.</p> ALTER</span> SUBSCRIPTION my_subscription </span>DISABLE</span>;</span></span></code></pre> As we hinted earlier, when disabled, the subscription stops consuming changes from the publisher, while keeping the replication slot. This means:</p> No new changes are applied on the subscriber.</li> WAL files will start to accumulate on the publisher (since the replication slot holds its position).</li> </ul> While disabled, you can't really check the status from the subscriber, as only the replication slot has the data. You can use the query we mentioned above on the publisher to check the status.</p> SELECT</span></span> slot_name,</span></span> active,</span></span> pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) </span>AS</span> wal_retained</span></span> FROM</span> pg_replication_slots;</span></span></code></pre> When ready, you can resume the subscription.</p> ALTER</span> SUBSCRIPTION my_subscription </span>ENABLE</span>;</span></span></code></pre>Coordinating Schema Changes</a> </h2> As we mentioned earlier, since logical replication does not automatically replicate DDL changes, you need to coordinate schema updates manually. Here's a basic overview of how to perform such a change, step by step.</p> First, disable the replication on the subscriber and apply your schema changes.</p> --- disable replication</span></span> ALTER</span> SUBSCRIPTION my_subscription </span>DISABLE</span>;</span></span> </span> --- apply changes on the subscriber</span></span> ALTER TABLE</span> products </span>ADD</span> COLUMN category_id </span>INTEGER</span>;</span></span> CREATE INDEX</span> products_by_category</span> ON</span> products(category_id);</span></span></code></pre> Only then can you apply the changes to the publisher.</p> ALTER TABLE</span> products </span>ADD</span> COLUMN category_id </span>INTEGER</span>;</span></span> CREATE INDEX</span> products_by_category</span> ON</span> products(category_id);</span></span></code></pre> And resume the replication.</p> ALTER</span> SUBSCRIPTION my_subscription </span>ENABLE</span>;</span></span></code></pre> If you applied the schema change to the publisher first, any new data using the new column would fail to replicate to the subscriber that doesn't have the column yet.</p> Handling Incompatible Changes</a> </h2> PostgreSQL's default conflict resolution is very simple. When a conflict occurs, logical replication stops</strong>, and the subscription enters an error state. For example, if you consider a product row already present on the subscriber, you will see something like this in the log files:</p> ERROR: duplicate key value violates unique constraint "products_pkey"</span></span> DETAIL: Key (id)=(452) already exists.</span></span> CONTEXT: processing remote data for replication origin "pg_16444" during message type "INSERT" for replication target relation "public.products" in transaction 783, finished at 0/1E020E8</span></span> LOG: background worker "logical replication apply worker" (PID 457) exited with exit code 1</span></span></code></pre> As you can see, logical replication stopped with an error code and won't continue until you manually resolve the conflict. To do so, you need to identify the conflicting data and decide what to do with it.</p> -- option 1: remove the conflicting data</span></span> DELETE FROM</span> products </span>WHERE</span> id </span>=</span> 452</span>;</span></span> </span> -- option 2: update the data to match the expected state</span></span> UPDATE</span> products </span>SET name =</span> 'New Product'</span>, price </span>=</span> 29</span>.</span>99</span> WHERE</span> id </span>=</span> 452</span>;</span></span></code></pre> Only after the conflict is resolved can the logical replication continue. This has been only a very simple example of a conflict that might arise during logical replication. Other examples might involve:</p> Data problems, constraint violations, or type mismatch</li> Schema conflicts (missing or renamed table/column)</li> Permissions issues or row-level security</li> </ul> In all those cases, you still need to go and fix the problem before logical replication can resume.</p> Wrap Up</a> </h2> Logical replication in PostgreSQL opens up powerful possibilities beyond traditional physical replication. We have covered the foundations that can get you started with setting up your own logical replication environment, including understanding the differences between physical and logical replication, configuring publishers and subscribers, and managing core components like publications, subscriptions, and replication slots.</p> This article is part of the upcoming guide Mastering Logical Replication in PostgreSQL</a>. If you are interested in the topic, please consider subscribing to get the latest articles as they are published. Or you can continue to Part2: Beyond The Basics of Logical Replication</a> available now.</p> PostgreSQL Service Connections 2025-05-15T00:00:00+00:00 There are so many ways to connect to PostgreSQL. One of my favourite, yet underutilised is using service definition, the feature of any application using the libpq</code> library. In this article we will explore what service definition is, where and how it can make your life with PostgreSQL much easier.</p> PostgreSQL connection methods</a> </h3> PostgreSQL offers several methods to establish connections to databases, each with its own benefits and use cases:</p> Providing individual parameters like host, port, database, username, and password directly</li> Using a URI format like postgresql://username:password@hostname:port/dbname</li> Using environment variables PGHOST, PGPORT, PGDATABASE, etc.</li> Command line arguments</li> And using pre-defined connection profiles stored in the service connection file(s)</li> </ul> While the first four methods are widely known and used, service definitions remain somewhat of a hidden gem despite offering significant advantages in terms of security, maintainability, and convenience.</p> Service definition and format</a> </h3> Connection service file</a> allows defining either per-user or system-wide connections (if both exist, the user one will take precedence). The service file uses INI format and employs the full list of libpq</code> parameters to configure the service. A sample file looks like:</p> [mydb]</span></span> host=localhost</span></span> port=5432</span></span> user=some_admin</span></span> password=password123</span></span> dbname=my_db</span></span></code></pre> You can consult the list of parameters, together with their description, directly using the documentation</a>. All you need to do is place the file in the correct location, which is: - either user-specific ~/.pg_service.conf</code> - or system-wide pg_service.conf in the PostgreSQL's configuration directory</p> and try it directly using:</p> psql service=mydb</span></span></code></pre>Why use service definitions?</a> </h2> With all other methods available to connect to PostgreSQL, why even bother with service definition? The primary benefit is the level of separation. While creating connection strings/DNS and environment variables is easy, they all suffer from the same fundamental problem: tight coupling between your application and database connection details.</p> Service definitions break this coupling by extracting connection parameters into a standardised configuration format that any libpq-based application can reference. Imagine using a single service definition across development, test, and production environments. All it takes is to define:</p> service=my_app_db</span></span></code></pre> and supply the correct service definition for the target environment. The application remains unaware of the underlying connection details, creating a clean abstraction layer with clear service name identification.</p> Another aspect is re-usability. While your application might rely on an easy to change configuration mechanism, you can easily spot the difference between these two commands:</p> pg_dump</span> --schema-only -h</span> my-app-db.internal</span> -p 5432 -U</span> some_user</span> -d</span> application</span> ></span> schema.dump</span></span> pg_dump</span> service=my_app_db</span> ></span> schema.dump</span></span></code></pre> The same applies to even internal mechanisms inside the running PostgreSQL (dblink</code> being a prime example - see below). This simplification extends to database migration, replication setup, and monitoring configuration. The consistency of service-based connections reduces errors and streamlines operational procedures.</p> The benefits increase with improved security. Having a service definition file stored outside your project GIT repository significantly reduces accidental credentials sneaking inside the repository. Separation (which you can further increase by use of passfile</code> instead of password</code>) extends flexibility, while improving the security baseline.</p> Using Service Definitions</a> </h2> Here are some ways to use service definitions:</p> With DSN keywords:</p> service=my_app_db</span></span> service=my_app_db application_name=myapp</span></span></code></pre> In connection URLs:</p> postgresql:///?service=my_app_db</span></span> postgresql:///?service=my_app_db&application_name=myapp</span></span></code></pre> Via environment variables:</p> PGSERVICE=my_app_db</span></span></code></pre> Given that libpq</code> is the foundation of most programming language drivers, the same is applicable almost across the board:</p> In Go:</p> // Go</span></span> db, err</span> :=</span> sql.</span>Open</span>(</span>"postgres"</span>,</span> "service=myservice"</span>)</span></span> </span> // Go (pgx)</span></span> conn, err</span> :=</span> pgx.</span>Connect</span>(context.</span>Background</span>(),</span> "service=my_app_db"</span>)</span></span></code></pre> Same with Python:</p> # Python (with psycopg2)</span></span> conn</span> =</span> psycopg2.connect(</span>"service=my_app_db"</span>)</span></span></code></pre> Java:</p> // Java</span></span> System.</span>setProperty</span>(</span></span> "org.postgresql.pgservicefile"</span>,</span></span> "/path/to/pg_service.conf"</span>);</span></span> Connection conn</span> =</span> DriverManager.</span>getConnection</span>(</span>"jdbc:postgresql:///?service=my_app_db"</span></span></code></pre> Not forgetting PHP:</p> // PHP</span></span> $conn</span> =</span> pg_connect</span>(</span>"service=my_app_db"</span>);</span></span> </span> // PHP PDO</span></span> $pdo</span> = new</span> PDO</span>(</span>"pgsql:service=my_app_db"</span>);</span></span></code></pre> Or Rust:</p> // Rust</span></span> let</span> (client, connection)</span> =</span></span> tokio_postgres</span>::</span>connect</span>(</span>"service=my_app_db"</span>,</span> NoTls</span>)</span>.await?</span>;</span></span></code></pre> Unfortunately, the services definitions are not available in drivers/modules not based on libpq</code>, which includes, for example, Node's pg module.</p> Using service definition inside PostgreSQL</a> </h2> A notable use case for service definition is within PostgreSQL itself. Hardcoding configuration details - either directly or using idempotent schema files - is not particularly fun. Service definition significantly improves configuration management in these cases.</p> This is the case for dblink</code>:</p> SELECT * FROM dblink('service=my_app_db', 'SELECT user_id, email FROM users');</span></span></code></pre> As well as FDW configuration:</p> CREATE SERVER foreign_server</span></span> FOREIGN DATA WRAPPER postgres_fdw</span></span> OPTIONS (service 'my_app_db');</span></span></code></pre> While postgres_fdw</code> naturally supports service definition, being based on libpq</code>, please note this is an exception and you can't expect it for other foreign data wrappers.</p> Similar to that, you need to consider the validity of the credentials, as libpq</code> reads the service definition at connection time, not at definition time. I.e., for Postgres FDW:</p> Service name is stored on CREATE SERVER</code> execution</li> Service definition is read when connection is established (foreign table accessed for the first time or service connection is validated)</li> </ul> Similar for dblink</code> where the definition is read each time it establishes a new connection. Don't forget the existing connections are not affected.</p> Partial Service Definitions</a> </h2> The flexibility offered by service definition does not end there. The libpq</code> library follows a specific order when determining connection parameters:</p> Command-line parameters take highest precedence</li> Environment variables come next</li> Service definitions provide the next level</li> Default values are used as a last resort</li> </ol> This hierarchy enables you to define only some parameters in the service definition:</p> [mydb]</span></span> host=localhost</span></span> port=5432</span></span> dbname=my_db</span></span></code></pre> With no credentials defined, you can later present the specific keywords (like credentials in this example) using other means:</p> PGUSER=user1</span></span> PGPASSWORD=pass1 psql service=my_db</span></span></code></pre> The same applies to connection strings/DSNs, etc. This allows you to use a "best of both worlds" approach, combining the consistency of service definition with the flexibility to override it.</p> Conclusion</a> </h2> Service definitions represent a powerful but often overlooked feature of PostgreSQL's connection system. They provide a clean separation of connection details from application code, enhance security by centralising credential management, and simplify connection management across different environments and applications.</p> Time to Better Know The Time in PostgreSQL 2025-04-06T00:00:00+00:00 To honor the name of the site (boringSQL) let's deep dive into a topic which might sound obvious, but it might be never ending source of surprises and misunderstanding.</p> Simple things</a> </h2> We can start with the simple statement like</p> SELECT</span> '2025-03-30 00:30'</span> as</span> t;</span></span></code></pre> Result:</p> t</span></span> ------------------</span></span> 2025-03-30 00:30</span></span></code></pre> which gives you (and I do hope that's not a big surprise) a simple text literal, rather than anything to do with date and time. As soon as you use the string literal in queries similar to</p> SELECT * FROM</span> events </span>WHERE</span> start_at </span>></span> '2025-03-30 00:30'</span>;</span></span></code></pre> PostgreSQL will implicitly convert the string into timestamp without time zone. The reason why it can happen is the fact start_at</code> is most likely timestamp based field, allowing automatic cast to match the column's data type. Which is unlike the query</p> SELECT</span> '2025-03-30 01:00'</span> -</span> interval </span>'15 minutes'</span>;</span></span></code></pre> Result:</p> ERROR: invalid input syntax for type interval: "2025-03-30 01:00"</span></span></code></pre> which will simply not work, as PostgreSQL can't perform automatic cast (which as we will cover later might be surprising behavior, but that's how things are). The correct way to get this example query to work is to use either one of two ways (both functionally equivalent but different notation).</p> -- PostgreSQL cast notation</span></span> SELECT</span> '2025-03-30 01:00'</span>::</span>timestamp -</span> interval </span>'15 minutes'</span>;</span></span> </span> -- SQL standard explicit type notation</span></span> SELECT timestamp</span> '2025-03-03 01:00'</span> -</span> interval </span>'15 minutes'</span>;</span></span></code></pre>PostgreSQL timestamp vs timestamptz</a> </h2> The next possible source of confusion when working with time in PostgreSQL is presence of two distinct data types:</p> timestamp</code> (or timestamp without time zone</code>)</li> timestamptz</code> (or timestamp with time zone</code>) Despite what the names suggest, the key difference isn't whether they store timezone information</strong>, but rather how they handle it during storage and retrieval.</li> </ul> IMPORTANT: before we cover the details, remember to always use timestamptz</code>. As official Don't Do This</a> page shows, using timestamp</code> is like storing picture of the clock, instead of point in time. And while this article might use timestamp</code> it's purely for demonstration purposes.</p> </blockquote> All you need to remember is the fact timestamp</code> stores the exact datetime value as entered</strong> without any timezone context or adjustments. It's essentially a snapshot of a calendar date and wall clock time, disconnected from any particular geographic location.</p> -- this stores what you provided</span></span> SELECT</span> '2025-03-30 01:30'</span>::</span>timestamp</span>;</span></span></code></pre> Result:</p> timestamp</span></span> ---------------------</span></span> 2025-03-30 01:30:00</span></span></code></pre> On the other hand timestamptz</code> normalizes all input to UTC internally</strong> and then converts values to the session's timezone for display. This provides geographic context to your datetime values.</p> -- this converts your input to UTC based on your session timezone</span></span> SELECT</span> '2025-03-30 01:30'</span>::</span>timestamptz</span>;</span></span></code></pre> Result:</p> timestamptz</span></span> -----------------------</span></span> 2025-03-30 01:30:00+01</span></span></code></pre> And this is where potential confusion might start. Let's take this example</p> -- With UTC timezone</span></span> SET</span> timezone </span>=</span> 'UTC'</span>;</span></span> SELECT</span> '2025-03-30 01:30'</span>::</span>timestamptz</span>;</span></span></code></pre> Result:</p> timestamptz</span></span> -----------------------</span></span> 2025-03-30 01:30:00+00</span></span></code></pre>-- With Tokyo timezone</span></span> SET</span> timezone </span>=</span> 'Asia/Tokyo'</span>;</span></span> SELECT</span> '2025-03-30 01:30'</span>::</span>timestamptz</span>;</span></span></code></pre> Result:</p> timestamptz</span></span> -----------------------</span></span> 2025-03-30 01:30:00+09</span></span></code></pre> which gives you correct representation based on the session timezone. This comes in handy when we add the actual storage.</p> -- Create table and view with Berlin timezone</span></span> SET</span> timezone </span>=</span> 'Europe/Berlin'</span>;</span></span> CREATE TABLE</span> time_flies</span>(moment </span>timestamp with time zone</span>);</span></span> INSERT INTO</span> time_flies </span>VALUES</span> (</span>'2025-03-30 00:30'</span>);</span></span> SELECT</span> moment </span>FROM</span> time_flies;</span></span></code></pre> Result:</p> moment</span></span> ------------------------</span></span> 2025-03-30 00:30:00+01</span></span></code></pre>-- View the same data with New York timezone</span></span> SET</span> timezone </span>=</span> 'America/New_York'</span>;</span></span> SELECT</span> moment </span>FROM</span> time_flies;</span></span></code></pre> Result:</p> moment</span></span> ------------------------</span></span> 2025-03-29 19:30:00-04</span></span></code></pre> This demonstrates the advantage of using automatic time zone conversion, allowing you to avoid not only time-zone bugs, but also DST related issues. While most of us might be fast asleep during our local DST changes, it's no fun to account for the time difference in a wrong. Using timestamptz</code> is sure way how to avoid it. Let's take an example of recent (at the time of writing the article) DST change.</p> -- Set Berlin timezone (during DST change)</span></span> SET</span> timezone </span>=</span> 'Europe/Berlin'</span>;</span></span> </span> -- Using timestamp (without timezone awareness)</span></span> SELECT</span> '2025-03-30 01:30'</span>::</span>timestamp +</span> interval </span>'45 minutes'</span> as</span> end_time;</span></span></code></pre> Result:</p> end_time</span></span> ---------------------</span></span> 2025-03-30 02:15:00</span></span></code></pre>-- Using timestamptz (with timezone awareness)</span></span> SELECT</span> '2025-03-30 01:30'</span>::</span>timestamptz +</span> interval </span>'45 minutes'</span> as</span> end_time;</span></span></code></pre> Result:</p> end_time</span></span> ------------------------</span></span> 2025-03-30 03:15:00+02</span></span></code></pre>AT TIME ZONE</a> </h2> Another powerful but potentially confusing operator in PostgreSQL is AT TIME ZONE</code> which allows you to convert timestamps between different time zones, but its behavior differs depending on the input data type.</p> When applied to timestamp (without time zone)</strong> When you apply AT TIME ZONE to a timestamp, PostgreSQL interprets the input as being in the specified time zone</strong> and converts it to a timestamptz:</p> -- First, set UTC timezone</span></span> SET</span> timezone</span>=</span>'UTC'</span>;</span></span> </span> -- Interpret '2025-03-30 01:30' as if it were in the 'Europe/Paris' time zone</span></span> SELECT</span> '2025-03-30 01:30'</span>::</span>timestamp AT TIME ZONE</span> 'Europe/Paris'</span>;</span></span></code></pre> Result:</p> timezone</span></span> ------------------------</span></span> 2025-03-30 00:30:00+00</span></span></code></pre> What it does is effectively "Take this wall clock time, consider it as being in the specified time zone, and give me the corresponding moment in time (as a timestamptz)". The representation here is based on session time zone settings.</p> When applied to timestamptz (with time zone)</strong> Conversely, when you apply AT TIME ZONE to a timestamptz, PostgreSQL converts the timestamp to the specified time zone and returns a timestamp without time zone</strong>:</p> -- Set timezone</span></span> SET</span> timezone</span>=</span>'America/Port_of_Spain'</span>;</span></span> </span> -- Convert the timestamptz to how it would appear on wall clocks in Tokyo</span></span> SELECT</span> '2025-03-30 01:30+01:00'</span>::</span>timestamptz AT TIME ZONE</span> 'Asia/Tokyo'</span>;</span></span></code></pre> Result:</p> timezone</span></span> ---------------------</span></span> 2025-03-30 09:30:00</span></span></code></pre> This operation says: "Take this moment in time and show me what wall clock time it would be in the specified time zone."</p> Practical usage</strong> Probably the only use when correctly using timestamps</code> is to provide "wall clock" representation of the times at various time zones at once for (view) representational purposes.</p> -- What time is the company-wide meeting in various offices?</span></span> SELECT</span></span> '2025-03-30 15:00'</span>::</span>timestamptz AS</span> meeting_time_utc,</span></span> '2025-03-30 15:00'</span>::</span>timestamptz AT TIME ZONE</span> 'Europe/London'</span> AS</span> london_office_time,</span></span> '2025-03-30 15:00'</span>::</span>timestamptz AT TIME ZONE</span> 'Asia/Tokyo'</span> AS</span> tokyo_office_time;</span></span></code></pre> Result:</p> meeting_time_utc \| london_office_time \| tokyo_office_time</span></span> ------------------------+---------------------+---------------------</span></span> 2025-03-30 15:00:00+02 \| 2025-03-30 14:00:00 \| 2025-03-30 22:00:00</span></span></code></pre> It's important to note that in API based clients, it's always better to represent full time notation and leave the clients with the representational layer. Similar to that, unless you need to work with multiple time zones, and you depend on the correct local time representation it's always better to use local session timezone</code> setting instead of using AT TIME ZONE</code> operator. This applies for both time input and output.</p> Common mistake</strong> Let's consider the example where you might consider "magically casting" already stored time using AT TIME ZONE</code>.</p> -- Setup example</span></span> SET</span> timezone</span>=</span>'Europe/Berlin'</span>;</span></span> CREATE TABLE</span> different_moments</span>(moment1 </span>timestamptz</span>, moment2 </span>timestamptz</span>);</span></span> INSERT INTO</span> different_moments (moment1) </span>values</span> (</span>'2025-03-30 01:30'</span>);</span></span> </span> -- Apply double time zone conversion</span></span> UPDATE</span> different_moments </span>SET</span> moment2 </span>=</span> moment1 </span>AT TIME ZONE</span> 'Europe/Berlin'</span> AT TIME ZONE</span> 'America/New_York'</span>;</span></span> </span> -- View the results</span></span> SELECT * FROM</span> different_moments;</span></span></code></pre> Result:</p> moment1 \| moment2</span></span> ------------------------+------------------------</span></span> 2025-03-30 01:30:00+01 \| 2025-03-30 07:30:00+02</span></span></code></pre> Unless you are aware of the implicit conversion between timestamps</code> and timestamp</code> and vice-versa you might be set for a lot of surprises. In this particular case the first conversion performed redundant conversion and removed the time zone offset. While second "forced" it to NYC time, while the subsequent select rendered it in local session timezone.</p> Timestamps and the storage</a> </h2> While you will use timestamp with time zone</code> from now on, PostgreSQL still comes with several nuances related to the way it can store the data. First lesser known feature might be timestamp precision. Let's consider following table.</p> -- Table with various timestamp precision specifications</span></span> CREATE TABLE</span> precision</span> (</span></span> t1 </span>timestamp with time zone</span>, </span>-- default precision (6)</span></span> t2 </span>timestamp</span>(</span>0</span>)</span> with time zone</span>, </span>-- seconds precision</span></span> t3 </span>timestamp</span>(</span>2</span>)</span> with time zone</span>, </span>-- centiseconds precision</span></span> t4 </span>timestamp</span>(</span>4</span>)</span> with time zone</span> -- 10 microseconds precision</span></span> );</span></span></code></pre> Effectively the timestamp</code> comes with default precision of 6 digits (microseconds), but you can specify anywhere from 0 to 6 for both timestamp</code> and timestamptz</code> types. At the same time it does not have any impact on the storage (8 bytes) regardless the precision specified.</p> -- Insert the same timestamp into all columns</span></span> INSERT INTO precision VALUES</span> (current_timestamp, current_timestamp, current_timestamp, current_timestamp);</span></span> SELECT * FROM precision</span>;</span></span></code></pre> Result:</p> -[ RECORD 1 ]---------------------</span></span> t1 \| 2025-04-03 21:19:20.952354+02</span></span> t2 \| 2025-04-04 21:19:21+02</span></span> t3 \| 2025-04-04 21:19:20.95+02</span></span> t4 \| 2025-04-04 21:19:20.9524+02</span></span></code></pre> Other than precision there are some other interesting aspects of the timestamps storage in PostgreSQL:</p> It has finite range</strong> of timestamptz '4713-01-01 BC'</code> and timestamptz '294276-12-31'</code></li> and at the same time supports positive and negative infinity</strong> with 'infinity'::timestamptz</code> and '-infinity'::timestamptz</code> that might provide alternative to NULL</code> for open ended intervals. If you choose to use infinity instead of NULL</code> values, you can test whatever the provided date/timestamp is finite value or not using isfinite(timestamp)</code>.</li> </ul> Getting the current time</a> </h2> Not many developers are aware that there are different ways to get the current datetime in SQL. These methods not only differ in syntax but also in their underlying logic.</p> CURRENT_TIMESTAMP vs NOW()</a> </h3> CURRENT_TIMESTAMP</strong> is part of the SQL Standard and interestingly, it's defined as both a "time-varying variable" and a "datetime value function". In PostgreSQL, you can use it with or without parentheses, making both these forms valid:</p> SELECT</span> CURRENT_TIMESTAMP;</span></span> SELECT</span> CURRENT_TIMESTAMP</span>(</span>2</span>); </span>-- (2) specifies precision</span></span></code></pre> Along with CURRENT_TIMESTAMP, the SQL Standard also defines CURRENT_TIME</code> and CURRENT_DATE</code> for working with individual time and date components respectively.</p> NOW()</code>, while not part of the SQL Standard, is a widely used alternative across many database systems. Unlike CURRENT_TIMESTAMP, it's strictly a function and therefore always requires parentheses when called. Its straightforward syntax - NOW()</code> - makes it a popular choice among developers.</p> Transaction Behavior</a> </h3> Both NOW()</code> and CURRENT_TIMESTAMP</code> share the same transaction behavior: they return the timestamp from the start of the current transaction, not the moment of execution:</p> BEGIN</span>;</span></span> SELECT NOW</span>(); </span>-- Returns transaction start time</span></span> SELECT</span> pg_sleep(</span>5</span>); </span>-- Wait 5 seconds</span></span> SELECT NOW</span>(); </span>-- Still returns the same time</span></span> COMMIT</span>;</span></span></code></pre> This behavior ensures data consistency within transactions but might be unexpected when measuring elapsed time.</p> Alternative Time Functions</a> </h3> When you need the actual current time regardless of transaction status, CLOCK_TIMESTAMP()</code> is your best choice. It returns the precise moment of execution, making it particularly useful for measuring real elapsed time. Here's how it differs from NOW()</code>:</p> BEGIN</span>;</span></span> SELECT</span> pg_sleep(</span>1</span>);</span></span> SELECT NOW</span>()</span> AS</span> transaction_time,</span></span> CLOCK_TIMESTAMP()</span> AS</span> actual_time;</span></span> COMMIT</span>;</span></span></code></pre> Result:</p> transaction_time \| actual_time</span></span> ------------------------------+-------------------------------</span></span> 2025-04-03 19:28:25.310254+00 \| 2025-04-03 19:28:26.311917+00</span></span></code></pre> Another useful function is STATEMENT_TIMESTAMP()</code>, which captures the time when the current statement began executing. This differs subtly from CLOCK_TIMESTAMP()</code>, which gives you the exact moment of function execution. The difference becomes clear in this example:</p> SELECT</span></span> pg_sleep(</span>2</span>),</span></span> STATEMENT_TIMESTAMP()</span> AS statement</span>,</span></span> CLOCK_TIMESTAMP()</span> AS</span> wall_time;</span></span></code></pre> Result:</p> pg_sleep \| statement \| wall_time</span></span> ----------+-------------------------------+------------------------------</span></span> \| 2025-04-03 19:32:15.984459+00 \| 2025-04-03 19:32:17.98661+00</span></span></code></pre> These timestamp functions serve different purposes and are invaluable when you need to measure transaction timing or analyze statement-level performance. Each provides a different perspective on time within your database operations.</p> Intervals</a> </h2> Guess if you have worked with time in PostgreSQL you have come across with intervals. You probably just entered the it using something like interval '1 year 2 months 3 days 4 hours'</code>. Technical that's representation driven by setting intervalstyle</code> which (as in this case) is set to postgres_verbose</code>. But you have much more options.</p> -- PostgreSQL default style</span></span> SET</span> intervalstyle </span>=</span> 'postgres'</span>;</span></span> SELECT</span> interval </span>'1 year 2 months 3 days 4 hours'</span>;</span></span></code></pre> Result:</p> interval</span></span> -------------------------------</span></span> 1 year 2 mons 3 days 04:00:00</span></span></code></pre>-- SQL standard style</span></span> SET</span> intervalstyle </span>=</span> 'sql_standard'</span>;</span></span> SELECT</span> interval </span>'1 year 2 months 3 days 4 hours'</span>;</span></span></code></pre> Result:</p> interval</span></span> ------------------</span></span> +1-2 +3 +4:00:00</span></span></code></pre>-- ISO 8601 style</span></span> SET</span> intervalstyle </span>=</span> 'iso_8601'</span>;</span></span> SELECT</span> interval </span>'1 year 2 months 3 days 4 hours'</span>;</span></span></code></pre> Result:</p> interval</span></span> ------------</span></span> P1Y2M3DT4H</span></span></code></pre> With the session intervalstyle</code> you only change the interval format representation, not the actual value. Nevertheless the interval representation you might run into an interval definition that might not longer make it obvious (both in input or output) to interpret. That's where function justify_interval</code> comes in play - helping you to normalize time expression into conventional formats.</p> -- Normalize complex interval</span></span> SELECT</span> justify_interval(interval </span>'15 months 40 days 30 hours'</span>);</span></span></code></pre> Result:</p> justify_interval</span></span> ---------------------------------</span></span> @ 1 year 4 mons 11 days 6 hours</span></span></code></pre>-- Practical example: normalize project durations</span></span> SELECT</span></span> project_name,</span></span> total_hours </span>\|\|</span> ' hours'</span> AS</span> raw_duration,</span></span> justify_interval(interval </span>'1 hour'</span> </span> total_hours) </span>AS</span> normalized_duration</span></span> FROM</span></span> (</span>VALUES</span> (</span>'Database Migration'</span>, </span>1850</span>),</span></span> (</span>'API Development'</span>, </span>720</span>),</span></span> (</span>'UI Redesign'</span>, </span>340</span>))</span></span> AS</span> projects(project_name, total_hours);</span></span></code></pre> Result:</p> project_name \| raw_duration \| normalized_duration</span></span> --------------------+--------------+--------------------------</span></span> Database Migration \| 1850 hours \| @ 2 mons 17 days 2 hours</span></span> API Development \| 720 hours \| @ 1 mon</span></span> UI Redesign \| 340 hours \| @ 14 days 4 hours</span></span></code></pre> One of the important specific with internal normalization is the fact it uses 30-days time periods as a month definition.</p> -- Month definition in normalize intervals</span></span> SELECT</span> justify_interval(interval </span>'30 days'</span>);</span></span></code></pre> Result:</p> justify_interval</span></span> ------------------</span></span> @ 1 mon</span></span></code></pre> Which might be understandable, but you need to beware of the edge cases when using days and months intervals.</p> -- Edge case comparison: month vs 30 days</span></span> SELECT</span></span> date</span> '2025-01-31'</span> +</span> interval </span>'1 month'</span> as</span> example1,</span></span> date</span> '2025-01-31'</span> +</span> interval </span>'30 days'</span> as</span> example2;</span></span></code></pre> Result:</p> -[ RECORD 1 ]-----------------</span></span> example1 \| 2025-02-28 00:00:00</span></span> example2 \| 2025-03-02 00:00:00</span></span></code></pre> And one last thing, before wrapping up the interval - you can (same as with dates/timestamp) get specific parts using extract</code> function.</p> -- Extract specific parts from intervals</span></span> SELECT</span></span> extract(</span>days from</span> interval </span>'1 year 3 months 21 days'</span>) </span>AS days</span>,</span></span> extract(</span>hours from</span> interval </span>'25:30:45'</span>) </span>AS hours</span>;</span></span></code></pre> Result:</p> days \| hours</span></span> ------+-------</span></span> 21 \| 25</span></span></code></pre>Time ranges</a> </h2> While working with timestamp comes natural, it's time ranges which often feel like ugly ducklings. And quite unfairly so - timestamp ranges in PostgreSQL are powerful yet underutilized features that elegantly solve common time-based challenges. PostgreSQL offers dedicated range types that are perfect for modeling time periods, reservations, schedules, and any situation where you need to track intervals with defined start and end points:</p> tsrange</code> - range of timestamp without time zone</li> tstzrange</code> range of timestamp with time zone And to follow the earlier advice - just as you should default to timestamps</code>, always use tstzrange</code> for time ranges unless you have VERY specific reason for it. Timestamp ranges aren't merely convenient syntax - they provide a complete set of operations for determining relationships between time periods and enable powerful constraints that handle complex business rules without reinventing the wheel. Timestamp ranges can be created using the range constructor syntax or string syntax.</li> </ul> -- Range constructor syntax</span></span> SELECT</span> tstzrange(</span></span> '2025-04-01 09:00:00+02'</span>::</span>timestamptz</span>,</span></span> '2025-04-01 17:30:00+02'</span>::</span>timestamptz</span>,</span></span> '[)'</span></span> ) </span>AS</span> workday;</span></span> </span> -- String syntax</span></span> SELECT</span> '[2025-04-01 09:00:00+02, 2025-04-01 17:30:00+02)'</span>::tstzrange </span>AS</span> workday;</span></span></code></pre> Result:</p> workday</span></span> ["2025-04-01 07:00:00+00","2025-04-01 15:30:00+00")</span></span></code></pre> The default boundary notation for timestamp ranges is [)</code> - lower bound inclusive (includes the start time) and upper bound exclusive (excludes the end time). For time-based applications, this convention is particularly valuable. Consider scheduling consecutive one-hour meetings from 9:00 to 10:00 and 10:00 to 11:00. With [)</code> notation, these are represented as [09:00, 10:00)</code> and [10:00, 11:00)</code>, creating perfect adjacency without overlap or gaps. If you used inclusive bounds [09:00, 10:00]</code> and [10:00, 11:00]</code>, the moment 10:00 would technically belong to both ranges - a logical impossibility for scheduling. This natural alignment with how we conceptualize time slots makes [)</code> notation the default and recommended choice for most of the application use cases. Timestamp ranges support numerous operations for checking relationships:</p> -- 1. Check if a timestamp is within a range</span></span> SELECT</span></span> '[2025-04-01 09:00, 2025-04-01 17:00)'</span>::tstzrange @</span>></span> '2025-04-01 12:30'</span>::</span>timestamptz AS</span> is_during_workday;</span></span></code></pre> Result:</p> is_during_workday</span></span> ------------------</span></span> t</span></span></code></pre>-- 2. Check if two ranges overlap</span></span> SELECT</span></span> '[2025-04-01 09:00, 2025-04-01 12:00)'</span>::tstzrange &&</span></span> '[2025-04-01 11:00, 2025-04-01 14:00)'</span>::tstzrange </span>AS</span> meetings_overlap;</span></span></code></pre> Result:</p> meetings_overlap</span></span> -----------------</span></span> t</span></span></code></pre>-- 3. Extract time range bounds</span></span> SELECT</span></span> lower</span>(</span>'[2025-04-01 09:00, 2025-04-01 17:00)'</span>::tstzrange) </span>AS</span> start_time,</span></span> upper</span>(</span>'[2025-04-01 09:00, 2025-04-01 17:00)'</span>::tstzrange) </span>AS</span> end_time;</span></span></code></pre> Result:</p> start_time \| end_time</span></span> --------------------------+--------------------------</span></span> 2025-04-01 09:00:00+00 \| 2025-04-01 17:00:00+00</span></span></code></pre> As well as obvious manipulations:</p> -- 1. Merge adjacent ranges</span></span> SELECT</span></span> '[2025-04-01 09:00, 2025-04-01 12:00)'</span>::tstzrange </span>+</span></span> '[2025-04-01 12:00, 2025-04-01 17:00)'</span>::tstzrange </span>AS</span> full_day;</span></span></code></pre> Result:</p> full_day</span></span> -------------------------------------------------------------</span></span> ["2025-04-01 09:00:00+00","2025-04-01 17:00:00+00")</span></span></code></pre>-- 2. Find intersection between overlapping ranges</span></span> SELECT</span></span> '[2025-04-01 09:00, 2025-04-01 14:00)'</span>::tstzrange </span></span></span> '[2025-04-01 12:00, 2025-04-01 17:00)'</span>::tstzrange </span>AS</span> overlap_period;</span></span></code></pre> Result:</p> overlap_period</span></span> -------------------------------------------------------------</span></span> ["2025-04-01 12:00:00+00","2025-04-01 14:00:00+00")</span></span></code></pre>-- 3. Find gap between non-overlapping ranges</span></span> SELECT</span></span> '[2025-04-01 09:00, 2025-04-01 11:00)'</span>::tstzrange </span>-</span></span> '[2025-04-01 14:00, 2025-04-01 17:00)'</span>::tstzrange </span>AS</span> gap;</span></span></code></pre> Result:</p> gap</span></span> -------------------------------------------------------------</span></span> ["2025-04-01 11:00:00+00","2025-04-01 14:00:00+00")</span></span></code></pre> And if you followed the previously mentioned logic, you wouldn't double guess tstzrange</code> handling of the DST transitions.</p> -- Testing DST transition handling</span></span> SET</span> timezone </span>=</span> 'Europe/Berlin'</span>;</span></span> SELECT</span></span> '[2025-03-30 01:00:00, 2025-03-30 03:00:00)'</span>::tstzrange </span>AS</span> dst_transition_range,</span></span> upper</span>(</span>'[2025-03-30 01:00:00, 2025-03-30 03:00:00)'</span>::tstzrange) </span>-</span></span> lower</span>(</span>'[2025-03-30 01:00:00, 2025-03-30 03:00:00)'</span>::tstzrange) </span>AS</span> actual_duration;</span></span></code></pre> Result:</p> dst_transition_range \| actual_duration</span></span> -----------------------------------------------------+-----------------</span></span> ["2025-03-30 01:00:00+01","2025-03-30 03:00:00+02") \| @ 1 hour</span></span></code></pre> The area where time ranges win in almost all scenarios is indexing. If you consider a manual implementation of the range using start_at</code> and end_at</code> columns you typically rely on B-tree indexes. While they might work for simple queries on just one of the values, they fall short for time ranges. Consider this example:</p> -- Traditional approach with two columns and B-tree indexes</span></span> SELECT * FROM</span> events_columns</span></span> WHERE</span> start_at </span><=</span> '2025-04-01 17:00'</span></span> AND</span> end_at </span>>=</span> '2025-04-01 09:00'</span>;</span></span></code></pre> For this query, PostgreSQL might use one of the indexes if it's selective enough, but the fundamental problem remains: each B-tree index can only efficiently filter on its own column. The planner might use the start_at index to find events starting before 17:00, or the end_at index to find events ending after 09:00, but it can't use both indexes simultaneously to find the intersection. With range types, we can use GiST</strong> (Generalized Search Tree) indexes that are specifically designed for range operations:</p> -- Create specialized GiST index for time range operations</span></span> CREATE INDEX</span> idx_events_range</span> ON</span> events_range </span>USING</span> GIST(time_period);</span></span></code></pre> This allows you to transform the sample query into:</p> -- Range-based approach using the && (overlap) operator</span></span> SELECT * FROM</span> events_range</span></span> WHERE</span> time_period && tstzrange(</span>'2025-04-01 09:00'</span>, </span>'2025-04-01 17:00'</span>);</span></span></code></pre> The performance gap between these approaches is substantial – range queries with GiST indexes typically outperform</strong> the traditional approach by 2-10x on large datasets. This difference becomes even more pronounced as your data grows.</p> Time to wrap up</a> </h2> Working with time in PostgreSQL might seem straightforward on the surface, but as we've explored, it's filled with nuances that can make or break your application's reliability. The key takeaways from this deep dive should be:</p> Always use timestamptz instead of timestamp</strong></li> Be intentional about time zone handling</strong></li> Leverage PostgreSQL's specialized time tools</strong></li> Mind the edge cases</strong></li> </ul> Time handling remains one of the most deceptively complex aspects of software engineering, but PostgreSQL offers a robust toolkit for managing it effectively. By understanding these "boring" but essential concepts, you'll avoid common pitfalls and build applications that handle time with confidence and precision.</p> VIEW inlining in PostgreSQL 2025-02-08T00:00:00+00:00 Database VIEWs are powerful tools that often don't get the attention they deserve when building database-driven applications. They make our database work easier in several ways:</p> They let us reuse common query patterns instead of writing them over and over</li> They give us a place to define business rules once and use them everywhere</li> They help us write cleaner, more organized queries</li> </ul> Let's see how this works with a practical example. Imagine we want to work with active users</em> - users who have used the application within the last 7 days. Instead of writing this condition in every query, we can define a view:</p> CREATE VIEW</span> active_users</span> AS</span></span> SELECT</span></span> </span></span> FROM</span> users;</span></span> WHERE</span></span> last_login </span>></span> current_date </span>-</span> INTERVAL </span>'7 days'</span>;</span></span></code></pre> Now we can easily use this view whenever we need active users. For example, if we want to find active users from Germany:</p> SELECT</span></span> user_id</span></span> FROM</span> active_users</span></span> WHERE</span></span> country </span>=</span> 'Germany'</span>;</span></span></code></pre> While this simple example shows how views can make developers' lives easier by organizing and reusing common logic, there's more to the story. In this article, we'll explore something even more interesting: how PostgreSQL can optimize these views through a process called "inlining" - making them not just convenient, but fast too.</p> What is VIEW inlining</a> </h2> When you use a view, it acts like a building block in SQL queries. Taking our previous example, PostgreSQL effectively transforms the query by replacing the view with its underlying subquery.</p> SELECT</span></span> user_id</span></span> FROM</span> (</span></span> SELECT</span></span> </span></span> FROM</span> users</span></span> WHERE</span></span> last_login </span>></span> current_date </span>-</span> INTERVAL </span>'7 days'</span></span> ) active users</span></span> WHERE</span></span> country </span>=</span> 'Germany'</span>;</span></span></code></pre> While we write the query using active_users</code> VIEW, PostgreSQL query planner will see it as an opportunity to optimize it further. Instead of treating the sub-query as separate step (and retrieving large subset of the users), it effectively transforms the query and inlines</strong> the view into the query itself. Behind the scenes, PostgreSQL will execute the query similar to:</p> SELECT</span></span> user_id</span></span> FROM</span> users</span></span> WHERE</span></span> last_login </span>></span> current_date </span>-</span> INTERVAL </span>'7 days'</span></span> AND</span> country </span>=</span> 'Germany'</span>;</span></span></code></pre> The inlining process</strong> allows the PostgreSQL query planner to optimize the entire query as a single unit. This allows it to select the best indexes, possible join strategies and other aspects together, rather than having to execute the individual parts separately and then filtering the data that won't be otherwise used. This is exactly what VIEWs are for - we get the best of both worlds - clean, modular code for developers and optimal performance for the database.</p> Inlining in action</a> </h2> The easiest way how to preview the VIEW inlining is to run EXPLAIN</p> Seq Scan on users (cost=0.00..19.20 rows=1 width=4)</span></span> Filter: ((country = 'Germany'::text) AND (last_login > (CURRENT_DATE - '7 days'::interval)))</span></span></code></pre> The Filter part here is the one showing how the filtering conditions from both the query and the view got merged together by the query planner.</p> This works especially well with complex queries involving joins, aggregations and filters. PostgreSQL can optimize entire query chain as one unit instead of executing individual pieces separately.</p> CREATE VIEW</span> order_analytics</span> AS</span></span> SELECT</span></span> o</span>.</span>customer_id</span>,</span></span> c</span>.</span>country</span>,</span></span> DATE_TRUNC(</span>'month'</span>, </span>o</span>.</span>created_at</span>) </span>as month</span>,</span></span> COUNT</span>(</span></span>) </span>as</span> orders,</span></span> SUM</span>(</span>o</span>.</span>total_amount</span>) </span>as</span> revenue</span></span> FROM</span> orders o</span></span> JOIN</span> customers c </span>ON</span> c</span>.</span>id</span> =</span> o</span>.</span>customer_id</span></span> GROUP BY</span> 1</span>, </span>2</span>, </span>3</span>;</span></span></code></pre> When you use this VIEW in the query</p> SELECT</span> customer_id, revenue</span></span> FROM</span> order_analytics</span></span> WHERE</span> country </span>=</span> 'Germany'</span></span> AND month >=</span> '2024-01-01'</span></span> AND</span> revenue </span>></span> 1000</span>;</span></span></code></pre> you will see how PostgreSQL effectively inlines the filter conditions where they belong.</p> HashAggregate (cost=205.97..207.97 rows=200 width=16)</span></span> Group Key: o.customer_id, c.country, (date_trunc('month'::text, o.created_at))</span></span> Filter: (sum(o.total_amount) > 1000)</span></span> -> Hash Join (cost=16.12..185.25 rows=1235 width=20)</span></span> Hash Cond: (o.customer_id = c.id)</span></span> -> Seq Scan on orders o</span></span> Filter: (created_at >= '2024-01-01'::date)</span></span> -> Hash</span></span> -> Seq Scan on customers c</span></span> Filter: (country = 'Germany'::text)</span></span></code></pre> This works especially well with increasing query complexity. Particulary involving joins and further filters. PostgreSQL can optimize entire query chain as one unit instead of executing individual pieces separately.</p> CREATE VIEW completed_orders AS</span></span> SELECT</span></span> o.id,</span></span> o.customer_id,</span></span> o.total_amount,</span></span> o.created_at,</span></span> o.status,</span></span> p.name as product_name,</span></span> c.email,</span></span> c.country</span></span> FROM orders o</span></span> JOIN customers c ON c.id = o.customer_id</span></span> JOIN order_items oi ON oi.order_id = o.id</span></span> JOIN products p ON p.id = oi.product_id</span></span> WHERE</span></span> o.status = 'completed';</span></span> </span> SELECT</span></span> </span></span> FROM completed_orders</span></span> WHERE</span></span> country = 'Germany';</span></span> </span> CREATE INDEX idx_orders_customer_id ON orders(customer_id); CREATE INDEX idx_order_items_order_id ON order_items(order_id); CREATE INDEX idx_order_items_product_id ON order_items(product_id);</span></span></code></pre> as demonstrated in the query plan</p> Nested Loop (cost=0.30..19.70 rows=1 width=176)</span></span> -> Nested Loop (cost=0.15..13.52 rows=1 width=148)</span></span> Join Filter: (o.id = oi.order_id)</span></span> -> Nested Loop (cost=0.15..12.43 rows=1 width=144)</span></span> -> Seq Scan on orders o (cost=0.00..1.05 rows=1 width=80)</span></span> Filter: (status = 'completed'::text)</span></span> -> Index Scan using customers_pkey on customers c (cost=0.15..8.17 rows=1 width=68)</span></span> Index Cond: (id = o.customer_id)</span></span> Filter: (country = 'Germany'::text)</span></span> -> Seq Scan on order_items oi (cost=0.00..1.04 rows=4 width=8)</span></span> -> Index Scan using products_pkey on products p (cost=0.15..6.17 rows=1 width=36)</span></span> Index Cond: (id = oi.product_id)</span></span></code></pre>Planner barriers</a> </h2> While PostgreSQL is very good at inlining views, certain operations create "planner barrier" that prevent the optimization. In case of views some of the conditions that will cause it are</p> DISTINCT ON operations</li> Window functions (OVER clauses)</li> Set operations (UNION/INTERSECT/EXCEPT)</li> More complex aggregations</li> Common Table Expressions that are materialized (either with MATERIALIZED hint or by the decision of the query planner</a>)</li> Use of VOLATILE functions</li> And in some cases complex subqueries</li> </ul> When planner is not able to inline VIEWs it might lead to the unnecessary performance and resource impact. I.e. each sub query might materialize results separately, leading to the already retrieved rows to be discarded by another part of the query as one of the most common examples.</p> When using EXPLAIN this is demostated by one of the following options:</p> Subquery scan on...</code> nodes</li> Materialization nodes</li> Separate aggregation/sorting steps before the main execution</li> </ul> As mentioned above the window function acts as such a planner barrier.</p> CREATE VIEW order_insights AS</span></span> SELECT</span></span> o.customer_id,</span></span> o.total_amount,</span></span> o.created_at,</span></span> ROW_NUMBER() OVER (PARTITION BY o.customer_id ORDER BY o.created_at) as order_sequence,</span></span> SUM(o.total_amount) OVER (PARTITION BY o.customer_id ORDER BY o.created_at) as running_total</span></span> FROM orders o;</span></span> </span> EXPLAIN</span></span> SELECT * FROM order_insights</span></span> WHERE customer_id = 1</span></span> AND total_amount > 500;</span></span></code></pre> Giving us easy to spot Subquery scan</code> node.</p> Subquery Scan on order_insights (cost=1.06..1.10 rows=1 width=84)</span></span> Filter: (order_insights.total_amount > '500'::numeric)</span></span> -> WindowAgg (cost=1.06..1.08 rows=1 width=84)</span></span> -> Sort (cost=1.06..1.06 rows=1 width=44)</span></span> Sort Key: o.created_at</span></span> -> Seq Scan on orders o (cost=0.00..1.05 rows=1 width=44)</span></span> Filter: (customer_id = 1)</span></span></code></pre>Runtime Materialization (Not MATERIALIZED VIEWs)</a> </h2> Let's clear up a possible confusion - we're not talking about CREATE MATERIALIZED VIEW here. This is about PostgreSQL's runtime decision to cache view results in memory during query execution. Query planner might use explicit materialization when it needs to reference view results multiple times or prevent repeated expensive computations. The query planner shows this through a Materialize node:</p> CREATE VIEW</span> customer_summary</span> AS</span></span> SELECT</span></span> customer_id,</span></span> COUNT</span>(</span></span>) </span>as</span> orders,</span></span> SUM</span>(total_amount) </span>as</span> total_spent</span></span> FROM</span> orders</span></span> GROUP BY</span> 1</span>;</span></span> </span> EXPLAIN </span>SELECT</span></span> a</span>.</span>customer_id</span>,</span></span> a</span>.</span>total_spent</span>,</span></span> b</span>.</span>total_spent</span> as</span> other_customer_spent</span></span> FROM</span> customer_summary a</span></span> JOIN</span> customer_summary b </span>ON</span> b</span>.</span>total_spent</span> ></span> a</span>.</span>total_spent</span>;</span></span></code></pre> Giving the perfect example of materialization.</p> Nested Loop (cost=46.00..200.75 rows=3333 width=68)</span></span> Join Filter: (b.total_spent > (sum(orders.total_amount)))</span></span> -> HashAggregate (cost=23.00..24.25 rows=100 width=44)</span></span> Group Key: orders.customer_id</span></span> -> Seq Scan on orders (cost=0.00..18.00 rows=1000 width=15)</span></span> -> Materialize (cost=23.00..25.75 rows=100 width=32)</span></span> -> Subquery Scan on b (cost=23.00..25.25 rows=100 width=32)</span></span> -> HashAggregate (cost=23.00..24.25 rows=100 width=44)</span></span> Group Key: orders_1.customer_id</span></span> -> Seq Scan on orders orders_1 (cost=0.00..18.00 rows=1000 width=15)</span></span></code></pre> Next to the materialization - the materialized VIEWs</strong> (CREATE MATERIALIZED VIEW</code>) are for practical purposes "just another table" they present an ultimate planner barrier. PostgreSQL must scan the materialized data directly, trading query flexibility for performance</p> Tips for writing inlining friendly VIEWs</a> </h2> Well, there's not a single "right" way how to write views. And don't forget - VIEWs should make your database easier to work with, not harder. When designing views, focus on single responsibility - each view should handle one aspect of business logic rather than trying to optimize everything at once. Simple views are more likely to get inlined by PostgreSQL's query planner.</p> There are several common patterns that prevent inlining</p> Avoid window functions, distinct and set operations completely</li> Split complex logic into multiple views if possible</li> Minimize subquery usage</li> Consider materialized views for aggregations</li> </ol> View dependencies are a critical consideration. Even minor schema changes can trigger cascading view modifications across your database. Instead of creating deep view hierarchies, split complex logic into smaller, composable views. For critical views that many applications depend on, consider versioning (v1, v2) rather than modifying existing ones.</p> When in doubt, write the plain SQL first, then extract common patterns into views. This helps avoid overengineering and keeps your database schema maintainable.</p> DELETEs are difficult 2024-11-23T00:00:00+00:00 Your database is ticking along nicely - until a simple DELETE brings it to its knees. What went wrong? While we tend to focus on optimizing SELECT and INSERT operations, we often overlook the hidden complexities of DELETE. Yet, removing unnecessary data is just as critical. Outdated or irrelevant data can bloat your database, degrade performance, and make maintenance a nightmare. Worse, retaining some types of data without valid justification might even lead to compliance issues.</p> At first glance, the DELETE command seems straightforward. Even the PostgreSQL documentation</a> provides simple examples like:</p> DELETE FROM</span> films </span>WHERE</span> kind </span><></span> 'Musical'</span>;</span></span> DELETE FROM</span> films;</span></span></code></pre> These queries might work effortlessly on your development machine, where only a few hundred records exist. But what happens when you try running a similar DELETE in production, where datasets are orders of magnitude larger?</p> In this article, we’ll uncover why DELETE operations demand careful consideration and explore how to handle them effectively.</p> What really happens when you DELETE the data?</a> </h2> At first glance, a DELETE query might seem straightforward. However, once the query is executed, a series of intricate steps occur:</p> Row Identification</strong>: Similar to a SELECT operation, the query identifies rows visible to the current transaction (considering MVCC) and checks for locks.</li> Lock Acquisition</strong>: The database acquires row-level exclusive locks to prevent other operations on the targeted rows.</li> BEFORE DELETE trigger</strong>: If a BEFORE DELETE trigger is defined, it is executed at this point.</li> Marking Rows as Deleted</strong>: Instead of being physically removed, the rows are marked as deleted in the current transaction, rendering them invisible to future queries (depending on transaction isolation). If table has large data objects, TOAST table will have to be involved too.</li> Index Updates</strong>: The corresponding index entries are also marked for deletion (if applicable).</li> Cascaded Actions</strong>: Cascading operations, such as ON DELETE CASCADE, are performed on related tables.</li> AFTER DELETE Trigger</strong>: If an AFTER DELETE trigger is defined, it is executed.</li> Write-Ahead Log (WAL)</strong>: Changes are recorded in the WAL-first at the row level, followed by index-level updates.</li> </ol> Only when the transaction is committed do these changes become permanent and visible to transactions starting afterward. However, even at this point, the data is not physically removed</strong>. This is how bloat</strong> is created.</p> Until the autovacuum</strong> process or a manual VACUUM operation reclaims the space, the “deleted” data remains. This leftover data contributes to bloat, which can degrade query performance over time.</p> The key question now is whether DELETEs are truly the hardest operation we can subject our database to. The answer? Quite possibly. While UPDATEs come close in complexity, they’re typically designed in ways that make them less challenging:</p> UPDATEs usually modify only a limited number of columns, reducing the potential number of index updates lower.</li> Not all UPDATEs trigger a full row (COLD) update, where the old row is marked as dead and a new row is created. With careful table and query design, you can minimise these cases. HOT (Heap-Only Tuple) updates, for instance, are easier to achieve with fixed-length columns.</li> Unlike DELETEs, UPDATEs don’t trigger cascaded actions - they only involve triggers that are explicitly defined.</li> </ul> And then comes AUTOVACUUM</a> </h2> When autovacuum kicks in (and you really want it to kick in) - typically triggered by thresholds for the number of dead tuples or changes to the table - a significant amount of work is required to clean them up. Let’s break it down step by step:</p> The process begins by scanning the table. While it’s not always a full table scan, the autovacuum checks the visibility map and pages where dead tuples might exist. This can happen incrementally, allowing even large tables to be processed - provided your autovacuum settings allow it to run frequently enough.</li> Each tuple is checked to ensure it’s no longer visible to any active or pending transactions.</li> Dead tuples that pass the visibility check are physically removed from the table.</li> Corresponding index entries for the removed tuples are updated.</li> The now-empty space is marked for reuse in future INSERT or UPDATE operations.</li> Table statistics are updated to reflect the current state, helping the query planner make better decisions.</li> The changes, including tuple removals and index updates, are logged in the Write-Ahead Log (WAL) for durability and replication.</li> If the table has TOASTed data (large objects), the associated TOAST tables are processed.</li> The visibility map is updated to mark cleaned pages as fully visible again.</li> Autovacuum resets thresholds to determine when the next vacuum operation should occur.</li> </ul> This process continues until it reaches the configured vacuum cost limit (in the case of autovacuum), at which point it pauses or stops. While autovacuum helps keep your database in check, it’s clear that reclaiming dead tuples is no small task - and it underscores why DELETE operations can have lasting effects on database performance.</p> While the AUTOVACUUM</code> might sounds as a bad news, without it, your database would quickly become bloated with dead tuples, leading to degraded performance, slower queries, increased storage usage, and even the potential for out-of-disk errors as unused space cannot be reclaimed.</p> Further considerations</a> </h2> For what seems like a simple DELETE, a surprising amount of work has already taken place, but the complexity doesn’t stop there. DELETE operations can introduce additional challenges, particularly when replication, resource contention, or the size of the operation comes into play.</p> In environments with replication to hot standby or replicas, DELETEs become more time-sensitive. The transaction cannot complete until the corresponding WAL (Write-Ahead Log) records have been written to disk on the standby. This is a fundamental requirement for maintaining data consistency in high-availability setups, where at least one standby server is typically involved. Additionally, if the standby is actively serving read operations, it must account for DELETEs before confirming the changes, potentially introducing further delays.</p> The size of the DELETE operation also plays a critical role. Small DELETEs, such as removing a single row, tend to have minimal impact. However, as the size of the operation grows, so does the volume of WAL records generated. Large DELETEs can overwhelm the system, slowing down transactions and straining the replication process. Standby servers must work harder to process the incoming WAL stream, which can bottleneck performance if their throughput is insufficient.</p> Resource contention adds yet another layer of complexity, particularly for large DELETEs. Generating WAL records, handling regular transactional workloads, and running background processes can collectively push the system towards I/O saturation. This creates competition for CPU and memory resources, leading to slower operations across the board.</p> Finally, once the data is marked for deletion, the autovacuum process must eventually step in to physically remove it. This introduces its own set of challenges, as autovacuum must deal with the same resource contention and I/O demands, compounding the overall impact of the initial DELETE operation.</p> Soft-deletes are not the solution</a> </h2> Soft-deletes might seem like an easy way to sidestep the complexities of traditional DELETE operations. After all, updating a deleted_at</code> field is straightforward and unlikely to trigger a COLD update. However, soft-deletes are not a true mechanism for data removal, and they come with their own set of complications.</p> While soft-deletes can provide a simple way to implement “undo” functionality, they raise serious questions about data consistency. For instance, do you only mark the main entity as deleted, or do you also cascade the status to all related records in referenced tables? Failing to cascade properly can leave your database in an inconsistent state, making it difficult to maintain data integrity.</p> Soft-deletes also require consideration in your application logic. Every query must include appropriate filters to exclude “deleted” rows, which can complicate query design and increase the risk of oversights. One missed filter could expose data that should no longer be visible, leading to potential security or business logic issues.</p> Finally, soft-deletes don’t solve the problem - they do merely postpone it. The data is still in your database, consuming storage and potentially contributing to performance degradation over time. Sooner or later, you’ll need to deal with the actual removal of this data, bringing you back to the same challenges that DELETEs pose in the first place.</p> At the time of writing this article we can only speculate how much will support of temporal PRIMARY KEY and UNIQUE constriants in PostgreSQL 18 will transform the balances in future. But given the complexity of the feature I wouldn't bet on it just yet.</p> Batching is the answer</a> </h2> Giving PostgreSQL time to process and catch up with large-scale changes is critical when dealing with operations like DELETEs. The core issue here is the duration and magnitude of the transaction. The shorter the transaction and the fewer changes made, the better PostgreSQL can manage and reconcile those changes.</strong> This principle is universal across all database operations and underscores the importance of minimizing the impact of individual transactions.</p> While you can optimize certain aspects, like row identification (using indexes, clustering, or similar techniques), larger datasets demand a more strategic approach - batching. For example, deleting 1 million rows in a single transaction is a textbook case of what not to do. Instead, splitting the operation into smaller batches, such as deleting 10,000 rows across 100 iterations, is far more effective.</p> Will this method be faster than performing one massive DELETE? Likely not, especially if you include a wait time between batches to allow PostgreSQL to handle other workloads. However, the trade-off is worthwhile. By batching, you give PostgreSQL more breathing room to manage changes without overwhelming regular transactional workloads - unless, of course, you’ve scheduled dedicated maintenance time for the operation.</p> How to Batch DELETEs</a> </h3> The easiest way to batch DELETEs is to use a subquery or a Common Table Expression (CTE)</a> to limit the number of rows affected in each iteration. For example, instead of executing a bulk DELETE like this:</p> DELETE FROM</span> films </span>WHERE</span> kind </span><></span> 'Musical'</span>;</span></span></code></pre> You can break the operation into smaller chunks. Using a query like the following, you can repeatedly delete rows in manageable batches (for instance, using \watch in psql to automate iterations):</p> DELETE FROM</span> films</span></span> WHERE</span> ctid </span>IN</span> (</span></span> SELECT</span> ctid </span>FROM</span> films</span></span> WHERE</span> kind </span><></span> 'Musical'</span></span> LIMIT</span> 250</span></span> );</span></span></code></pre> The use of ctid</code> in this example is PostgreSQL system column which provides a unique identifier for each row. By selecting ctid</code> values in the subquery, you can limit the number of rows affected in each iteration. This approach is more efficient than using LIMIT</code> directly in the main query, as it avoids the need to re-scan the table for each batch.</p> If you don't feel comfortable with ctid</code> (which might deserve article on its own) you can use regular lookup by primary key and LIMIT</code>.</p> Planning for Autovacuum</a> </h3> Batching alone doesn’t directly solve the issue of autovacuum catching up with the changes. You’ll need to plan for that separately. Adjusting autovacuum settings or triggering manual VACUUM</code> and VACUUM ANALYZE</code> runs can help manage bloat created during the DELETE process. However, disabling autovacuum is rarely advisable unless you’ve carefully planned for manual maintenance throughout the batch operations. Skipping this step risks leaving behind performance-impacting bloat that will require even more effort to address later.</p> Drop whole range of data with partitioning</a> </h2> Data that is naturally segmented - for example by time of the creation - makes it an excellent candidate for removal through partitioning. Partitioning allows you to bypass DELETE operations altogether by simply dropping or truncating the relevant partitions. This approach is far more efficient and avoids the overhead of scanning, locking, and marking rows as deleted, effectively eliminating the problem with the bloat.</p> While partitioning adds some complexity to schema design and query planning, it can provide significant performance benefits for DELETE-heavy workloads, especially when combined with automated partition management.</p> Conclusion</a> </h2> DELETE operations are often a source of unpleasant surprises - not just by affecting performance and creating bloat, but by striking back at the times we least expect. To handle them effectively, focus on strategies such as batching, monitoring autovacuum, or leveraging partitioning for large datasets. By considering DELETE operations during schema design, you can maintain an efficient database, reduce maintenance headaches, and ensure it continues to run smoothly as your data grows.</p> Text identifiers in PostgreSQL database design 2024-11-09T00:00:00+00:00 Whether you are designing a standalone application or a microservice, you will inevitably encounter the topic of sharing identifiers. Whether it’s URLs of web pages, RESTful API resources, JSON documents, CSV exports, or something else, the identifier of specific resources will be exposed.</p> /orders/123</span></span> /products/345/variants/1</span></span></code></pre> While an identifier is just a number and does not carry any negative connotations, there are valid reasons why you might want to avoid exposing them. These reasons include:</p> Security and Data Exposure</strong>: Numerical identifiers are sequential and predictable, which can expose information about the underlying data source (e.g., the volume of the data) and provide a basis for ID enumeration.</li> Privacy and Confidentiality</strong>: Concerns may arise about concealing the volume of referenced data. For example, the number of customers, clients, or orders might be information a business prefers to keep private.</li> Non-descriptive Nature</strong>: Integers as identifiers can lead to confusion. An ID like 123 does not convey any additional information, making debugging edge cases more challenging.</li> </ol> These and other reasons (like SEO optimisation) have led to the increased use of text-based identifiers. Their readability and versatility make them ideal for external data sharing.</p> However, in database (or data model) design, the advantages of text identifiers are often overshadowed by the problems they introduce. While text identifiers improve interoperability, they frequently come with performance and storage trade-offs. In contrast, integers are naturally faster and more efficient to process, resulting in lower storage requirements and faster indexing, sorting, and searching-tasks that computers are optimised for.</p> In this article, we will explore scenarios where using text identifiers directly in the database design might seem natural and discuss strategies for using them effectively.</p> What Makes Text Identifiers Appealing?</a> </h3> Let’s be honest-text identifiers are popular for a reason. For humans, they are much more readable and, in some cases, add extra context. (Raise your hand if you don’t appreciate Heroku’s quirky and memorable names!) If chosen carefully, they can also be easier to recall.</p> Text identifiers can embed additional context. Consider the order number APAC-20241103-8237, which encodes both the region and the order date. Another popular reason for using text identifiers is their uniqueness in distributed environments.</p> They’re especially handy when people need to interact with them directly. For example, customers copying an order number from an email or support teams discussing an issue benefit from a readable, meaningful identifier. It’s simpler, more intuitive, and less likely to cause headaches when someone tries to recall or share it.</p> When an Identifier Isn’t Just an Identifier</a> </h3> Problems with text identifiers arise when they are used as natural keys in your data or database model. Despite their benefits, text identifiers often make poor primary keys for several reasons:</p> Context Changes</strong>: The additional context provided by text identifiers will likely change, necessitating updates. Despite assurances to the contrary, changes are inevitable.</li> Sorting Issues</strong>: Sorting text identifiers can be tricky, particularly with locale-based sorting or when numbers are embedded within the identifier (e.g., order1434 vs. order349).</li> </ul> The ultimate issue is efficiency:</p> Text identifiers typically require more storage space</strong> than numeric ones. Each character in a text field occupies more bytes than a simple integer, leading to larger database sizes and slower performance, especially during indexing or handling large datasets.</li> Databases are optimised for numeric operations, making searches, joins, and indexing on text fields inherently slower</strong>. This performance gap can significantly affect large datasets, impacting application efficiency.</li> Text identifiers complicate managing relationships between tables</strong>. The additional storage requirements affect not only the source table but also all referencing entities. Multiply this increased storage by the number of referencing tables, and you’ll get a sense of the overall impact.</li> </ul> As databases grow, these issues become more pronounced. The combination of increased storage needs and slower operations contributes to database bloat</strong>, which can degrade system performance. Additionally, bloated indexes can mislead the query planner into making suboptimal choices, such as favouring sequential scans over index scans, complicating regular operations further.</p> Aren't UUIDs the Solution to All This?</a> </h3> It depends. While UUIDs offer numerical representation and are excellent for unique key generation across distributed systems, they are not always the best choice:</p> In comparison to BIGINT, there are few real-world scenarios that necessitate the full range of UUIDs. Premature optimisation most often isn’t worthwhile for most solutions.</li> UUIDs’ 16-byte (128-bit) storage size can be less efficient than many text identifiers. Even BIGINT, at 8 bytes (64 bits), is more storage-efficient. This inefficiency extends to indexes, joins, and other operations.</li> </ul> A matter of opinion: UUIDs are plain ugly.</li> </ul> Moreover, different versions of UUIDs offer varying benefits. For example, UUIDv1 includes a timestamp component, making it somewhat sortable, while UUIDv4 is entirely random. Even sortable UUIDs may not offer significant advantages over more streamlined options like BIGINT or carefully structured text identifiers.</p> In most cases, the trade-offs in performance and readability make UUIDs less appealing unless their globally unique nature is essential across distributed systems.</p> Real-Life Examples</a> </h3> Let’s move beyond theory and explore practical examples:</p> CREATE TABLE</span> sessions</span> (</span></span> token </span>TEXT PRIMARY KEY</span>,</span></span> user_id </span>INT NOT NULL REFERENCES</span> users(user_id),</span></span> ...</span></span> );</span></span> </span> CREATE TABLE</span> products</span> (</span></span> sku </span>TEXT PRIMARY KEY</span>,</span></span> label </span>TEXT NOT NULL</span>,</span></span> ...</span></span> );</span></span> </span> CREATE TABLE</span> documents</span> (</span></span> document_id </span>TEXT PRIMARY KEY</span>,</span></span> ...</span></span> );</span></span></code></pre> In these cases, using text identifiers as primary keys might seem logical because:</p> They serve as natural keys for the entity.</li> </ul> They are likely propagated outside the service scope, hence needing storage anyway.</li> They are not expected to change.</li> The storage impact for a single table seems negligible.</li> </ul> However, this logic falters under scrutiny. Text identifiers often propagate beyond the service scope, leading to pressure to update them. Updating primary keys is no trivial matter. Consider:</p> While SKUs ideally never change, real-world scenarios like rebranding, product consolidation, or supplier changes can necessitate updates. Despite their appeal, SKUs are poor primary key candidates.</li> Randomly generated text identifiers (like session tokens) will… TBD.</li> </ul> The real issue arises when referencing these identifiers across tables:</p> CREATE TABLE</span> session_logs</span> (</span></span> log_id </span>BIGINT GENERATED ALWAYS AS IDENTITY</span>,</span></span> token </span>TEXT NOT NULL REFERENCES sessions</span>(token),</span></span> ...</span></span> );</span></span> </span> CREATE TABLE</span> product_reviews</span> (</span></span> review_id </span>INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> product_sku </span>TEXT NOT NULL REFERENCES</span> products(sku),</span></span> ...</span></span> );</span></span> </span> CREATE TABLE</span> customer_orders</span> (</span></span> order_id </span>TEXT PRIMARY KEY</span>,</span></span> customer_id </span>TEXT NOT NULL REFERENCES</span> customers(customer_id),</span></span> ...</span></span> );</span></span> </span> CREATE TABLE</span> document_revisions</span> (</span></span> revision_id </span>INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> document_id </span>TEXT NOT NULL REFERENCES</span> documents(document_id),</span></span> ...</span></span> );</span></span></code></pre> In such scenarios, text identifiers can become problematic. Firstly, you lose the ability to modify them</strong>. Secondly, the increased storage requirements</strong> become evident. For instance, an additional 100 bytes per reference across a million records results in an extra 100 MB of storage for just one reference.</p> However, storage isn’t the biggest concern; indexing</strong> is. In PostgreSQL, indexing text fields - whether as primary or foreign keys - requires significantly more space than indexing numeric fields. This leads to bloated indexes, slower lookups, and more fragmented operations, particularly in queries relying on foreign key relationships. As a result, the query planner might resort to full table scans instead of index scans, further degrading performance.</p> These performance issues often remain undetected in development or testing environments but can cause significant disruptions in production.</p> Reintroducing Text Identifiers Effectively</a> </h3> The goal of this post is not to discourage the use of text identifiers entirely but to highlight why they are unsuitable as primary keys. Here are some strategies to handle them effectively:</p> 1. Introduce a Surrogate Key</a> </h4> One straightforward solution is to introduce a surrogate primary key, replacing references to text identifiers:</p> CREATE TABLE</span> products</span> (</span></span> product_id </span>INT GENERATED BY DEFAULT AS IDENTITY</span>,</span></span> sku </span>TEXT NOT NULL</span>,</span></span> label </span>TEXT NOT NULL</span>,</span></span> ...</span></span> );</span></span> </span> CREATE INDEX</span> products_by_sku</span> ON</span> products(sku);</span></span> </span> CREATE TABLE</span> product_reviews</span> (</span></span> review_id </span>SERIAL PRIMARY KEY</span>,</span></span> product_id </span>INT REFERENCES</span> products(product_id),</span></span> ...</span></span> );</span></span></code></pre> This approach maintains efficient retrieval by SKU via an index.</p> 2. Use Mapping Tables for Greater Flexibility</a> </h4> Mapping tables allow you to:</p> Update identifiers without affecting the parent entity.</span></span> * Maintain a history of text identifiers linked to a specific entity.</span></span></code></pre>CREATE TABLE</span> products</span> (</span></span> product_id </span>INT GENERATED BY DEFAULT AS IDENTITY</span>,</span></span> label </span>TEXT NOT NULL</span>,</span></span> ...</span></span> );</span></span> </span> CREATE TABLE</span> product_skus</span> (</span></span> product_sku_id </span>INT GENERATED BY DEFAULT AS IDENTITY</span>,</span></span> product_id </span>INT REFERENCES</span> products(product_id),</span></span> sku </span>TEXT NOT NULL</span>,</span></span> created_at </span>TIMESTAMPTZ DEFAULT</span> CURRENT_TIMESTAMP,</span></span> deleted_at </span>TIMESTAMPTZ</span></span> );</span></span> </span> CREATE UNIQUE INDEX</span> unique_product_skus</span> ON</span> product_skus (product_id, sku) </span>WHERE</span> deleted_at </span>IS NULL</span>;</span></span></code></pre> This approach accommodates changing SKUs without compromising data integrity. A related use case could be linking a mapping table to a product variant:</p> CREATE TABLE</span> product_variants</span> (</span></span> variant_id </span>INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY</span>,</span></span> product_id </span>INT NOT NULL REFERENCES</span> products(product_id),</span></span> sku </span>TEXT NOT NULL</span>,</span></span> name TEXT NOT NULL</span>,</span></span> ...</span></span> );</span></span></code></pre>3. Decompose the Text Identifier’s Meaning</a> </h4> Text identifiers often embed meaningful information, such as regions, dates, or categories. By decomposing the identifier, you can store this contextual data separately, improving both flexibility and performance.</p> For example, instead of storing an order identifier like ORD-EMEA-00789 directly, you can design a more robust schema:</p> CREATE TABLE</span> orders</span> (</span></span> order_id </span>INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY</span>,</span></span> region_id </span>INT NOT NULL REFERENCES</span> regions(region_id),</span></span> order_date </span>DATE NOT NULL</span>,</span></span> ...</span></span> );</span></span> </span> CREATE TABLE</span> regions</span> (</span></span> region_id </span>INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY</span>,</span></span> name TEXT NOT NULL</span>,</span></span> code </span>TEXT NOT NULL</span>,</span></span> ...</span></span> );</span></span> </span> INSERT INTO</span> regions (</span>name</span>, code) </span>VALUES</span></span> (</span>'Europe, the Middle East, and Africa'</span>, </span>'EMEA'</span>),</span></span> (</span>'Asia Pacific'</span>, </span>'APAC'</span>);</span></span> </span> To</span> generate a user</span>-</span>friendly order </span>number</span>, you can </span>create</span> a </span>function</span>:</span></span> </span> CREATE OR REPLACE FUNCTION</span> get_order_number</span>(p_order_id </span>INT</span>)</span></span> RETURNS TEXT AS</span> $$</span></span> DECLARE</span></span> v_region_code </span>TEXT</span>;</span></span> v_formatted_order_id </span>TEXT</span>;</span></span> v_order_number </span>TEXT</span>;</span></span> BEGIN</span></span> SELECT</span> r</span>.</span>code</span> INTO</span> v_region_code</span></span> FROM</span> orders o</span></span> JOIN</span> regions r </span>ON</span> o</span>.</span>region_id</span> =</span> r</span>.</span>region_id</span></span> WHERE</span> o</span>.</span>order_id</span> =</span> p_order_id;</span></span> </span> IF NOT</span> FOUND </span>THEN</span></span> RETURN NULL</span>;</span></span> END IF</span>;</span></span> </span> v_formatted_order_id :</span>=</span> TO_CHAR(p_order_id, </span>'FM00000'</span>);</span></span> v_order_number :</span>=</span> 'ORD-'</span> \|\|</span> v_region_code </span>\|\|</span> '-'</span> \|\|</span> v_formatted_order_id;</span></span> </span> RETURN</span> v_order_number;</span></span> END</span>;</span></span> $$ </span>LANGUAGE</span> plpgsql;</span></span></code></pre> This method enables you to store and retrieve structured data efficiently while still providing a readable and meaningful identifier for external use.</p> 4. Reversible Text IDs</a> </h4> For scenarios requiring enhanced security or privacy, where identifiers should not be easily enumerable or predictable, you can use reversible text-based IDs. These allow you to present users with seemingly random text representations while maintaining efficient numerical storage internally.</p> Sqids</a></strong> is a project that can help achieve this. It generates URL-friendly unique identifiers from numbers and can encode multiple numerical identifiers into a single string. Here’s an example using the Sqids project:</p> [42] -> JgaEBgznCpUZo3Kk</span></span> [42, 430004] -> lTiYlvsGkh59m1PQ</span></span></code></pre> The generated identifiers are reversible with knowledge of the shared alphabet, allowing you to decode requests without hitting the database, which can be beneficial in high-throughput environments. This technique is particularly useful for user, account, or session identifiers, balancing the need for security with operational efficiency.</p> However, it’s crucial to remember that this is not a substitute for robust security practices. Proper authentication and authorisation mechanisms are still necessary to secure your application.</p> By thoughtfully integrating these strategies, you can leverage the benefits of text identifiers where appropriate while avoiding common pitfalls in database design. This balance ensures efficient, maintainable systems that meet both technical and business needs.</p> 5. Leverage Generate columns</a> </h4> In edge case scenarios, where keeping text identifiers with embedded context is necessary, generated columns</strong> in PostgreSQL can be a valuable feature. Since version 12, PostgreSQL allows defining columns whose values are automatically computed from other columns in the table. This ensures consistency without manual intervention.</p> For example, you can define a function to handle the formatting logic:</p> CREATE OR REPLACE FUNCTION</span> get_formatted_order_id</span>(p_region_id </span>INT</span>, p_order_id </span>INT</span>)</span></span> RETURNS TEXT AS</span> $$</span></span> DECLARE</span></span> v_region_code </span>TEXT</span>;</span></span> BEGIN</span></span> v_region_code :</span>= CASE</span> p_region_id</span></span> WHEN</span> 1</span> THEN</span> 'EMEA'</span></span> WHEN</span> 2</span> THEN</span> 'APAC'</span></span> ELSE</span> 'OTHER'</span></span> END</span>;</span></span> </span> RETURN</span> 'ORD-'</span> \|\|</span> v_region_code </span>\|\|</span> '-'</span> \|\|</span> LPAD(p_order_id::</span>TEXT</span>, </span>5</span>, </span>'0'</span>);</span></span> END</span>;</span></span> $$ </span>LANGUAGE</span> plpgsql IMMUTABLE;</span></span></code></pre> The function is marked IMMUTABLE because PostgreSQL requires generated columns to use only immutable functions-those guaranteed to return the same result for the same input every time.</p> CREATE TABLE</span> orders</span> (</span></span> order_id </span>INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY</span>,</span></span> region_id </span>INT NOT NULL</span>,</span></span> order_date </span>DATE NOT NULL</span>,</span></span> formatted_order_id </span>TEXT GENERATED ALWAYS AS</span> (</span></span> get_formatted_order_id(region_id, order_id)</span></span> ) STORED</span></span> );</span></span></code></pre> This setup ensures that formatted_order_id</code> is automatically updated whenever region_id</code> or order_id</code> changes. The column is physically stored, enhancing read performance for frequently queried data.</p> Using generated columns can simplify maintaining consistency in derived values like formatted text identifiers. However, be mindful of their characteristics, such as:</p> Automated Updates</strong>: The system automatically recomputes the column’s value when the referenced columns change.</li> Immutability Requirements</strong>: Only immutable functions can be used, ensuring reliable and consistent computation.</li> </ul> Wrapping up</a> </h3> Text identifiers will stay with us, and that's great. They’re human-readable, memorable, and can pack a lot of meaningful context into a simple string. They make external interactions smoother, whether it’s a customer referencing an order number or a support team tracing an issue. They even add a bit of charm and personality to otherwise sterile identifiers.</p> However it's important to keep their usage in control. The nice rule I've heard is</p> Use text identifiers when communicating externally</strong> - for example, in URLs or API responses</li> Internally, always rely on numerical IDs</strong> - surrogate keys like INT or BIGINT (even UUID if you go for planet dominanance) where necessary, to maintain database efficiency and integrity.</li> </ul> This approach allows you to leverage the strengths of text identifiers for external communication while keeping your database optimised for performance and scalability. By storing text identifiers in dedicated fields and using numeric primary keys for internal operations, you achieve a the right balance and do best to maintain the system performance.</p> We need to talk about ENUMs 2024-09-04T00:00:00+00:00 Designing a database schema, whether for a new application or a new feature, always raises a lot of questions. The choices you make can have a big impact on how well your database performs and how easy it is to maintain and scale. Whether you’re just getting started with PostgreSQL or consider yourself a seasoned pro, it’s easy to rely on old habits or outdated advice. In this article, I want to take a fresh look at one of those topics that often sparks debate: the use of ENUMs in PostgreSQL.</p> I have to admit, not so long ago, I would advise "don't use ENUMs" without thinking about it too much. Relying only on random articles and some personal but outdated experience, this had been my go-to answer for some years. And while there were several limitations of ENUMs in PostgreSQL in the (distant) past, the support has improved a long time ago.</p> The improvements are so long-standing that:</p> PostgreSQL 9.1 (released in 2011) introduced the option to add new values to ENUMs without a table rewrite</strong>. From what I can say this fact alone remained the source of biggest misconception when thinking about ENUMs.</li> Renaming of values</strong> was added in version 10 (2017).</li> The ability to ALTER TYPE ... ADD VALUE</code> in a transaction block was introduced in PostgreSQL 12 (2019).</li> </ul> And new features coming (at the time of writing this article) with PostgreSQL 17</strong> allow the use of newly added values within the same transaction block (previously not possible without an explicit commit).</p> Therefore, I want to correct even myself and say—let's give ENUMs another chance. This article will go into detail and help us correctly decide when it makes sense to use them or not.</p> How are ENUMs Implemented?</a> </h2> Every stored ENUM value occupies 4 bytes on disk. This is the size of the OID, representing the actual ENUM value, stored as a row within the pg_enum</code> table. You can see this yourself by querying the table directly, but it will come by default with no data defined.</p> # select * from pg_catalog.pg_enum;</span></span> </span> oid \| enumtypid \| enumsortorder \| enumlabel</span></span> -----+-----------+---------------+-----------</span></span> (0 rows)</span></span></code></pre> But once you define a new ENUM data type, using:</p> CREATE TYPE</span> order_status</span> AS</span> ENUM (</span>'new'</span>, </span>'pending'</span>, </span>'processing'</span>, </span>'shipped'</span>, </span>'delivered'</span>);</span></span></code></pre> You will get the actual OIDs.</p> # select * from pg_catalog.pg_enum;</span></span> </span> oid \| enumtypid \| enumsortorder \| enumlabel</span></span> --------+-----------+---------------+------------</span></span> 211018 \| 211016 \| 1 \| new</span></span> 211020 \| 211016 \| 2 \| pending</span></span> 211022 \| 211016 \| 3 \| processing</span></span> 211024 \| 211016 \| 4 \| shipped</span></span> 211026 \| 211016 \| 5 \| delivered</span></span></code></pre> The table straightforwardly identifies both the type and individual values, but also the enumsortorder</code>, which is the foundation for BEFORE/AFTER</code> new value(s) definition.</p> ALTER TYPE</span> order_status </span>ADD VALUE</span> 'cancelled'</span> BEFORE</span> 'shipped'</span>;</span></span></code></pre> Resulting in:</p> oid \| enumtypid \| enumsortorder \| enumlabel</span></span> --------+-----------+---------------+------------</span></span> 211018 \| 211016 \| 1 \| new</span></span> 211020 \| 211016 \| 2 \| pending</span></span> 211022 \| 211016 \| 3 \| processing</span></span> 211024 \| 211016 \| 4 \| shipped</span></span> 211026 \| 211016 \| 5 \| delivered</span></span> 211027 \| 211016 \| 3.5 \| cancelled</span></span></code></pre> To confirm the actual storage size of the ENUM value, you can use a sample table, the previously defined type order_status</code>, a single data row, and the pg_column_size</code> function.</p> CREATE TABLE</span> orders</span> (</span></span> id </span>SERIAL PRIMARY KEY</span>,</span></span> status</span> order_status </span>NOT NULL</span>,</span></span> order_date </span>DATE NOT NULL DEFAULT</span> CURRENT_DATE</span></span> );</span></span> </span> INSERT INTO</span> orders (</span>status</span>, order_date) </span>VALUES</span></span> (</span>'new'</span>, </span>'2024-09-01'</span>);</span></span></code></pre> This gives you confirmation of the actual 4-byte storage.</p> # SELECT id, status, pg_column_size(status) AS status_size</span></span> </span> id \| status \| status_size</span></span> ----+--------+-------------</span></span> 1 \| new \| 4</span></span></code></pre> Returning to the pg_enum</code> table, the structure of the table provides other important clues:</p> enumlabel</code> is of type name</code>, which is effectively a 63-byte varchar for storing system identifiers.</li> The UNIQUE CONSTRAINT pg_enum_typid_label_index</code> prevents us from creating two values with the same name.</li> However, given the nature of VARCHAR</code>, it is case-sensitive, allowing you to define new</code> and NEW</code> as two distinct values.</li> </ul> Life with ENUMs</a> </h2> Now that we understand how ENUMs are implemented, let's review the lifecycle of such a data type in a typical database.</p> As demonstrated above, creation of new ENUMs</strong> is simple, and since version 12 (and expanded in 17), it's possible even in transaction blocks, making it easier to work with database migration tools without needing to explicitly worry about transaction management.</p> Adding new values</strong> does not require a table rewrite and can be done without further hesitation. You can also safely rename existing values</strong>. Similarly, you can use ENUMs in arrays, indexes, or compare them as needed (based on their order).</p> ALTER TYPE name ADD VALUE</span> [ IF NOT EXISTS ] new_enum_value [ { BEFORE \| AFTER } neighbor_enum_value ]</span></span> </span> ALTER TYPE name</span> RENAME </span>VALUE</span> existing_enum_value </span>TO</span> new_enum_value</span></span></code></pre> It's also important to mention that ENUMs allow you to use a default value, like this:</p> status order_status NOT NULL DEFAULT 'new'</span></span></code></pre> Unfortunately, this is where the flexibility ends at the moment. There's no easy way to remove or re-order existing values (please, don't modify pg_enum</code>—or any system tables, for that matter). The topic of removing existing ENUM value(s) can easily open up a whole discussion about the best way to 'deprecate' and drop a value over time. The only suitable way to perform these operations is by creating a new type and altering the column. Please be aware of the ways described in How not to change PostgreSQL column type</a></strong> will apply in this scenario.</p> Using CHECK Constraints as an Alternative to ENUMs</a> </h2> While ENUMs are a powerful tool in PostgreSQL for enforcing a fixed set of values in a column, they come with the limitations mentioned above. An alternative approach to achieving similar functionality is to use a CHECK constraint with a TEXT column. This method offers more flexibility when managing the allowed values, especially in scenarios where the set of valid values might change frequently.</p> A CHECK constraint can be applied to a column to enforce that its value must be one of a predefined set of values. Here’s how you can define a table using a CHECK constraint as an alternative to an ENUM:</p> CREATE TABLE</span> orders</span> (</span></span> id </span>SERIAL PRIMARY KEY</span>,</span></span> status TEXT NOT NULL</span>,</span></span> order_date </span>DATE NOT NULL DEFAULT</span> CURRENT_DATE,</span></span> CHECK</span> (</span>status IN</span> (</span>'new'</span>, </span>'pending'</span>, </span>'processing'</span>, </span>'shipped'</span>, </span>'delivered'</span>))</span></span> );</span></span></code></pre> Modification of the constraints is possible, but you will always have to DROP CONSTRAINT</code> and then ADD CONSTRAINT</code>.</p> ALTER TABLE</span> orders </span>DROP CONSTRAINT</span> orders_status_check;</span></span> </span> ALTER TABLE</span> orders </span>ADD CONSTRAINT</span> orders_status_check </span>CHECK</span> (</span>status IN</span> (</span>'new'</span>, </span>'pending'</span>, </span>'processing'</span>, </span>'shipped'</span>, </span>'delivered'</span>, </span>'returned'</span>));</span></span></code></pre> While this operation is not as 'heavyweight' as changing the data type, please be aware that an ACCESS EXCLUSIVE</strong> lock will be acquired for the entire table—meaning no other operations can be performed on the table. With a growing table size and increasing concurrency, this might still lead to application-level downtime.</p> Another limitation is the lack of direct support for sorting, meaning you will have to implement it manually in every query or abstract the sorting detail using a view.</p> SELECT</span> id, </span>status</span>, order_date</span></span> FROM</span> orders</span></span> ORDER BY</span></span> ARRAY_POSITION(</span></span> ARRAY</span>['new', 'pending', 'processing', 'shipped', 'delivered', 'returned'],</span></span> status</span></span> );</span></span></code></pre>Reference Tables as the Real Alternative</a> </h2> It might be surprising, but one of the motivations for using either ENUMs or CHECK constraints is to avoid additional JOINs. It's no surprise that JOINs are a source of pain for many people who struggle with SQL, and therefore they try to limit their use. While there might be negligible impact on performing such an operation, there are no other reasons not to use reference tables—it’s not storage (unless you over-optimise with smallint</code> and extremely large datasets) and definitely not operational considerations.</p> In fact, reference tables are often better in most cases. They not only address all the use cases supported by both ENUMs and CHECK constraints, but they can also resolve the limitations of these methods.</p> Creating and Using a Reference Table</a> </h3> Let’s create a reference table similar to the ENUM data type we’ve discussed above:</p> CREATE TABLE</span> order_statuses</span> (</span></span> status_id </span>INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> status_name </span>TEXT NOT NULL</span></span> );</span></span> </span> CREATE UNIQUE INDEX</span> unique_status_name</span> ON</span> order_statuses (</span>LOWER</span>(status_name));</span></span> </span> INSERT INTO</span> order_statuses (status_name) </span>VALUES</span></span> (</span>'new'</span>),</span></span> (</span>'pending'</span>),</span></span> (</span>'processing'</span>),</span></span> (</span>'shipped'</span>),</span></span> (</span>'delivered'</span>),</span></span> (</span>'cancelled'</span>);</span></span></code></pre>Updating the Orders Table to Use the Reference Table</a> </h3> To use the reference table, you’ll need to alter your existing orders</code> table. This involves dropping the current status</code> column if it’s using an ENUM or CHECK constraint, and replacing it with a foreign key that references the order_statuses</code> table:</p> ALTER TABLE</span> orders </span>DROP</span> COLUMN </span>IF EXISTS status</span>;</span></span> </span> ALTER TABLE</span> orders </span>ADD</span> COLUMN status_id </span>INT</span>,</span></span> ADD CONSTRAINT</span> fk_order_status</span></span> FOREIGN KEY</span> (status_id) </span>REFERENCES</span> order_statuses(status_id);</span></span></code></pre>Impact of Using Reference Tables</a> </h3> While reference tables do not provide anything for free, such as ENUM ordering or simple CHECK constraints, they offer far greater flexibility. Here are some examples of what you can achieve with reference tables:</p> Custom Sorting Logic:</strong> You can implement specific sorting by adding an additional sort_order</code> column to the order_statuses</code> table. This allows you to easily change the order of statuses without altering the schema.</li> Deprecating Values:</strong> By adding a deleted_at</code> column (or a similar field), you can deprecate certain status values. Combined with a trigger, you can prevent these deprecated values from being used in new data while maintaining historical records.</li> Renaming Values:</strong> Unlike ENUMs, renaming a status is straightforward - simply update the value in the order_statuses</code> table.</li> Internationalisation:</strong> If your application needs to support multiple languages, you can extend the order_statuses</code> table with additional columns for different languages or even create a separate lookup table that stores translations.</li> Additional Metadata:</strong> Reference tables can store extra information about each status, such as descriptions, colours (for UI purposes), or associated icons, which can be particularly useful in applications with rich user interfaces.</li> </ul> Having mentioned all advantages of using Reference Tables, it's also important to mention the fact it's not straightforward to set the default values for such a references. Definitely not in obvious way which ENUMs provide. The real alternative is to create table/column without default value and alter it specifically for given database using ALTER COLUMN ... SET DEFAULT</code></p> ALTER TABLE</span> orders</span></span> ALTER</span> COLUMN status_id </span>SET DEFAULT</span> (</span>SELECT</span> status_id </span>FROM</span> order_statuses </span>WHERE</span> status_name </span>=</span> 'new'</span>);</span></span></code></pre>Performance Considerations: ENUMs, CHECK Constraints, and Reference Tables</a> </h2> Thanks to their internal representation as OIDs, ENUMs offer predictable performance and indexing strategies, often outperforming text-based CHECK constraints. Since ENUM comparisons rely on integers, they tend to be faster than string comparisons required for CHECK constraints. Reference tables, on the other hand, inherently require a JOIN, which introduces the overhead of accessing two tables and potentially working with multiple indexes.</p> While I initially planned to publish benchmarks comparing the performance of all three approaches, I decided not to include specific numbers. It’s true that ENUMs consistently performed the fastest, and reference tables were somewhat slower due to the necessary JOIN. However, as with most benchmarks, the scenarios were artificial. The actual performance difference, though measurable, is unlikely to be significant in practical applications. In any reasonably complex query, other factors are far more likely to impact performance than whether you use ENUMs or not.</p> Conclusion: Use ENUMs Where They Make Sense</a> </h2> As with many decisions in software development, the use of ENUMs comes down to "it depends." This article has provided clarity on how ENUMs work, where their limitations lie, and how to make an informed decision about when to use them.</p> To recap:</p> ENUMs</strong> are ideal for small, static sets of values that require strict type enforcement and minimal changes, preferably data types that are not directly tied to the business logic.</li> CHECK constraints</strong> offer more flexibility than ENUMs, making them a good choice when the allowed values might change, but the data type remains simple, and there's no</li> Reference tables</strong> provide the most flexibility and scalability, making them the go-to choice for dynamic or complex value sets and when you want to avoid the limitations of ENUMs and CHECK constraints.</li> </ul> But in the end, no plan ever survives first contact with reality. It’s essential to revisit and reconsider your decisions as your application evolves, ensuring that your database schema continues to serve the needs of your project as effectively as possible.</p> And while it's not meant as advice - personally I'm still going to gravitate towards the advice "don't use ENUMs" though. I'm yet to see value-based field that does not change over time.</em></p> Beyond Simple Upserts with MERGE in PostgreSQL 2024-08-25T00:00:00+00:00 Understanding how comfortable someone is with databases and SQL often comes down to the features they use. In PostgreSQL, one such feature that distinguishes more advanced users is the MERGE</code> command, introduced in version 15 and expanded in version 17 (in beta at the time of writing this article). Before MERGE</code>, developers typically relied on INSERT ... ON CONFLICT DO UPDATE</code> for upserts—a method introduced in PostgreSQL 9.5 that has since become a staple in many developers' toolkits.</p> While ON CONFLICT</code> offers a straightforward solution for simple upsert scenarios, it can quickly become limiting as business logic grows in complexity. This is where the MERGE</code> command excels. Introduced in the SQL:2003 standard, MERGE</code> allows for more sophisticated data synchronisation tasks by combining multiple operations—such as conditional inserts, updates, and deletes—into a single, atomic statement.</p> In this article, we’ll explore the capabilities of the MERGE</code> command, comparing it with traditional upsert methods and examining how it can streamline database operations. Through practical examples, we'll illustrate how MERGE</code> can simplify even the most complex workflows, making it an indispensable tool for developers working with PostgreSQL.</p> This introduction sets the stage by highlighting the evolution from ON CONFLICT</code> to MERGE</code>, explains the context in which MERGE</code> becomes valuable, and clearly outlines what the reader will gain from the article.</p> Hands-on Example: Managing a Score Points System for a Mobile Game</a> </h2> Imagine you’re managing a tournament score system for an online game, where players earn special status by participating in weekly tournaments. The simple business logic would be:</p> Players can earn score points by participating in the tournament.</li> Players must participate in all tournaments to maintain their score (i.e., maintain the streak).</li> Players who participate in a second consecutive tournament can achieve the status of veteran</code> (if they reach 1,000 score points) or gain the status of star</code> (if they reach 100 or more score points).</li> </ul> The score points are updated weekly, based on the outcome of the previous week's participation.</p> We will start with a simple database schema consisting of a table tournament_scores</code> that tracks (as the name suggests) only active users.</p> CREATE TABLE</span> tournament_scores</span> (</span></span> player_id </span>INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> player_name </span>TEXT UNIQUE NOT NULL</span>,</span></span> score </span>INT NOT NULL DEFAULT</span> 0</span>,</span></span> status TEXT NOT NULL DEFAULT</span> 'newbie'</span> CHECK</span> (</span>status IN</span> (</span>'newbie'</span>, </span>'veteran'</span>, </span>'star'</span>))</span></span> );</span></span></code></pre> And populate it with some sample data:</p> INSERT INTO</span> tournament_scores (player_name, score, </span>status</span>)</span></span> VALUES</span></span> (</span>'PlayerOne'</span>, </span>900</span>, </span>'newbie'</span>), </span>-- Regular player, close to Veteran promotion</span></span> (</span>'PlayerTwo'</span>, </span>1200</span>, </span>'veteran'</span>), </span>-- Already a Veteran player</span></span> (</span>'PlayerThree'</span>, </span>300</span>, </span>'newbie'</span>); </span>-- Regular player with a lower score</span></span></code></pre>Upsert using ON CONFLICT</code></a> </h2> When activity data is received, there are multiple ways to process it. ON CONFLICT</code> is used here for demonstration purposes.</p> INSERT INTO</span> tournament_scores (player_name, score)</span></span> VALUES</span></span> (</span>'PlayerOne'</span>, </span>50</span>), </span>-- Add points for PlayerOne</span></span> (</span>'PlayerTwo'</span>, </span>120</span>), </span>-- Add points for PlayerTwo</span></span> (</span>'PlayerFour'</span>, </span>70</span>) </span>-- New player without an account, PlayerFour</span></span> ON</span> CONFLICT (player_name)</span></span> DO </span>UPDATE SET</span></span> score </span>=</span> tournament_scores</span>.</span>score</span> +</span> EXCLUDED</span>.</span>score</span>,</span></span> status = CASE WHEN</span> (</span>tournament_scores</span>.</span>score</span> +</span> EXCLUDED</span>.</span>score</span>) </span>></span> 1000</span> THEN</span> 'veteran'</span> ELSE</span> 'newbie'</span> END</span>;</span></span></code></pre> This performs the basic requirements, updating the scores and possibly evaluating the status for existing tournament users. To remove users who no longer participated (and hence broke the streak) and apply other conditional logic, you would need to use separate statements.</p> Handling Upserts with MERGE</code></a> </h2> Now, let’s introduce the MERGE</code> command and implement the same logic as above.</p> MERGE INTO</span> tournament_scores ts</span></span> USING</span> (</span></span> VALUES</span></span> (</span>'PlayerOne'</span>, </span>50</span>), </span>-- Add points for PlayerOne</span></span> (</span>'PlayerTwo'</span>, </span>120</span>), </span>-- Add points for PlayerTwo</span></span> (</span>'PlayerFour'</span>, </span>70</span>) </span>-- New player without an account, PlayerFour</span></span> ) </span>AS</span> v(player_name, score_added)</span></span> ON</span> ts</span>.</span>player_name</span> =</span> v</span>.</span>player_name</span></span> WHEN MATCHED THEN</span></span> UPDATE SET</span></span> score </span>=</span> ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span>,</span></span> status = CASE WHEN</span> (</span>ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span>) </span>></span> 1000</span> THEN</span> 'veteran'</span> ELSE</span> 'newbie'</span> END</span></span> WHEN NOT MATCHED THEN</span></span> INSERT</span> (player_name, score)</span></span> VALUES</span> (</span>v</span>.</span>player_name</span>, </span>v</span>.</span>score_added</span>);</span></span></code></pre> Here, we define the table tournament_scores</code> as the target</strong> and the list of VALUES as the source</strong>. Using a conditional clause, we define the logic for both matched and unmatched entries, giving us both INSERT and UPDATE paths.</p> The evaluation paths in this case are called when_clauses</code>. The MERGE</code> statement allows you to specify multiple clauses with different conditions, for example, allowing you to award users star</code> status if they gain more than 100 score points within a given tournament.</p> MERGE INTO</span> tournament_scores ts</span></span> USING</span> (</span></span> VALUES</span></span> (</span>'PlayerOne'</span>, </span>50</span>), </span>-- Add points for PlayerOne</span></span> (</span>'PlayerTwo'</span>, </span>120</span>), </span>-- Add points for PlayerTwo</span></span> (</span>'PlayerFour'</span>, </span>70</span>) </span>-- New player without an account, PlayerFour</span></span> ) </span>AS</span> v(player_name, score_added)</span></span> ON</span> ts</span>.</span>player_name</span> =</span> v</span>.</span>player_name</span></span> WHEN MATCHED AND</span> v</span>.</span>score_added</span> ></span> 100</span> THEN</span></span> UPDATE SET</span> score </span>=</span> ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span>, </span>status =</span> 'star'</span></span> WHEN MATCHED THEN</span></span> UPDATE SET</span></span> score </span>=</span> ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span>,</span></span> status = CASE WHEN</span> (</span>ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span>) </span>></span> 1000</span> THEN</span> 'veteran'</span> ELSE</span> 'newbie'</span> END</span></span> WHEN NOT MATCHED THEN</span></span> INSERT</span> (player_name, score)</span></span> VALUES</span> (</span>v</span>.</span>player_name</span>, </span>v</span>.</span>score_added</span>);</span></span></code></pre> Here’s a summary of how it works:</p> The MERGE</code> command evaluates the when_clauses</code> in the order they are written.</li> If a row matches the condition specified in a clause, the action defined in that clause is performed, and the row is no longer eligible to be matched against subsequent when_clauses</code>.</li> </ol> While we introduced multiple evaluation paths, the MERGE</code> command in PostgreSQL goes beyond that and expands the WHEN NOT MATCHED</code> clause, which effectively becomes WHEN NOT MATCHED [BY TARGET]</code>. PostgreSQL allows you to specify WHEN NOT MATCHED BY SOURCE</code> to perform the necessary merge statement for the data not present in the source table.</p> Handling DELETEs with MERGE</code></a> </h2> The functionality completely missing from regular upserts with ON CONFLICT</code> is the ability to delete missing entries. In our sample scenario, we want to penalise players who broke their streak and did not participate in last week's tournament by effectively deleting their entries from the target table.</p> MERGE INTO</span> tournament_scores ts</span></span> USING</span> (</span></span> VALUES</span></span> (</span>'PlayerOne'</span>, </span>50</span>), </span>-- Add points for PlayerOne</span></span> (</span>'PlayerTwo'</span>, </span>120</span>), </span>-- Add points for PlayerTwo</span></span> (</span>'PlayerFour'</span>, </span>70</span>) </span>-- New player without an account, PlayerFour</span></span> ) </span>AS</span> v(player_name, score_added)</span></span> ON</span> ts</span>.</span>player_name</span> =</span> v</span>.</span>player_name</span></span> WHEN MATCHED AND</span> v</span>.</span>score_added</span> ></span> 100</span> THEN</span></span> UPDATE SET</span> score </span>=</span> ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span>, </span>status =</span> 'star'</span></span> WHEN MATCHED THEN</span></span> UPDATE SET</span> score </span>=</span> ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span></span> WHEN NOT MATCHED THEN</span></span> INSERT</span> (player_name, score)</span></span> VALUES</span> (</span>v</span>.</span>player_name</span>, </span>v</span>.</span>score_added</span>)</span></span> WHEN NOT MATCHED BY</span> SOURCE </span>THEN</span></span> DELETE</span>;</span></span></code></pre> The introduction of the DELETE</code> merge operation complements all possible MERGE</code> outcomes:</p> The first is merge_update</code>, applicable to WHEN MATCHED</code> and WHEN NOT MATCHED</code> clauses, consisting of regular UPDATE</code> operations.</li> The second is merge_insert</code> for the WHEN NOT MATCHED</code> clause, allowing (as the name implies) data to be inserted.</li> The DELETE</code> we introduce is part of merge_delete</code>.</li> </ul> Technically, there's also the ability to specify DO NOTHING</code> for all available when_clauses</code>, similar to upserts using ON CONFLICT</code>.</p> Using MERGE</code> Output</a> </h2> Starting from PostgreSQL 17, the MERGE</code> command has been extended to support RETURNING</code> clauses to process merged data further. When an INSERT</code> or UPDATE</code> action is performed, the new values of the target table's columns are used. When a DELETE</code> is performed, the old values of the target table's columns are used.</p> Conclusion</a> </h2> The MERGE</code> command significantly enhances the ability to handle complex INSERT</code> and UPDATE</code> logic by allowing multiple operations within a single query while also capturing even the most intricate scenarios. In this article, we have explored the syntax and common use cases of MERGE</code>. For further exploration, note that the source</code> dataset is where you can perform any data transformations required, making MERGE</code> particularly well-suited for ETL operations, data archival, cleanup tasks, and more.</p> Incorporating MERGE</code> into your PostgreSQL toolkit can simplify database operations, reduce the risk of data inconsistencies, and streamline your</p> Gentle Introduction to Window Functions in PostgreSQL 2024-07-07T00:00:00+00:00 Understanding the relationship between data points is crucial. For instance, you might need to identify the most recent orders for each customer or track changes in sensor readings over time. Unlike aggregate functions, which summarise data into a single row, it is window functions that allow you to analyse data while preserving each row’s details. This is the core of the logic, but don’t worry if you struggle to imagine the difference, as we will cover all of it in this article.</p> PostgreSQL supports SQL window functions, facilitating complex calculations across related rows within a table. These functions are particularly useful for tasks such as ranking entries, calculating running totals, finding moving averages, and comparing individual entries. Mastering window functions can significantly enhance your data analysis capabilities.</p> You can easily use similar window functions as you would with aggregation. Let’s take a simple example of sensor readings:</p> CREATE TABLE</span> sensor_readings</span> (</span></span> sensor_id </span>bigint</span>,</span></span> reading_value </span>decimal</span>,</span></span> reading_time </span>timestamp with time zone default</span> current_timestamp</span></span> );</span></span> </span> INSERT INTO</span> sensor_readings (sensor_id, reading_value, reading_time) </span>VALUES</span></span> (</span>1</span>, </span>32</span>.</span>7</span>, </span>'2024-07-01 11:24:34'</span>),</span></span> (</span>1</span>, </span>33</span>.</span>1</span>, </span>'2024-07-02 11:29:01'</span>),</span></span> (</span>1</span>, </span>33</span>.</span>1</span>, </span>'2024-07-02 12:03:59'</span>),</span></span> (</span>1</span>, </span>33</span>.</span>0</span>, </span>'2024-07-03 10:12:15'</span>),</span></span> (</span>2</span>, </span>32</span>.</span>8</span>, </span>'2024-07-01 13:17:01'</span>),</span></span> (</span>2</span>, </span>35</span>.</span>8</span>, </span>'2024-07-02 09:18:11'</span>),</span></span> (</span>3</span>, </span>29</span>.</span>1</span>, </span>'2024-07-01 13:54:03'</span>),</span></span> (</span>3</span>, </span>30</span>.</span>3</span>, </span>'2024-07-01 14:12:09'</span>),</span></span> (</span>3</span>, </span>31</span>.</span>5</span>, </span>'2024-07-02 16:07:43'</span>);</span></span></code></pre> When you start with the aggregation functions, it is straightforward for anybody familiar with SQL:</p> SELECT</span></span> sensor_id,</span></span> AVG</span>(reading_value)</span></span> FROM</span> sensor_readings</span></span> GROUP BY</span> sensor_id;</span></span></code></pre> with the output</p> sensor_id \| avg</span></span> -----------+---------------------</span></span> 3 \| 30.3000000000000000</span></span> 2 \| 34.3000000000000000</span></span> 1 \| 32.9750000000000000</span></span></code></pre> While similar, using the window function AVG we can get almost same result:</p> SELECT</span></span> sensor_id,</span></span> reading_value,</span></span> AVG</span>(reading_value) </span>OVER</span> (</span>PARTITION BY</span> sensor_id) </span>AS</span> avg_reading_value</span></span> FROM</span> sensor_readings;</span></span></code></pre> such as</p> sensor_id \| reading_value \| avg_reading_value</span></span> -----------+---------------+---------------------</span></span> 1 \| 32.7 \| 32.9750000000000000</span></span> 1 \| 33.1 \| 32.9750000000000000</span></span> 1 \| 33.1 \| 32.9750000000000000</span></span> 1 \| 33.0 \| 32.9750000000000000</span></span> 2 \| 32.8 \| 34.3000000000000000</span></span> 2 \| 35.8 \| 34.3000000000000000</span></span> 3 \| 29.1 \| 30.3000000000000000</span></span> 3 \| 30.3 \| 30.3000000000000000</span></span> 3 \| 31.5 \| 30.3000000000000000</span></span></code></pre> This reiterates the fundamental difference between the two sets of functions. As mentioned earlier, while the results for sensor_id</code> are the same in both cases, the aggregate function</strong> summarised it into a single row (grouped by sensor_id</code>), whereas the window function</strong> provides the value for the set of rows in the partition defined by sensor_id</code>.</p> After showing the average in a window function, it’s important to note that traditional aggregation functions might not be the most helpful in the context of window function logic. Despite functions like AVG</code>, COUNT</code>, and SUM</code>—which are the most used aggregation functions—being available as window functions, we used the above only to demonstrate the difference.</p> OVER clause</a> </h2> Before diving into the individual functions, let’s cover the syntax first. From the sample query above, you already get the basic syntax of the window functions, with the OVER</code> clause being a primary identification of windowing functionality.</p> window_function ([expression...]) OVER window_definition</span></span></code></pre> In our example, the basic window functions can be similar to the aggregate ones— like AVG</code>, SUM</code> and COUNT</code> - and a number of the functions we will cover shortly.</p> The window definition part specifies how the window function will see the data it works over. It can include:</p> Partitioning</strong> to divide the result sets into separate partitions (think groups in aggregate functions).</li> Definition of the ordering</strong> of the rows within each partition.</li> Specification of how to apply framing</strong> of the subset of rows for each row’s calculation.</li> </ol> From all the components of the window syntax, only partitioning is mandatory.</p> If we revisit our first window function example above, you can identify the partitioning</strong> part (PARTITION BY sensor_id</code>). As already mentioned, you can easily compare the partitioning logic to the grouping used in aggregate functions, with the same properties—like partitioning data segments by multiple fields, using functions and other logic.</p> With partitioning comes hand in hand ordering</strong> of the data. When you start thinking of row properties, instead of grouping data into a single row, order starts to make a difference. Let’s take the following example:</p> window_function ([expression...]) </span>OVER</span> (</span>PARTITION BY</span> sensor_id </span>ORDER BY</span> reading_time)</span></span></code></pre> This will provide different data compared to unsorted ones. If you struggle to find the application for this, think of the moving average or row numbering.</p> The last component of the window definition is framing</strong>, allowing you to further limit the data over which the window function is calculated. There are two framing expressions: ROWS BETWEEN</code> and RANGE BETWEEN</code>. Using the framing, we can turn AVG from the above example to provide our first real use case when you might want to use window functions.</p> SELECT</span></span> reading_time,</span></span> sensor_id,</span></span> reading_value,</span></span> AVG</span>(reading_value) </span>OVER</span> (</span></span> PARTITION BY</span> sensor_id</span></span> ORDER BY</span> reading_time</span></span> ROWS BETWEEN</span> 1</span> PRECEDING AND</span> 1</span> FOLLOWING</span></span> ) </span>AS</span> moving_avg_reading_value</span></span> FROM</span> sensor_readings;</span></span></code></pre> Implementing the moving average of the individual readings, taking into account a maximum of 3 rows including the current one.</p> reading_time \| sensor_id \| reading_value \| moving_avg_reading_value</span></span> ------------------------+-----------+---------------+--------------------------</span></span> 2024-07-01 11:24:34+02 \| 1 \| 32.7 \| 32.9000000000000000</span></span> 2024-07-02 11:29:01+02 \| 1 \| 33.1 \| 32.9666666666666667</span></span> 2024-07-02 12:03:59+02 \| 1 \| 33.1 \| 33.0666666666666667</span></span> 2024-07-03 10:12:15+02 \| 1 \| 33.0 \| 33.0500000000000000</span></span> 2024-07-01 13:17:01+02 \| 2 \| 32.8 \| 34.3000000000000000</span></span> 2024-07-02 09:18:11+02 \| 2 \| 35.8 \| 34.3000000000000000</span></span> 2024-07-01 13:54:03+02 \| 3 \| 29.1 \| 29.7000000000000000</span></span> 2024-07-01 14:12:09+02 \| 3 \| 30.3 \| 30.3000000000000000</span></span> 2024-07-02 16:07:43+02 \| 3 \| 31.5 \| 30.9000000000000000</span></span></code></pre>Exploring Window functions</a> </h2> Now that we have a foundational understanding of window functions and their components, let’s dive into specific window functions. We’ll cover some of the most commonly used functions, such as ROW_NUMBER()</code>, RANK()</code>, DENSE_RANK()</code>, LAG()</code>, LEAD()</code>, FIRST_VALUE()</code>, LAST_VALUE()</code> and more. For each function, we’ll provide examples to illustrate their practical applications.</p> ROW_NUMBER</a> </h3> The ROW_NUMBER()</code> function assigns a unique sequential integer to rows within a partition of a result set, starting with one for the first row in each partition.</p> SELECT</span></span> sensor_id,</span></span> reading_time,</span></span> reading_value,</span></span> ROW_NUMBER</span>() </span>OVER</span> (</span>PARTITION BY</span> sensor_id </span>ORDER BY</span> reading_time) </span>AS</span> row_num</span></span> FROM</span> sensor_readings;</span></span></code></pre> This makes it easy to find the first/last entries for a specified window. As an example, you can experiment with getting only the last reading for each hour.</p> SELECT</span></span> sensor_id,</span></span> reading_time,</span></span> reading_value</span></span> FROM</span> (</span></span> SELECT</span></span> sensor_id,</span></span> reading_time,</span></span> reading_value,</span></span> ROW_NUMBER</span>() </span>OVER</span> (</span></span> PARTITION BY</span> sensor_id, date_trunc(</span>'hour'</span>, reading_time)</span></span> ORDER BY</span> reading_time </span>DESC</span></span> ) </span>AS</span> row_num</span></span> FROM</span> sensor_readings</span></span> ) </span>AS</span> ranked_readings</span></span> WHERE</span> row_num </span>=</span> 1</span>;</span></span></code></pre>RANK and DENSE_RANK</a> </h3> The RANK()</code> and DENSE_RANK()</code> functions are used to assign a rank to each row within a partition, based on the order of one or more values. These functions are useful when scoring the values and handling ties. The main difference between RANK</code> and DENSE_RANK</code> is how they deal with gaps.</p> Example use to find the highest reading_value per sensor:</p> SELECT</span></span> sensor_id,</span></span> reading_time,</span></span> reading_value,</span></span> RANK</span>() </span>OVER</span> (</span>PARTITION BY</span> sensor_id </span>ORDER BY</span> reading_value </span>DESC</span>) </span>AS</span> rank</span></span> FROM</span> sensor_readings;</span></span></code></pre> If you consider the sorting for the sensor_id</code> 1 in our sample seed the difference between RANK</code> and DENSE_RANK</code> is easy to demonstrate.</p> sensor_id \| reading_time \| reading_value \| rank</span></span> -----------+------------------------+---------------+------</span></span> 1 \| 2024-07-02 11:29:01+02 \| 33.1 \| 1</span></span> 1 \| 2024-07-02 12:03:59+02 \| 33.1 \| 1</span></span> 1 \| 2024-07-03 10:12:15+02 \| 33.0 \| 3</span></span> 1 \| 2024-07-01 11:24:34+02 \| 32.7 \| 4</span></span></code></pre> Giving a natural ranking with tie on the reading value 33.1 and leaving a gap on 2nd rank, whereas DENSE_RANK</code> wouldn't include the gap.</p> sensor_id \| reading_time \| reading_value \| rank</span></span> -----------+------------------------+---------------+------</span></span> 1 \| 2024-07-02 11:29:01+02 \| 33.1 \| 1</span></span> 1 \| 2024-07-02 12:03:59+02 \| 33.1 \| 1</span></span> 1 \| 2024-07-03 10:12:15+02 \| 33.0 \| 2</span></span> 1 \| 2024-07-01 11:24:34+02 \| 32.7 \| 3</span></span></code></pre>LAG</code> and LEAD</code></a> </h3> To evaluate previous or subsequent rows without the need to self-join the dataset, you can utilise the power of the window functions LAG</code> and LEAD</code>. These functions are particularly useful for calculating differences between rows, comparing current and previous values, or fetching future values for comparison.</p> The LAG</code> function provides access to a value in a previous row within the partition, while LEAD</code> provides access to a subsequent row.</p> SELECT</span></span> sensor_id,</span></span> reading_time,</span></span> reading_value,</span></span> LAG</span>(reading_value, </span>1</span>) </span>OVER</span> (</span>PARTITION BY</span> sensor_id </span>ORDER BY</span> reading_time) </span>AS</span> previous_value,</span></span> LEAD</span>(reading_value, </span>1</span>) </span>OVER</span> (</span>PARTITION BY</span> sensor_id </span>ORDER BY</span> reading_time) </span>AS</span> next_value</span></span> FROM</span> sensor_readings;</span></span></code></pre> In this query, we are using a value of 1, but you can choose any position necessary. By using LAG()</code> and LEAD()</code>, you can perform advanced analyses that require looking backward or forward within your dataset, making it easier to derive meaningful insights and trends.</p> FIRST_value</code> and LAST_VALUES</code></a> </h3> The functions FIRST_value</code> and LAST_VALUES</code> are similar, except they (as the name says) give the first/last value of the partition.</p> The FIRST_VALUE</code> is useful when comparing the partition values to the first value (for example opening price for a day), and LAST_VALUE</code> to identity the closing values.</p> Other Window functions</a> </h3> The complete list of the window functions is available in the documentation</a>.</p> Re-using the window definition</a> </h3> As you might have noticed, the query we used to demonstrate LAG</code> and LEAD</code> was rather verbose. This was due to the repeated definition of the window for both columns. Luckily, the syntax of the window functions allows you to define and re-use the window definition.</p> SELECT</span></span> sensor_id,</span></span> reading_time,</span></span> reading_value,</span></span> LAG</span>(reading_value, </span>1</span>) </span>OVER</span> readings_window </span>AS</span> previous_value,</span></span> LEAD</span>(reading_value, </span>1</span>) </span>OVER</span> readings_window </span>AS</span> next_value</span></span> FROM</span> sensor_readings</span></span> WINDOW</span> readings_window </span>AS</span> (</span>PARTITION BY</span> sensor_id </span>ORDER BY</span> reading_time);</span></span></code></pre> This way you can ensure the consistent window definition and consistency in complex queries.</p> The time keepers: pg_cron and pg_timetable 2024-06-15T00:00:00+00:00 Working with PostgreSQL, and virtually any database system, extends far beyond merely inserting and retrieving data. Many application and business processes, maintenance tasks, reporting, and orchestration tasks require the integration of a job scheduler. While third-party tools can drive automation, you can also automate the execution of predefined tasks directly within the database environment. Although system-level cron might be a starting point, the power of the database system lies in its ability to store all the necessary information alongside your data/schema. In this article, we will explore pg_cron</code> and pg_timetable</code> as two distinct PostgreSQL-specific tools for scheduled task automation.</p> The Many Roles of Job Scheduling</a> </h2> Usually, the first requirement to automate job execution is the optimisation of the PostgreSQL cluster and databases. Except in very low usage scenarios, routine maintenance</strong> is the foundation of all database deployments. Whether it is VACUUMing, index rebuilding, or updating statistics, these are tasks that you will eventually need to automate to maintain operational efficiency.</p> Job scheduling is also indispensable for maintaining data quality</strong>, particularly through operations like refreshing materialised views or batch removal of outdated data from the system. From there, it is just a step to reporting</strong>, which can involve the automation of generating operational and business reports.</p> I also believe that databases are an excellent place to coordinate business processes</strong>. Automating data flows between components, teams, and departments helps create agile systems. As mentioned above, there is no better place to store the description of such automations than alongside your data.</p> pg_cron: Automation's First Gear</a> </h2> Traditionally, the role of automation started with operating system tools like cron or Task Scheduler. That's where pg_cron</code></a> comes in. It's a PostgreSQL extension that provides the simplicity and familiarity of cron's scheduling directly within the database environment.</p> Due to its dependency on a shared library (because of the use of the background worker), it requires a full cluster restart. Nevertheless, due to its popularity, it is available within most managed cloud environments.</p> # requires </span>configuration update in</span> postgresql</span>.</span>conf</span> (</span>or</span> conf</span>.</span>d</span>)</span></span> shared_preload_libraries </span>=</span> 'pg_cron'</span></span> </span> #</span> add</span> extension </span>as</span> superuser</span></span> CREATE</span> EXTENSION pg_cron;</span></span></code></pre> As long as you are familiar with cron-like syntax, you can get started immediately by using commands like:</p> SELECT</span> cron</span>.</span>schedule</span>(</span>'30 3 * * 6'</span>, $$</span>DELETE FROM</span> events </span>WHERE</span> event_time </span>< now</span>()</span> -</span> interval </span>'1 week'</span>$$);</span></span></code></pre> By default, pg_cron</code> exposes its functionality in the postgres</code> database (configurable), where it expects its metadata tables. Personally, I consider this a drawback, as it makes it less obvious to the casual DBA who might not be aware of the scheduling logic present.</p> In the same database, you can perform basic monitoring, for example, getting details of running and recently completed jobs:</p> SELECT * FROM</span> cron</span>.</span>job_run_details</span> ORDER BY</span> start_time </span>DESC LIMIT</span> 5</span>;</span></span></code></pre> There is no direct support to talk to other PostgreSQL clusters (you would have to facilitate this, for example, using Foreign Data Wrappers).</p> While cron-like syntax is beneficial for adoption, it is where pg_cron</code> falls short, as it does not support advanced cases like task chaining, dependencies, and triggers.</p> Full Speed with pg_timetable</a> </h2> If pg_cron</code> offers automation in first gear, pg_timetable</code> is where you can go full speed ahead. It not only provides cron-like syntax but elevates it to a whole new level with task chaining, parameter support, multiple execution clients, enhanced scheduling, and much more.</p> The first difference you might notice is in its distribution. Timetable is not a PostgreSQL extension but a standalone binary (or available as a Docker image) that you have to configure and run. This immediately raises the bar in terms of the infrastructure it requires. On the other hand, not being distributed as an extension makes it compatible with all managed services by default (if you can get the process up and running).</p> While the basic syntax might be similar to pg_cron</code>:</p> SELECT</span> timetable</span>.</span>add_job</span>(</span>'execute-func'</span>, </span>'5 0 * 8 '</span>, </span>'SELECT public.my_func()'</span>);</span></span></code></pre> this is not anywhere near the limits of pg_timetable</code>. add_job</code> is a helper function that creates a simple one-task chain. For task execution, it directly supports more options, including built-in tasks (like sending emails if properly configured, downloading files, sleeping, copying files, etc.), external commands, the ability to choose which client should execute the job, concurrency, and more.</p> In its full definition, add_job</code> is quite powerful (see documentation</a> for details):</p> SELECT</span> timetable</span>.</span>add_job</span>(</span></span> job_name </span>=></span> 'notify every minute'</span>,</span></span> job_schedule </span>=></span> ' * * * '</span>,</span></span> job_command </span>=></span> 'SELECT pg_notify($1, $2)'</span>,</span></span> job_parameters </span>=></span> '["TT_CHANNEL", "Hello World!"]'</span>::jsonb,</span></span> job_kind </span>=></span> 'SQL'</span>::</span>timetable</span>.</span>command_kind</span>,</span></span> job_client_name </span>=> NULL</span>,</span></span> job_max_instances </span>=></span> 1</span>,</span></span> job_live </span>=></span> TRUE,</span></span> job_self_destruct </span>=></span> FALSE,</span></span> job_ignore_errors </span>=></span> TRUE</span></span> ) </span>AS</span> chain_id;</span></span></code></pre> The components of pg_timetable</code> consist of the command</strong> (SQL/program or built-in), task</strong> controlling the configuration for the command execution (error handling, timeout, or database connection to use), and chain</strong> which can contain a number of tasks chained together.</p> The big difference comes in the scheduling options. While pg_cron</code> is limited (as its name suggests) to cron-like syntax, pg_timetable</code> uses it for simple use cases but offers much more. It supports schedules @every</code> and @after</code>, allowing repeated execution and breaking away from the limitations of cron notation as it can go down to custom intervals (including second intervals). Another case is @reboot</code> for instances when the pg_timetable</code> controller reconnects to the database.</p> The timetable setup manages own schema migrations and stores all the configuration in schema timetable</code> (by default), where you can find the definitions of chains/tasks and relevant auditing information (logs).</p> And yes, if you have been paying attention, pg_timetable</code> architecture allows for running tasks across multiple PostgreSQL clusters, making it an advanced orchestration tool. The database connection, which can be configured in-place or via a drop-in connection service file</a>, can be set on a per-task basis.</p> With chains, the setup can be much more complex:</p> DO $$</span></span> DECLARE</span></span> v_chain_id </span>BIGINT</span>;</span></span> v_notify_task_id </span>BIGINT</span>;</span></span> BEGIN</span></span> INSERT INTO</span> timetable</span>.</span>chain</span> (</span></span> chain_name,</span></span> run_at,</span></span> max_instances,</span></span> live)</span></span> VALUES</span> (</span></span> 'Generate Weekly Report'</span>,</span></span> '5 4 * 1'</span>,</span></span> 1</span>,</span></span> TRUE</span></span> )</span></span> RETURNING chain_id </span>INTO</span> v_chain_id;</span></span> </span> INSERT INTO</span> timetable</span>.</span>task</span> (</span></span> chain_id,</span></span> task_order,</span></span> task_name,</span></span> command,</span></span> database_connection</span></span> )</span></span> VALUES</span> (</span></span> v_chain_id,</span></span> 1</span>, </span>-- task_order</span></span> 'generate_report'</span>, </span>-- task_name</span></span> 'SELECT generate_weekly_report();'</span>, </span>-- command</span></span> NULL</span> -- database_connection</span></span> );</span></span> </span> INSERT INTO</span> timetable</span>.</span>task</span> (</span></span> chain_id,</span></span> task_order,</span></span> task_name,</span></span> command,</span></span> database_connection</span></span> )</span></span> VALUES</span> (</span></span> v_chain_id,</span></span> 2</span>, </span>-- task_order (second task in the chain)</span></span> 'notify_management'</span>, </span>-- task_name</span></span> 'SELECT pg_notify($1, $2)'</span>, </span>-- command</span></span> 'service=service_db'</span> -- database_connection</span></span> )</span></span> RETURNING task_id </span>INTO</span> v_notify_task_id;</span></span> </span> INSERT INTO</span> timetable</span>.</span>task_parameter</span> (task_id, order_id, </span>value</span>)</span></span> VALUES</span></span> (v_notify_task_id, </span>1</span>, </span>'management_notifications'</span>),</span></span> (v_notify_task_id, </span>2</span>, </span>'Weekly report'</span>);</span></span> END</span>;</span></span> $$ </span>LANGUAGE</span> plpgsql;</span></span></code></pre>The Comparison</a> </h2> When deciding between pg_cron</code> and pg_timetable</code>, it's essential to consider the specific needs of your use case, as both tools target different scenarios and scales of deployment. Here’s a comparison to help you decide which tool is more suitable for your requirements:</p> Feature</th> pg_cron</th> pg_timetable</th></tr></thead> Distribution</td> PostgreSQL extension</td> Standalone executable</td></tr> Scheduling</td> Limited (cron-like)</td> Extensive (intervals, calendar, custom)</td></tr> Job Types</td> SQL</td> SQL, Built-in, Shell</td></tr> Job Chaining</td> No</td> Yes</td></tr> Error Handling</td> Basic</td> Intermediate</td></tr> Logging</td> Basic</td> Intermediate</td></tr> Dependencies</td> No</td> Yes</td></tr> Ease of Use</td> Easy</td> Moderate</td></tr> Configuration</td> Easy</td> Moderate</td></tr> Orchestration</td> Single Cluster</td> Advanced (multiple clusters)</td></tr> </tbody></table> When to Use pg_cron</code></a> </h3> If your requirements are straightforward, such as running maintenance tasks, refreshing materialised views, or generating periodic reports using simple SQL commands, pg_cron</code> is an excellent choice. Its cron-like syntax is familiar and easy to use</strong>.</li> pg_cron</code> is a PostgreSQL extension, making it easier to install</strong> and configure within your existing PostgreSQL environment. This makes it ideal for users who prefer minimal setup effort.</li> When your environment is limited to a single PostgreSQL cluster</strong>, pg_cron</code> is well-suited for the job. It does not support orchestration across multiple clusters, so it’s best used in environments where all operations are confined to one database cluster.</li> If Basic Error Handling and Logging</strong> is sufficient, pg_cron</code> provides the necessary functionalities without additional complexity.</li> </ol> When to Use pg_timetable</code></a> </h3> If you need more advanced scheduling capabilities</strong> and task depedencies and chaining</strong>, such as task chaining, complex intervals, and custom execution times, pg_timetable</code> is the better choice. It supports extensive scheduling options beyond the typical cron syntax.</li> pg_timetable</code> supports a variety of job types</strong>, including SQL commands, built-in tasks (like sending emails or downloading files), and shell commands. This makes it ideal for more complex automation needs that go beyond simple SQL execution.</li> If your environment spans multiple PostgreSQL clusters, pg_timetable</code> can handle advanced orchestration</strong>, allowing tasks to be executed across different clusters seamlessly.</li> Intermediate Error Handling and Logging</strong> provides better insights and control over job executions.</li> Although bit more complex to setup, pg_timetable</code> works as standalone Service</strong>, which makes it suitable for environments where extensions cannot be installed directly.</li> </ol> Choosing between pg_cron</code> and pg_timetable</code> depends on your specific needs. For simpler, single-cluster tasks, pg_cron</code> offers ease of use and straightforward setup. For more complex requirements involving advanced scheduling, task dependencies, and multi-cluster orchestration, pg_timetable</code> provides a more powerful and flexible solution.</p> By understanding the strengths and limitations of each tool, you can make an informed decision that best suits your database automation needs.</p> PS: there's also pgAgent</a>, but it is generally not used and does not seem to be actively maintained.</p> Deep Dive into PostgREST - Time Off Manager (Part 3) 2024-06-06T00:00:00+00:00 This is the third and final instalment of "Deep Dive into PostgREST". In the first part</a>, we explored basic CRUD functionalities. In the second part</a>, we moved forward with abstraction and used the acquired knowledge to create a simple request/approval workflow.</p> In Part 3, we will explore authentication and authorisation options to finish something that might resemble a real-world application.</p> Authentication with pgcrypto</a> </h2> There's no authorisation without knowing user identity. So let's start there. Our users table from the first part had an email</code> to establish the identity, but no way to verify it. We will address this by adding a password column. Of course, nobody in their right mind would store passwords in plain text.</p> To securely store users' passwords, we are going to utilise PostgreSQL pgcrypto</code> extension (documentation</a>). This built-in extension provides a suite of cryptographic functions for hashing, encryption, and more. In our case, we will leverage crypt</code> with the gen_salt</code> function to generate password hash using bcrypt algorithm.</p> Let's start with loading the extension and adding password column:</p> CREATE</span> EXTENSION </span>IF NOT EXISTS</span> pgcrypto;</span></span> ALTER TABLE</span> users </span>ADD</span> COLUMN password_hash </span>TEXT NOT NULL DEFAULT</span> gen_salt(</span>'bf'</span>); </span></span></code></pre> The new column password_hash</code> will accommodate the variable-length bcrypt hash, while default function gen_salt('bf')</code> creates a unique bcrypt salt for every user.</p> Now that our table structure is set, let's see how we can securely set and verify passwords using pgcrypto.</p> When a user sets or changes their password, we'll hash it using crypt before storing it in the password_hash column.Here's how:</p> UPDATE users SET password_hash = crypt('new_password', gen_salt('bf')) WHERE email = 'user@example.com';</span></span></code></pre> and during login, we will verify the hash of the password the user enters with the stored password_hash:</p> SELECT</span> user_id </span>FROM</span> users </span>WHERE</span> email </span>=</span> 'user@example.com'</span> AND</span> password_hash </span>=</span> crypt(</span>'entered_password'</span>, password_hash);</span></span></code></pre>JWT Authentication</a> </h2> With the ability to securely verify users' identity, let's move to the next step and build stateless authentication. The de-facto standard is JSON Web Tokens (JWTs). Represented by digitally signed information, they contain claims about user identity. The digital signature ensures the token's integrity and verifies that it hasn't been tampered with. You can find more in official Introduction to JSON Web Tokens</a>.</p> While PostgreSQL doesn't have built-in JWT support, we will have to either rely on pgjwt</code> extension (GitHub repo</a>)</p> CREATE EXTENSION IF NOT EXISTS pgjwt;</span></span></code></pre> or you can re-create the logic by including PL/pgSQL that comes with it</a> (please, make sure you replace/remove the @extschema@</code> to match the schema you are using). In this article we will use schema jwt</code>.</p> To make the token signature, we need to re-configure PostgREST to decode JWT tokens and configure the secret inside the database (to be able to use it in our code). For security reasons, the key must be at least 32 characters long. You can either use your own method to generate (for example, your password manager) or add it using the following CLI commands</p> export</span> LC_CTYPE</span>=</span>C</span></span> echo</span> "jwt-secret = </span>\"</span>$(</span>LC_ALL</span>=</span>C</span> tr</span> -dc</span> 'A-Za-z0-9'</span> <</span>/dev/urandom</span> \|</span> head</span> -c32</span>)</span>\"</span>"</span> >></span> postgrest.conf</span></span></code></pre> And use the generated secret for the database level configuration parameter, using</p> ALTER DATABASE</span> time_off_manager </span>SET</span> "pgrst.jwt_secret"</span> to</span> "your-jwt-secret-generated-above1"</span>;</span></span></code></pre> Please, make sure the database name is updated accordingly to set it correctly. And, I cannot stress it enough, make sure the secret is really 32 characters</strong>.</p> The web user role</a> </h2> In previous parts of this guide, we have worked with two user roles. The first, in PostgREST terminology, called authenticator, is the role used in the db-uri</code> parameter. It's the role used to access the database and its job is to impersonate other users based on the authentication (or lack thereof) of the HTTP requests. In a real production application, this role should be configured to have limited access.</p> The second role, called anonymous, is used in db-anon-role</code> parameter. This is the role impersonated for all unauthenticated HTTP requests.</p> The third role, or roles, representing authenticated web users. In our JWT tokens, we will use one specifically designed for PostgREST.</p> {</span></span> "role": "time_off_user"</span></span> }</span></span></code></pre> When JWT is successfully validated, with a role claim, PostgREST will switch to the database role with the provided name for the duration of the HTTP request. While PostgREST is quite flexible, we will limit the impersonated role for authenticated requests to a single hard-coded role. For our application, it's going to be called time_off_user</code>.</p> Let's generate some JWTs</a> </h2> The first step is to implement logic to create a JWT token asserting all claims</strong> we need for our application - in the case of Time Off Manager, we will rely on user_id and role. As discussed in the previous section, we will use the hard-coded role time_off_user</code>. Let's create the role and grant our authenticator the permissions.</p> CREATE ROLE</span> time_off_user NOLOGIN;</span></span> GRANT</span> time_off_user </span>TO</span> time_off_manager;</span></span></code></pre> Now create a function, users can call directly to verify the identity and generate the JWT token.</p> CREATE OR REPLACE</span> FUNCTION api.login(email text, password text)</span></span> RETURNS text</span></span> LANGUAGE</span> plpgsql</span></span> SECURITY DEFINER</span></span> AS</span> $function$</span></span> DECLARE</span></span> user_record public.users;</span></span> BEGIN</span></span> SELECT * INTO</span> user_record</span></span> FROM</span> public.users</span></span> WHERE</span> users.email </span>=</span> login.email;</span></span> </span> IF</span> user_record.password_hash </span>=</span> crypt(password, user_record.password_hash) </span>THEN</span></span> RETURN</span> create_jwt(user_record.user_id, </span>'user_id'</span>);</span></span> ELSE</span></span> RAISE</span> EXCEPTION</span> 'Invalid email or password'</span>;</span></span> END IF</span>;</span></span> END</span>;</span></span> $function$;</span></span></code></pre> Missing there is helper create_jwt</code></p> CREATE OR REPLACE</span> FUNCTION public.create_jwt(user_id </span>integer</span>, role text)</span></span> RETURNS text</span></span> LANGUAGE</span> plpgsql</span></span> AS</span> $function$</span></span> DECLARE</span></span> payload JSON;</span></span> BEGIN</span></span> payload </span>:=</span> json_build_object(</span></span> 'user_id'</span>, user_id,</span></span> 'role'</span>, </span>'time_off_user'</span>,</span></span> 'exp'</span>, </span>extract</span>(epoch </span>from</span> now()) </span>+</span> 3600</span> -- 1-hour expiration</span></span> );</span></span> RETURN</span> jwt.</span>sign</span>(payload, current_setting(</span>'pgrst.jwt_secret'</span>)); </span>-- Use configuration value</span></span> END</span>;</span></span> $function$;</span></span></code></pre> Restart the PostgREST instance to apply the changes, and you can try to generate a token using cURL (assuming you have changed the password as demonstrated in the first section ):</p> curl</span> -X</span> POST "http://localhost:3000/rpc/login"</span> \</span></span> -d</span> '{"email":"manager2@example.com", "password": "new_password"}'</span> \</span></span> -H</span> "Content-Type: application/json"</span></span></code></pre> If you set everything as expected you will get a base64 encoded token. To verify/debug it, you can try to use JWT.io</a>.</p> Prepare the permissions</a> </h2> The next step is where things get really interesting. First, we need to clean up excessive permissions. Some obvious, some less so.</p> First let's start with the revoking all the permissions for time_off_anonymous</code> we provided in previous part.</p> REVOKE</span> ALL PRIVILEGES </span>ON</span> ALL TABLES </span>IN SCHEMA</span> api </span>FROM</span> time_off_anonymous;</span></span> REVOKE EXECUTE ON</span> ALL FUNCTIONS </span>IN SCHEMA</span> api </span>FROM</span> time_off_anonymous;</span></span></code></pre> and same we need to adjust the default privileges</p> ALTER DEFAULT</span> PRIVILEGES </span>IN SCHEMA</span> api </span>REVOKE SELECT ON</span> TABLES </span>FROM</span> time_off_anonymous;</span></span> ALTER DEFAULT</span> PRIVILEGES </span>IN SCHEMA</span> api </span>REVOKE EXECUTE ON</span> FUNCTIONS </span>FROM</span> time_off_anonymous;</span></span> </span></code></pre> Which is the obvious part. The most likely surprising part is the need to remove EXECUTE</code> permissions for PUBLIC</code>. In PostgreSQL, granting usage on a schema also grants the ability to execute functions to PUBLIC</code>. Unless you revoke these privileges, all users will be able to execute the functions.</p> For our purposes we want to REVOKE the access for PUBLIC</code> and only grant EXECUTE</code> on function api.login()</code></p> GRANT</span> USAGE </span>ON SCHEMA</span> api </span>TO</span> time_off_user;</span></span> </span> -- REVOKE EXECUTE</span></span> ALTER DEFAULT</span> PRIVILEGES </span>IN SCHEMA</span> api </span>REVOKE EXECUTE ON</span> FUNCTIONS </span>FROM</span> PUBLIC;</span></span> REVOKE EXECUTE ON</span> ALL FUNCTIONS </span>IN SCHEMA</span> api </span>FROM</span> PUBLIC;</span></span> </span> -- GRANT EXECUTE </span></span> GRANT EXECUTE ON FUNCTION</span> api</span>.</span>login</span> to</span> time_off_anonymous;</span></span></code></pre> And finally, we need to establish the necessary permissions for time_off_user</code>, the role which will be used for authenticated requests.</p> GRANT SELECT ON</span> ALL TABLES </span>IN SCHEMA</span> public </span>TO</span> time_off_user;</span></span> GRANT SELECT ON</span> ALL TABLES </span>IN SCHEMA</span> api </span>TO</span> time_off_user;</span></span> GRANT EXECUTE ON</span> ALL FUNCTIONS </span>IN SCHEMA</span> api </span>TO</span> time_off_user;</span></span> </span> ALTER DEFAULT</span> PRIVILEGES </span>IN SCHEMA</span> public </span>GRANT SELECT ON</span> TABLES </span>TO</span> time_off_user;</span></span> ALTER DEFAULT</span> PRIVILEGES </span>IN SCHEMA</span> api </span>GRANT SELECT ON</span> TABLES </span>TO</span> time_off_user;</span></span> ALTER DEFAULT</span> PRIVILEGES </span>IN SCHEMA</span> api </span>GRANT EXECUTE ON</span> FUNCTIONS </span>TO</span> time_off_user;</span></span></code></pre> We will now use this to get an overview of only relevant vacation balances for the authenticated user.</p> Authorization with PostgREST</a> </h2> With authentication working, let's start with the implementation of fine-grained authorization in PostgREST. As a first step, the goal is to ensure that users only see the vacation balances they are supposed to. I.e. either their own (for the regular employees) or their own and the people the user manages (for the managers).</p> For this we will need to access JWT token claims, specifically user_id</code>. To avoid relatively complex notation, let's setup a helper</p> CREATE OR REPLACE FUNCTION</span> public</span>.current_user_id()</span></span> RETURNS integer</span></span> LANGUAGE</span> plpgsql</span></span> SECURITY</span> DEFINER</span></span> AS</span> $</span>function</span>$</span></span> DECLARE</span></span> user_id </span>INTEGER</span>;</span></span> BEGIN</span></span> user_id :</span>=</span> current_setting(</span>'request.jwt.claims'</span>, true)::</span>json->></span>'user_id'</span>;</span></span> </span> IF</span> user_id </span>IS NULL THEN</span></span> RAISE EXCEPTION </span>'User ID not found in JWT claims'</span>;</span></span> END IF</span>;</span></span> </span> RETURN</span> user_id::</span>integer</span>;</span></span> END</span>;</span></span> $</span>function</span>$;</span></span></code></pre> PostgREST seamlessly integrates with PostgreSQL's Row Level Security (RLS) to filter data based on user permissions. RLS policies are rules applied at the row level to determine if a user can access a particular row in a table.</p> ALTER TABLE</span> public</span>.</span>time_off_transactions</span> ENABLE ROW LEVEL SECURITY</span>;</span></span> </span> CREATE POLICY</span> select_own_balance </span>ON</span> public</span>.</span>time_off_transactions</span> </span></span> FOR SELECT USING</span> (</span>public</span>.</span>current_user_id</span>()</span> =</span> user_id);</span></span> </span> CREATE POLICY</span> select_supervised_balance </span>ON</span> public</span>.</span>time_off_transactions</span></span> FOR SELECT USING</span> (</span>EXISTS</span> (</span>SELECT</span> 1</span> FROM</span> api</span>.</span>users</span> WHERE</span> users</span>.</span>user_id</span> =</span> time_off_transactions</span>.</span>user_id</span> AND</span> users</span>.</span>manager_user_id</span> =</span> public</span>.</span>current_user_id</span>()));</span></span></code></pre> If you are (and I do hope so) using PostgreSQL 15 and higher, you need to switch the view from the default security definer to invoker.</p> ALTER VIEW</span> api</span>.</span>vacation_balances</span> SET</span> (security_invoker </span>=</span> true);</span></span></code></pre> This way the view, and the underlying tables will be evaluated using the permissions of the user querying the view, not the view owner. This is the behaviour we want. For versions beyond PostgreSQL 15 and more complex use cases, you might need to implement function-based security instead.</p> But without further delay, let's test "the magic" and (assuming you have authenticated as a manager) you might try to retrieve the vacation balances using cURL</p> curl</span> "http://localhost:3000/vacation_balances"</span> \</span></span> -H</span> "Authorization: Bearer ${</span>JWT_TOKEN</span>}"</span></span></code></pre> to get a result similar to</p> [{</span>"year"</span>:</span>2024</span>,</span>"user_id"</span>:</span>8</span>,</span>"total_amount"</span>:</span>25</span>},</span></span> {</span>"year"</span>:</span>2024</span>,</span>"user_id"</span>:</span>9</span>,</span>"total_amount"</span>:</span>25</span>},</span></span> {</span>"year"</span>:</span>2024</span>,</span>"user_id"</span>:</span>10</span>,</span>"total_amount"</span>:</span>25</span>},</span></span> {</span>"year"</span>:</span>2024</span>,</span>"user_id"</span>:</span>11</span>,</span>"total_amount"</span>:</span>25</span>},</span></span> {</span>"year"</span>:</span>2024</span>,</span>"user_id"</span>:</span>12</span>,</span>"total_amount"</span>:</span>25</span>},</span></span> {</span>"year"</span>:</span>2024</span>,</span>"user_id"</span>:</span>13</span>,</span>"total_amount"</span>:</span>25</span>}]</span></span></code></pre> Guess, it's important to mention the Row Security Policies which are used are much more complex, and the whole topic would deserve another article. For now, I do recommend you to consult the documentation</a></del>, to get more understanding. With the multi-role access you can implement with PostgREST, it might be interesting for you to focus on BYPASSRLS</code> which might be applied to certain role(s).</p> As mentioned before, Row Level Security is just one way to solve this. The traditional approach would be to use functions to hide the separation logic.</p> Wrapping up the Time Off Manager</a> </h2> Before we wrap up our tutorial, let's finish the core functionality of the Time Off Manager - the approval workflow. Given the basic concepts covered above, this is going to be a good exercise to use all of it.</p> Similar how we updated time_off_transactions</code>, we will apply Row Level Security to time_off_requests</code>.</p> ALTER TABLE</span> public</span>.</span>time_off_requests</span> ENABLE ROW LEVEL SECURITY</span>; </span></span> </span> CREATE POLICY</span> select_own_requests </span>ON</span> public</span>.</span>time_off_requests</span> </span></span> FOR SELECT USING</span> (</span>public</span>.</span>current_user_id</span>()</span> =</span> user_id); </span></span> </span> CREATE POLICY</span> select_subordinate_requests </span>ON</span> public</span>.</span>time_off_requests</span></span> FOR SELECT USING</span> (</span>EXISTS</span> (</span>SELECT</span> 1</span> FROM</span> public</span>.</span>users</span> WHERE</span> users</span>.</span>user_id</span> =</span> time_off_requests</span>.</span>user_id</span> AND</span> users</span>.</span>manager_user_id</span> =</span> public</span>.</span>current_user_id</span>()));</span></span></code></pre> Don't forget to switch the view to security_invoker</code> model.</p> ALTER VIEW</span> api</span>.</span>pending_requests</span> SET</span> (security_invoker </span>=</span> true);</span></span></code></pre> And we wouldn't be done with requests creation, if we wouldn't update function api.request_time_off</code> to take the advantage of the newly propagated user_id</code> from authentication.</p> CREATE OR REPLACE FUNCTION</span> api</span>.request_time_off(leave_type </span>text</span>, </span>period</span> daterange)</span></span> RETURNS integer</span></span> LANGUAGE</span> plpgsql</span></span> SECURITY</span> DEFINER</span></span> AS</span> $</span>function</span>$</span></span> DECLARE</span></span> v_leave_type_id </span>INT</span>;</span></span> v_request_id </span>INT</span>;</span></span> BEGIN</span></span> -- original validation logic </span></span> </span> -- Ensure the user is requesting for themselves</span></span> IF</span> public</span>.</span>current_user_id</span>()</span> !=</span> request_time_off</span>.</span>user_id</span> THEN</span></span> RAISE EXCEPTION </span>'User can only request time off for themselves'</span>;</span></span> END IF</span>;</span></span> </span> -- the rest of the function</span></span> END</span>;</span></span> $</span>function</span>$;</span></span></code></pre> The similar update then applies to the api.update_request</code> function to ensure only managers can approve/reject time off requests.</p> CREATE OR REPLACE FUNCTION</span> api</span>.update_request(request_id </span>integer</span>, new_status </span>text</span>)</span></span> RETURNS</span> void</span></span> LANGUAGE</span> plpgsql</span></span> SECURITY</span> DEFINER</span></span> AS</span> $</span>function</span>$</span></span> DECLARE</span></span> v_requested_user_id </span>INT</span>;</span></span> v_request_manager_id </span>INT</span>;</span></span> BEGIN</span></span> -- the original code up to the retrival of the requests</span></span> </span></span> IF</span> public</span>.</span>current_user_id</span>()</span> !=</span> v_request_manager_id </span>THEN</span></span> RAISE EXCEPTION </span>'Only the manager can approve or reject this request'</span>;</span></span> END IF</span>;</span></span> </span> -- update the request status</span></span> END</span>;</span></span> $</span>function</span>$;</span></span></code></pre> And that's about it! Obviously, we can't pretend Time Off Manager is anywhere complete, and the real-life application would require much more than what we have covered. But it should provide a good training platform to allow you to write a real API-based backends using just PostgREST.</p> Correction 2024-06-11</em>: during the final edits the function public.current_user_id</code> somehow got missing from the Markdown [FIXED].</p> Custom PostgreSQL extensions with Rust 2024-05-24T00:00:00+00:00 This article explores the pgrx framework, which simplifies the creation of custom PostgreSQL extensions to bring more logic closer to your database. Traditionally, writing such extensions required familiarity with C and a deep understanding of PostgreSQL internals, which could be quite challenging. pgrx</code> lowers the barrier and allows developers to use Rust, known for its safety and performance, making the process of creating efficient and safe database extensions much more accessible.</p> pg_sysload</a> </h2> When working with large datasets and migrations (as discussed in How Not to Change PostgreSQL Column Type</a>), or during resource-intensive maintenance tasks, you'll want to optimise speed and minimise disruption to other processes. One way to control the pace of batch operations is to consider the load on the underlying system.</p> Many Unix-based systems (we will focus on Linux) provide a valuable metric called the system load average</strong>. This average consists of three values: the 1-minute, 5-minute, and 15-minute load averages. The load average is not normalised for the number of CPU cores, so a load average of 1 on a single-core system means full utilisation, while on a quad-core system, it indicates 25% utilisation.</p> In many cases, the system load average is also an excellent indicator of how ongoing operations are impacting a busy database cluster. In this article, we will create a PostgreSQL extension with a function called sys_loadavg()</code> that retrieves this load information. We will use the /proc/loadavg</code> file (part of the proc filesystem</strong>), which exposes underlying system details.</p> Getting Started with pgrx</code></a> </h2> Before we start, ensure you have:</p> Rust installed (Get Started with Rust</a>)</li> System dependencies</a> installed for pgrx</code></li> </ul> With these prerequisites in place, you can install pgrx</code> itself and create a new extension skeleton:</p> cargo</span> install</span> --locked</span> cargo-pgrx</span></span> cargo</span> pgrx new pg_sysload</span></span> cd</span> pg_sysload</span></span></code></pre> This gives you a complete environment for developing your own PostgreSQL extensions in Rust. While Rust might seem daunting at first (especially with its async and other features), the language itself is quite powerful. Its ease of extension creation makes it worth a second look.</p> Setting Up the Extension</a> </h2> While the pgrx new</code> command sets up a basic template, we will create a more sophisticated extension. Its primary logic involves parsing the /proc/loadavg</code> file, extracting its values, and returning them to the user.</p> The logic itself is straightforward:</p> #[pg_extern]</span></span> fn</span> sys_loadavg</span>()</span> -></span> Option</span><</span>Vec</span><</span>f64</span>>> {</span></span> // Read the contents of the /proc/loadavg</span></span> let mut</span> file</span> = match</span> fs</span>::</span>File</span>::</span>open</span>(</span>"/proc/loadavg"</span>) {</span></span> Ok</span>(file)</span> =></span> file,</span></span> Err</span>(err)</span> =></span> {</span></span> pgrx</span>::</span>error!</span>(</span>"Error reading /proc/loadavg: {}"</span>, err);</span></span> }</span></span> };</span></span> let mut</span> contents</span> =</span> String</span>::</span>new</span>();</span></span> if let</span> Err</span>(err)</span> =</span> file</span>.</span>read_to_string</span>(</span>&mut</span> contents) {</span></span> pgrx</span>::</span>error!</span>(</span>"Error reading /proc/loadavg: {}"</span>, err);</span></span> }</span></span> </span> // Extract the load average fields</span></span> let</span> fields</span> =</span> contents</span>.</span>split_whitespace</span>()</span>.</span>collect</span>::</span><</span>Vec</span><_>>();</span></span> if</span> fields</span>.</span>len</span>()</span> >=</span> 3</span> {</span></span> Some</span>(</span></span> fields[</span>..</span>3</span>]</span></span> .</span>iter</span>()</span></span> .</span>filter_map</span>(</span>\|</span>s</span>\|</span> f64</span>::</span>from_str</span>(s)</span>.</span>ok</span>())</span></span> .</span>collect</span>(),</span></span> )</span></span> }</span> else</span> {</span></span> pgrx</span>::</span>error!</span>(</span>"Invalid format in /proc/loadavg"</span>);</span></span> }</span></span> }</span></span></code></pre> To limit usage to systems supporting the proc filesystem</strong>, we check for the presence of the file when the extension is loaded:</p> #[pg_guard]</span></span> fn</span> _PG_init</span>() {</span></span> INIT</span>.</span>call_once</span>(</span>\|\|</span> {</span></span> let</span> loadavg_available</span> =</span> fs</span>::</span>metadata</span>(</span>"/proc/loadavg"</span>)</span>.</span>is_ok</span>();</span></span> if !</span>loadavg_available {</span></span> pgrx</span>::</span>error!</span>(</span>"/proc/loadavg not found. Extension cannot load."</span>);</span></span> }</span></span> });</span></span> }</span></span></code></pre> This basic check prevents users from loading the extension on incompatible systems. With minor details omitted, that's almost all there is to it.</p> You can find the complete lib.rs</code> file in the accompanying GitHub repository</a>.</p> Running the Extension</a> </h2> Where pgrx</code> truly shines is in all the heavy lifting it does for you. It leverages the cargo</code> command. To initialise the extension, run:</p> cargo</span> pgrx init</span></span></code></pre> This downloads and prepares PostgreSQL source code for versions 12 to 16 (at the time of writing). After a bit of waiting (hopefully successfully, if you have all the system dependencies), run the extension:</p> cargo</span> pgrx run</span></span></code></pre> You'll get a psql</code> prompt for a dedicated PostgreSQL instance. Create and use the extension:</p> CREATE</span> EXTENSION pg_sysload;</span></span></code></pre> Now you can try the newly exposed function:</p> SELECT</span> sys_loadavg();</span></span> sys_loadavg</span></span> ------------------</span></span> {</span>1</span>.</span>17</span>, </span>0</span>.</span>57</span>, </span>0</span>.</span>32</span>}</span></span> (</span>1</span> row</span>)</span></span></code></pre> And that's all. Your very first PostgreSQL extension is working.</p> Throttling Batch Processing</a> </h2> With the extension ready, let's revisit our original goal: throttling long-running batch data processing. The data from sys_loadavg</code> can be used in various ways:</p> Dynamic Sleep Times:</strong> Insert calculated sleep intervals based on the system load.</li> Dynamic Batch Sizes:</strong> Adjust the number of rows processed per batch based on available resources.</li> </ul> Here's an example:</p> -- Sleep for 5 seconds multiplied by the 1-minute load of the system</span></span> SELECT</span> pg_sleep((sys_loadavg())[1]</span> </span> 5</span>);</span></span> </span> -- Assume 20 cores and for each one available, add 100 rows to process</span></span> SELECT</span> ... </span>LIMIT</span> (</span>SELECT</span> (</span>20</span> -</span> (sys_loadavg())[1])::</span>int </span> 100</span>);</span></span></code></pre> This allows you to fine-tune processing for unsupervised operation, automatically adapting to the current system load.</p> Conclusion</a> </h2> And there you have it! You've just built your first PostgreSQL extension using Rust and the pgrx</code> framework. I have always found the prospect of writing an extension quite daunting (probably because I'm long gone from the C-ecosystem), but this wasn't so bad. You've now got a handy tool for monitoring system load, perfect for keeping tabs on how your database processes long-running migrations.</p> This is just scratching the surface. We haven't even touched on the really fun stuff – extending PostgreSQL's internals with custom data types, operators, or indexes. You could build extensions that transform or aggregate data, hook into external APIs, or create background workers, bringing various business logic directly into the database engine. And yes, tapping into the database's core can be a bit risky, and it is always important to assess risks and recovery options. But whatever you choose to create, remember, with pgrx</code> at your side, you've got the power of Rust to keep things safe and sound.</p> Deep Dive into PostgREST - Time Off Manager (Part 2) 2024-05-18T00:00:00+00:00 Let's recap the first part</a> of "Deep Dive into PostgREST," where we explored the basic functionality to expose and query any table using an API, demonstrated using cURL</code>. All it took was to set up a db-schema</code> and give the db-anon-role</code> some permissions. But unless you are creating the simplest of CRUD applications, this only scratches the surface.</p> In Part 2, we will expand APIs, provide better abstraction, and implement the foundation of what can be considered business logic, all while extending the sample "Time Off Manager" application. While the previous instalment being introductiory only, make sure you don't miss the important details in this one.</p> Before we move on, let's do a bit of housekeeping and clean up the permissions setup from the first part. This way, we can start with a clean slate (when it comes to the permissions) and avoid any lingering rights that could confuse us later.</p> REVOKE</span> ALL PRIVILEGES </span>ON</span> ALL TABLES </span>IN SCHEMA</span> public </span>FROM</span> time_off_anonymous;</span></span></code></pre>Dedicated schema for the API</a> </h2> Using the public</code> schema (or any schema(s) where your core data model resides) is a fast way to get started. However, using a dedicated schema for the API is beneficial for several reasons:</p> It provides options for better abstraction, which you might appreciate later when refactoring the original data model.</li> Data customisation is also a requirement unless you prefer building a "fat" client and thus transferring the majority of the business logic there. Combining data from multiple tables helps shield the consumer from complex queries and relations.</li> While we won't explore it as a security feature, it can also provide relevant boundaries.</li> </ul> Let's get started with the creation of the schema itself, and expose the users using a view and setting basic permissions. We will do this by setting default permissions, so we don't have to repeat the same for all objects as we create them. Please note that default privileges are applied only to new objects created.</p> CREATE SCHEMA</span> api</span>;</span></span> </span> GRANT</span> USAGE </span>ON SCHEMA</span> api </span>TO</span> time_off_anonymous;</span></span> ALTER DEFAULT</span> PRIVILEGES </span>IN SCHEMA</span> api </span>GRANT SELECT ON</span> TABLES </span>TO</span> time_off_anonymous;</span></span> ALTER DEFAULT</span> PRIVILEGES </span>IN SCHEMA</span> api </span>GRANT EXECUTE ON</span> FUNCTIONS </span>TO</span> time_off_anonymous;</span></span></code></pre> Ok, let's create our first view:</p> CREATE VIEW</span> api</span>.users </span>AS</span></span> SELECT</span> </span></span> u</span>.</span>user_id</span>,</span></span> u</span>.</span>email</span>,</span></span> m</span>.</span>user_id</span> as</span> manager_user_id,</span></span> m</span>.</span>email</span> AS</span> manager_email,</span></span> u</span>.</span>created_at</span></span> FROM</span> </span></span> public</span>.</span>users</span> AS</span> u</span></span> LEFT JOIN</span> public</span>.</span>users</span> AS</span> m </span>ON</span> u</span>.</span>manager_id</span> =</span> m</span>.</span>user_id</span></span> WHERE</span></span> u</span>.</span>deleted_at</span> is null</span>;</span></span></code></pre> This way, we established the foundation for listing users while providing much richer information about a person's manager and preventing potential additional roundtrips between client and API. We also included simple business logic—by omitting the soft-deleted employees.</p> After creating the views, it's important to update the postgrest.conf</code> file to use the new api</code> schema:</p> db-uri = "postgres://username:password@localhost/time_off_manager"</span></span> db-schema = "api"</span></span> db-anon-role = "time_off_anonymous"</span></span></code></pre> This change tells PostgREST to treat the api</code> schema as the primary interface for API requests, rather than the public schema. Once done, feel free to restart the PostgREST server and test the new setup.</p> $</span> curl "http://localhost:3000/users"</span></span></code></pre> We can further extend the example by provisioning a view with a summary of the vacation days available per calendar year (of course, for simplicity we won't consider transfers between years, etc.).</p> CREATE VIEW</span> api</span>.vacation_balances </span>AS</span></span> SELECT</span> </span></span> EXTRACT(</span>YEAR FROM</span> transaction_date) </span>AS year</span>,</span></span> user_id,</span></span> SUM</span>(amount) </span>AS</span> total_amount</span></span> FROM</span> public</span>.</span>time_off_transactions</span></span> JOIN</span> api</span>.</span>users</span> USING</span> (user_id)</span></span> WHERE</span> </span></span> leave_type_id </span>=</span> (</span>SELECT</span> leave_type_id </span>FROM</span> public</span>.</span>leave_types</span> WHERE</span> label </span>=</span> 'vacation'</span>)</span></span> GROUP BY</span> </span></span> EXTRACT(</span>YEAR FROM</span> transaction_date), user_id;</span></span></code></pre> While this is nothing groundbreaking, we could (and should) have certainly used views in the first part. So what makes this important now? As mentioned above, the main use cases are abstraction, data access, and security. Additional benefits might include other concerns like derived columns (e.g., full name if we were using first and last name columns), enriching data (e.g., building full URLs from fragments), and other logic specific to the presentation layer.</p> For practice, we can expose possible time off types via a simple view:</p> CREATE VIEW</span> api</span>.leave_types </span>AS</span> </span></span> SELECT</span></span> label</span></span> FROM</span> public</span>.</span>leave_types</span></span> WHERE</span> deleted_at </span>is null</span>;</span></span></code></pre>Let's modify the data</a> </h2> While the simplicity of performing CRUD operations on tables in the public schema was straightforward in the first part of the tutorial, using VIEWs introduces certain complexities. One key limitation of VIEWs is their restricted capability for data modification. Let's look at how to encapsulate the logic.</p> You might have guessed it—the most obvious way forward is to introduce a database stored procedure for any business logic we want to expose.</p> CREATE OR REPLACE FUNCTION</span> api</span>.add_user(</span></span> email </span>text</span>,</span></span> manager_id </span>integer</span></span> ) </span>RETURNS integer AS</span> $$</span></span> DECLARE</span></span> v_new_user_id </span>integer</span>;</span></span> BEGIN</span></span> -- check if email already exists</span></span> PERFORM </span>1</span> FROM</span> api</span>.</span>users</span> WHERE</span> users</span>.</span>email</span> =</span> add_user</span>.</span>email</span>;</span></span> IF</span> FOUND </span>THEN</span></span> RAISE EXCEPTION </span>'The email address % is already in use'</span>, </span>add_user</span>.</span>email</span></span> USING</span> ERRCODE </span>=</span> 'unique_violation'</span>;</span></span> END IF</span>;</span></span> </span></span> -- sample business logic: all employees must have manager</span></span> IF</span> manager_id </span>IS NULL THEN</span></span> RAISE EXCEPTION </span>'manager_id must be provided and cannot be null'</span>;</span></span> END IF</span>;</span></span> </span></span> INSERT INTO</span> public</span>.</span>users</span> (email, manager_id)</span></span> VALUES</span> (</span>add_user</span>.</span>email</span>, </span>add_user</span>.</span>manager_id</span>)</span></span> RETURNING user_id </span>INTO</span> v_new_user_id;</span></span> </span></span> RETURN</span> v_new_user_id;</span></span> END</span>;</span></span> $$ </span>LANGUAGE</span> plpgsql </span>SECURITY</span> DEFINER;</span></span></code></pre> Ok, we have introduced a rather complex piece of logic there. While it might look simple (if you've ever seen a PL/pgSQL function), it comes with a couple of important details.</p> First and foremost, the function api.add_user</code> presents an example of how to implement business logic. In this case, it's providing a custom error message should the employee's email already exist and enforcing the logic that manager_id</code> must be provided. You can probably think of more cases to enrich data.</p> Now for the more difficult part—you might not be familiar with the SECURITY</code> attribute. Every function created can have two modes, INVOKER</code> (which is the default) and DEFINER</code>. If you look above, when we removed all the permissions for our db-anon-role</code> from schema public</code>, the user lost privileges to perform any operations there. Without modifying the security attribute, the function would result in an error message permission denied for table users</code>. Using SECURITY DEFINER</code> guarantees your application user's (the one used for the creating the view) permissions will be used when calling the function.</p> This is the opposite of how VIEWs work in this example, as they come with the default option security_invoker</code> disabled by default. You can refer to the documentation</a> to learn more.</p> If you are not familiar with PL/pgSQL, I recommend you check the basic statements</a>.</p> With the function in place, let's call it using cURL:</p> $</span> curl "http://localhost:3000/rpc/add_user"</span> -X</span> POST</span> \</span></span> -d</span> '{ "email": "admin2@example.com", "manager_id": 2 }'</span> \</span></span> -H</span> "Content-Type: application/json"</span></span></code></pre> This is the second time we need to deconstruct the seemingly simple logic. Let's start with the URL. The /rpc/</code> prefix is important. Every stored procedure in the schema defined using db-schema</code> is accessible under this prefix. The prefix helps to differentiate object types, as in PostgreSQL you are allowed to have the same name for a table/view and a function.</p> The second part is the request method, in this case POST</code>. While you can use both GET</code> and POST</code>, it's important to understand the implications. Using the GET</code> method sets the PostgREST transaction to read-only mode and is a powerful security measure—explicitly preventing the operation from performing any modification of the data (even indirectly via functions or triggers). Should you perform the function using GET</code>, you would get a cannot execute INSERT in a read-only transaction</code> error message to confirm it.</p> The third part of the call is the way the data is presented. In this example, it's using JSON (declared as Content-Type: application/json</code> header) and passed within the request body. If required, you can choose other formats—like application/x-www-form-urlencoded</code>, text/xml</code>, application/octet-stream</code> for bytea</code>, and more using custom media type handlers</a>.</p> Therefore, the same can be achieved using:</p> $</span> curl "http://localhost:3000/rpc/add_user"</span> -X</span> POST</span> \</span></span> -d</span> "email=admin2%40example.com&manager_id=2"</span> \</span></span> -H</span> "Content-Type: application/x-www-form-urlencoded"</span></span></code></pre> Let's omit the XML example completely [sic].</p> As we have introduced the function that modifies the data, we should explore the option to do the same with read-only functions.</p> CREATE OR REPLACE FUNCTION</span> api</span>.get_max_vacation_days()</span></span> RETURNS INTEGER</span> STABLE </span>AS</span> $$</span></span> SELECT</span> max_days</span></span> FROM</span> public</span>.</span>leave_types</span></span> WHERE</span> label </span>=</span> 'vacation'</span>;</span></span> $$ </span>LANGUAGE sql SECURITY</span> DEFINER;</span></span></code></pre> Here, we defined the function get_max_vacation_days</code>, which enables retrieving the current number of vacation days for all employees. To call the function, we can simply run:</p> $</span> curl "http://localhost:3000/rpc/get_max_vacation_days"</span></span></code></pre> This summarises the basics of using a dedicated schema and VIEWs. While using stored procedures is the most obvious way, you can also use updatable views and INSTEAD OF TRIGGERS. This approach allows you to hide the possible transition from table to views and implement the insert logic using the trigger function. While this is a powerful technique, I recommend using direct function calls for clarity (at least when you are getting started).</p> Simple workflow management</a> </h2> As we have introduced quite a lot of (potentially new) concepts, let's practice more and get our hands dirty with the implementation of a simple workflow management system. In our case, it's functionality for employees to request time off and manage approval flow. The premise is simple:</p> Employee can request a time off of a certain type</li> Employee can approve/reject the time off request for their reports</li> The boss can do whatever they want</li> </ul> Let's start with defining the table for time off requests. Please notice we are going to create it again in schema public</code> and not directly expose it via the API.</p> CREATE TABLE</span> public</span>.time_off_requests (</span></span> request_id </span>SERIAL PRIMARY KEY</span>,</span></span> user_id </span>INT REFERENCES</span> public</span>.</span>users</span>(user_id),</span></span> leave_type_id </span>INT REFERENCES</span> public</span>.</span>leave_types</span>(leave_type_id),</span></span> requested_date </span>DATE</span>,</span></span> period</span> DATERANGE,</span></span> status TEXT CHECK</span> (</span>status IN</span> (</span>'pending'</span>, </span>'approved'</span>, </span>'rejected'</span>)),</span></span> created_at </span>TIMESTAMP WITH TIME ZONE DEFAULT</span> current_timestamp</span></span> );</span></span></code></pre> We will add functionality to request time off via a function:</p> CREATE OR REPLACE FUNCTION</span> api</span>.request_time_off(</span></span> user_id </span>INT</span>,</span></span> leave_type </span>TEXT</span>,</span></span> period</span> DATERANGE</span></span> ) </span>RETURNS INTEGER AS</span> $$</span></span> DECLARE</span></span> v_leave_type_id </span>INT</span>;</span></span> v_request_id </span>INT</span>;</span></span> BEGIN</span></span> -- validate the leave type</span></span> SELECT</span> leave_type_id </span>INTO</span> v_leave_type_id </span></span> FROM</span> public</span>.</span>leave_types</span> </span></span> WHERE</span> label </span>=</span> request_time_off</span>.</span>leave_type</span>;</span></span> </span></span> IF</span> v_leave_type_id </span>IS NULL THEN</span></span> RAISE EXCEPTION </span>'Invalid leave type: %'</span>, </span>request_time_off</span>.</span>leave_type</span>;</span></span> END IF</span>;</span></span> </span> -- check if the user ID is valid</span></span> PERFORM </span>1</span> FROM</span> public</span>.</span>users</span> WHERE</span> users</span>.</span>user_id</span> =</span> request_time_off</span>.</span>user_id</span>;</span></span> IF NOT</span> FOUND </span>THEN</span></span> RAISE EXCEPTION </span>'Invalid user ID: %'</span>, </span>request_time_off</span>.</span>user_id</span>;</span></span> END IF</span>;</span></span> </span></span> -- insert the new time off request</span></span> INSERT INTO</span> public</span>.</span>time_off_requests</span> (</span></span> user_id, leave_type_id, requested_date, </span>period</span>, </span>status</span></span> ) </span>VALUES</span> (</span></span> request_time_off</span>.</span>user_id</span>, </span></span> v_leave_type_id,</span></span> CURRENT_DATE,</span></span> request_time_off</span>.</span>period</span>,</span></span> 'pending'</span></span> ) RETURNING request_id </span>INTO</span> v_request_id;</span></span> </span></span> RETURN</span> v_request_id;</span></span> END</span>;</span></span> $$ </span>LANGUAGE</span> plpgsql </span>SECURITY</span> DEFINER;</span></span></code></pre> For the purposes of this guide, let's leave details out, and we would expect the period</code> to be the applicable working days. To reiterate, the function above:</p> Provides basic application logic, setting the status to 'pending'</li> Explicitly declares SECURITY DEFINER</code> to facilitate permissions to access the newly created table.</li> </ul> We can call it using:</p> $</span> curl</span> -X</span> POST "http://localhost:3000/rpc/request_time_off"</span> \</span></span> -d</span> '{"user_id": 6, "leave_type": "vacation", "period": "[2024-05-20,2024-05-21]"} '</span> \</span></span> -H</span> "Content-Type: application/json"</span></span></code></pre> For the client to be able to display the pending requests, let's introduce the view using the request fields and adding the manager ID of the employee, which we will utilise in the next step:</p> CREATE VIEW</span> api</span>.pending_requests </span>AS</span></span> SELECT</span></span> r</span>.</span>request_id</span>,</span></span> r</span>.</span>user_id</span>,</span></span> r</span>.</span>leave_type_id</span>,</span></span> r</span>.</span>requested_date</span>,</span></span> r</span>.</span>period</span>,</span></span> r</span>.</span>status</span>,</span></span> r</span>.</span>created_at</span>,</span></span> u</span>.</span>manager_id</span></span> FROM</span> public</span>.</span>time_off_requests</span> r</span></span> JOIN</span> public</span>.</span>users</span> u </span>ON</span> r</span>.</span>user_id</span> =</span> u</span>.</span>user_id</span></span> WHERE</span> r</span>.</span>status</span> =</span> 'pending'</span></span> ORDER BY</span> r</span>.</span>created_at</span>;</span></span></code></pre> With this in place, we can move forward with implementing the function api.update_request</code> to provide the necessary business logic to approve/reject the requests.</p> CREATE OR REPLACE FUNCTION</span> api</span>.update_request(</span></span> request_id </span>INT</span>,</span></span> user_id </span>INT</span>,</span></span> new_status </span>TEXT</span></span> ) </span>RETURNS</span> VOID </span>AS</span> $$</span></span> DECLARE</span></span> v_requested_user_id </span>INT</span>;</span></span> v_request_manager_id </span>INT</span>;</span></span> BEGIN</span></span> -- validate the new status</span></span> IF</span> new_status </span>NOT IN</span> (</span>'approved'</span>, </span>'rejected'</span>) </span>THEN</span></span> RAISE EXCEPTION </span>'Invalid status: %. Only "approved" or "rejected" are allowed'</span>, new_status;</span></span> END IF</span>;</span></span> </span> -- retrieve the request together with the users associated with it</span></span> SELECT</span> pr</span>.</span>user_id</span>, </span>pr</span>.</span>manager_id</span> INTO</span> v_requested_user_id, v_request_manager_id</span></span> FROM</span> api</span>.</span>pending_requests</span> pr</span></span> WHERE</span> pr</span>.</span>request_id</span> =</span> update_request</span>.</span>request_id</span>;</span></span> </span> IF NOT</span> FOUND </span>THEN</span></span> RAISE EXCEPTION </span>'There''s no pending Time off request ID %'</span>, request_id;</span></span> END IF</span>;</span></span> </span> -- prevent users from self-approving their requests</span></span> IF</span> user_id </span>=</span> v_requested_user_id </span>THEN</span></span> RAISE EXCEPTION </span>'User cannot approve or reject their own request'</span>;</span></span> END IF</span>;</span></span> </span> -- check if the user is either the requester’s manager or the boss</span></span> IF</span> (v_request_manager_id </span>IS NOT NULL AND</span> user_id </span><></span> v_request_manager_id) </span>THEN</span></span> RAISE EXCEPTION </span>'Only the manager or The Boss can approve or reject the request'</span>;</span></span> END IF</span>;</span></span> </span> -- update the request status</span></span> UPDATE</span> public</span>.</span>time_off_requests</span></span> SET status =</span> new_status</span></span> WHERE</span> time_off_requests</span>.</span>request_id</span> =</span> update_request</span>.</span>request_id</span>;</span></span> END</span>;</span></span> $$ </span>LANGUAGE</span> plpgsql </span>SECURITY</span> DEFINER;</span></span></code></pre> This time, the function shouldn't have any surprises—just ensure you follow all the points addressed previously. With the functionality to approve/reject the requests, we are missing the last crucial step: deducting the balance upon approval. There are two ways to go about this: either add the logic to the function above or implement triggers. While a full discussion goes beyond this guide, let's provide a brief summary of when to choose which solution.</p> Choose a function if you have complex logic reusable by many functions (which does not prevent re-use in triggers), and use triggers if you prefer the automatic business rule enforcement and consistency on the database level. For this tutorial let's do triggers to expand further the use of different SQL techniques (plus I would personally choose this solution anyway).</p> Before jumping in, let's create a helper function that will calculate the number of days to deduct from the balance.</p> CREATE OR REPLACE FUNCTION</span> public</span>.days_in_daterange(</span>period</span> daterange) </span>RETURNS INT AS</span> $$</span></span> SELECT</span> upper</span>(</span>period</span>) </span>-</span> lower</span>(</span>period</span>)</span></span> $$ </span>LANGUAGE sql</span>;</span></span></code></pre> With it in place, we can create the function:</p> CREATE OR REPLACE FUNCTION</span> public</span>.create_transaction_on_approval()</span> RETURNS</span> TRIGGER </span>AS</span> $$</span></span> BEGIN</span></span> IF</span> NEW</span>.</span>status</span> =</span> 'approved'</span> THEN</span></span> INSERT INTO</span> time_off_transactions (</span></span> user_id, </span></span> leave_type_id, </span></span> transaction_date, </span></span> time_off_period, </span></span> amount</span></span> ) </span>VALUES</span> (</span></span> NEW</span>.</span>user_id</span>,</span></span> NEW</span>.</span>leave_type_id</span>,</span></span> CURRENT_DATE,</span></span> NEW</span>.</span>period</span>,</span></span> -</span>public</span>.</span>days_in_daterange</span>(</span>NEW</span>.</span>period</span>)</span></span> );</span></span> END IF</span>;</span></span> </span></span> RETURN</span> NEW;</span></span> END</span>;</span></span> $$ </span>LANGUAGE</span> plpgsql;</span></span></code></pre> And set up the trigger itself:</p> CREATE TRIGGER</span> create_transaction_on_approval</span></span> AFTER UPDATE ON</span> time_off_requests</span></span> FOR</span> EACH </span>ROW</span></span> WHEN</span> (</span>NEW</span>.</span>status</span> =</span> 'approved'</span> AND</span> OLD</span>.</span>status</span> =</span> 'pending'</span>)</span></span> EXECUTE FUNCTION</span> create_transaction_on_approval();</span></span></code></pre> With the approval logic in place, we can try the original function via API:</p> $</span> curl</span> -X</span> POST "http://localhost:3000/rpc/update_request"</span> \</span></span> -d</span> '{"request_id": 1, "user_id": 2, "new_status": "approved"}'</span> \</span></span> -H</span> "Content-Type: application/json"</span></span></code></pre> If you have used the correct IDs (both for request ID and user ID), you can now verify the balances using the pre-defined view vacation_balances</code>.</p> $</span> curl "http://localhost:3000/vacation_balances"</span></span></code></pre> This demonstrates the basic workflow and how it can be exposed using PostgREST as an API. If you want to continue practising more database logic, you can add the following logic:</p> Prevent negative vacation balances (intermediate)</li> Prevent employees of a single manager from requesting overlapping vacations (advanced)</li> </ul> End of Part 2</a> </h2> By introducing a dedicated API schema, we have added an abstraction layer that provides an effective boundary between the model/logic itself and the API specifics. While this is a good practice, it's up to you how to expose functionality. The main benefit of a dedicated schema is simplicity, allowing you to grant privileges on the schema in bulk rather than maintaining granular permissions. It also provides an easy way to expose existing databases with PostgREST.</p> While the current state of the sample API for Time Off Manager would work for small teams, in the next part, we will move towards authentication for added security and privacy.</p> And don't forget you can find the source code in the GitHub repository</a>.</p> Deep Dive into PostgREST - Time Off Manager (Part 1) 2024-05-11T00:00:00+00:00 The primary motivation behind boringSQL</strong> is to explore the robust world of SQL and the PostgreSQL ecosystem, demonstrating how these "boring" tools can cut through the ever-increasing noise and complexity of modern software development. In this series, I'll guide you through building a simple yet fully functional application—a Time Off Manager. The goal of this project is not only to demonstrate practical database/SQL approaches but also to provide a complete, extendable solution that you can immediately build upon. Each part of this series will deliver a self-contained application, setting the stage for introducing more complex functionalities and practices.</p> The first part of this guide focuses mainly on the application logic and exposing raw data using postgREST</strong>, allowing you to grasp the concept.</p> Requirements</a> </h2> This tutorial assumes you can install PostgreSQL, connect to it using your preferred DB client, have permissions to create a new database, and understand the basics of schema operations and SQL. Similarly, you should be able to follow the installation instructions for postgREST</a>.</p> The complete source code for the guide is available at GitHub</a>.</p> The Time Off Manager</a> </h2> When choosing a sample application for this tutorial, I was torn between the popular but simple TodoMVC and a slightly more complex application. In the end, a sample like Time Off Manager provides a much richer database scheme, going beyond the basics and offering a solution closer to real-life systems.</p> Time Off Manager is also an excellent example that combines simple business logic, workflows, and practicality. The main requirements are:</p> Maintaining the list of users (employees) with their respective managers</li> Allowing the maintenance of the time off balance, with an audit history</li> Introducing an approval workflow and automation</li> </ul> While the application is intended for learning purposes, the database and API can be easily exposed and built upon.</p> The Database</a> </h2> To get started, you will need to create a database. You can do this in two ways, either using the CREATE DATABASE statement:</p> CREATE DATABASE time_off_manager;</span></span></code></pre>Users model</a> </h2> The foundation and first sample functionality exposed by postgREST is the users themselves. The table schema behind it represents the employees and manages hierarchical relations, which will become important in the second part when we set up the approval workflow.</p> CREATE TABLE</span> users</span> (</span></span> user_id </span>int GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY</span>,</span></span> email </span>text NOT NULL UNIQUE</span>,</span></span> manager_id </span>int REFERENCES</span> users(user_id),</span></span> created_at </span>timestamp with time zone DEFAULT</span> current_timestamp,</span></span> deleted_at </span>timestamp with time zone</span></span> );</span></span></code></pre> For testing purposes, let's populate the data using a seed of 3 managers (one being "the boss" and hence not having a manager) and 10 employees.</p> DO $$</span></span> DECLARE</span></span> v_boss_id </span>int</span>;</span></span> v_manager_id </span>int</span>;</span></span> i </span>int</span>;</span></span> BEGIN</span></span> -- create "the boss"</span></span> INSERT INTO</span> users (email, manager_id)</span></span> VALUES</span> (</span>'owner@example.com'</span>, </span>NULL</span>)</span></span> RETURNING user_id </span>INTO</span> v_boss_id;</span></span> </span></span> -- setup 2 managers </span></span> FOR</span> i </span>IN</span> 1</span>..</span>2</span> LOOP</span></span> INSERT INTO</span> users (email, manager_id)</span></span> VALUES</span> (</span>'manager'</span> \|\|</span> i </span>\|\|</span> '@example.com'</span>, v_boss_id)</span></span> RETURNING user_id </span>INTO</span> v_manager_id;</span></span> </span> -- and 5 employees for each one of them</span></span> FOR</span> j </span>IN</span> 1</span>..</span>5</span> LOOP</span></span> INSERT INTO</span> users (email, manager_id)</span></span> VALUES</span> (</span>'employee'</span> \|\|</span> (</span>5</span> </span> (i </span>-</span> 1</span>) </span>+</span> j) </span>\|\|</span> '@example.com'</span>, v_manager_id);</span></span> END LOOP</span>;</span></span> END LOOP</span>;</span></span> END</span> $$;</span></span></code></pre> In case you are not aware, the seed data is produced by an anonymous block</a>. This way, we executed the logic to create the relationship required without the need to create (and later drop) the PL/pgSQL</em> function.</p> Expose users using postgREST</a> </h2> Having the users table in place, together with sample data, we are ready to expose the data using the REST API. To start, you only need to create a very simple configuration for postgREST</strong> using the file postgrest.conf (you can place the file in the folder created for this sample project).</p> Sample postgrest.conf</code>:</p> db-uri = "postgres://username:password@localhost/time_off_manager"</span></span> db-schema = "public"</span></span> db-anon-role = "time_off_anonymous"</span></span></code></pre> The most important part is the db-uri, which follows the standard PostgreSQL connection string format:</p> postgres://username:password@host/database_name</span></span></code></pre> You need to replace:</p> username</code> - with your actual database username. Ideally, you will use a separate user (for example, time_off_manager) for each application, rather than your personal account.</li> password</code> - the password for the user</li> host</code> - either localhost (if running locally) or a remote server name or IP address</li> database_name</code> - in this case, time_off_manager we created earlier.</li> </ul> NOTE: We are using hardcoded data (which can potentially get stored in your source code repository), and this is mainly for learning purposes. Any deployment resembling a production environment should follow best practices to reduce security risks.</em></p> The other configuration options are db-schema</code>, which we will leave pointing to the public schema for this part, and db-anon-role</code>, configuring the role postgREST should use when executing requests on behalf of unauthenticated clients (effectively all requests in Part 1).</p> For PostgreSQL 15 and higher, please make sure your username</code> is the owner of the database created (should you create it alternatively) in order to be able to create objects in public</code> schema.</p> To set up the database role, you need to run the following commands:</p> CREATE ROLE</span> time_off_anonymous NOLOGIN;</span></span> GRANT SELECT</span>, </span>INSERT</span>, </span>UPDATE ON</span> users </span>TO</span> time_off_anonymous;</span></span> GRANT</span> time_off_anonymous </span>TO</span> CURRENT_USER;</span></span></code></pre> Please note you this example assumes CURRENT_USER</code> is same as the username used in the db-uri</code> configured above. The latter GRANT statement assigns the role time_off_anonymous to the configured user to execute the anonymous commands.</p> Once you have the configuration file ready (assuming it's in the current directory), you can start the postgREST server and expose it locally on port 3000 (by default):</p> $ postgrest postgrest.con</span></span> 10/May/2024:22:06:12 +0200: Starting PostgREST 12.0.3...</span></span> 10/May/2024:22:06:12 +0200: Attempting to connect to the database...</span></span> 10/May/2024:22:06:12 +0200: Connection successful</span></span> 10/May/2024:22:06:12 +0200: Listening on port 3000</span></span> 10/May/2024:22:06:12 +0200: Listening for notifications on the pgrst channel</span></span> 10/May/2024:22:06:12 +0200: Config reloaded</span></span> 10/May/2024:22:06:12 +0200: Schema cache loaded</span></span></code></pre>Exploring the Users Model Using cURL</a> </h2> When you have the server successfully running, you can try to access the data using:</p> $</span> curl "http://localhost:3000/users"</span></span></code></pre> and get the list of all the sample data we created above. The API for reading data is described in detail</a> in the documentation, but you can try different alternatives.</p> # get one specific user</span></span> $</span> curl http://localhost:3000/users?user_id=eq.10</span></span> # query only active (i.e. non-deleted users)</span></span> curl</span> "http://localhost:3000/users?deleted_at=is.null"</span></span> # or display users without any manager (i.e. boss)</span></span> curl</span> "http://localhost:3000/users?manager_id=is.null"</span></span></code></pre> Similarly, you can create, update, and delete users as required—giving you full CRUD functionality with rich filtering options.</p> Create user:</p> $ curl -X POST "http://localhost:3000/users" \</span></span> -H "Content-Type: application/json" \</span></span> -d '{"email": "admin1@example.com", "manager_id": 1}'</span></span></code></pre> Update user:</p> $ curl -X PATCH "http://localhost:3000/users?user_id=eq.10" \</span></span> -H "Content-Type: application/json" \</span></span> -d '{"email": "updateduser@example.com"}'</span></span></code></pre> or, delete one:</p> $ curl -X DELETE "http://localhost:3000/users?user_id=eq.10"</span></span></code></pre> I haven't mentioned CRUD by accident; that's the primary use-case of postgREST—exposing CRUD operations on the tables within the configured schema (public</code> in this part). You can delegate authorisation at the database level.</p> Please note, the IDs are hardcoded, and you might need to check the output of the commands to get the correct ones (should you have truncated the table before or otherwise manipulated the data).</p> Basic models for managing time off</a> </h2> With the User model set up and tested via the API, it's time to return to the core of the Time Off Manager functionality: absence tracking itself. For these purposes, we will define two tables—leave types (identifying the reason or type of the leave) and the time off transactions (or updates) themselves.</p> CREATE TABLE</span> leave_types</span> (</span></span> leave_type_id </span>int GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY</span>,</span></span> label </span>text NOT NULL</span>,</span></span> description text</span>,</span></span> max_days </span>int</span>,</span></span> created_at </span>timestamp with time zone DEFAULT</span> current_timestamp,</span></span> deleted_at </span>timestamp with time zone</span></span> );</span></span> </span> CREATE TABLE</span> time_off_transactions</span> (</span></span> transaction_id </span>int GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY</span>,</span></span> created_at </span>timestamp with time zone DEFAULT</span> current_timestamp,</span></span> user_id </span>int NOT NULL REFERENCES</span> users(user_id),</span></span> leave_type_id </span>int NOT NULL REFERENCES</span> leave_types(leave_type_id),</span></span> transaction_date </span>date</span>,</span></span> time_off_period daterange,</span></span> amount </span>int NOT NULL</span>,</span></span> description text</span></span> );</span></span> </span> CREATE INDEX</span> transaction_for_user</span> ON</span> time_off_transactions(user_id);</span></span></code></pre> The tables should be self-explanatory, but for clarity, let's delve into the detail of the transaction tracking. Time off is tracked for individual users, the time off transaction type to identify the reason for the change, the period during which the absence is recorded, and the amount of effective days (simplified logic to account for weekends, public holidays, and similar).</p> To get started we also need to seed the data:</p> INSERT INTO</span> leave_types (label, </span>description</span>, max_days) </span>VALUES</span> </span></span> (</span>'vacation'</span>, </span>'Annual vacation leave'</span>, </span>25</span>),</span></span> (</span>'sick-leave'</span>, </span>'Leave for health reasons'</span>, </span>10</span>),</span></span> (</span>'unpaid-leave'</span>, </span>'Leave without pay'</span>, </span>NULL</span>),</span></span> (</span>'sabbatical'</span>, </span>'Extended leave for study or travel'</span>, </span>NULL</span>);</span></span></code></pre> and create the initial time off balances (this is a simplified version, assuming the employee always has the full balance, ignoring possible mid-year joiners, etc.):</p> DO $$</span></span> DECLARE</span></span> v_leave_type record;</span></span> BEGIN</span></span> FOR</span> v_leave_type </span>IN SELECT FROM</span> leave_types </span>WHERE</span> label </span>=</span> 'vacation'</span> LOOP</span></span> INSERT INTO</span> time_off_transactions (user_id, leave_type_id, transaction_date, amount, </span>description</span>)</span></span> SELECT</span> user_id, </span>v_leave_type</span>.</span>leave_type_id</span>, </span>'2024-01-01'</span>, </span>v_leave_type</span>.</span>max_days</span>, </span>'Initial balance for year 2024'</span></span> FROM</span> users;</span></span> END LOOP</span>;</span></span> END</span> $$;</span></span> </span></code></pre> This now gives us the ability to start tracking the leave of absence for users and keep a full audit log of it.</p> Before we can start using the table via the API, we need to complete the last important step: giving access to the configured db-anon-role for the tables. We can do this by assigning the grant to individual tables using:</p> GRANT SELECT</span>, </span>INSERT</span>, </span>UPDATE ON</span> leave_types </span>TO</span> time_off_anonymous;</span></span> GRANT SELECT</span>, </span>INSERT</span>, </span>UPDATE ON</span> time_off_transactions </span>TO</span> time_off_anonymous;</span></span></code></pre> or modify the default privileges. Spoiler alert—we will move away from using the public schema in Part 2, so let's not do more than we need at the moment :)</p> Working with absence</a> </h2> Similar to the users example, we can now track absence directly using the API. Before firing off cURL commands, there's one last thing to remember. postgREST</strong> requires metadata about the database schema itself, and it might be expensive to perform on the fly, hence to avoid this, the server caches the schema.</p> To reload the schema (after making schema changes), it is required to reload the cache. This can be done via several basic options:</p> restarting the server</li> issuing a (SIGUSR1) signal to the server process</li> </ul> and more advanced ways</a> (outside Part 1 of this tutorial).</p> Once reloaded, you can start exploring more:</p> # getting a time off transaction for particular user</span></span> $ curl "http://localhost:3000/time_off_transactions?user_id=eq.1"</span></span> # submitting new leave of absence</span></span> $ curl -X POST http://localhost:3000/time_off_transactions \</span></span> -H "Content-Type: application/json" \</span></span> -d '{"user_id": 5, "leave_type_id": 1, "transaction_date": "2024-02-26", "time_off_period": "[2024-02-28,2024-03-04]", "amount": -4, "description": "Vacation to avoid panic over leap year"}'</span></span></code></pre> All these commands work as expected. Should you forget to reload the schema, you might face an HTTP Status 404 (Not Found) on the newly created objects.</p> End of Part 1</a> </h2> While not being very sophisticated, I hope this first part has demonstrated the CRUD capabilities postgREST</strong> can offer with minimal effort. In the next part, we will move away from accessing the database tables directly and provide more functionality via a new schema api, providing a per-user time off balance view and introducing workflow management.</p> How not to change PostgreSQL column type 2024-05-04T00:00:00+00:00 One of the surprises that comes with developing applications and operating a database cluster behind them is the discrepancy between practice and theory, development environment and the production. A perfect example of such a mismatch is changing a column type.</p> The conventional knowledge on how to change a column type in PostgreSQL (and other systems compliant with the SQL standard) is to:</p> ALTER TABLE</span> table_name</span></span> ALTER</span> COLUMN column_name</span></span> [SET DATA]</span> TYPE</span> new_data_type</span></span></code></pre> which is obviously the semantically correct way, but given the right circumstances, you might be set for a rather unpleasant surprise.</p> The Problem</a> </h2> Let's create a sample table and demonstrate the isolated behaviour that you might observe. Let's start with 10 million rows (which really is just a drop in the whole world of data).</p> -- create very simple table</span></span> CREATE TABLE</span> sample_table</span> (</span></span> id </span>INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> label </span>TEXT</span>,</span></span> created_at </span>TIMESTAMP WITHOUT TIME ZONE DEFAULT NOW</span>(),</span></span> updated_at </span>TIMESTAMP WITHOUT TIME ZONE DEFAULT NOW</span>()</span></span> );</span></span> </span> -- populate with 10m records</span></span> INSERT INTO</span> sample_table (label)</span></span> SELECT</span></span> 'hash: '</span> \|\|</span> md5(random()::</span>text</span>)</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>7000000</span>);</span></span></code></pre> and let's change the id type from INT to BIGINT.</p> alter_type_demo=# ALTER TABLE sample_table ALTER COLUMN id TYPE bigint;</span></span> ALTER TABLE</span></span> Time: 21592.190 ms (00:21.592)</span></span></code></pre> And ... 21 seconds later, you have your change. Mind you, this is a small table with roughly 600 MB of data in it. What if you going to face 100x that amount? Let's have a look what went on behind the scene.</p> The things PostgreSQL must do</a> </h2> Changing a data type (and many other operations you might experience) is no simple task, and the PostgreSQL engine has to perform several tasks:</p> Rewrite the table</strong> is the most obvious culprit. Changing a column from INT to BIGINT requires 4 additional bytes to be allocated for every single tuple (ahem, think row). As the original table schema required a fixed amount of bytes, the cluster stored them in the most efficient way. i.e., every single row, in our example 10 million of them, must be read and re-written using the correct tuple size.</li> Locks</strong> might not be a problem in our synthetic example, but should you execute the ALTER command in production with hundreds or thousands of concurrent queries, you will have to wait for all of them to release the locks.</li> Indexes and constraints</strong> - if the column being altered is indexed or has constraints, they need to be rebuilt/revalidated. This is additional overhead.</li> Transactions & Write-Ahead Log</strong> is another big part of the problem. To guarantee durability (the 'D' in ACID), PostgreSQL has to record each change in WAL files. This way, if the database crashes, the system can replay the WAL files to reconstruct the lost modifications since the last checkpoint.</li> </ul> As you can see, there's quite a bit involved in performing something that might be understood as routine table maintenance. The size of the data being modified, disk I/O and capacity, and general system congestion come into play.</p> But the real problem does not end here. If we are talking about any sort of serious production deployment, you must consider more things:</p> Real-time replication</strong>, both physical and logical: This adds an additional layer of complexity. For read replicas, the default behaviour ensures that a synchronous commit is maintained to achieve consistency across the database cluster. This setup guarantees that a transaction is only finalised once all standby replicas have confirmed receipt of the changes. However, this introduces new challenges, as the performance now also depends on network throughput—including potential congestion—and the latency and I/O performance of the standby nodes.</li> Recovery and backups</strong> are another important area to consider. While the regular backups' size might not be affected that much, you have to consider everything that happens between the last backup before the change and the next one and ensure point-in-time consistency.</li> Less common but not unheard of might be asynchronous replicas</strong> or reserved slots for logical replication</strong>. Generating a large volume of changes (and therefore WAL files) can leave the less performant (or infrequent) replication systems behind for a considerable amount of time. While this might be acceptable, you need to make sure the source system has enough disk space to hold the WAL files for a long enough time</li> </ul> As you can see, altering the column data type is not as straightforward as it might seem. Current CI/CD practices often make it very easy for software developers to commit and roll out database migrations to a production environment, only to find themselves in the middle of a production incident minutes later. While a staging deployment might help, it's not guaranteed to share the same characteristics as production (either due to the level of load or monetary constraints).</p> The problem is, therefore (and I will repeat myself), the scale of the amount of data being modified, overall congestion of the system, I/O capacity, and the target table's importance in the application design.</p> At the end of the day, it translates to the total time</strong> needed for the migration to finish, and the unique constraints your business can or maybe cannot afford. The simplest solution to the problem is to schedule the planned maintenance during low traffic periods and get it done.</p> How to Safely Change a PostgreSQL Column Type</strong></a> </h2> What if you need to rewrite hundreds of gigabytes or even terabytes of data and can't afford anything more than minimal downtime? Let's explore how to change a column type properly.</p> Let's start with The Bad News</strong> - you cannot avoid rewriting the entire table, which will generate a significant amount of WAL files in the process. This is a given, and you must plan how to manage it.</p> The Good News:</strong> You can spread the potential downtime over a much longer period than it might take to process the data. The specific requirements and constraints will vary based on individual business needs, so careful planning is essential.</p> The full migration can be summarised as series of following steps:</p> Add a new column</strong> to the target table with the correct type. Ensure the column is NULLable and does not have a default value to avoid forcing a full table rewrite[^1]. For example, should you need to increase ID of order_id</code> you will end up with new column new_order_id</code>.</li> Setup a trigger</strong> to update the new column as new data comes in. This ensures that all new data during the migration will have the new column populated.</li> Implement a function or logic to batch migrate</strong> the values from old to new column over time. The size of the batch and the timing should align with the operational constraints of your business/environment.</li> Migrate Old Values:</strong> Depending on your constraints, data size, and I/O capabilities, this process could take anywhere from hours to several weeks, or possibly longer. While a SQL or PL/pgSQL function running in a terminal session (consider using tmux) might suffice for shorter migrations, more extended migrations may require a more sophisticated approach. This topic alone could be a good subject for a separate blog post or guide.</li> Once the migration is complete, create constraints and indexes</strong> that reflect the new column. Be aware of potential locking issues, especially if the field is part of any foreign keys.</li> </ul> At this point you are ready to perform the switch itself. If you can verify all rows have correctly populated new column, it's time to embrace the most difficult part. If possible in one transaction and or smaller scheduled downtime</p> Drop the legacy column</strong>. This operation usually only locks the table for a short duration.</li> After dropping the old column, rename the new column</strong>. This step completes most of the migration process.</li> </ul> It's good practise to consider the restart of all the application relying on the changed table, as some tools (ORMs... I'm looking at you) might cache the OIDs and not handle the change gracefully.</p> And that would be it - except not really. Dropping the column only removes the reference and the data itself will remain physically on the disk. This is the scenario where you will might need to perform VACUUM FULL</code> - which could lock the table and rewrite it completely—potentially defeating the purpose of a concurrent migration. This brings us back to the original article which motivated me to write this guide - [[The Bloat Busters: pg_repack vs pg_squeeze]] is the way to go. Preparation and familiarity with these tools in advance are highly recommended.</p> Conclusion</a> </h2> While changing the column type in PostgreSQL can be as simple as issuing an ALTER TABLE command, it is important for everyone involved to understand the complexities attached to it. Whether you are the software developer requesting the change, someone reviewing it, or the individual tasked with resolving incidents when such changes are deployed to the production environment without careful planning, a deep understanding of this process is crucial. Moreover, grasping this particular change enables you to easily project insights onto other potentially costly operations.</p> [^1] Correction: it's actually done without full table rewrite since PostgreSQL 11 (Fast Column Creation with Defaults</a>)</p> The Bloat Busters: pg_repack vs pg_squeeze 2024-04-27T00:00:00+00:00 As the database size increases and the number of transactions per second rise, you'll inevitably face the challenge of the table bloat. Although PostgreSQL assists as much as possible with its auto-vacuum feature</a>, there will come a time when you will compel whether to run VACUUM FULL</code>. Unless you have option of longish downtime windows, this is not an easy decision.</p> Thankfully, the rich ecosystem of PostgreSQL offers more than one solution how to make it simpler. Ignoring older tools like pg_reorg</code>, two contenders worth considering are pg_repack</strong> and pg_squeeze</strong>. This article dives into their strengths and weaknesses to help you decide which one is better for your specific use case.</p> Cases of Heavy Duty Maintenance</a> </h2> In scenarios with a continuous and predictable transaction pattern, you can usually rely on auto-vacuum. However, there are use cases where this won't be sufficient. Such examples typically involve "bulk" operations—whether it's bulk imports, deletions, or a combination of both. Imagine scenarios where:</p> You migrate one or more column data types</a> over a longer period (and no, using ALTER COLUMN name TYPE new_type</code> is not the best option).</li> You drop a column that has been moved to a different table.</li> You need to modify a large amount of data, using soft-deletes and later actual DELETEs</a>.</li> Due to changes in compliance requirements, you need to DELETE a massive amount of data.</li> </ul> In all these cases, you might end up with a huge table and the traditional solution would be VACUUM FULL</code>, which is associated with significant downtime due to table locking.</p> Alternatives to VACUUM FULL</a> </h2> Due to the limitations of VACUUM FULL</code>, several alternatives have emerged. The first contender in this space was pg_reorg</code> (which this comparison will not cover), later superseded by pg_repack</code>. The relatively new kid on the block is pg_squeeze</code>.</p> Each solution comes with different architectures, deployment methods, and sets of pros and cons.</p> As an added benefit, both tools can effectively serve as alternatives to CLUSTER</code> (which also requires the exclusive table lock similar to full vacuum), making them effective in optimising data storage based on the given order.</p> Both tools also share common behaviour. They won’t magically remove the bloat—you need to have at least the same amount of space available as the new table will require. Therefore, deploying both tools needs to be considered well before you reach critical levels of available disk space. Both will also generate a significant volume of WAL files — affecting every 8KB page</a> along the way — which need to be stored, processed, backed up, etc.</p> While not directly comparable to these tools, another method that can be used manually is table partitioning. Although it won’t be included in this comparison, if planned in advance, it can simplify certain maintenance tasks, improve performance, and make routine vacuuming faster and more efficient. It's only fair to say - it all depends on the specific scenarios and usage.</p> pg_repack</a> </h2> At the time of writing this article, pg_repack</code> can be considered the most well-known solution for combating table bloat in the PostgreSQL ecosystem. Built as an extension, it's easy to install (either from package repositories or via a self-compiled artifact) and its setup does not require a cluster restart.</p> pg_repack</code> works by creating a new copy of the table being processed, setting up triggers to replicate new data while it fills up with the existing data. An exclusive full table lock is required both at the start and finish of the process when swapping the old and new tables.</p> The maintenance is initiated from the CLI and allows you to specify a number of arguments to fine-tune the repacking of individual tables to match the requirements. pg_repack</code> allows re-clustering of data based on columns only, i.e., it does not explicitly require an index and can therefore overcome some limitations of pg_squeeze</code> in this regard.</p> pg_repack</code> is very effective in reclaiming space and can work with all types of bloat. It's available out of the box both on Amazon RDS and Google Cloud SQL.</p> The drawback you might experience when terminating the process (which may be necessary for various reasons, such as impact on the running environment) is that it won’t clean up all the fragments as it won't remain connected to the target database.</p> pg_squeeze</a> </h2> Compared to the previous solution, pg_squeeze</code> is built differently, relying on logical decoding instead of triggers. Its main benefit is a lower impact on the host system during table rebuilding, improving availability and stability.</p> Like pg_repack</code>, pg_squeeze</code> creates a new table and copies the existing data from the bloated table. Logical replication is involved in streaming changes from the original table to the newly created one in real-time. This allows the new table to stay up-to-date during the process without unnecessary impact on the regular operations performed on the bloated table. Thus, pg_squeeze</code> significantly reduces the need for locking. The exclusive lock is needed only during the final phase of the operation, when the old table is swapped out for the new, optimized table. The duration of the exclusive lock can also be configured.</p> While pg_squeeze</code> is also available as an extension, its deployment necessitates configuration changes involving wal_level</code>, max_replication_slots</code>, and shared_preload_libraries</code> — the same settings used for logical replication</a>. Due to this, a restart of the cluster is required.</p> On the other hand, pg_squeeze</code> is designed for regular, rather than ad-hoc processing only. You can register a table for regular processing, and whenever the table meets the criteria to be "squeezed," a task will be added to a queue, where it will be sequentially processed in the order they were created. The automated processing offers basic options usable in most scenarios, but you might find it limited if you need to work around other operational constraints of the cluster/environment. Having said that, the maintenance of the table can also be triggered manually.</p> While pg_squeeze</code> might be considered superior to pg_repack</code> in terms of maintenance operations and impact, it comes with a significant caveat when reclaiming space—compared to pg_repack</code>, it copies the full rows as they are. This behaviour renders it ineffective at removing bloat created due to dropped columns (behaviour still present at the time of the writing of this article at the end of April 2024).</p> As already mentioned, pg_squeeze</code> can use a specified index for clustering when needed. The limitation you might find is that clustering cannot be performed on a partial index. Compared to pg_repack</code>, it always seems to clean up all the artefacts accordingly (thanks to the always-running worker process).</p> Unfortunately at the moment of writing the article pg_squeeze</code> is not available for Amazon RDS, only Google Cloud SQL.</p> Conclusion</a> </h2> It's remarkable to have two mature and production-tested tools at your disposal. Deciding between them comes down to the specific requirements, the use cases, and the operational specifics of the business services.</p> The very opinionated difference between the tools can be made, using pg_squeeze</code> for automated, continuous cleaning of specific tables, and pg_repack</code> as the heavyweight champion of controlled setups.</p> (Edit) Added availability on Amazon RDS and Google Cloud SQL.</p> Are SQL & Databases Boring? Absolutely - and That's a Good Thing! 2024-04-21T00:00:00+00:00 Twenty years ago, I would have laughed if you had told me I'd be promoting databases as the technology to turn to. Back then, databases were just a 'dummy' storage for me—a necessary evil, a sentiment shared by many developers. At that point, I felt so strongly about it that I was a trainer for Hibernate, a tool which helped me keep those pesky databases at arm's length.</p> Fast forward, and here I am, starting boringSQL</strong>, a site dedicated to helping demystify the depths of SQL and databases in general. Whether you've just started or want to gain expert knowledge. What changed, you might ask? Nothing actually. Well, a few things did. The frameworks changed. I moved from the Java world to Rails, jumped to Python and then to Go, worked with countless frontend paradigms, and oversaw the implementation of a number of technologies. The same experiments were tried again and again. Only one part of the tech stood the test of time. Yes, it's the database. And out of many, one in particular—PostgreSQL. And while alternatives promised, and still promise, that the grass would be greener—I can honestly say that whatever revolution comes in the technology landscape, databases will hold the same place in the next twenty years.</p> This is not the first boring</strong> site, and rest assured, the concept of celebrating the mundane in technology is far from new. The tech landscape is paved with many "boring" things, from frameworks to methodologies to complete stacks. Businesses depend on these technologies to run their operations day in, day out. And while it might be fun to try new things every month, it's not the headline-driven development that is the way forward.</p> Hence, if you ask—is SQL boring? Are databases boring? I would also say YES. They are among the finalists of the least popular conversation starters. But it's the quiet power of reliability that has a healing effect in a world where technology changes at the speed of light.</p> When and Why PostgreSQL Indexes Are Ignored 2024-04-14T00:00:00+00:00 While it's true the most problems can be solved by the appropriate use of the index, there are cases where you will just waste resources doing so. For casual developer it might seems like PostgreSQL decided to do its own thing, but when you look behind the scenes it all makes perfect sense. Here's a quick run down of the some reasons why planner might pass on index and rather do things without it.</p> The case of missing condition</a> </h2> In the evolution of the software developer journey into the realms of databases, the partial indexes are the next best thing. Why to create the full index where you can easily restrict it to the sub-set of the records? The trick is ensure every query aligns with the condition given during the index creation.</p> Think of an order management system, where most active orders are going to those in 'open' status. Most of the day-to-day operational tasks will resolve around those entries, a partial index like</p> CREATE INDEX</span> open_orders</span> ON</span> orders(order_id) </span>WHERE state =</span> 'open'</span>;</span></span></code></pre> provides a perfect match. It will allow to iterate only on a relatively small subset of the data, hence saving the disc space. As long as the application developers are aware of the partial condition, and how to use it.</p> The column order matters</a> </h2> The second common case is just a step away from the partial index. The compound indexes, or multi-column indexes, are versatile and powerful - that is when used correctly. The crucial element is the order in which the columns are indexed. The order impacts whatever it gets used or not at all.</p> The key is to match the left-most columns of the index in your queries. Once these are aligned, you can optionally skip or include additional columns in the filter, but the initial columns must be used to leverage the index effectively.</p> While it might be easy not to miss the condition of the country, when filtering for cities</p> CREATE INDEX</span> contact_location</span> ON</span> contact_details (country_id, city);</span></span></code></pre> consider the scenario which might not be so obvious.</p> CREATE INDEX</span> brand_products_per_category</span> ON</span> products (category_id, brand_id);</span></span></code></pre> In this case the index will only get used if either both category_id</code> and brand_id</code> are used, or at least category_id</code> - but not for brand_id</code> alone. In latter case PostgreSQL planner might (unless another index is available) to opt in for scan of the entire table.</p> In this case alternative strategy would be to switch the column order (should the application logic support it).</p> Low Selectivity</a> </h2> If we use the example of the partial index from the first section, we can easily demonstrate another case when index just might get ignored forever. It's the case for the columns with low selectivity, where a predominant value overshadows others, making an index less useful.</p> Take example of the ordering system. While having index for open</code> orders is helpful, in most scenarios majority of the records will end up in delivered</code> state. Hence</p> CREATE INDEX</span> delivered_orders</span> ON</span> orders(order_id) </span>WHERE state =</span> 'delivered'</span>;</span></span></code></pre> might seem beneficial, but in reality PostgreSQL will most likely chose to skip it and instead to scan the table. The reason? Should you consider the regular business operations, you might find vast majority (let's say 95%) or orders shipped and considered finished. With such a high percentage of the records orders, the index will do little to narrow the search for the records. Planner hence will perform the sequential scan directly. Why? It's all to do with the statistics that are available on the table.</p> table_name \| orders</span></span> column_name \| state</span></span> n_distinct \| 4</span></span> most_common_vals \| {delivered,cancelled,pending,open}</span></span> most_common_freqs \| {0.95023334,0.030166665,0.013,0.0096}</span></span></code></pre> Those are fictional statistics matching the order distribution described above. From there you can see that delivered</code> orders dominate the dataset, compromising approximately out of 95.02% of the records. The other statuses make up just for a small fraction.</p> The above data sample comes from the query similar to</p> SELECT</span></span> tablename </span>AS</span> table_name,</span></span> attname </span>AS</span> column_name,</span></span> n_distinct,</span></span> most_common_vals,</span></span> most_common_freqs</span></span> FROM</span></span> pg_stats</span></span> WHERE</span></span> tablename </span>=</span> 'orders'</span></span> AND</span> schemaname </span>=</span> 'public'</span>;</span></span></code></pre>Outdated Statistics</strong></a> </h2> As demonstrated low selectivity to addressed by statistics, instead of index, it's not the only case where stale data can lead to inefficient query planning. Keeping database statistics current is crucial for PostgreSQL to make informed decisions about whether to use an index or opt for a sequential scan.</p> PostgreSQL heavily relies on statistics to estimate costs and make the right decisions across different query execution plans. The statistics cover various data points, like total number of rows in table(s), distinct values, their distribution, histogram bounds, correlation between physical row ordering and column values, and much more.</p> The trouble starts when statistics get out-of-date. It's similar as navigating using old maps. It just might you send you going in circles. How might the statistics in PostgreSQL get dated? Most cases involve high volume of data modifications</strong> or bulk operations</strong>, but will be affected also by schema changes. For all those operations it's always good idea to ANALYZE</code> the table to keep the DB up-to-date.</p> The prevent the statistics outdated, PostgreSQL alone either manual ANALYZE</code> or periodically tries to trigger it as part of autovacuum daemon. The key factor is whatever it can run fast and frequently enough to keep up with the changes. You can adjust the autovaccuum settings globally or per-table.</p> Keeping statistics up-to-date is crucial for maintaining good query performance, as it ensures that the query planner has accurate information to base its decisions on.</p> Summary</a> </h2> As demonstrated creating index might not be always the best course of action. Understanding those edge-cases is crucial not only for DBAs but also for developers. By keeping those considerations you can better design the schema and indexing strategy.</p> Indexes are still the next best thing in database world, but they require thoughtful implementation and maintenance. The goal is not only to create indexes, but to create the right indexes</strong> based on accurate, up-to-date data and aligned with your specific query patterns.</p>

pg_regresql: truly portable PostgreSQL statistics

Production query plans without production data

PostgreSQL Statistics: Why queries run slow

Inside PostgreSQL's 8KB Page

Reading Buffer statistics in EXPLAIN output

Introduction to Buffers in PostgreSQL

The hidden cost of PostgreSQL arrays

Instant database clones with PostgreSQL 18

VACUUM Is a Lie (About Your Indexes)

RegreSQL: Regression Testing for PostgreSQL Queries

Beyond Start and End: PostgreSQL Range Types

PostgreSQL maintenance without superuser

Beyond the Basics of Logical Replication

First steps with Logical Replication in PostgreSQL

PostgreSQL Service Connections

Time to Better Know The Time in PostgreSQL

VIEW inlining in PostgreSQL

DELETEs are difficult

Text identifiers in PostgreSQL database design

We need to talk about ENUMs

Beyond Simple Upserts with MERGE in PostgreSQL

Gentle Introduction to Window Functions in PostgreSQL

The time keepers: pg_cron and pg_timetable

Deep Dive into PostgREST - Time Off Manager (Part 3)

Custom PostgreSQL extensions with Rust

Deep Dive into PostgREST - Time Off Manager (Part 2)

Deep Dive into PostgREST - Time Off Manager (Part 1)

How not to change PostgreSQL column type

The Bloat Busters: pg_repack vs pg_squeeze

Are SQL & Databases Boring? Absolutely - and That's a Good Thing!

When and Why PostgreSQL Indexes Are Ignored

boringSQL | Supercharge your SQL & PostgreSQL powers

Good CTE, bad CTE