Keel Reference Guide

Overview

Keel is a high-performance, database-agnostic connection pooler and proxy written in modern C (C23). It supports both PostgreSQL (v3 wire protocol) and MySQL (client/server protocol) from a single binary, with native io_uring support, transparent read/write routing, prepared-statement virtualization, and full TLS/mTLS.

The core design principle is share-nothing: each worker thread owns its reactor, session slab, backend pool, and timer wheel — eliminating cross-thread locking in the fast path.

Production Status

Installation

Debian / Ubuntu

wget https://github.com/virtlabs-io/keel/releases/download/v0.2.0/keel_0.2.0_amd64.deb
sudo dpkg -i keel_0.2.0_amd64.deb

Build from Source

# Dependencies
sudo apt install cmake gcc libssl-dev liburing-dev

# Clone and build
git clone https://github.com/virtlabs-io/keel.git
cd keel
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)
sudo make install

Docker

docker pull ghcr.io/virtlabs-io/keel:latest

Build Options

Quick Start

The fastest path to running Keel as a PostgreSQL connection pooler:

[keel]
experimental_features = false

[worker_group.main]
protocol          = postgresql
bind_addr         = 0.0.0.0
bind_port         = 6432
num_workers       = 0             # auto: one per CPU core
mode              = pool          # production default
prepared_statement = virtualize

min_pool_size     = 10
max_pool_size     = 50

[worker_group.main.servers]
primary = host=127.0.0.1 port=5432 dbname=mydb user=app password=secret role=RW weight=100

# Start
sudo systemctl start keel

# Connect through Keel (port 6432) instead of direct (5432)
psql "host=localhost port=6432 dbname=mydb user=app"

# Verify via admin console (Prometheus port)
curl http://localhost:9187/metrics | grep keel_connections

Architecture

Keel uses a share-nothing, per-worker architecture. Each worker thread is a completely independent unit — it owns its reactor, session slab, backend pool, and timer wheel. There is no cross-thread locking in the query path.

Worker Components

Request Flow

Client
  → accept (SO_REUSEPORT) → Worker assignment
  → Startup + Auth (SCRAM-SHA-256 / MD5 / caching_sha2 / LDAP / PAM / mTLS / Cloud IAM)
  → Session created from slab
  → Query Rules (declarative INI rules — before SQL parse)
  → SQL Lexer → Parser → Query Tree    [skipped in PROXY mode]
  → Throttle check (token-bucket rate limiter)
  → OSC / NOTIFY / LISTEN / keel.* GUC intercept
  → Router (R/W split, shard dispatch, hook chain, plugin)
  → Backend Pool borrow (or wait queue + async refill)
  → Backend Protocol (proxy query, relay results, zero-copy splice)
  → Backend Pool return (transaction complete)
  → Stats update (counters, histogram, tracing span)

Connection Multiplexing

Workers use SO_REUSEPORT on the listen socket so the kernel distributes new connections across workers without any userspace balancing. Each worker independently runs its own io_uring ring, so there are zero cross-worker lock acquisitions on the query hot path.

Reactor Model

Keel abstracts platform I/O behind a unified keel_reactor_t API. Three backends are supported, selected automatically at startup:

io_uring Optimizations

• Registered FDs — file descriptors pre-registered with the ring, skipping per-operation fd table lookups
• Linked SQEs — backend connect + SCRAM auth chained as atomic io_uring sequences (IOSQE_IO_LINK), eliminating one syscall per round-trip
• MSG_PEEK + Splice bypass — when fast_network_path = on, DataRow frames are spliced directly from backend socket → kernel pipe → client socket with zero userspace copies
• Single-shot accept — fair connection distribution across workers

Connection Pooling

Keel supports transaction pooling as the primary mode: backend connections are borrowed from the pool at the start of each transaction and returned when the transaction completes. This allows hundreds of application threads to share a small pool of backend connections.

Pool Lifecycle

1. Client sends a query
2. Engine borrows a backend connection from the per-worker pool
3. If no idle connection is available, a waiter is queued and an async refill is triggered
4. Before forwarding the query, the engine syncs any session state differences (SET parameters, search_path) to the borrowed backend
5. Query is forwarded; results relayed back to client
6. On ReadyForQuery('I'), the backend connection is returned to the pool

Pool Sizing

DISCARD ALL on Return

When a dirty backend connection (one that has SET parameters or other state) is returned to the pool, Keel sends DISCARD ALL asynchronously and only returns it to the available pool after PostgreSQL confirms readiness with ReadyForQuery('I'). This state is tracked by the CLEANING pool entry state.

Session State — SSV

Semantic State Virtualization (SSV) is Keel's consistency model for preserving session state (GUC parameters, search_path, prepared statements) across transaction-pooled backend reassignments.

How It Works

1. Each client session tracks a sorted key-value state profile (SET parameters, search_path, etc.)
2. Each backend connection also tracks its current server-side state
3. On pool borrow, Keel computes a two-pointer merge diff between client state and backend state
4. If the states match (XXHash64 fingerprint check), the borrow is on the fast path — zero SQL sent
5. If they differ, minimal SQL (SET x = 'v') is generated and sent as a pre-query phase

Client state:   search_path='myschema', statement_timeout='5000', work_mem='256MB'
Backend state:  search_path='public',                              work_mem='128MB'
───────────────────────────────────────────────────────────────────────────────────
Sync SQL:       SET search_path='myschema'; SET statement_timeout='5000';
                                            -- work_mem differs; RESET work_mem sent

70–90% of borrows hit the hash-match fast path (zero sync SQL needed).

5-Tier Borrow Strategy

The backend pool borrow logic uses a 5-tier strategy, attempting each tier in order until a suitable connection is found:

1. Exact hash match — state already identical, zero sync SQL
2. Subset match — backend state is a superset; only additions needed
3. Compatible match — small diff; generate minimal SET commands
4. DISCARD candidate — send DISCARD ALL to clean and reuse
5. New connection — establish fresh backend connection

Runtime Modes

Keel has four runtime tiers, each activating additional features. Disabled features cost at most one cmp + jcc instruction — effectively zero overhead.

[worker_group.main]
mode = pool    # proxy | pool | smart | full  (default: pool)

Configuration Reference

Keel uses INI-format configuration files. Sections map to component areas. Indented keys belong to the nearest section above them.

[keel] — Global Settings

[worker_group.NAME] — Worker Group

[worker_group.NAME.servers] — Backends

[worker_group.main.servers]
# name = host=HOST port=PORT dbname=DB user=USER password=PASS role=ROLE weight=N
primary  = host=db1.local port=5432 dbname=app user=app password=s role=RW weight=100
replica1 = host=db2.local port=5432 dbname=app user=app password=s role=RO weight=100
replica2 = host=db3.local port=5432 dbname=app user=app password=s role=RO weight=50

Role values: RW (primary, read+write), RO (replica, read-only), auto (detected via pg_is_in_recovery() or Patroni REST API).

[probe] — Health Checks

Prepared Statements

Prepared statements are a challenge for connection pooling because they are bound to a specific backend connection. Keel offers four strategies:

Query Rules

Declarative INI-based rules evaluated before normal routing. No hook code required.

# Block a specific query pattern
[query_rule.block_truncate]
match      = ^\s*TRUNCATE
block      = true
block_msg  = "TRUNCATE is not allowed through the proxy"

# Route analytics queries to replicas
[query_rule.analytics_replica]
match      = ^\s*SELECT.*FROM\s+(events|metrics|logs)
route_to   = replica

# Rewrite legacy table name
[query_rule.rewrite_legacy]
match      = FROM old_users(\b.*)
rewrite_to = FROM users\1

# Throttle heavy reports
[throttle.reports]
match          = SELECT.*GENERATE_SERIES
rate_per_second = 5
burst          = 10

Read/Write Splitting

Available in smart and full modes. Keel's SQL parser classifies every query and routes it to the appropriate backend.

Routing Rules

• Writes → primary: INSERT, UPDATE, DELETE, DDL, CALL, MERGE
• Reads → replica (by weight): SELECT (without FOR UPDATE)
• Locking reads → primary: SELECT ... FOR UPDATE, SELECT ... FOR SHARE
• Sticky-primary window: After a write, reads are temporarily pinned to the primary (100 ms default) before replica routing resumes
• Open transactions → pinned to whichever backend received BEGIN

Cross-Service Read-After-Write

-- After writing in service A, propagate the WAL position:
SHOW keel.write_lsn   -- returns e.g. '0/15E3B20'

-- In service B, enforce read-after-write consistency:
SET keel.read_after_lsn = '0/15E3B20'
-- Keel will route to primary if no replica is caught up to that LSN

Sharding

Keel supports transparent horizontal sharding via SQL AST shard-key extraction. Applications connect to a single Keel endpoint and Keel automatically routes queries to the correct shard.

Configuration

[keel]
experimental_features = true

[shard_rule.orders]
table        = orders
column       = user_id
shard_count  = 4
strategy     = hash    # hash | range | modulo

[shard_backend.0]
host = shard0.local
port = 5432
database = mydb
user = app

[shard_backend.1]
host = shard1.local
port = 5432
database = mydb
user = app

# ... shard_backend.2, shard_backend.3

Admin Commands

SHOW SHARD RULES;
EXPLAIN SHARD PLAN FOR 'SELECT * FROM orders WHERE user_id = 42';

Scatter-Merge Queries

When sharding is enabled, Keel can fan out aggregation queries to all shards and merge the results transparently — no application changes needed.

Supported Aggregations

• COUNT, SUM, AVG, MIN, MAX, COUNT(DISTINCT col)
• GROUP BY, HAVING, ORDER BY
• LIMIT / OFFSET with correctness fix (strips per-shard limits before merging)

-- Application sends this single query to Keel:
SELECT status, COUNT(*) AS cnt, SUM(amount) AS total
FROM orders
GROUP BY status
ORDER BY total DESC
LIMIT 5;

-- Keel fans out to all 4 shards, merges groups, sorts, returns top-5.

Authentication Methods

TLS / mTLS / kTLS

[worker_group.main]
# Frontend TLS (clients → Keel)
tls_mode        = require          # disable | allow | prefer | require
tls_cert_file   = /etc/keel/server.crt
tls_key_file    = /etc/keel/server.key
tls_ca_file     = /etc/keel/ca.crt  # required for mTLS
tls_min_version = 1.3              # 1.2 | 1.3
tls_ciphers     = TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

# Backend TLS (Keel → PostgreSQL)
backend_tls     = require
backend_tls_ca  = /etc/keel/backend-ca.crt

# Kernel TLS offload (kTLS) — Linux 4.13+
ktls            = on

Certificate Hot-Reload

Send SIGHUP to reload certificates without restarting:

sudo systemctl reload keel
# Or directly:
kill -HUP $(cat /var/run/keel/keel.pid)

# Verify via admin console:
psql -h localhost -p 9187 -c "SHOW CERTIFICATES"
psql -h localhost -p 9187 -c "RELOAD CERTS"

Cloud IAM Authentication

Keel handles cloud-native authentication token generation and rotation automatically. No external scripts needed.

AWS RDS IAM

[worker_group.main.servers]
primary = host=mydb.cluster-xxx.us-east-1.rds.amazonaws.com port=5432 \
          dbname=app user=app role=RW \
          auth=aws_iam region=us-east-1

# IAM tokens cached for 14 minutes and auto-refreshed.

GCP Cloud SQL IAM

[worker_group.main.servers]
primary = host=/cloudsql/project:region:instance port=5432 \
          dbname=app user=app@project.iam role=RW auth=gcp_iam

Azure AD / Entra ID

[worker_group.main.servers]
primary = host=myserver.postgres.database.azure.com port=5432 \
          dbname=app user=app@myserver role=RW auth=azure_ad

Prometheus Metrics

Metrics are exposed at GET /metrics on the admin port (default: 9187).

keel_connections_active{role="client"}         # active client sessions
keel_connections_active{role="backend"}        # active backend connections
keel_pool_size{worker="0",state="idle"}        # idle pool entries
keel_pool_size{worker="0",state="active"}      # borrowed pool entries
keel_pool_borrows_total                        # total pool borrows
keel_pool_wait_queue_enqueued                  # clients waiting for a connection
keel_queries_total{type="read"}                # total read queries
keel_queries_total{type="write"}               # total write queries
keel_query_duration_seconds{quantile="0.99"}   # P99 query latency
keel_tls_handshakes_total                      # TLS handshakes
keel_migrations_sent_total                     # session migrations sent
keel_router_scatter_merge_duration_seconds     # scatter-merge latency histogram

Grafana Dashboard

A pre-built Grafana dashboard is available at etc/grafana/keel-dashboard.json. Import it directly into Grafana.

Admin Console

Connect with any PostgreSQL client to the admin port:

psql -h localhost -p 9187

SHOW POOL;          -- per-worker pool status
SHOW WORKERS;       -- worker statistics
SHOW REBALANCE;     -- rebalance metrics
SHOW STATS;         -- aggregate counters
SHOW SHARD RULES;   -- shard routing rules
SHOW CERTIFICATES;  -- loaded TLS certs
EXPLAIN SHARD PLAN FOR 'SELECT ...';  -- routing plan
RELOAD CERTS;       -- hot-reload TLS certs

OpenTelemetry Tracing

Keel injects W3C traceparent headers as SQL block comments and exports spans via OTLP/HTTP.

[worker_group.main]
tracing           = on
trace_sample_rate = 0.1              # 10% head-based sampling
otlp_endpoint     = http://otel-collector:4318/v1/traces

Each traced query will appear in the PostgreSQL log as:

/* traceparent=00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 */
SELECT * FROM users WHERE id = $1

Logging

Query Logging

[worker_group.main]
query_log        = on
query_log_mode   = all    # all | read | write | none
query_log_file   = /var/log/keel/queries.log

Audit Logging

[audit]
enabled       = true
events        = auth,admin,query     # auth | admin | query | pool
output        = file
file          = /var/log/keel/audit.log
format        = json

Hook System

Available in full mode. Four hook points in the query pipeline:

Hooks can return true (continue) or false (abort). Multiple hooks at the same point execute in priority order. Lua, Python, and native .so hooks can be mixed at any point.

Lua & Python Hooks

Lua Hook Example

-- Route queries from 'analytics' user to replica
function before_route(ctx)
    if ctx.user == "analytics" then
        ctx.route = "replica"
    end
    return true  -- continue
end

[worker_group.main]
mode = full
hook.before_route = /etc/keel/hooks/route.lua

Python Hook Example

def after_query_parse(ctx):
    # Block DROP TABLE in production
    if ctx.query_type == "DDL" and "DROP TABLE" in ctx.sql.upper():
        ctx.abort_message = "DROP TABLE is not allowed"
        return False
    return True

Hot Reload

Send SIGHUP to reload configuration without restarting:

sudo systemctl reload keel
# Or:
kill -HUP $(pgrep keel)

What Can Be Hot-Reloaded

• Pool sizes (min_pool_size, max_pool_size)
• Timeouts and probe settings
• Server weights
• TLS certificates (atomic SSL_CTX swap)
• Log level and format
• Shard rules
• Query rules and throttle rules
• Audit configuration
• Rebalancing configuration

Multi-Proxy HA Cluster

Two or three Keel instances can form a cluster with heartbeat-based peer monitoring and configuration gossip. Each node runs independently — the cluster layer propagates config changes and detects peer failures without affecting the query hot path.

[cluster]
enabled              = true
node_id              = keel-0
gossip_port          = 7000
peers                = keel-1.local:7000,keel-2.local:7000
heartbeat_interval   = 2s
compression          = zstd   # zlib | zstd | none (for WAN links)

Kubernetes

Helm Chart

helm install keel ./helm/keel \
  --set image.tag=latest \
  --set config.bindPort=6432 \
  --set config.numWorkers=4 \
  --set config.maxPoolSize=50

Environment Variables

All INI settings can be overridden with KEEL_* environment variables:

env:
  - name: KEEL_BIND_PORT
    value: "6432"
  - name: KEEL_NUM_WORKERS
    value: "4"
  - name: KEEL_MAX_POOL_SIZE
    value: "50"
  - name: KEEL_MODE
    value: "pool"
  - name: KEEL_SERVER_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-credentials
        key: password

Health Endpoints

livenessProbe:
  httpGet:
    path: /healthz
    port: 9187
  initialDelaySeconds: 5
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /ready
    port: 9187

HPA Scaling

Scale based on keel_pool_wait_queue_enqueued metric — clients waiting for a connection is the best signal that the proxy is under load.

Docker

services:
  keel:
    image: ghcr.io/virtlabs-io/keel:latest
    ports:
      - "6432:6432"
      - "9187:9187"
    volumes:
      - ./keel.ini:/etc/keel/keel.ini:ro
    environment:
      KEEL_LOG_LEVEL: info
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9187/healthz"]
      interval: 10s
      timeout: 3s
      retries: 3

Multi-arch images are available for linux/amd64 and linux/arm64.

Health & Failover

Probe Types

Graceful Drain

Lifecycle: CREATED → ACTIVE → DRAINING → STOPPING → STOPPED. During DRAINING, Keel rejects new connections with PostgreSQL error 57P03 (cannot_connect_now) and waits for in-flight transactions to complete before shutting down backends.