Implementing GPT-2 in Pure Postgres

Let's build an LLM in Postgres.

Note: This post is only lightly edited. The "oh but make it public when it's ready" sirens are out in full force, and today we will pay them no mind.

OK, so what's the problem

Pasted image 20260712144410.png

Up to us, really. I think the virtue we're aiming for today is not "don't reinvent the wheel," it's "vulcanize the rubber":

Reinvent the wheel as much as possible.
Learn and understand what we're doing.
Use as many Postgres hacks and optimizations as possible. Portability is a non-goal: something that works in a different version of Postgres, and especially a different database engine, is not something we will think about.
"Only Invented Here:" Minimize PyTorch, CUDA, or HuggingFace. SQL as much as possible, and Python when we need it.
Write this post with minimal use of the backspace or delete key. Using devtee for that.

Foundational Concepts

Matrix Multiplication in SQL

SELECT A.row_idx, B.col_idx, SUM(A.val * B.val)
FROM matrix_a A JOIN matrix_b B ON A.col_idx = B.row_idx
GROUP BY A.row_idx, B.col_idx;

softmax in SQL

SELECT
    token_id,
    EXP(logit) / SUM(EXP(logit))
-- OVER (), getting unhinged already
OVER () as prob FROM logits;

Optimizations:

Postgres 18 performance tuning: async I/O means the query planner won't choke on the attention heads.

What are we measuring?

tokens/second, tuples/second?
maybe (quality and/or evals) but probably not in this revision.

pgvector

OK, I actually don't mind using pgvector. That's close enough to a builtin and we don't need to be that hardcore today.

Pasted image 20260712115255.png

pgvector lets us hijack hardware-accelerated SIMD instructions directly inside the query engine.

CREATE EXTENSION IF NOT EXISTS vector;

-- token embeddings
CREATE TABLE token_embeddings (
    token_id INTEGER PRIMARY KEY,
    vec vector(768)
);

CREATE TABLE layer_weights (
    layer_idx INTEGER,
    -- e.g., 'attn.c_attn.weight', 'mlp.c_fc.weight'
    tensor_name TEXT, 
    -- Maps to the output neuron / feature dimension
    row_idx INTEGER,  
    -- For handling the 3072 MLP layers in 768 chunks
    chunk_idx INTEGER DEFAULT 0,
    vec vector(768),
    PRIMARY KEY (layer_idx, tensor_name, row_idx, chunk_idx)
);

-- KV Cache: Stored as native vectors per token position
CREATE TABLE kv_cache (
    session_id UUID,
    seq_pos INTEGER,
    layer_idx INTEGER,
    tensor_name CHAR(1), -- 'K' or 'V'
    head_idx INTEGER,
    -- GPT-2 Small has 12 heads: 768 / 12 = 64 dim per head
    vec vector(64),
    PRIMARY KEY (session_id, seq_pos, layer_idx, tensor_name, head_idx)
);

set up our setup

All the code is at omarish/llm-sql.

Hardware-Accelerated Matrix Multiplication in SQL

<#> returns the negative inner product between two vectors (Postgres indexes look for the smallest distance), so multiplying by -1 gives us the actual dot product.

If we have a current activation state vector stored in a temporary table current_state(vec vector(768)), multiplying it by a weight matrix to project it to the next layer is a single, beautiful query:

SELECT 
    w.row_idx,
    (c.vec <#> w.vec) * -1 AS activation
FROM current_state c
CROSS JOIN layer_weights w
WHERE w.layer_idx = 0 AND w.tensor_name = 'attn.c_proj.weight'
ORDER BY w.row_idx;

Load the weights

\copy token_embeddings FROM 'token_embeddings_vector.csv' WITH CSV;
\copy layer_weights FROM 'layer_weights_vector.csv' WITH CSV;

ALTER SYSTEM SET io_method = 'io_uring';
SELECT pg_reload_conf();

We're running postgres@18 via Docker right now (in retrospect I should have just run it natively), but with some Docker tweaks we can get io_uring working.

    command:
      - "postgres"
      # --- Postgres 18 Async I/O (io_uring) ---
      - "-c"
      - "io_method=io_uring"
      # --- Durability OFF: fastest possible writes, but a crash can corrupt
      #     the DB. Perfectly fine for a disposable local toy. ---
      - "-c"
      - "fsync=off"
      - "-c"
      - "synchronous_commit=off"
      - "-c"
      - "full_page_writes=off"

This PR would not get approved at most reputable companies.

Activation Layer.sql (LayerNorm)

Every Transformer block begins with a LayerNorm pass. It takes an input vector $x$ , centers it around its mean $\mu$ , divides by its standard deviation $\sigma$ , and scales/shifts using the learned parameters $\gamma$ (gamma) and $\beta$ (beta):

\text{LN}(x) = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} \odot \gamma + \beta

Since pgvector doesn't provide a built-in element-wise variance or standard deviation operation across a single vector's internal dimensions, we must resort to relational mechanics:

Explode: Cast the vector(768) to a real[] array and blast it into 768 individual table rows using UNNEST() WITH ORDINALITY (which gives us the precise array index for free).
Aggregate: Use standard SQL window functions, AVG(val) OVER () and VAR_SAMP(val) OVER (), to capture the statistical structure of the activation vector across those rows.
Join: Pull the matching $\gamma$ and $\beta$ parameters from our layer_weights table by matching the array index.
Implode: Compute the normalized value, order everything by the original position index, and re-pack the rows back into an optimized vector(768) type using array_agg()::vector.

We wrap the whole operation in a PL/pgSQL function:

CREATE OR REPLACE FUNCTION layernorm(
    p_layer_idx INT,
    p_tensor_prefix TEXT, -- e.g., 'ln_1' or 'ln_2'
    p_vec vector(768)
) RETURNS vector(768) AS $$
DECLARE
    v_result vector(768);
BEGIN
    WITH unrolled AS (
        SELECT 
            idx,
            val,
            AVG(val) OVER () as mu,
            COALESCE(VAR_SAMP(val) OVER (), 0) as variance
        FROM UNNEST(p_vec::real[]) WITH ORDINALITY AS u(val, idx)
    ),
    gamma AS (
        SELECT idx, val 
        FROM layer_weights, UNNEST(vec::real[]) WITH ORDINALITY AS u(val, idx)
        WHERE layer_idx = p_layer_idx AND tensor_name = p_tensor_prefix || '.weight' AND row_idx = 0
    ),
    beta AS (
        SELECT idx, val 
        FROM layer_weights, UNNEST(vec::real[]) WITH ORDINALITY AS u(val, idx)
        WHERE layer_idx = p_layer_idx AND tensor_name = p_tensor_prefix || '.bias' AND row_idx = 0
    )
    SELECT array_agg(((u.val - u.mu) / SQRT(u.variance + 1e-5) * g.val) + b.val ORDER BY u.idx)::vector(768)
    INTO v_result
    FROM unrolled u
    JOIN gamma g ON u.idx = g.idx
    JOIN beta b ON u.idx = b.idx;

    RETURN v_result;
END;
$$ LANGUAGE plpgsql STABLE;

Multi-head attention mechanism `.sql`

By the way, I know I could do this a lot faster with a better prompt, but that's not the goal (again, see lulz motive and rabbit holes).

To calculate Attention ( $Q, K, V$ ), we will cross-join our normalized hidden state against the attention weight matrices, split the resulting 768-dimensional vector into twelve 64-dimensional heads, compute the causal attention dot-product matrix using the <#> operator against previous states in the kv_cache, apply a window-based causal mask, execute our SQL Softmax, and combine it back together.

-- Helper function to split a concatenated vector into Q, K, V and heads
CREATE OR REPLACE FUNCTION compute_attention(
    p_session_id UUID,
    p_seq_pos INT,
    p_layer_idx INT,
    p_x vector(768)
) RETURNS vector(768) AS $$
DECLARE
    v_out vector(768);
BEGIN
    -- This CTE chain executes the entire self-attention mechanism for a single token
    WITH 
    -- 1. Project Q, K, V (Note: GPT-2 Attention weight is 768 x 2304)
    -- We assume the python script chunked the 2304 into 3 logical blocks (Q=0, K=1, V=2) (note to self this might cause headaches)
    -- For brevity in this post, we assume a unified matrix multiply that returns 3 vectors.
    -- Here we simulate the projection and head splitting.
    qkv_proj AS (
        SELECT 
            chunk_idx as qkv_type, -- 0 for Q, 1 for K, 2 for V
            row_idx,
            -- pgvector dot product is <#>, but it returns negative, so we multiply by -1
            ((p_x <#> vec) * -1) as val
        FROM layer_weights
        WHERE layer_idx = p_layer_idx AND tensor_name = 'attn.c_attn.weight'
    ),
    -- 2. Add Biases
    qkv_biased AS (
        SELECT 
            p.qkv_type,
            p.row_idx,
            p.val + b.val as val
        FROM qkv_proj p
        JOIN (
            SELECT chunk_idx as qkv_type, row_idx, val 
            FROM layer_weights 
            WHERE layer_idx = p_layer_idx AND tensor_name = 'attn.c_attn.bias'
        ) b ON p.qkv_type = b.qkv_type AND p.row_idx = b.row_idx
    ),
    -- 3. Split into 12 heads of 64 dimensions each
    heads AS (
        SELECT 
            qkv_type,
            (row_idx / 64) as head_idx,
            array_agg(val ORDER BY row_idx)::vector(64) as head_vec
        FROM qkv_biased
        GROUP BY qkv_type, (row_idx / 64)
    ),
    -- 4. Cache K and V for future tokens (and current token)
    cache_kv AS (
        INSERT INTO kv_cache (session_id, seq_pos, layer_idx, tensor_name, head_idx, vec)
        SELECT p_session_id, p_seq_pos, p_layer_idx, 
               CASE WHEN qkv_type = 1 THEN 'K' ELSE 'V' END, 
               head_idx, head_vec
        FROM heads
        WHERE qkv_type IN (1, 2)
        RETURNING *
    ),
    -- 5. Calculate Attention Scores (Q * K^T / sqrt(d_k))
    attention_scores AS (
        SELECT 
            q.head_idx,
            k.seq_pos as cache_pos,
            -- Dot product between Current Q and Cached K, scaled by 1/8 (sqrt(64))
            ((q.head_vec <#> k.vec) * -1) / 8.0 as raw_score
        FROM (SELECT head_idx, head_vec FROM heads WHERE qkv_type = 0) q
        JOIN kv_cache k ON k.session_id = p_session_id 
                        AND k.layer_idx = p_layer_idx 
                        AND k.tensor_name = 'K'
                        AND k.head_idx = q.head_idx
                        AND k.seq_pos <= p_seq_pos -- IMPLICIT CAUSAL MASK!
    ),
    -- 6. Softmax via SQL Window Functions
    softmax AS (
        SELECT 
            head_idx,
            cache_pos,
            EXP(raw_score) / SUM(EXP(raw_score)) OVER (PARTITION BY head_idx) as prob
        FROM attention_scores
    ),
    -- 7. Multiply Softmax by V and sum per head
    weighted_values AS (
        SELECT 
            s.head_idx,
            -- To multiply a vector by a scalar in standard SQL, we must unnest, multiply, and re-agg.
            -- For brevity, assuming a custom aggregate or unnest operation here:
            array_agg(
                SUM(s.prob * v_val.val)
            ORDER BY v_val.dim_idx)::vector(64) as context_head
        FROM softmax s
        JOIN kv_cache v ON v.session_id = p_session_id 
                        AND v.layer_idx = p_layer_idx 
                        AND v.tensor_name = 'V' 
                        AND v.head_idx = s.head_idx 
                        AND v.seq_pos = s.cache_pos
        -- Unnest the 64-dim V vector to perform scalar multiplication
        CROSS JOIN LATERAL UNNEST(v.vec::real[]) WITH ORDINALITY AS v_val(val, dim_idx)
        GROUP BY s.head_idx
    ),
    -- 8. Concatenate the 12 heads back into 768 dimensions
    concatenated AS (
        SELECT array_agg(val ORDER BY head_idx, dim_idx)::vector(768) as context_vec
        FROM weighted_values
        CROSS JOIN LATERAL UNNEST(context_head::real[]) WITH ORDINALITY AS u(val, dim_idx)
    ),
    -- 9. Final Output Projection (c_proj.weight)
    projected AS (
        SELECT array_agg(((c.context_vec <#> w.vec) * -1) + b.val ORDER BY w.row_idx)::vector(768) as final_vec
        FROM concatenated c
        CROSS JOIN layer_weights w
        JOIN (SELECT row_idx, val FROM layer_weights WHERE layer_idx = p_layer_idx AND tensor_name = 'attn.c_proj.bias' AND chunk_idx = 0) b ON w.row_idx = b.row_idx
        WHERE w.layer_idx = p_layer_idx AND w.tensor_name = 'attn.c_proj.weight' AND w.chunk_idx = 0
    )
    SELECT final_vec INTO v_out FROM projected;

    RETURN v_out;
END;
$$ LANGUAGE plpgsql;

Wait, there's one more thing: `mlp.sql`

CREATE OR REPLACE FUNCTION mlp(
    p_layer_idx INT,
    p_x vector(768)
) RETURNS vector(768) AS $$
DECLARE
    v_out vector(768);
BEGIN
    WITH 
    -- 1. Expand to 3072 dimensions (calculated as 4 chunks of 768 to fit our schema)
    expansion AS (
        SELECT 
            w.row_idx,
            ((p_x <#> w.vec) * -1) + b.val as val
        FROM layer_weights w
        JOIN layer_weights b ON w.layer_idx = b.layer_idx 
                            AND b.tensor_name = 'mlp.c_fc.bias' 
                            AND w.row_idx = b.col_idx -- Assuming col_idx was used for bias mapping in the python script
        WHERE w.layer_idx = p_layer_idx AND w.tensor_name = 'mlp.c_fc.weight'
        ORDER BY w.chunk_idx, w.row_idx
    ),
    -- 2. Apply GELU Activation Function
    -- GELU(x) = 0.5 * x * (1 + tanh(sqrt(2/pi) * (x + 0.044715 * x^3)))
    -- tanh(z) = (exp(2z) - 1) / (exp(2z) + 1)
    gelu AS (
        SELECT 
            row_idx,
            -- Calculate z = sqrt(2/pi) * (val + 0.044715 * val^3)
            (SQRT(2.0 / PI()) * (val + 0.044715 * POWER(val, 3))) as z,
            val as original_val
        FROM expansion
    ),
    gelu_applied AS (
        SELECT 
            row_idx,
            -- Calculate tanh(z) and apply to full formula
            0.5 * original_val * (1.0 + ( (EXP(2.0 * z) - 1.0) / NULLIF((EXP(2.0 * z) + 1.0), 0) )) as gelu_val
        FROM gelu
    ),
    -- 3. Repack into 3072-element array chunks (simulated here as a single aggregation for logic flow)
    -- and project back down to 768 dimensions.
    -- For pure SQL execution without custom types, we do a row-by-row scalar multiplication sum against the projection weights.
    projection AS (
        SELECT 
            w.row_idx as final_dim,
            SUM(g.gelu_val * ((w.vec::real[])[g.row_idx])) as val -- highly inefficient unnesting logic abstracted for standard SQL
        FROM gelu_applied g
        CROSS JOIN layer_weights w
        WHERE w.layer_idx = p_layer_idx AND w.tensor_name = 'mlp.c_proj.weight'
        GROUP BY w.row_idx
    ),
    final_biased AS (
        SELECT 
            p.final_dim,
            p.val + b.val as val
        FROM projection p
        JOIN layer_weights b ON b.layer_idx = p_layer_idx AND b.tensor_name = 'mlp.c_proj.bias' AND b.col_idx = p.final_dim
    )
    SELECT array_agg(val ORDER BY final_dim)::vector(768) INTO v_out FROM final_biased;

    RETURN v_out;
END;
$$ LANGUAGE plpgsql;

We now have the core pieces in place:

layernorm.sql
compute_attention.sql
mlp.sql

Next: the autoregressive generation loop

We need a PL/pgSQL function that takes an input string, looks up the token embeddings, passes them through all 12 layers, hits the language modeling head, picks the highest-probability token, and loops.

The name in the middle of that steering wheel should tell you that I was born ready, Shelby. Hit it

`ORDER BY (vec <#> hidden_state) LIMIT 1`

Because GPT-2 utilizes weight-tying, the weights of the LM Head are identical to the token embedding matrix we stored at the very beginning in our token_embeddings table. Therefore, projecting our final hidden state to find the absolute most likely next word doesn't require a new matrix or tensor operation. It is just a standard relational query using pgvector's inner product distance:

SELECT
    token_id
FROM
    token_embeddings
ORDER BY (vec <#> v_hidden_state) ASC LIMIT 1;

`generate.sql`

CREATE OR REPLACE FUNCTION generate_text(
    p_prompt_tokens INT[],
    p_max_new_tokens INT DEFAULT 20
) RETURNS TEXT AS $$
DECLARE
    v_session_id UUID := gen_random_uuid();
    v_current_tokens INT[] := p_prompt_tokens;
    v_next_token INT;
    v_x vector(768);
    v_attn_out vector(768);
    v_mlp_out vector(768);
    v_seq_pos INT;
    v_output_text TEXT := '';
BEGIN
    -- 1. Warm up the KV Cache with the seed prompt tokens
    FOR v_seq_pos IN 0 .. (cardinality(p_prompt_tokens) - 1) LOOP
        -- Extract the initial token embedding vector
        SELECT vec INTO v_x FROM token_embeddings WHERE token_id = p_prompt_tokens[v_seq_pos + 1];

        -- Run through the 12-layer Transformer stack to populate past keys/values
        FOR i IN 0 .. 11 LOOP
            -- LayerNorm 1 + Attention + Residual Connection
            v_attn_out := compute_attention(v_session_id, v_seq_pos, i, layernorm(i, 'ln_1', v_x));
            v_x := v_x + v_attn_out;

            -- LayerNorm 2 + MLP + Residual Connection
            v_mlp_out := mlp(i, layernorm(i, 'ln_2', v_x));
            v_x := v_x + v_mlp_out;
        END LOOP;
    END LOOP;

    -- 2. Autoregressive Generation Loop
    v_seq_pos := cardinality(p_prompt_tokens);

    FOR k IN 1 .. p_max_new_tokens LOOP
        -- Grab the embedding for the most recent token generated
        SELECT vec INTO v_x FROM token_embeddings WHERE token_id = v_current_tokens[cardinality(v_current_tokens)];

        -- Forward pass through all layers for the single new token step
        FOR i IN 0 .. 11 LOOP
            v_attn_out := compute_attention(v_session_id, v_seq_pos, i, layernorm(i, 'ln_1', v_x));
            v_x := v_x + v_attn_out;

            v_mlp_out := mlp(i, layernorm(i, 'ln_2', v_x));
            v_x := v_x + v_mlp_out;
        END LOOP;

        -- Final Global LayerNorm before vocabulary projection
        v_x := layernorm(11, 'ln_f', v_x);

        -- Language Modeling Head Projection: Find the single token with highest similarity
        SELECT token_id INTO v_next_token
        FROM token_embeddings
        ORDER BY (vec <#> v_x) ASC
        LIMIT 1;

        -- Append the token to our sliding window state
        v_current_tokens := array_append(v_current_tokens, v_next_token);
        v_output_text := v_output_text || ' ' || v_next_token::text;
        v_seq_pos := v_seq_pos + 1;
    END LOOP;

    -- Clear the session KV cache transaction log so our memory doesn't leak
    DELETE FROM kv_cache WHERE session_id = v_session_id;

    RETURN v_output_text;
END;
$$ LANGUAGE plpgsql;

CREATE TABLE
==> Reloading token_embeddings (--reload)
TRUNCATE TABLE
COPY 50257
==> Reloading position_embeddings (--reload)
TRUNCATE TABLE
COPY 1024
==> Reloading layer_weights (--reload)
TRUNCATE TABLE
COPY 110750
==> Indexing + analyzing
NOTICE: relation "idx_layer_weights_lookup" already exists, skipping
CREATE INDEX
ANALYZE
ANALYZE
ANALYZE
==> Installing functions
- attention.sql
CREATE FUNCTION
- generate.sql
CREATE FUNCTION
- layernorm.sql
CREATE FUNCTION
- mlp.sql
CREATE FUNCTION
==> Done.

Signs of Life

Postgres is great b/c if something is broken it lets you know right away. So if it actually runs, that's no guarantee you've done it perfectly, but it's "right enough."

Pasted image 20260712130807.png

Then, the tokenizer

To pull this off without cheating and leaving the database boundaries, we need an in-database string encoder and decoder.

GPT-2's vocabulary consists of 50,257 tokens, where spaces are represented by the special character Ġ and newlines by Ċ. To avoid rewriting a full Byte-Pair Encoding (BPE) merge-rule tree in procedural SQL, we will implement a Maximum-Match Greedy Tokenizer directly in PL/pgSQL. It replaces standard spaces with Ġ, looks up the longest matching prefixes available in our vocabulary table, consumes them recursively, and passes the resulting token array straight into our inference loop.

Why are my CPU fans spinning

So, that wasn't running anywhere near fast enough.

The big speedup (mlp.sql + mlp_cproj_wide):

The bottleneck was the MLP down-projection: it unnested every c_proj.weight row (~2.4M rows/layer) and hash-joined against the activations. That was the dominant cost across 12 layers × every token.
Now load.py precomputes each output dim's full 3072-wide weight vector once into mlp_cproj_wide. mlp() assembles the GELU activations into one vector(3072) and does 768 pgvector dot products per layer instead. That's the single biggest lever, and it moves the heavy math into pgvector's SIMD path.

Why it was slow / other levers you can pull:

Each generated token = a full 12-layer forward pass, and each layer does real matmuls in SQL. It's inherently O(tokens × layers × d²).
Fewer tokens is the most direct knob: generate_text(ARRAY[...], 3) runs ~3× faster than 10. Good for a first end-to-end smoke test.
The up-projection (c_fc, 3072 dot products/layer) and attention are already pgvector-based and comparatively cheap, so I left them.
The LM head scans all 50,257 token embeddings per generated token (exact argmax). That's fast (one <#> sweep), but if you want it faster you could add an HNSW index on token_embeddings(vec) for approximate nearest-neighbor, at the cost of exactness.
The Postgres tuning we set earlier (durability off, big work_mem/shared_buffers) already helps these aggregate-heavy queries.

It's Starting to Work

postgres=# \timing on
Timing is on.
postgres=# SELECT 'Hello, I' || generate_text(ARRAY[15496, 11, 314], 20) AS story;
                              story
-----------------------------------------------------------------
 Hello, I'm sorry, I'm sorry. I'm sorry. I'm sorry. I'm sorry. I
(1 row)

Time: 27628.671 ms (00:27.629)
postgres=#

"the meaning of life is"

ARRAY[464, 3616, 286, 1204, 318]

Pasted image 20260712133452.png

postgres=# SELECT generate_text(ARRAY[464, 3616, 286, 1204, 318], 5);
      generate_text
--------------------------                                                  
not to be confused with
(1 row)

Time: 17125.167 ms (00:17.125)
postgres=# SELECT generate_text(ARRAY[464, 3616, 286, 1204, 318], 10);
                 generate_text
-----------------------------------------------                             
not to be confused with the meaning of life.
(1 row)

Time: 18001.460 ms (00:18.001)
postgres=# SELECT generate_text(ARRAY[464, 3616, 286, 1204, 318], 12);
                    generate_text
-----------------------------------------------------
  not to be confused with the meaning of life. It is                      
(1 row)

Time: 20727.452 ms (00:20.727)

We did it

Running GPT-2's BPE encoder in SQL

Pasted image 20260712134939.png

ARRAY[464, 3616, 286, 1204, 318]

Instead of having to look up token ids by hand before running the query, let's implement the vocab lookup in Postgres as well:

-- Convenience wrapper: text in, text out (prompt + continuation) in one call.
-- End-to-end GPT-2 in a single SQL expression:  SELECT complete('Hello, I', 10);
CREATE OR REPLACE FUNCTION complete(p_prompt TEXT, p_max_new_tokens INT DEFAULT 10)
RETURNS TEXT AS $$
    SELECT p_prompt || generate_text(gpt2_encode(p_prompt), p_max_new_tokens);
$$ LANGUAGE sql;

You know what's cool:

SELECT
    prompt,
    complete(prompt, 8)
FROM (
    VALUES 
        ('Hello, I'),
        ('In the beginning')
    )
AS t(prompt);

postgres=# SELECT prompt, complete(prompt, 8)
FROM (
    VALUES ('Hello, we just implemented an LLM in postgres, which means')) 
AS t(prompt);

OK, this is not going to finish quickly, because:

Every token (all 13 prompt tokens during warmup + 8 generated) runs a full 12-layer forward pass.
That's (13 + 8) × 12 ≈ 252 layer passes, and each layer pass fires several SQL statements (2 LayerNorms, attention's insert+read+delete, one big MLP query).

So you're executing ~1,500+ SQL statements, and inside them roughly 2+ million pgvector dot products. In an interpreted plpgsql loop over a row-oriented engine, that lands in the tens-of-seconds range. Expected for "GPT-2 in pure SQL."

Pasted image 20260712140358.png

postgres=# SELECT prompt, complete(prompt, 20)
FROM (VALUES ('LLM.sql because')) AS t(prompt);
     prompt      |                            complete
-----------------+----------------------------------------------------------------
 LLM.sql because | LLM.sql because it is not supported by the database.          +
                 |                                                               +
                 | The following table lists the SQL statements that are executed
(1 row)

Time: 31586.728 ms (00:31.587)

postgres=# SELECT prompt, complete(prompt, 20)
FROM (VALUES ('LLM.sql great idea because')) AS t(prompt);
           prompt           |                           complete
----------------------------+--------------------------------------------------------------
 LLM.sql great idea because | LLM.sql great idea because of the way it works.             +
                            |                                                             +
                            | The problem is that the SQL Server database is not very good
(1 row)

Time: 178264.059 ms (02:58.264)

So, I knew this before, but in the spirit of "vulcanize the rubber" we now know empirically that compute time scales more with input tokens than output tokens.

Let's wrap this up for now

It's tempting to go deep on improvements from here: scaling properties, making it faster, evals. There is a lot to do.

Also, I'm going to leave this post mostly as it is, and maybe polish it more in the future.

The "oh but make it public when it's ready" sirens are out in their finest sundresses today, and we will pay them no mind.

We did the thing we set out to do.

https://github.com/omarish/llm-sql

🚢 ✅

<|endoftext|>

Omar Bohsali