LLM.sql

Let's build an LLM in Postgres.

Note: This post is not edited. The "oh but make it public when it's ready" sirens are out in full force, and today we will pay them no mind. Also, a few LLMs wrote most of the SQL here.

OK, so what's the problem

Pasted image 20260712144410.png

Up to us, really. I think the virtue we're aiming for today is "not not reinvent the wheel; vulcanize rubber"

reinvent the wheel as much as possible.
learn and understand what we're doing
use as many postgres hacks and optimizations as possible
No pytorch, CUDA, huggingface. SQL as much as possible, and python when we need to take the easy way out.
max(lulz)

Some Guiding Principles

Write this post with minimal use of backspace or delete key. Using devtee for that.

Foundational things

Matrix Multiplication in SQL

SELECT A.row_idx, B.col_idx, SUM(A.val * B.val)
FROM matrix_a A JOIN matrix_b B ON A.col_idx = B.row_idx
GROUP BY A.row_idx, B.col_idx;

softmax in SQL

SELECT
    token_id,
    EXP(logit) / SUM(EXP(logit))
-- OVER (), getting unhinged already
OVER () as prob FROM logits;

Other principles we need to know:

Optimizations:

pg18 performance tuning - async io in postgres 18 -> query planner won't choke on the attention heads.

What are we measuring:

tokens/second, tuples/second?
maybe (quality and/or evals) but probably not in this revision.
let's start small:
- does our SQL actually work (aka no syntax errors)
- lulz/second

pgvector

OK, I actually don't mind using pgvector. That's close enough to a builtin and we don't need to be that hardcore today.

Pasted image 20260712115255.png

pgvector lets us hijack hardware-accelerated SIMD instructions directly inside the query engine.

CREATE EXTENSION IF NOT EXISTS vector;

-- token embeddings
CREATE TABLE token_embeddings (
    token_id INTEGER PRIMARY KEY,
    vec vector(768)
);

CREATE TABLE layer_weights (
    layer_idx INTEGER,
    -- e.g., 'attn.c_attn.weight', 'mlp.c_fc.weight'
    tensor_name TEXT, 
    -- Maps to the output neuron / feature dimension
    row_idx INTEGER,  
    -- For handling the 3072 MLP layers in 768 chunks
    chunk_idx INTEGER DEFAULT 0,
    vec vector(768),
    PRIMARY KEY (layer_idx, tensor_name, row_idx, chunk_idx)
);

-- 3. KV Cache: Stored as native vectors per token position
CREATE TABLE kv_cache (
    session_id UUID,
    seq_pos INTEGER,
    layer_idx INTEGER,
    tensor_name CHAR(1), -- 'K' or 'V'
    head_idx INTEGER,
    -- GPT-2 Small has 12 heads: 768 / 12 = 64 dim per head
    vec vector(64),
    PRIMARY KEY (session_id, seq_pos, layer_idx, tensor_name, head_idx)
);

set up our setup

https://github.com/omarish/llm-sql

Hardware-Accelerated Matrix Multiplication in SQL

<#> calculates the negative inner product between two vectors since pg indexes look for the smallest distance, so multiplying by -1 gives us the actual result.

If we have a current activation state vector stored in a temporary table current_state(vec vector(768)), multiplying it by a weight matrix to project it to the next layer is a single, beautiful query:

SELECT 
    w.row_idx,
    (c.vec <#> w.vec) * -1 AS activation
FROM current_state c
CROSS JOIN layer_weights w
WHERE w.layer_idx = 0 AND w.tensor_name = 'attn.c_proj.weight'
ORDER BY w.row_idx;

Load the weights

\copy token_embeddings FROM 'token_embeddings_vector.csv' WITH CSV;
\copy layer_weights FROM 'layer_weights_vector.csv' WITH CSV;

ALTER SYSTEM SET io_method = 'io_uring';
SELECT pg_reload_conf();

We're running postgres@18 via docker right now (in retrospect should have just run it natively), but with some docker tweaks we can get io_uring working.

    command:
      - "postgres"
      # --- Postgres 18 Async I/O (io_uring) ---
      - "-c"
      - "io_method=io_uring"
      # --- Durability OFF: fastest possible writes, but a crash can corrupt
      #     the DB. Perfectly fine for a disposable local toy. ---
      - "-c"
      - "fsync=off"
      - "-c"
      - "synchronous_commit=off"
      - "-c"
      - "full_page_writes=off"

This PR would not get approved at most reputable companies.

Activation Layer.sql (LayerNorm)

A standard LayerNorm scales an activation vector $x$ across its hidden dimension $d$ using a learned weight ( $\gamma$ ) and bias ( $\beta$ ):

\text{LN}(x) = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} \odot \gamma + \beta

Because pgvector doesn't have a native variance aggregation function, we have to unroll the vector array components into temporary rows, perform the standard SQL window calculations for AVG() and VAR_SAMP(), apply the scaling math, and then pack it back into an optimized vector(768) row using array aggregations.

Every Transformer block begins with a LayerNorm pass. Mathematically, it takes an input vector $x$ , centers it around its mean $\mu$ , divides it by its standard deviation $\sigma$ , and scales/shifts it using the learned parameters $\gamma$ (gamma) and $\beta$ (beta):

Since pgvector doesn't provide a built-in element-wise variance or standard deviation operation across a single vector's internal dimensions, we must resort to relational mechanics:

Explode: Cast the vector(768) to a real[] array and blast it into 768 individual table rows using UNNEST() WITH ORDINALITY (which gives us the precise array index for free).
Aggregate: Use standard SQL window functions—AVG(val) OVER () and VAR_SAMP(val) OVER ()—to capture the statistical structure of the activation vector across those rows.
Join: Pull the matching $\gamma$ and $\beta$ parameters from our layer_weights table by matching the array index.
Implode: Compute the normalized value, order everything by the original position index, and re-pack the rows back into an optimized vector(768) type using array_agg()::vector.

We encapsulate this entire operation inside a clean, high-performance PL/pgSQL function:

CREATE OR REPLACE FUNCTION layernorm(
    p_layer_idx INT,
    p_tensor_prefix TEXT, -- e.g., 'ln_1' or 'ln_2'
    p_vec vector(768)
) RETURNS vector(768) AS $$
DECLARE
    v_result vector(768);
BEGIN
    WITH unrolled AS (
        SELECT 
            idx,
            val,
            AVG(val) OVER () as mu,
            COALESCE(VAR_SAMP(val) OVER (), 0) as variance
        FROM UNNEST(p_vec::real[]) WITH ORDINALITY AS u(val, idx)
    ),
    gamma AS (
        SELECT idx, val 
        FROM layer_weights, UNNEST(vec::real[]) WITH ORDINALITY AS u(val, idx)
        WHERE layer_idx = p_layer_idx AND tensor_name = p_tensor_prefix || '.weight' AND row_idx = 0
    ),
    beta AS (
        SELECT idx, val 
        FROM layer_weights, UNNEST(vec::real[]) WITH ORDINALITY AS u(val, idx)
        WHERE layer_idx = p_layer_idx AND tensor_name = p_tensor_prefix || '.bias' AND row_idx = 0
    )
    SELECT array_agg(((u.val - u.mu) / SQRT(u.variance + 1e-5) * g.val) + b.val ORDER BY u.idx)::vector(768)
    INTO v_result
    FROM unrolled u
    JOIN gamma g ON u.idx = g.idx
    JOIN beta b ON u.idx = b.idx;

    RETURN v_result;
END;
$$ LANGUAGE plpgsql STABLE;

Multi-head attention mechanism `.sql`

By the way, I know I could do this a lot faster with a better prompt, but that's not the goal (again, see lulz motive and rabbit holes)

To calculate Attention ( $Q, K, V$ ), we will cross-join our normalized hidden state against the attention weight matrices, split the resulting 768-dimensional vector into twelve 64-dimensional heads, compute the causal attention dot-product matrix using the <#> operator against previous states in the kv_cache, apply a window-based causal mask, execute our SQL Softmax, and combine it back together.

-- Helper function to split a concatenated vector into Q, K, V and heads
CREATE OR REPLACE FUNCTION compute_attention(
    p_session_id UUID,
    p_seq_pos INT,
    p_layer_idx INT,
    p_x vector(768)
) RETURNS vector(768) AS $$
DECLARE
    v_out vector(768);
BEGIN
    -- This CTE chain executes the entire self-attention mechanism for a single token
    WITH 
    -- 1. Project Q, K, V (Note: GPT-2 Attention weight is 768 x 2304)
    -- We assume the python script chunked the 2304 into 3 logical blocks (Q=0, K=1, V=2) (note to self this might cause headaches)
    -- For brevity in this post, we assume a unified matrix multiply that returns 3 vectors.
    -- Here we simulate the projection and head splitting.
    qkv_proj AS (
        SELECT 
            chunk_idx as qkv_type, -- 0 for Q, 1 for K, 2 for V
            row_idx,
            -- pgvector dot product is <#>, but it returns negative, so we multiply by -1
            ((p_x <#> vec) * -1) as val
        FROM layer_weights
        WHERE layer_idx = p_layer_idx AND tensor_name = 'attn.c_attn.weight'
    ),
    -- 2. Add Biases
    qkv_biased AS (
        SELECT 
            p.qkv_type,
            p.row_idx,
            p.val + b.val as val
        FROM qkv_proj p
        JOIN (
            SELECT chunk_idx as qkv_type, row_idx, val 
            FROM layer_weights 
            WHERE layer_idx = p_layer_idx AND tensor_name = 'attn.c_attn.bias'
        ) b ON p.qkv_type = b.qkv_type AND p.row_idx = b.row_idx
    ),
    -- 3. Split into 12 heads of 64 dimensions each
    heads AS (
        SELECT 
            qkv_type,
            (row_idx / 64) as head_idx,
            array_agg(val ORDER BY row_idx)::vector(64) as head_vec
        FROM qkv_biased
        GROUP BY qkv_type, (row_idx / 64)
    ),
    -- 4. Cache K and V for future tokens (and current token)
    cache_kv AS (
        INSERT INTO kv_cache (session_id, seq_pos, layer_idx, tensor_name, head_idx, vec)
        SELECT p_session_id, p_seq_pos, p_layer_idx, 
               CASE WHEN qkv_type = 1 THEN 'K' ELSE 'V' END, 
               head_idx, head_vec
        FROM heads
        WHERE qkv_type IN (1, 2)
        RETURNING *
    ),
    -- 5. Calculate Attention Scores (Q * K^T / sqrt(d_k))
    attention_scores AS (
        SELECT 
            q.head_idx,
            k.seq_pos as cache_pos,
            -- Dot product between Current Q and Cached K, scaled by 1/8 (sqrt(64))
            ((q.head_vec <#> k.vec) * -1) / 8.0 as raw_score
        FROM (SELECT head_idx, head_vec FROM heads WHERE qkv_type = 0) q
        JOIN kv_cache k ON k.session_id = p_session_id 
                        AND k.layer_idx = p_layer_idx 
                        AND k.tensor_name = 'K'
                        AND k.head_idx = q.head_idx
                        AND k.seq_pos <= p_seq_pos -- IMPLICIT CAUSAL MASK!
    ),
    -- 6. Softmax via SQL Window Functions
    softmax AS (
        SELECT 
            head_idx,
            cache_pos,
            EXP(raw_score) / SUM(EXP(raw_score)) OVER (PARTITION BY head_idx) as prob
        FROM attention_scores
    ),
    -- 7. Multiply Softmax by V and sum per head
    weighted_values AS (
        SELECT 
            s.head_idx,
            -- To multiply a vector by a scalar in standard SQL, we must unnest, multiply, and re-agg.
            -- For brevity, assuming a custom aggregate or unnest operation here:
            array_agg(
                SUM(s.prob * v_val.val)
            ORDER BY v_val.dim_idx)::vector(64) as context_head
        FROM softmax s
        JOIN kv_cache v ON v.session_id = p_session_id 
                        AND v.layer_idx = p_layer_idx 
                        AND v.tensor_name = 'V' 
                        AND v.head_idx = s.head_idx 
                        AND v.seq_pos = s.cache_pos
        -- Unnest the 64-dim V vector to perform scalar multiplication
        CROSS JOIN LATERAL UNNEST(v.vec::real[]) WITH ORDINALITY AS v_val(val, dim_idx)
        GROUP BY s.head_idx
    ),
    -- 8. Concatenate the 12 heads back into 768 dimensions
    concatenated AS (
        SELECT array_agg(val ORDER BY head_idx, dim_idx)::vector(768) as context_vec
        FROM weighted_values
        CROSS JOIN LATERAL UNNEST(context_head::real[]) WITH ORDINALITY AS u(val, dim_idx)
    ),
    -- 9. Final Output Projection (c_proj.weight)
    projected AS (
        SELECT array_agg(((c.context_vec <#> w.vec) * -1) + b.val ORDER BY w.row_idx)::vector(768) as final_vec
        FROM concatenated c
        CROSS JOIN layer_weights w
        JOIN (SELECT row_idx, val FROM layer_weights WHERE layer_idx = p_layer_idx AND tensor_name = 'attn.c_proj.bias' AND chunk_idx = 0) b ON w.row_idx = b.row_idx
        WHERE w.layer_idx = p_layer_idx AND w.tensor_name = 'attn.c_proj.weight' AND w.chunk_idx = 0
    )
    SELECT final_vec INTO v_out FROM projected;

    RETURN v_out;
END;
$$ LANGUAGE plpgsql;

one more thing -- `mlp.sql`

CREATE OR REPLACE FUNCTION mlp(
    p_layer_idx INT,
    p_x vector(768)
) RETURNS vector(768) AS $$
DECLARE
    v_out vector(768);
BEGIN
    WITH 
    -- 1. Expand to 3072 dimensions (calculated as 4 chunks of 768 to fit our schema)
    expansion AS (
        SELECT 
            w.row_idx,
            ((p_x <#> w.vec) * -1) + b.val as val
        FROM layer_weights w
        JOIN layer_weights b ON w.layer_idx = b.layer_idx 
                            AND b.tensor_name = 'mlp.c_fc.bias' 
                            AND w.row_idx = b.col_idx -- Assuming col_idx was used for bias mapping in the python script
        WHERE w.layer_idx = p_layer_idx AND w.tensor_name = 'mlp.c_fc.weight'
        ORDER BY w.chunk_idx, w.row_idx
    ),
    -- 2. Apply GELU Activation Function
    -- GELU(x) = 0.5 * x * (1 + tanh(sqrt(2/pi) * (x + 0.044715 * x^3)))
    -- tanh(z) = (exp(2z) - 1) / (exp(2z) + 1)
    gelu AS (
        SELECT 
            row_idx,
            -- Calculate z = sqrt(2/pi) * (val + 0.044715 * val^3)
            (SQRT(2.0 / PI()) * (val + 0.044715 * POWER(val, 3))) as z,
            val as original_val
        FROM expansion
    ),
    gelu_applied AS (
        SELECT 
            row_idx,
            -- Calculate tanh(z) and apply to full formula
            0.5 * original_val * (1.0 + ( (EXP(2.0 * z) - 1.0) / NULLIF((EXP(2.0 * z) + 1.0), 0) )) as gelu_val
        FROM gelu
    ),
    -- 3. Repack into 3072-element array chunks (simulated here as a single aggregation for logic flow)
    -- and project back down to 768 dimensions.
    -- For pure SQL execution without custom types, we do a row-by-row scalar multiplication sum against the projection weights.
    projection AS (
        SELECT 
            w.row_idx as final_dim,
            SUM(g.gelu_val * ((w.vec::real[])[g.row_idx])) as val -- highly inefficient unnesting logic abstracted for standard SQL
        FROM gelu_applied g
        CROSS JOIN layer_weights w
        WHERE w.layer_idx = p_layer_idx AND w.tensor_name = 'mlp.c_proj.weight'
        GROUP BY w.row_idx
    ),
    final_biased AS (
        SELECT 
            p.final_dim,
            p.val + b.val as val
        FROM projection p
        JOIN layer_weights b ON b.layer_idx = p_layer_idx AND b.tensor_name = 'mlp.c_proj.bias' AND b.col_idx = p.final_dim
    )
    SELECT array_agg(val ORDER BY final_dim)::vector(768) INTO v_out FROM final_biased;

    RETURN v_out;
END;
$$ LANGUAGE plpgsql;

ok, it's coming together

We now have the core pieces

layernorm.sql
compute_attention.sql
mlp.sql

Next - the autoregressive generation loop.

need: pl/pgsql function that takes input string, looks up the token embeddings, passes them through all 12 layers, hits the language modeling head, picks the highest probability tokens, and loops

The name in the middle of that steering wheel should tell you that I was born ready, Shelby. Hit it

`ORDER BY (vec <#> hidden_state) LIMIT 1`

Because GPT-2 utilizes weight-tying, the weights of the LM Head are identical to the token embedding matrix we stored at the very beginning in our token_embeddings table. Therefore, projecting our final hidden state to find the absolute most likely next word doesn't require a new matrix or tensor operation. It is just a standard relational query using pgvector's inner product distance:

SELECT
    token_id
FROM
    token_embeddings
ORDER BY (vec <#> v_hidden_state) ASC LIMIT 1;

`generate.sql`

CREATE OR REPLACE FUNCTION generate_text(
    p_prompt_tokens INT[],
    p_max_new_tokens INT DEFAULT 20
) RETURNS TEXT AS $$
DECLARE
    v_session_id UUID := gen_random_uuid();
    v_current_tokens INT[] := p_prompt_tokens;
    v_next_token INT;
    v_x vector(768);
    v_attn_out vector(768);
    v_mlp_out vector(768);
    v_seq_pos INT;
    v_output_text TEXT := '';
BEGIN
    -- 1. Warm up the KV Cache with the seed prompt tokens
    FOR v_seq_pos IN 0 .. (cardinality(p_prompt_tokens) - 1) LOOP
        -- Extract the initial token embedding vector
        SELECT vec INTO v_x FROM token_embeddings WHERE token_id = p_prompt_tokens[v_seq_pos + 1];

        -- Run through the 12-layer Transformer stack to populate past keys/values
        FOR i IN 0 .. 11 LOOP
            -- LayerNorm 1 + Attention + Residual Connection
            v_attn_out := compute_attention(v_session_id, v_seq_pos, i, layernorm(i, 'ln_1', v_x));
            v_x := v_x + v_attn_out;

            -- LayerNorm 2 + MLP + Residual Connection
            v_mlp_out := mlp(i, layernorm(i, 'ln_2', v_x));
            v_x := v_x + v_mlp_out;
        END LOOP;
    END LOOP;

    -- 2. Autoregressive Generation Loop
    v_seq_pos := cardinality(p_prompt_tokens);

    FOR k IN 1 .. p_max_new_tokens LOOP
        -- Grab the embedding for the most recent token generated
        SELECT vec INTO v_x FROM token_embeddings WHERE token_id = v_current_tokens[cardinality(v_current_tokens)];

        -- Forward pass through all layers for the single new token step
        FOR i IN 0 .. 11 LOOP
            v_attn_out := compute_attention(v_session_id, v_seq_pos, i, layernorm(i, 'ln_1', v_x));
            v_x := v_x + v_attn_out;

            v_mlp_out := mlp(i, layernorm(i, 'ln_2', v_x));
            v_x := v_x + v_mlp_out;
        END LOOP;

        -- Final Global LayerNorm before vocabulary projection
        v_x := layernorm(11, 'ln_f', v_x);

        -- Language Modeling Head Projection: Find the single token with highest similarity
        SELECT token_id INTO v_next_token
        FROM token_embeddings
        ORDER BY (vec <#> v_x) ASC
        LIMIT 1;

        -- Append the token to our sliding window state
        v_current_tokens := array_append(v_current_tokens, v_next_token);
        v_output_text := v_output_text || ' ' || v_next_token::text;
        v_seq_pos := v_seq_pos + 1;
    END LOOP;

    -- Clear the session KV cache transaction log so our memory doesn't leak
    DELETE FROM kv_cache WHERE session_id = v_session_id;

    RETURN v_output_text;
END;
$$ LANGUAGE plpgsql;

OK, running into a small issue with column naming, fixing that now. That means most of this code won't work if you copy-paste exactly, but again, goal is to finish this first.

CREATE TABLE
==> Reloading token_embeddings (--reload)
TRUNCATE TABLE
COPY 50257
==> Reloading position_embeddings (--reload)
TRUNCATE TABLE
COPY 1024
==> Reloading layer_weights (--reload)
TRUNCATE TABLE
COPY 110750
==> Indexing + analyzing
NOTICE: relation "idx_layer_weights_lookup" already exists, skipping
CREATE INDEX
ANALYZE
ANALYZE
ANALYZE
==> Installing functions
- attention.sql
CREATE FUNCTION
- generate.sql
CREATE FUNCTION
- layernorm.sql
CREATE FUNCTION
- mlp.sql
CREATE FUNCTION
==> Done.

signs of progress

postgres is great b/c if something is broken it lets you know right away. so if it actually runs, while not guaranteed that you've done it perfectly, it's "right enough."

Pasted image 20260712130807.png

Then, the tokenizer

To pull this off without cheating and leaving the database boundaries, we need an in-database string encoder and decoder.

GPT-2's vocabulary consists of 50,257 tokens, where spaces are represented by the special character Ġ and newlines by Ċ. To avoid rewriting a full Byte-Pair Encoding (BPE) merge-rule tree in procedural SQL, we will implement a Maximum-Match Greedy Tokenizer directly in PL/pgSQL. It replaces standard spaces with Ġ, looks up the longest matching prefixes available in our vocabulary table, consumes them recursively, and passes the resulting token array straight into our inference loop.

Hmm, I can hear my CPU fans spinning

making it fast enough to run

So, that wasn't working reasonably fast enough:

The big speedup (mlp.sql + mlp_cproj_wide):

The bottleneck was the MLP down-projection: it unnested every c_proj.weight row (~2.4M rows/layer) and hash-joined against the activations — the dominant cost across 12 layers × every token.
Now load.py precomputes each output dim's full 3072-wide weight vector once into mlp_cproj_wide. mlp() assembles the GELU activations into one vector(3072) and does 768 pgvector dot products per layer instead. That's the single biggest lever, and it moves the heavy math into pgvector's SIMD path.

Why it was slow / other levers you can pull:

Each generated token = a full 12-layer forward pass, and each layer does real matmuls in SQL. It's inherently O(tokens × layers × d²).
Fewer tokens is the most direct knob: generate_text(ARRAY[...], 3) runs ~3× faster than 10. Good for a first end-to-end smoke test.
The up-projection (c_fc, 3072 dot products/layer) and attention are already pgvector-based and comparatively cheap, so I left them.
LM head scans all 50257 token embeddings per generated token (exact argmax). That's fast (one <#> sweep), but if you want it faster you could add an HNSW index on token_embeddings(vec) for approximate nearest-neighbor — at the cost of exactness.
The Postgres tuning we set earlier (durability off, big work_mem/shared_buffers) already helps these aggregate-heavy queries.

Wow it's starting to work

postgres=# \timing on
Timing is on.
postgres=# SELECT 'Hello, I' || generate_text(ARRAY[15496, 11, 314], 20) AS story;
                              story
-----------------------------------------------------------------
 Hello, I'm sorry, I'm sorry. I'm sorry. I'm sorry. I'm sorry. I
(1 row)

Time: 27628.671 ms (00:27.629)
postgres=#

"the meaning of life is"

ARRAY[464, 3616, 286, 1204, 318]

Pasted image 20260712133452.png

postgres=# SELECT generate_text(ARRAY[464, 3616, 286, 1204, 318], 5);
      generate_text
--------------------------                                                  
not to be confused with
(1 row)

Time: 17125.167 ms (00:17.125)
postgres=# SELECT generate_text(ARRAY[464, 3616, 286, 1204, 318], 10);
                 generate_text
-----------------------------------------------                             
not to be confused with the meaning of life.
(1 row)

Time: 18001.460 ms (00:18.001)
postgres=# SELECT generate_text(ARRAY[464, 3616, 286, 1204, 318], 12);
                    generate_text
-----------------------------------------------------
  not to be confused with the meaning of life. It is                      
(1 row)

Time: 20727.452 ms (00:20.727)

We did it

check-in

This is enough to declare a victory (we've implemented a basic LLM in sql), but let's do a couple more things first:

Running GPT2's BPE-encoder in SQL

Pasted image 20260712134939.png

ARRAY[464, 3616, 286, 1204, 318]

you know what's cool

SELECT
    prompt,
    complete(prompt, 8)
FROM (
    VALUES 
        ('Hello, I'),
        ('In the beginning')
    )
AS t(prompt);

Instead of having to look up the token ids -> running postgres, let's implement the vocab lookup in postgres as well

-- Convenience wrapper: text in, text out (prompt + continuation) in one call.
-- End-to-end GPT-2 in a single SQL expression:  SELECT complete('Hello, I', 10);
CREATE OR REPLACE FUNCTION complete(p_prompt TEXT, p_max_new_tokens INT DEFAULT 10)
RETURNS TEXT AS $$
    SELECT p_prompt || generate_text(gpt2_encode(p_prompt), p_max_new_tokens);
$$ LANGUAGE sql;

postgres=# SELECT prompt, complete(prompt, 8)
FROM (
    VALUES ('Hello, we just implemented an LLM in postgres, which means')) 
AS t(prompt);

OK this is not going to finish fast enough because

Every token (all 13 prompt tokens during warmup + 8 generated) runs a full 12-layer forward pass.
That's (13 + 8) × 12 ≈ 252 layer passes, and each layer pass fires several SQL statements (2 LayerNorms, attention's insert+read+delete, one big MLP query).

So you're executing ~1,500+ SQL statements, and inside them roughly 2+ million pgvector dot products. In an interpreted plpgsql loop over a row-oriented engine, that lands in the tens-of-seconds range. Expected for "GPT-2 in pure SQL."

Pasted image 20260712140358.png

postgres=# SELECT prompt, complete(prompt, 20)
FROM (VALUES ('LLM.sql because')) AS t(prompt);
     prompt      |                            complete
-----------------+----------------------------------------------------------------
 LLM.sql because | LLM.sql because it is not supported by the database.          +
                 |                                                               +
                 | The following table lists the SQL statements that are executed
(1 row)

Time: 31586.728 ms (00:31.587)

postgres=# SELECT prompt, complete(prompt, 20)
FROM (VALUES ('LLM.sql great idea because')) AS t(prompt);
           prompt           |                           complete
----------------------------+--------------------------------------------------------------
 LLM.sql great idea because | LLM.sql great idea because of the way it works.             +
                            |                                                             +
                            | The problem is that the SQL Server database is not very good
(1 row)

Time: 178264.059 ms (02:58.264)

So - I knew this before but in the spirit of "vulcanize rubber" empirically we learn that compute time scales more based on input tokens and not output tokens.

Let's wrap this up for now

Tempting to go deep into improvements, curious about scaling properties, thinking about how to make this faster, evals, there is a lot to do from here.

Also, I'm going to leave this post as it is and maybe edit/polish it in the future.

The "oh but make it public when it's ready" sirens are out in their finest sundresses today, and we will pay them no mind.

We did the thing we set out to do.