Spike: sqlc -> Clojure query codegen with Malli schemas

Generate Clojure query fns with Malli result schemas from .sql files, using
sqlc for SQL parsing and type inference and sqlc-gen-json to expose its
analysis as JSON.

Verified end to end: migrations against Postgres 16 (1427 changesets, 190
tables), pg_dump fed to sqlc unmodified, real queries ported from the
glossary/sso HugSQL spike branches, and the generated file compiles with
schemas that accept valid rows and reject bad ones.

See dev/src/dev/sqlc/README.md for open issues (no H2 engine, weak param
type inference, no dynamic SQL).

dev/src/dev/sqlc/EXAMPLE_OUTPUT.clj.txt

@@ -0,0 +1,97 @@
+;; GENERATED by dev.sqlc.printer -- do not edit.
+(ns metabase.glossary.generated-queries
+  (:require [dev.sqlc.runtime :refer [execute!]]))
+
+(defn glossary-entry
+  "Generated from SQL. Do not edit.
+
+  SELECT id, term, definition, created_at, updated_at, creator_id FROM glossary WHERE id = $1
+
+  Returns: one row or nil"
+  {:malli/schema [:=> [:cat :any :any]
+                  [:maybe [:map [:id :int] [:term :string] [:definition :string] [:created_at inst?] [:updated_at inst?] [:creator_id :int]]]]}
+  [db id]
+  (execute! db
+            ["SELECT id, term, definition, created_at, updated_at, creator_id FROM glossary WHERE id = $1"
+             id]
+            :one))
+
+(defn glossary-entries-search
+  "Generated from SQL. Do not edit.
+
+  SELECT id, term, definition, created_at, updated_at, creator_id FROM glossary
+  WHERE (LOWER(term) LIKE $1 ESCAPE '!' OR LOWER(definition) LIKE $1 ESCAPE '!')
+  ORDER BY term ASC
+
+  Returns: a vector of rows"
+  {:malli/schema [:=> [:cat :any :any]
+                  [:sequential [:map [:id :int] [:term :string] [:definition :string] [:created_at inst?] [:updated_at inst?] [:creator_id :int]]]]}
+  [db like-escape]
+  (execute! db
+            ["SELECT id, term, definition, created_at, updated_at, creator_id FROM glossary\nWHERE (LOWER(term) LIKE $1 ESCAPE '!' OR LOWER(definition) LIKE $1 ESCAPE '!')\nORDER BY term ASC"
+             like-escape]
+            :many))
+
+(defn users-by-id
+  "Generated from SQL. Do not edit.
+
+  SELECT id, email, first_name, last_name
+  FROM core_user WHERE id = ANY($1::int[])
+
+  Returns: a vector of rows"
+  {:malli/schema [:=> [:cat :any :any]
+                  [:sequential [:map [:id :int] [:email :string] [:first_name [:maybe :string]] [:last_name [:maybe :string]]]]]}
+  [db arg-1]
+  (execute! db
+            ["SELECT id, email, first_name, last_name\nFROM core_user WHERE id = ANY($1::int[])"
+             arg-1]
+            :many))
+
+(defn user-group-ids-excluding
+  "Generated from SQL. Do not edit.
+
+  SELECT group_id FROM permissions_group_membership
+  WHERE user_id = $1 AND group_id <> ALL($2::int[])
+
+  Returns: a vector of rows"
+  {:malli/schema [:=> [:cat :any :any :any]
+                  [:sequential [:map [:group_id :int]]]]}
+  [db user-id arg-2]
+  (execute! db
+            ["SELECT group_id FROM permissions_group_membership\nWHERE user_id = $1 AND group_id <> ALL($2::int[])"
+             user-id
+             arg-2]
+            :many))
+
+(defn auth-identity-exists
+  "Generated from SQL. Do not edit.
+
+  SELECT 1 AS one FROM auth_identity
+  WHERE user_id = $1 AND provider = $2 LIMIT 1
+
+  Returns: one row or nil"
+  {:malli/schema [:=> [:cat :any :any :any]
+                  [:maybe [:map [:one :int]]]]}
+  [db user-id provider]
+  (execute! db
+            ["SELECT 1 AS one FROM auth_identity\nWHERE user_id = $1 AND provider = $2 LIMIT 1"
+             user-id
+             provider]
+            :one))
+
+(defn insert-glossary-entry
+  "Generated from SQL. Do not edit.
+
+  INSERT INTO glossary (term, definition, creator_id, created_at, updated_at)
+  VALUES ($1, $2, $3, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)
+
+  Returns: update count"
+  {:malli/schema [:=> [:cat :any :any :any :any]
+                  :int]}
+  [db term definition creator-id]
+  (execute! db
+            ["INSERT INTO glossary (term, definition, creator_id, created_at, updated_at)\nVALUES ($1, $2, $3, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)"
+             term
+             definition
+             creator-id]
+            :exec))

dev/src/dev/sqlc/README.md

@@ -0,0 +1,52 @@
+# sqlc -> Clojure codegen spike
+
+Generates Clojure query fns with Malli result schemas from `.sql` files, using
+[sqlc](https://sqlc.dev) for SQL parsing and type inference.
+
+## Why sqlc
+
+sqlc resolves `SELECT *` against a catalog and infers expression types (`count(*)` -> `bigint`),
+which is the hard part of typing SQL. It reads a schema dump rather than connecting to a live
+database, so generation needs no running server. `sqlc-gen-json` is a WASM plugin that emits
+sqlc's whole analysis as JSON, so the Clojure side only has to render it.
+
+## Pipeline
+
+    migrated app-db -> pg_dump --schema-only -> sqlc + sqlc-gen-json -> generate_request.json
+      -> dev.sqlc.printer -> generated .clj (defn + :malli/schema + docstring)
+
+The schema comes from a *migrated database*, not from the Liquibase YAML: the YAML is a program
+(conditional changesets, custom Clojure migrations) and rendering it statically would mean
+reimplementing Liquibase. Dumping a migrated instance asks the database what actually exists.
+
+## Verified in this spike
+
+- Migrations ran against Postgres 16: 1427 changesets, 190 tables (67 filtered as DBMS mismatch).
+- `pg_dump` output (20,501 lines) fed to sqlc unmodified -- no massaging needed.
+- Real queries ported from the glossary/sso HugSQL spike branches resolved correctly:
+  `SELECT *` expanded, `citext` on `core_user.email`, nullability right on `first_name`.
+- Generated file parses, compiles, and its Malli schemas accept valid rows / reject bad ones.
+
+## Reproduce
+
+    docker run -d --name sqlc-spike-pg -e POSTGRES_PASSWORD=spike -e POSTGRES_USER=spike \
+      -e POSTGRES_DB=mbspike -p 55432:5432 postgres:16
+    MB_DB_TYPE=postgres MB_DB_HOST=localhost MB_DB_PORT=55432 MB_DB_DBNAME=mbspike \
+      MB_DB_USER=spike MB_DB_PASS=spike clojure -M -e "(require '[metabase.app-db.core :as mdb]) (mdb/setup-db! :create-sample-content? false)"
+    PGPASSWORD=spike pg_dump -h localhost -p 55432 -U spike -d mbspike --schema-only \
+      --no-owner --no-privileges -f schema/schema.sql
+    sqlc generate   # with sqlc.yaml in this dir
+
+## Open issues
+
+- **No H2 engine.** sqlc supports postgresql/mysql/sqlite only. H2 types would have to be
+  borrowed from the Postgres analysis or read via JDBC introspection against in-process H2.
+- **Param type inference is weaker than column inference.** `LIKE $1 ESCAPE '!'` inferred the
+  param as `bytea`. Result columns were correct in every case tested; params were not.
+  This spike only emits `:any` for params, which sidesteps it.
+- **Query names must be Go identifiers.** `-- name: glossary-entry` is rejected; the printer
+  kebab-cases `GlossaryEntry` on the way out, so `.sql` files carry CamelCase names.
+- **No dynamic SQL.** sqlc has no equivalent of HugSQL's `:value*:ids` expansion. Postgres
+  `= ANY($1::int[])` covers the IN-list case and types correctly; other dynamic shapes do not
+  translate.
+- Schema freshness needs a CI check (regenerate, `git diff --exit-code`).

dev/src/dev/sqlc/example_queries.sql

@@ -0,0 +1,23 @@
+-- name: GlossaryEntry :one
+SELECT * FROM glossary WHERE id = $1;
+
+-- name: GlossaryEntriesSearch :many
+SELECT * FROM glossary
+WHERE (LOWER(term) LIKE $1 ESCAPE '!' OR LOWER(definition) LIKE $1 ESCAPE '!')
+ORDER BY term ASC;
+
+-- name: UsersById :many
+SELECT id, email, first_name, last_name
+FROM core_user WHERE id = ANY($1::int[]);
+
+-- name: UserGroupIdsExcluding :many
+SELECT group_id FROM permissions_group_membership
+WHERE user_id = $1 AND group_id <> ALL($2::int[]);
+
+-- name: AuthIdentityExists :one
+SELECT 1 AS one FROM auth_identity
+WHERE user_id = $1 AND provider = $2 LIMIT 1;
+
+-- name: InsertGlossaryEntry :exec
+INSERT INTO glossary (term, definition, creator_id, created_at, updated_at)
+VALUES ($1, $2, $3, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP);

dev/src/dev/sqlc/printer.clj

@@ -0,0 +1,119 @@
+(ns dev.sqlc.printer
+  "Spike: turn `sqlc-gen-json` output into Clojure query fns with Malli result schemas.
+
+  Pipeline: migrated app-db -> pg_dump -> sqlc (+ sqlc-gen-json) -> generate_request.json -> here.
+  sqlc resolves `SELECT *` against the catalog and infers expression types, so the JSON already
+  carries per-column names, types, and nullability; this namespace only renders them."
+  (:require
+   [camel-snake-kebab.core :as csk]
+   [cheshire.core :as json]
+   [clojure.java.io :as io]
+   [clojure.string :as str]))
+
+(def ^:private pg-type->malli
+  "Postgres type name -> Malli schema. Names arrive bare (`int4`) or schema-qualified
+  (`pg_catalog.int4`); `pg-type` strips the qualifier before lookup."
+  {"int2"        :int
+   "int4"        :int
+   "int8"        :int
+   "int"         :int
+   "bigint"      :int
+   "serial"      :int
+   "float4"      :double
+   "float8"      :double
+   "numeric"     'decimal?
+   "bool"        :boolean
+   "text"        :string
+   "varchar"     :string
+   "char"        :string
+   "citext"      :string
+   "uuid"        :uuid
+   "date"        'inst?
+   "timestamp"   'inst?
+   "timestamptz" 'inst?
+   "json"        :any
+   "jsonb"       :any
+   "bytea"       'bytes?})
+
+(defn- pg-type
+  "Bare type name for a sqlc column, e.g. `pg_catalog.int4` -> `int4`."
+  [col]
+  (-> col :type :name (str/split #"\.") last))
+
+(defn- col->malli
+  "Malli schema for one sqlc column. Unknown types degrade to :any with a marker so the
+  generated file shows what was not mapped rather than silently widening."
+  [col]
+  (let [base    (get pg-type->malli (pg-type col) :any)
+        base    (if (:isArray col) [:sequential base] base)]
+    (if (:notNull col)
+      base
+      [:maybe base])))
+
+(defn- row-schema [columns]
+  (into [:map] (map (fn [c] [(keyword (:name c)) (col->malli c)])) columns))
+
+(defn- result-schema
+  "Result schema for a query, keyed off the sqlc command: `:one` is a single row or nil,
+  `:many` a vector of rows, `:exec` an update count."
+  [{:keys [cmd columns]}]
+  (let [row (row-schema columns)]
+    (case cmd
+      ":one"  [:maybe row]
+      ":many" [:sequential row]
+      ":exec" :int
+      :any)))
+
+(defn- unmapped-types
+  "Type names in `columns` that have no Malli mapping -- surfaced as a docstring warning."
+  [columns]
+  (->> columns (map pg-type) (remove pg-type->malli) distinct sort))
+
+(defn- docstring [{:keys [text cmd columns]}]
+  (let [unmapped (unmapped-types columns)]
+    (str "Generated from SQL. Do not edit.\n\n  "
+         (str/replace (str/trim text) #"\n" "\n  ")
+         "\n\n  Returns: " (case cmd ":one" "one row or nil" ":many" "a vector of rows"
+                                    ":exec" "update count" "unknown")
+         (when (seq unmapped)
+           (str "\n  UNMAPPED TYPES (schema widened to :any): " (str/join ", " unmapped))))))
+
+(defn- fn-params
+  "Parameter symbols for a query. sqlc names a parameter after the column it is compared to;
+  unnamed or duplicate names (a param used in two places) fall back to positional argN."
+  [parameters]
+  (let [named (mapv #(csk/->kebab-case-symbol (or (get-in % [:column :name])
+                                                  (str "arg" (:number %))))
+                    parameters)]
+    (if (= (count named) (count (distinct named)))
+      named
+      (mapv #(symbol (str "arg" (inc %))) (range (count named))))))
+
+(defn- query->defn-str
+  "Render one sqlc query as Clojure source text. Emitted as text rather than via pprint so the
+  Malli metadata survives and the docstring keeps real newlines."
+  [{:keys [name cmd] :as q}]
+  (let [fn-name (csk/->kebab-case-symbol name)
+        params  (fn-params (:parameters q))
+        schema  (result-schema q)]
+    (str "(defn " fn-name "\n"
+         "  \"" (str/replace (docstring q) #"\"" "\\\"") "\"\n"
+         "  {:malli/schema [:=> [:cat :any" (str/join (repeat (count params) " :any")) "]\n"
+         "                  " (pr-str schema) "]}\n"
+         "  [db" (str/join (map #(str " " %) params)) "]\n"
+         "  (execute! db\n"
+         "            [" (pr-str (:text q)) (str/join (map #(str "\n             " %) params)) "]\n"
+         "            " (keyword (subs cmd 1)) "))")))
+
+(defn render
+  "Read a sqlc generate_request.json and return the generated namespace as a string."
+  [json-path ns-sym]
+  (let [{:keys [queries]} (json/parse-string (slurp json-path) true)]
+    (str ";; GENERATED by dev.sqlc.printer -- do not edit.\n"
+         "(ns " ns-sym "\n  (:require [dev.sqlc.runtime :refer [execute!]]))\n\n"
+         (str/join "\n\n" (map query->defn-str queries)) "\n")))
+
+(defn -main [json-path ns-sym out-path]
+  (io/make-parents out-path)
+  (spit out-path (render json-path (symbol ns-sym)))
+  (println "wrote" out-path))

dev/src/dev/sqlc/sqlc.yaml

@@ -0,0 +1,13 @@
+version: "2"
+plugins:
+  - name: jsonplug
+    wasm:
+      url: file:///tmp/sgj.wasm
+      sha256: d35f71b18c2db3bb13356340c9b0454af2a4a7fec230f4903b3b39665bf8895a
+sql:
+  - engine: postgresql
+    schema: schema
+    queries: queries
+    codegen:
+      - plugin: jsonplug
+        out: out