2550ef763e
* Add a versioned embedding provider SPI Summary - Add a runtime registry for embedding providers with readiness, model resolution, preparation, and batch embedding. - Define immutable embedding-space descriptors and legacy deterministic identities. - Validate provider results, dimensions, counts, and finite numeric vector components. - Make SPI version 1 a compile-time constant; provider artifacts must embed the literal version they were built against. Scope This defines the neutral provider contract only. It does not migrate semantic-search consumers or ship a provider plugin. Testing - Registration and version compatibility. - Identity stability and revision round trips. - Registry snapshot consistency and malformed provider output. * Rename shadowing local in provider/prepare! The local bound from :prepare! shadowed the prepare! var (eastwood local-shadows-var). * Add the Arctic embedder plugin Summary - Build a separate runtime plugin containing the pinned Snowflake Arctic embedding model and DJL/ONNX dependencies. - Bundle arm64 and AVX2 exports, lazy-load native inference, and cap local batches. - Register literal embedding SPI version 1 and derive immutable space identity from the exact model/runtime bundle. - Fail readiness closed for unsupported platforms, musl, and glibc older than 2.34. - Disable DJL network telemetry and revision-address bundle URLs so model upgrades cannot reuse stale cached content. Scope This PR ships and registers the provider but does not yet select it from semantic-search settings. Consumer integration follows upstack. Testing - Model bundle and plugin uberjar build tests. - Catalog, classloader, platform-readiness, identity, cache-key, and batching coverage. - Artifact-only real inference with zero outbound network attempts. * Follow prune-provided-libs! rename in the embedder plugin build * Merge nested let in embedder-plugin artifact test Kondo flagged the inner let as redundant; fold both binding vectors into one. * Deref the now-delayed core provided-libs in the embedder build * Bundle MiniLM for in-process complexity scoring * Activate the embedder plugin in its artifact smoke test * Activate lazy embedding providers on demand * Use plugin initializer in artifact smoke test * Make lazy provider activation best effort * Log lazy embedding provider activation failures * Declare the embeddings plugin dependency * Keep embeddings provider internal * Activate the in-process embedder during bootstrap * Fix embedder validation and optional plugin startup * Declare the embedder plugin as a module The deps graph only scanned src, enterprise/backend/src, and modules/drivers, so the plugin's namespaces belonged to no module and anyone linting them got six unresolved-dependency warnings. Scanning modules/embedder/src too lets the generator record what the plugin actually takes from the host. * Move embedder model caching into the weekly cache generator Review feedback on #78310: - Drop the redundant persist-credentials: false checkout options. - Stop writing the model cache from the CI job. The weekly cache-generator workflow is the single cache writer; the build job now restores only. The downloader is content-addressed (SHA-256 verified, refetches deltas), so a stale weekly cache still protects against HuggingFace rate limits without affecting correctness. * Bundle snowflake-arctic-embed-xs instead of Arctic-L The bundled models are a test tier: they exercise the packaging, DJL runtime, tokenizer, and pooling machinery without shipping the 1.1 GB Arctic-L artifacts. xs keeps the same family, query-prefix scheme, and cls pooling as Arctic-L at 23 MB per architecture. A release-tier build with the full model comes later, when the plugin is published for customers or offline use. At ~90 MB of pinned downloads there is nothing worth caching, so the cache-generator wiring is reverted and the CI job downloads directly. * Fail loudly when a catalog entry lacks this architecture's export resolved-model computed the embedding-space-id from get-in spec [:architectures arch :sha256] without checking it was present. A model missing this architecture's export would silently bake a nil into the identity hash instead of failing, producing a syntactically valid but bogus embedding-space-id. * Pin the embedder plugin manifest name to one constant "Metabase In-Process Embedder" was duplicated in core.clj (the registered?/load-plugin! lookup key) and the artifact test. A rename in modules/embedder/resources/metabase/embedder/metabase-plugin.yaml without updating both call sites would silently disable the plugin, since registered? just returns false for a name it doesn't recognize. * Downgrade the hashed-table-name log line to debug Hashing fires deterministically for every identifier derived from a long provider/model pairing (e.g. "in-process" + the bundled xs model), so it was logging a warning on every lookup rather than once at creation. * Cover the missing-architecture catalog guard The error path added alongside the guard had no test, so the earlier silent nil-hashing behaviour could come back unnoticed. * Assert the reason on the missing-architecture guard The message alone leaves `:reason :architecture-not-bundled` untested, which is the part callers actually branch on. * Drop the hashed-table-name log line It fires from a pure name-derivation helper on every lookup. The table creation it hints at is already logged at info by the caller, and the parameters behind the hash are in index_metadata. * Extract the missing-architecture check out of the smoke test The guard's assertions pushed the smoke test to 155 lines, past the 150-line deftest limit, so Clj-Kondo failed. Pull it into a helper the way the platform-support checks already are.
15 lines
280 B
Bash
Executable File
15 lines
280 B
Bash
Executable File
#! /usr/bin/env bash
|
|
|
|
set -eo pipefail
|
|
|
|
script_directory=`dirname "${BASH_SOURCE[0]}"`
|
|
cd "$script_directory/.."
|
|
|
|
source "./bin/check-clojure-cli.sh"
|
|
check_clojure_cli
|
|
|
|
source "./bin/clear-outdated-cpcaches.sh"
|
|
clear_outdated_cpcaches
|
|
|
|
clojure -X:build:build/embedder-plugin "$@"
|