Plugin SDK
Testen & veröffentlichen
Verträge, Zugriffsrechte, MCP und Browser testen. Externe Modell-Evals einordnen.
Diese technische Anleitung wird derzeit auf Englisch gepflegt. Navigation und Nutzer-Anleitungen sind auf Deutsch verfügbar.
Test the independent plugin before testing companions. Separate deterministic correctness, authorization, client behavior and live deployment evidence. One green score does not prove all four.
Ordered test layers
| Layer | Minimum proof |
|---|---|
| 1. Files and workflow | Strict schemas, unsupported versions, codec round trips, deterministic plans, domain refusal cases, unchanged unrelated records. |
| 2. Contract/compiler | Shipped schemas match runtime, versions/IDs resolve, required cycles rejected, optional integrations degrade on absence/incompatibility. |
| 3. Real host | Postgres persistence, private bootstrap, revision conflicts, atomic failure, idempotency, protected parents, denial/exclusion/expiry, reader vs editor, current-rights replay, export/disable. |
| 4. OAuth/MCP | Actual authenticated discovery, create/read/query/edit, retry, cross-user denial, revocation, disabled reads and writes. |
| 5. Companion integration | Both directions, canonical link owner, no source mutation, per-resource rights, disabled counterpart, missing public contract, no stale content. |
| 6. Browser | Real forms/navigation, validation/input retention, search/counts, no-JS, keyboard/mobile and unavailable states. |
| 7. External client | Fresh connected client understands indirect requests, loads guide, chooses tools, asks when ambiguous, and achieves verified persisted state. |
| 8. Live rollout | Deployed commit/images, actual connected-account smoke and separately authorized second-user collaboration. |
Layers 1–6 should be deterministic and normally run in CI. Model calls are opt-in and belong to an external runner. Live tests must use labelled synthetic records and explicit authorization; never send someone a real assignment or invitation as an incidental test.
Repository commands
Run from the source repo root with compose running and migrations applied:
pnpm plugin:check
pnpm plugin:verify-external
pnpm typecheck
pnpm test
pnpm build
sh scripts/browser-smoke.sh
pnpm plugin:verify-isolationplugin:check checks every reviewed definition from plugins/catalog.json, each
in isolation and all together. Package boundaries, packaged artifacts and browser
entrypoints also follow that catalog; a declared public contract requires a runtime
projection. Host guards cover every registered plugin namespace and route, and
reject relative package escapes as well as forbidden package imports.
The isolation command derives its modes from the reviewed catalog, builds and
drives the real server/web with each plugin alone, then with an empty reviewed
catalog for declarative plugins. Explorer exercises activation, file editing,
publication and export through its browser rig; Tasks, Hello World, Calendar and
Messages use their independent browser flows. Wiki, Contacts and newly
registered plugins without a dedicated browser flow get public-documentation
boot checks; Secrets and Integrations get the registry checks only. Their
domain behavior needs separate acceptance. In every mode it also checks each
v2 catalog of that build: the api/worker, mcp and browser compositions
offer only that plugin and refuse every other id, composing everything offered
loads no other reviewed package (a module log of each probe process), jobs and
procedures belong to that plugin, and the emitted OpenAPI document names
exactly the default selection's namespaces. The command
restores the normal registries and build in finally.
plugin:verify-external is the external author path: the packed SDK and the
example in a clean directory with no other @meinkontext/* package, the
example plus a module over every SDK v2 entrypoint compiled with
skipLibCheck: false, and the v2 OpenAPI documents. The external client
(packages/plugin-client: OAuth, plugin discovery, a restricted execution
grant, read-back and revocation) is exercised against the built v2 auth and
mcp fleets by apps/server/src/entrypoints/plugin-client.integration.test.ts
(store suite). Do not run another build concurrently in the same worktree. A descriptor-only check is not a substitute for these actual builds.
For the current Calendar pilot, focused rechecks are:
pnpm test packages/plugin-calendar/src/runtime.test.ts tooling/plugins/contracts.test.ts
CI=true pnpm test packages/core/src/plugins/host.test.ts apps/server/src/e2e/oauth-e2e.smoke.test.ts -t 'Calendar|Tasks and Calendar integrate' --maxWorkers=1Root pnpm test builds packages first. Smoke servers resolve package exports from dist; testing a source change without rebuilding can falsely pass. Store suites fail when prerequisites are absent in CI. Skipped store/model suites must be reported as skipped, not green evidence.
Client evaluation and Langfuse
A plugin owns its PluginEvalSuite: scenario setup, prompt, expected behavior and independent state graders. The generic SDK only defines the evaluator interface. The external testkit provides execution; meinKontext itself runs no generative AI (its one exception, a local search model under temporary test, computes search vectors only).
Assert actual stored state, revisions, denied effects and tool calls. An assistant saying “done” is not proof. Include indirect discovery, ambiguity, malicious source instructions, forbidden actions and loss of rights. evalPassed rejects empty checks, incomplete runs, execution errors and failed checks; do not average a critical failure into a passing score.
The repository's docs/plugins/CONTEXT-QUALITY.md documents the external gateway runner and optional Langfuse export. Langfuse records traces/scores/usage for comparison; it does not enforce RBAC or replace deterministic tests. Keep provider credentials in the external client, never plugin manifests, canonical files, prompts or test reports. Record dataset version, model, source revision, measured scope and redacted outcomes. Do not claim Langfuse evidence for a native client run that did not export traces.
Calendar's opt-in native Codex cases run with:
CI=true CALENDAR_NATIVE_CODEX=1 pnpm test apps/server/src/e2e/oauth-e2e.smoke.test.ts -t 'native Codex Calendar context acceptance'This consumes the connected Codex account's external model usage. It is not the default CI gate or an unattended-import safety claim. The separate native evaluator accepts synthetic local OAuth only on loopback; the production import launcher still requires a restricted execution credential.
Evidence and release review
For the planned cross-owner integration gate, use separate OAuth identities and owner organizations. Verify destination write rights independently of source read/contract rights; cover revocation between checks and commit, replay after revocation, expiry, disabled providers and same-named resources in different scopes. Check search/count/export/report visibility as well as record reads. A shared destination must never disclose a private linked source. This matrix extends the current same-owner integration tests; it is not a claim that cross-owner links already work.
Record the exact commit and environment for each result, links to CI, test counts, skips, and any timeout or failed attempt. Keep a local two-user OAuth test distinct from a live second-user test. Attach actual labelled screenshots for UI changes; do not present illustrations as product evidence.
A passing small fixture does not close public distribution/license gates, migration/restore readiness, enterprise load gates or a real independent plugin-author pilot. Release notes should say which of these remain open.
Reviewed release actions
See fixed audiences and reviewed commands for requiresApproval, immutable collection ceilings, live revision dependencies, web review and the trusted MCP command-grant client. These generic primitives never put domain semantics in the host.
Cross-plugin agent acceptance
Calendar and Messages ship versioned ./evals datasets. Use pnpm plugin:eval:composition:codex for fresh native clients, or PLUGIN_EVAL_SUITE=calendar|messages pnpm plugin:eval:local for the dedicated gateway/Langfuse profile. These are paid, opt-in evaluations; normal CI runs deterministic negative controls that reject fabricated success. The fixture-only executeReviewed callback prepares synthetic approved messages outside the evaluated model. Fixture cleanup restores temporary activation state in a finally block. An approvedCommand supplies exactly reviewed synthetic input to a restricted client; it is mutually exclusive with an intake executionScope. expectedReadDenials may name only known private fixture IDs: their explicit UNAVAILABLE reads remain visible as tool errors, while a correctly completed explanation can pass. Network errors, failed writes, other IDs, incomplete turns and missing answers still fail. PLUGIN_EVAL_CASES selects diagnostic gateway cases, but a subset can never pass overall acceptance. Gateway results are checkpointed after every completed case. Scope and results.
Fixture setup, restricted-credential setup, grading and cleanup can fail independently of model behavior. The repository's external evaluator records each as a failed case with a safe stage category; it never exports raw exception bodies or lets the experiment backend silently drop the scenario. Compare expected scenario counts with recorded results and independently read back check arrays and case scores. A successful subset or a budget-blocked suite is not a passing full run.
Reading evaluation results in Langfuse
Start in Datasets: each item describes a task and its expected outcome. Experiments are concrete runs against that contract. Open an item to compare its expected output, recorded outcome and Scores. Tracing contains the sanitized case protocol. A hosted dataset is not itself evidence that the product passed.
| Signal | Meaning |
|---|---|
case_passed = 1 | Complete agent execution and all independent state/behavior checks passed. |
case_passed = 0, execution_complete = 1 | A completed run violated at least one expectation. Investigate actual effects and the failed checks. |
execution_complete = 0 | Infrastructure, fixture or execution did not complete. No full product-quality verdict. |
suite_passed = 1 | Every case in the entire versioned catalog passed. A subset cannot pass this gate. |
execution_coverage | Fraction of the whole catalog fully executed and evaluated, including completed cases that failed expectations. |
BOOLEAN values may display as true/false. A passing average cannot cancel a failed permission check. Distinguish a model mistake from HTTP 429, timeout and fixture failure. Compare the same dataset version, client, code and scenario scope; repeat runs before claiming reliability. UI behavior and deployed two-user access remain separate acceptance gates.
Plugin authors can supply optional public presentation metadata on PluginEvalSuite (title, description) and each case (title, scenario, expected, category). The external runner uses these for readable hosted datasets and experiments, never as hints to the evaluated agent. Keep all domain scenarios and graders in the plugin. The generic SDK does not contain Tasks, Calendar or Messages knowledge.
The Langfuse exporter synchronizes immutable, versioned dataset contracts and links executions to their hosted items. Changes to a published contract require a deliberate new datasetVersion. Gateway repeats become separately named runs. Native composition export is opt-in with PLUGIN_EVAL_LANGFUSE=true and explicit Langfuse project credentials. An Archiv run imports existing evidence without model calls; its Langfuse duration is import duration, not model latency. Missing original timing stays unknown.
External evaluators and the Evaluators page differ. Our authoritative evaluators run beside the synthetic fixture, inspect actual persisted state and publish scores through the Langfuse SDK. They are not automatically UI-managed evaluator definitions. Hosted code evaluators require a separately configured execution backend; a visible Evaluators navigation entry does not prove this backend is enabled. The meinKontext instance enabled its protocol-consistency evaluator on 2026-09-30 and verified six positive/negative protocols, filter exclusion and runtime errors. It rechecks exported checks, but cannot inspect meinKontext state. A separate pilot LLM judge assesses usefulness and unsupported claims on six synthetic controls; ongoing product scoring remains disabled pending human calibration. It never overrides failed authorization or state checks.
Default export omits prompts, answers and tool payloads. Gateway runs now record real model generations, MCP calls and normalized token counts. Missing price information is not zero cost; gateway billing remains authoritative. Full conversation content requires explicit PLUGIN_EVAL_TRACE_CONTENT=true and synthetic fixtures. meinKontext itself continues to run no inference.
Sessions and Prompt Management are integrated in the external gateway runner. One case/repetition gets one session; fixtures may return followUps: string[] for genuine user turns sharing history and a single step/time budget. AgentResult.turns records actual answers and half-open indexes into calls, so graders can enforce a boundary between turns. Keep domain expectations inside the plugin. Native composition rejects follow-up fixtures until client resume is implemented; archived runs do not get invented sessions.
Pin PLUGIN_EVAL_PROMPT_NAME=meinKontext/eval/agent and PLUGIN_EVAL_PROMPT_VERSION=1 to use the hosted baseline. Both are required for remote selection; there is no latest-label resolution or silent fallback. The resolved prefix hash/name/version goes into the report and each gateway generation links to its version. MCP instructions still come unchanged from the server under test. Candidate prefixes can be authored in Langfuse; never place hidden graders/answers in them.
With synthetic content exported, PLUGIN_EVAL_HUMAN_REVIEW=true enqueues the actual session for human review. Human scores and pilot judge scores are separate quality signals and cannot change deterministic acceptance. See the operator walkthrough and commands and boundaries.
Core-context acceptance
The independent Kernkontext dataset tests the platform's core promise without installing a domain plugin. Its eight cases cover skill discovery/application, current wiki retrieval among 42 articles, contradictory evidence, memory provenance and cross-project recall, whole-wiki sharing, revoked access and injected source instructions.
The core scenarios live in the external testkit, not a domain plugin or the production host. PluginEvalCase<Environment> and PluginEvalSuite<Environment> accept an explicit fixture environment; the default remains PluginEvalEnvironment, so existing plugin suites need no changes. Reporters consume the static EvalCaseDefinition/EvalSuiteDefinition rather than requiring domain fixture capabilities. scope: "core" is reporting metadata, never a plugin installation.
Run deterministic reference/negative controls with:
pnpm test apps/server/src/e2e/core-context.smoke.test.tsThe graders' own contract (a read counts only through the server's answer, only a refusal naming the target counts as an access check) is no PR-chain test: assertGraderContracts() runs as the evaluation's precondition, before any model call.
Run an explicit paid gateway evaluation with pnpm context:eval:local using the existing dedicated evaluation profile. The two accounts and stores are local and synthetic. Pin the agent prompt version and opt into full synthetic content only if required. Structured read answers are checked against actual source retrieval; memory mutations are independently read back. A correct-looking answer without evidence fails. Reference controls are never exported as LLM runs.
In Langfuse, open Datasets → Kernkontext → Experiments, then a scenario's Session to inspect model steps and MCP reads/writes. Each scenario has German task/expectation text and named checks. Compare the same dataset/client/model/version; suite_passed requires all eight cases. A published dataset, a scripted control or a partial experiment is not full model acceptance. The independent 36-case plugin catalog and human quality calibration remain separate.
Scenario details and commands · Operator guide.
Repeated product baselines
The external testkit combines the eight core-context and 36 plugin scenarios into a campaign with 2–5 repetitions. pnpm eval:baseline plan lists the contract without model calls; run DIRECTORY makes explicitly requested model calls through the existing bounded profile. Each suite retains its plugin-owned fixtures and graders.
The standalone HTML/JSON/Markdown reports distinguish complete success, violated expectations and missing or aborted executions. A comparison checks dataset/grader identity, client, model, prompt, execution limits and cache policy; a critical failure cannot be hidden by an average. Per-generation gateway usage/cost and serialized request-component bytes explain efficiency without treating bytes as billable tokens. The optional generic on-demand tool catalog is an external-client experiment, not a host capability or a dependency on any domain.
See the repository's docs/plugins/EVALUATION-BASELINES.md for commands, interpretation and evidence limits. Baseline campaigns and synthetic test results do not prove every native client or deployed account behaves identically.
The operator dashboards separate planned coverage, observed failures and measured generation costs. Their source-controlled definitions and verified application procedure are documented in the repository at docs/plugins/LANGFUSE-DASHBOARDS.md. Aggregate summary scores are explicitly separate from case evidence and human judgments.
Tasks format v2 has a separate nine-scenario acceptance dataset (2026-10-01.workflow.4), leaving the historical twelve-case Tasks suite intact. The lease case sends a second real user turn; a single atomic claim+note is not evidence of follow-up coordination. Its grader accepts any successful authorized command with the exact required state and same lease, including task.note and task.status; it does not mandate one tool path. Run deterministic grader controls before spending model budget:
CI=true pnpm test apps/server/src/e2e/oauth-e2e.smoke.test.ts -t 'plugin-owned state grader|workflow lease grader'
PLUGIN_EVAL_SUITE=tasks-workflow PLUGIN_EVAL_CATALOG_MODE=on-demand \
PLUGIN_EVAL_PROMPT_NAME=meinKontext/eval/agent PLUGIN_EVAL_PROMPT_VERSION=1 \
PLUGIN_EVAL_TRACE_CONTENT=true pnpm plugin:eval:localThe second command consumes the dedicated approved EU gateway budget, uses the already-published immutable agent prompt and exports synthetic conversations to Langfuse. Never use that content flag for private production context. Stage scenarios require actual state read-back, not text-only success; they run with Tasks alone. Complement them with the strict format v1/v2 persistence regression, hidden-policy/revocation replay and companion contract matrix. Preserve failed experiments and distinguish a corrected grader, an agent failure and an incomplete/budget-limited execution; see docs/plugins/LANGFUSE-GUIDE.md for the verified results.
Managed-agent acceptance
Experimental actor contract v1 has a separate four-case Agenten-Zugänge und Urheberschaft dataset. Its external fixture uses real scoped MCP credentials and distinguishes owner, authenticated actor, credential and server-bound work session. No delegated subject exists yet. Real-store tests cover revoked/expired access, protected parents, alternate read surfaces, concurrent sessions, replay and forged human answers; Tasks also tests custodian recovery of a stranded lease. Missing plugin adoption fails closed.
Model scenarios check service attribution, owner-answer refusal, same-agent session fencing and lost write access. The first grader has positive and negative real-MCP controls. Run PLUGIN_EVAL_SUITE=managed-agents pnpm plugin:eval:local with the documented pinned prompt/limits only within the approved budget. Implementation and scripted controls do not prove model acceptance. Reproduction commands, current results and limits.