Product expert: every GoodQA feature, API, tool, command, setting and page, searchable, with honest status

Status: on by default (platform setting mcp.productExpert, read only; it rides the agent-ops block, so mcp.agentOps must be on too). Off: none of the four tools is registered and the MCP tool list is exactly what it was.

QA Commander, the coordinator and any agent connected to GoodQA are product experts as well as assistants: they look up how GoodQA works instead of guessing. Everything GoodQA offers is in one versioned product pack:

Kind	What is in it	From
API operations	every route the server registers and every documented operation: method and path, what it is for, who may call it (session only, or the key family), the user-token abilities, and whether it is on by default	the served and all-features OpenAPI documents and the router itself
MCP tools, prompts, resources	every tool of every group (remote and stdio), with the switch that registers its group	the MCP register functions
CLI and slash commands	every `goodqa` command and every agent-harness slash command	the CLI's own help and the plugin's commands
Settings and flags	every platform setting, org entitlement and deployment switch, with its default and who turns it on	the settings registry, the entitlements and the server configuration
Pages	every web page: what you do there and how to reach it	the web app's routes and its navigation catalog
Systems	QA Commander, its harnesses and desktop, the event bridge, follow-ups and the follow-up graph, escalation, Connect, private networks, Machines, the apps and extension, the vault and secret requests, custom data, CRM, outreach, Discord, Slack, GitHub, agent knowledge, the coordinator, agent launches, agent chat, user tokens, notes to self, billing, MCP, the CLI, the harness plugin, discovery, testing, issues	a hand-written layer whose status is read from the switches' defaults in code
How-to and why-can't-I	step-by-step answers naming the page, the API operation, the MCP tool and the CLI command	the hand-written layer
Permissions	org keys, key families, runner keys, persona keys, user tokens and abilities, roles, session-only routes, who turns switches on	the auth policy and the hand-written layer
Docs and what is new	every public docs page and agent skill (by section), the changelog and the What's New cards	the docs pages, skills, changelog and cards

Every entry carries an honest status for a default organization: live (on by default), needs_admin (someone must set it up first: connect an account, run setup), needs_config (the deployment is missing configuration; its routes answer 501), dark (off by default; a GoodQA platform admin turns it on), coming_soon (not in this build), or internal (GoodQA's own operation). Whether a switch is on for YOUR organization comes from your own capabilities at call time, never from the pack. The pack carries no organization data, secrets, environment values, internal ticket numbers or people's names; the same pack serves every organization.

How agents use it

MCP (POST /v1/mcp):
- product_search {query, kind?, limit?}: the best matches, each with its status and source.
- product_describe {name, kind?}: one route, tool, setting, page, system, flow or command in full, related entries, and forYourOrg (read from the caller's own GET /v1/capabilities when the entry has switches).
- product_whats_new {since?, limit?}: what shipped, newest first.
- product_how_do_i {task, limit?}: the best how-to (steps, page, API, tool) plus alternatives; also "why can't I".
CLI: goodqa product search "<words>", goodqa product describe <name>, goodqa product how-do-i "<task>", goodqa product whats-new --since YYYY-MM-DD, and goodqa help <topic> (the matching commands plus the how-to).
Slash commands (agent-harness plugin): /goodqa-product <question> and /goodqa-help <topic>.
QA Commander and coordinator turns carry the compact PRODUCT MAP (about 6,000 tokens: systems, how-tos, where things are, tool groups, commands, API areas, switches that are not on, permissions) after their own rules, and their rules say: you are the product expert; look it up before answering a product question or guessing an API; never invent a feature; say "dark" or "not enabled for your organization" honestly.
Persona memory: a persona with memory gets the full corpus loaded into its own agent knowledge (about 50 rows tagged goodqa-product, each with provenance: the pack version, the GoodQA build that served the load and the source files) and refreshed automatically when the pack version changes (and re-checked hourly, so rows someone erased come back). The persona finds them with memory_search; the per-turn memory packet keeps recalling the persona's own notes only, so product text never crowds them out.

Answers are documentation, not permission: every action still goes through its own route, which decides access.

Freshness

The pack is generated from the sources above by a deterministic build, and a guard fails the build when:

a route, MCP tool, setting, entitlement, deployment switch, CLI command, slash command or web page exists without a pack entry (or a page without a description);
the pack is stale against its sources (a content and sources-hash check);
the pack carries a secret shape, an environment value, an unknown host, an organization id, an internal ticket number, a release-train name or a person's name;
the map, a chunk or the pack is over its size bound.

Every release regenerates the pack and checks it before promotion, so the tools describe the build they ship in.

Eval

A deterministic, model-free retrieval eval: realistic user and admin questions ("how do I pair Discord", "how do I paste a secret into the remote desktop", "which API creates an issue", "what does machines.enabled do", "why can't my runner key use the CRM"), each with the entries that answer it and the facts the answer must contain. A question passes retrieval when product_search's top 5 contain an expected entry, and passes the fact check when every expected fact is in the text of those top 5.

Set	Questions	recall@5	fact check	Threshold
main (the search was tuned on it)	135	1.000	1.000	recall@5 >= 0.9 and facts >= 0.9
held-out (written after tuning, never tuned on)	25	0.800	1.000	recall@5 >= 0.75

Scores measured 2026-10-05 on the pack this change generates (pnpm --filter @bugtracker/api exec tsx test/product-expert-eval/score.ts prints them). The held-out misses are honest ones: a question about "who decides whether coordinator is on" retrieved permission-adjacent docs, not the entitlement; "the list of agent chat tools" retrieved one agent chat tool but not the system entry.

Reference

API reference: the product expert adds no REST route; its tools describe the operations in /v1/openapi.json and the all-features document.
MCP tools: product_search, product_describe, product_whats_new, product_how_do_i in the MCP tool reference; which keys get them: MCP tools.
Agent skill: rule 10 and section 9 of the conversations skill.
Runbook: product pack runbook (regenerate, check, promotion step, troubleshooting); the switch row in feature switches.

Product expert: every GoodQA feature, API, tool, command, setting and page, searchable, with honest status

Kind

What is in it

From

API operations

every route the server registers and every documented operation: method and path, what it is for, who may call it (session only, or the key family), the user-token abilities, and whether it is on by default

the served and all-features OpenAPI documents and the router itself

MCP tools, prompts, resources

every tool of every group (remote and stdio), with the switch that registers its group

the MCP register functions

CLI and slash commands

every goodqa command and every agent-harness slash command

the CLI's own help and the plugin's commands

Settings and flags

every platform setting, org entitlement and deployment switch, with its default and who turns it on

the settings registry, the entitlements and the server configuration

Pages

every web page: what you do there and how to reach it

the web app's routes and its navigation catalog

Systems

QA Commander, its harnesses and desktop, the event bridge, follow-ups and the follow-up graph, escalation, Connect, private networks, Machines, the apps and extension, the vault and secret requests, custom data, CRM, outreach, Discord, Slack, GitHub, agent knowledge, the coordinator, agent launches, agent chat, user tokens, notes to self, billing, MCP, the CLI, the harness plugin, discovery, testing, issues

a hand-written layer whose status is read from the switches' defaults in code

How-to and why-can't-I

step-by-step answers naming the page, the API operation, the MCP tool and the CLI command

the hand-written layer

Permissions

org keys, key families, runner keys, persona keys, user tokens and abilities, roles, session-only routes, who turns switches on

the auth policy and the hand-written layer

Docs and what is new

every public docs page and agent skill (by section), the changelog and the What's New cards

the docs pages, skills, changelog and cards

How agents use it

MCP (POST /v1/mcp):

product_search {query, kind?, limit?}: the best matches, each with its status and source.
product_describe {name, kind?}: one route, tool, setting, page, system, flow or command in full, related entries, and forYourOrg (read from the caller's own GET /v1/capabilities when the entry has switches).
product_whats_new {since?, limit?}: what shipped, newest first.
product_how_do_i {task, limit?}: the best how-to (steps, page, API, tool) plus alternatives; also "why can't I".

CLI: goodqa product search "<words>", goodqa product describe <name>, goodqa product how-do-i "<task>", goodqa product whats-new --since YYYY-MM-DD, and goodqa help <topic> (the matching commands plus the how-to).

Slash commands (agent-harness plugin): /goodqa-product <question> and /goodqa-help <topic>.

QA Commander and coordinator turns carry the compact PRODUCT MAP (about 6,000 tokens: systems, how-tos, where things are, tool groups, commands, API areas, switches that are not on, permissions) after their own rules, and their rules say: you are the product expert; look it up before answering a product question or guessing an API; never invent a feature; say "dark" or "not enabled for your organization" honestly.

Persona memory: a persona with memory gets the full corpus loaded into its own agent knowledge (about 50 rows tagged goodqa-product, each with provenance: the pack version, the GoodQA build that served the load and the source files) and refreshed automatically when the pack version changes (and re-checked hourly, so rows someone erased come back). The persona finds them with memory_search; the per-turn memory packet keeps recalling the persona's own notes only, so product text never crowds them out.

Answers are documentation, not permission: every action still goes through its own route, which decides access.

Freshness

The pack is generated from the sources above by a deterministic build, and a guard fails the build when:

a route, MCP tool, setting, entitlement, deployment switch, CLI command, slash command or web page exists without a pack entry (or a page without a description);

the pack is stale against its sources (a content and sources-hash check);

the pack carries a secret shape, an environment value, an unknown host, an organization id, an internal ticket number, a release-train name or a person's name;

the map, a chunk or the pack is over its size bound.

Every release regenerates the pack and checks it before promotion, so the tools describe the build they ship in.

Eval

Set

Questions

recall@5

fact check

Threshold

main (the search was tuned on it)

135

1.000

recall@5 >= 0.9 and facts >= 0.9

held-out (written after tuning, never tuned on)

0.800

1.000

recall@5 >= 0.75

Reference

API reference: the product expert adds no REST route; its tools describe the operations in /v1/openapi.json and the all-features document.

MCP tools: product_search, product_describe, product_whats_new, product_how_do_i in the MCP tool reference; which keys get them: MCP tools.

Agent skill: rule 10 and section 9 of the conversations skill.

Runbook: product pack runbook (regenerate, check, promotion step, troubleshooting); the switch row in feature switches.

Product expert: every GoodQA feature, API, tool, command, setting and page, searchable, with honest status

Kind	What is in it	From
API operations	every route the server registers and every documented operation: method and path, what it is for, who may call it (session only, or the key family), the user-token abilities, and whether it is on by default	the served and all-features OpenAPI documents and the router itself
MCP tools, prompts, resources	every tool of every group (remote and stdio), with the switch that registers its group	the MCP register functions
CLI and slash commands	every `goodqa` command and every agent-harness slash command	the CLI's own help and the plugin's commands
Settings and flags	every platform setting, org entitlement and deployment switch, with its default and who turns it on	the settings registry, the entitlements and the server configuration
Pages	every web page: what you do there and how to reach it	the web app's routes and its navigation catalog
Systems	QA Commander, its harnesses and desktop, the event bridge, follow-ups and the follow-up graph, escalation, Connect, private networks, Machines, the apps and extension, the vault and secret requests, custom data, CRM, outreach, Discord, Slack, GitHub, agent knowledge, the coordinator, agent launches, agent chat, user tokens, notes to self, billing, MCP, the CLI, the harness plugin, discovery, testing, issues	a hand-written layer whose status is read from the switches' defaults in code
How-to and why-can't-I	step-by-step answers naming the page, the API operation, the MCP tool and the CLI command	the hand-written layer
Permissions	org keys, key families, runner keys, persona keys, user tokens and abilities, roles, session-only routes, who turns switches on	the auth policy and the hand-written layer
Docs and what is new	every public docs page and agent skill (by section), the changelog and the What's New cards	the docs pages, skills, changelog and cards

How agents use it

MCP (POST /v1/mcp):
- product_search {query, kind?, limit?}: the best matches, each with its status and source.
- product_describe {name, kind?}: one route, tool, setting, page, system, flow or command in full, related entries, and forYourOrg (read from the caller's own GET /v1/capabilities when the entry has switches).
- product_whats_new {since?, limit?}: what shipped, newest first.
- product_how_do_i {task, limit?}: the best how-to (steps, page, API, tool) plus alternatives; also "why can't I".
CLI: goodqa product search "<words>", goodqa product describe <name>, goodqa product how-do-i "<task>", goodqa product whats-new --since YYYY-MM-DD, and goodqa help <topic> (the matching commands plus the how-to).
Slash commands (agent-harness plugin): /goodqa-product <question> and /goodqa-help <topic>.
QA Commander and coordinator turns carry the compact PRODUCT MAP (about 6,000 tokens: systems, how-tos, where things are, tool groups, commands, API areas, switches that are not on, permissions) after their own rules, and their rules say: you are the product expert; look it up before answering a product question or guessing an API; never invent a feature; say "dark" or "not enabled for your organization" honestly.
Persona memory: a persona with memory gets the full corpus loaded into its own agent knowledge (about 50 rows tagged goodqa-product, each with provenance: the pack version, the GoodQA build that served the load and the source files) and refreshed automatically when the pack version changes (and re-checked hourly, so rows someone erased come back). The persona finds them with memory_search; the per-turn memory packet keeps recalling the persona's own notes only, so product text never crowds them out.

Answers are documentation, not permission: every action still goes through its own route, which decides access.

Freshness

The pack is generated from the sources above by a deterministic build, and a guard fails the build when:

a route, MCP tool, setting, entitlement, deployment switch, CLI command, slash command or web page exists without a pack entry (or a page without a description);
the pack is stale against its sources (a content and sources-hash check);
the pack carries a secret shape, an environment value, an unknown host, an organization id, an internal ticket number, a release-train name or a person's name;
the map, a chunk or the pack is over its size bound.

Every release regenerates the pack and checks it before promotion, so the tools describe the build they ship in.

Eval

Set	Questions	recall@5	fact check	Threshold
main (the search was tuned on it)	135	1.000	1.000	recall@5 >= 0.9 and facts >= 0.9
held-out (written after tuning, never tuned on)	25	0.800	1.000	recall@5 >= 0.75

Reference

API reference: the product expert adds no REST route; its tools describe the operations in /v1/openapi.json and the all-features document.
MCP tools: product_search, product_describe, product_whats_new, product_how_do_i in the MCP tool reference; which keys get them: MCP tools.
Agent skill: rule 10 and section 9 of the conversations skill.
Runbook: product pack runbook (regenerate, check, promotion step, troubleshooting); the switch row in feature switches.