Step 6 · Evaluate your vendors

Available now by hand. The goal is a comparison on evidence, not on

feature lists, that you can revisit when SOM or a product changes.

The evidence

Evidence	From	What it proves	Status
A results statement shared with your organisation	The vendor, from its complete test runs: consumer scenarios and skill cases, graded by the bus against a named suite	Exactly which cases passed, in fixed words, until a date the vendor chose. See Results statements	Available now
A readiness write-up	The vendor, from the readiness checklist	The producer checks the bus can't grade by itself, ticked or explained	Available now
Your house's counts	The portal: accepted, refused and duplicate messages per vendor connection	How the product behaved in your house, in numbers	Available now
Your run sheets	Your scenario runs from step 4	What each product did in your workflows, signed off by your people	Available now
The coverage matrix	The portal, from statements shared with you and your house's traffic	Every app in your house against the message families, consumer scenarios and skill cases, with every gap	Available now: Reading the coverage matrix

Also ask each vendor for its SOM roadmap: what it doesn't support yet, and when.

A vendor that shares a statement with you before it connects has it shown beside its connection request, so the evidence is there when you decide what to grant.

Scorecard

One per vendor product. Score each line pass, partial, fail or n/a, and link the evidence.

Area	Check	Evidence
Validity	Every message the product sent in your runs was accepted, or refused only where the scenario meant it	House counts; the vendor's own activity
	Negative cases can't be produced through the product's screens or API	Readiness write-up
Snapshots (story owners)	Whole snapshots, never deltas; sequence and updated_at right across restarts	Timeline; S5
	Killed stories stay killed; reopening is handled	S2
Linking	correlation_id shared across a story's families; causation_id on every reaction	Raw JSON on the timeline
Resilience	Safe retries with the same message_id; no retries on 4xx	Readiness write-up; ask for a demo
	Catches up in order after an outage	S6; statement
Receiving	Snapshots replace, never merge; highest sequence wins; duplicates dropped	Statement (consumer scenarios); S1, S2
	Unknown types, fields, extensions and newer 1.x versions ignored	Statement; S7
Governance	Honours editorial gates and standards holds	S4
	Warnings valid, traceable, at the right severity (skill executors)	Statement (skill cases); Skills in your scenarios
Your skills	Covers the skills your house runs, with your values	Which executors cover your skills
Operations	system_id configurable; credentials in a secret store; clear errors for users	Configuration review
Honesty	Results described as "passes som-bus suite som-1.0.0+lib-0.2.2", with gaps stated; no "certified" or "compliant" claims	Write-up, marketing

Reading the results

• A refusal isn't automatically a failure. Ask for its source. schema, conformance and sequence mean the product broke SOM; policy usually means set-up: grants, credentials, ownership, size. See Verdicts.
• Partial is normal early on. What matters is whether the gap is known, stated and on the roadmap.
• Pending spec isn't a failure. Cases that need a message type SOM 1.0 doesn't define are shown but never counted: see Topics without a 1.0 schema. They're open questions for the SOM working group, and every vendor faces them.
• A statement names exactly what passed. A case left out of it is a gap, even if the vendor ran it: ask for a statement that includes it.
• Re-test on change. Record the suite, the product version and the date on every scorecard. A new release or a new SOM 1.x version means a new run of the relevant scenarios.

Sharing results

Scorecards are yours. A vendor's statement is shared with your organisation until the date it chose, and it can revoke it. If you share a vendor's results outside your organisation, ask the vendor first.

Next: Step 7 · Go live.