How to QA Product Recommendation Accuracy Before Going Live on a Shopify Store

The short answer: run structured test cases against your live catalog before any customer sees a recommendation

A recommendation engine is only as good as what it surfaces. If it suggests an out-of-stock item, ignores a shopper's budget, or picks the wrong category, you lose the sale — and often the customer. QA for product recommendations means deliberately probing the system with realistic buyer inputs and verifying that what comes back is accurate, available, and appropriate. This guide walks through how to do that systematically.

What is product recommendation accuracy?

Product recommendation accuracy is the degree to which a recommendation engine surfaces items that match the buyer's stated intent, budget, and context — while only drawing from products that are actually in stock and available for purchase. It is distinct from relevance (which is subjective) and from CTR (which is a downstream metric). Accuracy is measured before launch, in a controlled QA environment.

In a server-side recommendation architecture — like the one SmartBrain uses, where the server queries the real catalog and enforces constraints before the AI writes any copy — accuracy failures show up as catalog mismatches rather than hallucinations. That distinction changes how you test.

Why recommendation QA fails in practice

Most teams skip structured QA because the demo looked fine. The demo used one or two products, a clean budget, and a category the system knows well. Real buyers are messier: they have edge-case budgets, they ask for discontinued SKUs by name, they cross category lines. Common failure modes include:

• Out-of-stock leakage — the engine recommends an item that was in inventory during setup but is now depleted
• Budget ceiling violations — a shopper says "under $40" and receives a $47 item because the filter was applied at display time, not query time
• Category bleed — a query for "running shoes" surfaces hiking boots because the catalog taxonomy is flat or ambiguous
• Null results with no fallback — no matching product exists and the system returns nothing, crashing the conversation
• Variant confusion — the correct product is recommended but the wrong size or color variant is surfaced

How to build a QA test suite for product recommendations

Step 1 — Define your buyer persona test cases

Write at least 20 test inputs before you run a single test. Cover these categories:

• Happy path: clear category, reasonable budget, item clearly in stock (e.g., "I want a moisturizer under $30 for dry skin")
• Edge budget: budget exactly at or just below a price threshold (e.g., "under $25" when your cheapest item is $24.99)
• Out-of-stock trap: a product you know is depleted, named explicitly or by type
• Cross-category ambiguity: inputs where two categories could match ("gift for a runner" — apparel or gear?)
• Zero-result scenario: a query that should return nothing, to verify graceful handling
• Multi-constraint query: budget + category + attribute at once (e.g., "vegan lip balm under $15 in a travel size")

Step 2 — Define a pass/fail standard before you test

Without a pre-defined standard, QA becomes subjective. A workable baseline for Shopify stores:

• 100% of recommendations must be in-stock at the moment of the test
• 100% of recommendations must fall within the stated budget (no rounding up)
• At least 90% of recommendations must be in the correct primary category
• Zero-result queries must trigger a defined fallback (a redirect, a category browse, or a "tell me more" prompt)
• Variant specificity: if the buyer specifies a size, the correct variant must be pre-selected, not defaulted to the first available

Document these before the first test run. Agencies using SmartBrain for client stores typically lock in the pass criteria during the briefing call, so there is no negotiation after results come in.

Step 3 — Run tests against the live catalog, not a snapshot

This is where most teams go wrong. Testing against a product export from last Tuesday means you are not testing the system customers will actually experience. Run every test case against the live Shopify catalog, during a time window that reflects real inventory levels. If your store has flash sales, run a subset of tests during one.

Step 4 — Audit the recommendation output, not just the conversation

Log the actual product ID, variant ID, price, and stock level returned for each test case. Do not rely on visual inspection of the chat UI. A well-architected system — again, server-decides engines like SmartBrain make this straightforward because the catalog query is separate from the copy generation — will let you pull structured output logs per session. Compare those logs against your pass/fail criteria row by row.

Server-decides vs. AI-decides: a key architectural comparison

Two broad architectures exist for conversational product recommendations. Understanding which one you are testing changes your QA focus.

• AI-decides: the language model chooses which product to name based on training data and the conversation. Accuracy failures here include hallucinated products, wrong prices, and items that exist in training data but not in your catalog. QA must include hallucination checks.
• Server-decides: the server queries the real catalog with hard constraints (price, stock, category), then passes the result to the AI which only writes the copy. Accuracy failures are catalog or filter logic failures — no hallucinations possible. QA focuses on filter correctness and fallback handling.

The server-decides model is significantly easier to QA because failures are deterministic and reproducible. SmartBrain operates on this principle: the engine never asks the AI to pick a product; it picks the product itself and hands it off. That means a failed test is a bug you can fix, not a probability you have to manage.

Red flags that mean you are not ready to go live

• Any out-of-stock item appearing in recommendations during the test window
• Budget violations on more than 0% of budget-constrained test cases
• No documented fallback for zero-result scenarios
• Variant defaults that ignore buyer-specified attributes
• Recommendation logs that do not include product IDs (you cannot audit what you cannot trace)

FAQ

How many test cases do I need before going live?

A minimum of 20 structured cases covering happy path, edge budget, out-of-stock, zero-result, and multi-constraint scenarios. For stores with more than 500 SKUs or complex variant structures, 40–60 cases is a more realistic floor.

Should I QA again after a catalog update?

Yes. Run a targeted subset — at minimum the out-of-stock and budget-edge cases — after any bulk product import, price change, or sale configuration. These events are the most common trigger for recommendation drift.

What is the difference between QA and A/B testing for recommendations?

QA is pre-launch and binary: the recommendation is either accurate or it is not. A/B testing is post-launch and comparative: it measures which of two accurate approaches drives better conversion. Do not use A/B testing as a substitute for QA — you cannot optimize accuracy by splitting traffic on a broken baseline.

Can I automate product recommendation QA?

Partially. You can automate catalog-state checks (stock, price, variant availability) and log parsing. The conversational input layer still benefits from human review, particularly for ambiguous or multi-constraint queries where the "correct" answer requires judgment. A hybrid approach — automated catalog assertions, human review of edge cases — is the practical standard.

How do agencies handle QA across multiple client stores?

Agencies running SmartBrain across multiple Shopify clients typically maintain a shared test case template with a client-specific catalog layer. The pass/fail criteria stay constant; the product IDs and categories are swapped per store. This reduces QA time per new client to under two hours once the template is established.