Playbooks
Analysing Customer Reviews at Scale with AI: Causes and a Fix List
Turn reviews into checkable data, let AI apply fixed labels with evidence, then rank fixes by frequency, impact and how much you control.
Your reviews usually already contain the next round of product fixes, the next support FAQ and the next ad angle. But dumping a few thousand of them into a model and asking it to "summarise" produces exactly the sentences you can't act on: quality is good, shipping needs improvement, customers want more colours.
The better route is to turn reviews into structured data first, let AI do theme labelling and evidence collection, and let the team rank the results by frequency, impact and how much you actually control.
Decide what question you're answering
The same batch of reviews can support very different decisions. Pick one before you start:
- why low-star reviews are increasing;
- which problems cause returns;
- the reasons customers say they bought;
- which parts of the product description are unclear;
- which feature requests deserve a place in the next version;
- which customer phrases could become product page or ad language.
Ask a model to do product, support, advertising and market research at once and the themes become too broad to verify.
Step 1: build the minimum data table
Export from the platform and keep only the fields the analysis needs:
| Field | Example |
|---|---|
| review_id | Platform review identifier |
| sku | Product or variant |
| rating | 1–5 |
| review_date | Date posted |
| review_text | Body |
| country / language | Market and language |
| verified_purchase | Platform's purchase flag |
| return_reason | If your systems can join it |
Strip names, emails, order numbers, addresses and phone numbers. Before putting anything into an external AI tool, confirm your company's data policy and the service's data settings; if you can't confirm them, work on a redacted sample or in an approved environment.
Step 2: clean before you analyse
Review exports routinely contain duplicates, empty text, templated incentivised reviews, mixed languages, and ratings that contradict the text.
Do this first:
- de-duplicate on
review_idand on body text; - count empty and emoji-only reviews separately and exclude them from theme analysis;
- keep the original text; put translations in a separate column;
- group by SKU, market and month so different products don't merge into one conclusion;
- flag "five stars but complaining" for human review instead of trusting the star rating as sentiment.
Step 3: ask for fixed labels, not prose
Take a sample of 100–200 reviews and have a human define the first label set, for example:
- size doesn't match;
- material not as expected;
- difficult to assemble;
- damaged in transit;
- missing parts;
- product page unclear;
- positive: easy to use, portable, appearance, value.
Then run a fixed-format pass:
You are a customer review analyst. Label strictly from the review text; do not infer causes that aren't written.
For each review output: review_id, primary theme, secondary theme, sentiment, product part involved,
whether human review is needed, and the phrase supporting your judgement.
Use the supplied labels first; if nothing fits, mark new_theme. Do not merge labels on your own.
The supporting phrase matters. Without an evidence snippet, nobody can spot-check whether the model mislabelled. Keep only the short quotes you need for internal verification — don't republish large amounts of customer text.
Step 4: rank on three dimensions
The most frequent theme isn't automatically the most worth fixing. Score each one:
- Frequency: share of the relevant SKU's reviews;
- Impact: does it link to one-star reviews, returns, refunds or safety;
- Controllability: can the team improve it via product, packaging, page or support.
A problem appearing in 5% of reviews that makes the product unusable outranks a 20% colour preference. A simple formula is enough:
priority = frequency score x impact score x controllability score
The score orders the list; it doesn't replace a decision-maker. Safety and compliance issues escalate immediately, regardless of rank.
Step 5: convert themes into actions
Every high-priority theme gets an owner and a verification metric:
| Review theme | Action | Owner | Verification metric |
|---|---|---|---|
| Size misunderstood | Add a size comparison graphic and measuring guide | Product page | Related enquiries and return rate |
| Hard to assemble | Reshoot a 60-second assembly video | Content | Share of assembly-related negatives |
| Missing parts | Add packing scan and contents card | Warehouse | Missing-item ticket rate |
| Delivery time misread | Rewrite shipping copy and checkout notice | Ops | Shipping enquiry volume |
That's what stops the analysis ending as a nice report nobody uses.
Step 6: close the loop with support and the product page
Push high-frequency questions into the FAQ, then run them through the knowledge-base-plus-AI-draft-plus-human-approval pattern in how small Shopify teams automate customer support with AI. Product page problems go back into the fact templates in turning one product photo into a full Amazon and Etsy listing.
Re-run the same labels monthly and watch whether the share of a theme falls after a change. Don't delete old reviews to make the trend look better — trend analysis needs a stable baseline.
Quality checklist
- personal information and unnecessary order data removed;
- SKUs, markets and languages not incorrectly merged;
- label definitions and examples confirmed by a human;
- every theme retains a checkable evidence phrase;
- random spot-checks on high-impact themes and on
new_theme; - frequency uses the relevant review count as the denominator;
- every conclusion has an owner, an action and a verification metric;
- AI output does not stand in for product safety or compliance judgement.
AI is good at turning a lot of text into a checkable structure. The original value comes from your own SKU data, evidence and the results of the changes you make. Recording "what we changed and whether the metric moved" beats generating one more review summary.
Method compiled 2026-07.