[object Object]

Hallucination Risk

Generated content makes claims with the confidence of an authoritative source. Customer-data hallucinations — wrong order number, fictional policy clause, invented refund window, misquoted product spec — convert quickly from quality issue to legal exposure. The 2024 Air Canada case (the carrier was held to a chatbot’s invented bereavement-fare policy) is the canonical precedent: courts treat AI-generated commitments as company commitments.

Mitigations that work:

  • Ground every factual claim in retrieved context (RAG with provenance).
  • Refuse to answer when retrieval confidence is below threshold.
  • Add an LLM-as-judge step that checks generated text against the source documents before send.
  • For high-stakes content, human review gate.
  • Log the source material that produced each output for audit and dispute.

Verification before customer-facing send is non-negotiable for transactional and policy content. Marketing and educational content tolerates more model autonomy.

Tone Drift

Brand voice erodes when AI-generated content goes unreviewed. “Feels AI-generated” — formulaic structure, predictable transitions, hedged conclusions, em-dash overuse — is a death signal for outbound communications. Open rates and reply rates on AI-flavored sales emails collapsed measurably across 2025 cohorts (Lavender and Apollo data both show 30–50% degradation when readers detect AI patterns).

Mitigations:

  • Style guide encoded in the system prompt with concrete do/don’t examples.
  • Few-shot examples of approved past content.
  • Periodic spot-check by editorial — sampled review of the worst-performing 5% of outputs.
  • Human editorial review for higher-stakes content categories (executive communications, regulated disclosures, large-account messaging).
  • Monitor reply rate, unsubscribe rate, and customer-effort signals as tone proxies.

Brand Damage

Off-color, off-brand, or factually embarrassing outputs that slip through. The DPD chatbot writing a poem critical of its own employer (January 2024), the Chevrolet dealership chatbot agreeing to sell a Tahoe for $1, the airline chatbots inventing routes and prices — each cost real money and meaningful brand equity. Brand reputation repair takes years; prevention is cheap in comparison.

Defensive layers:

  • Content moderation pre- and post-generation (Anthropic’s safety classifiers, OpenAI’s moderation API, Azure Content Safety, Lakera Guard).
  • Refusal patterns for off-topic, sensitive, or escalating user inputs.
  • Topic constraints limiting what the agent will discuss.
  • Rate-limited public deployment after launch, scaling only after baseline shows clean.

Regulatory Exposure

Specific jurisdictions require disclosure of AI-generated content. EU AI Act Article 50 mandates that users know they’re talking to AI for certain interaction types; Article 50(2) requires AI-generated content to be marked machine-readable as such. California’s BOT Disclosure Law (SB-1001) has been on the books since 2019. Korea’s PIPA, China’s deep-synthesis rules, and the FTC’s enforcement on undisclosed AI endorsements all add layers.

Email-specific: the 2024 Yahoo/Google bulk-sender requirements care about compliance posture; mass AI-generated sends without quality controls can trip spam filters at scale. Compliance review before automating high-volume content generation.

Common Failure Modes

  • Treating “the model said it” as a defense in customer disputes — courts don’t accept this.
  • No provenance trail; can’t reconstruct what the AI said and what data it had.
  • Style guide as document, not as system prompt — humans read it; the model never sees it.
  • Approval workflow that’s a checkbox auto-clicked — meaningfulness fails.
  • Disclosure language buried in legal footer — fails Article 50’s “clear and distinguishable” test.

What to Do This Week

Pull a sample of last week’s AI-generated customer emails. Read 20 with fresh eyes. Note every unsupported claim. That’s your remediation list.

[object Object]
Share