Freshness
When did this data last update? A CRM lead score stale by 3 days misleads reps. Track freshness at column-level for critical fields. Alert when stale.
Define SLAs per field, not per table. Lead score may need a 4-hour SLA; account hierarchy may tolerate 24 hours. Monte Carlo, Bigeye, and Soda all support column-level freshness checks. Implement with MAX(updated_at) queries scheduled via Airflow or dbt-cloud, with PagerDuty routing for breaches over 2x SLA. Tag critical CRM fields (opportunity_amount, lead_score, account_owner_id) with a freshness_critical=true metadata flag so dashboards filter to what reps actually consume.
Quality
Completeness (required fields populated), accuracy (values match business rules), uniqueness (no unintended duplicates). Sample and validate continuously. Quality drops silently; monitoring catches it.
Use Great Expectations or dbt tests for declarative validation: expect_column_values_to_match_regex for emails, expect_column_values_to_be_in_set for stages, expect_compound_columns_to_be_unique for natural keys. Run on every load with row-level failure logging to a data_quality_failures table. Anomaly detection (z-score on row counts, null-rate trend) catches the 30% of failures that pass schema validation but break business logic — for example, a sudden spike of opportunity.amount=0 after a Salesforce permission change.
Schema Drift
Upstream system adds a column. Pipeline ignores it silently. New data isn’t captured. Schema registry + drift detection prevents silent degradation.
Confluent Schema Registry, Atlan, or DataHub track Avro/JSON schemas with semver. Fivetran and Airbyte both emit schema_change events you can route to Slack. Distinguish additive drift (safe, auto-merge) from breaking drift (column removed, type narrowed) — the latter pages the on-call. Pin a schema contract between source and destination teams; require a PR to amend it. Most production CRM data incidents trace to silent drift, not pipeline failures.
Cost
Data pipelines consume compute. Warehouse bills climb. Attribute cost per pipeline, per team. FinOps for data just like AI.
Snowflake QUERY_HISTORY and BigQuery INFORMATION_SCHEMA.JOBS expose per-job bytes scanned and credit burn. Tag warehouses with team labels; chargeback monthly. Common wins: convert full reloads to incremental, partition large fact tables on event_date, and kill ad-hoc SELECT * against Account dumps. A single mis-clustered Salesforce sync can burn $4K/month before anyone notices.
Common Failure Modes
Five recurring patterns. Timezone mismatch making data appear stale (UTC vs local). API rate limits causing partial syncs marked successful. Soft-delete fields not propagated, leaving zombies in CRM. Webhook backpressure during outages. Backfills overwriting fresher downstream edits — always check system_modstamp before upsert.
What to Do This Week
Add column-level freshness checks for your top five rep-facing fields and route breaches to a dedicated Slack channel.
