Webhooks When Real-Time
Webhooks suit event-driven flows where the CRM tells you the moment a record changes. Salesforce Platform Events, HubSpot Webhooks v3, Dynamics 365 Service Bus topics, and Zendesk Triggers all fit this pattern. The handler must be idempotent because duplicate deliveries are a feature, not a bug — most platforms guarantee at-least-once delivery, which means at some point you will receive the same payload twice. The handler should verify the signature, deduplicate on the event ID, then process. A 200 response within 5 seconds is the typical SLA the sender expects; anything heavier than that goes onto an internal queue.
// Salesforce Platform Event handler — idempotent shape
async function handle(event) {
const eventId = event.replayId; // unique per event
if (await seen(eventId)) return ok(); // duplicate, drop
await markSeen(eventId, ttl='7d');
await enqueue(event.payload); // hand off to worker
return ok(); // ack within 5s
}Polling for Stability
Polling is the fallback when webhooks misbehave or the source system does not emit events. A cron job that pulls every N minutes against a lastModifiedDate > X filter tolerates flaky networks better than webhooks alone. The right N depends on freshness need: 5 minutes for sales pipeline; 60 minutes for analytics; daily for finance closeout. Always run polling in addition to webhooks for any integration that matters — the belt-and-suspenders approach catches the missed delivery you did not know happened. Track the watermark in a dedicated table, not in code, so a redeploy does not lose state.
Bulk APIs for Volume
Initial loads, full-refresh syncs, and bulk updates require bulk APIs. The Salesforce Bulk API 2.0 handles up to 150 million records per day on a standard org; HubSpot’s batch endpoints accept 100 records per call; Dynamics 365 supports OData $batch with up to 1,000 changes per request. Per-record APIs at volume produce rate-limit failures, retry storms, and integration-user lockouts. The break-even is roughly 5,000 records — below that, single-record calls with a worker pool are fine; above, switch to bulk. Plan for the bulk job’s monitoring story: most platforms return a job ID and require polling for status, not a webhook on completion.
Idempotency
Every integration must tolerate replay. Use external IDs (Salesforce external ID fields, HubSpot unique value properties) rather than internal record IDs as the key. Upsert is the default operation, not create — a PATCH /Account/External_Id__c/ACC-12345 with the desired state succeeds whether the record exists or not. Assume at-least-once delivery on every queue and every webhook; design the downstream so receiving the same message twice produces the same end state.
Error Handling
Retry with exponential backoff (typical: 1s, 4s, 16s, 64s, 256s, then dead-letter). Dead-letter queues hold terminal failures for human inspection. Alert on DLQ growth above a threshold — silent failure is the worst failure because it manifests as quiet data divergence weeks later when an exec asks why two reports disagree. Log every retry attempt with the failure reason; HTTP 401 is a credential rotation issue, 429 is rate limiting, 500 is a partner outage, and the response strategy differs.
Common Failure Modes
The recurring failures: trusting webhook delivery alone, building per-record sync loops that run head-first into the API rate limit during quarter-end, missing the watermark table after a deploy, and treating idempotency as a “we’ll add it later” item. The most expensive of these is the missing DLQ — a CRM integration silently dropping 3% of records will not be discovered until the next forecast review goes sideways.
What to do this week
Run a query against your DLQ table (or stand one up if it does not exist). If the result is empty for an integration that has been running for a year, that is suspicious — silent successes do not exist at scale. Investigate whether failures are being logged at all.
