Integration patterns that actually scale
Webhooks, queues, ETLs, reverse ETLs — picking the wrong pattern costs more than picking the wrong vendor. Here's the decision tree we use.
Most integration projects fail quietly. The MVP ships, the demo works, and then — three months in — events start dropping, the queue backs up, and someone discovers that the 'real-time sync' has been running an hour behind for two weeks. Nobody noticed because there was no alert. There was no alert because the pattern was wrong from day one.
Picking the right integration pattern isn't a technical detail — it's an architecture decision that determines what kinds of failures you'll have, and whether you'll find out about them.
Webhooks: real-time, low-volume, idempotent
Webhooks are the right answer when you need to react to an event within seconds, the volume is manageable (less than a few hundred per minute sustained), and the operation is safe to retry. They fail when you treat them as fire-and-forget. The internet drops packets; receivers go down for deploys; senders give up after one retry.
Non-negotiables for webhooks in production:
- Signature verification on every payload — never trust the sender's word
- Idempotency keys so duplicate deliveries don't double-charge or double-ship
- A retry queue on the receiving side that catches and replays failures
- An alert when delivery lag exceeds your SLA, not just when delivery stops
Queues: ordering, bursts, decoupling
When the order of events matters, or when you need to absorb traffic spikes without dropping work, queues are the right shape. They decouple the producer's pace from the consumer's pace — the producer can fire as fast as it wants while the consumer drains at a sustainable rate.
The trap with queues is unbounded growth. A queue that fills faster than it drains looks healthy for a while, then suddenly isn't. Set max-depth alerts at 50% of capacity, not 95%. By the time you hit 95%, you have 30 minutes to fix a problem that took weeks to develop.
ETL: batch is fine, actually
Not everything needs to be real-time. If you're moving operational data into a warehouse for analysis, a nightly ETL is simpler, cheaper, and more reliable than a streaming pipeline — and your dashboards will look the same in the morning. Real-time is a cost, not a feature.
Reverse ETL: warehouse to operational tools
The newer pattern: pushing data from the warehouse back into operational tools (CRM, marketing automation, support). Useful when analytical context — lifetime value, churn risk, propensity scores — needs to be visible to the people doing day-to-day work. Treat the warehouse as the source of truth and the operational tool as a cache.
"Every integration is a contract. The pattern you pick is what happens when the contract is broken."
The decision in one sentence
If the answer to 'what happens if this fails silently for an hour?' is 'we lose money or trust,' you need monitoring and retries — regardless of pattern. If the answer is 'nothing,' you can be sloppier than the architecture diagrams suggest. Most teams are too sloppy in the first case and too rigorous in the second.