Five years ago, enterprise integration meant connecting two or three systems. An ERP to a website. A CRM to an email platform. Today, the average enterprise client we work with has 15-20 systems that need to exchange data, and the number is growing. The integration landscape has become the most complex and least appreciated layer of enterprise architecture.
What You Need to Know
- Enterprise integration complexity is growing faster than most organisations' ability to manage it
- The three patterns that work at scale: API gateway, event-driven messaging, and ETL for batch data
- Point-to-point integrations create an exponential maintenance burden. Every new system multiplies the connections
- Integration testing and monitoring are as important as the integrations themselves
The Complexity Problem
When you have three systems, you have three potential integration points. When you have ten systems, you have 45. When you have twenty, you have 190. The number of potential connections grows exponentially, and every connection is a potential failure point.
Most enterprises have grown their integration landscape organically. System A was connected to System B when the CRM was installed. System C was connected to System A when the reporting tool was added. System D was connected to B and C when the mobile app launched. Nobody drew the full picture because the full picture was never planned. It emerged.
190
potential point-to-point connections between 20 systems
The result is a web of integrations where changing anything is risky because nobody fully understands the downstream effects. Updating an API on one system breaks a process on another system that nobody knew depended on it. A data format change in the ERP causes the reporting dashboard to show incorrect figures for two weeks before anyone notices.
The Patterns That Work
API Gateway
An API gateway sits between your systems and provides a single point of entry for all integration traffic. Instead of System A calling System B directly, System A calls the gateway, and the gateway routes the request to System B.
This adds a layer of indirection, but that indirection provides significant benefits:
Visibility. All integration traffic flows through a single point where it can be logged, monitored, and analysed. You can see exactly what's calling what, how often, and how fast.
Transformation. The gateway can transform data between formats. System A sends dates in one format; System B expects another. The gateway handles the translation without either system needing to change.
Rate limiting and throttling. Protect downstream systems from being overwhelmed by upstream requests. When the batch process that syncs 50,000 records fires, the gateway can throttle the flow to a rate the receiving system can handle.
Authentication. Centralise authentication and authorisation at the gateway rather than implementing it separately in every system.
Tools like Kong, Apigee, and AWS API Gateway make this pattern accessible without building a gateway from scratch. For most NZ enterprise teams, a managed gateway service is the pragmatic choice.
Event-Driven Architecture
Instead of systems calling each other directly (request-response), systems publish events to a message broker, and other systems subscribe to the events they care about.
When a new customer is created in the CRM, the CRM publishes a "customer created" event. The billing system, the email platform, and the reporting system each subscribe to that event and process it independently. The CRM doesn't know or care how many systems are listening. The subscribing systems don't know or care where the event originated.
Event-driven architecture turns a web of dependencies into a collection of independent systems that react to shared information. Adding a new system means adding a new subscriber, not modifying every existing system.
John Li
Chief Technology Officer
This decoupling is the primary benefit. Adding a new system to the landscape means subscribing to relevant events, not building new point-to-point connections to every existing system.
RabbitMQ and Apache Kafka are the two most common message brokers in enterprise contexts. RabbitMQ is simpler and sufficient for most NZ enterprise volumes. Kafka is more complex but handles higher volumes and provides event replay capabilities.
ETL for Batch Data
Not all data needs to flow in real time. Financial reconciliation, reporting aggregation, and data warehouse loading are typically batch processes that run on a schedule.
Extract, Transform, Load (ETL) is the established pattern. Extract data from source systems. Transform it into the target format (cleaning, aggregating, enriching). Load it into the destination.
The key is to keep ETL processes visible and monitored. Scheduled jobs that run overnight and fail silently are one of the most common sources of data quality issues in enterprise. Every ETL process needs alerting on failure, data quality checks on output, and logging that makes troubleshooting feasible.
Integration Anti-Patterns
The File Drop
System A writes a CSV file to a shared folder. System B polls the folder and processes the file. This pattern is remarkably common in enterprise, remarkably fragile, and remarkably difficult to troubleshoot. File format changes, timing issues, partial writes, and permission problems create a steady stream of silent failures.
If you have file-drop integrations, plan to replace them. They're technical debt that compounds.
The Direct Database Connection
System A reads directly from System B's database. This creates a tight coupling between the two systems that makes either one impossible to change independently. When System B restructures its database, System A breaks. When System A's queries are inefficient, System B's performance degrades.
Databases are internal implementation details. Expose data through APIs, not through direct database access.
The "We'll Document It Later" Integration
An integration built quickly to solve an urgent problem, without documentation, without error handling, without monitoring. It works fine until it doesn't, and when it doesn't, nobody knows how it works or what it's supposed to do.
Every integration needs documentation of what it does, what data it moves, what format it expects, and what happens when it fails. This isn't optional. It's the difference between a manageable integration landscape and an unmanageable one.
Monitoring and Observability
Integration monitoring is not optional. Every integration needs:
Health checks. Is the integration running? When did it last succeed? When did it last fail?
Data quality checks. Is the data that's flowing accurate and complete? Are there records that failed to transfer? Are there transformations producing unexpected results?
Latency tracking. How long are integrations taking? Is performance degrading over time? Are there bottlenecks?
Alerting. When an integration fails, someone needs to know immediately. Not when a user reports that the dashboard is showing stale data three days later.
The integration landscape is the circulatory system of enterprise IT. When it works, nobody notices. When it doesn't, everything stops. Invest in monitoring proportional to that criticality.