The Integration Layer
Relays
Relays sit at the boundary between 3forge and external systems. Their role is to bring live data into the fabric, and to implement specific architectural patterns for scalability and data processing.
Function inside the fabric
At the integration boundary
Relays connect to real-time feeds, messaging systems, databases, files and local applications through feed handlers and adapters. As information arrives, a Relay can parse, validate and transform it before routing it to one or more Centers. Performing this work at the integration boundary reduces the amount of source-specific logic that must be implemented downstream.
Relays can also provide a controlled route from the Center to external resources. They can issue database queries, upload data, access files and logs, invoke local processes or send commands to downstream applications.
This makes Relays the integration and transport layer of the fabric: close to the source, aware of the message, and responsible for delivering it in the form and direction required by the rest of the system.
Relays are used for the following:
Real-time feeds
Centers and Relays can both connect directly to databases. Relays are specifically designed to also connect to real-time feeds, terminating the protocol and parsing, validating and normalizing each message before any Center sees it.
Scaling patterns
Distributing work across several Centers, balancing load, replicating a stream to a warm standby or holding messages while a destination is down all rest on Relay routing rules. The seven patterns below are that second use, drawn out.
Example patterns
Seven ways Relays are deployed
Relays are not limited to simple message forwarding. Routing rules can determine which messages are sent to which Centers, replicate selected streams to several destinations or discard messages that do not meet defined criteria. This supports distributed processing patterns in which work is mapped across several Centers and the resulting state is subsequently combined.
Dynamic message routing and Map Reduce
- Rules dispatch each message to a specialized processing Center by content, load or custom logic, configurable in real time
- Each Center processes its own share, and a map-reduce operation merges the results
Guaranteed messaging
- Every critical update is durably persisted and delivered, whatever the load or transient network conditions
- Messages are journaled to a write-ahead log, so a Center that was offline can reconnect and replay what it missed
- No duplicate suppression, but the durability and continuity that real-time trading, risk and compliance systems require
Message transformation and routing
- Inspect, modify, enrich or filter messages in-flight, with no custom code
- Built for FIX and other financial protocols, where counterparties differ and formats need normalizing
- Redact sensitive fields or route on content
Load balancing based on routing rules
- Replication load-balances between Centers on the Relay's routing rules
- More Centers, more redundancy and scalability
- Multi-region and global replication
Data replication
- Primary and warm standby Center databases, from a simple configuration
- The standby checks the primary's health and takes over on failure
- All the data is already loaded in memory on the standby
Market data aggregation
- Converges many latency-sensitive sources onto one or several Centers
- For when an aggregated view is the advantage, such as liquidity across multiple execution venues
- Builds level 2 market data depth
Trading data aggregation
- Converges many trading data sources for real-time and historical query
- Order books from many clients, and executions from several venues
- Aggregation of order books across asset classes
Cache, persistence, and processing
Relays are close to the source. Centers are where the state lives.
In-memory SQL over live state, a petabyte-scale historical database, and a complex event-processing engine, all inside the same governed runtime.