Leveraging Cloud Technology as Trading Infrastructure

Trading systems have long exploited new technologies to give market participants advantages in understanding, strategy, and timing to improve outcomes. The increasing exploitation of latency reductions for market advantage has made co-location with major venues critical — leading market participants to make increasing capital expenses for on-premise and co-lo infrastructure.

For much of IT, the major trend in the second decade of the century is the move to leverage major cloud service providers to provide flexibility, innovation, and capital efficiency in providing services. Regulatory, control, and latency requirements for trading systems have made them one of the last application areas to move to the cloud. OpStack believes that with thoughtful architecture it can be remarkably effective to build and operate trading systems using a hybrid cloud approach.

In consultation with the lead architect at a financial services firm, OpStack developed this reference infrastructure to allow leveraging public cloud services to substantially reduce the capital investment needed to enter a new market without sacrificing any competitive advantage to other participants fully co-located with the electronic trading venues. In this architecture, the presumption was to use cloud neutral technologies (Docker, Kubernetes, et al) with the possibility of replacing some of the less latency sensitive elements with PaaS elements — we envisioned the trade history databases leveraging a cloud DBaaS (Database as a Service) offering rather than building and maintaining dedicated clusters from scratch on cloud IaaS (Infrastructure as a Service).

OpStack uses discussion documents, like this one, as a means of creating conversations with partners that lead to business-specific solutions. We specified this candidate architecture in a fair amount of detail in order to illustrate its viability and cost-effectiveness, with the understanding that a final solution for any client and market will be at least adjusted if not redirected by that ongoing discussion.

Requirements and Best Practices

Solution Elements

• Integrated CI/CD pipeline built to support the languages, platform, and architecture selected by the development team. Critical components include:
  • GIt-based source code management
  • Automated unit testing on check-ins
  • Integrated static code analysis
  • Scheduled, triggered and on-demand integration/regression testing
  • Automated security scanning - both static and of deployed pre-prod instances
  • ITIL change process automated in the implementation of the pipeline
• Fully Software Defined Infrastructure
  • Encompasses network, firewall rules, server configuration, operating system, layered products, security, and monitoring
  • Buildable on physical or virtual hardware, in a co-lo or on IaaS cloud
  • Under full source code management, making infrastructure repeatable, inspectable, and auditable
  • Applicable to physical servers, firewalls, and switches, to virtualized environments (VMware, KVM, ...), and both public and private (Aprenda, Open Stack...) clouds
  • ITIL change process automated in the implementation of the infrastructure deployment process
• Operations Automation
  • Operations orchestration technology centralizes control of all components with a stateful, scripted execution capability
  • Create all new non-prod and production instances with an automated process — cloud, virtual, and physical
  • Every server or container starts fully loaded and fully patched with all security, monitoring, and admin packages installed and tested — and is in the CMDB
  • Access controls, group privileges, auditing, firewall configurations are all set as part of the build and adjustable centrally with changes applied to any logical group of assets
  • Make every server image easily replaceable with a clean build at any time
  • Automation components log every action, providing real-time monitoring, anomaly detection, and corrective feedback loops for infrastructure and application changes
  • Choose to patch server images or simply redeploy fresh infrastructure with all application components
  • Orchestrate changes to allow for continuous availability

OpStack Baseline Operations Stack for Trading Systems

The tools below have been proven and tested in complex, regulated infrastructure environments. They are a combination of commercially supported open source and best of breed. Substitutions to provide desired capabilities or to conform to existing corporate standards is always possible and often desirable. Capacity modeling, an important capability for trading systems are currently all proprietary but should be part of the solution stack.

The accompanying Stack Builder is solution tailored, but it includes many of the technologies and tools used or incorporated into recent OpStack Engagements to meet the needs of our clients. Not all of the tools in any category are required to automate infrastructure for any single environment or application, OpStack will select the smallest set that fully meets the requirements of the application suite being hosted.

Co-Lo versus Cloud or Co-Lo with Cloud Deployment

The approach proposed allows for development, test, and production environments to be deployed on in the cloud or in the desired trading-advanted co-lo (e.g. the Equinix NY4 data center.) There are cost, complexity, and opportunity trade-offs to be made in the proximity of hosting of trading engines to the ECNs (NYSE Bonds, Bloomberg, Instinet, …). Only as the trading strategies and algorithms are developed and the engine deployed to execute them within the trading application will it be definitive as to the advantage or disadvantage of relative differences in latency. The number of firms that have chosen to co-locate with trading venues in order to effectively pursue low-latency and high-frequency trading strategies argues for opting to minimize latency.

The OpStack approach to defining, deploying, and operating the trading infrastructure is equally applicable to any mix of cloud and co-lo deployments, the table below lays out some of those options.

Hosting Options

The optimal solution is to mix and match the above options as the application is developed. The strength of the design is that the infrastructure is fungible with little or no change required to the trading applications.

Development and functional QA environments can be provisioned in cloud or on-prem virtual environments that are software identical to production. As the number of these environments will be variable with the development and testing cycle (and multiple parallel development streams are common amongst trading shops) they can be created, spun up, spun down, and released at any time with nominal effort. Algorithm testing against real-time data or capture/playback facilities would benefit from environments identical to production — whether cloud or co-lo, but on high performance hardware and production latencies.

A hybrid-cloud approach, using:

• AWS for development, monitoring, logging, and post-trade analytics
• with all low-latency trading components running in containers on dedicated hardware co-located with the ECNs

is OpStack’s recommendation as the optimal solution in 2020’s technical and trading environment that would be both market competitive and cost-effective.

Representative Co-Lo Configuration

This hardware configuration provides a representative hardware and system software configuration for the Co-Lo Physical and CoLo Virtual options. The equipment proposed is state-of-the-practice and the configurations based on the assumptions laid out by our client prior to the final definition and creation of the platform. The pricing provided was quoted by the manufacturers and is not reflective of competitive procurement practices or negotiated discounts.

Design Approach

The infrastructure is designed to:

1. Provide high-performance execution engines to be housed in a single rack within the Equinix NY-4 data center.
2. Redundant UPS-protected power distribution is provided in rack as standard at Equinix.
3. High-performance, high reliability, industry-standard components have been selected. No GPU co-processors have been specified, but they could easily be added to the strawman design if advantageous.
4. The design assumes connectivity to both AWS (Direct Connect) infrastructure and to dedicated fibre drops to ECNs co-located at NY-4 or any other latency-advantaged data center.
5. All hardware is redundant and configured so that the loss of one component of any type or of any one connection would not disrupt operations.
6. AWS or another cloud provider will be used to house administrative systems, provide a storage target for backups, and host development as well as monitoring and control infrastructure — only the low-latency components need premium hardware and minimum latency location.
7. Our assumption is that the ECN connections will terminate at the Arista switches (for lowest latency) rather than at the Palo Alto firewalls. This is a design decision to revisit as the application topology is finalized.
8. The network design provides for external connections at the standard speeds coming from those providers with a 40 gigabit internal network to provide for low latency, DMA-capable, communications between all elements of the trading systems.
9. There is expansion space both in the rack and in the networking components (including ports on the servers) to support the addition of additional servers, specialized GPU compute hardware, or a SAN storage array.

Hardware Components

System Software Components

In addition to the relevant operations configuration and management components selected from the Operations Stack, the licensing and support for a container-based low-latency trading application include the following two items. Both Real-Time Linux and Kubernetes are open source products and there is a very wide range of list-prices for support and commercial licensing. The choice of distribution for these products will have little to no impact on performance and little impact on the overall cost of environment setup.

OpStack Services

Project Phases - Months not Years

Assumptions and Notes

• Splunk and other monitoring hardware has not been specified. The design forwards both log records and SNMP traps to monitoring software in the AWS cloud, Splunk/SignalFX is state-of-the-practice but OpStack has had excellent results with Elastic, and Prometheus/Grafana as well.
• All systems will be built locked-down to minimize security threats and will be scanned for relevant compliance testing. No additional security monitoring or mitigation software (e.g Carbon Black) or malware detection and prevention (e.g. Symantec Endpoint Protection) has been specified, but can be added at the customer’s discretion.
• No DR (Disaster Recovery) environment has been specified or priced. The OpStack SDI implementation can (and should) be extended to define such an environment should it be desired. The key additional information needed by OpStack to estimate those costs for any customer solution would be RTO (Recovery Time Objective), RPO (Recovery point Objective), and desired location of capacity based on the DR facilities of the connected trading venues.
• Additional server capacity for the production co-lo compute complex can be extrapolated from the server hardware and system software prices for the initial five servers. Any deployment costs (other than onsite time to rack, stack, wire, and light) are nominal.
• A current best practice is to have sufficient capacity to run two instances of production with the second available for A/B testing of new code, trading strategies, system software, or firmware. Should A/B capacity be desired it can be specified once the actual application hardware requirements are better understood.
• Trading simulation (tick capture and replay) is a capability that is likely to be desirable and may require additional CPU and storage capacity.
• A trade history database should be provisioned in the cloud. A variety of options exist for DbaaS, whether an open solution like PostgreSQL, or an industry-advantaged option like InterSystems IRIS, TimescaleDB, or Snowflake.