How we helped Improve Performance and Minimizing Licensing Costs by Enhancing Trade Validation Process
The client is a large independent securities regulator.
Industry
Capital markets
Business challenge
This client relies on its audit trail application as a source of timed, sequenced-order events that contain market quotation and trading information submitted by member firms. This key application is used to verify and track member firms’ submission, reporting, and processing statistics for compliance, repair, and resubmission.
The existing technology—a platform built on a DW/BI appliance and the DataStage ETL tool—was expensive, with limited scalability options, and adversely impacted trade validation with ever-increasing volumes (consistently growing by about 45% year on year). There were frequent misses in SLA due to this growing volume—and application volume was expected to grow five to six folds over the next two years as additional asset classes were included. The client also needed to create a central repository to receive and store consolidated audit trail data in order for regulators to view cross-market data.
Our Approach
In collaboration with the client’s developers, We engaged in a proof-of-concept and evaluated new designs based on Hive, NoSQL (HBase), and in-memory-based custom Map/ Reduce jobs. Based on the findings, we created a multi-phased solution to migrate the trade validation process to Hadoop.
Our Solution
To address the client’s business needs, we developed a multi-phased approach to port the client’s existing DataStage application to a Hadoop-based platform:
- Phase 1: Proof-of-concept to evaluate designs based on Hadoop, HDFS, Hive, and HBase.
- Phase 2: Migrate from current ETL platform (DataStage) to a core Java-based ETL solution.
- Phase 3: Migrate from existing DW/BI to an in-memory based custom Map/Reduce Hadoop solution, which includes Hive and HBase.
This solution enabled the client to move member firms’ trade data from the proprietary NAS system to Hadoop HDFS, along with the required reference data. The data validation engine is fully customizable, and rules are created using an XMLbased template that runs on a custom-built Map/ Reduce framework, as shown in the below chart.
Business Impact
Implementation of our trade validation process migration to Hadoop had an immediate positive impact:
- Resolution of capacity problems, with performance improvement over 10x.
- Lower license costs for DW/BI and DataStage by using more affordable Hadoop platform.
- Affordable scalability potential for expected long-term audit trail data volume growth of 10–15 billion transactions per day.
- Long-term, cost-effective Tier 3 storage platform using Hadoop HDFS.
- Ability to port the new Hadoop-based solution to cloud-based systems like AWS without design changes.
