Innovation In Quality Assurance: What Is The Impact On Trading Technology?

By Iosif Itkin, CEO, Exactpro
This article focuses on how particular changes in quality assurance might affect trading technology. There are plenty of discussions on Internet-of-Things, Cloud and Web. However, software verification for exchanges and brokerage platforms is closer to home for Exactpro. Here are some interesting QA trends:
• FrAgile process
• Crowd-sourced testing
• Formal methodologies
• Cognitive technology
When it comes to start-ups and web-industry, agile methods are favoured. They do not necessarily mean better products, but they can make software developers much happier. The outstanding keynote address of GTAC 2011, Google’s test automation conference, claimed that “test is dead”, and that FrAgile is the new agile. Reid Hoffman asserted that “if you are not embarrassed by the first version of your product, then you’ve launched too late”.

He went on stating that we need to test our ideas first before building the product right. The point is to have an established user community by the time your competitors get a well tested product. Extensive user pool is one of the key elements of crowd-testing.
The idea is simple – just deliver your product to the community and collect instant feedback on your software, both the upsides and downsides. To do it efficiently, one would have to rely on sophisticated code instrumentation and data collection. Otherwise it impedes you from thoroughly processing crowd-sourcing output and becomes an exercise in futility.
Is it feasible to use FrAgile process and Crowd-testing in finance? Can we afford an algo running amok burning money which could very well result in prosecution? Maybe we should take a different course of action. Perhaps, what we need is the very opposite of agile.
Areas like transport, medicine, nuclear power should exercise more rigorous controls. This leads to the creation of an environment, conducive to the evolution of quality assurance methods. Advances in model checking mean the end of guessing game. Last year, one TABB Forum posts pondered on why we are not using mathematics and computer science similar to what NASA uses to design a safe autopilot?
Is it not embarrassing that we have financial market crashes at the same time as spaceships explore the heliosphere? We can invest heavily into formal methods and expect financial software that is more reliable. Nevertheless, one has to keep in mind that in defiance of all of the formal methods, the New Horizons probe experienced a shutdown that made it lose contact with Earth for three days. Thus, expecting your systems to fail is the only way to enhance the quality of your software.
So where do we go from here? Is it possible to be faster without compromising safety? Can other industries teach us anything? If not the computer industry, maybe the show business has an answer. Some of you may be familiar with the characters of Battlestar Galactica and Sarah Connor Chronicles, their Doomsday scenario and the people who ran into some serious confrontation with technology. Had they not relied on the same technology that tried to kill them, the characters would not have lasted until the finale. They were guarded by systems with the same level of sophistication as those they were fighting against. This is the mentality that we should adopt. In the context of a complex platform, it is imperative to build software to test our software. Our risk control and test instruments should not be inferior to what will hit us. Having a good robot on your side is the only way to survive the robot apocalypse.
So, can we estimate the impact on the trading technology? Developing testing instruments and monitoring tools should have the same priority as developing the trading platform itself. It is possible to use agile methods and continuous integration if the system is designed with testability and risk control in mind. At the very least you can deploy, start, restart and configure it automatically. In addition, it is crucial to be able to shut the system down if necessary. Do not expect formal artefacts from the agile process. Instead, have a parallel stream to develop the necessary test harness. Software engineering in the test approach proposed by Google is very close to building software to test software.
We can implement formal verification, theorem proving, static analysis and other approaches, but the software will break anyway. What’s more, the absence of sufficient monitoring and kill switches is what turns a problem into a disaster.
For the sake of audit and regulatory requirements, we develop passive testing tools. Instead of generating messages themselves, they capture the traffic and store it for further analysis. Passive testing tools can collect all the evidence you need.

It is paramount to ensure that your trading system can co-exist with the tools. Passive testing tools can serve as both surveillance and client on-boarding systems. They can also be used for crowd-sourcing. If your test environment is open to others and you have a strong passive test capability you will be able to gain a lot of valuable feedback without even having to ask.

The Technology Services division of LSEG incorporates MillenniumIT, GATElab and Exactpro.

This gives the Group a unique set of software from all three companies at our disposal, which we can use to build test infrastructure for any system. Take testing of trading algos, for example. We can deploy one or several scalable matching engines to function as market simulators. We can utilise GATElab’s algopath or an Exactpro tool called Minirobots to act as market participants. They can both replay some data and at the same time mimic a realistic market impact. In addition, we can have latency and front-running proxies for better test diversity.
It is possible to take the next version of the algo and put it to the test. However, what is the point of running it once? Markets are not deterministic, neither are well-simulated markets. Every time tests are executed, there will be a slightly different result. For this reason, multiple replays are necessary to assess the quality and efficiency of the software.
This is where everything falls into place:

1. Efficient restart and testability to enable unattended test execution
2. Machine-readable specifications to synchronise all the tools and formally iterate the possible tests
3. Passive testing capabilities and, potentially, market surveillance to capture and store all the relevant metrics (both P&L and system related ones)

Hypothetically speaking, one can run an infinite amount of tests and execute a wide range of scenarios, except that we all have limited time as well as limited hardware. There are two more quality assurance trends, though, cognitive technology and mutation testing, that allow us selecting an efficient limited subset of tests from an infinite number of possibilities. They are used to find the optimal test scenario subset for a trading platform. They can also be utilised in enhancing the test set along the changes in the system under test and in the surrounding markets.
We’d love to hear your feedback on this article. Please click here