You should read the entire detailed analysis at the Nanex site, but here are the two big items that jumped out at me:
- Inaccurate quotes from the NYSE caused other trading systems to believe they were observing a downward momentum that wasn't (initially) actually present, but when those other trading systems acted on that inaccurate data, they in fact caused the behavior that they thought they were merely observing (i.e., a feedback loop):
quotes from NYSE began to queue, but because they were time stamped after exiting the queue, the delay was undetectable to systems processing those quotes. The delay was small enough to cause the NYSE bid to be just slightly higher than the lowest offer price from competing exchanges. This caused sell order flow to route to NYSE -- thus removing any buying power that existed on other exchanges. When these sell orders arrived at NYSE, the actual bid price was lower because new lower quotes were still waiting to exit a queue for dissemination.
Because many of the stocks involved were high capitalization bellwether stocks and represented a wide range of industries, and because quotes and trades from the NYSE are given higher credibility in many HFT systems, when the results of these trades were published, the HFT systems detected the sudden price drop and automatically went short, betting on capturing the developing downward momentum. This caused a short term feed-back loop to develop and panic ensued.
- Certain trading systems exhibited strange and unexpected behavior for which there still does not appear to be a valid explanation:
We decided to analyze a handful of these cases in detail and graphed the sequential bid/offers to better understand them. What we discovered was even more bizarre and can only be evidence of either faulty programming, a virus or a manipulative device aimed at overloading the quotation system.
The quote stuffing analysis is quite disturbing and serious. Was this in fact a malicious event? I am inclined to think it was more likely a mistake. As a software engineer, my first reaction is to favor the "faulty programming" hypothesis.
When you look at the charts and graphs in the quote stuffing section of the analysis, you'll see that they certainly look like artificially-induced, synthetically-generated data patterns.
Like test data, that is.
At work, we have an ultra-high-volume testing tool that we call the "submit-a-tron", which is designed to stress-test our server by intentionally subjecting it to implausible and overwhelming activity. Once in a while, somebody accidentally mis-configures their test system and mistakenly points an instance of the submit-a-tron against a valid production server, which then has us all running around cleaning up the mess for a few days.
Perhaps the software development staff at one of these trading houses accidentally did something similar (that is, accidentally performed an internal stress test intended for use with internal testing systems while pointed at the real production stock exchange channels)?