Markets: Very Big Multithreaded Software Apps That Crash
At one point in history, equity markets were giant discounting mechanisms, taking in all available information about the economy, earnings, sentiment, then spitting out future expectations of value.
The proliferation of HFT and Algo traders, according to a fascinating take from Ars Technica, has changed that. Our stock markets now behave no differently than a giant multithreaded software application. And on May 6th, 2010, that software app crashed:
“To be a single multithreaded app, as opposed to an unrelated collection of multithreaded apps, the different threads must somehow interact with one another. In other words, the threads must share and jointly modify some kind of state.What state do the various apps and algorithms that run on Wall Street’s machines share? At the very least, every part of the market shares the quote feed, and some parts are even more tightly coupled than that. But let’s focus on the quotes.
The price of, say, AAPL at any given moment is a numerical value that represents the output of one set of concurrently running processes, and it also acts as the input for another set of processes. AAPL, then, is one of many hundreds of thousands of global variables that the market-as-software uses for message-passing among its billions of simultaneously running threads. Does it really matter that those threads are running on separate machines at different institutions?
. . . [If] the market really is essentially an enormous piece of multithreaded software, then I’m not entirely sure what kind of rabbit hole we’ve all gone down.”
Fascinating metaphor . . .
>
Source:
The stock market as a single, very big piece of multithreaded software
Jon Stokes
Ars Technica, October 7, 2010
http://arstechnica.com/business/news/2010/10/you-say-stock-market-i-say-ginormous-multithreaded-application.ars
Tweet
Facebook
Reddit
Digg this!
October 8th, 2010 at 8:10 am
Oh great. So now does one take the blue pill, or the red pill?
Unfortunately, this pretty accurately describes what I think of the market at this point. The question then becomes that soooooo much of modern neoliberal economic theory is based on market efficiency and perfect information getting incorporated into market prices — if this is true, what does it mean? Can one ever really put this genie back in the bottle?
October 8th, 2010 at 8:52 am
Now, consider the fact that there is 223 Trillion in outstanding US derivative exposure ,a record. It appears that TBTF are ganbling with our money again. Wordwide derivative exposure is 3X us. Sleep tight, hope the computer programs don’t bite.
October 8th, 2010 at 9:12 am
It’s a pity all the HFT machines are running Vista – should have seen this one coming.
October 8th, 2010 at 10:18 am
A rock-ribbed Midwesterner sold too many futures. The system survived. End of May 6th story.
Instead of the “Flash Crash” it should be called the “Guy Fry.”
Though it is true that the system is a big distributed, networked software system. …Oh, that’s a good thing.
October 8th, 2010 at 10:52 am
Liked this article when I read it. Nice to see ARS is on your reading list.
My other favorite quote that I’ve seen is that SkyNet caused the crash.
October 8th, 2010 at 11:08 am
[...] 60 Minutes is going to talk about high frequency trading this weekend. (The Reformed Broker also Big Picture) [...]
October 8th, 2010 at 11:09 am
If the market is a distributed, concurrent system then it’s the regulatory agencies and exchange rules that should generally be coordinating the threads. That being said, the threads (i.e. the market participants) should try to ensure their own safety.
I find it difficult to get too worked up about HFT’s. At the end of the day if they occasionally introduce market inefficiency then that provides opportunities for long-term investors. If somebody wants to sell me an asset for a fraction of my perceived value I will be only too happy to oblige. It just seems unfair that the rules allow market participants to yank back quotes that are below the top of the book; orders should sweep the book.
October 8th, 2010 at 11:34 am
It’s a pity all the HFT machines are running Vista
I know you’re snarking, but it is worth mentioning: I have never seen windows used in an HFT system.
Haven’t seen all of them, but I’d bet that anyone who tries will be naturally selected.
October 8th, 2010 at 11:52 am
As some of the comments there point out, it really sounds like someone who either doesn’t really understand common multithreaded software, or is just stretching the concept too far. If you have one piece of software with a large number of threads, it would usually be silly for them to all be acting on the same data set, unless you’re doing some kind of modeling. On the other hand, if you have 40 (pick your favorite number) pieces of software with hundreds of threads each, acting on the same set of data, you’re describing complex adaptive systems. The “one piece of software” bit doesn’t make a lot of sense.
Is this really much different from what was there before? If you had a bunch of *really* unsophisticated traders at 40 different trading firms (say, 1000 traders per firm), with a set of fairly simple instructions for what to do, and someone walked in with a large sell order, you’d get basically the same behavior, but on a much slower scale.
The real differences are the speed of the transactions and the lack of sophistication amongst the traders. Human traders are smart enough to see prices dropping like that and say “WTF?” Software traders aren’t that smart, but they can make trades in milliseconds (probably microseconds between traders in the same application). So where the 29 crash took years for the humans to fully realize the crash, and the 87 crash took a few days with human and computer trading, now we can do the same thing in minutes! Heck, in a few years we’ll be able to crash the market faster than you can blink. Gotta love progress!
October 8th, 2010 at 11:55 am
There is nothing sinister about multi-threaded apps. Every modern web-server is multi-threaded and the world doesn’t seem the worse for the wear. The operating system I’m working on has 632 threads running at this moment. The computer is still functional.
The core of the problem is simply that markets are complex and the various moving parts sometimes interact in ways that produces a surprisingly bad outcome. The fact that we use computers is irrelevant. I’m much more disturbed that prop trading is legal. It’s like going into wal-mart and finding that half the people in the store are all about buying goods, then returning them 2 minutes later, over and over again.
October 8th, 2010 at 12:12 pm
dlipton – I agree in theory with your point “If somebody wants to sell me an asset for a fraction of my perceived value I will be only too happy to oblige.”
But the skeptic in me thinks that if there is some temporary market inefficiency – the “house” will make it work for them – not you.
Ie – a 20% trailing stop loss will be triggered and those transactions will stick. Ie the house is happy to buy your shares in a panic sell situation.
But if YOU step in a buy at the bottom – those transaction won’t be honored, because, well, any reason the house can think of…
October 8th, 2010 at 1:39 pm
The author makes some reasonable points (and, folks, the issue is the algorithms (think cooking recipe) and the relative speed of execution (how fast the machines are running and how quickly they are able to access and modify data — so “choice of operating system” is really irrelevant — sorry, 2+2 doesn’t get a better answer under Linux than Windows, etc.).
What would make HFT’s into a large, single multithreaded application is if their algorithms are virtually identical (otherwise one would expect somewhat different behavior from each).
This is very possible, given that programmers and consultants move from company to company implementing the same stuff. And, they read the same papers, and talk to each other. And their masters want to copy all of their competitors.
This is why, in aeronautics, redundant systems are created with completely different algorithms, so that if one fails (say, due to a programming bug), the others can correct it before you fly into the sun.
As things (seem to) stand now, if one goes berserk, they all do.
Screw this, I’m starting my own bank.
October 8th, 2010 at 2:59 pm
Excellent article! I think the point the author was trying to make with “multi-threaded” was that the stock market is a multi-headed Hydra (from mythology) and one head doesn’t know what the other head is doing.
As in many blogs, the comments attached to the source article were great reading, maybe more enlightening than the article itself.