HN Books @HNBooksMonth

Ha, Flash Boys 2.0 was from 2019. This blog post is literally "I read a thing from three years ago and have lost faith in something that I never believed in in the first place".

As someone who knows and has worked with all the people involved with the write-ups referenced in this blog post, I can pretty confidently say that the author is way way behind here. There have been some insane advancements in recent years that would blow his mind.

MEV harvesting is actually an insanely fascinating field of study, and it comes down to consensus in a world where information can only move as fast as the speed of light.

Front-running is a general problem where, if you can find out that someone wants to buy something, and can buy it first and sell it to the person who actually want it at a premium. This gets really interesting in the blockchain space, because Miners (Validators) decide the order of transactions, and in some cases validators can be incentivized/coerced to move around transactions in a way that might be beneficial for certain users. This is what is referred to as "Miner Extractable Value".

Yes it's a problem, yes there are mitigations, yes there are fun and weird ways around those mitigations, etc.

This is what happens when people are incentivized to take a really good hard look at what global consensus really means. At a high level, most users don't really need to know or care about any of it, but it exists as a parasitic force constantly extracting value from the network. Still, IMO the parasites that feed off MEV are at a significant disadvantage compared with those who feed of legacy financial networks (for more on that, you can read the actual Flash Boys book https://www.amazon.com/Flash-Boys-Wall-Street-Revolt/dp/0393...).

⬐ throwaway82652

>it comes down to consensus in a world where information can only move as fast as the speed of light.

Yes and now you're circling back around to the usual problems with algorithmic high-frequency trading talked about in Flash Boys, which has nothing to do with blockchains or cryptocurrency. Unless you can cite some "insane advancement" in consensus algorithms that solves these problems everywhere then I doubt there is much more that can be said here. Because AFAICT everything you're talking about is just piling more mitigations and edge cases on top of an existing consensus algorithm. That work is important and it needs to happen on any distributed system. It's not some kind of world-changing thing, it's just necessary maintenance (and cost) to keep the system going.

>This is what happens when people are incentivized to take a really good hard look at what global consensus really means.

They were already incentivized to do that without blockchains. This sentence could be more accurately stated: This is what happens when people deploy an experimental solution that claims to solve global consensus, but as we find out from experience that it actually doesn't, in some ways it actually makes it worse, and generally speaking those problems are still just as hard as ever.

⬐ tornato7

There are actually some insane advancements coming out in the form of zero-knowledge order books, Dusk Network being the most well known. It essentially removes frontrunning as a possibility because buyers and sellers are matched using ZK proofs, but are not able to see any orders before they are matched.

⬐ jkhdigital

The real question here is not whether one can design a market that prevents bad behavior, but whether any of the major players will ever bother using it. If the “little guy” is always on the losing end of market shenanigans, then market makers will never have an incentive to move their liquidity to the “fair” market.

⬐ pcthrowaway

The "little guy" is always at a disadvantage in capitalism (or small business, or mom & pop shop)

I suspect there's a disconnect between people talking about democratization of access, and people pointing out crypto doesn't actually make things fair.

Crypto doesn't make things fair for the little guy.

It does add additional transparency to financial markets though, as well as things like open, verifiable, non-custodial, programmable financial contracts (as long as all the inputs can be verified reliably on-chain)

This gives people equal access (with some limitations on what that means)

⬐ throwaway82652

>It does add additional transparency to financial markets though

No, not really. This comment is more crypto myth-building. There's nothing technical about blockchains that adds transparency. Any company that wants to publish all its financial statements publicly can technically already do so and could always do so. They don't for many reasons, the most important ones being that customers overwhelmingly want financial privacy, and financial companies are required by law to provide a certain level of privacy for customers.

>open, verifiable, non-custodial, programmable financial contracts

No, smart contracts are technically not "contracts" in the legal sense and are also not "non-custodial" because they require middlemen to run the blockchain. There is also nothing more open or verifiable about smart contracts compared to any other program. They're just programs. I absolutely hate that they used the name "smart contract" for this because it's so misleading.

>as long as all the inputs can be verified reliably on-chain

This will never happen because the only reason they're useful is because they take inputs from off the chain. The marketing is that users will be able to actually use these things to perform services. Of course that is self-contradictory because of the previous reason.

>This gives people equal access

No it doesn't, because only a small fraction of people are going to become smart contract programmers or are going to become skilled in auditing smart contracts. You might as well say all finance gives people equal access because anyone can become a CPA.

⬐ 3np

> There's nothing technical about blockchains that adds transparency

False. On a public chain like Ethereum, every single historical operation and its outcome events and state can be traced and verified. If the source code is published, the running bytecode can be verified to match the source code.

> smart contracts are [...] not "non-custodial" because they require middlemen to run the blockchain

Also false - miners/validators have no opportunity to seize or freeze assets

> There is also nothing more open or verifiable about smart contracts compared to any other program

See above. You can verify what code was running when and with what input.

⬐ lottin

Right... why do you think the perpetrators or ransomware attacks demand to be paid in crypto-currency, instead of via a plain old wire transfer which according to you is more opaque?

⬐ eptcyka

Because it's still less tracesble than a bank account, but unless one converts it in a dex to something truly untraceablr like monero or zcash, ones attempts to transact with it will be traced, so you won't be able to do much else with it. Coinbase and all other exchanges are known to freeze coins linked to illegal activity.

⬐ 3np

For other reasons than the ones I am replying on.

For one, accounts on it are pseudonymous. If by "transparency", you mean "every activity can be traced back to a meatspace individual or institution as recognized by a national government and/or law enforcement", that is a different level of "transparency".

It's not being compared to wire transfers but to server-side software (including those responsible for executing legacy wire transfers).

⬐ throwaway82652

>False. On a public chain like Ethereum, every single historical operation and its outcome events and state can be traced and verified. If the source code is published, the running bytecode can be verified to match the source code.

No it's not false. Ethereum doesn't matter, there are a ton of sidechains and mixers that obfuscate transactions and you can't stop people from using them, you also can't guarantee they will provide source code or provide any means to verify their own smart contracts which they don't have to build on top of Ethereum's platform. And many of them don't anyway, specifically because of deficiencies in Ethereum.

>Also false - miners/validators have no opportunity to seize or freeze assets

Actually they do, if they all decide they don't like you then they can blacklist your wallet address. For practical purposes it's the same as a frozen asset. The inability to seize assets is actually a bad thing because it means the network operators have no effective way to confiscate stolen money.

>See above. You can verify what code was running when and with what input.

And you can also do that with literally any other program regardless of whether it's on a blockchain or not.

⬐ ClumsyPilot

'Any company that wants to publish all its financial statements publicly can technically already do so and could always do so.'

What, in PDF? and store them on a website or FTP server? Totally discoverable and analysable?

And if two different companies publish statements (inevitsbly in slightly dofferent formats) you need a team of analysts working for 6 months to match up the transactions?

⬐ mw888

1. Saying that transparency isn’t desired does not refuse his point. Not only that but the off topic declaration where you imply privacy is impossible on blockchain is incorrect.

2. Miners and block verifiers are not middle men. They have negligible control where as a middle man has full control.

3. The verifiably of smart contracts is not exceptional for its own sake, but for the fact that it prevents cheating in a system where cheating is heavily incentivized.

4. Auditing smart contracts will generally be much easier than auditing EVM bytecode. Ethereum isn’t the only game in town and the most sophisticated layer 1 smart contract languages converge toward functional, locked down languages which lend themselves to automated and manual auditing. The same way open source increases user security even though 99% percent of users won’t build or read its code.

You are so off base on all of your point I seriously doubt you don’t have an emotional stake in your position. Perhaps the classic “I’m a smart tech person yet I missed out, so really it’s always been bad and I’ll be proven right someday,” syndrome. You know other people that missed out simply developed some grace, and others still that identified problems went to work on them rather than pretending they were unsolvable.

The dogmatic crypto skeptic is as bad as the crypto shill.

⬐ throwaway82652

>Saying that transparency isn’t desired does not refuse his point.

That's not what I was saying, at all.

>Not only that but the off topic declaration where you imply privacy is impossible on blockchain is incorrect.

Please do not argue straw men. I never implied privacy is impossible. I know about things like privacycoins. Some have even been mentioned in this comment chain. My actual implication here is that blockchains cannot promise either privacy or transparency. Those who want to use mixers and privacycoins will do so and you won't have any transparency into their activities. Those who want to insist you use KYC exchanges and traceable coins will do so and you won't be able to have privacy there if you want to transact with them. So basically, an ordinary person has no control over how much transparency or privacy there actually is on the network. Unless you have an outsized level of control on the network (and therefore the network is not decentralized) then you're completely dependent on the other more powerful party. So basically blockchains are providing nothing of value here compared to an ordinary financial service.

>2. Miners and block verifiers are not middle men. They have negligible control where as a middle man has full control.

No, they have full control. Like actual full control over the network. Individually they don't, but as a whole they do, that's literally how the network functions.

>3. The verifiably of smart contracts is not exceptional for its own sake, but for the fact that it prevents cheating in a system where cheating is heavily incentivized.

No, this is wrong. Verifying a smart contract does not prevent cheating. Even if you verify that the smart contract technically has no bugs, it could still do the wrong thing, or the other party could just commit good old-fashioned normal fraud and never hold up their end of the bargain. Smart contracts are not actually smart not are they contracts.

>4. Auditing smart contracts will generally be much easier than auditing EVM bytecode. Ethereum isn’t the only game in town and the most sophisticated layer 1 smart contract languages converge toward functional, locked down languages which lend themselves to automated and manual auditing. The same way open source increases user security even though 99% percent of users won’t build or read its code.

This whole paragraph is based on a falsehood. Open source does not increase user security, go look at the recent log4j disaster. What actually increases security is having security engineers being paid to look for and fix security issues, the cost of which is not significantly different between open or closed source. I have seen no reason to believe it's any different in smart contracts. Functional languages can be good for proving certain types of code with fixed requirement, but the same thing also applies to any of that category of software, financial or otherwise. It's again not related to blockchains at all.

>I seriously doubt you don’t have an emotional stake in your position. Perhaps the classic “I’m a smart tech person yet I missed out, so really it’s always been bad and I’ll be proven right someday,” syndrome.

This is a totally wrong, nonsensical and completely rude comment, please never say anything like this again. Never even let the words cross your mind. Your intelligence is capable of much better things.

>and others still that identified problems went to work on them rather than pretending they were unsolvable

This also makes no sense. I've been looking at this for 10 years. The problems aren't unsolvable. There are plenty of problems to solve, the reason why you shouldn't bother solving them is because all of those problems are intentionally caused by the bad design of blockchains. Our work is hard enough without purposefully making it harder, but that's the only thing blockchains do.

>The dogmatic crypto skeptic is as bad as the crypto shill.

I'm not a "crypto skeptic" and I don't care to discuss dogma. This is about the facts. Please avoid making these ridiculous accusations, please stop trying to psychoanalyze me, and just stick to the facts. That will help both of us.

⬐ throwaway82652

That sounds good for any kind of HFT platform matching buyers and sellers. I don't see how that is an "insane advancement" or has anything to do with blockchains.

⬐ pazimzadeh

Because the miner wouldn't be able to see the order book?..

⬐ throwaway82652

Sure, if you care about HFT you probably want the same guarantee to apply to any market maker, not just ones who operate on blockchains. Unless I missed something here.

⬐ polygamous_bat

So, the blockchain bros have reinvented... dark pools, which has been around since 1979? Because that's what you just described.

⬐ CryptoPunk

Dark pools rely on the operator being trustworthy. Distributed encryption relies only on cryptography and the validator set being too large, and costly to join, to allow cooption, and both assumptions are highly trustworthy.

⬐ spopejoy

Even if this is true, it is still just a dark pool, which is trivial to game. One ping is all you need.

I think the race to put order books on blockchain is misguided. Constant-product DEXes can offer way less gameable best-execution by simply trading more cautiously for the swapper.

Problem is, on Ethereum, gas makes it too expensive _not to trade_ once you get to the blockchain. MEV is a problem that can be managed effectively if the blockchain is scalable (thus avoiding gas problems).

⬐ CryptoPunk

>>Even if this is true, it is still just a dark pool, which is trivial to game. One ping is all you need.

If that's true, then you're right. Assuming an effective decentralized dark pool, that is not vulnerable to side-channel attacks, is possible, it is preferrable to the centralized variety that traditional finance uses.

>>Constant-product DEXes

What is a constant-product DEX?

⬐ spopejoy

> it is preferrable to the centralized variety that traditional finance uses

Citation needed :) . Seriously, crypto needs to do better than just assert that ownerless decentralization is instantly better. Traditional dark pools work just fine once you accept that with the right model, a single fill can tell you loads about the order book.

> What is a constant-product DEX?

That's the model used by Uniswap and almost every other major swap-based DEX. MEV as described in the TFA is preying largely on these DEXes and people conclude that it's the fault of the constant product model, or of being transparent. It's not: it's the fault of gas making it impossibly expensive to use the most basic tool in execution, splitting your order into smaller trades and applying logic to when you choose to trade.

I'm actually kind of a constant-product maximalist because it shows you CAN have a public, transparent market, which is something genuinely new to finance. IMO the last thing the world needs are old-fashioned order books and dark pools on blockchain.

⬐ CryptoPunk

On the basis that the cryptography remaining unbroken and the protocol-genetated incentives creating a validator set too large to coopt are more reliable trust assumptions than the trusted third party controlling a traditional dark pool being competent and trustworthy.

>>It's not: it's the fault of gas making it impossibly expensive to use the most basic tool in execution, splitting your order into smaller trades and applying logic to when you choose to trade.

Yes that makes sense.

⬐ xyzzy123

No, you can still be front-run in a dark pool. Worth doing a bit of research rather than reflexive dismissal.

⬐ RC_ITR

The entire story of crypto is that it’s a somewhat volatile new form of security (I know I said it) that has two disadvantages vs. traditional securities.

1) It lacks the sophisticated financial tools and derivatives that have been developed over the past century

2) It does not have enough volume to keep up with the new sources of arbitrage that come with every new DeFi product (looking at you DEX’s and synthetic stocks!).

Every time a crypto asset “solves” one of those two problems (almost always incompletely) the crypto verse acts like they’ve proven p=np.

⬐ FabHK

> This blog post is literally "I read a thing from three years ago and have lost faith in something that I never believed in in the first place".

No, this blog post is "Look, here, several papers from last month showing there are still massive problems (that were presaged accurately in papers from three years ago)."

(and it's not literally that, of course. Why do people use "literally" to mean not literally?)

> Still, IMO the parasites that feed off MEV are at a significant disadvantage compared with those who feed of legacy financial networks.

Well, fortunately one can regulate these financial networks, right? Well, the legacy ones at least.

⬐ mw888

And you can’t regulate crypto? How did China do it? Would it somehow be harder when it’s so transparent? Of course not. Anyways you have only a surface level understanding of the post and crypto currencies if you think the way forward is not a technological advancement. People outside the space seem to discover these issues and think that those in it aren’t acutely aware and working on them, which they are.

⬐ FabHK

Yeah, you can regulate crypto. But then, what's the point? Just have a permissioned regulated blockchain, without all that Proof of Waste nonsense.

⬐ calsy

Saying insane a lot, pretty confident, and cherry picking some text to 'literally' dismiss the entire blog post with a single sentence that just serves to validate your flimsy reasoning doesn't make a good argument.

⬐ toolz

Do we have to accept that MEV is a problem? Just because it's been a tactic used in the shadows with traditional finance doesn't mean it's a problem in an open ecosystem where anyone willing to pay can play. There isn't some shadow elite funding a central government here lobbying for restrictive legislation to keep competition at arms length. In my opinion the crypto ecosystem in general is much more understanding and capable of finding ways to distribute information and trust, which prevents this from becoming a bad thing like we see in traditional finance.

⬐ jkhdigital

The real problem with MEV is that it distorts the consensus mechanism in weird and unpredictable ways. MEV is essentially free money that could be considered part of the block reward, but it exists at the application layer so it really can’t be accounted for in the consensus protocol incentive design. Application layer logic (“smart contracts”) can move arbitrary value in arbitrary ways, which can create wild volatility from block to block.

A similar problem appears to emerge when Bitcoin’s block reward disappears, as high volatility in block rewards would cause rational miners to behave in suboptimal ways.

⬐ w_TF

Something that rarely gets addressed in these discussions is why Ethereum even has fee market to begin with instead of a cheap FIFO model which ideally would be more fair.

To answer that I would invite anyone to try using a blockchain under heavy load without any way for users to prioritize transactions. What you will find is the network becomes vulnerable to transaction spamming; possibly to point of breaking consensus making it so nobody can transact at all.

MEV is a consequence of a design decision to address this. And that's not to say Ethereum has everything figured out and a better FIFO model can't solve this, but afaik this is the current state of things.

⬐ 99112000

FIFO? First in first out? The blockchain is essentially the timestamping machine. The timestamp of a transaction are untrusted until mined into blocks, even then a fork can rid them.

The blockchain is essentially solving the ordering problem, one big timestamp machine. The solution you propose would require a timestamp machine in the first place lol.

⬐ delaaxe

It’s what Cardano does

⬐ lukeramsden

> Do we have to accept that MEV is a problem?

Yes and no. On the one hand, one does have to accept that "MEV" in the "_Maximum_ extractable value" is always going to be potentially present on a public blockchain - but often times, this is a good thing. Liquidity pool arbitrage is also "MEV", but is MEV that is absolutely vital to efficient markets, and the more advanced the players in that game get, the more everybody gets to enjoy liquid and efficient markets. On the other hand, more and more DeFi projects take in to consideration possible MEV extraction vectors when designing applications, and some even use "MEV protection" as a USP (such as CoWswap)

⬐ Geee

The referenced paper is dated 29 March 2022. https://arxiv.org/abs/2203.15930

⬐ otterley

Can you elaborate on some of these “insane advancements”?

⬐ crazypython

It means they worked on it and acknowledge it now. The fundamental problem is still very much there.

⬐ repomies69

That sounds insance

⬐ px43

Flashbots gets a minor mention in the article, but it doesn't do justice to how huge the Flashbots ecosystem has gotten:

https://docs.flashbots.net

To be fair, there isn't a ton of information widely available, because talking about Flashbots in public is considered "alpha leakage". You see glimpses on Twitter, Discord, Twitch, etc though. Every once in a while someone will dump their finder code to burn alpha when they get squeezed out of some opportunity set.

People have spent years fine tuning bots looking for MEV opportunities, and they have gotten really really good. It also plays a ton into cross-bridge arbitrage, taking advantage of quirks in the interchain ecosystem to squeeze out opportunities.

Like I said, the author has dipped their big toe into the MEV rabbit hole, but it goes so, so much deeper.

⬐ yunohn

> Like I said, the author has dipped their big toe into the MEV rabbit hole, but it goes so, so much deeper.

So, both you and GP are saying that MEV is a bigger problem than this article says?

⬐ akyu

A unsolved problem from three years ago is still an unsolved problem. And you don't actually rebuke any of his claims, you just give a poor summary of MEV.

⬐ uncomputation

I think you’re over-exaggerating about the “insane advancements in recent years that would blow his mind.” There was a recent overview of different MEV preventive measures [0] which came to the conclusion that there is currently no silver bullet. This is still a large and unsolved issue with competing mitigations with large drawbacks, far from any “insane advancements.” You make a good point that these algorithmic trading techniques are not as big of a deal for everyday users though, and mostly bot wars.

[0]: https://arxiv.org/pdf/2203.11520.pdf

⬐ nurettin

> This gets really interesting in the blockchain space, because Miners (Validators) decide the order of transactions

Traders use exchanges. I always thought *coin transactions are too slow to execute at trading speeds, so exchanges somehow buffer transactions using their own pools without ever sending your request to the network at large.

⬐ yunohn

> Still, IMO the parasites that feed off MEV are at a significant disadvantage compared with those who feed of legacy financial networks

Literally whataboutism.

⬐ sl3232323

Crypto is a scam. Why do you need all those words to just say somethign so simple, px43? Sell all your crypto stock (aka skams), and learn something new for once.

⬐ _Nat_

> As someone who knows and has worked with all the people involved with the write-ups referenced in this blog post, I can pretty confidently say that the author is way way behind here. There have been some insane advancements in recent years that would blow his mind.

Looks like they link a lot of stuff from the past month.

Examples of links in the article from within the last month:

1. https://arxiv.org/abs/2203.15930

2. https://davidgerard.co.uk/blockchain/2022/04/04/if-you-want-...

3. https://datafinnovation.medium.com/the-consequences-of-scala...

4. https://web3isgoinggreat.com/?id=2022-04-05-0

5. https://ethresear.ch/t/np-completeness-of-a-strong-form-of-s...

⬐ aaaaaaaaata

If it's published, it's already lost alpha to the publisher, in 90%+ of cases.

This article explains what the stock market pretends to be.

This book explains what the stock market actually is: https://www.amazon.com/Flash-Boys-Wall-Street-Revolt/dp/0393...

It's much less friendly than it seems and only "efficient" for a select few.

⬐ starkd

I don't think that's the spirit in which the article was introduced. It's an educational piece. Of course, it's about the ideal scenario.

⬐ khold_stare

I greatly enjoy Michael Lewis and his books, but Flash boys was extremely inaccurate, full of factual errors. I've worked in the finance industry, in HFT at one of the firms mentioned in the book. I joined around the time Flash Boys came out, and it was required reading in the firm. Here are some points:

- Michael Lewis really only got one side of the story - that of Brad Katsuyama, who had a vested interest in casting HFT players in a bad light to promote his own business - building the new exchange IEX.

- Brad also blamed HFTs for systems at RBC failing to make massive trades like they used to. There was nothing nefarious here - RBC had just fallen behind the time in technology, like trying to send a Fax in a world where everyone already uses Email. If Brad, or RBC, or RBC software engineers picked up the phone and called any of the exchanges, they would probably gladly update them on the industry and save them all the work of re-discovering it themselves.

- The claims about front-running are completely false. Front running would mean that a market maker somehow knows someone's orders at two different exchanges and somehow is able to "get in front of the line" or even know that those orders belong to the same person. This would mean the exchanges leak information or allow certain users "ahead of the queue". None of this is true. What Michael Lewis called front-running, was HFT firms reducing their risk on other exchanges when they would get traded against on one exchange. They did this without any knowledge that Brad Katsuyama was on the other end, or that he was just late trying to make the same trade at another exchange at a later time. There are no guarantees that you can make the same trade at different exchanges - the same rules apply to everybody.

- Unsurprisingly, IEX as an exchange is no different from others, in that they need market makers (a.k.a. HFTs) to provide liquidity on their exchange. I wrote the code for the FIX gateways to connect our firm to IEX, and it was all business as usual.

⬐ Rimpinths

I work in the industry too and the things that people mislabel as "front running" is really aggravating. At worst, you could call it "order anticipating": using publicly available knowledge to figure out that if someone hit Exchange A and B, they're probably headed to Exchange C next. But they have no inside knowledge that the same party will in fact send an order to Exchange C next. They're taking a risk by anticipating that.

"Front running" as defined by the SEC has a more narrow definition. It basically means that you have a customer that has placed an order for XYZ and you aware of the order, but you placed your own order to be executed in front them, thus forcing them to buy it from you at a higher price than if their order was executed first. HFTs are not "front running" anybody.

IMO Google and Facebook have their hands remarkably clean relative to the rest of the Adtech industry.

I've heard some crazy shit about how much money comes from adtech to fund blackhat data brokers. Adtech buys hacked databases on underground markets, but more than that they fund supply chain attacks to get highly intrusive adware into popular apps. They frequently buy up applications that have a wide install base on phones and browser extensions, and then on the next update, request maximum privileges and use it to loot as much as they can from user systems.

It's a symbiotic relationship. Shady ad networks are often used by criminals for narrowly targeted attacks (advertise this crafted phishing site to women aged 25-35 in the greater Dallas Fort Worth area who are recently married). Those criminals use that access to obtain more private data which they sell to adtech companies. It's a pretty gross business.

In other news, HFT isn't bad because it's HFT, it's bad because order matching services have a bunch of shady, undocumented order types that are designed to allow HFT firms to specifically extract winnings from retail investors. They are absolutely economic parasites, and no one has any incentive to stop them.

https://www.amazon.com/Flash-Boys-Wall-Street-Revolt/dp/0393...

https://www.amazon.com/Dark-Pools-Machine-Traders-Rigging/dp...

⬐ deadmutex

> bunch of shady, undocumented order types that are designed to allow HFT firms to specifically extract winnings from retail investors

Ehn, most of those order types are to try to beat other HFTs. When talking about order types, it's HFT vs HFT.

Michael Lewis' "Flash Boys" is a great read on the subject of building the high speed trading networks. https://www.amazon.com/Flash-Boys-Wall-Street-Revolt/dp/0393...

HFT capitalizes on a kind of time arbitrage. They take the data from one market, and then transport that data using specialized, custom built networks that move the data from one place to another a tiny fraction of a second quicker than other networks. They then capitalize on having that data earlier to eek out small profits at scale.

Imagine two investors sitting at a restaurant table discussing trades they are about to make. The trades they are making will be significant, in the sense that their trades will then impact the value of the stocks they're trading. Meanwhile, a waiter at the restaurant makes a habit of eavesdropping on the conversations of these investors. When he gets the information, he runs to the phone and effects his own, smaller trade.

It's not that the waiter happened to overhear something. The waiter makes it his business to "overhear." The waiter adds no real value. He's a parasite on the people who do add value. The HFT traders are likewise.

What they do is documented in Michael Lewis's Flashboys.

https://www.amazon.com/Flash-Boys-Wall-Street-Revolt/dp/0393...

⬐ pgwhalen

I work in the industry, have read Flash Boys, and am aware of how misleading and occasionally false the book is. It's always disheartening to see someone be so sure of themselves while also being incorrect.

Your metaphor is inaccurate. Your metaphor describes actual front running, which is indeed illegal. It does not describe the sort of latency arbitrage that HFTs do, which Michael Lewis has unfortunately also dubbed "front running" (and succeeded at changing the lay population's definition of the term).

I would try to adjust your metaphor to be correct but it isn't really possible. There is no eavesdropping. HFTs do not know anything about trades before they happen. If they see a large trade, they adjust their prices after the trade happens (possibly on another exchange). This isn't eavesdropping, they are operating on public information, just very quickly. It's the foundation of all market making: adjusting prices according to order flow.

⬐ mariodiana

I'm going by memory of what I read in the book and would be happy to have any misunderstanding on my part clarified. Put simply, I think the book described things this way. (I'll use all convenient numbers and fanciful descriptions.)

A buy is tendered from Timbuktu. The offer will need to travel from Timbuktu to the NYSE. The Flash Boys find out about this buy offer before the sellers using the NYSE system do, so they insert themselves.

The buyer in Timbuktu sees a price of 100 dollars. As are all prices, this 100 dollar quote is "historical." The price will be what it is when the trade is actually effected. The buyer offers 101 dollars. The Flash Boys get ahead of the game, buy the stock at 100.50 and resell it for 101.

That's boiling it down to the essentials. I could be wrong about what the book says. The book could be wrong in its characterization of what goes on. Please, you tell me.

⬐ pgwhalen

I haven't read Flash Boys since it came out, but I'm pretty sure it doesn't actually inaccurately describe how latency arbitrage works the way you just did, it just blurs together similar concepts enough to create confusion about the details, which results in outrage.

The key part that you're missing is that latency arbitrage works only when there are multiple exchanges, and there is only one exchange in your example. Your example suggests that HFTs somehow see the buy order before it reaches the NYSE, but that is not possible. In reality it would be something like this:

- Buyer wants to buy 100,000 shares at $100, but no single stock exchange (there are 13 in the US I believe, soon to be 14) has that many shares available at that price. - But there are 50,000 shares each available at NYSE and NASDAQ each, so they send orders to each. - Their NYSE order arrives first, and the trade happens at $100 for all 50,000 shares - The HFT notices, and seeing that demand is high for the stock, increases their price on NASDAQ to $100.01 for those remaining 50,000 shares. - The buyer's order on NASDAQ does not trade, because the $100 is no longer available

The "information leakage" is from public information only - that a trade happened on another exchange.

Note also that no part of this example discusses the HFT buying a certain price, then selling back immediately at a higher price.

⬐ pgwhalen

Sorry I can't edit, I reformatted the bullets below:

- Buyer wants to buy 100,000 shares at $100, but no single stock exchange (there are 13 in the US I believe, soon to be 14) has that many shares available at that price.

- But there are 50,000 shares each available at NYSE and NASDAQ each, so they send orders to each.

- Their NYSE order arrives first, and the trade happens at $100 for all 50,000 shares

- The HFT notices, and seeing that demand is high for the stock, increases their price on NASDAQ to $100.01 for those remaining 50,000 shares.

- The buyer's order on NASDAQ does not trade, because the $100 is no longer available

It means they are handing over your transaction to HFTs, who can then see your transactions ahead of everyone, and buy the stock right before and then sell to you for a higher price. Imagine you just issued a BUY order at market price; they can buy at $99.9 and sell to you for $100, making a $0.01/stock profit. Rinse and repeat millions of times per day.

This is very similar to "front running" [1], but because a) this happens privately and in small quantities, and b) it is not necessarily privileged information, it's considered legal by the SEC. But it is highly questionable for sure.

"Flash Boys", by Michael Lewis [2] is a great read, and goes into detail on the world of high frequency traders, and how they operate.

[1] https://en.wikipedia.org/wiki/Front_running

[2] https://www.amazon.com/Flash-Boys-Wall-Street-Revolt/dp/0393...

⬐ pochamago

This doesn't sound like it's fucking me, though. I'm still getting the stock for exactly what I asked, correct? I mean, hasn't Robin Hood always been upfront that they make money on the discrepancy between what buyers are willing to pay versus what sellers are willing to take? It's still a free service to me.

⬐ wodenokoto

> hasn't Robin Hood always been upfront that they make money on the discrepancy between what buyers are willing to pay versus what sellers are willing to take?

No, they are quite clear on how they make money and that is not it. They, officially, make money on interests and a subscription service [1].

[1] https://support.robinhood.com/hc/en-us/articles/360001226106...

⬐ noobermin

As long as it's 1 cent no single individual will care. As long as it's less than what a person would pay per trade elsewhere then both sides sort of win I guess...

The problem I suppose is the poor chaps don't know this is what is happening.

https://www.amazon.com/Flash-Boys-Wall-Street-Revolt/dp/0393...

> Maybe you can tell us something about their coding practices. A friend of mine who work at GS told me they have an absolutely massive ball of spaghetti that takes 7 hours to compile. That's for their trading systems.

In case no one with actual experience responds to you, you might want to check out Flash Boys[0], which spends a little time talking about GS's code. From what I've read in that book and in other places, what you heard seems to be the correct.

[0] https://www.amazon.com/Flash-Boys-Wall-Street-Revolt/dp/0393...

⬐ lordnacho

The way they treated Aleynikov is a minus in my account as well.

⬐ kchoudhu

It depends on what system you're talking about.

If you're talking about the core risk management system, yeah, seven hours sounds about right. But the risk management system is a platform that you can use to build other systems that can be iterated upon very, very quickly: we regularly made changes, reviewed them and made them production visible globally in under two minutes).

Flash Boys' technical discussion is, frankly, crap. The larger point about GS' treatment of Aleynikov (sp?) is, however, correct and relevant.

Source: Worked there for 8 years, still suffering from withdrawal.

⬐ cheez

Withdrawal meaning you want to go back? Hard to find a similar environment anywhere else?

That's me in a nutshell, if not you.

⬐ kchoudhu

The amount of irritation GS abstracts away from the programmer is amazing -- and I do miss it very much. I'd like to go back, but circumstances appears to be taking me elsewhere.

It was a good run, and unlike OP's friend, I'd recommend it wholeheartedly.

⬐ cheez

My experience as well.

I'm fascinated by exposes of stock market rigging. Michael Lewis's "Flash Boys" was one of the best things I've read about it.

http://www.amazon.com/gp/product/0393351599/ref=as_li_tl?ie=...

This columnist seems to be using the word "rigged" in a much narrower sense (a small discount being given to large purchases of foreign shares). When talking about stock market rigging, this really seems to be just a small part of the larger problem.