During Christmas, we were treated to an EPIC day of spectacular proportions and this post relives that moment for those that were not around to bear witness to it.
While betting exchanges and the Betfair exchange, in particular, have brought many opportunities. As complex systems, they sometimes throw a bit of a curveball. Complex systems tend to be difficult to trap errors, simply because you can’t tell how they will manifest themselves. Sometimes these can throw up something quite spectacular.
Now and again you see odd moments on Betfair trading markets but in 2011 we hit a whole new ball game. Throughout my Betfair trading career, I’ve seen some odd things. But Boxing Day 2011 was on a whole new level!
Leopardstown Christmas festival
Each year we get a busy week between Christmas and the New year and once out of the Kempton King George meeting, we move into a quieter period. But the Leopardstown Christmas Festival still brings us some quality and is worthy of a look.
The are many big races at Leopardstown during this week and one of them is the Christmas Hurdle. On the day of the hurdle, the meeting overall has prize money that often exceeds all the other meetings combined. That’s why it is a focal point for me and a key trading target during this busy Christmas period.
Leopardstown Christmas Hurdle 2011
The 2011 meeting at Leopardstown and the Christmas hurdle, in particular, will always go down in Betfair history for something that went very, very wrong.
The exchange is the epitome of fairness. You lodge money upfront to place your bets and all bets are honoured. With a third party bookmaker, there are reasons not to pay out but not on the exchange.
On an exchange, you see a price and you can take it safely in the knowledge that it will be honoured. Sometimes there is the odd error, but rather than being a Betfair error. It’s usually an error on behalf of an errant bot or person on the other side of the exchange.
But regardless of the error, the bet stands. That is one of the beauties of exchanges.
Christmas hurdle 2011
In a fairly standard race, Voler La Vedette ran home to take the win. There was nothing unusual about the win and I pretty much ignored the race, focused on the next market that was about to arrive. But there was something really odd about the in-play market.
For most of the race, £21m was available to back at good odds. Even as the horse pulled away to stride past the post. You could be forgiven for thinking that it should be 1.01 to lay as it strode miles past the post. However, it was still available to back at odds of 29, not only that you could do it for incredible amounts of money! Never before had such a thing been seen before on a betting exchange. Yes, you see some odd things, but this was bizarre. You had £21.4m available to back at decimal odds of 29! Not only that but this was on a feature race, the Christmas hurdle at Leopardstown racecourse.
There was so much money that nobody was able to take all that money in-running. ‘Only’ £1.6m was matched, and the loser would have to pay out a cool £23m to the backers. It seemed too good to be true.
Unfortunately, it was, the market was never paid out.
Betfair did, eventually, give a rough explanation of a technical glitch that occurred that had been spotted and corrected so it could never happen again. But there weren’t huge amounts of depth to the explanation, though there were a few plausible theories.
When we looked at the data we could see, honestly by coincidence, that the amount of money that was being offered was the maximum positive value for a 32-bit signed integer. For the less technical amongst you, the maximum amount allowed by some computer systems. Was this the cause? Some really obscure bit of code, a lack of error trapping or a weird database issue. Most likely, but I guess the reason for not offering a full explanation was to avoid future exploits? Since then the API and the infrastructure behind the exchange has changed significantly. So I am guessing that this exploit probably doesn’t exist anymore. But it was a ‘funny’ thing to see.
Meltdown & IBAS
Everything went into melt-down that day. For the full story, you need to read the full blog posts, but also the forum posts for the race. I’ve listed them all for your enjoyment. It will be a day never to forget!
As though the meltdown in the Betfair trading community wasn’t enough. The meltdown in the wider betting and trading community went on for some time. Betfair took the unprecedented step off voiding all bets in the market citing a software failure caused by “a unique set of circumstances’.
People who had, legitimately, backed Voler La Vedette for the win at 29 were left wondering why they hadn’t paid out, as other exchange errors always had been. This brought up the spectre of counterparty risk on the exchange.
Who actually was on the other side? It turned out an account with just £1k in it had accidentally exposed a freaky error on the API.
200 punters took advantage of the error. When Betfair voided some of those decided to go to IBAS, the independent betting adjudication service. The full reason for the error was never fully made public, but for anybody that has coded or knows something other than denary in number terms. It pointed clearly towards a certain type of error. Which was subsequently closed down by Betfair. No further action was required by the gambling commission, it was just one of those freaky things that can happen with technology.
A betting exchange equivalent of a ‘flash crash’.
The trigger was a guy running a ridiculous bot on the API on one of his unfunded sub accounts. It was doing a martingale over and over again, going from tiny to huge stakes in a cycle. The max stake on the exchange was some very large number, I don’t remember what, but bigger than 2e31. Two things happened at the exact same moment: this martingale bot tried to place some huge bet way over the numeric max (and crucially, over that subaccount’s exposure limit) this caused a numeric error inside the trading engine exposure checking code. This bet was rejected and you never saw it: all good. Unfortunately, the long dormant bug activated and allowed another bet which was not over the numeric max stake but was over the exposure limit for the account (it had nothing like the 20m to bet in it) to skip the exposure check. Because it was a valid stake and the exposure check was missed, it went into the market and became good value pretty quickly. The nature of the bug meant that both bets had to be placed effectively at the same time, otherwise the tiny window of time that the numeric error opened up for the second bet to skip the exposure check would be closed again. All involved in all aspects of handling this bug at the time were given the usual spiel of secrecy, which was fair because the bug was exploitable for a little while until fixed. However, it was fixed long ago (although similar bugs probably existed because the codebase was very difficult to reason over) and now that whole trading system has been rewritten in a more modern language with a safer architecture (and it’s much much faster). I imagine there are all sorts of bugs in it, but they won’t be this type.