{"id":41888,"date":"2023-03-15T14:46:35","date_gmt":"2023-03-15T18:46:35","guid":{"rendered":"https:\/\/centricconsulting.com\/?p=41888"},"modified":"2023-04-11T14:48:33","modified_gmt":"2023-04-11T18:48:33","slug":"automation-challenges-the-tech-view-on-the-ticketmaster-meltdown","status":"publish","type":"post","link":"https:\/\/centricconsulting.com\/blog\/automation-challenges-the-tech-view-on-the-ticketmaster-meltdown\/","title":{"rendered":"Automation Challenges: The Tech View on the Ticketmaster Failure"},"content":{"rendered":"

We address the Taylor Swift Ticketmaster meltdown from a tech perspective, unpacking how bots can be both problems and solutions and how automation can help prevent a similar disaster.<\/h2>\n
\n

She can make the whole place shimmer, and she can make the whole site break. On November 15, 2022, millions of fans entered Ticketmaster\u2019s online queue for the Eras Tour, Taylor Swift’s first headlining concert tour since 2018. Fans spent hours waiting for tickets, while Ticketmaster, overwhelmed by the volume of users, paused the queue several times and delayed ticket sales for hours before finally shutting down the sale altogether.<\/p>\n

By the time Ticketmaster stopped the presale, it had already sold approximately 97 percent of tickets for the Eras tour \u2014 that’s 2 million tickets, the highest number of tickets sold for a single artist in history. While 1.5 million fans received access to the November 15 \u201cpresale\u201d of Eras tickets, 14 million users, including bots, swarmed Ticketmaster all at once.<\/p>\n

In its apology blog post<\/a>, Ticketmaster emphasized the amount of web traffic flooding its site, \u201cBased on the volume of traffic to our site, Taylor would need to perform over 900 stadium shows\u2026that\u2019s a stadium show every single night for the next 2.5 years.\u201d<\/p>\n

Ticketmaster\u2019s meltdown<\/a> sparked a barrage of tweets, posts, and TikToks from Swift fans, as well as an ominous warning from Congress.<\/strong> Last month, members of the Senate antitrust committee met with executives of LiveNation, Ticketmaster\u2019s parent company. How Ticketmaster prepared for Taylor Swift\u2019s presale wasn\u2019t the topic of conversation \u2014 monopolies were. LiveNation controls roughly 70 percent of ticket sales in the events industry, and members of Congress questioned whether Ticketmaster had something to gain from the bot-driven Eras presale chaos.<\/p>\n To understand what happened during the Taylor Swift Ticketmaster meltdown, we need to look at the build-up to the presale from two perspectives: marketing and tech. <\/span>\n

Marketing to the Masses<\/h2>\n

In August 2022, \u201cAll Too Well (10 Minute Version) (Taylor\u2019s Version)\u201d won a VMA for video of the year. Taylor ended her acceptance speech<\/a> by announcing her new album, \u201cMidnights\u201d:<\/p>\n

\u201cI had sort of made up my mind that if you were going to be this generous and give us this [award], I thought it might be a fun moment to tell you that my brand-new album comes out October 21. And I will tell you more at midnight.\u201d<\/p>\n

An accompanying Instagram post soon followed, Swift describing \u201cMidnights\u201d as a collection of \u201c13 sleepless nights scattered throughout my life.\u201d From then on, Taylor\u2019s marketing campaign swept the Internet, encouraging fans to experience their own sleepless nights.<\/p>\n

In late September, Swift began \u201cMidnights Mayhem with Me,\u201d a series of TikToks<\/a> set to snappy seventies game show tunes. Every few days, always at midnight sharp, Swift released a TikTok in which she would draw a numbered ball from a bingo cage and announce the name of the track that corresponded to the number. The TikToks, later posted to her Instagram story, drew massive attention, with most TikToks blowing past the 10 million viewer mark within a day. The night before her album\u2019s release, Taylor reached millions of people outside her fanbase (affectionately known as Swifties) by releasing a teaser trailer of \u201cMidnights\u201d music videos during the third quarter of Thursday Night Football.<\/p>\n

Then, only ten days after the release of record-breaking \u201cMidnights,\u201d whispers of a tour appeared on Twitter. Swifties spent Halloween leaking various combinations of potential venues and dates before Swift herself announced the Eras Tour, a \u201cjourney through the musical eras of [her] career (past and present!),\u201d a day later. Several factors beyond the album\u2019s extensive marketing campaign foreshadowed that this tour would have unprecedented demand.<\/strong><\/p>\n

It\u2019s no surprise that more people want to purchase concert tickets after the pandemic, but Swifties are especially invested in the Eras Tour. Lover Fest, a worldwide tour of Swift\u2019s 2019 \u201cLover\u201d album, was slated to take place during the summer of 2020. Swift postponted the tour, then canceled in early 2021, giving all fans refunds.<\/p>\n

Since the would-have-been Lover Fest, Swift has released three new albums, \u201cfolklore,\u201d \u201cevermore\u201d and \u201cMidnights,\u201d and has re-recorded two of her most famous albums, \u201cFearless (Taylor\u2019s Version)\u201d and \u201cRed (Taylor\u2019s Version).\u201d All of these details \u2014 a large and extremely loyal fanbase, an exceptional amount of new music, a well-executed marketing campaign, and a tour that will speak to every \u201cera\u201d of Taylor\u2019s career \u2014 contributed to the unprecedented demand for Eras tickets.<\/p>\n

To purchase tickets, fans were required to register with Ticketmaster\u2019s Verified Fan program, an effort to ensure that real people, not bots, would log on to purchase tickets.<\/strong> On the day Swift announced the Eras Tour, over 3.5 million people registered for the Eras presale \u2014 the largest registration in history \u2014 but during the first few hours of registration, Ticketmaster lagged. And glitched. And paused.<\/p>\n

The demand was there \u2014 so much, in fact, that Swift added seventeen extra shows to accommodate these large registration numbers. And, Ticketmaster\u2019s site faltered under the number of eager fans long before the day of the presale.<\/p>\n

The Tech and Automation Challenges<\/h2>\n

Swifties anticipated competition amongst themselves, but they didn\u2019t predict their biggest competition: bots, software designed to carry out task-oriented actions by mimicking how we click, type and interact with interfaces. While bots sometimes have a negative connotation, they are like any other tool.<\/strong> Developers can program bots to surf the Internet for information, identify specific services and report data to people who can use it to evaluate options and make decisions.<\/p>\n

Approximately 40 percent of Internet traffic is bots, and most of us interact with them daily, whether using a chatbot to make an online return or asking Alexa to play our favorite song. People frequently use bots in the ticket sales industry, sweeping the Internet for information on ticket inventory and pricing to determine how to get the best deal.<\/p>\n

Put bots in non-ethical hands, though, and things can get tricky. By programming bots to create hundreds of new accounts, each of which can purchase six tickets or hack into real people\u2019s accounts by inputting easy-to-guess passwords, people can use bots to buy highly sought-after tickets in bulk.<\/p>\n

Some bots can bring in hundreds of tickets in less than a minute, allowing the people who programmed them to resell those tickets at a marked-up price. Additionally, bots seek out the best seats, trying to scrape up VIP tickets or front-row seats while leaving the back of the stadium untouched.<\/strong> The more demand there is for an event, the more likely an army of bots is at the ready.<\/p>\n

Thinking Back: How Ticketmaster Could Have Anticipated these Automation Challenges<\/h2>\n

We\u2019ve examined the dynamics of the flood and demand for Taylor Swift tickets from a marketing and tech perspective. Now, let\u2019s look at the dam, the architecture of Ticketmaster\u2019s website.<\/p>\n

Unlike many Swifties, Ticketmaster knew bots would be an issue during the Eras Tour presale. To combat unwanted bot traffic, Ticketmaster offered their Verified Fan service to Taylor Swift. Artists who opt to use Verified Fan want to limit, if not eliminate, buyers who use bots to resell tickets for profit. By requiring people to register themselves as a \u201cverified fan\u201d in advance of the day of ticket sales, Ticketmaster has the opportunity to \u201cverify\u201d registration requests and filter out the bots from the fans, so on the day of ticket sales, wait times are shorter and the only people waiting are real people.<\/strong><\/p>\n

As another bot-preventing measure<\/a>, Ticketmaster uses CAPTCHAs before a fan can purchase tickets. A CAPTCHA is an automated test designed to determine whether you are doing something or whether a bot pretending to be you is attempting to access your account. CAPTCHAs often appear when you log into an account you haven\u2019t used in a while or when attempting to purchase something on a website.<\/p>\n

This test can either be a random assortment of distorted letters and numbers you must type into a text box or a grid of images, accompanied with an instruction like \u201cselect all images with crosswalks.\u201d Bots cannot identify distortions or evaluate images like people can, so while Verified Fan registrations help weed out bots on the front end, CAPTCHAs work on the back end as a final bot buffer.<\/p>\n

Despite these efforts, millions of bots could enter the Taylor Swift presale. What went wrong?<\/p>\n

Demand<\/h3>\n

While Ticketmaster recognized the level of ticket demand, they didn\u2019t anticipate the extent to which demand would drive enormous bot presence.<\/strong> Experienced resellers would have recognized the combination of a loyal fanbase, a successful marketing campaign, and an extended time away from touring. These resellers would have inferred that fans would be willing to pay a large amount to see Taylor Swift in concert.<\/p>\n

Such a demand also incentivized more people to try their hand at reselling for a large profit. Additionally, loyal fans who are adept at programming might have wanted to ensure they would get their hands on a ticket, and they knew how to use bots to do it.<\/p>\n

While Ticketmaster was prepared for 15 million fans and the predictable number of bots, they didn\u2019t have an architecture that could scale if online presence was double, triple, or in this case, nine times the anticipated amount. In other words, Ticketmaster prepared for big, but they couldn\u2019t handle massive.<\/p>\n

Verified Fan Program<\/h3>\n

Fans registered and verified as \u201cnon-bots\u201d received an access code, which they would type on their website to enter the ticket queue. These access codes were a combination of letters and numbers. However, because of their random nature, these combinations would be easy for bots to guess and replicate, much like a password of 1234.<\/p>\n

CAPTCHAs<\/h3>\n

Automation is not the only tech tool people can use to purchase tickets. While CAPTCHAs can prevent bots, they are not foolproof against AI, whose capabilities include image recognition. Most images on CAPTCHAS are from Google Earth, a platform easily accessible to AI programmers.<\/p>\n

Several YouTube videos show how a bot that used AI clicked through a complicated CAPTCHA without issue. Most sophisticated websites no longer use CAPTCHAs for this reason.<\/p>\n

Looking Forward: How You Can Use Tech to Reduce Bot Problems<\/h2>\n

Any website like Ticketmaster that offers access to high-demand activities like plays, sporting events or concerts must have a sophisticated and flexible response to an automation challenge like bots. Bots didn\u2019t appear from thin air. They are intelligent tools used by intelligent people, and you can use them too.<\/p>\n

Fortifying your website against unwanted bots is like playing chess. You must prepare for someone to uncover your strategy and beat it, and you must have a backup strategy on hand to counter your opponent\u2019s next move. There is no easy, one-size-fits-all way to eradicate bots from your website because people can automate anything easy and repetitive<\/a>.<\/strong><\/p>\n

Before jumping to preventative measures, you should first assess company policy on automation. If you decide you will not allow external bots \u2013 automation that isn’t from your organization \u2013 you can gather information about the types of users on your website by programming bots to analyze:<\/p>\n