Seetickets - what to do next year...

[matt_berr]

So being in the IT industry and working for a company that does high volume online sales - I thought I'd try and explain why the backdoor worked and why seetickets should employ me (that has got to be surefire way to guarantee tickets).

This will end up with Seetickets management via some route or the other.

------------------------------------------------------------

What was the backdoor (or more accurately the other front door)?

When you type in web address - in this case glastonbury.seetickets.com - your PC references a DNS record on the internet to get information about where to go to get a connection - this is an IP address, a numerical location where a website is located. Think of it as a phone book giving a number for a named address.

In this instance Seetickets had two IP addresses that the address glastonbury.seetickets.com should resolve to (techy term for returning an IP address for a web address).

What should have happened when I typed in glastonbury.seetickets.com was that either:

194.168.202.201

194.168.202.202

Would be returned to my browser and I would connect to that location, these are allocated to visitors on a round robin basis - so 50% get one IP address and 50% the other.

What happened was that a typo meant that only one correct address was returned when going to glastonbury.seetickets.com (a bogus address 192.168.202.201 was put in by error, an assumption on my part is that if a DNS record doesn't resolve to anything then the next correct IP address would be returned, feel free to correct if anyone knows better).

In this case 194.168.202.202 was returned to all users who went to the ticket site.

What did this mean?

Seetickets, as is common, split their internal systems into two halves - each of these halves was addressed by one of the two IP addresses.

So in theory going to glastonbury.seetickets.com would send 50% of users to 194.168.202.201 and the other 50% to 194.168.202.202.

What happened was 100% went to 194.168.202.202.

This left half of Seetickets internal systems 50% under utilised.

The backdoor fix involved changing a file on your PC called 'hosts' - this file is consulted by your PC every time you go to a web address. If there is an IP address for a web address specified in your host file then instead of going to the Internet to get the IP address you will use the one on your PC.

So by having the line below in your host file:

194.168.202.201 glastonbury.seetickets.com

You would go to the half of the Seetickets systems that were massively underused (assumption is internal Seetickets telephone bookings were captured here).

I used this and got 7 tickets 8-)

So here is some thoughts on this:

This is poor IT management by Seetickets, why:

Don't use DNS to balance load on your internal systems (use something like http://www.riverbed....ducts/stingray/), few reasons behind this:

1. It is more difficult to control and test (see what happened + other reasons)

2. DNS will continue to send requests to systems that are down or struggling on a round robin basis - better systems hold traffic back in a more graceful way (not the standard unavailable screen but something meaningful and attractive) and balance where customers go based on how well the systems are holding up. So if one half of your system is working quicker than the other more users go there.

3. DNS info can get cached which means even if you change it - that change may not be reflected to your customers for some time.

Where was your monitoring?

This was your biggest day of the year, for days like this you should monitor everything. If 50% of your systems were idling you should have known and pretty quickly fixed it.

My assumption is you didn't even know this and was only alerted when the backdoor method hit twitter.

So if you weren't monitoring how did you know what your capacity was and when you would breach it?

Based on my experience because 100% of the traffic hit one half of the systems some customers didn't get tickets, at 8:55 I reached the registration check screen only to have it 'lock up' and freeze (before I got the backdoor).

This froze because Seetickets servers got too busy, why did you get too busy - because you were not monitoring the right things and 100% of the traffic hit 50% of the infrastructure.Imagine how gutted I was to get to this screen only to not get past it because of amateur systems management - luckily I found another way.

This easy to address.

I could go on.....

Looking at Linkedin the Senior IT Mgmt at Seetickets have all been there for about 10 years, the methods used by the systems are archaic and give a bad customer experience you need some fresh blood from companies that do this better.

With little investment and bit of thought you could sell all of the tickets in half the time with no backdoors and without system issues at Seetickets impacting customers.

This is fairer all round.

NFRNFC

Matt

[whisty]

The assumption is that tickets would have sold out in an hour had the IP been set up correctly. Failing to get tickets in an hour or 2 hours makes no difference to the applicant. I couldn't care less how they set their system up as long as I get a ticket. Personally (and I know it's probably statistically untrue) I'd feel like I had more of a chance with the 2 hour window.

Edited October 27, 2012 by whisty

[timbrass]

Can anyone here explain what happens when routers get overloaded (i.e. before the traffic gets to the Seetickets load balancers). So there are users on different ISPs (BT, Virgin etc) - do the routers along the paths to the Seetickets load balancers communicate with each other to tell routers further up the chain to "back off". In other words are you at an advantage if you are "closer" (number of router hops) to the Seetickets load balancers? Or do all requests hit Seetickets and it really is random who gets in. For example would Virgin routers send a messages to BT routers to tell them the destination address is under excessive load. In other words do nodes on a Virgin network (where Seetickets live) have an advantage?

[matt_berr]

The restricting of load will be done at see tickets, they will have a firewall which restricts the number of concurrent connections.

The best ways to get in are:

Be trying when they open the gates, before 9am, in theory this is when the most free sessions are available at one

Have lots of browser sessions refreshing at once on lots of devices

There isn't real ISP or broadband speed advantage once your bb speed is over 1mb or so

[dingbat2]

Errrr, if you are an IT wiz, and know all this stuff, how come you didn't get a ticket in the first sale?! I am IT illiterate and I got a ticket using the good ole method of loads of windows and F5

[matt_berr]

I did get a ticket using the back door method, else I would have done the same as you which is the best method.

I did get tickets via the backdoor method, this was afeter being gutted by see tickets letting their servers getting overloaded which as half their system was getting 100% of the load which initially resulted in a lot of people getting frozen on the registration check screen and them limiting the traffic to cope with this. This initially happened to me

Was a bit bored at work today so thought I'd work through what happened based on what I saw and what I know.

As has been noted if everyone has the same issues then its fair, in this case those with a bit of tech know how and efestivals got an advantage which worked out well for me

[abdoujaparov]

*yawn*

bored with this now...

[Jitterbug_87]

i would prefer if everyone got in a good old fashioned queue (real life, phone or net i dont care) but if everyone was in a queu i think t would be fair

[markeee]

The restricting of load will be done at see tickets, they will have a firewall which restricts the number of concurrent connections.

The best ways to get in are:

Be trying when they open the gates, before 9am, in theory this is when the most free sessions are available at one

Have lots of browser sessions refreshing at once on lots of devices

There isn't real ISP or broadband speed advantage once your bb speed is over 1mb or so

[loosechange]

GFL employed See to sell all the tickets.

See sold all the tickets - probably not as quickly as they could've done ideally but it's probably not one of the contract KPIs.

The end.

[markeee]

I think it's fine as it is normally - ok the DNS issue was annoying but normally site going down..well happens to everyone so we're all in the same boat

its frustrating it dies a lot on t day..but it does for everyone..so we;'re all on a level playing field!

Edited October 27, 2012 by markeee

[Craigston]

I failed for the first time ever to get tickets and although I've got over it, I need to understand this.

Would using a LAN of pc's at work be a disadvantage? They Seemed to be all blocked for 90 minutes by which time I'd left for home to try there also without success. The workstations were 'locked' so the back door thing wouldn't work!

Also why did we get the booking page - which in previous years usually means the rest of the buying process was rock solid - and this year get it a few times put in all the details only to be thrown out?

I think see have definitely changed their system - Emily alluded to this in a tweet - they have levelled the playing field somehow to make it more of a lottery.

This is fairer of course but doesn't make you feel any better when you're among the several hundred thousand who aren't going. Just have to dredge up happy memories of the nine previous fests I've been to - which helps a lot.

Resales/oxfam working pass permitting, still hope to join you all

Edited October 28, 2012 by Craigston

[mandolin]

Ok, I'm about to really show up my stupidity but, anyway, here goes.

What is the f5 button supposed to do? I've read that it refreshes the page you are looking at but on my laptop it always brings me back to my homepage. Is there any downside to using the refresh icon instead of f5? Or is there something I should tweak (and, if so, how?). Just think of this as ridiculous forward planning for next year!

[UEF]

It depends how the site is written to be honest. If the site you're on updates the URL in the address bar as you move through the process then you may find it useful, but if the URL stays the same the whole time it may or may not help.

[Becoming a veteran]

I

Ok, I'm about to really show up my stupidity but, anyway, here goes.

What is the f5 button supposed to do? I've read that it refreshes the page you are looking at but on my laptop it always brings me back to my homepage. Is there any downside to using the refresh icon instead of f5? Or is there something I should tweak (and, if so, how?). Just think of this as ridiculous forward planning for next year!

[Kareneh]

Can anyone here explain what happens when routers get overloaded (i.e. before the traffic gets to the Seetickets load balancers). So there are users on different ISPs (BT, Virgin etc) - do the routers along the paths to the Seetickets load balancers communicate with each other to tell routers further up the chain to "back off". In other words are you at an advantage if you are "closer" (number of router hops) to the Seetickets load balancers? Or do all requests hit Seetickets and it really is random who gets in. For example would Virgin routers send a messages to BT routers to tell them the destination address is under excessive load. In other words do nodes on a Virgin network (where Seetickets live) have an advantage?

[BA4 4BY]

The more I read about this the luckier I feel to have got a ticket.

I'm no expert when it comes to computers, so my 'logic' process led me to believe that constantly refreshing several browsers on the same computer would just lead to them fighting each other and slowing them all down. I thought it would be a similar story when it came to trying with more than one computer on the same internet account, i.e. they would have to share the bandwidth between them, hence reducing the likelihood of either of them getting through.

So having reached this conclusion, I sat with one browser (Chrome) open on my laptop and kept F5-ing until I got the 20 seconds auto re-attempt screen, then I would leave it to re-attempt connection automatically every 20 seconds until I got thrown out of that page, then back to F5-ing again and so on. After a handful of false starts and frozen screens etc I eventually got my ticket at 10.10 am, which I now understand was around the time that See fixed the problem.

Ignorance is bliss as they say, but I think I'll prepare myself a bit better for next year after realising how lucky I was this time!

[jimmyk83]

My Virgin router got me nowhere