server upgrade / system reconfig

[eFestivals]

I've just ordered another server, to take the efestivals systems up-to four beefy hi-spec web-servers.

Because of the traffic that efests gets, and because of the loading on the current servers at peak times, putting this new server in place will require a re-configuration of how each server interacts with the other servers, meaning that the forums and other website features might be offline for several hours over the coming days at some point (I can't give a time for when as yet).

The result from having this new server will be a much better distribution of the load on each of the servers, meaning that at peak times - such as the days tickets for major festivals go on sale - everything should cope much better than it does currently.

The new server should be in place before Friday - when tickets are going on sale for Reading/Leeds, T in the Park, Big Chill, Download, and other fests - so hopefully the new set-up will get tested to the limit then, and hopefully the improvement will be noticeable by a significant amount.

This extra server is just stage one of a number of changes and upgrades to the efests server setup, so while there should be a very noticeable improvement on Friday, there's further improvements to come over the coming months which should improve things further. How the new system set-up copes on Friday will be used to help plan the other forthcoming changes.

[ampersand]

noooooooooooooooooooooo!!!!

nah, hopefully it all goes smoothly for you

luckily i'll not be stressing out on ticket day so see how everyone else copes.

[Ed209]

I reckon Friday 3rd Dec, at around 9am would be a great time to do the upgrade . Lots of teenagers in tears

Edited November 29, 2010 by Ed209

[rabid]

The new server should be in place before Friday - when tickets are going on sale for Reading/Leeds, T in the Park, Big Chill, Download, and other fests - so hopefully the new set-up will get tested to the limit then, and hopefully the improvement will be noticeable by a significant amount.

[eFestivals]

Really a good time to be doing this?

yep - 100%, absolutely.

The changes that are planned to be made this week (providing the new server gets set-up in time) won't have any negative impact at all. None of the changes that are being made are things which are accessed directly by website users, so there's no impact caused by user-caching (or more correctly, the need to re-cache cos things have changed from the users' side).

[rabid]

yep - 100%, absolutely.

The changes that are planned to be made this week (providing the new server gets set-up in time) won't have any negative impact at all. None of the changes that are being made are things which are accessed directly by website users, so there's no impact caused by user-caching (or more correctly, the need to re-cache cos things have changed from the users' side).

[eFestivals]

Fair enough, you know your systems. Just couldn't imagine getting that one past any change control board I've dealt with.

It would only get stopped by change control if those change control people didn't properly understand the technical aspects of the change. It's a huge win, with no downside.

A change I've just made but didn't want to be making this week is an upgrade to the forums software - but as it included security patches it wasn't something I could ignore. The main downside with having had to do that is that every user will be re-caching the skins, which adds to the overall server load.

(BTW, the bit of downtime just now wasn't anything to do with either of these things - that was the datacentre (or a server it hosts) being DoS attacked).

[rabid]

(BTW, the bit of downtime just now wasn't anything to do with either of these things - that was the datacentre (or a server it hosts) being DoS attacked).

[eFestivals]

PS: ... I guess I should put my confidence in it being sensible to do install & use the new server this week in a bit of context, before I look foolish.

It's quite possible that after doing what I'll be doing, when things get busy on Friday some users might get one or more server error messages and not get the access to efestivals they were expecting.

If that does happen then it'll mean that efestivals has surpassed it previous successful-use record for any moment, and flagged up a further use limitation elsewhere ... but I've taken the decision to not play around with the set-up of the other parts involved and to see what happens; it could be the case that nothing else needs changing and everything is set-up now in the best way possible.

[LondonTom]

Excelllent news, could we schedule the downtime for tommorow? I am in need of doing an essay and the less distractions the better

[windy_miller]

It would only get stopped by change control if those change control people didn't properly understand the technical aspects of the change. It's a huge win, with no downside.

[rabid]

That's the trouble with change management though, the managers rarely know the technical aspects of the change and always err on the side of caution. A good thing from one point of view, fecking annoying from another point of view

[eFestivals]

That's the trouble with change management though, the managers rarely know the technical aspects of the change and always err on the side of caution. A good thing from one point of view, fecking annoying from another point of view

Yep, the "it ain't broke, so don't fix it" view is generally a good one. But with the situation here, things ain't broke most of the time, but they are when things get busy - so it needs fixing.

The change I'm making is all very simple, with no possibility of going wrong outside of an unlikely fault with the new server (and server failures can happen at any time anyway) - while the pay-off in increased capacity is massive (tho as I've intimated above, it might be the case that not all of the possible extra capacity will be realised on Friday as further tweeking might be required).

[eFestivals]

because someone somewhere doesn't seem to have two brain cells to rub together, this is now not happening.

[eFestivals]

It's back on .... and I haven't even had to shout at anybody.

As a result, the forums will be offline for 30 minutes to an hour at some point later today. I can't give a time for when this will be as yet, but I'd guess at around 3pm.

At some point before that, some functionality of efests will be lost for a few minutes when the new server is installed and the connections made to it.

Apologies for any inconvenience with all of this.

[eFestivals]

Hmmmmm .... not sure if this is going to happen or not right now.

[LondonTom]

O_o so did it happen or should I stay off efests tomoz and let those trying for tickets grab space?

[eFestivals]

It didn't happen.

When the customer is having to tell the people who are employed as the technical experts what they should be doing then this customer knows to give up.

To put it mildly, I am absolutely seething.

- They spent two days with the order in front of them but did nothing with it. So when they picked it up to do it they found they didn't have the network card they'd been told they needed so that they could actually fulfil the order.

- They claimed to be building the server when they clearly weren't.

- They couldn't even cobble together properly the most simple network that can be made. I'm not sure why yet, but I'm guessing they have no idea what an uplink port is on a network switch.

The data centre I've been using for the last 10 years very happily have changed a lot of staff recently, and they don't know shit. I'm guessing that it's time for this customer to move on elsewhere.

[mikeb]

New server coming, yay ! ... Oh no it isn't ... Oh yes it is ... Oh no it isn't ... Oh yes it is ... Dontcha just love it when panto season comes around again or perhaps not as the case may be

I feel your pain as I've been suffering similarly over recent weeks with much the same stylee problems on several occasions although with completely unrelated technical stuff. There is nothing that p*sses me off more than finding out that the so-called 'experts' and 'professionals' know considerably less about what they're supposed to be experts in than even a slightly educated punter who doesn't in any way claim to be anything remotely close to being an expert. Always a bl**dy nightmare when previously good companies lose (or quite often dispose of) competent (and no doubt expensive) staff only to replace them with spotty think-they-know-it-all teens (or equivalent) who not only know next to b*gger all in reality but wont even listen to what they're being told or asked to do so end up making a complete b*llock of things over an unnecessarily extended period of time

[eFestivals]

while there's been a few issues related to the necessary hardware for this upgrade (which they didn't bloody order, grrrr), my anger at their failing is much increased because they even-now still haven't done all of the other bits that they could have done in the meantime.

This server was ordered on Monday morning (after a few days of chatting thru various minor options, but them knowing from the off that it was wanting to go ahead), and ordered on the proviso that it was delivered to me no later than 5pm Wednesday. The useless f**kers talked me into allowing them to get it sorted into yesterday, but then it became patently clear they didn't actually have the (not hi-level) skills to do what was necessary.

They're supposedly still working on it and getting it finished ... but I know they're talking crap, I'm logged onto the new server and can see they've not touched it for hours at a time.

The boss at the company is back off his hols next week ... I think these new peeps might find their arses kicked back to the dole office soon after.

[eFestivals]

When are you going to sort out a new URL for me? That dionys-something-or-other.co.uk worked a treat before ... might even donate *gasp* if there could be work-blocking work arounds!

once this upgrade has been done I'll be happy to sort you out with something for a small fee.

[eFestivals]

right, time to get the forums moved across to the new server .... which means them going offline for around 20 to 30 minutes I guess ... apologies for any inconvenience.

[eFestivals]

well, that was painless.

Hopefully you'll now be finding the forums a little speedier.

[windy_miller]

It didn't happen.

When the customer is having to tell the people who are employed as the technical experts what they should be doing then this customer knows to give up.

To put it mildly, I am absolutely seething.

- They spent two days with the order in front of them but did nothing with it. So when they picked it up to do it they found they didn't have the network card they'd been told they needed so that they could actually fulfil the order.

- They claimed to be building the server when they clearly weren't.

- They couldn't even cobble together properly the most simple network that can be made. I'm not sure why yet, but I'm guessing they have no idea what an uplink port is on a network switch.

The data centre I've been using for the last 10 years very happily have changed a lot of staff recently, and they don't know shit. I'm guessing that it's time for this customer to move on elsewhere.

[eFestivals]

stage two of four of the server upgrades and efestivals system reconfig will be happening tomorrow, with the replacement and upgrade of another of the servers that are a part of this whole system.

Hopefully, disruption to people's use of eFestivals will be minimal, and most people are unlikely to notice any issues at all. Any users who do notice any problems shouldn't experience those problems for more than 5 minutes at very most.

Users of festivals.co.uk email will notice a greater amount of disruption. Please see the post just made in the 'Website wishlist and problems' forum for further details.

The date for stage three of the upgrade and reconfiguration process hasn't yet been set. It might happen before xmas, or it might wait until early in the new year.