Skip to main content

Making live changes on large high-traffic sites

Somehow I seem to often be in the position where I need to make changes to large, complex, high-traffic sites - changes to the live site that is... Yes, they should have dev/staging servers, but they don't, and I'm not in a position to make them set it up...

So changes must be made with a minimum of disruption to the site. Usually the best way to do this is to make a test copy of the page that's going to change and work with that - so that if you break something it doesn't affect the live page. But in some cases this isn't feasible - for one large site a change will involve a cascading set of php files and smarty templates - and I'd have to make copies of all of those, plus change the includes in the right place etc. to make it all work. Then undo all that to make the changes live...

Not the best way to set up a system, but that's a topic for another day. So how to minimize disruption?

One way I've alluded to before - take advantage of Firebug by putting the new section or widget or whatever into a div or table or whatever that's set to display: none - that way this possible screwed up code won't show up because it's hidden on the client-side. It's still generated by the server, but it won't affect layout etc of the live page. Then you can use Firebug to make it visible (set it to display: block;) and see how it looks.

It can also be tricky to output error messages when debugging changes. Obviously you don't want them showing up to users. Sometimes you can just put them in as commented out html and view the source to see them, perhaps because they're coming from deeply buried php code and won't actually make it to the browser. In this case, sometimes you can dump them into a variable in the templating system. Smarty is great for this - you can put them in something like 'aaaaaa' and then set smarty to show it's debug screen only for your IP address. Then you can look on that debug screen for your 'aaaaaa' var and see what's in it.

Perhaps those tips can help someone out there make the best of a bad situation - this is why staging servers should always be available, but in the real world that's not always an option!
Labels:

Comments

Post a Comment

Popular posts from this blog

Another VI tip - using macros, an example

God I love VI. Well, actually, vim but whatever. Here's another reason why. Suppose you need to perform some repetitive task over and over, such as updating the copyright date in the footer of a static website. (Yes, yes I know you could do a javascript thing or whatever, just bear with me.) Of course you could just search and replace in some text editor, changing "2007" to "2008" (if you're stupid) - and you'll end up with a bunch of incorrect dates being changed, most likely. What you need to do is only change that date at the bottom. And suppose that because of the formatting, you can't use the "Copy" part of the string in a search replace - perhaps some of the pages use "©", some spell out "Copyright" etc. This is where vi macros come in handy. A macro in vi is exactly what you expect, it records your actions and allows you to play them back. To start recording, press q followed by a character to use to "stor
6 comments
Read more

Using FIle FIlters in FileZilla

Here's a handy tip for situations when you want to download a large number of files - but only of a certain type. For example, perhaps you want to download all the PHP files from a largish website, scattered through many subdirectories. Perhaps you're making a backup and don't want any image files, etc. FileZilla (still the best FTP in my opinion) has a handy feature called filename filters - located under the Edit menu. Here you can set various filters that filter out files based on their filename. Took me a minute to figure that out - you're saying show only PHP files, rather you're saying filter out files that do not have ".php" as their suffix. For some reason, that seems a little backwards to me, but whatever. It works quite well. You can also check whether the filter applies only to files, only to directories - or both. In this example, you'd want to check only files, as otherwise you won't see any directories unless they happen to end in
1 comment
Read more

Debugging a DOS

I'm not a sysadmin, but I end up doing my best now and then when one of my sites gets into trouble. This is a sort of "after action report" of an incident that I just resolved (hopefully). I woke up and happened to check email on my phone (don't always do this, will now) and was greeted with a uptime robot email that one of my sites was down, and had been for about 4 hours. I quickly checked the site on my phone and yup, it wasn't loading. Ran to the office and hopped on my laptop. SSH to the server, and everything seems fine. Very little load on the server (AWS instance). Did a restart of apache/php/mysql and the site is still down. Weird. Running the site's index.php file on the command line works as expected and fast. Ask a few other people to check, and it's down for them. Then I logged into the AWS console and checked on status there - everything is up and running.... WTF? This is a lightsail instance, and then I noticed the outgoing network traffic h
Post a Comment
Read more