Tuesday, January 25, 2011 at 1:00 AM
Webmaster level: Intermediate to Advanced
Once in a while we get asked whether a site’s visibility in Google’s search results can be impacted in a negative way if it’s unavailable when Googlebot tries to crawl it. Sometimes downtime is unavoidable: a webmaster might decide to take a site down due to ongoing site maintenance, or legal or cultural requirements. Outages that are not clearly marked as such can negatively affect a site’s reputation. While we cannot guarantee any crawling, indexing or ranking, there are methods to deal with planned website downtime in a way that will generally not negatively affect your site’s visibility in the search results.
For example, instead of returning an HTTP result code 404 (Not Found) or showing an error page with the status code 200 (OK) when a page is requested, it’s better to return a 503 HTTP result code (Service Unavailable) which tells search engine crawlers that the downtime is temporary. Moreover, it allows webmasters to provide visitors and bots with an estimated time when the site will be up and running again. If known, the length of the downtime in seconds or the estimated date and time when the downtime will be complete can be specified in an optional Retry-After header, which Googlebot may use to determine when to recrawl the URL.
Returning a 503 HTTP result code can be a great solution for a number of other situations. We encounter a lot of problems with sites that return 200 (OK) result codes for server errors, downtime, bandwidth-overruns or for temporary placeholder pages (“Under Construction”). The 503 HTTP result code is the webmaster’s solution of choice for all these situations. As for planned server downtime like hardware maintenance, it’s a good idea to have a separate
server available to actually return the 503 HTTP result code. It is important, however, to not treat 503 as a permanent solution: lasting 503s can eventually be seen as a sign that the server is now permanently unavailable and can result in us removing URLs from Google’s index.
header('HTTP/1.1 503 Service Temporarily Unavailable');
header('Retry-After: Sat, 8 Oct 2011 18:27:00 GMT');
If you set up a 503 (Service Unavailable) response, the header information might look like this when using PHP.
Similar to how you can make 404 pages more useful to users, it’s also a good idea to provide a customized 503 message explaining the situation to users and letting them know when the site will be available again. For further information regarding HTTP result codes, please see RFC 2616.
16 comments:
We are on shared hosting and do not always know when the server goes down because of failures of other websites.
When the server goes down/hangs, how do I send a 503? That's impossible as far as I can tell because there are no pages served...
Does this also apply for people who hosts their sites on the Blogger environment?
I think Google can do better than this. Just exactly how many web site owners do you think are capable of returning another obscure code that no one but the W3C has ever heard of?
Here's what we would like to hear from Google about crawl errors which are the result of a site being temporarily down:
"Google understands that web sites are run by humans and computers require maintenance from time to time. We won't penalize a web site that is down occasionally. If you would like to tell Google about planned maintenance incidents for your website, you may do so in our Webmaster Tools website."
We got pretty much killed by a period of site downtime a year or two back and never recovered.
A few months previously our traffic had dropped to near zero and we'd found some 7000 broken links in webmaster tools. Turned out the Googlebot had deep indexed thousands of theoretical search results in our store and when we change store software they all came up as broken links.
So we fixed that and most of our Google traffic returned - but then our webserver fell over and our hosts took 24 hours to bring it back up. Nothing we could do about it. Google traffic went through the floor again and that was that - never recovered.
So, yes, dealing with planned downtime is essential - but unplanned downtime needs looking at too, and that may be something that Google needs to talk to webhosts about - especially with blogs, personal sites and Mom and Pop businesses where the site owner has little or no control over the webserver.
This is so much better than just, say, throwing in an appOffline.htm in an ASP.NET installation, which, I believe, triggers a faulty 404 repsonse.
@Challenge This is about planned downtime. Unplanned downtime usually gets a 5xx response anyway.
@Outta Why should Google implement something of its own, when there is a perfectly good standardized way of doing exactly that with HTTP headers? Also, Google isn't the only heartbeat tool that you would have to notify that way. Make your servers respond with the correct status codes to tell auxiliary systems about your downtime in a centralized controlled way.
If you are using WordPress you can use "Site Maintenance" plugin which automatically creates a 503 page for you. http://wordpress.org/extend/plugins/wet-maintenance/
Note: upon activating this plugin puts your site into maintenance mode. simply deactivate the plugin to bring your site back online.
Hi!
This might help. On Apache Servers, it's usually not possible to redirect a user with any other HTTP Status Code than of the 3xx class.
Users that visit a site under maintenance deserve to get a useful information page.
Do this with a 302 redirect first that leads to a 503 page that outputs the 503 header (i.e. via PHP or Perl). This will work fine with Google since Google will assign all target page properties to the source page when they encounter a 302 - including the header response. I've tested this on Virtual Hosts and it works as desired. If you can, choose a time with few requests for the service actions.
Regards, Thomas
My site is on a shared hosting platform and I have no idea when and how long the downtime lasts whenever their servers crap out. Very frustrating. Just how badly does downtime affect rankings?
So, yes, dealing with planned downtime is essential - but unplanned downtime needs looking at too, and that may be something that Google needs to talk to webhosts about - especially with blogs, personal sites and Mom and Pop businesses where the site owner has little or no control over the webserver.
Guys what do you think doing 302 redirection for all pages to 503 page which generate 503 http response code is good or not?
(This is only for planned server maintenance / server down / database down cases)
Are there restrictions on the period? I need to close a website in 3 days. Is it too long if I use 503?
Are there restrictions on the period? (yes, I'm another asking!)
We need to close down our blog for a FULL MONTH. How should this be handled? 503 of course, if it isn't for too long?
Google haven't provided any specific info about time periods :( Not that I'm aware of.
Any info would be appreciated!
Just for an FYI to anyone interested in my solution.
I host several web sites. I have set up one special box with a single purpose. When the main web server goes down (for any reason) I manually tell my router to redirect all HTTP/HTTPS traffic to this special box.
This box responds to all requests by default. And any page not found (pretty much every request) will kick out the same error page.
I have a friendly looking page that tells the user to try back in a few hours.
This same page regardless of the issue returns the 503 header (even for page not found because that is expected) and tells the bots to try back in 4 hours.
Currently, this switch over is a manual process because I have to switch my router. But it could be done with a content switch.
If you host on a remote machine, this still works but you need to change your DNS entry which can take many hours to echo through the internet. So I would only do that if things were going to be down for a long time.
When the main server is back, I just change the router to point back to that box and all the sites stop reporting 503 because the real server is responding now.
I took my site down twice in the last couple months for various types of maintenance. I used the 503 method and I didn’t see any bad impact on Google or other search engines. I was rather annoyed that Google Webmaster Tools insists on logging thousands of 503 messages as errors in my reports and emailing me to let me know that my site is down. Of course if a 503 is given the operator knows it is down. 503 can’t just happen by mistake.
It would be VERY HELPFUL if Webmaster Tools had a way to tell Google to not send us those emails and disregard 503 messages because they are not actually errors. But they are currently showing up in the error graphs. 503 is not an error it is a directive saying "not available right now" and often "come back later".
How frigging annoying!
I had a minor database update on my site and of course my site issues 503 and a 2 hour retry directive.
In the 3 minutes my site was down, Google web bot scanned 19,000 pages and logged them all in my Webmaster Tools as errors and now I will spend the next 19 days clearing those errors 1,000 at a time.
This is highly annoying. Can we clear all errors rather than only 1,000 per day or get the 503 out of the error log or actually retry as directed by the web site? Clearing all errors would be my preferred fix and let it work on all errors not just 503.
Post a Comment