Thursday, December 04, 2008 at 12:11 AM
We've upgraded the crawl rate setting in Webmaster Tools so that webmasters experiencing problems with Googlebot can now provide us more specific information. Crawl rate for your site determines the time used by Googlebot to crawl your site on each visit. Our goal is to thoroughly crawl your site (so your pages can be indexed and returned in search results!) without creating a noticeable impact on your server's bandwidth. While most webmasters are fine using the default crawl setting (i.e. no changes needed, more on that below), some webmasters may have more specific needs.
Googlebot determines the range of crawl rate values you'll have available in Webmaster Tools. This is based on our understanding of your server's capabilities. This range may vary from one site to another and across time based on several factors. Setting the crawl rate to a lower-than-default value may affect the coverage and freshness of your site in Google's search results. However, setting it to higher value than the default won't improve your coverage or ranking. If you do set a custom crawl rate, the new rate will be in effect for 90 days after which it resets to Google's recommended value.
Googlebot employs sophisticated algorithms that determine how much to crawl each site it visits. For a vast majority of sites, it's probably best to choose the "Let Google determine my crawl rate" option, which is the default. However, if you're an advanced user or if you're facing bandwidth issues with your server, you can customize your crawl rate to the speed most optimal for your web server(s). The custom crawl rate option allows you to provide Googlebot insight to the maximum number of requests per second and the number of seconds between requests that you feel are best for your environment.
Googlebot determines the range of crawl rate values you'll have available in Webmaster Tools. This is based on our understanding of your server's capabilities. This range may vary from one site to another and across time based on several factors. Setting the crawl rate to a lower-than-default value may affect the coverage and freshness of your site in Google's search results. However, setting it to higher value than the default won't improve your coverage or ranking. If you do set a custom crawl rate, the new rate will be in effect for 90 days after which it resets to Google's recommended value.
You may use this setting only for root level sites and sites not hosted on a large domain like blogspot.com (we have special settings assigned for them). To check the crawl rate setting, sign in to Webmaster Tools and visit the Settings tab. If you have additional questions, visit the Webmaster Help Center to learn more about how Google crawls your site or post your questions in the Webmaster Help Forum.
17 comments:
This is a good idea, but i think the optimum solution would be to let the user choose pages or directories to scan.
i.e. if i have a website with turistic informations about a town or region, it's useful to scan frequently the news and events section of the site, but probably the monuments and museums section will not change very often.
I am still confused how its going to work, need more explain on it.
Anyway what does this mean "Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate."
Is my site having some trouble, or not compatible with googlebot or its a good news for me
It would be nice that google is crawl faster during the night when server loads are low, and during the day to crawl fast only indexes and news.
What a complete waste of time.
Nothing of any value as usual.
Argh...
This is a good idea
Pooja, I am also seeing a special crawl rate option. Please clarify why it is so and what we can do about it. Is there a way to know if the special is slower, medium or faster? Please let me know these details.
The blog I am talking about is : http://cutewriting.blogspot.com
Is the previous 'Faster' setting equivalent to the fastest setting that you can now set?
i can't change the crawl rate even after 90 days. does anyone can help me ?
Why do the crawl rates expire after 3 months? Every three months I have to log back in and update this? Why not make them permanent and perhaps send an email every 6 months to a year asking for a confirmation.
Is there any specific time limit for Google crawl rate. I think when getting quality links to our sites that means from higher page ranks sites in our niche..Google will definitely crawl our site in decent way. But the site structure is also important in crawling. If the web site structure is creating problem for Google from its crawling then Google bot will not visit the web site frequently.
This one is good idea. But i am confused for that which type of crawl rates are very useful to website. So suggest me ...
this blog is very helpful,,, especially for me, I also want to ask how often google crawl a site or blog??? Thanks and more power...
Hello Pooja,
What actually happens is whenever I post a new article on my website. Google takes about 3-4 hours to crawl/index it. Before it was never like this. When ever I used to post something it used to crawl/index immediately (probably in a minute). Now what happens when Google index my page in 3-4 hours! 1st: I normally post technology news, so in 3-4 hours the post becomes useless as everyone has read it already! 2. Huge drop in traffic! 3. I never come on page 1 anymore, however I used to be on page 1 before.
P.S: I've have a PR 4 website. So if someone wants to say that google crawls according to your PageRank then it doesn't make sense to me :S
Hi. I need the explanation about Crawl rate.
on my website settings, there are no option anymore to choose, there are only said "Your site has been assigned special crawl rate settings. You will not be able to change the crawl rate."
I confused about that, is that good or bad for my website.
Thank you
Hi,
Frankly, I think google SHOULD follow the crawl delay directive by robots.txt, but it doesnt! Alas! Because of this, google actually almost DDOSes my websites! I have more than 500 websites, so setting each of them up individually on webmaster tools is a PITA! Any advise would be of great help!
Thanks
Webmaster~
I still can't understand the logic, in one hand you are giving the feature of increasing the crawl rate and on another Google is resetting the changes after 90 days, pretty strange.
Hi everyone,
Since over a year has passed since we published this post, we're closing the comments to help us focus on the work ahead. If you still have a question or comment you'd like to discuss, free to visit and/or post your topic in our Webmaster Central Help Forum.
Thanks and take care,
The Webmaster Central Team
Post a Comment