Tuesday, March 06, 2007 at 9:30 AM
Search engine robots, including our very own Googlebot, are incredibly polite. They work hard to respect your every wish regarding what pages they should and should not crawl. How can they tell the difference? You have to tell them, and you have to speak their language, which is an industry standard called the Robots Exclusion Protocol.
Dan Crow has written about this on the Google Blog recently, including an introduction to setting up your own rules for robots and a description of some of the more advanced options. His first two posts in the series are:
Controlling how search engines access and index your website
The Robots Exclusion Protocol
Stay tuned for the next installment.
While we're on the topic, I'd also like to point you to the robots section of our help center and our earlier posts on this topic:
Debugging Blocked URLs
All About Googlebot
Using a robots.txt File
Update: For more information, please see our robots.txt documentation.
Dan Crow has written about this on the Google Blog recently, including an introduction to setting up your own rules for robots and a description of some of the more advanced options. His first two posts in the series are:
Controlling how search engines access and index your website
The Robots Exclusion Protocol
Stay tuned for the next installment.
While we're on the topic, I'd also like to point you to the robots section of our help center and our earlier posts on this topic:
Debugging Blocked URLs
All About Googlebot
Using a robots.txt File
Update: For more information, please see our robots.txt documentation.
10 comments:
Howdy Folks...need to pick your brains please...
1. does site map = navigational buttons
2. when google says " try to use text to display important names, contents or links" does the pic that has a link to another page interfere with the text box right below it since I have both the visual and the readable to give my customers both. In other words do I need to loose the image or is it harmles as long as it is in a different text box that can be read by the spiders?
3. How do you use a text browser like Lynx to examine your site?
4. What are "session IDs or arguments that track their path through the site" and how do I know if I have them?
5. How do I know if my webserver supports "If-modified-since HTTP header"?
Thank you for any and all input you can provide for me...Have a Great Day!!!
I've a question on Googlebot's capability to run Javascript.
Actually, I have a problem with some of my webpages which should not happen unless Googlebot can actually run JavaScripts.
I saw a few forum posts that suggest, URLs can be extracted from the JS code. But the problem with my webpage shows that Googlebot can actually execute the javascript code.
Can you please clarify?
Thanks,
Ram
Suddenly Google can't find my website. After coming up as number one on the search page for two years now, this is a real comedown. Two years of work down the drain.
The last time Google found my homepage was 3/1/07 - the day before Blogger forced me to switch to the new improved program. Irony of irony, I hear Google owns Blogger now.
Instead of my blog - which is updated every single day - coming up first, my static website - which is never updated and gets no hits - comes up first. Blogger was better before Google bought it.
Howdy!
How does Googlebot handle a robots.txt file on a FTP site? There is a post telling Googlebot doesn't obey robots.txt when fetching via ftp:
http://groups.google.com/group/Google_Webmaster_Help-Indexing/browse_thread/thread/c05bc8336babcfb3/#
TIA
Sebastian
I HAVE A PROBLEM...
I had " blogspot " since 2006. I was post 511 messages.
But, my blog (http://alexandrecabreira.blogspot.com) become very, very slowly when charged...
Somebody could I help me?
(Sorry my ba english - I write from Brasil)
: - )
is it possible to add mirrors for the same website ? i have an other mirror for my blogspot blog and i want google understeand this as it helps my page ranking (some sites link to my mirror and google does not get it)!
Why can't the googlebot check to see if a site is build with all JavaScript, and if so index the content in the noscript tag without penalty. It is easy to see if it is just keyword stuffing vs if it is legitimate markup?
Updated robots.txt example for blogs..
I have a question as to why it googlebots wont come by our site. It's been almost 3 months. I have been submitting url add requests every couple weeks, added adsense, added a google sitemap and nothing! We want traffic!
http://www.atmospheretv.com
Thanks for listening to my ranting!
Hi everyone,
Since over a year has passed since we published this post, we're closing the comments to help us focus on the work ahead. If you still have a question or comment you'd like to discuss, free to visit and/or post your topic in our Webmaster Help Group.
Thanks and take care,
The Webmaster Central Team
Post a Comment