Google Webmaster Central Blog - Official news on crawling and indexing sites for the Google index

Feeling lucky at PubCon

Tuesday, January 22, 2008 at 11:15 AM

Last month, several of us with Webmaster Central hit the "good times" jackpot at PubCon Vegas 2007. We realize not all of you could join us, so instead of returning home with fuzzy dice for everyone, we've got souvenir conference notes.

Listening to the Q&A, I was pleased to hear the major search engines agreeing on best practices for many webmaster issues. In fact, the presentations in the duplicate content session were mostly, well, duplicate. When I wasn't sitting in on one of the many valuable sessions, I was chatting with webmasters either at the Google booth, or at Google's "Meet the Engineers" event. It was exciting to hear from so many different webmasters, and to help them with Google-related issues. Here are a few things that were on the minds of webmasters, along with our responses:

Site Verification Files and Meta Tags
Several webmasters asked, "Is it necessary to keep the verification meta tag or HTML file in place to remain a verified owner in Webmaster Tools?" The answer is yes, you should keep your verification file or meta tag live to maintain your status as a verified owner. These verification codes are used to control who has access to the owner-specific tools for your site in Webmaster Tools. To ensure that only current owners of a site are verified, we periodically re-check to see if the verification code is in place, and if it is not, you will get unverified for that site. While we're on the topic:

Site Verification Best Practices
  • If you have multiple people working on your site with Webmaster Tools, it's a good idea to have each person verify the site with his or her own account, rather than using a shared login. That way, as people come and go, you can control the access appropriately by adding or removing verification files or meta tags for each account.
  • You may want to keep a list of these verification codes and which owner they are connected to, so you can easily control access later. If you lose track, you can always use the "Manage site verification" option in Webmaster Tools, which allows you to force all site owners to reverify their accounts.
Subdomains vs. Subdirectories
What's the difference between using subdomains and subdirectories? When it comes to Google, there aren't major differences between the two, so when you're making that decision, do what works for you and your visitors. Following PubCon, our very own Matt Cutts outlined many of the key issues in a post on his personal blog. In addition to those considerations, if you use Webmaster Tools (which we hope you do!), keep in mind that you'll automatically be verified for deeper subdirectories of any sites you've verified, but subdomains need to be verified separately.

Underscores vs. Dashes
Webmasters asked about the difference between how Google interprets underscores and dashes in URLs. In general, we break words on punctuation, so if you use punctuation as separators, you're providing Google a useful signal for parsing your URLs. Currently, dashes in URLs are consistently treated as separators while underscores are not. Keep in mind our technology is constantly improving, so this distinction between underscores and dashes may decrease over time. Even without punctuation, there's a good chance we'll be able to figure out that bigleopard.html is about a "big leopard" and not a "bigle opard." While using separators is a good practice, it's likely unnecessary to place a high priority on changing your existing URLs just to convert underscores to dashes.

Keywords in URLs
We were also asked if it is useful to have relevant keywords in URLs. It's always a good idea to be descriptive across your site, with titles, ALT attributes, and yes, even URLs, as they can be useful signals for users and search engines. This can be especially true with image files, which otherwise may not have any text for a search engine to consider. Imagine you've taken a picture of your cat asleep on the sofa. Your digital camera will likely name it something like IMG_2937.jpg. Not exactly the most descriptive name. So unless your cat really looks like an IMG_2937, consider changing the filename to something more relevant, like adorable-kitten.jpg. And, if you have a post about your favorite cat names, it's much easier to guess that a URL ending in my-favorite-cat-names would be the relevant page, rather than a URL ending in postid=8652. For more information regarding issues with how Google understands your content, check out our new content analysis feature in Webmaster Tools, as well as our post on the URL suggestions feature of the new Google Toolbar.

Moving to a new IP address
We got a question about changing a site's IP address, and provided a few steps you can take as a webmaster to make sure things go smoothly. Here's what you can do:
  1. Change the TTL (Time To Live) value of your DNS configuration to something short, like five minutes (300 seconds). This will tell web browsers to re-check the IP address for your site every five minutes.
  2. Copy your content to the new hosting environment, and make sure it is live on the new IP address.
  3. Change your DNS settings so your hostname points to the new IP address.
  4. Check your logs to see when Googlebot starts crawling your site on the new IP address. To make sure it's really Googlebot who's visiting, you can verify Googlebot by following these instructions. You can then log into Webmaster Tools and monitor any crawl errors. Once Googlebot is happily crawling on the new IP address, you should be all set as far as Google is concerned.
  5. To make sure everyone got the message of your move, you may want to keep an eye out for visits to your old IP address before shutting it down.
Proxies
A few webmasters were concerned that proxy services are being indexed with copies of their content. While it's often possible to find duplicate copies of your content in our results if you look hard enough, the original source is most likely going to be ranked higher than a proxy copy. However, if you find this not to be the case, please drop us some URLs in the Webmaster Help Group. There are many Googlers including myself who monitor this group and escalate issues appropriately.

It was great talking with webmasters at the conference -- we hope those of you unable to join us found this post useful. If you want to continue to talk shop with me, other Googlers, and your fellow webmasters, join the follow-up conversation in the Webmaster Help Group.

Update: Additional PubCon notes from Jonathan Simon are available in our discussion group.
The comments you read here belong only to the person who posted them. We do, however, reserve the right to remove off-topic comments.

37 comments:

JLH said...

Thank you, Wysz.

Phoenix Arizona Auto, Car, Home Owner Insurance Quote said...

I have been trying hard to get Google to index the following site. I have asked my webmaster to see if there is something he did that is preventing the process from happening. I know nothing about programming so at a loss.
Anyone else have an idea what would stop Google from indexing a site. I've tried to make it a site with some content, as you can see.

http://www.phoenix-life-insurance.com

http://search-engines-web.com/ said...

Did anyone ask why Google does NOT recognize the META KEYWORD tag?

If not, could you please expand.

Also - is Concept Search on the horizon - how long until it will be incorporated into the organic SERPs?.

Next time, please make a Video or MP3 of the conference to share with the world.

ddwebdesign said...

Thanks Michael, it's nice information to us.
Deb

bestaffiliate said...

My name is robin. I am new in internet word. I have a site that try to be indexed with google and other search engine. If you are webmaster, please help me. You can email me at robinlaki_laki@yahoo.co.id

My site is www.1st-lifeinsurance.net

incrediblehelp said...

A little off topic but why cant we export the content analysis results to a .csv file? VERY frustrating.

Tanya V said...

When will Google offer bulk verification of subdomains? It would be to Google's benefit as enterprise sites are not able to communicate with Google via webmaster tools and aren't those the sites that are most difficult for Google to efficiently crawl and index?

Michael Martinez said...

When will Google return to promoting the most relevant results to the top of search results?

I'm getting tired of clicking through to deeper search results pages.

Staten Island Real Estate Agent said...

I'm not sure if i have a problem, I have a web design company whose hosting my site. I puchased my own domian name changed the dns server which is realestatesiny.com and pointed to the hosting company server. The problem I believe is the hosting has alicciardello.topproducerwebsite.com live, which is exactly the same as realestatesiny.com. it appears both sites are being indexed, is it considerd a mirror site that will be penalized. Thanks for any insight you can give.

Get Paid For EVERY Blog Visitor - Ask Me How... said...

Thanks for the updates!

Chris Blanc said...

Great post. I like how the "search engine game" has brought some fun into web design.

Should an image Alt tag include the descriptive terms used in the filename?

Oscar said...

Hello, nice post.

I use google webmasters and "yahoo webmaster", with yahoo is more easy, fast to remove old urls and directories. With Google its very dificult, and some urls need to have the 404 error.

There are an easy way? Why google make this so hard.

Thanks,

TOR Hershman said...

Howdy do, I wrote the words for the song parody "What A Friend We Have In Google."

Stay on groovin' safari,
Tor

PM said...

Have google tried to create their own audio advertising yet?
http://sellingppp.com/a.cgi?ppp=1210551888
Adsense doesn't have audio adds like this yet does it?

John said...

See Google supplemental results are a great headache for me, i tried several proven methods of interlinking with content and played with robots. txt still no use, i want to know the exact logic behind supplemental results. I know the factors but the exact algo of google how it makes a page as supplemental.

My site http://about.infocrystals.com has been affected severely by supplemental results.

Susan Moskwa said...

Hi John:

While you probably won't get "the exact algo of Google" any time soon, this article on supplemental results may be of interest.

Bijay Rungta said...

Subdomains vs. Subdirectories

Underscores vs. Dashes

Two isues I was skeptic about for long..
And was going to ask it in teh Google Webmaster Group.

Talking about the Underscores and dashes, How does Google Treats urls in Wiki and CamelCase Combination Style as in all Help files in Ubuntu for example.
https://help.ubuntu.com/community/ApacheMySQLPHP

I was Working on a Project and had adopted a Strategy to have urls with underscores and words starting with Capital letters..
As it was still in dev mode I converted the Underscores to dashes.. and Also changed the Cases to all small...
The only reason for me to choose the Capital initial letter was its readability but it makes difficult to type..
I think the Cases will have little or no effect in the Indexing Context.. am I right??

What do you suggest for the cases of the Words??

Susan Moskwa said...

Generally you're right that case is not a big factor. You might want to take into consideration whether your server is case-sensitive or not, though (if it is, it's more important to make sure that any links to your site use the correct case).

Greynium said...

Google should provide a way to share the Sitemap accounts exactly on the same lines of sharing Google Analytics.

You folks have cracked GA sharing so well, why is that feature not being included for Webmaster Tools :-(

Alberta said...

Webmaster Tools shows www.humboldt.info was indexed 2007yet humboldt.info has been indexed in the last month (new pages included) - it is only one site.
I have set preferred Domain to humboldt.info - so why doesn't the the Tools show the last Crawl; why does Google consider the site as 2 sites?

Adnan said...

Hi Google.

According to my webmaster tools my site Linkspub.com should come up on our title searches, but stopped doing so about a few weeks ago, nowhere to be found.

On Yahoo, MsN and Baidu it comes up just like it used to.

I don't see any negative messages or anything on my webmaster tools and it says that my site is indexed and gives me all those terms for which it should pull up.

I tried to get someone from Google at Google Groups to help me on why all of a sudden the change, but no luck.

I've always followed all search engines rules and policies whether I like it or not, cause I understand it's a relationship which works both ways.

The site is around 2.5 years old, and from the very beginning I've submitted my content to Google Sitemaps and Google Analytics.

I have no idea if my site was handpicked and if so, what I did wrong and how to fix it and get it so the homepage comes up for our main term, like it used to.


Thank You in Advance
Adnan

noertz said...
This comment has been removed by the author.
SmarTy said...

Hi!
I have a problem with Sitemaps, and it seems there is no one to tell about it :).
Hope you can help.

I receive a sitemap error. The sitemaps are the same that crawled fine
until a couple of days ago when I first got the message.

The web-site is http://smartssex.com/
The sitemap I add is an index, UTF-8 encoded:
http://smartssex.com/sitemap.xml

I have checked that Google can access the site (at least verification
processes and robots.txt analysis work fine), but the google bot has
never accessed sitemaps for the past couple of days.
But it tells me it did, and encountered an error :).

The server was up for last 24 hours, since my team is here, and 5-10
re-submissions of a sitemap today all resulted in "Network
unreachable: Network unreachable".

Thanks in advance!

rena said...

OK! I'm going to cry if I can't even get a PR 1 sometime soon! I have a site that I've been working on myself (because every computer person I've ever hired has flaked out on me). I really want to build a sitemap (anything to help) but am completely lost. Anyone interested in being somewhat of a mentor for a sweet girl in Los Angeles (who will one day soon be wealthy and want to hire someone full time to do all my web stuff and SEO stuff). I have an aircraft charter company and a Non-Profit organization that raises money for air ambulance flights for people who cannot afford them :)

Rena@ExquisiteAirCharter.com

Vanessa Alexander said...

Speaking of meta tags from Webmaster tools. My blog was recently indexed but now Google keeps asking me to put the tag in. Its already there. I've replaced it again but I go back and am asked to verify again.

This began to happen after I registered for Google Analytics. So I removed the analytics code.Its still un-verifying me.

Another theory is that I sign into blogger.com with a non-Google account and after having to migrate to a google account recently I still log in to Blogger with my old account.

Yes I have the new blogger. Could the two different accounts be the problem?

If it is can you tell me simply how to migrate my blogs without a hassle of losing them or going round and round the mulberry bush with Google about sign in. Been there done it.

All my other accounts are Google but the blogger account.

Thanks...

Bijay Rungta said...

Keywords in URLs
Regarding Images.
Does the Search Engine also consider the path to the folder containing the image.
For example I want the images of our Products to be in the following format
http://example.com/images/..
..products/product-type/product-subtype/..
..product-category/product-code.jpg

Will that help me improve my indexing??

Thanks,
Bijay Rungta

Bijay Rungta said...
This comment has been removed by the author.
Bijay Rungta said...

Are white spaces ok in Urls???
For example..
http://www.templatekingdom.com/Template/Templates/tag/match%20making%20website%20templates/

Also is there any ways I can tell GoogleBot that my urls are case insensitive so
http://www.templatekingdom.com/Template/Templates/tag/match%20making%20website%20templates/

http://www.templatekingdom.com/template/templates/

are interpreted as a single Page???

Thanks a lot..
I will really appreciate if you could clarify my doubt.

bestaffiliate said...

Can someone tell me, how to optimize my site?

I am really new in internet, I have one site that I want to optimize, please help me.


robin
www.datinglovely.com

Mountain Bike and RPG Fan said...

Thanks for the clarification on the underscores/dashes issue. I often think that URLs with no punctuation are more memorable to users, ie. not having to describe exactly what a dash is every time when you tell them your URL, so it's good to know that google still picks up keywords in my URL even though I forgo dashes for my user's sake.

Cheers,

Colin
Mountain Bikes Apart

rena said...

I have a question - someone told me yesterday that Google only updates your backlinks once every two to three months and that they only update your page rank twice per year - can anyone expand on this?

http://www.exquisiteaircharter.com
rena@exquisiteaircharter.com

Owen said...

I was interested in your webcast a few days ago (the Google Trifecta) but gave up in irritation.
1. The prerequisites say Windows Media Player OR Real. However, when you come to view it, it requires BOTH. I refuse to install Real because of problems I've had in the past (and I think that would apply to others as well).
2. No real contact address on anything to complain about things like this (which is why I'm writing here).

Could you please direct this winge to the right person ASAP?

Ta!

kabonfootprint said...

It would be to Google's benefit as enterprise sites are not able to communicate with Google via webmaster tools. thanks

kabonfootprint | busby seo challenge

SEOHMH said...

today i change the domain sobookee.com to ufindbook.com

bestaffiliate said...

Can SEO become very eazy?

info said...

Can someone confirm whether Google ranking, search results, and indexing is affected when a site DNS are changed -
that is moved to a new host server? domain name remain the same including all contents.

Maile Ohye said...

Hi everyone,

Since some time has passed since we published this post, we're closing the comments to help us focus on the work ahead. If you still have a question or comment you'd like to discuss, free to visit and/or post your topic in our Webmaster Help Forum.

Thanks and take care,
The Webmaster Central Team