Google Webmaster Central Blog - Official news on crawling and indexing sites for the Google index

New: Content analysis and Sitemap details, plus more languages

Thursday, December 13, 2007 at 12:48 PM



We're always striving to help webmasters build outstanding websites, and in our latest release we have two new features: Content analysis and Sitemap details. We hope these features help you to build a site you could compare to a fine wine -- getting better and better over time.

Content analysis

To help you improve the quality of your site, our new content analysis feature should be a helpful addition to the crawl error diagnostics already provided in Webmaster Tools. Content analysis contains feedback about issues that may impact the user experience or that may make it difficult for Google to crawl and index pages on your site. By reviewing the areas we've highlighted, you can help eliminate potential issues that could affect your site's ability to be crawled and indexed. This results in better indexing of your site by Google and other search engines.

The Content analysis summary page within the Diagnostics section of Webmaster Tools features three main categories. Click on a particular issue type for more details:

  • Title tag issues
  • Meta description issues
  • Non-indexable content issues

content analysis usability section

Selecting "Duplicate title tags" displays a list of repeated page titles along with a count of how many pages contain that title. We currently present up to thirty duplicated page titles on the details page. If the duplicate title issues shown are corrected, we'll update the list to reflect any other pages that share duplicate titles the next time your website is crawled.

Also, in the Title tag issues category, we show "Long title tags" and "Short title tags." For these issue types we will identify title tags that are way too short (for example "IT" isn't generally a good title tag) or way too long (title tag was never intended to mean <insert epic novel here>). A similar algorithm identifies potentially problematic meta description tags. While these pointers won't directly help you rank better (i.e. pages with <title> length x aren't moved to the top of the search results), they may help your site display better titles and snippets in search results, and this can increase visitor traffic.

In the "Non-indexable content issues," we give you a heads-up of areas that aren't as friendly to our more text-based crawler. And be sure to check out our posts on Flash and images to learn how to make these items more search-engine friendly.


content analysis crawlability section


Sitemap details page

If you've submitted a Sitemap, you'll be happy when you see the additional information in Webmaster Tools revealing how your Sitemap was processed. You can find this information on the newly available Sitemap Details page which (along with information that was previously provided for each of your Sitemaps) shows you the number of the pages from your Sitemap that were indexed. Keep in mind the number of pages indexed from your Sitemap may not be 100% accurate because the indexed number is updated periodically, but it's more accurate than running a "site:example.com" query on Google.

The new Sitemap Details page also lists any errors or warnings that were encountered when specific pages from your Sitemap were crawled. So the time you might have previously spent on crafting custom Google queries to determine how many pages from your Sitemap were indexed, can now be spent on improving your site. If your site is already the crème de la crème, you might prefer to spend the extra free time mastering your ice-carving skills or blending the perfect eggnog.

Here's a view of the new Sitemap details page:


Sitemaps are an excellent way to tell Google about your site's most important pages, especially if you have new or updated content that we may not know about. If you haven't yet submitted a Sitemap or have questions about the process, visit our Webmaster Help Center to learn more.

Webmaster Tools now available in Czech & Hungarian

We love expanding our product to help more people and in their language of choice. We recently put in effort to expand the number of Webmaster Tools available languages to Czech and Hungarian, in addition to the 20 other languages we already support. We won't be stopping here. Our desire to support even more languages in the future means that if your language of choice isn't currently supported, stay tuned -- there'll be even more supported languages to come.

We always love to hear what you think. Please visit our Webmaster Help Group to share comments or ask questions.
The comments you read here belong only to the person who posted them. We do, however, reserve the right to remove off-topic comments.

64 comments:

Greg said...

Nice additions. Now if I could only find a page that tells me if I have been penalized or if my redirect error this weekend caused my complete fall from the SERPS. :)

Jennifer Mathews Somogyi said...

I have been a big fan of the webmaster tools from day one and this is yet another reason why webmasters should start utilizing what's available to them to help streamline their natural SEO.

I have been mocked for the past few years for still placing such an emphasis on the title, keyword, and description meta tags and it's nice to see Google supporting us "old school" SEO's with this great new feature.

There is an article posted about the meta tags on the semdiscussion.com website titled "Meta Tags or No Meta Tags?" back in December of 2006.

vanhantro said...
This comment has been removed by the author.
oroceo.ian said...

love the new tool but i have a question. is the content analysis tool only anlayze the index page of the site or the "WHOLE" pages of the site?

NadirG said...

Excellent, that's probably the best additions in GWC that I've ever seen. It certainly helps webmasters who don't have to develop scripts anymore to check that data.

EDI-L said...

Excellent addition, I made several changes to my titles and descriptions already. I would like to know more about why some pages are not indexed. In the example.com (umm) example, they have 2871 Total URLs and 2181 Indexed URLs. But the Content Analysis dosn't describe why 690 pages are not indexed.

Genevieve said...

The Meta Tag information is helpful, but will this feature be developed further to allow the user to see which links have duplicate/long/short meta tags? Right now, it just shows me the number of bad tags, but not which pages they belong to.

Susan Moskwa said...

Jennifer, thanks for all your positive comments!

oroceo.ian, this tool analyzes pages from all over your site, not just the homepage.

Edi-L, we address your question in our Help Center.

Genevieve, you should be able to click on each issue type (e.g. "Duplicate meta descriptions") to see details about that issue, and then click on the details to see the individual URLs on which we found that issue.

Rajesh Kumar said...

Really its a very helpful for especially for the dynamic site that difficult to crawl.

jevstar said...

Firstly I think the new additions are great and thank you for them. I have around 600,000 pages from which a few 404 errors and one 403 have been listed, however these pages are not in my site! It would be of great use to know what the referring page(s) were that caused the 403/404, or any other similar error for that matter so that they could be attended to.

I have used other tools like Xenu to attempt to locate the issue but they do not report the fault that site map does. I fear that it may be caused by an external reference beyond my control, but I don't know. If we could see where the reference came from it would be of great use.'

Bill said...

Please can you tell us, how long is too long, and how short is too short for the meata Title and Description.

Genevieve said...

Thanks Susan - I didn't realize I could click on the tag and it would drop-down the pages.

Dan said...

This is a great idea. However, I don't think that the Title tag detection is working 100% properly. It claims that this page does not have a title:

https://www.contractorsav.com/Default.aspx

Yet, looking in the source, it clearly does (and has for over a year).

Any ideas why?

Jonathan Simon said...

Hi dan,

The page you mention doesn't have a title tag. It contains a Javascript redirect pointing to: "http://www.contractorsav.com/Default.aspx"
which does contain a title tag.

If possible you should probably update the page:
"https://www.contractorsav.com/Default.aspx"
to instead use a server side 301 redirect. This is the best way to ensure that Google is properly directed to the page you want evaluated for indexing or in this case diagnosed for a missing title tag.

If you have a follow-up question or other questions the best place for answers is the Webmaster Help Group

support said...

Content analysis might be broken.
Hello Google,

I am reporting this possible bug in Webmaster Tools(Content analysis) here only because there appears to be no other way of contacting Google.

The Section that deals with "duplicate meta descriptions" is quick to find offending pages, but appears to tag these pages so they drop-out of the search results.

We found that correcting the duplicate meta descriptions does not remove the tag(s).

Our site provides Real Time Weather data and has was tagged 12-14-2007 for Duplicate Content or banned by this new feature, but when the pages are corrected, I would think the tag would be removed?

Regards,
Larry

Susan Moskwa said...

Hi Larry,

The data that we report to you in the content analysis tool may or may not have an effect on your site's search performance, but the actual tool doesn't "ban sites" or change our search results. That is to say, any change in your site's performance is not due to the launch of this tool.

If you've updated some of your meta descriptions, you'll need to wait awhile to see the changes reflected in your Webmaster Tools account because we have to a) recrawl your site (to detect the changes) and b) refresh the data shown in your account.

In the future, you can report potential issues or questions you have in our Webmaster Help Group.

baurum said...

Idea: Index of attendance. The site from 1 page is visited by 100 person and the site from 1000 pages is visited 100 person. The index of attendance is deduced by simple mathematics
- Excuse for bad English, can not in that blogs I write :(

Tara said...

Could someone offer some suggestions please:

My site is not coming up in the Google Search results when I type in the URL or keywords. Note: we switched from a static html site to a php site, domain stayed the same but we moved to a different hosting company. Before the switch, the site was number one and showing up in the results by keyword entry or url entry.

Another odd thing is that in webmaster tools, I can see the URLs from the site that Googlebot had trouble crawling. One of the sections is called “URLs not followed” and it lists the actual domain and a bunch of the pages with a detail of: “Redirect Error”.

Many of the urls in the site follow this format: /public/t/?f=1&s=101&t=201 Could that be causing trouble? But why wouldn't the index page work?

I have set up htaccess 301 redirects as well for the old html pages.

Any help would be so appreciated!

Thanks,
Tara

Susan Moskwa said...

Tara:
As stated several times above, the best place for troubleshooting and site-specific questions is our Webmaster Help Group. Please repost your question there.

ram said...

in my webmaster central the content analysis page is empty not showing any thing.
www.wwwportal.blogspot.com

Susan Moskwa said...

Ram, that's fine; it just means we haven't detected any content issues on your site.

info said...

Something drastic has now changed in your spidering. In our product blog, the title of most of the 150+ SERP entries is identical and so is the description! Only the link is different, naturally. There is no duplicate content on the site. I wish I knew how we could help you to correct this.

Godfather said...

The entire articles and suggestions were very helpful to me. However, one small point where I could not get the help is that after the contents in Meta tags the short Meta descriptions were changed as suggested by Google, the content analsys still shows the same error despite resubmitting the contents after correction. It is almost ten days I made these fresh corrections and submitted.

A guidance on how to resubmit the contents after making the needed corrections would be of most welcome one for new entrant like me.

TRS Iyengar
webmaster
www.trsiyengar.com

incrediblehelp said...

Any reason why you can download all of the content analysis results with one click? Makes it really hard when you have to download one by one by hand.

support77 said...

Susan,

You told us to post help questions in the groups, but no one responds?

Our site: http://broadcast-weather.net/win/ has six (6) pages that were flagged as having duplicate meta tag descriptions on or about 12/14/2007. We fixed the problems around 12/17/2007, but webmaster tools still shows the same six(6) errors today?

Google has crawled and indexed all six (6) of the pages, many times since we corrected it, but it still shows the errors?

From the outside, it looks as thou Content analysis (duplicate meta description is broken).

Could you please see if this is the case?

Best Regards,
Larry

Susan Martin said...

I had uploaded a new page to my site, and had second thoughts about the url (first I had it stem from the top level, then decided to stem it from a 2nd level) Now, google has this page indexed from the top level, but I've moved the page. Will google catch up with this?

incrediblehelp said...

yeah Google Groups is worthless to. I have posted various questions on many different products in there and never get quality or response sin general. I wish someone from Google would be helping out here.

Colin said...

My blog (wordpress) has been live for 3mths. The index page(s) as of today have been tagged as "Duplicate title tags" and "Duplicate meta descriptions" as of today. Previously no error.

The index page has excerpts of each post and I list 8 per page. I use wp-pagenavi to navigate from page to page. There are a total of 6 pages and pages 2-6 have been listed in WebMaster Tools as having the above errors.

Could somebody please advise what I should do to correct it, or is it just a hiccup in WTools?
Thanks in advance

Oleksiy said...

How to can I know a location to access from a command line to upload a sitemap?
It is very difficult explanation for me...
Thanks in advance Oleksiy

Partisimon Partogi said...

Dear Sir,

I find this blog, whan I'am confuse to follow one guide.

According to informatioan I read on a blog, he said, one of the key is personalization or localization your blog, that Matt Cutts said.

Because I make a blog and my content in indonesia language and I hope the reader from indonesia mostly.

In that blog I read, there guide to login to my google.com/webmaster , after that, in the menu 'tool', you must set "set geographic target" and choose "Associate a geographic location with this site" and then "save"

but After I login to My webmater tool, I did not find it, I only find : 'download data for all site', 'report spam in our index', report paid links and request the reconsideration.

please help me to find the guide in my webmaster tool, I mean, I want to optimize my blog for indonesia reader on search engine google.co.id

thanks
Partisimon Partogi

Susan Moskwa said...

Hi Partisimon Partogi:
Glad you're using our tools. To use the geographic targeting tool, you have to add your site to your Webmaster Tools account and verify it. Then you should be able to access the geographic targeting tool; see this blog post for more details.

incrediblehelp said...

Susan what about an export option for content analysis?

abhilash said...

I have my sitemap submitted for my website after changing the links but dont have all the links catched by google.Can anyone explain why only certain links are being scanned and why the rest are not? And how much time does google need to scan each ..

iliyas said...

I have more than 35 sites in my account in google web master tool. I want to share this webmaster tool wiht another gmail account, anybody know about this that how can i share it.

Help soon if anybody have suggestion.
thanks

Susan Moskwa said...

Hi iliyas,
You'll find the answer in our Help Group. In order for another person to see the data for a site, they have to verify that site in their own Webmaster Tools account.

Waiting for better... said...

Hello, I had a site drop off of Google and now when I try to VERIFY to attempt to fix the problem I get a 403 error. I tried the meta tag and html page on my index and still nothing. The site does have a certificate/https

I am at a loss.

Please help.

DS

IndianPie said...

nice info !!

Berk said...

As many people asked in webmaster tools sitemap statistics and index:www.xxxxxx.com is different. İndex url is very low according to total urls.
Like
in

http://www.buybestanabolicsteroids.com

Sitemap statistics:
Total URLs: 141
Indexed URLs: 36

In:

http://www.allaboutanabolicsteroids.com

Sitemap statistics:
Total URLs: 111
Indexed URLs: 46

Why does this difference occur?

Susan Moskwa said...

Hi Berk,
The Sitemap statistics only deal with URLs that are in your Sitemaps, whereas a site: query shows URLs from your site that are indexed, regardless of whether they're in your Sitemaps. If the number of site: query results is greater than the number of indexed URLs from your Sitemaps, it's because some of your indexed URLs aren't included in your Sitemaps.

Acumed said...

I have a dynamic site that is based on a shopping cart program. I noticed that last week Google Webmaster flagged me for alot of duplicate page titles and metatag descriptions. None of these pages has products for sale. I dropped page position from 5 to 13 (a big drop). How bad is this hurting me (I do not have access to the titles and meta description on those pages and should I exclude those pages from being indexed.

Thanks,
Scott
www.acumedsupplies.com

Arhan Efha said...

Wow, very useful! I hope Bahasa Indonesia can be add to Google Webmaster Tool. :)

PetCollection.com said...

once I fix the error in content analysis, how long does it take for google to notice it?

Google is used to crawl my site every week, but now it's been more than two weeks since the last time googlebot crawled.

Any idea?

Michele's Paint Shop Newbie 101 said...

Google Webmaster Analytics of Duplicate Meta Descriptions even after being fixed a month ago.

With regards to the comment on January 17, 2008 8:04 PM by the member Support77

Results state some of my sample pages had "duplicate meta descriptions" which yes they did and I fixed them.

I see that Google has crawled those pages again, but yet the pages are still listed as having duplicate meta descriptions. Their last analysis date(s)are weeks after my "fix"

The updated descriptions are as different as night and day. So what do I do now? Wait it out?

Should I worry that they are STILL telling me I have nine duplicate meta descriptions even after they have been fixed? I've been waiting four weeks and during that time the update date has changed several times.

Any Advice?

Krunal Jariwala said...

Hi,

this is a great tool, I had been using it since 3 years, but just to send sitemap...

I've found errors on all my sites which I'll correct.. as well as keyword tools is the most useful to find performing keywords...

Kmpfurniture said...

Please Help.

I got on the report that this Url http://www.kmpfurniture.com/earth_collection/products/night_table_51.html is missing the Title Tag but It has its Title Tag.

Can you explaing why?.

hc said...

check out my site at lonestarteachers.blogspot.com

Jagadeesh M said...

I love this tool, mainly crawl section :)

Thanks,
Jag
SEO Pro

Gis User said...

Different pages on my website are crawled at different intervals.

And old content is there in my webmaster results.

If my website has frequesnt updations, how can google track this any special tools or procedures to it.

The content analysis doesnt update due to this.

Answers ?

Al said...

Webmaster tools are useful... as far as they go. We just restructured the links into our ecommerce site. The new sitemap was crawled on 10/29. Results show 203 links found with no errors and 0 links indexed. All the other data in the tools is for old links that were removed and now return 404 codes. It would be considerably more helpful if the sitemaps had more status information about why some or all pages have not been indexed, and I don't mean detailed analysis. Just are they waiting for Google to get a round tuit, or have they been rejected (if so, why would be nice!).

JerrySchrader said...

I think you have done a great job covering the basics and not so basics of SEO and set up in general.

seoforward said...

cool man.you have done great job...

hey please support me at Busby seo test

Kmpfurniture said...

When I start with KMPFurniture.com, it had about 300 pages missing tag, 700 duplicate title and description but now it is 0.

Webmaster tool is great for SEO

seoforward said...

its great for seo realy....Busby Seo Test

seoforward said...

cool man


Busby Seo test

DebNCgal said...

Does anyone have an idea why Webmaster Tools is showing a duplicate title and description tags for the / and /index.php pages for my blog? Those are the only two pages that are problematic.

I've racked my brain trying to figure out what's causing the duplication error. I'm using WordPress and the All in One SEO plugin. I've made modifications to both the robots.txt file and the .htaccess file. But it's not clear to me if either of these files is causing the duplication error, or if the problem is something else.

Any good guesses that can lead me in a direction to investigate? Thanks.

Dustin said...

I see someone asking how to share their authenticated sites with other Google user accounts - I as well want to do this. Having that other users re-authenticate directly with the site is unacceptable since I don't have direct control over the site itself (I have to wait for the webmaster), hence I'm not directly controlling the ACL for this data, which needs to be the case.

google said...

I have a problem with how Google is interpreting my robots.txt file. The "Analyze Robots.txt" function in webmaster tools says that my file is formatted the way I want, but Google is still excluding many of the URLs that I'm explicitly allowing. Google doesn't allow people to contact them, so I'm stuck posting here.

Anybody have an idea or know how to contact Google?

MBE - Delhi University said...

Thank you Team for such a nice support . I am new to this word of SEO and find this article quite helpful.

Naveen Verma
www.insiderthings.com

timon said...

With the meta tag duplicate issue that can effect site pages or posts in the index (supplementary results for those in error) is there a way to let the google bot know these have been fixed. In webmaster central it keeps saying they are an issue when they are fixed..?

It is for my site occultblogger.com
Any feedback would be appreciated.

Kind Regards
Timon

KOESnadi said...

I'm already use GWS, i hope will make my blog more popular..http:innakoe.blogspot.com

Paul said...

I have a lot of work moving my company web site to new domain furnituredepot.com because of re-branding issues. Without webmaster tools the chance to screw everything is pretty big. Now it is absolutely first step to add web site to webmaster tools dashboard. I will feel myself blind without such useful tool.

Niksss..... said...

Excellent, It certainly helps webmasters who don't have to develop scripts anymore to check that data.

thanks a lot

Maen Mola said...

Good day all,
i have this "Duplicate tittle" warning for two pages on my website which they were detected late march 2009, i was happy that this amazing tool notified me to fix this thing and that what i did, currently in those pages i programatically specify my tittle tags and it is still the same warning in content analysis page!!!
my web pages are :
http://www.rugbyprofiler.com/RPDirectory/DirectoryDetails.aspx?dir=40
http://www.rugbyprofiler.com/RPDirectory/DirectoryDetails.aspx?dir=44

would appreciate your tips to fix this issue.

Many Thanks,

a2purn.com said...

its cools...
but my site have low PR in search engine always go to front pages.
How to solve?