Google Webmaster Central Blog - Official news on crawling and indexing sites for the Google index

Making more pages load instantly

Thursday, May 17, 2012 at 3:00 PM

Webmaster level: All


At Google we're obsessed with speed. We've long known that even seemingly minor speed increases can have surprisingly large impacts on user engagement and happiness. About a year ago we rolled out Instant Pages in pursuit of that goal. Instant Pages makes use of prerendering technology in Chrome to make your site appear to load instantly in some cases, with no need for any extra work on your part. Here's a video of it in action:



We've been closely watching performance and listening to webmaster feedback. Since Instant Pages rolled out we've saved more than a thousand years of ours users' time. We're very happy with the results so far, and we'll be gradually increasing how often we trigger the feature.

In the vast majority of cases, webmasters don't have to do anything for their sites to work correctly with prerendering. As we mentioned in our initial announcement of Instant Pages, search traffic will be measured in Webmaster Tools just like before this feature: only results the user visits will be counted. If your site keeps track of pageviews on its own, you might be interested in the Page Visibility API, which allows you to detect when prerendering is occurring and factor those out of your statistics. If you use an ads or analytics package, check with them to see if their solution is already prerender-aware; if it is, in many cases you won't need to make any changes at all. If you're interested in triggering Chrome's prerendering within your own site, see the Prerendering in Chrome article.

Instant Pages means that users arrive at your site happier and more engaged, which is great for everyone.


Sorting and Filtering Results in Custom Search

Tuesday, May 08, 2012 at 2:02 PM

Webmaster level: All
(Cross-posted on the Custom Search Blog)

Using Custom Search Engine (CSE), you can create rich search experiences that make it easier for visitors to find the information they’re looking for on your site. Today we’re announcing two improvements to sorting and filtering of search results in CSE.

First, CSE now supports UI-based results sorting, which you can enable in the Basics tab of the CSE control panel. Once you’ve updated the CSE element code on your site, a “sort by” picker will become visible at the top of the results section.


By default CSE supports sorting by date and relevance. In the control panel, you can specify additional “sort by” keys that are based on the structure of your site’s content, giving users more options to find the results that are most relevant to them. For example, if you’ve marked up pages for product rich snippets, you could enable sorting based on price as shown below:


Second, we’re introducing compact queries for filtering by attribute. Currently you can issue a query like
[more:pagemap:product-description:search more:pagemap:product-description:engine]
which will only show pages with a ‘product-description’ attribute that contains both ‘search’ and ‘engine’.

With a compact query, you can issue the same request as:
[more:p:product-description:search*engine]

We hope these new features help you create richer and more useful search experiences for your visitors. As always, if you have any questions or feedback please let us know via our Help Forum.

Navigation, Dashboard and Home page

Monday, May 07, 2012 at 4:10 PM

Webmaster level: All

After announcing Webmaster Tools spring cleaning earlier this quarter, it’s time to do the job. There are a few changes coming along: an updated navigation, revamped dashboard, and a compact view for the home page site-list.

Here's the new sample Webmaster Tools Dashboard for www.example.com

We’ve regrouped the features in Webmaster Tools to create an improved navigation structure (shown on the left-hand side of the above image). We distinguished the following groups: Configuration, Health, Traffic and Optimization. Each group represents a related set of functionality:
  • Configuration: Things you configure and generally don’t change very often.
  • Health: Where you look to make sure things are OK.
  • Traffic: Where you go to understand how your site is doing in Google search, who’s linking to you; where you can explore the data about your site.
  • Optimization: Where you can find ideas to enhance your site, which enables us to better understand and represent your site in Search and other services.

If you have a moment, please take time to familiarize yourself with the new Webmaster Tools navigation. Some features were slightly renamed, such as HTML Suggestions became HTML Improvements, however the functionality remains the same.

Hoping you’ll find the new navigation useful, we also think you’ll like the new Dashboard. At the top of the Dashboard you can see recent, important, prioritized messages regarding your site. Just below that, you’ll find another section which provides a brief summary of the current status of your site. There are three widgets displayed: Crawl Errors, Search Queries and Sitemaps, each representing a different navigation group: Health, Traffic and Optimization (respectively). We know your time is valuable. With the new Dashboard, we've surfaced more messages and charts to let you see how your site is doing at a glance. Take a quick look before diving into the details.

Finally, those of you who manage a large number of sites can choose to view your site-list in a 'Compact' layout, without the large site-preview thumbnails. Don't worry, if you want the more expanded layout you can always switch back.


Compact layout of the Home page

If you have questions or comments about these changes please post them in our Help Forum.

Coding guidelines for HTML and CSS

Wednesday, May 02, 2012 at 8:46 AM

Webmaster level: All

Great code has many attributes. It’s effective, efficient, maintainable, elegant. When working on code with many developers and teams and maybe even companies, great code needs to also be consistent and easy to understand. For that purpose there are style guides. We use style guides for a lot of languages, and our newest public style guide is the Google HTML and CSS Style Guide.

Our HTML and CSS Style Guide, just like other Google style guides, deals with a lot of formatting-related matters. It also hints at best practices so to encourage developers to go beyond indentation. Many style guide authors know the underlying motivation from the question whether to describe the code they write—or to prescribe what code they want to write. Not surprisingly then, in our HTML and CSS style guide you’ll find both (as much as you’ll still find a lot of different development styles in our not entirely small code base).

At this time we only want to introduce you to this new style guide. We hope to share more about its design decisions and future updates with you. In the meantime please share your thoughts and experiences, and as with the other style guides, feel free to use our style guide for your own projects, as you see fit.

Written by Jens O. Meiert, Senior Web Architect, Google Webmaster Team

Responsive design – harnessing the power of media queries

Monday, April 30, 2012 at 6:07 PM

Webmaster Level: Intermediate / Advanced

We love data, and spend a lot of time monitoring the analytics on our websites. Any web developer doing the same will have noticed the increase in traffic from mobile devices of late. Over the past year we’ve seen many key sites garner a significant percentage of pageviews from smartphones and tablets. These represent large numbers of visitors, with sophisticated browsers which support the latest HTML, CSS, and JavaScript, but which also have limited screen space with widths as narrow as 320 pixels.

Our commitment to accessibility means we strive to provide a good browsing experience for all our users. We faced a stark choice between creating mobile specific websites, or adapting existing sites and new launches to render well on both desktop and mobile. Creating two sites would allow us to better target specific hardware, but maintaining a single shared site preserves a canonical URL, avoiding any complicated redirects, and simplifies the sharing of web addresses. With a mind towards maintainability we leant towards using the same pages for both, and started thinking about how we could fulfill the following guidelines:
  1. Our pages should render legibly at any screen resolution
  2. We mark up one set of content, making it viewable on any device
  3. We should never show a horizontal scrollbar, whatever the window size


Stacked content, tweaked navigation and rescaled images – Chromebooks
Implementation

As a starting point, simple, semantic markup gives us pages which are more flexible and easier to reflow if the layout needs to be changed. By ensuring the stylesheet enables a liquid layout, we're already on the road to mobile-friendliness. Instead of specifying width for container elements, we started using max-width instead. In place of height we used min-height, so larger fonts or multi-line text don’t break the container’s boundaries. To prevent fixed width images “propping open” liquid columns, we apply the following CSS rule:

img {
  max-width: 100%;
}


Liquid layout is a good start, but can lack a certain finesse. Thankfully media queries are now well-supported in modern browsers including IE9+ and most mobile devices. These can make the difference between a site that degrades well on a mobile browser, vs. one that is enhanced to take advantage of the streamlined UI. But first we have to take into account how smartphones represent themselves to web servers.

Viewports

When is a pixel not a pixel? When it’s on a smartphone. By default, smartphone browsers pretend to be high-resolution desktop browsers, and lay out a page as if you were viewing it on a desktop monitor. This is why you get a tiny-text “overview mode” that’s impossible to read before zooming in. The default viewport width for the default Android browser is 800px, and 980px for iOS, regardless of the number of actual physical pixels on the screen.

In order to trigger the browser to render your page at a more readable scale, you need to use the viewport meta element:

<meta name="viewport" content="width=device-width, initial-scale=1">


Mobile screen resolutions vary widely, but most modern smartphone browsers currently report a standard device-width in the region of 320px. If your mobile device actually has a width of 640 physical pixels, then a 320px wide image would be sized to the full width of the screen, using double the number of pixels in the process. This is also the reason why text looks so much crisper on the small screen – double the pixel density as compared to a standard desktop monitor.

The useful thing about setting the width to device-width in the viewport meta tag is that it updates when the user changes the orientation of their smartphone or tablet. Combining this with media queries allows you to tweak the layout as the user rotates their device:

@media screen and (min-width:480px) and (max-width:800px) {
  /* Target landscape smartphones, portrait tablets, narrow desktops

  */
}

@media screen and (max-width:479px) {
  /* Target portrait smartphones */
}


In reality you may find you need to use different breakpoints depending on how your site flows and looks on various devices. You can also use the orientation media query to target specific orientations without referencing pixel dimensions, where supported.


@media all and (orientation: landscape) {
  /* Target device in landscape mode */
}

@media all and (orientation: portrait) {
  /* Target device in portrait mode */
}



Stacked content, smaller images – Cultural Institute
A media queries example

We recently re-launched the About Google page. Apart from setting up a liquid layout, we added a few media queries to provide an improved experience on smaller screens, like those on a tablet or smartphone.

Instead of targeting specific device resolutions we went with a relatively broad set of breakpoints. For a screen resolution wider than 1024 pixels, we render the page as it was originally designed, according to our 12-column grid. Between 801px and 1024px, you get to see a slightly squished version thanks to the liquid layout.

Only if the screen resolution drops to 800 pixels will content that’s not considered core content be sent to the bottom of the page:


@media screen and (max-width: 800px) {
  /* specific CSS */
}


With a final media query we enter smartphone territory:


@media screen and (max-width: 479px) {
  /* specific CSS */
}


At this point, we’re not loading the large image anymore and we stack the content blocks. We also added additional whitespace between the content items so they are more easily identified as different sections.

With these simple measures we made sure the site is usable on a wide range of devices.


Stacked content and the removal of large image – About Google
Conclusion

It’s worth bearing in mind that there’s no simple solution to making sites accessible on mobile devices and narrow viewports. Liquid layouts are a great starting point, but some design compromises may need to be made. Media queries are a useful way of adding polish for many devices, but remember that 25% of visits are made from those desktop browsers that do not currently support the technique and there are some performance implications. And if you have a fancy widget on your site, it might work beautifully with a mouse, but not so great on a touch device where fine control is more difficult.

The key is to test early and test often. Any time spent surfing your own sites with a smartphone or tablet will prove invaluable. When you can’t test on real devices, use the Android SDK or iOS Simulator. Ask friends and colleagues to view your sites on their devices, and watch how they interact too.

Mobile browsers are a great source of new traffic, and learning how best to support them is an exciting new area of professional development.

Some more examples of responsive design at Google:


Written by Rupert Breheny, Edward Jung, & Matt Zürrer, Google Webmaster Team

Even more Top Search Queries data

Thursday, April 26, 2012 at 1:48 PM

Webmaster level: All

We recently updated the Top Search Queries data to take into account the average top position, we enabled programmatic download and we made sure you could still get all the queries that drive traffic to your site. Well, now it’s time to give you more search queries data!

First, and most important, you can 
now see up to 90 days of historical data. If you click on the date picker in the top right of Search queries, you can go back three months instead of the previous 35 days.


And after you click:


In order to see 90 days, the option to view with changes will be disabled. If you want to see the changes with respect to the previous time period, the limit remains 30 days. Changes are disabled by default but you can switch them on and off with the button between the graph and the table. Top search queries data is normally available within 2 or 3 days.


Another big improvement in Webmaster Tools is that you can now see basic search query data as soon as you verify ownership of a site. No more waiting to see your information.

Finally, we're now collecting data for the top 2000 queries for which your site gets clicks. You may see less than 2000 if we didn’t record any clicks for a particular query in a given day, or if your query data is spread out among many countries or languages. For example, a search for [flowers] on Google Canada is counted separately from a search for [flowers] on google.com. Nevertheless, with this change 98% of sites will have complete coverage. Let us know what you think. We hope the new data will be useful.

Written by Javier Tordable, Tech Lead, Webmaster Tools

1000 Words About Images

Wednesday, April 25, 2012 at 1:08 AM

Webmaster level: All

Creativity is an important aspect of our lives and can enrich nearly everything we do. Say I'd like to make my teammate a cup of cool-looking coffee, but my creative batteries are empty; this would be (and is!) one of the many times when I look for inspiration on Google Images.


The images you see in our search results come from publishers of all sizes — bloggers, media outlets, stock photo sites — who have embedded these images in their HTML pages. Google can index image types formatted as BMP, GIF, JPEG, PNG and WebP, as well as SVG.

But how does Google know that the images are about coffee and not about tea? When our algorithms index images, they look at the textual content on the page the image was found on to learn more about the image. We also look at the page's title and its body; we might also learn more from the image’s filename, anchor text that points to it, and its "alt text;" we may use computer vision to learn more about the image and may also use the caption provided in the Image Sitemap if that text also exists on the page.

 To help us index your images, make sure that:
  • we can crawl both the HTML page the image is embedded in, and the image itself;
  • the image is in one of our supported formats: BMP, GIF, JPEG, PNG, WebP or SVG.
Additionally, we recommend:
  • that the image filename is related to the image’s content;
  • that the alt attribute of the image describes the image in a human-friendly way;
  • and finally, it also helps if the HTML page’s textual contents as well as the text near the image are related to the image.
Now some answers to questions we’ve seen many times:


Q: Why do I sometimes see Googlebot crawling my images, rather than Googlebot-Image?
A: Generally this happens when it’s not clear that a URL will lead to an image, so we crawl the URL with Googlebot first. If we find the URL leads to an image, we’ll usually revisit with Googlebot-Image. Because of this, it’s generally a good idea to allow crawling of your images and pages by both Googlebot and Googlebot-Image.

Q: Is it true that there’s a maximum file size for the images?
A: We’re happy to index images of any size; there’s no file size restriction.

Q: What happens to the EXIF, XMP and other metadata my images contain?
A: We may use any information we find to help our users find what they’re looking for more easily. Additionally, information like EXIF data may be displayed in the right-hand sidebar of the interstitial page that appears when you click on an image.


Q: Should I really submit an Image Sitemap? What are the benefits?
A: Yes! Image Sitemaps help us learn about your new images and may also help us learn what the images are about.


Q: I’m using a CDN to host my images; how can I still use an Image Sitemap?
A: Cross-domain restrictions apply only to the Sitemaps’ tag. In Image Sitemaps, the tag is allowed to point to a URL on another domain, so using a CDN for your images is fine. We also encourage you to verify the CDN’s domain name in Webmaster Tools so that we can inform you of any crawl errors that we might find.


Q: Is it a problem if my images can be found on multiple domains or subdomains I own — for example, CDNs or related sites?
A: Generally, the best practice is to have only one copy of any type of content. If you’re duplicating your images across multiple hostnames, our algorithms may pick one copy as the canonical copy of the image, which may not be your preferred version. This can also lead to slower crawling and indexing of your images.


Q: We sometimes see the original source of an image ranked lower than other sources; why is this?
A: Keep in mind that we use the textual content of a page when determining the context of an image. For example, if the original source is a page from an image gallery that has very little text, it can happen that a page with more textual context is chosen to be shown in search. If you feel you've identified very bad search results for a particular query, feel free to use the feedback link below the search results or to share your example in our Webmaster Help Forum.

SafeSearch

Our algorithms use a great variety of signals to decide whether an image — or a whole page, if we’re talking about Web Search — should be filtered from the results when the user’s SafeSearch filter is turned on. In the case of images some of these signals are generated using computer vision, but the SafeSearch algorithms also look at simpler things such as where the image was used previously and the context in which the image was used. 
One of the strongest signals, however, is self-marked adult pages. We recommend that webmasters who publish adult content mark up their pages with one of the following meta tags:
<meta name="rating" content="adult" />
<meta name="rating" content="RTA-5042-1996-1400-1577-RTA" />

Many users prefer not to have adult content included in their search results (especially if kids use the same computer). When a webmaster provides one of these meta tags, it helps to provide a better user experience because users don't see results which they don't want to or expect to see. 

As with all algorithms, sometimes it may happen that SafeSearch filters content inadvertently. If you think your images or pages are mistakenly being filtered by SafeSearch, please let us know using the following form

If you need more information about how we index images, please check out the section of our Help Center dedicated to images, read our SEO Starter Guide which contains lots of useful information, and if you have more questions please post them in the Webmaster Help Forum