This is an article about Sitemaps and Google Webmaster Tools.

This article won’t tell you how to make a Sitemap or how to use Google Webmaster Tools. There are plenty of good articles that explain the mechanics of Sitemaps and Webmaster Tools (see Add a Sitemap to Your Wordpress Site). But sometimes it’s good to get up above the trees and see the big picture.

In a Nutshell

When people search the web using Google, you want them to find your Site.

But if Google doesn’t have the right information, it won’t return links to your Site.

Yes, Google needs information about your Site. Why is this?

Well, let’s look at an example.

Suppose someone is a dog lover. This person wants Google to find links to pages with dogs. More specifically, suppose this person types “Boston Terrier” into the Google search box, and hits return.

So now the Google search engine has two problems to solve. First, it wants to find site with the “Boston Terrier” content. Second, it doesn’t want to return unpopular sites. It wants to return a site people are interested in, otherwise people will stop using Google and start using another search engine.

So the Google search engine really as two problems to solve — content and popularity.

How Google Keeps Track of Content

We’ll discuss popularity (called “ranking”) later in this article, but for now the topic is content.

How does the Google search engine keep track of everything on the Web?

Well, the truth is, the Google search engine doesn’t keep track of EVERYTHING on the Web; but it tries to.

Someplace (probably more than one place) Google corporation has an enormous number of big Server machines with lots of hard drives and backup media, just for keeping lists of what is on the World Wide Web.

But the problem (actually opportunity) for Google is that web content keeps changing. Each day trillions pieces of new information are added to the Web.

So the Google search engine is constantly combing through the web in a process called “crawling”.

Here is the definition of “crawling”, according to a Google Webmaster help page,:

“Crawling is the process by which Googlebot discovers new and updated pages to be added to the Google index.”

Google goes on to say the following:

We use a huge set of computers to fetch (or “crawl”) billions of pages on the web. The program that does the fetching is called Googlebot (also known as a robot, bot, or spider). Googlebot uses an algorithmic process: computer programs determine which sites to crawl, how often, and how many pages to fetch from each site.

Another Way Google Tracks Content

As described in the previous section, the Google search engine is constantly looking at sites on the Web to build indices of Web content. But there is still a lot of content that the Google search engine cannot process.

For instance, if you submit a video to YouTube, the Google search engine can’t (yet!) watch the video and create index data based on the content of the Video.

Also, more sophisticated Web sites build their pages dynamically, meaning that pages and content of pages change, depending upon user input. It is difficult or impossible for the search engine to build an index of such a site.

So there is some content that the Google search engine cannot find or index by crawling. Still, the authors of such sites want people to be able to find their content and the operators of the search engines want to be able to find the content. Therefore, Sitemaps were invented.

A Sitemap is a protocol webmasters use to tell search engines about Web pages that are available for crawling. Below is a very simple example of a Sitemap. Again, we leave it to other documents and articles to give detailed technical descriptions. Suffice it to say, the information in the Sitemap helps the Google search engine to find pages on your site. http://www.sitemaps.org has official specifications and other useful information.

Submitting the Sitemap

At the beginning of this article, I implied that Sitemaps and Google Webmaster Tools go together like milk and cookies. Here’s why — once your Sitemap is ready, you register your Site with Google Webmaster Tools and then submit your Sitemap.

You join Google Webmaster Tools pretty much the same way you join any other Google Application Service. Just go to the Google Webmaster Tools Site and sign in with an existing Google Account Name/Password or create a new one. Then follow the instructions for registering your Site and submitting your Sitemap.

If you have followed reasonable rules of SEO (Search Engine Optimization) in creating your site and your site has good quality content then you are off and running. Of course, Google makes different types of reports available. Demystifying Google Webmaster Tools Reports is a nice place to start reading about them.

Other Search Engines

As you know, Google isn’t the only search engine. Both Microsoft Bing and Yahoo Search have tools for registering your Site and submitting Sitemaps. Check them out too.

Ranking (popularity)

As promised earlier, we include here a brief section on the topic of ranking. It’s just the way it sounds. The Google search engine ranks pages. Google’s method is not disclosed but it is generally understood to be based on popularity. This includes factors like how many times people search for your site, how many other sites link to your site, and the ranking of the sites that have links to your site, and possibly how long your site has been around.

There is a marketing specialty called Search Engine Optimization (SEO) that deals with optimizing your site to improve its search engine rankings. See the Wikipedia Article on Search Engine Optimization.