How to Get Your Website Indexed

A question many affiliates ask is how to get the search engines crawling your website in the first place. In this lesson we look at How to Get Your Website Indexed.

Getting Seen

There are a couple of ways you can get the search engines to actually find your website:

1) Get a link

Search engine spiders are just basically tireless wanderers. They wander around the internet following links and seeing what they discover. Therefore the most straightforward way to get your site indexed is to get a link from another site. There are a couple of caveats here: The site that links to you has to be indexed by the search engines itself, and the link can't have a "nofollow" tag on it (if you want to get into Google).

One good way to get the search engines crawling your site is to set up a Twitter profile.The search engines love Twitter and the spiders spend a lot of time hanging out there. Once you've set up your profile, try to find some active members of the Twitter community to "follow" — which is basically like adding yourself to their friends list. This provides a link back to your Twitter profile, and allows the search engine spiders to start crawling your Twitter profile. Now just post a link to your new site and you should see little search engine bot footprints all through your stats in no time.

2) Submit your site to the search engines

Many of the search engines have "submit your site" pages where you can enter your site's URL and some other information as a suggestion that they include it in their index. It's not a guarantee that they will index it.

This used to be a good way of getting your site indexed back in the day, but these days all the major search engines find your site by the link method we talked about above, so there's not really much point.

Are You Indexed?

A quick trick to check if your website has been indexed is to go to Google and entersite:yourdomain.com into the search box. Google will return all the results they have for your website. The same trick applies for Yahoo and MSN.

You can also access this same information (and a bunch of other stuff) by signing up for Google's Webmaster Tools. Once you verify your site by adding a snippet of code to your HTML or uploading a file, you'll see an option to see which pages have been indexed by Google. This is pretty much the same as doing the site:yourdomain.com trick, but Google can also tell you if they're having difficulty crawling any of your pages, and the nature of the difficulty. This can be pretty handy for checking the overall health of your site structure.

How to get ALL your pages indexed

Getting the search engines to visit your site is one thing, but getting them to visit and index allthe pages on your website is a different matter. When you check how many pages of your site are indexed, you may notice that some of them are missing.

A search engine spider may visit your site but fail to index some of your pages for a number of reasons:

• It ran out of steam
  If you don't have many links to your site just yet (and consequently little PageRank) you might find the search engine spiders "not bothering" to dig too deeply into your site.

• You're using duplicate content

  If the search engines think that the content on your page is pretty much the same as content on another page (this could be on your website or on a completely different website) then it probably won't show up in their listings.

• Badly organized link structures

  Remember that the search engines find pages on your site by following links. A page on your site with more links to it is going to be easier to find than a page with only one link to it, buried in the deep recesses of your site. If a page is important, make sure it's easily accessible.

Suggestions for getting your pages indexed:

• Use unique content
  Make sure all your content is at least 25% different to other content on the internet, andpreferably at least 50% different. This is particularly true if you're using PLR content on your site — don't be lazy, reword it!

• Get more links to your site
  The more good links you have to your site, the more "energy" the search engine spiders will have to explore your pages. You can tell how strong your links are by looking at your PageRank. The higher your PageRank, the more chance that the spiders will go deeper into your site.

• Get links to your internal pages
  Rather than just having every link pointing at your home page, try getting links that point to your internal pages as well. This is called "deep linking" and it's a very useful strategy.

  Think of it this way: instead of conducting an all-out assault on your front door (and only getting as far as your living room), deep-linking means the spiders will also try the back door, the laundry window, and the ranch slider. They'll see a lot more of your house as a result!

PageRank Sculpting

PageRank "sculpting" is where you design your site so that as much PageRank "flows" to your important pages as possible. This means that these pages are more likely to appear in the search engines.

It sounds like a complex and painful procedure, but once you get past the fancy words it's really just what we were talking about before: making sure that your important pages are well-linked and easily accessible, and that the spiders have enough "energy" to reach your important pages.

Here's an example of a site that hasn't been "PageRank sculpted". It has a lot of links arriving at its home page, and then from the home page you can access the usual range of pages: a links page, a privacy policy page, a "contact us" page, a reviews page and an articles page.

From the articles page there are links to a bunch of articles - this is the main content of the site. The owner of this site expects that most of his traffic will arrive through through these articles, because they've been optimized for very popular keywords.

As you can see from the darkness of the shading, these articles aren't getting much PageRank flowing through to them. The PageRank is being split between the links at the top, but because the "articles" page only gets one fifth of the available PageRank, each of it's children (the articles) only receive one-tenth of one-fifth.

Chances are, until the website owner starts getting a lot of good links (and a lot of PageRank) to his home page, some of his articles might not be indexed because there's simply not enough PageRank flowing through to them.

Tip 1: Use "nofollow"

Does the website owner really need all that PageRank to go to his privacy policy and contact us page? Not really. So a good idea is to either remove links to these pages from his main home page, or use "nofollow" in his links. "Nofollow" tells the search engines not to bother exploring through this link, and means they can direct their attention to your more important pages. By blocking off those three unimportant pages, the website owner can increase the amount of PageRank flowing to his more important pages: The articles page and the reviews page.

Of course you probably want these pages indexed eventually. So it's a good idea to link to them from one of your less-important pages. It's just a way of telling the spiders which pages to pay attention to first.

Now we should note here: Google claims that PageRank sculpting doesn't work, and their engine does not direct PageRank evenly across all the links on a page; however we have run tests which show that this is, in fact, the case, so it's better to play safe than sorry and apply this trick anyway.

Tip 2: Get deep links

To add even more punch to this strategy, the website owner should work on getting links to his article pages as well as his home page. This benefits the article page itself, as well as any other pages the article might link to. You can see here that the PageRank spreads out through all pages the article links to, increasing the chance of those articles being indexed.

Create a Sitemap

Sitemaps help search engine spiders find pages on your website by listing every single page that you have. XML sitemaps can also include information like the importance of the page that can help the spiders know which pages they should pay most attention to. If you're having difficulty getting all your pages indexed, it may be worth creating a sitemap.

(For more information on how to do this, see our lesson on how to create a sitemap.)

Lesson Summary

In this lesson we've looked at how to get your page indexed, including:

• Get a link to your website
• Submit your site to the search engines

We also looked at checking if the search engine has indexed your site and what to fix if it isn't getting indexed correctly, including:

• Checking with "site:yourdomain.com"
• Ensuring you have unique content
• Getting more backlinks to your site
• Deep linking
• Creating a sitemap