Invisible Websites, Part II - Why Can’t the Search Engines Find My Site?

Recently we discussed the concept of “invisible websites.” These are sites that search engines don’t find at all, or sites for which only the home page appears –none of the interior pages shows up. Before you can worry about your site ranking highly for its key phrases, you need to make sure that search engines are aware that it exists, period.

You can spend hours researching key phrases that will attract targeted traffic and writing optimized web copy, but it will be wasted effort if the site has flaws that prevent it from being found.

There are several reasons why a site doesn’t show up. It may simply be too new; it may have been created using techniques that make it difficult for search engines to “see” its content; or it may violate the terms of search engines, which results in its being banned or penalized.

Getting into the Index Takes Time

You might think that once a site has been submitted to search engines, it should show up within a week or two. But it can take several weeks or months for a search engine to figure out that your site exists and put it in its index. Search engines create their own index of the web – sort of like a snapshot of the web – and that’s what they actually search. Search engine “spiders” update their index regularly.

The quickest way to get a site found by search engines, at no cost to you, is to get other, established sites to link to it. Imagine you’ve built a house in the wilderness. If there are no roads to it, how is anyone going to find it? It’s the same with websites. You’ve got to build roads (links) that show search engines how to find your site.

Don’t waste your money on services that submit your site to thousands of search engines, or spend time resubmitting your site over and over again. You don’t have to pay any money or even use a search engine’s submission forms. Just get links from established sites, and your site will be found.

If you’re in a huge rush, some search engines, such as Yahoo, offer a paid submission fee that gets your site included right away. In essence, though, you will be paying for something you’ll get for free if you just wait. Besides, Yahoo’s service only gets you into their index. It doesn’t make your site show up any higher in the search results.

Is it “Search-Engine-Friendly”?

Once a search engine finds your site, it has to be able to “read” it. Search engines care only about the text, not the graphics. (Text within graphics does not count.) If your home page is one big graphic or Flash file, the search engines may not be able to find the links to your inside pages.

Take a look at your site through a Lynx viewer (e.g., www.delorie.com/web/lynxview.html). If you can’t see the links to other pages, the search engines may not be able to, either.

Other reasons why a site might be keeping search engines at bay include the following:

• The site was temporarily down when the Google spider visited it. This problem will rectify itself the next time the Googlebot visits.
• Some sites have a robots.txt file that provides instructions to search engines. Perhaps your site’s robots file is inadvertently telling search engines to go away. Look for a phrase like “Disallow: /directoryname/,” which tells search engines to ignore that directory
• Framed sites are notoriously bad for search engines. Try the Lynx viewer referenced above to see how the search engines see your framed site.
• Dynamic, database-driven sites can cause problems, too. A site map linking to all your key pages can help them be found.

Has Your Site Been Banned or Penalized?

Websites that violate a search engine’s terms of service can be banned or penalized. Banned sites are removed from the search engine’s index. Penalized sites remain in the index, but don’t show up high in the results. Luckily, this is pretty rare, and you have to have done something really wrong for it to happen.

Recently, many sites belonging to the clients of a search engine optimization firm called Traffic Power were banned because the firm used “black hat” techniques (doorway pages, link farms, and redirects) to increase the sites’ rankings. It pays to choose an optimization firm carefully. Avoid anyone who tells you they can optimize your site without making any visible changes to it. It’s a clue that they may resort to shady techniques.

Another reason your site doesn’t show up could be that it is a duplicate. Some firms try to double their chances of being found by creating duplicate sites with different addresses. Search engines don’t like duplicates, and will simply ignore one of the sites.

Get Indexed, Then Get Found

When you’ve taken all of the right steps to ensure that your site will be indexed by search engines, all of your hard work writing content with key, phrase-rich copy will start to pay off. Your site will rise higher and higher in the rankings, and you will begin to receive your targeted traffic.