You'll want to sic a spider on your own site, before letting MSN, Google or Yahoo get at it. Why? To make sure its spiderable, that's why! Huh??? Didn't think about validating your linking structure, did you?

There are many page whackers available that download a whole site to your hard drive, but who knows what kind of "forgiving" spidering technology they're using. The best one, the one that replicates the crawling spiders like Google and Yahoo the closest, is a tool like OptiSpider.

Here's a tip, if OptiSpider can't spider your site... neither can Google. That's right, it's probably just about the best reason to get and use a tool like OptiSpider, just so you can see the same stuff the spiders see.

If you manage multiple sites for clients, it's a great weapon in your toolbox! I can't count how many times I've done coaching sessions for clients who claim search engines are not finding all their pages. A few seconds later - using OptiSpider - I can tell them... dude!!! You didn't link to the pages properly, or your custom 404 not found page is killing the search engine spiders.

No Google, Yahoo or MSN spider is going to find all your pages, when you put the crawler in an endless feedback loop. You're lucky if the spiders ever come back.

Stop doing whatever it is (even if it's reading this article ;-) and go look at your custom 404 page. Everyone loves to use them. Unfortunately, many people use them badly. The biggest sin of all - even I was guilty of this - is using a relative link on the 404 page. It's so common that relative linking should be banned altogether.

Imagine the spider is several directories down in your site and encounters a missing page or broken link, up comes your custom 404 error page. Too bad sucker! You used a relative link instead of an absolute one. The spider is totally miffed at your site and resets, leaving all the rest of your pages behind... unspidered, unindexed and unfound.

Stop whatever it is you're doing and go do it now. Put an absolute link on your 404 page, one that includes the http://www. and your full .com web address to your home page. That way, when your custom 404 error page comes up, the spider won't choke. Instead, it finds your home page from the custom 404 page and continues spidering like normal.

If you want to make spiders happy and be sure they can crawl your entire site, then use a spidering tool like OptiSpider to test your internal linking structure. Just don't be surprised if you end up getting a whole lot more pages indexed by the search engines. What's on your 404 page?

by Michael Campbell

Author of....
Revenge of the Mininet... Advanced search engine linking strategies and diagrams for increased revenue.

Clickin' it Rich... The complete work from home business training system for new affiliates.

Nothing but 'Net... Simple internet marketing strategy that made $750,000 in less than a year.

Internet Marketing Secrets Newsletter... Learn how to harness the money making power of multiple internet revenue streams like search engines, affiliate programs, paid advertising, opt-in email, newsletters, ebooks and lot more in internet marketing secrets. Subscribe now and get a free lifetime membership.

Copyright 2005 and beyond... internetmarketingtips.ws - All Rights Reserved