Crawling Difficulties and Error Pages

I like to put my error pages in a subdirectory usually named “error”. So any 401, 403, 404 or other errors will end up in http://www.example.com/error/404/ . I also use this for other thing like database failures or things so I end up with http://www.example.com/error/database-error/ with a hopefully human friendly explanation of what happened. In my robots.txt file I block the entire error directory for all crawlers. Google seems to be having a real issue with this structure as it will report the original page as blocked by robots.txt. So it looks like I may be shooting myself in the foot by trying to be more organized.

GraywolfSEO.com runs on the Genesis Framework

Genesis Framework

Genesis lets you to quickly and easily build amazing websites with WordPress. Whether you're a novice or advanced developer, Genesis provides the secure and search-engine-optimized foundation that takes WordPress to places you never thought it could go.
It's that simple - start using Genesis now!


Take advantage of the 6 default layout options, comprehensive SEO settings, rock-solid security, flexible theme options, cool custom widgets, custom design hooks, and a huge selection of child themes ("skins") that make your site look the way you want it to. With automatic theme updates and world-class support included, Genesis is the smart choice for your WordPress website or blog.


tla starter kit

Advertisers:

  1. Text Link Ads - New customers can get $100 in free text links.
  2. BOTW.org - Get a premier listing in the internet's oldest directory.
  3. Need an SEO Audit for your website, look at my SEO Consulting Services
  4. TigerTech - Great Web Hosting service at a great price.
More in Google, npb, npp, ntt, SEO
Is My Website Banned in Google?

One of the more difficult problems to diagnose is has your website been banned or is there something else wrong?...

Azoogle 2.0 Interface

Sorry I'm totally not liking the new Azoogle 2.0 interface. Eye candy for the sake of eye candy. Much slower...

Google Sitemaps for Large Scale Sites

If you've got a any first hand experience working with and submitting sitemaps for sites with over 50K pages drop...

Close