Crawling Difficulties and Error Pages

I like to put my error pages in a subdirectory usually named “error”. So any 401, 403, 404 or other errors will end up in http://www.example.com/error/404/ . I also use this for other thing like database failures or things so I end up with http://www.example.com/error/database-error/ with a hopefully human friendly explanation of what happened. In my robots.txt file I block the entire error directory for all crawlers. Google seems to be having a real issue with this structure as it will report the original page as blocked by robots.txt. So it looks like I may be shooting myself in the foot by trying to be more organized.

GraywolfSEO.com runs on the Genesis Framework

Genesis Framework

Genesis lets you to quickly and easily build amazing websites with WordPress. Whether you're a novice or advanced developer, Genesis provides the secure and search-engine-optimized foundation that takes WordPress to places you never thought it could go.
It's that simple - start using Genesis now!


Take advantage of the 6 default layout options, comprehensive SEO settings, rock-solid security, flexible theme options, cool custom widgets, custom design hooks, and a huge selection of child themes ("skins") that make your site look the way you want it to. With automatic theme updates and world-class support included, Genesis is the smart choice for your WordPress website or blog.


tla starter kit

Advertisers:

  1. Text Link Ads - New customers can get $100 in free text links.
  2. BOTW.org - Get a premier listing in the internet's oldest directory.
  3. Need an SEO Audit for your website, look at my SEO Consulting Services
  4. TigerTech - Great Web Hosting service at a great price.