Looking for a good robots.txt file for WordPress? Well here’s a good example, culled from a few different guides around the web and rolled into one file:
The crawl delay prevents bots using too much of your site bandwidth by over crawling.
You can auto generate your basic Sitemap here. Auto generate a 500 page Sitemap here.
1 response so far ↓
1 Zhdanov // Nov 5, 2008 at 8:46 PM
Fun. Add to your bookmarks. And how long it took to write articles?
Leave a Comment