Looking for a good robots.txt file for WordPress? Well here’s a good example, culled from a few different guides around the web and rolled into one file:
example_robots.txt
The crawl delay prevents bots using too much of your site bandwidth by over crawling.
example_sitemap.xml.txt
You can auto generate your basic Sitemap here. Auto generate a 500 page Sitemap here.
Tags: [...]