comments 4

WordPress robots.txt For Search Engine Optimization (SEO) – SEO robot.txt

Do you know that search engine optimisation (SEO) can also be done by having a neat robots.txt?

For your information, robots.txt is used to tell search engine robots about what files or folder on your website to be crawled so that certain pages consequently get indexed in the search engines.

To know what pages have been indexed by a search engine, type site:yoursite.com in the search box. A Google search on this website shows there is a total of 1490 indexed pages.

The more indexed pages you have, the greater the possibility that visitors would stumble upon your blog as they searched through certain keywords.
For those using WordPress content management system, you may adapt or adopt my version of robots.txt:

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /category/*/*
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Allow: /wp-content/uploads

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

# Adsbot-Google
User-agent: Adsbot-Google
Disallow:
Allow: /*

#Googlebot Mobile
User-agent: Googlebot-Mobile
Disallow:
Allow: /*

# Disable Internet Archiver Wayback Machine
User-agent: ia_archiver
Disallow: /

# Disable digg mirror
User-agent: duggmirror
Disallow: /

Sitemap: http://r3m1ck.us/sitemap.xml
Sitemap: http://r3m1ck.us/sitemap.xml.gz

Basically the robots.txt file allows the “googlebot”, which is the search engine bot of Google, to retrieve and index every page from my site except for directories “cgi-bin”, “wp-admin”, “wp-includes”, and so forth.

Google user-agents allowed here are Googlebot-Image, Mediapartners-Google, and Googlebot-Mobile.

Googlebot-image Googlebot-Mobile crawl pages for image index (images.google.com) and mobile index (m.google.com) respectively. Mediapartners-Google, which only oncerns Adsense users, crawls pages to determine Adsense content.

A robots.txt file should be placed at the root of the domain, e.g., r3m1ck.us/robots.txt.

4 Comments

  1. Haha, If you have a website in English language with 500 unique visitors per day, I can make you earn $200-1000 everyday and the request is after receiving the payment, we share the revenue 50 to 50. This is an invitation sent to you via a group-sending software, which helped me send more than 50,000 invitations to blog writers using wordpress, although only 5 of them established the cooperative relationship with us, they now get $2000-10000 every month. If you are interested in this invitation, please contact us. You will get an auto email reply with an url link liking to detailed information about this project. ;-)

Leave a Reply

Your email address will not be published. Required fields are marked *