ปกติแล้วเราสามารถทำการ block พวก bot ของ search engine ได้โดยการเขียน syntax ใส่ไฟล์ robots.txt
แล้วนำไฟล์นี้ไปไว้ใน path web ครับผม
ตัวอย่างเช่น
User-agent: *
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /rss/
Disallow: /comments/feed/
Disallow: /page/
Disallow: /date/
Disallow: /comments/
User-agent: Googlebot
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /*?*
Disallow: /*?
# This is the ad bot for google
User-agent: Mediapartners-Google*
# Allow Everything
Allow: /*
Test URLs against this robots.txt file
http://www.ghacks.net/http://www.ghacks.net/2007/05/20/support-ghacks/http://www.ghacks.net/tag/http://www.ghacks.net/category/http://www.ghacks.net/2007/05/20/flitter-a-flickr-twitter-realtime-screensaver/trackback/You may add a second search engine bot which should also try and crawl the site. It would be a good idea to select the Adsense bot for instance. Clicking on check displays the results if Google bot wanted to crawl your website.
Allowed means that Google Bot is able to crawl that type of sites will Blocked means that this type of sites will not be crawled. If the results are not to your satisfaction you can easily edit the robots.txt and check again until they are.
