Robot Exclusion file for GrabPERF

Had to slap up a robots.txt file into the GrabPERF configuration; the MSN Crawler is getting a might aggressive. You really don’t need to index every single measurement page every single day. You don’t compete with Google by saturating the Internet with crawlers.

Leave a Reply

Your email address will not be published. Required fields are marked *