# Core One Generated Robots.txt # Robot Exclusion File -- robots.txt # Author: Core One Team # Last Updated : August 21, 2010 User-Agent: * Disallow: /markup/ Disallow: /dogfoodie/ Disallow: /sociallyweb/ Disallow: /socially-web/ Disallow: /express/ Disallow: /curl/ Disallow: /cgi-bin/ Disallow: /shop-clone/ Disallow: /shop_bulletcable/ Disallow: /shop_corex2/ Disallow: /express2/ # Current Testing pages Disallow: /indexnewcoreone.html Disallow: /test.php Disallow: /inner.html Disallow: /branding/ Disallow: /core-one-notices/ Disallow: /dealer-locations/ Disallow: /includes/ Disallow: /press-release/ Disallow: /core-one-products/ # IF YOU DO NOT WISH TO HAVE THE GOOGLE IMAGE BOT SCAN YOUR DOMAIN FOR IMAGES # THEN YOU CAN INCLUDE THE FOLLOWING IN YOUR ROBOTS FILE. # I FOUND THAT MY BANDWIDTH USAGE DROPPED BY A MASSIVE AMOUNT AFTER I GOT RID # OF THE GOOGLE IMAGE BOT. ALL I HAD WAS IMAGE HUNTERS STEALING PRODUCT SHOTS # AND NOT EVEN BROWSING THE SITE. User-agent: Googlebot-Image Disallow: /nav-images/