# /robots.txt file for http://www.spu.edu/ # for more information/help: http://www.robotstxt.org/wc/exclusion-admin.html # google is always helpful: http://www.google.com/bot.html#norobots # this statement below is saying: any web crawler (*) CANNOT access the /classifieds/ folder: # but they CAN still access the /webApps/ folder... User-agent: * Disallow: /classifieds/ Disallow: /Classifieds/ Disallow: /CLASSIFIEDS/ # if there is another folder on the server, then just add the line: # Disallow: /folder_name/ # you can have subfolders as well: # Disallow: /folder_name/sub_folder_name/