Welcome to WebmasterWorld Guest from 18.104.22.168 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
How Do You Block Teoma Spider? akogo msg:32999 11:06 pm on Aug 3, 2003 (gmt 0) Sounds crazy to do right now, but I need to prevent Teoma from goin nutso on my bandwidth this early in the month. Anyone know if Teoma obeys robot.txt files? If so, what's the spider/crawler name?
jdMorgan msg:33000 1:09 am on Aug 4, 2003 (gmt 0)
Yes, they obey robots.txt
User-agent: Ask Jeeves/Teoma
Consider disallowing them only from certain subdirectories, while allowing them to continue spidering your index page and "important" top-level pages... Otherwise, you may have to wait several months to get back in if they drop you.
akogo msg:33001 3:04 am on Aug 4, 2003 (gmt 0)
Do you think if I disallow them now, that they will just consider for inclusion in their index what they have already taken from my site?