Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

How Do You Block Teoma Spider?



11:06 pm on Aug 3, 2003 (gmt 0)

10+ Year Member

Sounds crazy to do right now, but I need to prevent Teoma from goin nutso on my bandwidth this early in the month. Anyone know if Teoma obeys robot.txt files? If so, what's the spider/crawler name?


1:09 am on Aug 4, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member


Yes, they obey robots.txt

User-agent: Ask Jeeves/Teoma

Consider disallowing them only from certain subdirectories, while allowing them to continue spidering your index page and "important" top-level pages... Otherwise, you may have to wait several months to get back in if they drop you.



3:04 am on Aug 4, 2003 (gmt 0)

10+ Year Member

Thanks Jim,

Do you think if I disallow them now, that they will just consider for inclusion in their index what they have already taken from my site?


Featured Threads

Hot Threads This Week

Hot Threads This Month