Welcome to WebmasterWorld Guest from 54.166.179.165

Forum Moderators: open

Message Too Old, No Replies

How Do You Block Teoma Spider?

     
11:06 pm on Aug 3, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 19, 2002
posts:173
votes: 0


Sounds crazy to do right now, but I need to prevent Teoma from goin nutso on my bandwidth this early in the month. Anyone know if Teoma obeys robot.txt files? If so, what's the spider/crawler name?
1:09 am on Aug 4, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


akogo,

Yes, they obey robots.txt

User-agent: Ask Jeeves/Teoma

Consider disallowing them only from certain subdirectories, while allowing them to continue spidering your index page and "important" top-level pages... Otherwise, you may have to wait several months to get back in if they drop you.

Jim

3:04 am on Aug 4, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 19, 2002
posts:173
votes: 0


Thanks Jim,

Do you think if I disallow them now, that they will just consider for inclusion in their index what they have already taken from my site?