Welcome to WebmasterWorld Guest from 54.162.152.232

Forum Moderators: open

Message Too Old, No Replies

How Do You Block Teoma Spider?

     
11:06 pm on Aug 3, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 19, 2002
posts:173
votes: 0


Sounds crazy to do right now, but I need to prevent Teoma from goin nutso on my bandwidth this early in the month. Anyone know if Teoma obeys robot.txt files? If so, what's the spider/crawler name?
1:09 am on Aug 4, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


akogo,

Yes, they obey robots.txt

User-agent: Ask Jeeves/Teoma

Consider disallowing them only from certain subdirectories, while allowing them to continue spidering your index page and "important" top-level pages... Otherwise, you may have to wait several months to get back in if they drop you.

Jim

3:04 am on Aug 4, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:July 19, 2002
posts:173
votes: 0


Thanks Jim,

Do you think if I disallow them now, that they will just consider for inclusion in their index what they have already taken from my site?

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members