homepage Welcome to WebmasterWorld Guest from 54.167.182.201
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Ask - Teoma
Forum Library, Charter, Moderator: open

Ask - Teoma Forum

    
How Do You Block Teoma Spider?
akogo

10+ Year Member



 
Msg#: 312 posted 11:06 pm on Aug 3, 2003 (gmt 0)

Sounds crazy to do right now, but I need to prevent Teoma from goin nutso on my bandwidth this early in the month. Anyone know if Teoma obeys robot.txt files? If so, what's the spider/crawler name?

 

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 312 posted 1:09 am on Aug 4, 2003 (gmt 0)

akogo,

Yes, they obey robots.txt

User-agent: Ask Jeeves/Teoma

Consider disallowing them only from certain subdirectories, while allowing them to continue spidering your index page and "important" top-level pages... Otherwise, you may have to wait several months to get back in if they drop you.

Jim

akogo

10+ Year Member



 
Msg#: 312 posted 3:04 am on Aug 4, 2003 (gmt 0)

Thanks Jim,

Do you think if I disallow them now, that they will just consider for inclusion in their index what they have already taken from my site?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Ask - Teoma
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved