homepage Welcome to WebmasterWorld Guest from 54.161.192.130
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How to Disallow Jeeves?
skuba

10+ Year Member



 
Msg#: 311 posted 6:27 pm on Mar 5, 2004 (gmt 0)

I saw several different information on how the user agent for ask jeeves is called.
I want to disallow it in my robots.txt but I don't know how to call it.
On my stats it calls the robot just Jeeves. So is that how I should set on the robots.txt?
I also heard people saying it is Ask Jeves, or Ask Jeeves/Teome, or teomaagent1, or directhit..........................

So, what is the correct one?

Thanks

 

ncw164x

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 311 posted 6:38 pm on Mar 5, 2004 (gmt 0)

Why do you want to ban a spider from one of the larger search engines which is normally well behaved.

With the current changes going on I would of thought your site needs all the FREE exposure it can get because in the future it may not be FREE

ncw164x

skuba

10+ Year Member



 
Msg#: 311 posted 6:50 pm on Mar 5, 2004 (gmt 0)

Becuase Jeeves took 3GB of my bandwidth last month and I heard that most of the time the don't add your pages to their listings. I never got any traffic from them. I also asked other developers and they said that they get almost no traffic from Ask Jeeves.
Do you get?

TomV

10+ Year Member



 
Msg#: 311 posted 6:53 pm on Mar 5, 2004 (gmt 0)

Considering that AskJeeves just purchased Excite and Iwon and are expected to garner about 7% of se traffic you may want to reconsider.

skuba

10+ Year Member



 
Msg#: 311 posted 7:03 pm on Mar 5, 2004 (gmt 0)

Ok, I will give it a chance.

But, usually how long does it take to see your pages listed there after they are crawled?
Do you get traffic from them?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved