homepage Welcome to WebmasterWorld Guest from 107.22.25.119
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
Accredited PayPal World Seller

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library : Charter : Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
save and upload robots.txt
helenp




msg:1526671		 9:41 am on Feb 2, 2004 (gmt 0)

Hi,
I have the problem that Inktomi only reads my robots.txt.
So I checked my robots file and made some changes just to see if anything changes.
My robots.txt is validated.

And wondering how the file should be saved in textpad, read should be in unix mode, but should it be ansi, dos, utf-8, unicode or unicode (big indian)?
Then it should be uploaded in ascii mode.

Donīt know if done correct, know the file in log file is bigger in bit than before, and itīs bigger that on my pc.

 

ThomasB




msg:1526672		 11:19 am on Feb 2, 2004 (gmt 0)

feel free to sticky me the URL and I'll have a look at it.
helenp




msg:1526673		 7:41 pm on Feb 2, 2004 (gmt 0)

Hi,
I did send you the sticky, did you receive it?
sovidiu




msg:1526674		 8:27 pm on Feb 2, 2004 (gmt 0)

Taken from the inktomi web page:

"There is an Inktomi-specific extension to robots.txt which allows you to set a lower limit on our crawler request rate. You can add a "Crawl-delay: xx" instruction, where "xx" is the minimum delay in seconds between successive crawler accesses. Our default crawl-delay value is 1 second. If the crawler rate is a problem for your server, you can set the delay up to 60 or 300 or whatever value is comfortable for your server. Setting a crawl-delay of 20 seconds for Slurp would look something like:
User-agent: Slurp
Crawl-delay: 20
-------------------------- end ------------------------
You could use that specification inside your robots.txt file, but then that robots validator web page would show your file as invalid. So what'll be?

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved