Welcome to WebmasterWorld Guest from 54.144.107.83

Forum Moderators: goodroi

Message Too Old, No Replies

save and upload robots.txt

     
9:41 am on Feb 2, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 27, 2003
posts:1308
votes: 0


Hi,
I have the problem that Inktomi only reads my robots.txt.
So I checked my robots file and made some changes just to see if anything changes.
My robots.txt is validated.

And wondering how the file should be saved in textpad, read should be in unix mode, but should it be ansi, dos, utf-8, unicode or unicode (big indian)?
Then it should be uploaded in ascii mode.

Donīt know if done correct, know the file in log file is bigger in bit than before, and itīs bigger that on my pc.

11:19 am on Feb 2, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 5, 2002
posts:1562
votes: 0


feel free to sticky me the URL and I'll have a look at it.
7:41 pm on Feb 2, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 27, 2003
posts:1308
votes: 0


Hi,
I did send you the sticky, did you receive it?
8:27 pm on Feb 2, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 1, 2004
posts:47
votes: 0


Taken from the inktomi web page:

"There is an Inktomi-specific extension to robots.txt which allows you to set a lower limit on our crawler request rate. You can add a "Crawl-delay: xx" instruction, where "xx" is the minimum delay in seconds between successive crawler accesses. Our default crawl-delay value is 1 second. If the crawler rate is a problem for your server, you can set the delay up to 60 or 300 or whatever value is comfortable for your server. Setting a crawl-delay of 20 seconds for Slurp would look something like:
User-agent: Slurp
Crawl-delay: 20
-------------------------- end ------------------------
You could use that specification inside your robots.txt file, but then that robots validator web page would show your file as invalid. So what'll be?

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members