Robot Help

[Stra]

hello everyone,

I am pretty new at this stuff, And this is a great place to get information, but here is want I have. I have been trying to get a site indexed for about 3 weeks now I have noticed in my web stats in my cpanel that the googlebot has been at my site. Can anyone tell me about how long before it will be indexed. If it does get indexed..

And one more thing can any one tell me why a site will show up in yahoo search but won't show up in the main search. and my site that shows up in the search is a page that I had deleted. and I need it to pick up the page I have now.

I have got a robot.txt file in my root directory but I didn't put anything in it because I read in this forum that it would stop alot of the 404 errors. But should I put something in that file so the bot will know..

Can anyone help!

Thanks, Stra

[ncw164x]

hello stra

did you make a typo error by not putting an "s" robot.txt ;)

It is a robots.txt file and yes you are correct you can have a blank file to stop the crawlers from filling you sites log file with 404 errors when the file is requested.

To create a robots.txt file

Create a text file using a Word Processor or HTML editor using the required coding as examples below
Save the file as robots.txt
Upload the robots.txt file to the root directory using your FTP software in ACSII mode

A few Examples
To exclude all robots from parts of the server
User-agent: *
Disallow: /cgi-bin/
Disallow: /sitestats/

Exclude a specific spider from parts of the server
User-agent:slurp.so/
Disallow: /cgi-bin/
Disallow: /images/

This indicates that nothing is disallowed and the spider can follow all links
User-agent: *
Disallow:

To allow a single robot complete access and exclude all others
User-agent: Googlebot/1.0
Disallow:
User-agent: *
Disallow: /

This would prevent your entire web site from being indexed
User-agent: *
Disallow: /

Can anyone tell me about how long before it will be indexed

have you tried doing site:www.yoursiteurl.com at google to see if its been indexed and how many pages googlebot has taken

Do you have any links to your site
[webmasterworld.com...]

It could be sandboxed by google
[webmasterworld.com...]

hope this helps

ncw164x

[Stra]

Sorry That is a typo!

So if I put something like this

User-agent: *
Disallow: /index2.html/

Would this work..

[ncw164x]

Without the trailing slash

User-agent: *
Disallow: /index2.html

[Stra]

Thanks