Forum Moderators: phranque

Message Too Old, No Replies

Bad robots.txt - Please help!

Bad robots.txt

         

JackieB

4:22 am on Apr 7, 2005 (gmt 0)

10+ Year Member



I ram my robots.txt file through a "validator" and it may have errors.

Here is the text of the file
###############################
#
#
User-agent: *
#
# list folders robots are not allowed to index
Disallow: /web/images/
Disallow: /images/
Disallow: /slideshow/
Disallow: /galleries/
Disallow: /engagements/
Disallow: /gallery1/
Disallow: /gallery2/
Disallow: /gallery3/

# User-agent: Googlebot-Image
Disallow: /

#
#

It is telling me that each foolder path may be misintrupreted.
I copied the Google no image text directly from the google site.

Thanks,
Jackie
###############################

Code Sentinel

6:43 pm on Apr 7, 2005 (gmt 0)

10+ Year Member



remove the comment # before the google user agent and move the whole google section ABOVE the user agent * section.

JackieB

12:08 am on Apr 8, 2005 (gmt 0)

10+ Year Member



Thank you for your note. It appears that your site is no longer in our
index. Google will continue to exclude your site from successive crawls if
the robots.txt file exists in the web server root. Once you remove or
change your robot.txt file, your site should be picked up in future
crawls. More information on robots.txt is available at
[robotstxt.org...]

I read this. All I want to do is not have the images archived. What is wrong with this robots.txt?

Please help!

Jackie

JackieB

12:24 am on Apr 8, 2005 (gmt 0)

10+ Year Member



###############################
#
#

User-agent: Googlebot-Image
Disallow: /
#
#
User-agent: *
#
# list folders robots are not allowed to index
Disallow: /web/images/
Disallow: /images/
Disallow: /slideshow/
Disallow: /galleries/
Disallow: /engagements/
Disallow: /gallery1/
Disallow: /gallery2/
Disallow: /gallery3/
#
#
###############################

JackieB

1:01 am on Apr 8, 2005 (gmt 0)

10+ Year Member



I deleted the Googlebot line. Although I don't want Google to index the images, if I can't figure out why they won't crawl me, it is better than nothing.

Jackie

Code Sentinel

2:45 am on Apr 8, 2005 (gmt 0)

10+ Year Member



did you not read my post?

JackieB

3:01 am on Apr 8, 2005 (gmt 0)

10+ Year Member



I did. The Googlebot line was moved to the top.

However, that was before google told me they wouldn't ixex me again at all unless I remove the robots.txt file.

So, I am thinking it would be safer for me to remove the google reference altogether.

nothin' like being in the first 3 spots on my keywords to being removed completely!

I am scared to death!

Jackie

bill

8:33 am on Apr 8, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Your previous robots.txt effectively blocked all robots from your entire site. What the search engines would see is:

User-agent: *
Disallow: /

That disallows all robots from all directories on your site.

You'll have to wait for the robots to come back and see your new robots.txt file.

JackieB

3:52 pm on Apr 8, 2005 (gmt 0)

10+ Year Member



###############################
#
#
#
User-agent: *
#
# list folders robots are not allowed to index
Disallow: /images/
Disallow: /slideshow/
Disallow: /galleries/
Disallow: /engagements/
Disallow: /gallery1/
Disallow: /gallery2/
Disallow: /gallery3/
#
#
###############################

Latest version.

If this works, I will NEVER touch robots.txt again.

thanks,
jackie