Welcome to WebmasterWorld Guest from 23.22.182.29

Forum Moderators: goodroi

Message Too Old, No Replies

Site Pages are Blocked by Robots.txt

     
4:46 pm on May 1, 2008 (gmt 0)

Full Member

5+ Year Member

joined:Sept 11, 2007
posts: 303
votes: 0


Hello,

I have converted my site from asp to aspx(.net), since its conversion I continously facing problems. One of them is that my site 15 official pages are restricted by robots.txt file. It is happened when I got xml sitmap to resubmit on google. After resubmitting xml sitemap, a message was dispalyed there that 15 url(s) are restricted by rotbos.txt file. Even I used the following user agent:

user agent: *

disallow:

I gave open visits to all bots in the above mentioned.

Please help me out what to do?
Is .net is bad for google bot or any other bot?

Thanks,

Bilal

8:11 pm on May 4, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Dec 5, 2002
posts:1562
votes: 0


I would just remove the robots.txt file as you don't seem to need it. After that everything should be fine again.
8:26 pm on May 4, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


The code is invalid, with incorrect keywords, incorrect case, and incorrect use of line space. It must conform exactly with the format given in the robots.txt paper -- if you want it to be accepted and correctly-interpreted by all robots, there is no wiggle-room whatsoever.

User-agent: *
Disallow:


Note that blank line at the end. A blank line must appear after each 'record' in robots.txt.

Jim

10:03 am on May 6, 2008 (gmt 0)

Full Member

5+ Year Member

joined:Sept 11, 2007
posts:303
votes: 0


Hey thanks for the help, can you please explain what is the right code or syntax for robots.txt file. Actually I want to give an open visit to all crawler, and it seems that after implementing new robots.txt file no bots taking visits. Even google bot had its last visit on 23rd April, 2008. Please help me out my site is loosing its position in different search engines.

Thanks,

Bilal