homepage Welcome to WebmasterWorld Guest from 54.167.144.202
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Site Pages are Blocked by Robots.txt
bilalseo

5+ Year Member



 
Msg#: 3639586 posted 4:46 pm on May 1, 2008 (gmt 0)

Hello,

I have converted my site from asp to aspx(.net), since its conversion I continously facing problems. One of them is that my site 15 official pages are restricted by robots.txt file. It is happened when I got xml sitmap to resubmit on google. After resubmitting xml sitemap, a message was dispalyed there that 15 url(s) are restricted by rotbos.txt file. Even I used the following user agent:

user agent: *

disallow:

I gave open visits to all bots in the above mentioned.

Please help me out what to do?
Is .net is bad for google bot or any other bot?

Thanks,

Bilal

 

ThomasB

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3639586 posted 8:11 pm on May 4, 2008 (gmt 0)

I would just remove the robots.txt file as you don't seem to need it. After that everything should be fine again.

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3639586 posted 8:26 pm on May 4, 2008 (gmt 0)

The code is invalid, with incorrect keywords, incorrect case, and incorrect use of line space. It must conform exactly with the format given in the robots.txt paper -- if you want it to be accepted and correctly-interpreted by all robots, there is no wiggle-room whatsoever.

User-agent: *
Disallow:


Note that blank line at the end. A blank line must appear after each 'record' in robots.txt.

Jim

bilalseo

5+ Year Member



 
Msg#: 3639586 posted 10:03 am on May 6, 2008 (gmt 0)

Hey thanks for the help, can you please explain what is the right code or syntax for robots.txt file. Actually I want to give an open visit to all crawler, and it seems that after implementing new robots.txt file no bots taking visits. Even google bot had its last visit on 23rd April, 2008. Please help me out my site is loosing its position in different search engines.

Thanks,

Bilal

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved