homepage Welcome to WebmasterWorld Guest from 107.20.131.154
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Website
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Is this enough
dougie




msg:1525463
 9:57 am on Feb 2, 2004 (gmt 0)

Hi all,

All I have in my robots.txt file is the bit below - is it enough?

Dougie.

User-Agent: *
Disallow:

 

trillianjedi




msg:1525464
 11:07 am on Feb 2, 2004 (gmt 0)

is it enough?

What is it you want to do?

The robots.txt you have does nothing.

TJ

dougie




msg:1525465
 11:17 am on Feb 2, 2004 (gmt 0)

All I want is for every robot for every SE and every directory to visit my website.

trillianjedi




msg:1525466
 11:19 am on Feb 2, 2004 (gmt 0)

All I want is for every robot for every SE and every directory to visit my website.

You can safely just remove your robots.txt file completely in that case.

It's only used for listing search engine spiders that you do *not* want indexing your site.

TJ

sem4u




msg:1525467
 11:20 am on Feb 2, 2004 (gmt 0)

Well all the robots.txt file will do is to prevent certain spiders from visiting your website. At the moment you do allow all, but if you want to be in a directories you will have to submit to them :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved