homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Question about robots meta tag, as well as robots.txt....

 11:49 pm on May 19, 2004 (gmt 0)

Hello guys, i think i have the robots.txt setup right... but i basically dont want the robots to index ANYTHING in the /images/ folder...
sooo i have:

User-agent: *
Disallow: /images

i believe that will stop them from indexing anything inside of the /images directory? also, could i simply put: Disallow: /images /adminstuff to have it block both directorys?

as for the robots meta tag...

i have a header.php, and obviously all of my index.php files call it, now... would it be redundant or would it cause any harm to have:

<META name="ROBOTS" content="INDEX, FOLLOW">

being that every time it hits a index.php, it would index that, as well as follow to index2.php, which it would them index and follow back to index.php being every page I have also has a link back to the main index.php?

if it sounds confusing let me know and ill try and explain it better.

i thank you for your time.



 11:22 am on May 20, 2004 (gmt 0)



 11:48 am on May 20, 2004 (gmt 0)

Personally I'd use the following, just to ensure that only the directory was being disallowed (trailing slash). If you want to add a second disallowed item then create yourself a second entry;

User-agent: *
Disallow: /images/
Disallow: /adminstuff/

As for the meta tag, I'd consider "index, follow" as redundant because that's the default behaviour so you could say nothing and achieve the same effect - but if you want to keep it there feel free because the only harm it will do is add an extra line to your source code.

- Tony

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved