homepage Welcome to WebmasterWorld Guest from 54.234.128.25
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
SEO Robot.txt tag and robot meta tags... need advice
Help about robot.txt and robot meta tags
jamieabbott




msg:1528168
 5:15 pm on Jan 25, 2004 (gmt 0)

Hey Guys,

I own the site: www.example.co.uk and I'm setting it up for the Search Engines.

First question - should you ban spiders indexing contact forms and/or the thank you for submitting a form pages?

At the moment I'm using a robot.txt file, check the test page: www.example.co.uk/robots.txt

Should I use a robot.txt or robot meta tags or both to ban indexing of certian pages.

Final question - should you ban spiders from your image files?

Thanks

Jamie

 

Brett_Tabke




msg:1528169
 5:23 pm on Jan 25, 2004 (gmt 0)

First, welcome to WebmasterWorld.

>Should I use a robot.txt or robot meta tags

It's a toss up. Whichever is easier to setup and mantain. I think the real question is how many files you are talking?

> should you ban spiders from your image files?

I think so. However, there are some sites that seel product based on those images - so they would want to allow them.

Go2




msg:1528170
 2:09 pm on Jan 28, 2004 (gmt 0)

Should I use a robot.txt or robot meta tags or both to ban indexing of certian pages.

I agree with Brett, go with what is easiest to maintain. If you have many web pages on your site and the web pages have been organized in file directories, it may be easier to use robots.txt.

On a side note, if you choose to use both ways, and there is a conflict between the directive in robots.txt and the meta tag directive (e.g. if your robots.txt allows spidering but your page meta tag does not) the robots.txt directive has precedence.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved