Welcome to WebmasterWorld Guest from 54.160.131.144

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Why should I have a robots.txt file?
Isn't it OK if I don't?
23 beren 8:45 am Jan 11, 2007
Google Site Diagnostics - robots.txt blocks urls
4 akreider 7:44 pm Jan 5, 2007
Robots txt to solve gallery duplicates
can someone confirm this is correct
3 caines 10:54 am Jan 5, 2007
How many search engine understand crawl delay?
1 foxfox 3:38 pm Jan 4, 2007
Multiple Domains Issue
3 kdobson99 4:58 am Jan 4, 2007
Do spiders follow links from pages with the noindex tag?
As opposed to links from robots files?
4 bouncybunny 6:59 am Jan 2, 2007
Directing Spiders to specific Servers
2 TheSeoGuy 1:50 pm Dec 29, 2006
I had robots.txt blocking my phpbb folders which are now deleted
Should I remove that line from robots.txt now?
2 cmendla 4:16 pm Dec 24, 2006
Disallowing Indexed files in Google
5 leo11877 2:01 pm Dec 23, 2006
Disallowing by date.
Is it possible to keep some page and disallow new ones...?
3 bouncybunny 8:31 am Dec 20, 2006
Worried about profile links
4 Asia_Expat 3:53 pm Dec 19, 2006
Is it possible to exclude a URL by specifying an anchor?
Excluding #...?
3 bouncybunny 12:40 am Dec 17, 2006
Removing Pages - Does Robots.txt Help?
3 akreider 4:17 am Dec 16, 2006
Allowing only some pages for ia archive
5 sosoo 8:20 am Dec 13, 2006
Mediapartners-Google fellow the robots.txt?
2 AlphaWu 1:41 pm Dec 12, 2006
my robots file
need help
3 experienced 12:26 am Dec 12, 2006
Disagreement between Google and a validator
Robots.txt Checker says it's so, Google robots anayler says the other way
6 Gusgsm 9:49 am Dec 11, 2006
Googlebot tripping bad bot script
5 RobertRogers 5:29 pm Dec 10, 2006
Is this bad robots.txt
1 scoobydooaustralia 8:02 pm Dec 8, 2006
duplicate pages .aspx?id=
Not sure what to do next
2 schatzm 9:01 pm Dec 6, 2006
Is this a bot?
Unwanted visitor to my website
2 brancook 8:57 pm Dec 6, 2006
Two domains mirror each other
Is it possible to a command to a robots.txt file to apply to just one?
4 Jeremy_H 9:08 pm Dec 5, 2006
Do I need to disallow files that aren't linked to?
for example, 'included' PHP & javascript pages
9 sssweb 3:34 pm Dec 4, 2006
removing permanent redirect - honoring content in page
removing permanent redirect
2 thu_household 2:56 pm Dec 4, 2006
Do we need robots at all? New member here
Confused as to whether I need to add robots txt
5 plenusvita 9:23 pm Dec 2, 2006