Welcome to WebmasterWorld Guest from 54.163.35.238

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
Disallow Slurp from all files except two
Yahoo likes only one page, so I'd like to just 'give' it that.
2 tigertom 3:00 am Jan 20, 2007
Yahoo Slurp Bot misbehaving
.. or is it someone masquerading?
4 tigertom 9:27 am Jan 17, 2007
Blocking Site in some countries
3 eldinboy 5:55 pm Jan 15, 2007
What to do with duplicate pages
3 olba 2:12 am Jan 12, 2007
Why should I have a robots.txt file?
Isn't it OK if I don't?
23 beren 8:45 am Jan 11, 2007
Google Site Diagnostics - robots.txt blocks urls
4 akreider 7:44 pm Jan 5, 2007
Robots txt to solve gallery duplicates
can someone confirm this is correct
3 caines 10:54 am Jan 5, 2007
How many search engine understand crawl delay?
1 foxfox 3:38 pm Jan 4, 2007
Multiple Domains Issue
3 kdobson99 4:58 am Jan 4, 2007
Do spiders follow links from pages with the noindex tag?
As opposed to links from robots files?
4 bouncybunny 6:59 am Jan 2, 2007
Directing Spiders to specific Servers
2 TheSeoGuy 1:50 pm Dec 29, 2006
I had robots.txt blocking my phpbb folders which are now deleted
Should I remove that line from robots.txt now?
2 cmendla 4:16 pm Dec 24, 2006
Disallowing Indexed files in Google
5 leo11877 2:01 pm Dec 23, 2006
Disallowing by date.
Is it possible to keep some page and disallow new ones...?
3 bouncybunny 8:31 am Dec 20, 2006
Worried about profile links
4 Asia_Expat 3:53 pm Dec 19, 2006
Is it possible to exclude a URL by specifying an anchor?
Excluding #...?
3 bouncybunny 12:40 am Dec 17, 2006
Removing Pages - Does Robots.txt Help?
3 akreider 4:17 am Dec 16, 2006
Allowing only some pages for ia archive
5 sosoo 8:20 am Dec 13, 2006
Mediapartners-Google fellow the robots.txt?
2 AlphaWu 1:41 pm Dec 12, 2006
my robots file
need help
3 experienced 12:26 am Dec 12, 2006
Disagreement between Google and a validator
Robots.txt Checker says it's so, Google robots anayler says the other way
6 Gusgsm 9:49 am Dec 11, 2006
Googlebot tripping bad bot script
5 RobertRogers 5:29 pm Dec 10, 2006
Is this bad robots.txt
1 scoobydooaustralia 8:02 pm Dec 8, 2006
duplicate pages .aspx?id=
Not sure what to do next
2 schatzm 9:01 pm Dec 6, 2006
Is this a bot?
Unwanted visitor to my website
2 brancook 8:57 pm Dec 6, 2006

Featured Threads

Hot Threads This Week

Hot Threads This Month