Welcome to WebmasterWorld Guest from 18.104.22.168
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|Disallow Slurp from all files except two
Yahoo likes only one page, so I'd like to just 'give' it that.
|2||tigertom||3:00 am Jan 20, 2007|
|Yahoo Slurp Bot misbehaving
.. or is it someone masquerading?
|4||tigertom||9:27 am Jan 17, 2007|
|Blocking Site in some countries
||3||eldinboy||5:55 pm Jan 15, 2007|
|What to do with duplicate pages
||3||olba||2:12 am Jan 12, 2007|
|Why should I have a robots.txt file?
Isn't it OK if I don't?
|23||beren||8:45 am Jan 11, 2007|
|Google Site Diagnostics - robots.txt blocks urls
||4||akreider||7:44 pm Jan 5, 2007|
|Robots txt to solve gallery duplicates
can someone confirm this is correct
|3||caines||10:54 am Jan 5, 2007|
|How many search engine understand crawl delay?
||1||foxfox||3:38 pm Jan 4, 2007|
|Multiple Domains Issue
||3||kdobson99||4:58 am Jan 4, 2007|
|Do spiders follow links from pages with the noindex tag?
As opposed to links from robots files?
|4||bouncybunny||6:59 am Jan 2, 2007|
|Directing Spiders to specific Servers
||2||TheSeoGuy||1:50 pm Dec 29, 2006|
|I had robots.txt blocking my phpbb folders which are now deleted
Should I remove that line from robots.txt now?
|2||cmendla||4:16 pm Dec 24, 2006|
|Disallowing Indexed files in Google
||5||leo11877||2:01 pm Dec 23, 2006|
|Disallowing by date.
Is it possible to keep some page and disallow new ones...?
|3||bouncybunny||8:31 am Dec 20, 2006|
|Worried about profile links
||4||Asia_Expat||3:53 pm Dec 19, 2006|
|Is it possible to exclude a URL by specifying an anchor?
|3||bouncybunny||12:40 am Dec 17, 2006|
|Removing Pages - Does Robots.txt Help?
||3||akreider||4:17 am Dec 16, 2006|
|Allowing only some pages for ia archive
||5||sosoo||8:20 am Dec 13, 2006|
|Mediapartners-Google fellow the robots.txt?
||2||AlphaWu||1:41 pm Dec 12, 2006|
|my robots file
|3||experienced||12:26 am Dec 12, 2006|
|Disagreement between Google and a validator
Robots.txt Checker says it's so, Google robots anayler says the other way
|6||Gusgsm||9:49 am Dec 11, 2006|
|Googlebot tripping bad bot script
||5||RobertRogers||5:29 pm Dec 10, 2006|
|Is this bad robots.txt
||1||scoobydooaustralia||8:02 pm Dec 8, 2006|
|duplicate pages .aspx?id=
Not sure what to do next
|2||schatzm||9:01 pm Dec 6, 2006|
|Is this a bot?
Unwanted visitor to my website
|2||brancook||8:57 pm Dec 6, 2006|