Welcome to WebmasterWorld Guest from 188.8.131.52
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|why use and what is robots.txt
this maybe stupid question
|3||mick_simon||4:45 pm May 7, 2008|
|How To stop crawler to crawl https:// pages
||8||vreelmistee1||6:03 am May 7, 2008|
|Status: 404(not found) in Google Webmaster Tools
||3||Boulder90||6:39 pm May 6, 2008|
|Y! and robots.txt
|5||Staffa||6:36 pm May 6, 2008|
|Googlebot is Not Taking Visit to my site
robots.txt is restricted googlebot to do not visit
|4||bilalseo||1:54 pm May 6, 2008|
|Site Pages are Blocked by Robots.txt
||4||bilalseo||10:03 am May 6, 2008|
|robots.txt and IP addresses
||11||Maciej_Ziemczonek||10:11 pm May 2, 2008|
|Is this valid? wildcard in robots
|3||rudyten||1:47 pm Apr 20, 2008|
|Will spiders follow links on URL's blocked by robots.txt
||3||deanw||6:34 pm Apr 16, 2008|
||4||pageoneresults||3:49 pm Apr 2, 2008|
|Dumbest question eva: Do all major SEs support the Google guidelines?
I am testing mine on G's WebmasterCentral
|2||walkman||6:42 pm Mar 26, 2008|
|Is there any way to block everyone except.
for Google, Yahoo etc?
|2||mmiller||9:41 am Mar 21, 2008|
|need to block. and don't know - pls help
need help with blocking - Malware Ref
|4||Reilly||7:55 pm Mar 17, 2008|
|How to block original + localized pages?
|1||vanguard2000||6:19 pm Mar 17, 2008|
|How long before bots re-visit after they were told they are not allowed?
||3||boromir27||3:32 am Mar 17, 2008|
|Inktomi User-Agent blocked
|6||cyberdyne||8:36 am Mar 14, 2008|
|robots.txt - Disallow/Allow
||7||eyalkattan||1:14 am Mar 13, 2008|
|robots and redirecting
i am wondering how a robot would act on a page with a redirect
|2||melanger||8:56 pm Mar 8, 2008|
|Unusual log entries
||5||cyberdyne||6:17 pm Mar 8, 2008|
|Redirect robot.txt from http to https
how to redirect properly.
|8||clyde4210||2:59 pm Mar 6, 2008|
|Uh oh, it's happening again
robots.txt blocked out
|2||wesg||9:58 pm Feb 28, 2008|
|new badbot: EnaBot/1.0
||7||Achernar||8:31 am Feb 28, 2008|
|Need help blocking search results page
Lots of keyword search pages indexed, some with no content
|4||ecmedia||2:28 pm Feb 26, 2008|
|disallow ALL bots.and then allow the good ones?
||5||tokey666||8:40 pm Feb 25, 2008|
|Excluding a stub page with robots.txt
Dealing with trailing forward slash
|4||bouncybunny||12:41 am Feb 24, 2008|