Welcome to WebmasterWorld Guest from 54.145.209.34

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
why use and what is robots.txt
this maybe stupid question
3 mick_simon 4:45 pm May 7, 2008
How To stop crawler to crawl https:// pages
8 vreelmistee1 6:03 am May 7, 2008
Status: 404(not found) in Google Webmaster Tools
3 Boulder90 6:39 pm May 6, 2008
Y! and robots.txt
Partial ignore
5 Staffa 6:36 pm May 6, 2008
Googlebot is Not Taking Visit to my site
robots.txt is restricted googlebot to do not visit
4 bilalseo 1:54 pm May 6, 2008
Site Pages are Blocked by Robots.txt
4 bilalseo 10:03 am May 6, 2008
robots.txt and IP addresses
11 Maciej_Ziemczonek 10:11 pm May 2, 2008
Is this valid? wildcard in robots
Disallow: /*/edit.html
3 rudyten 1:47 pm Apr 20, 2008
Will spiders follow links on URL's blocked by robots.txt
3 deanw 6:34 pm Apr 16, 2008
Security Challenges
4 pageoneresults 3:49 pm Apr 2, 2008
Dumbest question eva: Do all major SEs support the Google guidelines?
I am testing mine on G's WebmasterCentral
2 walkman 6:42 pm Mar 26, 2008
Is there any way to block everyone except.
for Google, Yahoo etc?
2 mmiller 9:41 am Mar 21, 2008
need to block. and don't know - pls help
need help with blocking - Malware Ref
4 Reilly 7:55 pm Mar 17, 2008
How to block original + localized pages?
robots.txt localized
1 vanguard2000 6:19 pm Mar 17, 2008
How long before bots re-visit after they were told they are not allowed?
3 boromir27 3:32 am Mar 17, 2008
Inktomi User-Agent blocked
Inktomi blocked
6 cyberdyne 8:36 am Mar 14, 2008
robots.txt - Disallow/Allow
7 eyalkattan 1:14 am Mar 13, 2008
robots and redirecting
i am wondering how a robot would act on a page with a redirect
2 melanger 8:56 pm Mar 8, 2008
Unusual log entries
5 cyberdyne 6:17 pm Mar 8, 2008
Redirect robot.txt from http to https
how to redirect properly.
8 clyde4210 2:59 pm Mar 6, 2008
Uh oh, it's happening again
robots.txt blocked out
2 wesg 9:58 pm Feb 28, 2008
new badbot: EnaBot/1.0
7 Achernar 8:31 am Feb 28, 2008
Need help blocking search results page
Lots of keyword search pages indexed, some with no content
4 ecmedia 2:28 pm Feb 26, 2008
disallow ALL bots.and then allow the good ones?
5 tokey666 8:40 pm Feb 25, 2008
Excluding a stub page with robots.txt
Dealing with trailing forward slash
4 bouncybunny 12:41 am Feb 24, 2008

Featured Threads

Hot Threads This Week

Hot Threads This Month