Welcome to WebmasterWorld Guest from 54.162.239.134

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
4 Wordpress File's Link To Be Blocked
How should I block them?
5 aced84 3:54 pm May 19, 2008
I think there is a problem
Robots problem
2 AllanS 10:39 am May 17, 2008
robots.txt syntax for wildcards?
3 Marfola 7:15 am May 15, 2008
How many links can i place in one page of site map?
5 bava_seo 2:12 pm May 10, 2008
why use and what is robots.txt
this maybe stupid question
3 mick_simon 4:45 pm May 7, 2008
How To stop crawler to crawl https:// pages
8 vreelmistee1 6:03 am May 7, 2008
Status: 404(not found) in Google Webmaster Tools
3 Boulder90 6:39 pm May 6, 2008
Y! and robots.txt
Partial ignore
5 Staffa 6:36 pm May 6, 2008
Googlebot is Not Taking Visit to my site
robots.txt is restricted googlebot to do not visit
4 bilalseo 1:54 pm May 6, 2008
Site Pages are Blocked by Robots.txt
4 bilalseo 10:03 am May 6, 2008
robots.txt and IP addresses
11 Maciej_Ziemczonek 10:11 pm May 2, 2008
Is this valid? wildcard in robots
Disallow: /*/edit.html
3 rudyten 1:47 pm Apr 20, 2008
Will spiders follow links on URL's blocked by robots.txt
3 deanw 6:34 pm Apr 16, 2008
Security Challenges
4 pageoneresults 3:49 pm Apr 2, 2008
Dumbest question eva: Do all major SEs support the Google guidelines?
I am testing mine on G's WebmasterCentral
2 walkman 6:42 pm Mar 26, 2008
Is there any way to block everyone except.
for Google, Yahoo etc?
2 mmiller 9:41 am Mar 21, 2008
need to block. and don't know - pls help
need help with blocking - Malware Ref
4 Reilly 7:55 pm Mar 17, 2008
How to block original + localized pages?
robots.txt localized
1 vanguard2000 6:19 pm Mar 17, 2008
How long before bots re-visit after they were told they are not allowed?
3 boromir27 3:32 am Mar 17, 2008
Inktomi User-Agent blocked
Inktomi blocked
6 cyberdyne 8:36 am Mar 14, 2008
robots.txt - Disallow/Allow
7 eyalkattan 1:14 am Mar 13, 2008
robots and redirecting
i am wondering how a robot would act on a page with a redirect
2 melanger 8:56 pm Mar 8, 2008
Unusual log entries
5 cyberdyne 6:17 pm Mar 8, 2008
Redirect robot.txt from http to https
how to redirect properly.
8 clyde4210 2:59 pm Mar 6, 2008
Uh oh, it's happening again
robots.txt blocked out
2 wesg 9:58 pm Feb 28, 2008

Featured Threads

Hot Threads This Week

Hot Threads This Month