Welcome to WebmasterWorld Guest from 54.167.65.217

Forum Moderated by: goodroi

Sitemaps, Meta Data, and robots.txt


Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.

 
Thread SubjectMessagesStarted byLast Message
will all robots crawl the sites without robot.txt?
robot crawl, robot.txt
2 johnlim 3:08 am Feb 1, 2004
What will Inktomi do in this scenario?
18 flobaby 6:02 am Jan 31, 2004
Google spidering past secure server
Robots.txt file to boot the bot?
2 xbase234 12:52 am Jan 31, 2004
Logs keeping showing 404 error
I have a robots file and it looks ok
3 eesnard 12:03 am Jan 30, 2004
need robots.txt for pages that are not linked?
robots.txt
5 rld3 5:09 pm Jan 29, 2004
SEO Robot.txt tag and robot meta tags... need advice
Help about robot.txt and robot meta tags
3 jamieabbott 2:09 pm Jan 28, 2004
Robots.txt returning "test"
Host has really messed me up
4 GodLikeLotus 10:37 am Jan 28, 2004
Need quick help
Prevent robots from clicking on PPC links
2 hostlead 11:11 am Jan 27, 2004
how to let robot crawl the links at a page but not index this page?
index, robot
2 johnlim 11:39 pm Jan 26, 2004
Wildcard use in robots.txt
Need to disallow spidering of graphics files
2 Probably 4:58 am Jan 26, 2004
robots.txt and htaccess
.htaccess circumvents to site root
4 stefan009 5:35 pm Jan 24, 2004
Altavista Scooter eats up too much trafic
2 maxidrom11 5:28 am Jan 24, 2004
Does every site need a robots.txt?
I have one, but I'm not sure I need it
2 MatthewHSE 4:00 pm Jan 23, 2004
creating robots.txt
5 artdog 12:21 pm Jan 23, 2004
My page has never been indexed
2 kknusa 6:45 am Jan 23, 2004
Subdomains
2 BBB_Guy 6:38 am Jan 23, 2004
Indexed pages that are disalowed by robots.txt
is this normal?
10 WebJoe 7:03 pm Jan 20, 2004
Google ignoring robots.txt
No real surprise, but a real pain
9 voice220 4:12 pm Jan 19, 2004
Is there any bots to block?
Else than the infamous "Web Content International"?
3 Daemon 11:56 pm Jan 16, 2004
Robots.txt help
5 mikemcs 1:04 pm Jan 16, 2004
Who are these spiders and what do they want?
7 obsos 4:51 pm Jan 14, 2004
Revisit instruction for robots
I want to specify different revisit instruction different sections
3 adityamodi 5:13 am Jan 10, 2004
Have I been blocking Altavista spider from spidering my site?
6 irock 6:10 pm Jan 9, 2004
robot.txt to remove an alias
Afraid of duplicated content
2 Ulkari 5:48 pm Jan 8, 2004
Spider Names
I need the names of some SEs usernames
3 The_Young_One 2:47 am Jan 8, 2004