Welcome to WebmasterWorld Guest from 22.214.171.124
Using a robots.txt is all part of being a good SEO. Be sure to check yours in the robots.txt validator that is available to subscribers.
|Thread Subject||Messages||Started by||Last Message|
|will all robots crawl the sites without robot.txt?
robot crawl, robot.txt
|2||johnlim||3:08 am Feb 1, 2004|
|What will Inktomi do in this scenario?
||18||flobaby||6:02 am Jan 31, 2004|
|Google spidering past secure server
Robots.txt file to boot the bot?
|2||xbase234||12:52 am Jan 31, 2004|
|Logs keeping showing 404 error
I have a robots file and it looks ok
|3||eesnard||12:03 am Jan 30, 2004|
|need robots.txt for pages that are not linked?
|5||rld3||5:09 pm Jan 29, 2004|
|SEO Robot.txt tag and robot meta tags... need advice
Help about robot.txt and robot meta tags
|3||jamieabbott||2:09 pm Jan 28, 2004|
|Robots.txt returning "test"
Host has really messed me up
|4||GodLikeLotus||10:37 am Jan 28, 2004|
|Need quick help
Prevent robots from clicking on PPC links
|2||hostlead||11:11 am Jan 27, 2004|
|how to let robot crawl the links at a page but not index this page?
|2||johnlim||11:39 pm Jan 26, 2004|
|Wildcard use in robots.txt
Need to disallow spidering of graphics files
|2||Probably||4:58 am Jan 26, 2004|
|robots.txt and htaccess
.htaccess circumvents to site root
|4||stefan009||5:35 pm Jan 24, 2004|
|Altavista Scooter eats up too much trafic
||2||maxidrom11||5:28 am Jan 24, 2004|
|Does every site need a robots.txt?
I have one, but I'm not sure I need it
|2||MatthewHSE||4:00 pm Jan 23, 2004|
||5||artdog||12:21 pm Jan 23, 2004|
|My page has never been indexed
||2||kknusa||6:45 am Jan 23, 2004|
||2||BBB_Guy||6:38 am Jan 23, 2004|
|Indexed pages that are disalowed by robots.txt
is this normal?
|10||WebJoe||7:03 pm Jan 20, 2004|
|Google ignoring robots.txt
No real surprise, but a real pain
|9||voice220||4:12 pm Jan 19, 2004|
|Is there any bots to block?
Else than the infamous "Web Content International"?
|3||Daemon||11:56 pm Jan 16, 2004|
||5||mikemcs||1:04 pm Jan 16, 2004|
|Who are these spiders and what do they want?
||7||obsos||4:51 pm Jan 14, 2004|
|Revisit instruction for robots
I want to specify different revisit instruction different sections
|3||adityamodi||5:13 am Jan 10, 2004|
|Have I been blocking Altavista spider from spidering my site?
||6||irock||6:10 pm Jan 9, 2004|
|robot.txt to remove an alias
Afraid of duplicated content
|2||Ulkari||5:48 pm Jan 8, 2004|
I need the names of some SEs usernames
|3||The_Young_One||2:47 am Jan 8, 2004|