Forum Moderators: open
Please excuse my basic questions. I'm still learning and hope to contribute more when I truly know what I'm doing,
but for now I can only help by asking good questions.
I ran into something on a "Promotion Tips" site that I'd
never heard before. -Is it really true that some engines won't crawl your whole site without a robots.txt?
Following is the excerpt I read -------------------------
"The robots.txt file tells web spiders what to index and what not to. While most robots will index your page if you don't have a robot.txt file, they won't index your whole site without it!
Adding a robots.txt file to your webserver will increase the robot activity on your site and thus bring you more hits.
If you don't already have a robots.txt file on your server, then upload this file to the main directory of your site. You need a seperate robots.txt file for each one of your domains.
Copy and paste the code below and put it into a text file called robots.txt
# /robots.txt for [yourdomain.com...]
# comments to webmaster@yourdomain.com/
User-agent: *
Disallow:
Allow: *
------------------------------------------------------
I'm confused! It seems to be the "allow" tag that makes the difference, but what if I need to disallow one folder?
Would "allow" still work to make the spider crawl the rest of the pages?
and what about this stuff:
# /robots.txt for [yourdomain.com...]
# comments to webmaster@yourdomain.com/
Is that really necessary? It's commented out anyway,
so I don't see why it's needed. Is there some benefit
to having it there?
Could someone please explain the robots tag options,
and the RIGHT ways to use them for SEO?
Thanks in advance,
Chuck West
