new to robots

Forum Moderators: goodroi

Message Too Old, No Replies

new to robots

is this right?

sara2

6:14 pm on Apr 3, 2004 (gmt 0)

Hi, I have been reading a lot and added a robots.txt to my root directory yesterday snipped and it validates fine. Now, my question is, do I have it in the right spot?

and on every page of my site I have the meta tag

is this right?

In my root directory, there is one index.html file which is our site, but at the beginning of building it, this was a splash page (not any more) so I had the first page as index2.htm. I did not want to submit the splash page to search engines and such so I submitted the index2.htm page (which is indexed) and so now I have that page in my root directory also, along with one more page and some js files.

My main site is in a directory called page. Do I have this set right for the search engine or robots to crawl this site? Or do I need to change something?

Thanks!

[edited by: DaveAtIFG at 6:11 pm (utc) on April 5, 2004]
[edit reason] Url removed [/edit]

Symbios

6:26 pm on Apr 3, 2004 (gmt 0)

Looks ok to me, also I've never used the robots follow tag on my webpages as the crawlers follow without them.

PS. Thanks for the link to the midifiles, you may want to put a blank index.html in those folders.

sara2

6:30 pm on Apr 3, 2004 (gmt 0)

Hi, You're welcome for the midi's What do you mean about a blank index.html? And in what folders? Please explain in more detail if it is not too much trouble.

Symbios

6:33 pm on Apr 3, 2004 (gmt 0)

If you wanted to stop people looking at the content of the midi folder save a blank file as index.html and ftp it up to the midi folder, then when people find yoursitedotcom/midi/ they get presented with a blank page instead of a listing of the foleders contents.

sara2

6:43 pm on Apr 3, 2004 (gmt 0)

Ok, I think I get it,

and should I do this in the other directory's also that I would not want people to see?

Also
What about a doc tag

someone on another board told me I need a doc tag. Is that a doc tag above and do I need it?

Symbios

5:38 pm on Apr 5, 2004 (gmt 0)

Document headers are not essential, I use them sometimes but in my opinion they make no difference, a typical header would be;

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">

If you want to exclude crawlers modify your robots.txt file, if you don't have one create one in notepad, save it as robots.txt then ftp it up to your site, example below;

User-agent: *
Disallow: /tmp
Disallow: /logs

new to robots

is this right?

sara2

Symbios

sara2

Symbios

sara2

Symbios

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week