Robots.txt and frontpage

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt and frontpage

sunflower12

3:48 pm on Aug 14, 2005 (gmt 0)

Hi,
I have frontpage. I do all of my html by hand but I do not have a robots.txt file. Can someone please tell me how to create this in frontpage step by step. LOL I do not want to make a mistake. I want to start with a basic one allowing all pages to be spidered. Thanks,

sokybiz

3:56 pm on Aug 14, 2005 (gmt 0)

Open FrontPage
Open your website
Selection "create new" > page
Save as robots.txt in the root
Switch to HTML view
erase all
enter your properly formatted robots directives
save
publish web

sunflower12

4:12 pm on Aug 14, 2005 (gmt 0)

"Open FrontPage
Open your website
Selection "create new" > page
Save as robots.txt in the root
Switch to HTML view
erase all
enter your properly formatted robots directives
save
publish web"

Thanks, so do I erase from <html>to </html> completely? Then what do I put there, if I want the entire site to be spidered now.

sunflower12

4:16 pm on Aug 14, 2005 (gmt 0)

OK, I went to wedtoolcentral and used their tool. Is this what I put in there word for word?

# All robots will spider the domain
User-agent: *
Disallow:

ScottM

4:19 pm on Aug 14, 2005 (gmt 0)

If you want the entire site to be spidered then you don't need a robots.txt

Yes, it will show up as an error in your logs, but that's about it.

sunflower12

4:24 pm on Aug 14, 2005 (gmt 0)

Hi Scott,
Are you sure about that? My site was banned on July 28. I have no idea why it was banned. It's been on the first page of google for many years for all it's search terms. It's not a link farm, it's not a directory and I do not use adsense. I have no idea why it was banned.

So the robots.txt has been mentioned as the possible problem. I recently changed from a shared ip to a dedicated ip hoping that may be the problem. Thanks

jbinbpt

4:47 pm on Aug 14, 2005 (gmt 0)

Scott is correct.

The robots.txt is for exclusion. If you don't have one, thats ok. No spiders will then be excluded.

That said you should have one because it's shows you pay attention to details. The only thing in your file should be is the following

User-agent: *
Disallow:

Have you validated your website html? That might show up problems? The banned question is a hard one. Lots of factors involved.
hth
jb

mack

4:58 pm on Aug 14, 2005 (gmt 0)

If you want your entire site to be sidered whilst avoiding the 404 logs from bots you can simply use a blank file as your robots.txt

Mack.

sunflower12

5:46 pm on Aug 14, 2005 (gmt 0)

I tried to save it as robots.txt but it will only allow me to save it as robots.txt.htm So I deleted it for now.

jbinbpt

7:26 pm on Aug 14, 2005 (gmt 0)

Sorry if we've confused you.
try opening it notepad, save it as robots.txt and then upload it.

mack

9:36 pm on Aug 14, 2005 (gmt 0)

jbinbpt is right. You can simply make the file in notepad. Alternativly you can simple go to your folders view (in frontpage) and change the full file name from robots.txt.htm to robots.txt

Mack

sunflower12

11:24 pm on Aug 14, 2005 (gmt 0)

"jbinbpt is right. You can simply make the file in notepad. Alternativly you can simple go to your folders view (in frontpage) and change the full file name from robots.txt.htm to robots.txt
Mack "

I tried to do that initially but it kept saving it in robots.txt.htm

mack

1:45 am on Aug 15, 2005 (gmt 0)

If you click on view and make sure the folder list is visible to the left hand side of the screen. You should be able ro right click the file and change it's entire name (including file extension).

If this doesnt work simply make a text file in notepad and call it robots.txt then copy paste it into your mywebs/sitename folder on your computer. It will then be visible as robots.txt within frontpage.

Mack.

Kursak

1:32 pm on Sep 4, 2005 (gmt 0)

Thank You this helped me out, :)