Welcome to WebmasterWorld Guest from 23.22.46.195

Forum Moderators: goodroi

Robots.txt and frontpage

   
3:48 pm on Aug 14, 2005 (gmt 0)

5+ Year Member



Hi,
I have frontpage. I do all of my html by hand but I do not have a robots.txt file. Can someone please tell me how to create this in frontpage step by step. LOL I do not want to make a mistake. I want to start with a basic one allowing all pages to be spidered. Thanks,
3:56 pm on Aug 14, 2005 (gmt 0)

5+ Year Member



Open FrontPage
Open your website
Selection "create new" > page
Save as robots.txt in the root
Switch to HTML view
erase all
enter your properly formatted robots directives
save
publish web
4:12 pm on Aug 14, 2005 (gmt 0)

5+ Year Member



"Open FrontPage
Open your website
Selection "create new" > page
Save as robots.txt in the root
Switch to HTML view
erase all
enter your properly formatted robots directives
save
publish web"

Thanks, so do I erase from <html>to </html> completely? Then what do I put there, if I want the entire site to be spidered now.

4:16 pm on Aug 14, 2005 (gmt 0)

5+ Year Member



OK, I went to wedtoolcentral and used their tool. Is this what I put in there word for word?

# All robots will spider the domain
User-agent: *
Disallow:

4:19 pm on Aug 14, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If you want the entire site to be spidered then you don't need a robots.txt

Yes, it will show up as an error in your logs, but that's about it.

4:24 pm on Aug 14, 2005 (gmt 0)

5+ Year Member



Hi Scott,
Are you sure about that? My site was banned on July 28. I have no idea why it was banned. It's been on the first page of google for many years for all it's search terms. It's not a link farm, it's not a directory and I do not use adsense. I have no idea why it was banned.

So the robots.txt has been mentioned as the possible problem. I recently changed from a shared ip to a dedicated ip hoping that may be the problem. Thanks

4:47 pm on Aug 14, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Scott is correct.

The robots.txt is for exclusion. If you don't have one, thats ok. No spiders will then be excluded.

That said you should have one because it's shows you pay attention to details. The only thing in your file should be is the following

User-agent: *
Disallow:

Have you validated your website html? That might show up problems? The banned question is a hard one. Lots of factors involved.
hth
jb

4:58 pm on Aug 14, 2005 (gmt 0)

WebmasterWorld Administrator mack is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



If you want your entire site to be sidered whilst avoiding the 404 logs from bots you can simply use a blank file as your robots.txt

Mack.

5:46 pm on Aug 14, 2005 (gmt 0)

5+ Year Member



I tried to save it as robots.txt but it will only allow me to save it as robots.txt.htm So I deleted it for now.
7:26 pm on Aug 14, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Sorry if we've confused you.
try opening it notepad, save it as robots.txt and then upload it.
9:36 pm on Aug 14, 2005 (gmt 0)

WebmasterWorld Administrator mack is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



jbinbpt is right. You can simply make the file in notepad. Alternativly you can simple go to your folders view (in frontpage) and change the full file name from robots.txt.htm to robots.txt

Mack

11:24 pm on Aug 14, 2005 (gmt 0)

5+ Year Member



"jbinbpt is right. You can simply make the file in notepad. Alternativly you can simple go to your folders view (in frontpage) and change the full file name from robots.txt.htm to robots.txt
Mack "

I tried to do that initially but it kept saving it in robots.txt.htm

1:45 am on Aug 15, 2005 (gmt 0)

WebmasterWorld Administrator mack is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



If you click on view and make sure the folder list is visible to the left hand side of the screen. You should be able ro right click the file and change it's entire name (including file extension).

If this doesnt work simply make a text file in notepad and call it robots.txt then copy paste it into your mywebs/sitename folder on your computer. It will then be visible as robots.txt within frontpage.

Mack.

1:32 pm on Sep 4, 2005 (gmt 0)

5+ Year Member



Thank You this helped me out, :)
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month