homepage Welcome to WebmasterWorld Guest from 54.204.142.143
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt and frontpage
sunflower12




msg:1525674
 3:48 pm on Aug 14, 2005 (gmt 0)

Hi,
I have frontpage. I do all of my html by hand but I do not have a robots.txt file. Can someone please tell me how to create this in frontpage step by step. LOL I do not want to make a mistake. I want to start with a basic one allowing all pages to be spidered. Thanks,

 

sokybiz




msg:1525675
 3:56 pm on Aug 14, 2005 (gmt 0)

Open FrontPage
Open your website
Selection "create new" > page
Save as robots.txt in the root
Switch to HTML view
erase all
enter your properly formatted robots directives
save
publish web

sunflower12




msg:1525676
 4:12 pm on Aug 14, 2005 (gmt 0)

"Open FrontPage
Open your website
Selection "create new" > page
Save as robots.txt in the root
Switch to HTML view
erase all
enter your properly formatted robots directives
save
publish web"

Thanks, so do I erase from <html>to </html> completely? Then what do I put there, if I want the entire site to be spidered now.

sunflower12




msg:1525677
 4:16 pm on Aug 14, 2005 (gmt 0)

OK, I went to wedtoolcentral and used their tool. Is this what I put in there word for word?

# All robots will spider the domain
User-agent: *
Disallow:

ScottM




msg:1525678
 4:19 pm on Aug 14, 2005 (gmt 0)

If you want the entire site to be spidered then you don't need a robots.txt

Yes, it will show up as an error in your logs, but that's about it.

sunflower12




msg:1525679
 4:24 pm on Aug 14, 2005 (gmt 0)

Hi Scott,
Are you sure about that? My site was banned on July 28. I have no idea why it was banned. It's been on the first page of google for many years for all it's search terms. It's not a link farm, it's not a directory and I do not use adsense. I have no idea why it was banned.

So the robots.txt has been mentioned as the possible problem. I recently changed from a shared ip to a dedicated ip hoping that may be the problem. Thanks

jbinbpt




msg:1525680
 4:47 pm on Aug 14, 2005 (gmt 0)

Scott is correct.

The robots.txt is for exclusion. If you don't have one, thats ok. No spiders will then be excluded.

That said you should have one because it's shows you pay attention to details. The only thing in your file should be is the following
User-agent: *
Disallow:

Have you validated your website html? That might show up problems? The banned question is a hard one. Lots of factors involved.
hth
jb

mack




msg:1525681
 4:58 pm on Aug 14, 2005 (gmt 0)

If you want your entire site to be sidered whilst avoiding the 404 logs from bots you can simply use a blank file as your robots.txt

Mack.

sunflower12




msg:1525682
 5:46 pm on Aug 14, 2005 (gmt 0)

I tried to save it as robots.txt but it will only allow me to save it as robots.txt.htm So I deleted it for now.

jbinbpt




msg:1525683
 7:26 pm on Aug 14, 2005 (gmt 0)

Sorry if we've confused you.
try opening it notepad, save it as robots.txt and then upload it.

mack




msg:1525684
 9:36 pm on Aug 14, 2005 (gmt 0)

jbinbpt is right. You can simply make the file in notepad. Alternativly you can simple go to your folders view (in frontpage) and change the full file name from robots.txt.htm to robots.txt

Mack

sunflower12




msg:1525685
 11:24 pm on Aug 14, 2005 (gmt 0)

"jbinbpt is right. You can simply make the file in notepad. Alternativly you can simple go to your folders view (in frontpage) and change the full file name from robots.txt.htm to robots.txt
Mack "

I tried to do that initially but it kept saving it in robots.txt.htm

mack




msg:1525686
 1:45 am on Aug 15, 2005 (gmt 0)

If you click on view and make sure the folder list is visible to the left hand side of the screen. You should be able ro right click the file and change it's entire name (including file extension).

If this doesnt work simply make a text file in notepad and call it robots.txt then copy paste it into your mywebs/sitename folder on your computer. It will then be visible as robots.txt within frontpage.

Mack.

Kursak




msg:1525687
 1:32 pm on Sep 4, 2005 (gmt 0)

Thank You this helped me out, :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved