sokybiz
msg:1525675 | 3:56 pm on Aug 14, 2005 (gmt 0) |
Open FrontPage
Open your website
Selection "create new" > page
Save as robots.txt in the root
Switch to HTML view
erase all
enter your properly formatted robots directives
save
publish web
|
sunflower12
msg:1525676 | 4:12 pm on Aug 14, 2005 (gmt 0) |
"Open FrontPage
Open your website
Selection "create new" > page
Save as robots.txt in the root
Switch to HTML view
erase all
enter your properly formatted robots directives
save
publish web" Thanks, so do I erase from <html>to </html> completely? Then what do I put there, if I want the entire site to be spidered now.
|
sunflower12
msg:1525677 | 4:16 pm on Aug 14, 2005 (gmt 0) |
OK, I went to wedtoolcentral and used their tool. Is this what I put in there word for word? # All robots will spider the domain
User-agent: *
Disallow:
|
ScottM
msg:1525678 | 4:19 pm on Aug 14, 2005 (gmt 0) |
If you want the entire site to be spidered then you don't need a robots.txt Yes, it will show up as an error in your logs, but that's about it.
|
sunflower12
msg:1525679 | 4:24 pm on Aug 14, 2005 (gmt 0) |
Hi Scott,
Are you sure about that? My site was banned on July 28. I have no idea why it was banned. It's been on the first page of google for many years for all it's search terms. It's not a link farm, it's not a directory and I do not use adsense. I have no idea why it was banned. So the robots.txt has been mentioned as the possible problem. I recently changed from a shared ip to a dedicated ip hoping that may be the problem. Thanks
|
jbinbpt
msg:1525680 | 4:47 pm on Aug 14, 2005 (gmt 0) |
Scott is correct. The robots.txt is for exclusion. If you don't have one, thats ok. No spiders will then be excluded. That said you should have one because it's shows you pay attention to details. The only thing in your file should be is the following Have you validated your website html? That might show up problems? The banned question is a hard one. Lots of factors involved.
hth
jb
|
mack
msg:1525681 | 4:58 pm on Aug 14, 2005 (gmt 0) |
If you want your entire site to be sidered whilst avoiding the 404 logs from bots you can simply use a blank file as your robots.txt Mack.
|
sunflower12
msg:1525682 | 5:46 pm on Aug 14, 2005 (gmt 0) |
I tried to save it as robots.txt but it will only allow me to save it as robots.txt.htm So I deleted it for now.
|
jbinbpt
msg:1525683 | 7:26 pm on Aug 14, 2005 (gmt 0) |
Sorry if we've confused you.
try opening it notepad, save it as robots.txt and then upload it.
|
mack
msg:1525684 | 9:36 pm on Aug 14, 2005 (gmt 0) |
jbinbpt is right. You can simply make the file in notepad. Alternativly you can simple go to your folders view (in frontpage) and change the full file name from robots.txt.htm to robots.txt Mack
|
sunflower12
msg:1525685 | 11:24 pm on Aug 14, 2005 (gmt 0) |
"jbinbpt is right. You can simply make the file in notepad. Alternativly you can simple go to your folders view (in frontpage) and change the full file name from robots.txt.htm to robots.txt
Mack " I tried to do that initially but it kept saving it in robots.txt.htm
|
mack
msg:1525686 | 1:45 am on Aug 15, 2005 (gmt 0) |
If you click on view and make sure the folder list is visible to the left hand side of the screen. You should be able ro right click the file and change it's entire name (including file extension). If this doesnt work simply make a text file in notepad and call it robots.txt then copy paste it into your mywebs/sitename folder on your computer. It will then be visible as robots.txt within frontpage. Mack.
|
Kursak
msg:1525687 | 1:32 pm on Sep 4, 2005 (gmt 0) |
Thank You this helped me out, :)
|
|
