homepage Welcome to WebmasterWorld Guest from 54.211.97.242
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Marketing and Biz Dev / Deprecated - Search Engine Submission
Forum Library, Charter, Moderator: open

Deprecated - Search Engine Submission Forum

    
Spiders
do i need one
Andrew Thomas

10+ Year Member



 
Msg#: 345 posted 3:15 pm on Mar 12, 2002 (gmt 0)

Im totally confused on Spiders? Ive looked in the Forums, All im concerned about is getting as many SE to look at my site...

So do I need a robots.txt file, if so, is there a basic working template i can use...

thanks from someone who is totally confused

 

agerhart

WebmasterWorld Senior Member agerhart us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 345 posted 3:19 pm on Mar 12, 2002 (gmt 0)

[robotstxt.org...]

There is an example that you can use about half way down the page.....you need this for your site.

Andrew Thomas

10+ Year Member



 
Msg#: 345 posted 3:27 pm on Mar 12, 2002 (gmt 0)

thanks, but why would i want to Disallow, spiders from looking at my files, when I thought the idea was to promote sites as much as possable?

agerhart

WebmasterWorld Senior Member agerhart us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 345 posted 3:31 pm on Mar 12, 2002 (gmt 0)

you want to disallow certain spiders from certain section.....for instance, you don't want people to be able to access your stats, right? Use the "disallow" to keep spiders out of certain folders that you want to keep private.

Andrew Thomas

10+ Year Member



 
Msg#: 345 posted 3:35 pm on Mar 12, 2002 (gmt 0)

Oh thanks, sorry for sounding dumb... But i was looking at the forums and could only see people putting disalow on all folders and couldnt see why!

Apart from stats there isnt much else i want to protect.. All my images are in one folder, but with meaningfull names which hopefully SE may pick up as keywords, so should I leave this open for spiders to search though?

Ove

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 345 posted 3:42 pm on Mar 12, 2002 (gmt 0)

Ok Andrew

You dont wanna share your stats thats right.

But that folder who cares about that part does not be in the www folder in my case i got a folder called bin and it is ÷zldfjkgnao÷

i try again when i get in my ftp i have that bin folder and from there i have to go to the www do you mean i have to set a robot.txt to protect the bin folder.

Hope you get what i mean
/Ove

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Marketing and Biz Dev / Deprecated - Search Engine Submission
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved