homepage Welcome to WebmasterWorld Guest from 54.145.172.149
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt - newbie needs help
geezer_uk

10+ Year Member



 
Msg#: 472 posted 12:02 am on Oct 21, 2004 (gmt 0)

hi people i'm new to this website and new to web site stuff

robts.txt

the text you have in how hould it be Formatted?

free
internet
music
design
software

:::OR:::

free,internet,music,design,software

:::OR:::

free ,internet ,music ,design ,software

or doesnt it matter how its formatted?

 

encyclo

WebmasterWorld Senior Member encyclo us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 472 posted 12:09 am on Oct 21, 2004 (gmt 0)

Welcome to WebmasterWorld [webmasterworld.com], geezer_uk.

I think you've misunderstood the role of the robots.txt file - it's not to define keywords, but to define which parts of your site can and cannot be spidered by search engine robots.

The best place to start is the following robots.txt tutorial, complete with examples:

[searchengineworld.com...]

geezer_uk

10+ Year Member



 
Msg#: 472 posted 1:05 am on Oct 21, 2004 (gmt 0)

I didnt know that, Soo it doesnt pick any keywords from robots.txt at all?

ive added at the very top:
User-agent: *
Disallow:

& ive uploaded it, when I view it from my website
User-agent: *
Disallow:

it isnt at the top where I saved it, any ideas?

so is there no point putting any keywords in it at all,

so how does robots.txt work?
and shall I put all keywords into to meta? will search engines crawl to my site?

PaulHudson

10+ Year Member



 
Msg#: 472 posted 5:20 am on Nov 10, 2004 (gmt 0)

<< Soo it doesnt pick any keywords from robots.txt at all? >>

- No, it is just a set of instructions for the bots to follow.

=================================================

<< ive added at the very top:
User-agent: *
Disallow:

it isnt at the top where I saved it, any ideas?>>

- not entirely sure what you mean by this, but to clarify, the robots.txt

"User-agent: *
Disallow:"

Should be in a text file in your website directory called robots.txt. Not written on the actual web page.

If you knew this already, then check your host allows it, I can’t imagine that they would not allow it, but I have been caught with my pants down for assuming things before.

Please clarify what you mean by “ive added at the very top” and ill try to answer.

====================================================

<< and shall I put all keywords into to meta? >>

- Yes, this is becoming redundant on some search engines, Google for example. But, even if meta tags are not as important as they once were, they are still used and every little helps. Read more about meta tags because you can use them to help give good descriptions of your site when displayed in search results. Also you never know when they will make a comeback.

====================================================

<< will search engines crawl to my site? >>

- 1, if there are inbound links to your site (links from other websites to yours) then search engines will follow those links and crawl your site.

2, you can ensure engines crawl your site by adding your URL (www.domainname.com) to the search engines. There is normally a link for this on the search engines web site, or you can search add site or something to that affect.

Read, read, read… this is a great place to get all this information, if I were you I would just read every thing you can on webmasterworld.

p.s hello all, great forum, definatly worth signing up for the paid membership just our of appreciation!

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 472 posted 5:39 am on Nov 10, 2004 (gmt 0)

Definitive source: A Standard for Robots Exclusion [robotstxt.org]
And some more information [robotstxt.org]

PaulHudson, welcome to WebmasterWorld!

Jim

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved