Welcome to WebmasterWorld Guest from 54.162.203.39

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt - newbie needs help

     
12:02 am on Oct 21, 2004 (gmt 0)

New User

10+ Year Member

joined:Oct 20, 2004
posts:7
votes: 0


hi people i'm new to this website and new to web site stuff

robts.txt

the text you have in how hould it be Formatted?

free
internet
music
design
software

:::OR:::

free,internet,music,design,software

:::OR:::

free ,internet ,music ,design ,software

or doesnt it matter how its formatted?

12:09 am on Oct 21, 2004 (gmt 0)

Senior Member from CA 

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 31, 2003
posts:9074
votes: 6


Welcome to WebmasterWorld [webmasterworld.com], geezer_uk.

I think you've misunderstood the role of the robots.txt file - it's not to define keywords, but to define which parts of your site can and cannot be spidered by search engine robots.

The best place to start is the following robots.txt tutorial, complete with examples:

[searchengineworld.com...]

1:05 am on Oct 21, 2004 (gmt 0)

New User

10+ Year Member

joined:Oct 20, 2004
posts:7
votes: 0


I didnt know that, Soo it doesnt pick any keywords from robots.txt at all?

ive added at the very top:
User-agent: *
Disallow:

& ive uploaded it, when I view it from my website
User-agent: *
Disallow:

it isnt at the top where I saved it, any ideas?

so is there no point putting any keywords in it at all,

so how does robots.txt work?
and shall I put all keywords into to meta? will search engines crawl to my site?

5:20 am on Nov 10, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 9, 2004
posts:67
votes: 0


<< Soo it doesnt pick any keywords from robots.txt at all? >>

- No, it is just a set of instructions for the bots to follow.

=================================================

<< ive added at the very top:
User-agent: *
Disallow:

it isnt at the top where I saved it, any ideas?>>

- not entirely sure what you mean by this, but to clarify, the robots.txt

"User-agent: *
Disallow:"

Should be in a text file in your website directory called robots.txt. Not written on the actual web page.

If you knew this already, then check your host allows it, I can’t imagine that they would not allow it, but I have been caught with my pants down for assuming things before.

Please clarify what you mean by “ive added at the very top” and ill try to answer.

====================================================

<< and shall I put all keywords into to meta? >>

- Yes, this is becoming redundant on some search engines, Google for example. But, even if meta tags are not as important as they once were, they are still used and every little helps. Read more about meta tags because you can use them to help give good descriptions of your site when displayed in search results. Also you never know when they will make a comeback.

====================================================

<< will search engines crawl to my site? >>

- 1, if there are inbound links to your site (links from other websites to yours) then search engines will follow those links and crawl your site.

2, you can ensure engines crawl your site by adding your URL (www.domainname.com) to the search engines. There is normally a link for this on the search engines web site, or you can search add site or something to that affect.

Read, read, read… this is a great place to get all this information, if I were you I would just read every thing you can on webmasterworld.

p.s hello all, great forum, definatly worth signing up for the paid membership just our of appreciation!

5:39 am on Nov 10, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


Definitive source: A Standard for Robots Exclusion [robotstxt.org]
And some more information [robotstxt.org]

PaulHudson, welcome to WebmasterWorld!

Jim

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members