homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Robots.txt during development
Anyone advise on this?

WebmasterWorld Senior Member 10+ Year Member

Msg#: 181 posted 2:42 am on Oct 30, 2003 (gmt 0)

I like to develop my sites online, thus I usually register domains and start the development.

Sometimes companies will only do a splash page or something similar, or I would like to do the work without having it indexed right away in the engines.

Are there any advantages to waiting until completion to have the site spidered? Any disadvantages?



10+ Year Member

Msg#: 181 posted 3:30 am on Oct 30, 2003 (gmt 0)

What I did was password protect my domains as I worked on them online. Robots.txt is only obeyed by good bots not humans or bad bots. IMO the latter two are more of a problem when you're trying to put things together. A friend of mine was using robots.txt only, and a human happened to stumble on his new site. The visitor thought the info was great and placed a link to his site on a busy forum. He was getting a lot of traffic before he was ever ready.

Advantage of pp: I didn't have to worry about checking logs every day to see if some bad bot/surfer wondered in and was taking the info as I placed it online. I didn't have to wrestle with the problem that an SE might pick up a splash page and not come back for the rest when it was ready. I was also able to easily identify several bad covert bots and write to their ISPs/hosts to make them go away.

Disadvantage of pp: Any bots that came by when my site was closed eventually went away. Some of the good bots would stop by every 4-6 weeks to check on it. So, when the sites went live I had to lure them back again. I did this by getting a couple high PR sites to deep link.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved