homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Robots.txt during development
Anyone advise on this?

 2:42 am on Oct 30, 2003 (gmt 0)

I like to develop my sites online, thus I usually register domains and start the development.

Sometimes companies will only do a splash page or something similar, or I would like to do the work without having it indexed right away in the engines.

Are there any advantages to waiting until completion to have the site spidered? Any disadvantages?



 3:30 am on Oct 30, 2003 (gmt 0)

What I did was password protect my domains as I worked on them online. Robots.txt is only obeyed by good bots not humans or bad bots. IMO the latter two are more of a problem when you're trying to put things together. A friend of mine was using robots.txt only, and a human happened to stumble on his new site. The visitor thought the info was great and placed a link to his site on a busy forum. He was getting a lot of traffic before he was ever ready.

Advantage of pp: I didn't have to worry about checking logs every day to see if some bad bot/surfer wondered in and was taking the info as I placed it online. I didn't have to wrestle with the problem that an SE might pick up a splash page and not come back for the rest when it was ready. I was also able to easily identify several bad covert bots and write to their ISPs/hosts to make them go away.

Disadvantage of pp: Any bots that came by when my site was closed eventually went away. Some of the good bots would stop by every 4-6 weeks to check on it. So, when the sites went live I had to lure them back again. I did this by getting a couple high PR sites to deep link.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved