Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt during development

Anyone advise on this?

2:42 am on Oct 30, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 22, 2002
votes: 1

I like to develop my sites online, thus I usually register domains and start the development.

Sometimes companies will only do a splash page or something similar, or I would like to do the work without having it indexed right away in the engines.

Are there any advantages to waiting until completion to have the site spidered? Any disadvantages?

3:30 am on Oct 30, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 11, 2003
votes: 0

What I did was password protect my domains as I worked on them online. Robots.txt is only obeyed by good bots not humans or bad bots. IMO the latter two are more of a problem when you're trying to put things together. A friend of mine was using robots.txt only, and a human happened to stumble on his new site. The visitor thought the info was great and placed a link to his site on a busy forum. He was getting a lot of traffic before he was ever ready.

Advantage of pp: I didn't have to worry about checking logs every day to see if some bad bot/surfer wondered in and was taking the info as I placed it online. I didn't have to wrestle with the problem that an SE might pick up a splash page and not come back for the rest when it was ready. I was also able to easily identify several bad covert bots and write to their ISPs/hosts to make them go away.

Disadvantage of pp: Any bots that came by when my site was closed eventually went away. Some of the good bots would stop by every 4-6 weeks to check on it. So, when the sites went live I had to lure them back again. I did this by getting a couple high PR sites to deep link.