After using robots.txt none of the bots are crawling my site! Help

robots.txt is not working properly

[shamims]

I have created a blog and using Wordpress on there. Just 1 page was crawled and I changed my theme. But I don't know why the robots.txt file was changed automatically. It looks like:

User-agent: *
Disallow: /


However i changed my robots.txt file as:
User-agent: *
Disallow:

It is the correct format of allowing all of the robots I think.

But the sad things is that......though i have added lots of pages and posts on my blog; none of the posts are indexing. It is worth-mentioning that i have changed this just 1 day before.

So my question is that how long will the Search Engines take time to acquainted of my changed robots.txt?
Is there any wrong else?

Help.

[Staffa]

Disallow means not allowing, blocking
* means everybody

so basically you are telling every bot NOT to access pages on your site.

If you want every bot to crawl your site just leave your robots.txt file empty.

[shamims]

@ Staffa. Thanks for your quick response.

Suppose I changed my robots.txt file according to your tips. Now when the Google or other search engines will be notified that I have changed robots file ( as my site is new and just 1 page has been indexed )?

[phranque]

so basically you are telling every bot NOT to access pages on your site.

not true!
the robots exclusion protocol uses the most specific rule and matches the pattern from left to right.
a blank pattern for Disallow means "disallow NOTHING".

from "The Web Robots Pages" [robotstxt.org] of the "official" REP site:

To allow all robots complete access

User-agent: *
Disallow:

(or just create an empty "/robots.txt" file, or don't use one at all)

you will find similar information on SE help pages regarding robots.txt syntax.

[tangor]

I would go with a whitelist of allowed bots, disallowing all others... Along the lines of:

# Whitelisted user-agents are allowed

User-agent: googlebot
Disallow: /cgi-bin

User-agent: msnbot
Disallow: /cgi-bin

User-agent: teoma
Disallow: /cgi-bin

User-agent: slurp
Disallow: /cgi-bin

User-agent: atomz
Disallow: /cgi-bin

# Disallow all others
User-agent: *
Disallow: /

[penders]

I would go with a whitelist of allowed bots, disallowing all others...

So you allow very few robots? What about all the other 'good bots'? - or do they not count anything towards site promotion?

[tangor]

That is correct. I allow very few bots as most of the others do little toward site promotion. It also allows me to keep my last 13 hairs since I am not tempted to pull them out when things do not go well. :)

(and teoma is on a short list of things to get rid of...)

But that's just me!

Obviously one can whitelist what they like. The above is my list... and one of them is for an onsite search box that keeps the visitor on my site...

[jameswsparker]

If you want Google to start indexing your site again, then add it into:

[google.co.uk...]

It'll normally start crawling it within the next few days, and like always takes up to a month to process it in it's search results.

[martinibuster]

phranque is correct. Re-read the original post. The OP states they changed the Robots.txt to the version that allows bots to access the site after they initially displayed the wrong version. But that's not what is under discussion, despite what the title of this discussion shows, which is the source of the confusion. Here is what the discussion is about:

So my question is that how long will the Search Engines take time to acquainted of my changed robots.txt?

This is why it's important to accurately describe what the discussion is about. ;)

As for the answer, keep building links. Bots revisit a site according to how many links you have and how worth crawling they are. Robots.txt will not slow down how often a bot visitors your site. So once a bot follows a link to find your site they will see the correct robots.txt and proceed to index your site.

[goodroi]

Google and the other search engines strongly prefer to crawl based on links and not url submissions. I would not spend any time submitting a url to a search engine. Even if a mistake has been made you will be automatically crawled by the search engines as long as you have links pointing to the content.