Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: open
I have been having visits to my website from Fast PartnerSite Crawler, who have been looking for robots.txt
What is robots.txt, and what should I do?
All The Web also has a free submission here [alltheweb.com]. It takes time but it works.
The robots.txt file is explained in a lot of threads here. Here [webmasterworld.com]
To exclude all robots from the entire server
To allow all robots complete access
Or create an empty "/robots.txt" file.
There is lots of great community here. Look around, ask questions. There is always someone hanging out.
Unless you are specifically paying for SEO services from your Host, then i would'nt heed to much towards that approach.
If you want a quick inclusion then the PFI is a 48 hour inclusion via 'positiontech' obviously at cost.
Otherwise as the others have said, get reciprocated links in your industry, and get on the crawl/index cycle.
To get your subpages indexed, do what you would do with any index page - optimize it the best you can, and try to get some incoming links pointed at that subpaged with relevant anchor text.
But yeah, for the most part, I'm in the same boat as you are. All you can do is keep trying and be patient.
Does ATW have a paid option where I can give them URLs to spider?
ATW do offer PFI at $34 per url, but normally the bot would come and spider just the page that you have submitted. Many months will pass until he/she decides to spider and index the rest of the site.
At least this is what I have been observing for 6 months.
Don't even submit to them and they come crawling big time.
Whilst I have no proof, I am sure that ATW will have some kind of penalty for site URL's submitted repeatedly over a short period of time.
It is a must to invest in some dedicated SEO and wider e-marketing to promote your site effectively. I have only every used ATW's 'free' submission, and whilst it has been a bit patchy in listing the pages I want, they do get in - even if they flunctuate quite a lot.