By now it should be painfully obvious that crawling and indexing the web is a hard problem. Google and Yahoo have a lot of competent engineers working on this (praise be to Cutting), and you expected to do it in six months? One does not simply walk into Mordor.