BumbleBee Garage Bot Stop
Help
Stop
The Bot - If you feel the need to stop
our bot, you can just add our bot to your
disallow list. We are using our most recent bot
SmartyBot. The SmartyBee.com search is done for
the moment while we regroup and try something
less ambitious and more manageable. We have
already given webmasters a chance to exclude
SmartyBot if they wanted to. So in all fairness
we should continue with the same bot and exclude
all unwilling participants. That way, we do not
unleash a new bot onto the battlefield.
To disallow
SmartyBot add the following lines to your robots.txt
file:
User-agent:
SmartyBot
Disallow: /
We plan to not
crawl more than 50 pages from any given site in
any single day. And unlike slurp or googlebot, we
are running from one location, one database, so
we can actually limit crawls to one per day. But
if you need to slow us down you can set a page
delay by adding:
Crawl-Delay:
10
10 being the
number of seconds you want the bot to wait until
it crawls the next page. We are respectful up to
30 seconds. If you set a delay over that time we
will consider it a complete disallow.
At the moment,
we are not reading meta tags. since we have to
crawl the page before we can find out if you want
it indexed, I just think that is stupid.
We never
really had many complaints as we built
squigglebot and smartybee mainly because we do
not abuse people servers. As a hosting company
and web developers, we resect peoples hardware
and hard work. As well as understanding the cost
of bandwidth and the dreded "Bandwidth Limit
exceeded" website closures due to bot
activity. We are taking every step possible so we
do not get grouped into that category of
malicious $*@%
Bots!
The ironic
part about declaring our bot as the site visitor
is that if we wanted to crawl your pages and
steal your suff, we would with out notice or
regard for your content or bandwidth. But we know
how bad the bot problem is and we don't want to
contribute to it.
We expect this
search to be the next big thing at least for the
automotive industry. With an openly accessible
free database, there will be thousands of sites
using our service and that ultimately means
traffic for the webmasters. You can read more
about our business plans or other products using the same
technology.
|
