About our crawler

BidBeforeBot/1.0 is the crawler BidBefore uses to read public council committee pages.

What it reads

Public committee pages on London council websites (committee lists, meeting lists, meeting pages and lists of forthcoming decisions), and published notices on Contracts Finder and Find a Tender.

How it behaves

• On weekdays it only fetches between 01:00 and 05:00 UK time. At weekends it may fetch at any time.
• It makes one request at a time per site and follows any Crawl-delay in your robots.txt.
• It obeys robots.txt rules addressed to BidBeforeBot. It doesn't apply rules addressed to every crawler (User-agent: *): many council sites serve their supplier's default file, which blocks all bots from papers published for the public.
• If your site refuses it, for example with a 403, it doesn't try to get round that.
• Where possible it only downloads pages that have changed since its last visit.
• It doesn't keep copies of the pages it reads.

Stopping it

Add a Disallow rule for BidBeforeBot to your robots.txt and it will stop on its next visit:

User-agent: BidBeforeBot
Disallow: /

Or email info@bidbefore.co.uk and we'll stop reading your site.