Crawler information

BoldDataBot and BoldDataFetcher

Bold Data runs two web agents for Llamar, a Bold Data product. This page says what they do and how to block them.

BoldDataBot

BoldDataBot is Bold Data's crawler. It reads fixed public news sitemaps or RSS feeds from a small number of selected publishers. It uses article links and publication dates to discover recent ordinary news articles, then fetches eligible articles. Llamar publishes short automatically generated summaries with source attribution and a link to the original article. Bold Data does not use these articles to train or fine-tune models.

• Agent string: BoldDataBot/1.0 (+https://bolddata.org/bot)
• robots.txt token: BoldDataBot
• Discovery inputs: eligible for a check every 15 minutes; actual timing depends on shared capacity. Conditional requests are used when the publisher supplies caching information.
• Indexing: automatic discovery starts work on at most 30 new articles an hour and 400 in a rolling 24-hour period across all publishers, with at most two articles being processed or waiting. These limits count indexing work; requests also include discovery feeds, robots.txt checks and article fetches.
• robots.txt: obeyed for BoldDataBot and rechecked at least daily. It stops fetching anything that is disallowed.
• It does not log in, use cookies, fetch paywalled content or fetch personal pages.

Where a summary includes a publisher image, the reader's browser loads it directly from the publisher. Llamar shows the available image credit alongside the source link.

How to block BoldDataBot

Add this to your robots.txt:

User-agent: BoldDataBot
Disallow: /

BoldDataFetcher

BoldDataFetcher is a separate agent. It fetches one public page only when a Llamar reader asks for a summary of that page. It does not follow links.

• Agent string: BoldDataFetcher/1.0 (+https://bolddata.org/bot)

Blocking BoldDataBot does not block BoldDataFetcher.

Contact

Questions or concerns about either agent: christian@bolddata.biz.