<p><strong>Crawler information</strong></p>
<h1>BoldDataBot and BoldDataFetcher</h1>
<p>Bold Data runs two web agents for Llamar, a Bold Data product. This page says what they do and how to block them.</p>
<h2>BoldDataBot</h2>
<p>BoldDataBot is Bold Data's crawler. It reads the public news sitemaps of a small number of approved publishers and fetches newly listed articles. Llamar uses them to publish short automatically generated summaries that link to the original article.</p>
<ul class="ul-disc">
<li>Agent string: <code>BoldDataBot/1.0 (+https://bolddata.org/bot)</code></li>
<li>robots.txt token: <code>BoldDataBot</code></li>
<li>Sitemaps: checked every 15 minutes with conditional requests.</li>
<li>Articles: at most 30 an hour per publisher.</li>
<li>robots.txt: obeyed for <code>BoldDataBot</code> and rechecked at least daily. It stops fetching anything that is disallowed.</li>
<li>It does not log in, use cookies, fetch paywalled content or fetch personal pages.</li>
</ul>
<h2>How to block BoldDataBot</h2>
<p>Add this to your robots.txt:</p>
<pre><code>User-agent: BoldDataBot
Disallow: /
</code></pre>
<h2>BoldDataFetcher</h2>
<p>BoldDataFetcher is a separate agent. It fetches one public page only when a Llamar reader asks for a summary of that page. It does not follow links.</p>
<ul class="ul-disc">
<li>Agent string: <code>BoldDataFetcher/1.0 (+https://bolddata.org/bot)</code></li>
</ul>
<p>Blocking BoldDataBot does not block BoldDataFetcher.</p>
<h2>Contact</h2>
<p>Questions or concerns about either agent: <a href="mailto:christian@bolddata.biz" target="_blank" rel="noopener noreferrer">christian@bolddata.biz</a>.</p>
