The site reader.
When somebody adds a product, Billboard Badger reads that product's own site to set it up. It does not run yet: it arrives with set up from a URL.
What it does.
- Identifies as
BillboardBadger/1.0 (+https://billboardbadger.com/bot)- robots.txt
- Read first and obeyed. A site that asks not to be read is not read, and the person is told to fill the details in by hand
- How much
- Up to 30 pages of one site, at most 2 requests at a time, each given up after 10 seconds
- What it keeps
- The text of the pages it read, deleted after 30 days, and what the person approves from it
- What it never does
- Log in, fill in a form, follow a link off the site, or read a page behind a password
To block it, add this to the site's robots.txt:
User-agent: BillboardBadgerDisallow: /
Anything else about it goes to hello@billboardbadger.com.