The ELDB crawler

What it does

The ELDB crawler feeds the store database of Ecommerce Leads Database.

• It reads the home page to detect the platform, theme and apps.
• It reads the contact and legal notice pages, and the sitemaps.
• It reads the product catalog through the store's public feeds (for example products.json on Shopify, or the public WooCommerce API), up to 5,000 products.
• It comes back from time to time to check the data is still current.

What it collects

• Store data: platform, theme, apps, product count, public catalog (titles, prices, stock, brands).
• Business contact details published on the site: email, phone, address, social links. For each one, it notes the page it read it on and the date.
• A published email can name a person. It is then personal data, handled as our privacy policy describes.

It fills no forms, creates no accounts and accesses nothing behind a login.

How to recognise it

It sends the User-Agent header of desktop Chrome on Windows. We update the version about every three months. It does not announce itself under a crawler name. In your robots.txt, it applies the "User-agent: ecommerce-data-enrich" and "User-agent: ecommerce-data-scan" groups. Without those groups, it applies the "User-agent: *" group.

• It respects your robots.txt file, for every page and every redirect.
• It comes back to each store about every 30 days, to check the data is still current.
• Requests come from our servers and from those of technical service providers.
• If your site answers 403 or 429, the request is made once more through a technical service provider. If that works, the next requests to your site may go through it for 5 minutes. The same robots.txt rules apply.

How to block it

Add these lines to your robots.txt file. The crawler will no longer visit your site:

User-agent: ecommerce-data-scan
User-agent: ecommerce-data-enrich
Disallow: /

Blocking the crawler does not erase data already collected. For that, ask for your store's removal.

Remove your store from the database

You can remove your email address or your whole store. The removal takes effect as soon as you confirm it by email. Ask for removal.

A question? hello@ecommerce-leads-database.com