Skip to content
Tile Deal HubUK tile sales and clearance

Crawler

Tile Deal Hub reads public sale pages. It does not log in, fill baskets, or bypass access controls.

Identity

User-Agent: TileDealHubBot (+https://www.tiledealhub.co.uk/bot)

What it requests

  • robots.txt for the host, before any product URL.
  • Shopify collection JSON such as /collections/sale/products.json.
  • Public sale or clearance HTML, to read JSON-LD, microdata, or the embedded catalogue payload.
  • Individual product pages only when the listing has names and URLs but no prices, and only up to a fixed cap.

Politeness

  • Paths disallowed for this user agent are not fetched.
  • At least one second between requests to the same host, or the crawl-delay in robots.txt if that is longer.
  • Each source at most once every 20 hours.
  • The daily Vercel cron refreshes at most one source and skips retailers that need a product-by-product walk, so a short function budget still stays polite.

What it stores

Name, URL, image URL, prices, size, material classification, and the time of the fetch. Not the description HTML.

Opt out

Disallow TileDealHubBot in robots.txt, or email hello@tiledealhub.co.uk with the host to remove. A source can also be deleted from src/config/retailers.ts.