Skip to content
Selected work

Data & LLM pipeline

50,000 artworks aggregated, cleaned and published for $5.51.

Client
Art auction marketplace
Sector
E-commerce · art market
Duration
May – June 2026

The problem

A marketplace specialising in art auctions wanted to aggregate catalogues from several external sources. Each had its own structure, duplicates, missing fields and a different anti-bot posture. Manual cleaning at that scale was unthinkable — and AI cleaning at list price would have cost more than the value it added.

The architecture

  1. 01

    Collection

    protected sources, resumable

  2. 02

    GPT-4o-mini

    idempotent cleaning

  3. 03

    Image pipeline

    8 formats per artwork

  4. 04

    GPT-4o vision

    alt texts + biographies

  5. 05

    EC2 + RDS

    storage and monitoring

The real cost centre wasn't collection, it was inference. By batching records, splitting prompts by task and dropping to a lighter model wherever quality allowed, cleaning 64,602 records came to $5.51 in total. An idempotency marker guarantees an already-processed row is never paid for twice, even if the pipeline is re-run — that detail is what turns a runaway cost into a bounded one.

Collecting from protected sources

The sources weren't open: subscription systems, enterprise-grade anti-bot protection, and for some a sale lifecycle to track over time rather than a simple snapshot. Each collector manages its own checkpoints: an interrupted run restarts from the last confirmed batch, never from zero.

  • Sale lifecycle tracked from pending through to completed
  • Checkpoints on every batch, monitoring and failure logging
  • No source is named publicly, out of respect for the contractual scope

Making LLM cleaning economically viable

Cleaning titles, classifying mediums, parsing dimensions and signatures: trivial one at a time, unmanageable across 64,602. The pipeline batches records, applies task-specific prompts, and drops to the lightest model that still holds the required quality. Result: $5.51 of tokens in total, with an idempotency marker preventing any budget from being spent twice.

An image pipeline built for SEO

Storage migrated to shared hosting, then eight formats generated per artwork in AVIF and WebP — 400px, 800px, 1200px, thumbnail and original — with dynamic search-oriented naming such as artist-title_800w.avif. A taxonomy prefix bug was fixed along the way: the prefix now applies only to the correct nomenclature category, not the whole catalogue.

Editorial enrichment at scale

253,000 alt texts generated by a vision model, around 3,000 a night on a slot scheduled at 2am UTC so it never competes with real traffic. 11,301 artist biographies rewritten for search across five languages: French, English, German, Italian, Spanish. JSON-LD Person and Artist structures, semantic HTML5, mobile accordions.

A handover you can pick up without me

Everything shipped to GitHub with a README, an operations runbook and the database schema. Six bugs were found and fixed before the final push. That's the part of the work nobody sees in a demo but which decides whether the client is independent or dependent.

Stack

  • Python
  • AWS EC2
  • AWS RDS MySQL
  • GPT-4o-mini
  • GPT-4o vision
  • ScrapFly
  • Playwright
  • BeautifulSoup
  • JSON-LD

The outcome

  • 50,000

    artworks aggregated and published

  • 64,602

    records cleaned by AI for $5.51 in total

  • 253,000

    alt texts generated, around 3,000 a night

  • 11,301

    biographies rewritten for SEO across five languages

Need something similar?

I read every message myself and reply within 24 hours. No automated quote, no form disappearing into a void.

Start a project

Other work