MMO auction house price tracker public web httpx

A trader's notebook used to be a physical thing. Mine was a spiral-bound one next to the keyboard, and every Tuesday and Friday I'd flip through the auction house and scribble down which mats had drifted since the last check-in. By the third month it felt less like play and more like a second job. The swing I actually wanted to catch was already three days old by the time I logged it. This article is what happened when I finally decided the notebook could keep itself.

Every serious MMO trader keeps a notebook of some shape. The market shifts overnight, the same crafting mat that sold for 3g on Tuesday goes for 5g by Friday, and the only way to spot the swing is a paper trail. Reading the auction house in the client works fine for one snapshot, but doing it every six hours by hand is the kind of grind that turns a hobby into a chore. The game already publishes the same data on its official public web view, so the notebook can be automated without ever touching the client.

Here I'll walk through the small Python service I built to replace mine. It fetches the publisher's official auction page, parses the listings out of the raw HTML, and writes each fetch into SQLite as a versioned snapshot. Everything is publisher-permitted: the tracker reads exactly the same URL a browser would open, uses a descriptive User-Agent so the WAF can identify it, and touches the page at most four times per day per shard. There's no client injection, no packet capture, no API key, no reverse-engineered protocol. When we're done you'll have a companion repository at vytharion/mmo-auction-price-tracker-public-web-httpx you can clone, extend, and point at any publisher web view that lists prices in HTML.

The stack stays intentionally small. httpx gives us an async HTTP client with HTTP/2, redirects, and per-request timeouts. selectolax parses HTML using the Modest engine, which handles messy publisher markup an order of magnitude faster than lxml and never breaks on partial documents. sqlite3 from the standard library gives us a single-file store with WAL journaling, which means the tracker can read prices while a fetch is writing. typer wraps it all in a friendly CLI. The whole tree fits into one uv workspace and the test suite runs offline via pytest-httpx.

Lesson 1: fetch the page with httpx (commit 53cff33)

What does a well-behaved fetcher actually look like to a publisher's WAF? Three httpx settings and a User-Agent that names itself, and they matter more than the client.get call around them. The client.py module opens an httpx.AsyncClient with three settings that matter: a descriptive User-Agent that names the tracker and links back to the public repo, a hard Timeout on both connect and read, and follow_redirects=True so the fetcher survives the publisher moving auction pages between shards.

async with httpx.AsyncClient(
    http2=True,
    follow_redirects=True,
    timeout=httpx.Timeout(timeout_s, connect=min(5.0, timeout_s)),
    headers={
        "User-Agent": user_agent,
        "Accept": "text/html,application/xhtml+xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.7",
    },
) as client:
    response = await client.get(url)

Every failure path collapses into a single FetchError. The reason is upstream. The scheduler in Lesson 4 needs one exception type to decide whether to skip the tick or retry. Transport errors, 4xx status, 5xx status, and timeouts all raise FetchError with a short reason string. The public docs for httpx at https://www.python-httpx.org/ cover the timeout model in detail if you want to tune connect versus read independently. For most publisher web views a flat 15s ceiling with a 5s connect works fine, since the page itself is only a few hundred kilobytes of HTML.

Tests use pytest-httpx to inject responses without touching the network. That gives us three fast assertions: a 2xx returns the body, a 5xx raises FetchError, and the polite User-Agent reaches the wire. The whole test file runs in under 100 milliseconds on my laptop, which matters because CI is where flaky fetch code goes to die.

Lesson 2: parse listings with selectolax (commit 89e43ea)

The first parser I wrote broke two weeks in, when the publisher swapped one <td> for a <div> between patches; the one I ended up shipping doesn't care, and this section is a walk through why. Rows sit inside a <table>, prices are in the game's compound format (3g 50s 12c), and half the columns are empty on trade-blocked items. The parser has to be tolerant without silently dropping legitimate rows. selectolax.parser.HTMLParser handles the first half by parsing at Modest's speed and being forgiving about malformed close tags. A regex normaliser handles the second half.

_PRICE_RE = re.compile(
    r"(?:(?P<g>\d+)\s*g)?\s*(?:(?P<s>\d+)\s*s)?\s*(?:(?P<c>\d+)\s*c)?",
    re.IGNORECASE,
)

def parse_price_to_copper(text: str) -> int:
    match = _PRICE_RE.search(text.strip())
    if not match:
        return 0
    gold = int(match.group("g") or 0)
    silver = int(match.group("s") or 0)
    copper = int(match.group("c") or 0)
    return gold * 10_000 + silver * 100 + copper

Normalising every price to a single copper integer buys us two things. Analytics stay trivial, since median price is a plain SELECT in Lesson 3's schema, not a compound-format sort. And the data stays cheap: SQLite stores integers in one byte when they fit, so a year of ticks across the top 200 items comes out to a few dozen megabytes at most.

The parser skips rows where the data-item-id is empty or the quantity isn't a digit. That drops the header row automatically and also drops junk from mid-page ad banners some shards inject. Tests read a fixture HTML file from disk, run the parser, and assert both the row count and the specific normalised copper amounts for the Iron Ore listings. When the publisher changes their markup (and they do, every couple of patches), the fixture is the first thing you edit, so the parser stays a pure function.

If you're wondering why not BeautifulSoup: the selectolax README at https://github.com/rushter/selectolax shows a benchmark with 10x parsing speed on real-world HTML pages. For four fetches a day it doesn't matter, but for a batch job that reindexes 500 back-fetched pages during a schema migration, the difference is minutes versus an hour.

Lesson 3: persist snapshots atomically in SQLite (commit 31923df)

Why would a tracker that writes to disk four times a day need transactions and write-ahead logging? Because the first time you check prices from your phone at 6:01am, the scheduler is still finishing its 6:00am write, and the read has to see a consistent snapshot or nothing at all. Losing the child listings but keeping the parent snapshot would poison every downstream query, so the storage layer wraps both inserts in a single transaction:

@contextmanager
def _transaction(conn: sqlite3.Connection):
    cursor = conn.cursor()
    cursor.execute("BEGIN")
    try:
        yield cursor
    except Exception:
        cursor.execute("ROLLBACK")
        raise
    else:
        cursor.execute("COMMIT")

The connection sets PRAGMA journal_mode=WAL at startup. Write-ahead logging lets the query CLI in Lesson 5 read historical prices while a fetch tick is writing a new snapshot, which matters when you want to check prices from your phone at 6:01am and the scheduler is finishing its 6:00am tick. The SQLite team explains the guarantees on https://sqlite.org/wal.html.

The schema stays tiny: snapshots(id, fetched_at, source_url, row_count) and listings(id, snapshot_id, item_id, item_name, quantity, unit_copper, seller) with a foreign key from listings to snapshots and two indexes on the query paths that matter. idx_listings_item(item_id, snapshot_id) covers the price-history query, and idx_snapshots_fetched(fetched_at) covers the daily digest report you'll inevitably add later.

The atomicity test is the one I keep closest to the CI eye. It deliberately passes an object that will fail the INSERT binding, then asserts that zero snapshot rows exist afterwards. If the transaction ever regresses to autocommit-per-statement (easy to do by accident with a Python driver that changes defaults between versions), that test catches it before a corrupted snapshot ships to production.

Lesson 4: schedule ticks on a jittered 6-hour cadence (commit 827c225)

Publishers don't love synchronised traffic. A tracker that fires exactly at HH:00 every six hours will show up in their access logs as a suspicious clock-tick pattern. The scheduler adds a bounded random jitter on top of the base 6-hour interval so no two runs of the tracker align:

async def run_forever(settings, *, should_stop=lambda: False, rng=None):
    jitter = rng or random.Random()
    executed = 0
    while not should_stop():
        await run_tick(settings)
        executed += 1
        if should_stop():
            break
        delay = settings.tick_interval_s + jitter.uniform(0, settings.tick_jitter_s)
        await asyncio.sleep(delay)
    return executed

The should_stop callback is the design decision that makes this testable. Instead of installing a signal handler inside the scheduler (which the CLI in Lesson 5 does at the outermost layer), the loop asks a callable whether to keep going. Tests pass a should_stop that flips to True after one iteration, which lets us assert loop progress without touching wall-clock time. run_tick never raises on fetch failure; a bad tick logs a warning and lets the next window try again, so a single publisher hiccup doesn't crash the daemon.

Six hours is the operator's contract with the publisher. Four fetches per day per shard, with jitter up to five minutes, works out to less server load than a single logged-in player refreshing the auction page manually. That's the point of the whole design: the tracker leaves a lighter footprint on the publisher's infrastructure than the same behaviour done by hand.

Lesson 5: wrap it in a typer CLI (commit f91cc12)

The CLI is the operator's steering wheel. Three commands cover the whole workflow: tick runs one fetch cycle and exits (perfect for cron), watch runs the scheduler until Ctrl-C, and history --item "Iron Ore" prints the last N listings for a specific item, newest first.

@app.command()
def history(
    item: str = typer.Option(..., help="Exact item name, case-sensitive."),
    limit: int = typer.Option(10, help="How many recent listings to print."),
) -> None:
    settings = Settings.from_env()
    conn = connect(settings.db_path)
    try:
        rows = item_history(conn, item, limit=limit)
    finally:
        conn.close()
    for row in rows:
        typer.echo(
            f"{row.fetched_at}  qty={row.quantity:<4}  "
            f"price={_format_copper(row.unit_copper):<12}  seller={row.seller}"
        )

Settings.from_env() is the escape hatch. Everything the CLI needs (auction URL, timeout, DB path, tick interval, jitter) reads from environment variables with sane defaults, so the same binary works locally and inside a systemd unit or a launchd plist. The typer docs at https://typer.tiangolo.com/ cover the option and argument model if you want to extend the CLI. An export-to-CSV command is a five-minute addition, for instance.

The signal handler for watch is the piece that made me hesitate the longest. Installing SIGINT and SIGTERM handlers is fine inside a top-level entry point, but doing it inside a library function is a landmine. The scheduler in Lesson 4 stayed pure by taking a should_stop callback, and the signal wiring lives inside cli.watch() where it belongs.

Comparison: why this stack over the alternatives

Approach	Publisher-safety	Setup time	Failure blast radius
Client-side memory reading	Fails TOS on every publisher	Days (per-patch reverse engineering)	Account ban
Man-in-the-middle packet capture	Fails TOS on most publishers	Hours (mitmproxy + cert install)	IP ban plus account flag
Community-run price API scrape	Middle-man dependency	Minutes	API goes down, tracker dies
Publisher public web view + httpx	Publisher-permitted (public URL)	Under an hour	Publisher changes markup, parser needs one fixture update

I put the comparison here because every conversation about auction-house automation opens with the same question: why not read the client memory instead? The answer is that reading client memory is faster (real-time updates instead of six-hour snapshots) but it violates the publisher's TOS on every game I've shipped for. Reading the public web view is slower by design and legitimate by design. For a solo trader who wants to spot weekly trends, six-hour snapshots are already 168x more granular than the "check every week" workflow they were replacing.

A few numbers to close the loop: the whole test suite runs 14 tests in 1.34 seconds on an M2 laptop, the SQLite database grows about 40 KB per 200-item tick, and a full year of six-hour snapshots on the top 200 items lands around 24 MB before compression. Those numbers matter because they set an upper bound on how much this tracker can misbehave, and the upper bound stays inside a single free-tier t3.nano for years.

Repository

Full source at https://github.com/vytharion/mmo-auction-price-tracker-public-web-httpx.

• Scaffold (commit 0) → 87cb78d: README, .gitignore, pyproject.toml, empty package
• Lesson 1 → 53cff33: httpx fetch layer with polite headers + FetchError
• Lesson 2 → 89e43ea: selectolax parser + copper normalisation
• Lesson 3 → 31923df: SQLite snapshots + atomic writes
• Lesson 4 → 827c225: jittered 6-hour scheduler loop
• Lesson 5 → f91cc12: typer CLI: tick, watch, history

Next steps

Once the tracker is running, the interesting work starts. A median-by-week view on top of the listings table gives you the same weekly-trend chart traders keep in spreadsheets. A crafting-margin view joins listings twice (once for the ingredient item, once for the crafted item) and surfaces recipes whose ingredient cost currently sits below the sale price. A tiny Telegram bot that reads history every morning and pings you when your watchlist moves more than 20 percent turns the tracker from a passive log into an alerter.

Clone the repo, run uv sync, run uv run pytest -q, then point AUCTION_URL at whichever publisher web view lists prices in HTML. Every extension mentioned above sits behind less than 100 lines of code, since the boring parts (fetching, parsing, storing, scheduling) are already done for you.