CollabWORK
An AI Job-Data Pipeline Holding Five Systems Consistent
Section 1 of 4
CollabWORK is an AI-powered job search platform serving the US, Canada and Mexico. I built the ingestion and enrichment backend behind it — the system that takes raw job feeds from a dozen partner sources and turns them into the clean, classified, deduplicated dataset the product actually searches over.
The problem is deceptively plain when stated: read XML, call a model, save results. What makes it an engineering problem is everything the feeds do in practice. They are large enough that parsing them naively exhausts memory. They contain the same role posted twenty times with twenty different locations and slightly different wording. They re-publish listings unchanged, so a system that cannot tell “identical” from “new” rewrites its entire dataset every run. The model has per-region quota limits, occasionally refuses a batch outright for exceeding a token limit, and is sometimes confidently wrong about what a job is. And the result has to land consistently in five separate systems, each of which serves a different part of the product.
This backend is public. Every mechanism described in these sections can be read in the source, and the live product it feeds can be opened below.
29
Python modules, fully asynchronous
7
Google Cloud regions used in parallel
5
downstream systems held consistent
2
deduplication passes per run
800
concurrent model calls at peak
Pipeline Architecture
Seven stages, strictly ordered, with three points where records leave the main path rather than being dropped. Nothing is discarded silently — a rejected job is a stored rejected job, and a low-confidence job is a job waiting for a human.
The Constraints That Shaped It
Look at what a run actually spends its time doing: downloading a feed, calling a model, writing to Postgres, calling an external API, writing to a warehouse, writing to a search index. Essentially all of it is waiting on a network round trip, and essentially none of it is computation.
A synchronous implementation would spend the overwhelming majority of a run idle, holding a thread open while nothing happened. The whole system is therefore built on Python's async stack — an async web framework, an async Postgres driver, async model clients — so thousands of operations can be in flight simultaneously against a single event loop. The few genuinely CPU-bound steps, notably fuzzy deduplication, are explicitly pushed to worker threads so they cannot block that loop.
Concurrency is bounded rather than unlimited at every level: how many model calls may be in flight, how many calls to the external API, how many jobs go into one model call, how many jobs are in one pipeline batch. Unbounded concurrency does not go faster — it gets rate-limited, and then it fails in a way that is much harder to reason about than being slightly slower.