Ingestion & Deduplication
Reading Feeds Without Loading Them, and Recognising the Same Job Twice
Section 2 of 4
Everything downstream depends on this stage being right. If ingestion loads a feed into memory, large partners take the process down. If change detection is unreliable, every run rewrites the entire dataset and sends unchanged jobs back through an expensive model. And if deduplication is naive, the same role posted in twenty cities becomes twenty results, and the product looks broken to the user before any AI has been involved at all.
Streaming Ingestion
Partner feeds arrive as XML over HTTP and are large. The obvious implementation — fetch the document, parse the tree, iterate the results — has memory usage proportional to feed size, which means the system works fine until a partner grows and then fails on exactly the feeds that matter most.
Ingestion instead opens the response as a stream and parses incrementally, emitting jobs in fixed-size chunks as they are encountered and releasing each parsed element once it has been consumed. Memory usage is proportional to the chunk size, not to the feed. A feed ten times larger is ten times slower and uses the same memory.
Bytes read are tracked as the stream is consumed, which is how average feed size ends up in the run statistics — a number that turns out to be genuinely useful operationally, because a feed that suddenly halves in size is usually a partner problem that nothing else would have surfaced.
The same parser accepts a local file path, which is what makes reproducing a partner-specific parsing bug possible: capture the feed once, then iterate against it locally without hammering their endpoint.
Content-Addressed Change Detection
Every job is identified by a hash of its own meaningful content, not by a partner-assigned identifier. Two cosmetically different publications of the same role produce the same hash — which is what makes an unchanged job recognisable as unchanged.
Partner-assigned job identifiers would be the convenient key. They are also unreliable in every direction: some partners re-issue a new identifier for an unchanged listing, some reuse an identifier for a genuinely different role, and identifiers are not comparable across partners at all, so the same job syndicated through two of them looks like two jobs.
Hashing the content instead makes identity a property of the job itself. Anything that produces the same title, company, employment type and substantive description is the same job, regardless of who published it or what they called it.
The normalisation steps are the whole design. Feeds re-publish constantly with cosmetic differences — the description re-wrapped in different markup, tracking parameters rotated on the application URL, punctuation and whitespace changed. Hashing raw text treats every one of those as a new job, which means a full rewrite of the dataset every run, a full pass of model calls that were already paid for, and downstream systems churning on records that did not change. Stripping markup, URLs, punctuation and case before hashing means the hash tracks substance and ignores presentation.
The same hash then does double duty at the end of the run. The set of hashes currently active in the database, minus the set that just came through the feed successfully, is precisely the set of jobs that have disappeared from the partner's feed and should be archived. Change detection and archival detection are the same mechanism read in two directions.
Fuzzy Deduplication with Union-Find
Hashing catches jobs that are byte-identical after normalisation. It cannot catch the far more common case: the same role posted separately for each city it is open in, with the description reworded slightly each time. That needs similarity matching — and a way to merge clusters that were never directly compared.
Comparing every job against every other job is quadratic, and a feed with tens of thousands of listings makes that immediately impossible. So candidates are blocked first: jobs are grouped by an exact composite key of title, company and employment type, and similarity is only ever computed within a group.
The insight is that two postings for the same role at the same company will agree on those three fields exactly — it is the description and the location that vary. Blocking on the fields that do not vary reduces an intractable global comparison to many small local ones.