JobDedup: Semantic Job Posting Deduplication Engine for Job Boards

Here's a real one we hit, still only partly solved, and it generalises well beyond our case. The problem: deduplicating job postings that are the same job. We track a live index of postings and roughly two thirds of what comes in is a repost of something already in there. Same role, different day, sometimes a different board, sometimes the same board with a new id. Why it's harder than it sounds: - Titles are unstable. "Senior Backend Engineer", "Sr. Backend Engineer (Remote)", "Backend Engineer II" can be one job or three. - Descriptions get lightly reworded between posts, so exact hashing catches almost nothing and fuzzy matching over-merges. - Employers genuinely repost the same requisition after 30 days, and they also genuinely open a second headcount for the identical role. Those are different facts and they look identical. - The naive fix, embed everything and cluster, is expensive at volume and still can't tell a repost from a second opening. Current broken workflow: a mix of normalised-title plus company plus location keys, with a fuzzy pass on the description. It leaks in both directions, which is the worst outcome, and the thing that finally helped was realising distance can't decide it. A typo and a genuinely different role can score the same, so any threshold you pick is wrong somewhere. The signal I suspect is the answer and haven't built: the requisition identifier the employer's own system assigns, which is often sitting in the apply URL. It's messy and inconsistent per platform, which is presumably why nobody does it. Full disclosure, this is from https://aiapplyd.com which is what I work on, so I'd obviously benefit if you solved it. Take it as a genuinely interesting dedup problem rather than a request. Happy to describe the data shape if you want to poke at it.