Home › Our work › Defense Procurement ETL and Market Intelligence
Our work / Defense Procurement ETL and Market Intelligence
Turning federal contract data into a market you can see
The information a US defence supplier needs is out there. It is spread across four systems that use different formats, different identifiers and different update schedules. We built the pipeline that puts them together and answers the question a supplier actually has: where is the business, and who else is going for it.
The problem with the data
A manufacturer selling into defense logistics lives on the portal that lists the open requests for quote: which part, how many, by when. Reading it tells you what is available today and stops there.
The questions that decide whether a quote is worth preparing sit elsewhere. Has this part been bought before, and how often? Who won it, at what unit price, and how long ago? Is the buyer a regular or a one-off? Are we even an approved source, and who else is? Those answers live in other systems, each with its own export format and its own idea of how to spell a company name.
People do reconcile this by hand. It takes a morning per part, which means it happens for the handful of opportunities somebody already had a hunch about, and the rest go unexamined.
What we built
An extraction pipeline that pulls all four sources on a schedule, resolves them against each other, and loads the result into a single database that answers those questions in one query.
The join key is the National Stock Number. It is a thirteen-digit identifier for one specific manufactured part, and it is the one field every system in federal supply shares. A solicitation, an award five years ago and a supplier approval all point at the same NSN, which makes them the same part.
Getting to that key is most of the work. The NSN appears in a free-text description as often as it appears in a proper field, written five different ways: dashes, no dashes, prefixed with a label, buried in a sentence. The extractor reads the structured field where one exists and falls back to pattern matching over the description text, checking for the labelled form before the bare thirteen-digit form so a quantity or a date is never mistaken for a part number.
Company names are the hard part
The second reconciliation problem is the supplier. The same company appears as
ACME AEROSPACE INC in one dataset, Acme Aerospace, Inc. in another and
ACME AEROSPACE in a third, and a naive match treats them as three competitors.
The pipeline normalises names through a dedicated cleaning step and, where the data provides one, resolves against the CAGE code, the government's own unique identifier for a supplier facility. A CAGE match is authoritative. A name match is a candidate that gets recorded as such. Keeping those two apart is what stops a competitor analysis from quietly double-counting.
Designed to run unattended
Long scrapes over these portals fail partway through, and a pipeline that has to start over each time it hits a timeout will never finish. The extraction saves progress as it goes, so a run resumes where it stopped. It paces requests deliberately, staying polite to servers that belong to the government and are shared by every supplier in the country. It checkpoints large API pulls so a rate limit costs minutes rather than a day.
Raw pulls are written once and kept. Everything downstream reads from those stored files rather than re-fetching, which means an analysis can be re-run months later and produce the same numbers.
What it is used for
Three things, in practice.
Finding the opportunities worth quoting. Open solicitations arrive already joined to their own award history, so a buyer sees at a glance whether a part has been bought thirty times or once, and whether the prices have been climbing or falling.
Understanding the competition. Award history by NSN shows exactly who has won this part, how often, and whether the client is on the approved-source list alongside them. That turns a guess about the competitive field into a count.
Feeding the pricing model. The dataset this pipeline produces is the training data for our price prediction model, which recommends a unit price for an open opportunity from the award history of that exact part.
The value is not that the data was secret. It is that nobody had the hours to read all of it, in the several places it lives, before the quote was due.