📄 Turning SEC Filings Into Something You Can Query

In an earlier post I wrote about Metis, the news aggregation and classification platform I have been building in one form or another for years. It has grown a fair amount since then, and the part I want to write about now is the SEC side of it.

I went in thinking this was a fetching problem. Everything a US public company is legally required to disclose is free and public, the SEC publishes it in bulk, and pulling it down is a weekend. That part was a weekend. The rest of it has taken years.

The Problem

A filing is a document, not a record. It was written to satisfy a disclosure obligation and to be read by a person, and only incidentally to be parsed by me.

So it is not one job, it is a stack of separate ones that each need their own answer:

• Insider trades - who bought and sold, when, and at what price
• Institutional holdings - what the large funds hold, and what changed since last quarter
• Beneficial ownership, including the ownership chains, because it is rarely one entity holding one thing
• Executive compensation, proxy statements, registration statements, private placements, tender offers, proposed sales
• Late filing notices - a company saying it will miss its own deadline, which is often more interesting than what it eventually files
• Current report items - the 8-K, the “something happened” form, and therefore the one that actually moves

Each of those is a different form, with its own shape, its own edge cases, and its own idea of what counts as the same company.

The Stack

• Ingestion: a dedicated SEC queue worker, with a runner per form type
• Storage: PostgreSQL for the structured records, ClickHouse for analytics, Meilisearch for search, Qdrant for vectors
• Queue: BullMQ, with the SEC work isolated from the rest of the pipeline
• API and admin: NestJS, in an NX monorepo, with an admin surface per form type
• Models: self-hosted services for embeddings, clustering, named entity recognition, sentiment and language detection

The Rule: Keep the Raw, Reparse Later

The most useful decision in the whole pipeline is that fetching and parsing are separate, and the raw filing is kept forever.

The reason is not disk space. It is that my parser is wrong, and I know it is wrong before I have found out how. There is always a form variant I did not know existed, a field that meant something different before some rule change, an amendment restating a number I already stored. When I find one, the fix cannot be to re-download eleven years of filings, because the SEC is a public service and I am not going to hammer it to fix my own bug.

So nearly every form type has a reparse runner sitting next to its ingest path. When I learn something new about how a form works, I fix the parser and reprocess what I already have, and not a single new request goes out. That one decision is why the pipeline has survived several years of me being wrong about things.

The Things Nobody Warns You About

Knowing what is missing. There is a coverage layer whose only job is to answer which filings should exist that I have not got. Without it the gaps are invisible, because missing data does not raise an error. It just sits there, and I find out when a query returns a confidently incomplete answer.

Identity is the hard part. A filing names an issuer, sometimes a reporting person, sometimes a fund, in a format that changes over the years. It does not name a ticker. Linking a filing to the asset someone actually cares about turned into its own subsystem, and it is what separates a pile of documents from something queryable.

Errors are data. Processing failures do not get logged and forgotten, they go into a table and get retried by their own runner. A parse that fails today because a field was empty may succeed next month after a parser fix, and I would rather that happened on its own than because I remembered.

History gets rewritten. Filings are amended, dates get corrected, identities get merged. There are repair passes whose entire purpose is to go back over data that was correct when it was ingested and is not correct any more.

Newest first, while the backfill runs. The queue can run last in, first out, so a filing that lands right now jumps ahead of the multi-year backfill grinding away behind it. Fresh data stays fresh while history fills in underneath. I did not have that at first, and the backfill starved the live pipeline for about a week.

The Rate Limiting

Metis makes tens of thousands of outbound requests a day across its sources, and the SEC is a public service rather than an API I bought.

So every outbound request is recorded, fetch requests are tracked per source and per entry, there are domain blocks for hosts that should not be touched at all, and there is a proxy layer for the places that need one. I do not rate limit because I am worried about being banned. I rate limit because it is a shared resource and I am one of many people using it.

The side benefit of recording every request is that when a source quietly changes shape, I can see it. The requests still succeed, the parses start failing, and the two together tell me what happened.

The AI Part

Two places.

Text, not numbers. The structured fields are parsed deterministically, because a share count is not a judgement call and no model should be anywhere near it. Where the models earn their place is the prose: analysing the text of a filing, and scoring whether an 8-K item is material or routine. That is reading comprehension over unstructured English, which is what they are good at.

Writing the parsers, not being one. Working out how an unfamiliar form variant is laid out is high volume, mechanical, and needs someone to read a lot of tedious XML carefully without getting bored. The output is easy to check, because the parser either produces the right rows for a filing I already have or it does not. That is a good use of an agent. Asking a model to be the parser at runtime, for data that has to be correct, is not.

The Current Status

The SEC side now covers insider trades and insider scoring, institutional holdings and the changes between quarters, beneficial ownership chains, executive compensation, proxy and registration statements, private placements, tender offers, late filing notices, 8-K items with materiality scoring, and filing text analysis. There are backfill runners for coverage and for the master index, repair passes for dates and identities, and a retry runner for processing errors.

Metis is not open source and I have no immediate plans to change that, so unlike some of my other projects there is no repository to link here.

The End

The interesting-sounding part, the bit where I say I ingest SEC filings, was a couple of weeks. The years went into coverage, identity, reparsing, repair and error retries, which is another way of saying I was wrong a lot and had to build the tooling to fix it cheaply.

If I had to keep one thing from all of it, it would be the raw copy. Everything else only works because the original filing is still sitting there when I discover what I got wrong.

More of what I have built is on the projects page, and if you want to talk about data pipelines that have to stay correct, here is what I am looking for.

I hope you enjoyed this blog post and I will see you in the next one!