📰 Metis Revisited: The Platform Behind the News

In my original Metis post, published in April 2024, I went through the news aggregator I had been building for a few years. There were screenshots of the admin dashboard and a list of the things it could collect. Since then I have kept working on it, and that post has become a fairly incomplete description of the project.

I recently wrote about the SEC filings pipeline, which is one part of that change. This time I wanted to come back to the whole platform, because quite a lot of the work has been in how those parts fit together.

The Starting Point

The original version already collected articles and market data, assigned topics, and used AI to classify and summarize news. The admin interface was where I could inspect the sources and see what the queues were doing. That is still part of Metis, and the screenshots in the earlier post are a useful record of that version.

What has grown around it is a financial data platform. News sits alongside company disclosures and other datasets, with tools for searching and inspecting the results. This post describes Metis as it stands in September 2026.

The Data

The financial side now includes SEC filings and company financials, analyst ratings, congressional trading disclosures, government contracts, lobbying filings, economic forecasts and prediction markets. Each has its own fetching and processing work, and its own idea of what a useful record looks like.

A news article has a publication date and a publisher. A government contract has an award identifier and a recipient, while a congressional disclosure has a reporting date that can be later than the transaction it describes. Keeping those distinctions is part of making the data useful once it is in the same system.

Assets provide a way to connect records to the companies or instruments they concern. An article mentioning a company and a filing submitted by that company can then be found through the same asset, while keeping their original source information. There is also a separate distinction between an asset and the market where it trades, which was already present in the earlier version.

The News Pipeline

The news side has become more explicit about the individual processing steps. There are separate processors for language detection, sentiment, named entities, text preparation, asset matching, embeddings, topics and duplicates. That gives me specific pieces to rerun when the data or the processing changes.

One detail I like is that a failed page fetch does not always have to stop enrichment. When an attempted fetch has been recorded, the processor can continue with the title and excerpt it already has. A later successful fetch processes the full content again. The failure remains recorded, so a partially enriched article does not quietly become a successful download.

Embeddings also have a clearer place now. They are written to Qdrant for vector search, and natural-language search can use those vectors to find related entries. The platform still has Meilisearch and ClickHouse alongside PostgreSQL; adding a vector index did not remove the other storage and search paths.

The Model Services

The model-serving side is split into services for embeddings, named-entity recognition, sentiment analysis and language detection. Those use Python and FastAPI. Clustering now has a Rust service using Axum, so the old description of the backend as TypeScript with some Python leaves out a piece.

This is the part of the project that gets hidden by a phrase like “AI classification”. An embedding and a sentiment score are different outputs, with different consumers. Keeping their processing separate means the rest of the application can ask for the result it needs without putting every kind of analysis into the same call.

The Architecture

The diagram below is a simplified map of the current responsibilities. The arrows show the broad movement of data, rather than every queue or database connection.

The admin application remains React with React Router, backed by NestJS. The codebase also retains a Next.js research interface for local development. Shared libraries hold the processing and database code used across the applications, while some services run under Bun. The workspace tooling has also moved from the NX setup listed in the old post to Yarn workspaces and Turborepo.

Even the dependency installation needed attention. The Bun services run under Bun, but their container builds install dependencies with Yarn against the repository lockfile. That keeps the runtime choice from becoming a separate dependency-resolution path with its own idea of which versions to install.

The Scheduled Jobs

A less visible refactor was moving suitable scheduled work into the existing worker. The platform had accumulated Kubernetes CronJobs alongside BullMQ repeat jobs, so there were separate places to define schedules and find failures. Bounded jobs that can resume unfinished work now share the worker’s scheduling machinery.

That move also needed a shared execution wrapper. It records progress and failures, gives a run a time budget, and works with locks that prevent overlapping runs. Prometheus metrics and alerts cover the schedules, including jobs that have stopped succeeding. Moving a schedule out of Kubernetes would have been incomplete if its failure reporting had stayed behind.

Backups still have their own CronJobs. They have different runtime and memory requirements, and replacing the worker in the middle of a long backup would throw away work I need to keep. The split follows the work each job does.

The Subscriptions

Subscriptions have replaced the older broadcasting system. They match asset events and can deliver individual notifications or digests, with delivery records and retries. The delivery code supports email and webhooks as well as messaging channels. This puts the matching and delivery work in one system instead of maintaining a separate broadcaster for signals.

For private testing, this provides another way to follow the data without leaving the dashboard open. A subscription can describe the assets and event types to watch, and whether matching events should arrive individually or together in a digest. That makes delivery part of the research workflow being tested.

The Signal Research

Metis also has signal evaluation and strategy code, and some of the recent refactoring was about removing overlapping responsibilities there. Indicator calculations now belong to the technical-analysis library. Oracle owns signal policy and replay, while strategy code owns trade execution and backtesting. Unused copies of signal replay code were removed.

The distinction matters when I revisit a result. A change to indicator math needs to be distinguishable from a change to the policy that turns it into a signal. The system versions that evidence. Signals produced by corrected logic are evaluated together, and have to accumulate fresh results before they can contribute to recommendations.

I am also keeping the description of those results fairly literal. A recommendation’s confidence field is a heuristic score, and a better reported accuracy after fixing an evaluation bug does not establish better trading performance. This is an area of the platform I am still developing, with no profitability claim attached to it.

The Private Testing

Metis is in private testing, and I am not planning a public release. For now, I am keeping it focused on private research and evaluating specific uses directly. I use it internally; there is no public signup or invitation programme.

The End

The earlier post was mostly a tour of what Metis collected. The work since then has expanded that considerably, but it has also involved bringing overlapping code back into one place and making failures easier to inspect. There is a lot more behind the old dashboard screenshots now.

Metis remains closed source, so there is no repository link for this one. More of what I have built is on the projects page, and my engineering background is on the career page.

I hope you enjoyed this blog post and I will see you in the next one!