Skip to content
All case studies

Metis

Independent project 2020–present

I built and run Metis for my own market research. It collects news and SEC filings, tags material by topic and entity, and makes it searchable. I keep the source filings so I can replay them when a parser changes.

News processed
About 100,000 articles a day
Search
Vector search
Status
Private testing
Metis dashboard showing charts for entries, source fetches and outbound requests
The Metis dashboard shows ingestion activity, source coverage and market data.

Keeping the raw data

I built and run Metis alone, so parser mistakes need to be cheap to repair. Fetching and parsing are separate, and the raw SEC filings are retained. When I discover a form variant or fix a parser, a reparse runner can process the stored filing without downloading it again. Coverage checks and recorded processing errors make missing or failed records visible.

The same constraint shaped scheduled work. Bounded jobs that can resume now use the existing BullMQ worker, with progress, failure reporting and locks against overlapping runs. Backups keep their own Kubernetes CronJobs because their runtime and memory requirements differ, and replacing a worker during a long backup would lose work.

What I can inspect

The admin application shows coverage and processing failures alongside the collected data. Prometheus metrics and alerts cover scheduled jobs, including jobs that stop succeeding. Metis remains in private testing for my own research; there is no public signup.

Read about reparsing and data coverage, or see the current architecture and admin screenshots.

Pipeline and technology

The Evolution

My own project, built and run solo, and a spin-off from Plutus, a personal trading platform. The goal was to ingest news from thousands of sources in real time, understand it using AI, and correlate it with market data to find trading signals before the market reacts. Nobody was paying for it, so every architectural choice had to survive one person maintaining it.

Data Pipeline

  • Ingestion: 5000+ RSS feeds, newsletters, and archives monitored 24/7
  • Storage: Polyglot persistence using PostgreSQL (relational), ClickHouse (analytics), Meilisearch (search) and Qdrant (vector search)
  • Queueing: Event-driven processing on RabbitMQ and BullMQ

AI & Intelligence

  • Classification: Auto-tagging topics and detecting relations between entities
  • Retrieval: Article embeddings in Qdrant behind natural-language search and an LLM agent's news lookup. How it works
  • Summarisation: LLM-powered summaries (OpenAI, Anthropic, Ollama)
  • Market Data: Integration with Alpaca and Binance for real-time price feeds

The Tech Stack

TypeScript React NestJS FastAPI ClickHouse Meilisearch Qdrant PostgreSQL Redis RabbitMQ Docker Swarm Kubernetes Grafana OpenAI API