Metis
Independent project 2020–present
I built and run Metis for my own market research. It collects news and SEC filings, tags material by topic and entity, and makes it searchable. I keep the source filings so I can replay them when a parser changes.
- News processed
- About 100,000 articles a day
- Search
- Vector search
- Status
- Private testing
Keeping the raw data
I built and run Metis alone, so parser mistakes need to be cheap to repair. Fetching and parsing are separate, and the raw SEC filings are retained. When I discover a form variant or fix a parser, a reparse runner can process the stored filing without downloading it again. Coverage checks and recorded processing errors make missing or failed records visible.
The same constraint shaped scheduled work. Bounded jobs that can resume now use the existing BullMQ worker, with progress, failure reporting and locks against overlapping runs. Backups keep their own Kubernetes CronJobs because their runtime and memory requirements differ, and replacing a worker during a long backup would lose work.
What I can inspect
The admin application shows coverage and processing failures alongside the collected data. Prometheus metrics and alerts cover scheduled jobs, including jobs that stop succeeding. Metis remains in private testing for my own research; there is no public signup.
Read about reparsing and data coverage, or see the current architecture and admin screenshots.
Pipeline and technology
The Evolution
My own project, built and run solo, and a spin-off from Plutus, a personal trading platform. The goal was to ingest news from thousands of sources in real time, understand it using AI, and correlate it with market data to find trading signals before the market reacts. Nobody was paying for it, so every architectural choice had to survive one person maintaining it.
Data Pipeline
- Ingestion: 5000+ RSS feeds, newsletters, and archives monitored 24/7
- Storage: Polyglot persistence using PostgreSQL (relational), ClickHouse (analytics), Meilisearch (search) and Qdrant (vector search)
- Queueing: Event-driven processing on RabbitMQ and BullMQ
AI & Intelligence
- Classification: Auto-tagging topics and detecting relations between entities
- Retrieval: Article embeddings in Qdrant behind natural-language search and an LLM agent's news lookup. How it works
- Summarisation: LLM-powered summaries (OpenAI, Anthropic, Ollama)
- Market Data: Integration with Alpaca and Binance for real-time price feeds