Tracking Vietnamese news trends with embeddings, HDBSCAN and an LLM

A big event is usually covered by dozens of outlets at once, under different headlines. Reading article by article, it's hard to see the overall picture. The more useful question isn't "what's in the news today", but: which stories are heating up, which are cooling off, and why?

To answer that, I built Tin tức hôm nay có gì? ("What's in the news today?"), a system that reads about 95 Vietnamese RSS feeds, groups articles about the same thing into topics, tracks each topic's lifecycle, and has an LLM summarize what is driving it. This post covers how it works and the numbers I picked along the way.

Architecture at a glance

It's a single TypeScript service running on Bun, with its data in one SQLite file (bun:sqlite). The pipeline runs in passes, and each pass goes through these stages:

1. Ingest: read the RSS feeds and drop duplicates.
2. Embed: turn each article's title and description into a vector.
3. Cluster: group nearby vectors with HDBSCAN.
4. Lifecycle: update each topic's velocity, acceleration and stage.
5. Label and insight: an LLM names the topic and explains it.
6. Digest: put together an email summary.

The same image runs as web (dashboard and API), pipeline, or both. In production the two run separately, so the dashboard isn't affected while the pipeline is busy.

A schedule that follows the news cycle

News doesn't arrive evenly through the day, so the pipeline doesn't run evenly either. The schedule is computed in Vietnam time:

Passes are aligned to round times (7:00, 7:15, 7:30…), so the dashboard can always say when the next one will run.

Dropping duplicates

The same article often shows up in several feeds of one outlet (front page, section, latest news). I hash the URL into a content_hash and skip anything already in the database. It's simple and good enough, since one article nearly always keeps one URL. Two different outlets covering the same event are not duplicates, though: that's exactly the signal clustering needs.

Embeddings for Vietnamese

By default I use jina-embeddings-v3 through Jina AI's API; it's multilingual and handles Vietnamese well. Without an API key the service falls back to running LaBSE locally (via @xenova/transformers). It's slower, but the pipeline doesn't depend on an outside service to keep working.

Vectors are stored with the article, and each pass only embeds new articles, up to 1,000 at a time.

Why HDBSCAN

With news, I don't know in advance how many topics there are, and most articles are noise: one-off stories that don't belong to anything bigger. K-means makes you pick the number of clusters and forces every point into one. HDBSCAN finds the number of clusters on its own and labels points that don't fit anywhere as -1. For this problem, that's exactly the behavior I want.

It runs in a small Python process (the hdbscan library); Bun pipes the vectors in on stdin and reads the labels from stdout. The current settings:

HDBSCAN(
    min_cluster_size=8,
    min_samples=2,
    metric="euclidean",
    cluster_selection_method="leaf",
)

A few of these choices are worth explaining:

• L2-normalize first. For unit vectors, Euclidean distance and cosine similarity are directly related (d² = 2 − 2·cos), so euclidean ranks neighbors exactly as cosine would, and HDBSCAN is faster with it.
• leaf rather than eom. eom prefers large, stable clusters, so it tends to merge related but separate stories (say a storm, the floods after it and a landslide) into one. leaf gives smaller, more specific clusters, which suits tracking individual events.
• min_cluster_size=8. A topic needs at least 8 articles in a pass. Lower, and the dashboard fills up with trivia; higher, and new stories get missed.
• Each pass clusters at most 3,000 articles. Articles marked as noise get retried for the next 2 hours, in case their story is only just starting.

From clusters to topics that live across passes

HDBSCAN only knows about the articles in the current pass. For a topic to live for hours, each topic stores a centroid (the mean of its vectors), and every new cluster is compared against the centroids of active topics:

The merge threshold is higher than the join threshold on purpose: a wrong merge is hard to undo, while leaving two clusters apart for one more pass costs almost nothing. After each merge the centroid is recomputed right away, which stops cascades (A absorbs B, then AB absorbs C only because it grew).

A topic's lifecycle

Each topic moves through a simple state machine driven by velocity (how fast new articles arrive, measured over a 2-hour window) and acceleration (how velocity is changing):

• Signal → Emerging: at least 5 articles and positive velocity.
• Emerging → Growing: positive acceleration.
• Growing → Hot: at least 10 articles and velocity above 1.
• Hot → Peak: acceleration close to zero relative to velocity (under 10%).
• Declining: negative velocity for 2 windows in a row.
• Dead: no new articles for 3 windows in a row. If new ones arrive later, the topic comes back as Emerging.

Every transition is recorded with its reason (for example "Count >= 10 and velocity > 1.0"), so when a topic jumps around I can read its history and see why.

Scoring how hot a topic is

For ranking, each topic gets a score from 0 to 100, the sum of six parts normalized to 0–1:

Distinct sources get their own weight because 20 articles from one outlet are very different from 8 outlets covering the same story. The score is then multiplied by a decay factor based on the time since the latest article: no decay in the first hour, then 20% after 1 hour, 40% after 6, 60% after 12, 80% after 24, and zero after 48 hours.

The LLM: naming and explaining

The LLM is called through an OpenAI-compatible API (DeepSeek by default), for two kinds of requests:

• Naming: at most 5 words, in Vietnamese, sent in batches of many topics, each with at most 10 article headlines.
• Analysis: JSON with driving_force (the main cause), trend_rationale (why it's heating up or calming down), opportunity (a concrete opportunity or risk, and for which industry), watch_next (what to watch for) and an opportunity_score from 0 to 10.

LLM calls cost money, so choosing which topics to analyze matters as much as the prompt:

• Only active topics (Signal through Peak) are considered; Declining and Dead ones aren't.
• A topic is re-analyzed when it has never been analyzed, has 5 or more new articles since last time, has been flagged for a redo, or its last analysis is over 24 hours old.
• Candidates are ordered by hotness score, and each pass is capped (500 topics for naming, 300 for analysis).
• Content-based caching: each analysis stores a hash of the articles it was based on. If that set hasn't changed, the topic is skipped and no LLM call is made.

Token usage is recorded per pass to keep an eye on cost. Setting LLM_ENABLED=false keeps the whole pipeline running, just without names and analysis.

Running it, and what I'd improve

Day to day, the service needs little attention:

• Data: the SQLite database is about 3.8 GB. Dead topics, orphan articles and logs older than 30 days are deleted daily.
• Measuring: each pass records the time of every stage (embed, cluster, label, insight, lifecycle, digest) and the tokens it used. The app's dashboard shows them, and my shared monitoring tracks them too (see self-hosting side projects).

There is also room for improvement:

• Recluster over a sliding window instead of clustering only new articles and comparing centroids. The current approach is cheap, but centroids can drift as long-lived topics grow.
• A labeled evaluation set. The 0.75 / 0.78 / 0.82 thresholds are currently fixed values in the code. A few hundred article pairs labeled "same story / different story" would make those choices less of a guess.
• Deduplicate by content, not just by URL, to catch republished articles.

Summary

• A trend tracker answers "which stories are heating up, which are cooling off, and why", not just "what's in the news today".
• HDBSCAN fits news because it finds the number of clusters on its own and labels one-off stories as noise; leaf keeps related but separate events apart.
• Topics live across passes through stored centroids and cosine thresholds, with merging set stricter than joining because a wrong merge is hard to undo.
• Velocity and acceleration drive a simple lifecycle from Signal to Dead, and every transition records its reason so a topic's history can be read later.
• The hotness score gives distinct sources their own weight, since 8 outlets on one story mean more than 20 articles from one outlet.
• LLM cost is controlled by choosing what to analyze: only active topics, only when their articles change, ordered by score and capped per pass.

The live version is at news.vuminhhoa.com. If you're working on something with Vietnamese news too, I'd like to hear about it: my email is on the home page.