For most of my career the honest answer to “what should I try first on this series?” was a seasonal naive model and then some flavour of gradient boosting on lag features. Time-series foundation models (TSFMs) — pretrained on very large and very heterogeneous collections of series, then applied zero-shot — have moved that line. You can now point a pretrained model at a series it has never seen and get a forecast that is competitive with something you would have spent a week fitting.
That is a real shift, and I do not want to undersell it. But working with forecasts in production has made me sharply aware of what it does not solve.
A TSFM’s pretraining corpus has an end date. Everything after that is, from the model’s point of view, out of distribution.
For a lot of series that hardly matters — weekly seasonality in electricity demand is not going to be rewritten. For the series people actually care about commercially, it matters a great deal. Retail demand carries promotions, price changes, assortment churn, competitor behaviour, and the occasional structural break that makes the last three years of history actively misleading. The regularities the model learned are still true on average and wrong exactly where the money is.
So the same question I have been chasing in text streams shows up again: how does a pretrained model keep up with a world that moves after pretraining ends?
In the online setting I have worked in before, the loop is tight. The model predicts a label, the true label is revealed, the model updates. Uncomfortable, but clean.
Forecasting breaks that loop in a way I do not think is appreciated enough.
Supervision is delayed, and the delay is the horizon. Predict 28 days ahead and you find out how wrong you were in 28 days. At any moment, the most recent feedback you have concerns a forecast you made a month ago, about a world that has since moved on. Every update you make is an update based on stale evidence.
The feedback is staggered. Forecasts made on different days for different horizons resolve at different times, so the error signal arrives as a continuous dribble of partially-complete information rather than a clean batch.
Actions contaminate the labels. This one is specific to production and it is brutal. A demand forecast drives a replenishment decision. That decision changes what is on the shelf. What is on the shelf changes what sells. So the “true” demand you eventually score against is partly a consequence of your own earlier forecast. Under-forecast, stock out, observe low sales, conclude demand was low, under-forecast again — a feedback loop that looks like model drift and is actually a decision loop eating itself. Censored-demand correction is not a nice refinement here; without it, adaptation actively makes things worse.
And you cannot retrain your way out. Full fine-tuning of a foundation model on every batch of arriving truth is not affordable, which is why parameter-efficient approaches — LoRA and friends — are the obvious lever. But they only reframe the question: you still have to decide when to move the adapter weights, and by how much, on evidence that is late and incomplete.
I will not pretend this is settled — a chunk of it is work that is currently under review, and I would rather write about it once it has survived that. But the principles I have most confidence in are ones I would defend regardless of how the results land.
Distinguish a shock from a regime change. A single large error is usually not a reason to move. A run of same-signed errors probably is. Almost all the value in an adaptive forecaster lies in that distinction, and almost none of it lies in adapting faster.
Prefer agreement over speed. The pattern that keeps working — in AdaNEN, in Adaptimizer, in the forecasting setting — is to require two differently-tempered views of the data to agree before committing to an update. A fast estimator alone chases noise. A slow one alone never arrives. Requiring corroboration makes the system’s willingness to change a function of the evidence rather than of a hyperparameter.
Evaluate on the decision, not the metric. WMAPE going down is not the goal. Fewer stockouts and less dead stock is the goal, and the two come apart more often than is comfortable — most obviously because forecast error is not symmetric in cost. Being 10 units short is not the same kind of wrong as being 10 units long.
Assume you are inside a feedback loop. If your model’s output changes the world that generates your training data, you are not doing supervised learning in the sense the textbook means. Design the evaluation accordingly.
TSFMs move the starting line, and that is worth a lot: less bespoke modelling per series, a much stronger cold-start, genuinely useful zero-shot behaviour.
They do not remove the need to keep a model honest after deployment. If anything they sharpen it, because a large pretrained model is expensive enough that the retrain-often reflex stops being available and you are forced to think properly about what adaptation should mean.
Which is, more or less, the thing I have been working on from three different directions for six years.
]]>The usual answer is to retrain on a schedule. Quarterly, monthly, whenever someone notices the dashboards sagging. But that answer has a hole in the middle of it: between retrains, the model is wrong and nobody is doing anything about it. The longer the retraining cadence, the wider the hole.
Our CIKM 2025 paper, LLM-OFA, is about closing that gap.
We frame it as On-the-Fly Adaptation (OFA), and the framing matters more than any single technique in the paper.
Each incoming instance is first used for inference — the model predicts, and that prediction is what the user actually gets. Only afterwards is the true label revealed, and the model updated. Single-pass, sequential, no second look at the data.
That ordering is the whole point. It means:
It is a deliberately harsh setting, and it is the realistic one.
Most concept-drift benchmarks are small, synthetic, or both — you inject a distribution shift at instance 5,000 and see whether the detector notices. That tells you something, but it does not tell you how a model behaves over a decade of gradual semantic movement.
So we built 1M-News: one million New York Times headlines spanning 2005–2025, labelled by section. Two decisions in it were more consequential than they look.
Consolidating the label space. The raw corpus has 87 sections, many of them near-duplicates or vanishingly rare, and several that appear or disappear mid-stream. We consolidated them into a stable set (Real Estate folded into Business, and so on). Without that, you measure the newsroom reorganising its own taxonomy rather than the language changing — a real drift, but not the one under study.
Handling long items. News text items run long, and truncating at 512 tokens throws away exactly the context that disambiguates a drifting entity. We use Big Bird’s tokenizer so that the ultra-long tail stays intact.
The result is a benchmark where the drift is endogenous. Nobody injected it. It is simply what twenty years of news looks like.
The obvious way to adapt continuously is to keep running SGD on the incoming stream. The obvious problem with that is equally well known: you get catastrophic forgetting, and you get a model that chases every burst of noise. Turn the learning rate down instead and you get stability, and no adaptation.
That tension — plasticity against stability — is the actual problem, and it is not solved by finding a cleverer learning rate.
Adaptimizer keeps two sets of weights:
The update is then gated on elementwise agreement between the two. Where fast and slow both push a parameter the same way, that is evidence of real signal, and the parameter moves. Where they disagree, the fast set is reacting to something the slow set does not corroborate — most likely noise — and the update is damped.
The intuition is a consensus rule rather than a compromise. A blend of two learning rates is still one learning rate. Requiring two differently-tempered estimates to agree before committing is a different thing: it makes the optimizer’s confidence depend on the data rather than on a hyperparameter you guessed in advance.
Across the benchmark, OFA with Adaptimizer improves consistently over static baselines — models that were fine-tuned once and then left to face two decades of news on their own.
Three things generalise past the paper.
Prequential evaluation is not a nicety. If your offline number comes from a random split of a temporally-ordered dataset, it is measuring the wrong thing and it is measuring it optimistically. Score on the future, always.
“We retrain quarterly” is a statement about your error budget. It says you have accepted being progressively wrong for up to three months at a time. That may well be the right call — but it should be a decision, not a default.
Stability is a first-class objective. An adaptive system that occasionally lurches is worse than a static one, because nobody can reason about it. Most of the engineering in Adaptimizer is not about adapting faster; it is about not adapting to the wrong things.
The paper is at ACM CIKM 2025, and the code and the 1M-News data are at github.com/PouyaGhahramanian/LLM-OFA. Joint work with Sepehr Bakhshi and Prof. Fazlı Can at the Bilkent Information Retrieval Group.
]]>