Overview
Information relevant to different questions changes at different rates. Some answers may remain correct for only a few hours, while others remain valid for months, years, or may not be expected to become outdated.
Task 1 asks systems to estimate a question's recency requirement: how recent the information needs to be to answer the question correctly. If the answer changes often, more recent information is needed. If the answer is stable, older information may still be sufficient.
Where provided, temporal context gives additional information about the situation in which the question is asked. It may include temporal cues, events, or circumstances that affect how recent the information needs to be. The same question can therefore have different recency requirements under different contexts.
Systems assign one of twelve ordered recency classes, from information that needs to be very recent to information that is unlikely to become outdated. The task does not require systems to generate the answer itself; they only predict the appropriate recency class.
Why does recency requirement estimation matter?
Large language models (LLMs) can answer many questions using knowledge learned during training. However, the information needed to answer different questions can change at very different rates. A question about current weather, a recent event, or the current holder of a public office may require up-to-date information, while a stable scientific fact may remain correct for many years.
For question-answering systems, retrieval-augmented generation (RAG), and other LLM-based systems that can access external information, this raises an important question: when is fresh information needed, and when is existing knowledge still sufficient?
Estimating a question's recency requirement can help systems determine how recent the information should be before answering. This can support decisions about whether new retrieval is needed, how recent the retrieved evidence should be, and when previously retrieved or cached information should be refreshed.
A reliable recency estimate can also help systems avoid unnecessary retrieval. Questions with stable answers may not require fresh evidence every time, while highly time-sensitive questions may need recent information before an answer is produced. This can make retrieval more targeted and reduce the risk of using outdated information.
Rather than treating information as simply current or outdated, this task considers recency requirements across different temporal scales, from information that may change within hours to information that may remain valid for many years or indefinitely.
Task definition
Given a natural-language question and, where provided, its context, the goal is to predict one of twelve ordered recency classes.
The predicted class represents how recent the information needs to be to answer the question correctly under the given context.
Context provides information about the temporal or situational setting in which the question is asked. It may describe an event, stage, or circumstance that affects how recent the information should be.
For example:
Question: What is the latest stable version of Python?
Context: “A developer is preparing the software environment for a new project that will begin development soon.”
Recency class: A-Few-Months
Python releases change over time, but a stable version usually remains the latest stable release for a period of time before it is replaced by a newer version. Information from the last few months may therefore still be valid, while much older information is more likely to be outdated.
The task concerns the freshness of the information needed to answer the question, not how often a source reporting that information is published or updated.
Systems are not required to generate the factual answer itself.
Illustrative examples
| Question | Context | Recency class | Interpretation |
|---|---|---|---|
| What is the exchange rate between the euro and the US dollar? | “A traveler is checking how much money they would receive before making a currency exchange.” | A-Few-Hours | Exchange rates can change throughout the day, so highly recent information is needed. |
| What is the unemployment rate in Austria? | “An economist is preparing a report on the present labour-market situation.” | A-Month | The unemployment rate can change from one reporting period to the next, so information from the recent period is needed. |
| What is Geoff Hinton’s h-index? | “A conference organizer is preparing a speaker profile for an upcoming event.” | A-Few-Months | The value can change as new publications and citations are added, so relatively recent information is more suitable than much older information. |
| What is the chemical symbol for gold? | “During a chemistry class, a student asks about the chemical symbol for gold.” | Never | The answer is stable and does not depend on recent information. |
Recency classes
The twelve classes represent progressively longer expected validity periods, from highly dynamic information to information that is not expected to change.
| Recency Class | Expected Time Until Answer Change |
|---|---|
| An-Hour | Within an hour |
| A-Few-Hours | Within a few hours |
| A-Day | Within a day |
| A-Few-Days | Within a few days |
| A-Week | Within a week |
| A-Few-Weeks | Within a few weeks |
| A-Month | Within a month |
| A-Few-Months | Within a few months |
| A-Year | Within a year |
| A-Few-Years | Within a few years |
| Many-Years | After many years |
| Never | Not expected to change |
The scale uses finer distinctions for rapidly changing information and progressively broader intervals at longer temporal horizons.
Evaluation
The twelve recency classes form an ordered scale. Therefore, the evaluation considers both exact classification performance and how far a prediction is from the gold class.
| Metric | Purpose |
|---|---|
| Macro-F1 | Evaluates how well the system performs across the full range of recency classes, giving equal importance to each class regardless of its frequency in the dataset. |
| Strict accuracy | Evaluates the ability of a system to identify the exact recency class associated with a question. |
| Tolerant accuracy (±1) | Evaluates whether a system identifies the correct recency region, allowing for small disagreements between adjacent classes. |
| Mean Absolute Ordinal Distance (MAOD) | Evaluates how far system predictions are from the gold classes on the ordered recency scale, distinguishing small errors from larger ones. |
| Freshness under-estimation rate | Evaluates how often a system predicts that older information is sufficient when the gold class requires fresher information. |
Dataset and splits
The campaign dataset is derived from RecencyQA and contains questions with a wide range of recency requirements, from highly time-sensitive questions that require very recent information to stable questions that can be answered using older information.
Some questions are also accompanied by context describing the situation in which the question is asked. This context may affect how recent the information needs to be and, therefore, the appropriate recency class.
| Split | Labels | Purpose |
|---|---|---|
| Training | To be announced | System development and model training. |
| Development | To be announced | Local validation and model selection. |
| Test | To be announced | Official evaluation. |
Exact split sizes, file formats, release details, and the final set of context fields will be announced with the dataset release.
Baselines
Planned starter systems include a majority-class baseline, supervised multiclass classification, ordinal classification, and zero- and few-shot language-model prompting.
Baseline implementations and evaluation scripts will be released with the campaign resources.
Submission and participation
Teams will submit one valid recency class for every test question through TIRA. The current plan permits up to three runs per task, with one primary run identified for official reporting.
- Teams may participate in Task 1 without participating in Task 2.
- Each run must use the released question identifiers and exact recency-class labels.
- Systems are required to predict a recency class; generating the factual answer is not part of Task 1.
- System descriptions should report the training data, model, prompting or optimization method, and any external resources used.
Downloads and links
Question files, baseline systems, evaluation scripts, format validators, and the TIRA task link will be published on the Resources page.
Task 1 FAQ
Must systems answer the question?
No. Systems only need to predict the recency class, which represents how recent the information needs to be to answer the question correctly.
How should systems use the provided context?
When context is provided, systems should estimate the recency requirement of the question under that specific context. Context may describe an event, stage, or circumstance that affects how recent the information needs to be. The same question may therefore have different recency requirements under different contexts.
Is a neighbouring class accepted?
Exact-match metrics such as strict accuracy require the predicted class to match the gold class. Tolerant accuracy additionally considers predictions within one adjacent class of the gold label.
Why are the classes ordered?
The classes represent progressively different levels of information freshness. Predicting a neighbouring class represents a smaller error on the recency scale than predicting a class several positions away.
Can external models or data be used?
Permitted external resources will be defined in the final participation rules and must be documented in system descriptions.