New pilot evaluation labCLEF 2027

TemporalWebLab

Evaluating Recency and Change across the Web

  • 1 University of Innsbruck, Austria
  • 2 Internet Archive, USA
  • 3 University of Beira Interior / INESC TEC, Portugal
  • 4 Delft University of Technology, Netherlands
Explore the tasks How to participate Join our Google Group

Latest updates

TemporalWebLab accepted at CLEF 2027

View all updates
Why TemporalWebLab

Understanding recency and change

Information changes at different speeds. Some answers may need updating within hours, while others remain valid for years. At the same time, Web pages are revised, replaced, or removed, making it difficult to establish what information was available at a particular time and how it changed. Web archives preserve captures of these pages, providing evidence for answering questions about the past and comparing information across time.

TemporalWebLab addresses these challenges through two complementary tasks. Task 1: Question Recency Requirement Estimation asks systems to estimate the expected validity horizon of a currently correct answer under the given temporal context. Task 2: Question Answering over Web Archives asks systems to answer questions about information at specific times and changes across archived versions, supported by evidence from the archive collection.

Information RetrievalQuestion AnsweringWeb ArchivesTemporal ReasoningRAGLLMs
Two complementary challenges

Evaluation Tasks

Participate in either task independently or in both.

TASK 1
Classification · Ordinal prediction

Question Recency Requirement Estimation

Given a natural-language question and, where provided, its temporal context, estimate how long the currently correct answer is expected to remain valid.

Output
1 of 12 ordered validity-horizon classes
Primary metric
Macro-averaged F1
Data
Adapted from RecencyQA
View the 12 classes
  1. An hour
  2. A few hours
  3. A day
  4. A few days
  5. A week
  6. A few weeks
  7. A month
  8. A few months
  9. A year
  10. A few years
  11. Many years
  12. Never
View Task 1 details
TASK 2
Retrieval · Evidence-grounded QA

Question Answering over Web Archives

Retrieve temporally appropriate evidence from a frozen archive collection and answer questions about what information was online at a specified time or how it changed.

Output
Answer + supporting record IDs
Primary metric
Grounded answer correctness
Corpus
EOT and selected archive captures
Point in timeFirst / last observedCompare versionsVerify changeTrace changes
View Task 2 details
Campaign schedule

Important dates

Exact dates and remaining deadlines will be announced here.

01

Campaign launch

Website, initial documentation, and participant communication channels.

02

Resources released

Training data, baselines, and documentation.

03

Registration

Participant registration for the CLEF evaluation campaign.

04

Evaluation begins

Test questions released and TIRA submissions opened.

05

Run submission

Official runs submitted and evaluated.

06

Results

Results released and invitations to submit CLEF Working Notes papers.

07

CLEF 2027

TemporalWebLab session and presentation of official findings.

Participant resources

Datasets, baselines, and evaluation tools

Datasets, baseline systems, evaluation scripts, and submission instructions will be provided here as they become available.

DDatasetsTraining and development dataDetails BBaselinesStarter systems for both tasksDetails EEvaluationScorers and format validatorsDetails TTIRAOfficial run submission platformDetails

Organizing Team

Bhawna Piryani

Bhawna Piryani

University of Innsbruck
Jamshid Mozafari

Jamshid Mozafari

University of Innsbruck
Sawood Alam

Sawood Alam

Internet Archive
Ricardo Campos

Ricardo Campos

University of Beira Interior / INESC TEC
Avishek Anand

Avishek Anand

Delft University of Technology
Adam Jatowt

Adam Jatowt

University of Innsbruck

Frequently Asked Questions

Detailed participation rules will be added as the campaign resources are finalized.

Can we participate in only one task?+

Yes. Teams may participate in either Task 1 or Task 2 independently, or enter both.

How many runs can a team submit?+

Each team may submit up to three runs per task and designate one primary run for the official leaderboard.

Where will submissions be handled?+

Official submissions will be handled through TIRA to support secure, reproducible, and comparable evaluation.

When will data and baselines be available?+

The initial release is planned for October 2026. Exact release dates will be announced here.

Participate in TemporalWebLab

We welcome participation from researchers and practitioners interested in estimating question recency requirements and answering questions over Web archives.

Join our Google Group
  1. 1
    Choose a taskParticipate in Task 1, Task 2, or both.
  2. 2
    Follow updatesReceive announcements about resources, registration, and important deadlines.
  3. 3
    Participation detailsRegistration and submission information will be announced when confirmed.
Participation details