ACL 2026 Tutorial

The Data Frontier for Large Language Models

Selection, Synthesis, and Tools

July 2, 2026, 9:00 AM-12:30 PM

Lijun Wu portrait

Lijun Wu

Shanghai AI Laboratory

Young Scientist; data-centric intelligence, post-training, and evaluation

Wentao Zhang portrait

Wentao Zhang

Peking University

Assistant Professor; data-centric AI systems and executable workflows

Conghui He portrait

Conghui He

Shanghai AI Laboratory

Young Leading Scientist; open data, document intelligence, and infrastructure

Download Slides Watch Recording

Abstract

As the development of Large Language Models (LLMs) matures, the focus of the research community is undergoing a critical shift from a purely model-centric to a data-centric paradigm. It is now evident that the quality, diversity, and composition of training data—not merely its scale—are the primary drivers of a model's advanced capabilities, from complex reasoning to reliable instruction following. However, acquiring and curating such high-quality data remains a significant bottleneck.

This tutorial provides a comprehensive and practical guide to the state-of-the-art in data research directions for LLMs. We structure the tutorial around the two core pillars of modern data strategy: intelligent data selection and advanced data synthesis.

In the first part, we delve into methods for curating the most valuable information from vast, noisy datasets, covering techniques like LLM-as-a-judge for automated quality filtering and active learning for maximizing annotation efficiency. The second part explores the synthetic data revolution, detailing paradigms that range from generating complex reasoning traces, such as Chain-of-Thought, to deploying sophisticated multi-agent workflows that can autonomously create high-quality, diverse instruction data from raw seeds.

Finally, we will conclude with a practical overview of open-source tools and platforms that facilitate these data-centric workflows, empowering researchers and practitioners to build better models through better data. Attendees will leave with a principled framework and actionable insights for designing and implementing the advanced data strategies required to build the next generation of powerful, specialized, and aligned LLMs.

Tutorial Recording

The full tutorial recording is available for browser preview. The video is hosted as a GitHub Release asset, so initial loading may take a few seconds.

Open Video

Schedule

July 2, 2026, 9:00 AM-12:30 PM, with a 30-minute break from 10:30 AM to 11:00 AM.

Opening: why LLM progress is increasingly data-centric
Part 1-2: data lifecycle, selection, curation, and multimodal data filtering
Break
Part 3: data synthesis for instruction following, reasoning, multimodal tasks, and agents
Part 4: open-source tools and platforms for reproducible data work
Conclusion: data recipes, evidence, and Q&A

Outline

  1. Data-centric view How pretraining, post-training, evaluation, and deployment form a feedback loop around data.
  2. Selection and curation How quality, deduplication, diversity, safety, data value, and target relevance are measured as interventions.
  3. Data synthesis How synthetic instruction, reasoning, preference, multimodal, GUI, web, and agent data fill high-value gaps.
  4. Tools and platforms How DataFlow, MinerU, OpenDataLab, and OpenDataArena make data recipes executable, inspectable, and reusable.
  5. Tutorial takeaways How to diagnose a data problem, choose the right data operation, and look for evidence that model behavior changed.

Representative Examples

A few pages from the current deck illustrate the main arc: lifecycle framing, selection, multimodal data, our synthesis work, dataset-value benchmarking, and open-source tools.

Slide 11: The Lifecycle of an LLM Dataset

Example slide 11

The Lifecycle of an LLM Dataset

  • 1. Sourcing → Common Crawl, GitHub, arXiv, Books, Wikipedia
  • 2. Parsing → PDF/DOCX → structured text (MinerU)
Slide 24: Meta-rater [ACL 2025 Best Theme Paper]

Example slide 24

Meta-rater [ACL 2025 Best Theme Paper]

  • Meta-rater: A Multi-dimensional Data Selection Method for Pre-training LMs
  • Zhuang, Peng, Ma, Wang, Bai, Wei, Qiu, Zhang, Qian, Conghui He
Slide 38: IDEAL: Data Mixture Optimization for Multi-Capability SFT [ICLR 2026]

Example slide 38

IDEAL: Data Mixture Optimization for Multi-Capability SFT [ICLR 2026]

  • IDEAL: Data Equilibrium Adaptation for Multi-Capability Language Model Alignment
  • Ming, Qu, Cai, Pei, Pan, Li, Duan, Lijun Wu, Conghui He
Slide 45: OmniCorpus: 10B Images Interleaved with Text [ICLR 2025 Spotlight]

Example slide 45

OmniCorpus: 10B Images Interleaved with Text [ICLR 2025 Spotlight]

  • OmniCorpus: A Unified Multimodal Corpus of 10B-Level Images Interleaved with Text
  • Li, Chen, Wang, ..., Conghui He (co-corresponding), Dai
Slide 69: Caco: Code-Assisted CoT Synthesis [NeurIPS 2025]

Example slide 69

Caco: Code-Assisted CoT Synthesis [NeurIPS 2025]

  • Caco: Scaling Code-Assisted Chain-of-Thoughts for Model Reasoning
  • Lin, Pei, Gao, Pan, Li, Li, Conghui He, Lijun Wu
Slide 70: MMFineReason: Multimodal Reasoning via Data-Centric Methods [2026]

Example slide 70

MMFineReason: Multimodal Reasoning via Data-Centric Methods [2026]

  • MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods
  • Lin, Liu, Zhu, Qin, Lin, Shang, Conghui He, Wentao Zhang, Lijun Wu
Slide 84: OpenDataArena: Benchmarking Dataset Value

Example slide 84

OpenDataArena: Benchmarking Dataset Value

  • OpenDataArena: A Fair and Open Arena for Benchmarking Post-Training Dataset Value
  • Cai, Gao, Li, ..., Dahua Lin, Conghui He, Lijun Wu
Slide 103: DataFlow: Architecture Overview

Example slide 103

DataFlow: Architecture Overview

  • DataFlow: Unified LLM-Driven Framework for Data Preparation (github.com/OpenDCAI/DataFlow)

References

  1. Bai et al., 2022 Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.
  2. Bayer et al., 2024 ActiveLLM: Large Language Model-Based Active Learning for Textual Few-Shot Scenarios.
  3. Chen et al., 2023 AlpaGasus: Training a Better Alpaca with Fewer Data.
  4. Gao et al., 2020 The Pile: An 800GB Dataset of Diverse Text for Language Modeling.
  5. Gao et al., 2025 A Strategic Coordination Framework of Small LLMs Matches Large LLMs in Data Synthesis.
  6. Hubotter et al., 2024 Efficiently Learning at Test-Time: Active Fine-Tuning of LLMs.
  7. Lee et al., 2021 Deduplicating Training Data Makes Language Models Better.
  8. Li et al., 2022 Competition-Level Code Generation with AlphaCode.
  9. Liu et al., 2023 What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning.
  10. Mitra et al., 2024 AgentInstruct: Toward Generative Teaching with Agentic Flows.
  11. Mukherjee et al., 2023 Orca: Progressive Learning from Complex Explanation Traces of GPT-4.
  12. Radford et al., 2021 Learning Transferable Visual Models From Natural Language Supervision.
  13. Raffel et al., 2020 Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.
  14. Schuhmann et al., 2022 LAION-5B: An Open Large-Scale Dataset for Training Next Generation Image-Text Models.
  15. Singh et al., 2024 Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning.
  16. Touvron et al., 2023 Llama 2: Open Foundation and Fine-Tuned Chat Models.
  17. Wang et al., 2022 Self-Instruct: Aligning Language Models with Self-Generated Instructions.
  18. Wei et al., 2022 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.
  19. Wenzek et al., 2019 CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data.
  20. Xu et al., 2023 WizardLM: Empowering Large Language Models to Follow Complex Instructions.
  21. Zheng et al., 2023 Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.
  22. Zhou et al., 2023 LIMA: Less Is More for Alignment.

BibTeX

@inproceedings{wu2026datafrontier,
  title     = {The Data Frontier for Large Language Models: Selection, Synthesis, and Tools},
  author    = {Wu, Lijun and Zhang, Wentao and He, Conghui},
  booktitle = {Tutorial at the Annual Meeting of the Association for Computational Linguistics},
  year      = {2026},
  month     = {July},
  note      = {ACL 2026 Tutorial, July 2, 2026, 9:00 AM--12:30 PM},
  url       = {https://acl26datatutorial.github.io/acl-data-frontier-tutorial/}
}