DS-Lighting: Making Agent Harnesses Explicit for Data-Science Automation

A new toolkit called DS‑Lighting has been released to make the “harness” that underpins large‑language‑model (LLM) agents for data‑science automation explicit rather than implicit. The harness, which defines how tasks are represented, how execution state is managed, how output artifacts are constrained and how evaluation feedback is provided, has historically been hidden inside agents, hampering reproducibility and comparison across heterogeneous workflows. DS‑Lighting decomposes this harness into four reusable layers—data, workflow, execution and evaluation—and encodes diverse agents as executable operator programs that can run both fixed pipelines and adaptive search strategies. The toolkit also bundles several open‑source data‑science benchmarks into a unified MLE‑Bench‑style task format, offering a shared interface, sandboxed runtime and common metric protocol for systematic evaluation.

The primary benefit of making the harness explicit is a measurable improvement in reproducibility, comparability and reliability of end‑to‑end data‑science pipelines. Experiments that varied agents, harness configurations, underlying LLMs and ablation settings demonstrated fewer system‑level failures and more consistent results when the harness was defined according to DS‑Lighting’s layered schema. By providing a standardized way to describe and enforce task constraints, the toolkit enables researchers to isolate the contribution of the LLM agent itself from the surrounding infrastructure, facilitating fairer benchmarking and more transparent attribution of performance gains.

The release includes open‑source code hosted on GitHub, inviting the community to adopt and extend the framework. With DS‑Lighting, developers can plug in new benchmarks, customize harness layers, and share operator programs that adhere to the common task format, potentially accelerating progress in automated data‑science research. The authors anticipate that broader adoption will reduce the prevalence of avoidable failures in complex workflows, lower the barrier to reproducing published results, and create a more collaborative ecosystem for evaluating and improving LLM‑driven data‑science agents.

⚡ Effects Interpreter