Home / Analysis / Theoretical Motivation / Case Study

Case Study: xarray-Issues-3992

Failure mode: coverage gap ($g_{excl} > 0$) + weak test selection ($n/k_{test}$ small) causes $H_{indirect} > H_{direct}$, yielding negative gain $\Delta H$.

instance_id: pydata__xarray-3993 $k_{test}=5$ $k_{ret}=10$ $gt\\_threshold=1.0$ ext_failure: true

Issue summary

Dataset.integrate uses coord while DataArray.integrate uses dim; API inconsistency. The desired fix is to align the API (likely use coord consistently) and handle deprecation concerns.

Problem statement (raw excerpt)

DataArray.integrate has a 'dim' arg, but Dataset.integrate has a 'coord' arg

The API syntax is inconsistent:
ds.differentiate(coord='x')
da.differentiate(coord='x')
ds.integrate(coord='x')
da.integrate(dim='x')   # why dim??

It should definitely be `coord` ...
The only question is whether it requires a deprecation cycle?

One-screen takeaway

What went wrong: only 2/4 ground-truth locations fall inside the trace subspace induced by retrieved tests, so we incur an external penalty $g_{excl}=2$. Meanwhile only 1 of the 5 retrieved tests is effective, so stage 1 also costs extra entropy.

Observed entropies $H_{direct}=12.7035$
$H_{indirect}=14.2567$
$\Delta H = -1.5533$

Failure signatures $n=1$ effective test out of $k_{test}=5$
$N_{cov}=271 \ll N=2011$ (compression exists)
but $g_{excl}=2$ (coverage gap dominates)

Key quantities

Symbol	Value	Meaning
$N$	2011	Total candidate locations (functions/methods)
$k_{ret}$	10	Return budget for code locations
$k_{test}$	5	Return budget for tests
$\|G\|$	4	# ground-truth edit locations
$N_{cov}$	271	Trace subspace size ($\|L_{cov}\|$)
$g_{ret}^{direct}$	1	GT hit count in direct top-$k_{ret}$
$n$	1	# effective tests among top-$k_{test}$
$g_{ret}^{indirect}$	2	GT hit count in indirect top-$k_{ret}$ (within $L_{cov}$)
$g_{excl}$	2	GT outside trace coverage ($g_{ext}$ in script)

Note: Compression exists ($N_{cov}\ll N$), but it’s not sufficient: uncovered GT introduces a penalty term in stage 2.

Direct vs indirect: why indirect can be worse

Stage 1 cost: $H_{stage1}=-\log_2(n/k_{test})=-\log_2(1/5)=2.3219$.
Stage 2 cost: because $g_{excl}>0$, stage 2 includes a term roughly $-\log_2(g_{excl}/N_{excl})$ with $N_{excl}=N-N_{cov}$, which can be large.
Net effect: $H_{indirect}=H_{stage1}+H_{stage2}$ becomes larger than $H_{direct}$.

Ground-truth and coverage gap

GT locations

xarray/core/dataarray.py::DataArray.integrate
xarray/core/dataset.py::Dataset.integrate
xarray/core/dataarray.py::DataArray.map_blocks
xarray/core/dataarray.py::DataArray.unify_chunks

Uncovered GT (outside trace coverage)

These two are not covered by the union trace subspace from retrieved tests (hence $g_{excl}=2$):

xarray/core/dataarray.py::DataArray.map_blocks
xarray/core/dataarray.py::DataArray.unify_chunks

Interpretation

Even if test-driven localization ranks well within $L_{cov}$, it cannot “see” edits outside $L_{cov}$. When the evaluation requires covering all GT locations (here gt_threshold=1.0), uncovered GT forces an external penalty.

Retrieved sets

Selected tests (top-$k_{test}=5$)

xarray/tests/test_dataset.py::test_differentiate
xarray/tests/test_dataset.py::test_integrate   (effective)
xarray/tests/test_dataset.py::TestDataset.test_expand_dims_mixed_int_and_coords
xarray/tests/test_concat.py::test_concat_merge_single_non_dim_coord
xarray/tests/test_dataset.py::TestDataset.test_dataset_diff_n2

Effective tests Only 1 test covers any GT:

xarray/tests/test_dataset.py::test_integrate

Therefore $H_{stage1}=-\log_2(1/5)=2.3219$.

Trace subspace size $N_{cov}=271$ (union of coverage over selected tests).
$N_{excl}=N-N_{cov}=2011-271=1740$.

Direct top-$k_{ret}=10$ functions

Show list

xarray/core/coordinates.py::DataArrayCoordinates.dims
xarray/core/coordinates.py::DatasetCoordinates.dims
xarray/core/dataarray.py::DataArray.differentiate
xarray/core/dataset.py::Dataset.differentiate
xarray/core/dataarray.py::_infer_coords_and_dims
xarray/core/dataset.py::Dataset.dims
xarray/core/dataarray.py::DataArray.dims
xarray/core/dataarray.py::DataArray.dims
xarray/core/dataarray.py::DataArray.coords
xarray/core/dataarray.py::DataArray.integrate

Indirect top-$k_{ret}=10$ functions (within $L_{cov}$)

Show list

xarray/core/dataarray.py::DataArray.differentiate
xarray/core/dataset.py::Dataset.differentiate
xarray/core/dataarray.py::_infer_coords_and_dims
xarray/core/dataset.py::Dataset.dims
xarray/core/dataarray.py::DataArray.dims
xarray/core/dataarray.py::DataArray.dims
xarray/core/dataarray.py::DataArray.coords
xarray/core/dataarray.py::DataArray.integrate
xarray/core/dataset.py::Dataset.coords
xarray/core/dataset.py::Dataset.integrate

Entropy terms (numbers used in the artifact)

Direct Inputs: $N=2011$, $k_{ret}=10$, $|G|=4$, $g_{ret}^{direct}=1$.

$$H_{direct} = -\log_2\left(\frac{1}{10}\right) -\log_2\left(\frac{3}{2011-10}\right) \approx 12.7035$$

(Term 1: ambiguity among returned hits; Term 2: residual uncertainty for the remaining GT outside the returned set.)

Indirect Stage 1: $$H_{stage1}=-\log_2\left(\frac{n}{k_{test}}\right)= -\log_2(1/5)=2.3219$$ Stage 2 (script output): $$H_{stage2}\approx 11.9348$$ Therefore: $$H_{indirect}=H_{stage1}+H_{stage2}\approx 14.2567$$

$$\Delta H = H_{direct}-H_{indirect}\approx -1.5533$$

What exactly drives $H_{stage2}$ up here?

We have compression: $N_{cov}=271$ and $N_{excl}=1740$.
But uncovered GT yields $g_{excl}=2$, which contributes a large penalty term of the form $-\log_2(g_{excl}/N_{excl})$.
In addition, the GT distribution across partitions (ret/cov/excl) affects the sum of terms.

This is the canonical “coverage gap” bad case: trace-constrained localization can’t win if a meaningful portion of GT lies outside traces.