Home / Analysis / Empirical Study

Empirical Study

Empirical validation for the premise that tests act as executable requirements for issue localization. We evaluate three hypotheses: H1 Coverage feasibility, H2 Retrievability, and H3 Bridging effect.

← Back to Homepage

One-screen takeaway

Summary: Existing tests frequently reach ground-truth edits (H1), covering tests are more semantically aligned with issue descriptions than non-covering tests (H2), and the two-hop pathway Issue $\rightarrow$ Test $\rightarrow$ Location provides stronger semantic connectivity than direct Issue $\rightarrow$ Location matching (H3).

Key results (paper-ready)

H1: Mean coverage: 96.98% (file), 66.70% (function). Search space reduces to 58.39% on average.
H2: Covering tests win in 90.9% of instances (paired t-test $p < 0.001$; Cohen’s $d \approx 0.56$). Correlation: Spearman’s $\rho \approx 0.18$ ($p < 0.001$).
H3: Mediated pathway is stronger in 82.4% of (issue, location) pairs (paired t-test $p < 0.001$; Cohen’s $d \approx 0.93$).

Where this fits in the artifact

System pipeline overview: see the paper Section 4 and the Case Studies.
This page focuses on: empirical evidence and how it informs IssueExec’s design.

Quick links

1) Setup
2) Dynamic trace collection
3) H1 — Coverage feasibility
4) H2 — Retrievability
5) H3 — Bridging effect
6) Implications for IssueExec
7) Notes & limitations

1) Setup

We study whether tests can serve as executable requirements for localizing software engineering issues. The dataset includes 929 issue instances drawn from SWE-bench and SWE-bench Gym across 18 popular Python repositories. Each repository-issue pair is executed in a containerized environment reset to its pre-patch state. To prevent environment configuration errors from confounding results, we retain instances where the test environment is sufficiently robust (at least 50% of test functions execute successfully).

Ground truth (edit locations) Ground-truth edits are extracted from PR diffs associated with each issue, mapped to function-level locations using fully-qualified identifiers (module / class / function).

Semantic analysis We compute cosine similarity between embeddings of issue descriptions, tests, and code locations. Tests are represented by a textual form (e.g., signature/doc/body) truncated to fit embedding input limits. Code locations are represented by their fully-qualified signatures.

2) Dynamic trace collection

We collect function-level execution traces during test runs and retain only repository-internal functions. Standard library and third-party dependency calls are filtered out, and remaining functions are represented by their fully-qualified paths.

Architecture of the dynamic coverage collection framework (placeholder path)

Figure: Architecture of the dynamic coverage collection framework. The system operates in four phases: (1) parallel container startup with volume mounting, (2) in-container test execution with instrumentation, (3) trace output collection, and (4) result aggregation and formatting.

What is counted as "covered"?

A code location is considered covered if it appears in the repository-internal function call trace of at least one executed test. Coverage is evaluated at three granularities: file, module (class or top-level function), and function. For search space reduction, we compare the size of the union of test-covered internal functions to the total number of internal functions in the repository for each instance.

3) H1 — Coverage feasibility

Hypothesis H1: Existing tests execute the ground-truth edit locations.

H1 evaluates whether dynamic execution signals already exist in typical repositories. By intersecting test traces with the ground-truth edit sets, we measure how often tests reach the true edit locations and how much the candidate space can be compressed by trace-induced constraints.

Summary statistics (929 instances)

Metric	Mean	Median	Std	Min	Max
File-level coverage	0.9698 (96.98%)	1.0000 (100.00%)	0.1550	0.0000	1.0000
Module-level coverage	0.7804 (78.04%)	1.0000 (100.00%)	0.3933	0.0000	1.0000
Function-level coverage	0.6670 (66.70%)	1.0000 (100.00%)	0.4178	0.0000	1.0000
Search space (union coverage / total)	0.5839 (58.39%)	0.6188 (61.88%)	0.2268	0.1117	0.8523

Threshold hit rates

Granularity	≥ 75%	≥ 90%
File-level	898 / 929 (96.7%)	886 / 929 (95.4%)
Module-level	700 / 929 (75.3%)	686 / 929 (73.8%)
Function-level	546 / 929 (58.8%)	504 / 929 (54.3%)

CDF of coverage rates and space reduction (placeholder path)

Figure (H1): Cumulative distribution of test coverage rates and trace-induced search space reduction. Existing tests cover most ground-truth files and a substantial fraction of ground-truth functions, while compressing the candidate space for downstream localization stages.

Raw report (H1)

RQ1: Coverage Statistics Report (Generated: 2026-01-05)
- Total Instances: 929
- File-level Mean: 0.9698
- Module-level Mean: 0.7804
- Function-level Mean: 0.6670
- Space Reduction Mean: 0.5839

4) H2 — Retrievability

Hypothesis H2: Tests that cover ground-truth locations are more semantically aligned with the issue than non-covering tests.

H2 checks whether covering tests are distinguishable from non-covering tests via semantic signals. We compare similarity between the issue description and (i) tests that cover at least one ground-truth location and (ii) baseline tests sampled from the same repository that do not cover ground-truth locations.

Instance-level statistics (recommended)

Statistic	Value	Notes
Win rate (covering > non-covering mean)	90.9%	Across valid instances, covering tests show higher similarity than non-covering tests.
Paired t-test (per-instance means)	$p < 0.001$ (t = 38.8015)	Significant advantage of covering tests over non-covering tests.
Effect size (Cohen’s d)	0.5573	Medium-to-large practical effect.
Correlation (#GT covered vs similarity)	$\rho = 0.1809$, $p < 0.001$	Tests covering more ground-truth locations tend to be more semantically aligned with the issue.

Semantic similarity of covering vs non-covering tests (placeholder path)

Figure (H2): Semantic similarity of tests to issue descriptions. Covering tests exhibit higher similarity than non-covering tests, enabling retrieval-based selection.

Raw report (H2)

RQ2: Semantic Similarity Statistics Report (Generated: 2026-01-05)
- Valid instances: 536
- Win rate: 90.9%
- Paired t-test p-value: 1.203337e-157
- Cohen's d: 0.5573
- Spearman ρ: 0.1809 (p < 0.001)

Why instance-level analysis?

Similarity distributions vary across repositories and test suites. Comparing covering vs non-covering tests within each issue instance provides a more stable estimate of semantic distinguishability and reduces cross-project confounding.

5) H3 — Bridging effect

Hypothesis H3: The two-hop pathway Issue $\rightarrow$ Test $\rightarrow$ Location provides stronger semantic connectivity than direct Issue $\rightarrow$ Location matching.

For each (issue, location) pair where coverage exists, we compare direct similarity $s_{\text{direct}}=\mathrm{sim}(\text{issue}, \text{location})$ against a test-mediated pathway strength:

$$s_{\text{mediated}} = \max_{t \in \mathcal{T}_{\text{cover}}} \sqrt{\mathrm{sim}(\text{issue}, t)\cdot \mathrm{sim}(t,\text{location})}$$

This geometric-mean form ensures a strong bridge requires balanced semantic associations across both hops: Issue $\rightarrow$ Test and Test $\rightarrow$ Location.

Summary statistics (2416 pairs)

Statistic	Value	Notes
Direct pathway mean (std)	0.6745 (0.0681)	$s_{\text{direct}}$
Mediated pathway mean (std)	0.7271 (0.0576)	$s_{\text{mediated}}$
Win rate (mediated > direct)	82.4% (1991 / 2416)	Bridging effect holds for most pairs.
Paired t-test	$p < 0.001$ (t = 45.4641)	Highly significant difference.
Wilcoxon signed-rank	$p < 0.001$ (W = 2,633,234)	Non-parametric confirmation.
Effect size (Cohen’s d)	0.9251 (large)	Substantial practical improvement.

Direct vs mediated pathway scatter (placeholder path)

Figure (H3): Scatter plot of direct vs mediated pathway strength. Most points lie above the diagonal, indicating $s_{\text{mediated}} > s_{\text{direct}}$ is common.

Distribution of mediated-direct difference (placeholder path)

Figure (H3): Distribution of $\Delta = s_{\text{mediated}} - s_{\text{direct}}$ across pairs. The mass concentrates on $\Delta > 0$, consistent with the bridging effect.

Raw report (H3)

RQ3: Tests as Semantic Bridges (Generated: 2026-01-07)
- Pairs analyzed: 2416
- direct_mean: 0.6745, mediated_mean: 0.7271
- Win rate: 82.4%
- Paired t-test: p < 0.001, Cohen's d: 0.9251

Representative examples (format)

You can optionally place a small table of representative (issue, location) pairs here (e.g., a few with large positive deltas and a few with negative/near-zero deltas). Keep it short to avoid overwhelming the page.

Instance	Location	$s_{direct}$	$s_{mediated}$	Best test
`pytest-dev__pytest-11143`	`src/_pytest/assertion/rewrite.py::AssertionRewriter.run`	0.7222	0.7567	`testing/test_assertrewrite.py::TestAssertionRewrite.test_rewrites_plugin_as_a_package`
`sphinx-doc__sphinx-8548`	`sphinx/ext/autodoc/__init__.py::ClassDocumenter.get_object_members`	0.7573	0.8320	`tests/test_ext_autodoc.py::test_autodoc_inherited_members`

6) Implications for IssueExec

The empirical findings directly inform IssueExec’s design choices:

H1 → Trace-constrained search space High coverage feasibility implies traces can provide a high-recall candidate subspace with reduced entropy, enabling more efficient localization stages downstream.

H2 → Retrieval-based test selection Since covering tests tend to be more semantically aligned with the issue, semantic retrieval can prioritize requirement-relevant tests. The moderate effect size motivates domain knowledge enhancement to further strengthen Issue → Test alignment.

H3 → Two-hop localization strategy Tests act as semantic relays that link requirement-level language to code locations more effectively than direct matching, validating the Issue → Tests → Code decomposition.

Trace noise → Hierarchical trace analysis The gap between covering and non-covering code within traces motivates trace-guided pruning and hierarchical analysis to filter infrastructure noise and retain requirement-central functions.

7) Notes & limitations

Coverage feasibility varies by granularity: file-level coverage is typically higher than function-level coverage, which is expected because a single file may contain multiple functions and only some are executed by existing tests.
Semantic retrievability is evaluated with within-instance comparisons to mitigate cross-project distribution shifts in test suites.
The bridging effect is evaluated only on pairs where coverage exists (i.e., at least one covering test is available).

Where to see concrete end-to-end examples

See Case Studies for illustrative good and bad cases, with intermediate artifacts (retrieved tests, trace-guided candidates, and final ranked locations).