Illustrative Case Study (Bad Case)

sphinx-doc__sphinx-8474 — “no number is assigned for table” warnings

A failure case where the ground-truth edit location is covered by traces, but the call chain is too deep and the model fails to surface the GT location.

← Back to Homepage · ← Back to Case Studies

Outcome

❌ Missed GT location despite coverage

The retrieved tests’ traces do cover the ground-truth edit location (StandardDomain._resolve_numref_xref), but IssueExec did not surface it due to a deep and complex execution chain.

Failure type: Deep Call Chain / Trace Complexity (GT ∈ Cov(T_d), but not recovered as L*).

Basic Information

Dataset: SWE-bench Lite
Base commit: 3ea1ec84cc610f7a9f4f6b354e264565254923ff
Instance ID: sphinx-doc__sphinx-8474
Issue link: https://github.com/sphinx-doc/sphinx/issues/8360

Ground-truth (GT)

GT edit location: sphinx/domains/std.py:StandardDomain._resolve_numref_xref
Note: Unlike a coverage-gap failure, this GT is executed by the selected tests. The failure here is that the model does not correctly recover/surface the GT from a very deep trace chain.

Issue

Original issue description

v3.3 upgrade started generating "WARNING: no number is assigned for table" warnings
We've updated to Sphinx 3.3 in our documentation, and suddenly the following warning started popping up in our builds when we build either `singlehtml` or `latex`.:

`WARNING: no number is assigned for table:`

I looked through the changelog but it didn't seem like there was anything related to `numref` that was changed, but perhaps I missed something? Could anyone point me to a change in the numref logic so I can figure out where these warnings are coming from?

Stage 1 — Relevant Test Retrieval

BM25 Top-10

tests/test_setup_command.py::test_build_sphinx_warning_return_zero_status
tests/test_setup_command.py::test_build_sphinx_warning_is_error_return_nonzero_status
tests/test_intl.py::test_text_warning_node
tests/test_domain_std.py::test_glossary_warning
tests/test_config.py::test_conf_warning_message
tests/test_util_logging.py::test_info_and_warning
tests/test_util_logging.py::test_once_warning_log
tests/test_util_logging.py::test_is_suppressed_warning
tests/test_util_logging.py::test_warning_location
tests/test_intl.py::test_xml_warnings
...
tests/test_build_latex.py::test_numref (top-27)
tests/test_build_latex.py::test_numref_with_prefix1
tests/test_build_latex.py::test_numref_with_prefix2
tests/test_build_latex.py::test_numref_with_language_ja
tests/test_build_html.py::test_numfig_with_numbered_toctree_warn

Selected T_d (LLM)

tests/test_build_latex.py::test_numref
tests/test_build_latex.py::test_numref_with_prefix1
tests/test_build_latex.py::test_numref_with_prefix2
tests/test_build_latex.py::test_numref_with_language_ja
tests/test_build_html.py::test_numfig_with_numbered_toctree_warn

These tests are semantically aligned with numref/numfig and table numbering warnings, and their traces include the relevant numbering/reference resolution pipeline.

Stage 2 — Trace-Guided Analysis

Representative test snippet (entry point)

Test: tests/test_build_latex.py::test_numref

def test_numref(app, status, warning):
    app.builder.build_all()
    result = (app.outdir / 'python.tex').read_text()
    assert ('\\hyperref[\\detokenize{index:table-1}]'
            '{Table \\ref{\\detokenize{index:table-1}}}') in result
    ...

Trace observations + suspicious mapping (summary)

Observed trace property:
- The test triggers a long LaTeX build pipeline (builder → env → transforms → domains → writer).
- The GT location is executed only after many indirections (deep call chain).

Top suspicious locations surfaced by trace-guided mapping (high-level):
- sphinx/builders/latex/__init__.py::LaTeXBuilder.write
- sphinx/environment/collectors/toctree.py::TocTreeCollector.assign_figure_numbers
- sphinx/environment/__init__.py::BuildEnvironment.resolve_references

Failure mode:
- Although the trace eventually reaches `sphinx/domains/std.py::StandardDomain._resolve_numref_xref`,
  the chain is too deep/noisy for the model to correctly surface the GT as a final edit location.

Stage 3–4 — Contextual Refinement & Code-Aware Reranking

Contextual refinement performs supplemental retrieval around suspicious locations (e.g., class/module context), so the reranker can see the broader implementation surface beyond the raw trace nodes (which are often too granular or too many).

Contextual refinement (module/class context retrieved)

sphinx/environment/__init__.py::BuildEnvironment
sphinx/builders/html/__init__.py::StandaloneHTMLBuilder
sphinx/environment/collectors/toctree.py::TocTreeCollector
sphinx/builders/latex/__init__.py::LaTeXBuilder

Final reranked edit locations (L*)

(1) sphinx/environment/collectors/toctree.py::TocTreeCollector.assign_figure_numbers
(2) sphinx/environment/__init__.py::BuildEnvironment.get_doctree
(3) sphinx/environment/__init__.py::BuildEnvironment.resolve_references

The reranked results are plausible upstream “pipeline” components, but still do not include the GT edit location StandardDomain._resolve_numref_xref. This illustrates a limitation: when the root-cause hook sits deep in a long chain, the trace-constrained reasoning can become dominated by higher-level orchestration nodes (builders/env), failing to surface the precise domain-level resolver.

Takeaway

This case highlights a limitation different from coverage gaps: even if a correct edit location is executed by the selected tests, a deep and complex call chain can prevent the model from surfacing the correct GT location as the final edit target.