Illustrative Case Study (Bad Case)

django__django-10924 — Callable path for FilePathField

A failure case where the ground-truth edit location is not covered by the retrieved tests’ execution traces.

← Back to Homepage  ·  ← Back to Case Studies

Outcome

❌ Missed GT location due to coverage gap
IssueExec’s trace-guided localization could not reach the ground-truth edit location (FilePathField.formfield) because it was not executed by the selected tests (Td).

Failure type: Coverage Gap (GT location not in Cov(Td)).

Basic Information

Ground-truth (GT)
  • GT edit location: django/db/models/fields/__init__.py:FilePathField.formfield
  • Note: In this case, the selected tests did not execute FilePathField.formfield, so trace-guided localization could not include it in the suspicious set.

Issue

Original issue description 
Allow FilePathField path to accept a callable.
Description

I have a special case where I want to create a model containing the path to some local files on the server/dev machine. Seeing as the place where these files are stored is different on different machines I have the following:
import os
from django.conf import settings
from django.db import models
class LocalFiles(models.Model):
    name = models.CharField(max_length=255)
    file = models.FilePathField(path=os.path.join(settings.LOCAL_FILE_DIR, 'example_dir'))
Now when running manage.py makemigrations it will resolve the path based on the machine it is being run on. Eg: /home/<username>/server_files/example_dir
I had to manually change the migration to include the os.path.join() part to not break this when running the migration on production/other machine.

Stage 1 — Relevant Test Retrieval

BM25 Top-10 
tests/model_fields/test_promises.py::PromiseTest::test_FilePathField
tests/forms_tests/field_tests/test_filepathfield.py::FilePathFieldTest::test_recursive_no_folders_or_files
tests/forms_tests/field_tests/test_filepathfield.py::FilePathFieldTest::test_recursive_folders_without_files
tests/utils_tests/test_autoreload.py::TestIterModulesAndFiles::test_paths_are_pathlib_instances
tests/forms_tests/field_tests/test_filepathfield.py::FilePathFieldTest::test_fix_os_paths
tests/test_runner/test_discover_runner.py::DiscoverRunnerTests::test_file_path
tests/forms_tests/field_tests/test_filepathfield.py::FilePathFieldTest::test_allow_folders
tests/forms_tests/field_tests/test_filepathfield.py::FilePathFieldTest::test_clean
tests/forms_tests/field_tests/test_filepathfield.py::FilePathFieldTest::test_match
tests/forms_tests/field_tests/test_filepathfield.py::FilePathFieldTest::test_recursive
Selected Td (LLM) 
tests/model_fields/test_promises.py::PromiseTest::test_FilePathField
tests/forms_tests/field_tests/test_filepathfield.py::FilePathFieldTest::test_recursive_no_folders_or_files
tests/forms_tests/field_tests/test_filepathfield.py::FilePathFieldTest::test_recursive_folders_without_files
tests/forms_tests/field_tests/test_filepathfield.py::FilePathFieldTest::test_fix_os_paths
tests/forms_tests/field_tests/test_filepathfield.py::FilePathFieldTest::test_allow_folders

These tests are semantically related to FilePathField. However, they do not exercise the GT edit location FilePathField.formfield in the observed trace coverage.

Stage 2 — Trace-Guided Analysis

Example test snippet + partial coverage excerpt 
Test: tests/model_fields/test_promises.py::PromiseTest::test_FilePathField

def test_FilePathField(self):
    lazy_func = lazy(lambda: 'tests.py', str)
    self.assertIsInstance(FilePathField().get_prep_value(lazy_func()), str)
    lazy_func = lazy(lambda: 0, int)
    self.assertIsInstance(FilePathField().get_prep_value(lazy_func()), str)

Coverage excerpt:
- django/db/models/fields/__init__.py::FilePathField.__init__
- django/db/models/fields/__init__.py::FilePathField.get_prep_value
(...)
Trace-guided suspicious mapping (summary) 
Key suspicious locations (from covered traces):
- django/db/models/fields/__init__.py::FilePathField.__init__
- django/db/models/fields/__init__.py::FilePathField.get_prep_value
- django/forms/fields.py::FilePathField (init/choices-related logic)

Coverage limitation:
- NOT_IN_COVERAGE: GT edit location `django/db/models/fields/__init__.py:FilePathField.formfield` is not executed by selected tests, so it cannot be proposed by trace-constrained localization.

Stage 3–4 — Contextual Refinement & Code-Aware Reranking

Contextual refinement expands the candidate set by retrieving structurally related context (e.g., the containing class/module) for suspicious locations discovered from traces. This helps the reranker see the full API surface and nearby logic even when the trace stage only pinpoints coarse-grained entities.

Contextual refinement (module/class context retrieved) 
django/forms/fields.py::FilePathField
django/db/models/fields/__init__.py::FilePathField
Final reranked edit locations (L*) 
(1) django/db/models/fields/__init__.py::FilePathField
(2) django/forms/fields.py::FilePathField

Although these are plausible “surface” candidates for callable path handling, the GT method FilePathField.formfield is not included because it is not covered by the selected tests’ execution traces in this case.

Baseline (SWE-agent) — Failure/Success Summary

In contrast, SWE-agent reached the GT location by using content-based search in the repository (rather than relying on trace coverage).

Compact bullet summary
  • File-name hypothesis failed: searched for FilePathField.py (no match), then broadened to fields.py.
  • Key pivot: searched the codebase for class FilePathField and found django/db/models/fields/__init__.py.
  • Precise navigation: jumped to the class definition and inspected methods.
  • Located GT: identified and edited FilePathField.formfield (the ground-truth edit location).

Takeaway

This case highlights a key limitation of trace-guided localization: if the retrieved tests do not execute the true edit location, the GT cannot appear in the suspicious set. This failure mode aligns with our discussion of limitations related to incomplete coverage.