TL;DR
The paper introduces CWE-Trace, a framework for diagnosing LLM vulnerability detection, showing that fine-tuning calibrates outputs without fundamentally improving security reasoning.
Problem
It is unclear whether LLMs' strong performance on vulnerability benchmarks stems from genuine security reasoning or from pattern-matching on contaminated training data.
Approach
Develop CWE-Trace, a framework with 834 manually curated Linux kernel samples across 74 CWEs.
Enforce strict temporal splits and context-aware vulnerable-patched pairs.
Introduce two diagnostic metrics: Directional Failure Index (DFI) and Hierarchical Distance and Direction (HDD).
Evaluate 8 vanilla LLMs and 15 LoRA fine-tuned variants on non-targeted detection, targeted detection, and CWE classification.
Results & Contribution
Data contamination provides no measurable advantage: 84% of contaminated samples carry no usable memorization signal.
Fine-tuning shifts output thresholds without changing the decision policy (backbone directional priors dominate).
Detection and understanding are decoupled capabilities: the weakest detection model shows the most improvement in CWE classification.