Fault Localization on Verification Witnesses (Poster Paper)

Verifiers export violation witnesses, which help independent validators to confirm a reported specification violation. It is assumed that violation witnesses are helpful if they are very precise: ideally, they should describe a single program path for the validator to check. But we claim that this leads verifiers to produce large, detailed witnesses that include a lot of unnecessary information that actually hinders validation. We reduce violation witnesses with automated fault localization to only that information which fault localization suspects as fault. We performed a large experimental evaluation on the witnesses produced in the International Competition on Software Verification (SV-COMP 2023) to explore the effect of our reduction. Our experiments show that the witnesses reduced using our approach shrink considerably and can be confirmed better.


INTRODUCTION
When a formal software verifier reports an alarm, it produces a violation witness [6] to increase the confidence in its result.A violation witness describes a subset of program paths of which at least one contains the reported specification violation.This helps to reason about the alarm and allows to validate the alarm.All participants of the International Competition on Software Verification (SV-COMP) [3] produce such violation witnesses.
Contrary to a test case, violation witnesses are hardly readable for humans and can grow to thousands of lines.Additionally, we observe that witnesses contain unnecessary states and assumptions.We solve these problems through fault localization: Given a violation witness, we reconstruct a potential error path, apply fault localization to get suspects for faults along the path, and delete Figure 1: Workflow of fault localization on violation witnesses; after creating the product automaton of specification , witness  , and program , we apply fault localization on the obtained error path   to find suspects   ; these are used to reduce the violation witness  to  ′ all edges that are deemed irrelevant from the witness.Figure 1 illustrates the workflow of our approach.
Related Work.Many fault-localization techniques exist [1,10,13,14,17].Other approaches [5,11,15,16] provide step-by-step simulators for error paths.We reduce the size of witnesses to relevant information beyond input values.This works on witnesses that miss input values, and makes simulators focus on relevant information.

FAULT LOCALIZATION ON VIOLATION WITNESSES
Program Representation.Our implementation and experiments work on GNU C programs.For presentation, we consider an imperative, sequential programming language with two types of operations: assign operation (x = x + 1) and assume operation ([x <= 0]).Ops is the set of all valid operations.We represent programs as control-flow automata (CFA).A CFA (,  0 , ) consists of program locations , initial location  0 ∈  and edges  = ×Ops×.Our Witness Reduction.Figure 1 illustrates the workflow of our approach.Given a program , a violation witness , and a specification we build the product automaton, extract the error path   , run fault localization on it, and produce a set of suspects.We then take the suspects and the original witness to obtain the reduced witness  ′ .A transition is irrelevant if it is not part of the suspects.We experiment with three reduction strategies: Strategy r all deletes all irrelevant edges that contain no information about program branches.Strategy r state deletes irrelevant state-space guards and keeps all source-code guards.Strategy r match turns irrelevant statespace guards into trivial  state-space guards.The witnesses still steer the validation.We proved that all three strategies are sound.

EVALUATION
Experiment Setup.We conduct the experiments on machines with an Intel Xeon E3-1230 v5 @ 3.40 GHz 8-core processor and 33 GB memory, limited to 2 cores and 7 GB of memory.The timeout for validation is set to 90 s.This aligns with the SV-COMP 2023 setup.We use all violation witnesses [4] that were produced by 14 nonhors-concours verifiers on 3 225 non-recursive unsafe verification tasks with property unreach-call of SV-COMP 2023.For witness reduction, we use CPAchecker in revision 44 191 and our tool Flow 1 in revision cc5e4f8d.For witness validation, we use four validators from SV-COMP 2023: CPAchecker [7], UAutomizer [12], MetaVal [9], and Symbiotic-Witch [2].The presented data exclude witnesses for which the respective fault-localization approach did not work.Results.With reduction r all , we decrease the number of transitions in witnesses to about 45 % with MaxSat and MinUnsat, and to 43 % with Unsat-on average, across the three fault localization techniques (see Table 1).We exemplarily show data for the validators MetaVal and UAutomizer after applying r all and r state , respectively (Figs. 2 and 3).Our reduction techniques has positive and negative effects.However, up to 32 % more witnesses can be confirmed after reduction.

CONCLUSION
Applying fault localization to violation witnesses is an effective and sound way to increase the confirmation rate and to lower the number of unnecessary transitions in the witness automaton.Next to boosting the performance of validators, our approach makes witnesses easier to store and comprehend.
Data-Availability Statement.The experiment setup and all experimental data are archived and available at Zenodo [8].

Fault
Localization.A suspect is a set of program lines that, together, may be responsible for a program error.Given a feasible error path that ends in a program error, fault localization determines a finite set F = { 0 ,  1 , . . .,   } of suspects.All three different fault-localization techniques that we consider work on error paths with negated final assumption, yielding an infeasible error path: MaxSat[14] finds the maximum satisfiable subsets of the infeasible error path and returns the complement as suspects.MinUnsat computes minimal unsat cores as suspects.Unsat returns an arbitrary unsat core and serves as baseline in our experiments.Violation Witness.A violation witness describes program executions of which at least one leads to a specification violation by restricting the set of all possible program executions through source-code guards and state-space guards: Source-code guards restrict the control flow, and state-space guards restrict the potential program states.A witness validator[6] checks whether a violation witness describes any program execution that reaches the claimed specification violation.If it does, the violation witness is confirmed.If it does not, the violation witness is rejected.ICSE 2024, April 2024, Lisbon, Portugal Dirk Beyer, Matthias Kettl, and Thomas Lemberger

Table 1 :
Remaining transitions after reduction with r all