Abstraction by Term Rewriting for Malware Behavior Analysis - Extended Version

Philippe Beaucamps (1) , Isabelle Gnaedig (1) , Jean-Yves Marion (1)

1 CARTE - Theoretical adverse computations, and safety

We propose a formal approach for behavioral analysis of programs based on dynamic analysis. It works by abstracting execution traces with respect to given behavior patterns in order to produce a high level representation of a program behavior and then, by comparing this abstract form to signatures defining reference abstract malicious behaviors. Abstraction is performed by term rewriting using rules on terms with variables, which enables to handle the data used by behavior functionalities. This technique allows us to deal with interleaved behaviors. Successfully applied to malware detection, it allows us in particular to model and detect information leak.

Mots clés: malware; behavioral detection; abstraction; trace; term rewriting; finite state automaton; formal language; dynamic binary instrumentation; temporal logic; information leak

Domaines: Cryptographie et sécurité [cs.CR]

Philippe Beaucamps, Isabelle Gnaedig, Jean-Yves Marion. Abstraction by Term Rewriting for Malware Behavior Analysis - Extended Version. [Research Report] 2010. ⟨inria-00547884v3⟩