lirent/vatrog-vm-introspection-engine
1
0
Fork 0
mirror of https://dev.lirent.ru/Vatrog/vm-introspection-engine.git synced 2026-07-08 23:46:36 +03:00
Code Issues Packages Projects Releases Wiki Activity

Add readable operand decode and register def-use analysis

Browse Source 
insndec.h: insn_decode() decodes one instruction into a readable form -
concrete registers as canonical roots (al/ah/ax/eax/rax -> one root), memory
base/index/scale/disp, immediate values, operand sizes, per-operand read/write,
and implicit operands/effects (rsp on push/pop, rdx:rax on mul/div, FLAGS). It
is ABI-agnostic: a call reports only its architectural effects.

codetrace.h: func_usedef(fn, abi, ...) computes per-instruction register
use/def over a function view (reusing cfg_blocks), splitting writes into
full-kill / partial / conditional so a reaching-definitions pass can be built on
top. With TRACE_ABI_WIN64 it marks the Win64 caller-saved set
(rax,rcx,rdx,r8-r11,xmm0-5,flags) clobbered at each call site; the clobber table
is confined to codetrace.c and the generic layer stays OS-agnostic
(TRACE_ABI_NONE keeps the conservative behaviour).

Both ride the existing optional disassembler seam (a second insn_decode_full
is added to semsig_backend.h; sem_insn and semsig_hash are untouched), gated
behind VMIE_HAVE_DISASM; OFF builds a stub. Adds the vmie_win32_func_usedef
wrapper. No new build option - the feature shares VMIE_DISASM with semsig.
This commit is contained in:
Gregory Lirent
2026-06-28 23:02:40 +03:00
parent 51feb8d39c
commit 244511b66e
11 changed files with 1133 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
include/codetrace.h
+106
View File
@@ -0,0 +1,106 @@
/* codetrace.h - generic (OS-agnostic) register def-use over one function.
*
* Handler layer, gated like semsig.h: a process-scoped, STATIC register def-use
* analysis built on insn_decode (insndec.h) and the basic-block partition of
* cfg_blocks (codeanalysis.h, REUSED - this header writes no second splitter).
* For one function view it computes per-instruction register read (use) / write
* (def) sets - the building material a caller needs for reaching-definitions.
*
* It returns DATA. It does NOT execute or model values, does NOT unroll loops,
* does NOT build use-def CHAINS, and carries no dataflow solver: those are the
* caller's analysis. v1 deliberately stops at per-instruction use/def so the
* surface does not predetermine the caller's algorithm.
*
* Family placement. codeanalysis.h is PURE / always available; semsig.h is gated
* on the disassembler backend. codetrace.h sits in the second row with semsig: it
* needs the backend decode (via insndec.h) plus cfg_blocks. With no backend the
* symbol still exists (ABI stable) but returns the sentinel, and VMIE_HAVE_DISASM
* is 0. The chosen split keeps codeanalysis.h's "zero-dep, works always" contract
* literally true.
*
* GENERIC / OS-agnostic. The analysis names no OS. A calling convention enters
* ONLY through the explicit trace_abi argument, which selects a static caller-
* saved (volatile) clobber table applied at call sites. Win64 is just one enum
* value, not a hardcoded dependency; TRACE_ABI_NONE reproduces conservative
* behavior for any non-Windows consumer. The OS-specific CHOICE of Win64 is made
* by the win32 wrapper (vmie_win32_func_usedef, win32.h), not by this header.
*/
#ifndef VMIE_CODETRACE_H
#define VMIE_CODETRACE_H
#include <stdint.h>
#include <stddef.h>
#include "memmodel.h" /* mem_view_t (the single owner of the view type) */
/* Shared feature gate (see semsig.h / insndec.h). Default keeps the header valid
* when compiled outside the project build. */
#ifndef VMIE_HAVE_DISASM
#define VMIE_HAVE_DISASM 0
#endif
/* Calling convention applied to call sites when computing def-use. The analysis
* itself stays OS-agnostic: an ABI is just a parameter selecting a static
* caller-saved (volatile) clobber table. An unknown value is treated as
* TRACE_ABI_NONE (conservative), not an error. */
typedef enum {
TRACE_ABI_NONE = 0, /* conservative: a call clobbers only what insn_decode
* reports architecturally (rsp/rip write, indirect
* target read); no volatile registers are killed. */
TRACE_ABI_WIN64 /* standard Win64: volatile GPR/vec/FLAGS are killed
* (full-kill) across every call site (see codetrace.c
* for the exact set and its honest limits). */
} trace_abi;
/* A compact register set over canonical reg_id roots (a bitset). The field split
* is an implementation detail of the struct; consumers should treat it opaquely
* and compare/inspect via the documented def/use semantics, not the raw words.
* gpr - the 16 GPR roots (REGID_RAX..REGID_R15)
* vec_lo - vector roots REGID_VEC0..REGID_VEC31
* misc - mask roots (k0..k7), REGID_FLAGS, REGID_OTHER */
typedef struct {
uint16_t gpr; /* bit i set => REGID_RAX + i present (16 GPR roots) */
uint32_t vec_lo; /* bit i set => REGID_VEC0 + i present (32 vector roots) */
uint16_t misc; /* bit i set => (REGID_K0 + i) for i<8, then FLAGS, OTHER */
} regset;
/* Per-instruction use/def in the function view's coordinate space.
* off - byte offset of the instruction within fn
* len - instruction length
* use - roots READ (incl. implicit, incl. address regs of mem operands)
* def - roots WRITTEN as a FULL/zero-extending write (these KILL the
* enclosing root); for a call site under a non-NONE trace_abi the
* ABI volatile clobber is merged here as a full kill too
* def_partial - roots written PARTIALLY (8/16-bit GPR, legacy-SSE): they do NOT
* kill the enclosing root, so a wider live root survives
* cond_def - roots written CONDITIONALLY (cmovcc/setcc): do not kill either */
typedef struct {
uint32_t off;
uint8_t len;
regset use, def, def_partial, cond_def;
} insn_usedef;
/* Compute per-instruction use/def for one function view under calling convention
* `abi`. `fn` is a view spanning EXACTLY one function (as for cfg_blocks /
* semsig_hash). REUSES cfg_blocks only to validate the partition and iterate in
* block order; the use/def itself comes from insn_decode per instruction.
*
* `abi` selects the call-site volatile clobber: TRACE_ABI_NONE = conservative (no
* volatile kill, only the architectural rsp/rip write insn_decode reports);
* TRACE_ABI_WIN64 = full-kill of the Win64 volatile roots at every call site.
* The analysis stays OS-agnostic: `abi` is just a parameter, not a hardcoded OS.
*
* Writes up to `max` records in ascending off order (out=NULL to count). Returns
* the TOTAL instruction count, 0 if `fn` is empty, or -1 on a decode desync
* (cfg_blocks could not split, or the backend could not decode an instruction) /
* a build with no backend (VMIE_HAVE_DISASM==0). The sentinel convention mirrors
* cfg_blocks: -1 = desync/no-backend, 0 = empty.
*
* PURE: only the view and the decoder, no vmie_mem / no I/O (a stack buffer with
* a heap spill only when the block count overflows, like semsig_hash).
*
* Example - registers killed by each instruction of a function:
* insn_usedef ud[512];
* int n = func_usedef(fn, TRACE_ABI_WIN64, ud, 512);
* for (int i = 0; i < n && i < 512; i++) ... inspect ud[i].def ... */
int func_usedef(mem_view_t fn, trace_abi abi, insn_usedef* out, int max);
#endif /* VMIE_CODETRACE_H */
include/insndec.h
+194
View File
@@ -0,0 +1,194 @@
/* insndec.h - readable per-instruction operand decode (generic, ABI-agnostic).
*
* Handler layer: a RICH decode of ONE x86-64 instruction into concrete operands
* (concrete registers as a backend-neutral stable id, memory base/index/scale/
* disp, immediate VALUES, sizes), per-operand read/write, AND the full aggregate
* register read/write sets INCLUDING IMPLICIT operands/effects (rsp on push/pop,
* rdx:rax on mul/div, FLAGS on cmp/jcc, ...). This is the "perception" primitive
* the register def-use analysis (codetrace.h) is built on.
*
* Relationship to semsig.h (shared backend, DIFFERENT purpose). semsig_hash
* folds a NORMALIZED, LOSSY token stream: it erases register identity (keeps only
* a class), erases immediate/displacement values, and carries no read/write - by
* design, so the hash is register-/value-/reorder-stable. That makes semsig
* irreversible: you cannot trace registers through it. insn_decode is the
* opposite: it KEEPS register identity, values, and read/write, so an analysis
* can reason about WHICH physical register an instruction touches. Both ride the
* SAME optional external disassembler (Zydis/Capstone) behind the same private
* seam, but insn_decode is NOT a hash and NOT lossy.
*
* Operand decode is NOT in the light decoder (x86dec.h is deliberately pure and
* length-only) and NOT a second hand-rolled operand decoder: it is supplied by
* the optional backend, so this primitive is FEATURE-GATED exactly like semsig.
* With no backend the symbol still exists (ABI stable) but returns 0, and
* VMIE_HAVE_DISASM is 0.
*
* ABI-AGNOSTIC. insn_decode reports ONLY what an instruction does
* ARCHITECTURALLY. In particular a `call` writes rsp and rip and (for an
* indirect call) reads its target register/memory - that is ALL. It does NOT
* mark caller-saved (volatile) registers as written: that is a property of the
* CALLED function / the calling convention, not of the instruction. The Win64
* volatile clobber lives in the def-use layer (codetrace.h/.c) behind an explicit
* trace_abi, never here.
*/
#ifndef VMIE_INSNDEC_H
#define VMIE_INSNDEC_H
#include <stdint.h>
#include <stddef.h>
/* Compile-time feature availability, shared with semsig (one backend, one gate).
* Pushed from CMake as a PUBLIC compile definition so consumers see the SAME
* value the library was built with:
* VMIE_HAVE_DISASM == 1 built with a disassembler backend; insn_decode is
* live, insndec_backend_name() names the backend.
* VMIE_HAVE_DISASM == 0 built without a backend (the default); insn_decode
* always returns 0 and insndec_backend_name() == "none".
* The default here keeps the header valid when compiled outside the project. */
#ifndef VMIE_HAVE_DISASM
#define VMIE_HAVE_DISASM 0
#endif
/* Canonical full-register root id. al/ah/ax/eax/rax all map to REGID_RAX;
* xmm0/ymm0/zmm0 -> REGID_VEC0; etc. Identity is KEPT here (unlike sem_insn,
* which erases it) so def-use can reason about WHICH physical register. The
* concrete backend register enum is NEVER carried across the seam - the backend
* TU normalizes into THIS stable id, exactly as it does for mnemonic classes. */
typedef enum {
REGID_NONE = 0,
/* 16 GPR roots; al/ax/eax/rax -> REGID_RAX, etc. */
REGID_RAX, REGID_RCX, REGID_RDX, REGID_RBX,
REGID_RSP, REGID_RBP, REGID_RSI, REGID_RDI,
REGID_R8, REGID_R9, REGID_R10, REGID_R11,
REGID_R12, REGID_R13, REGID_R14, REGID_R15,
/* 32 vector roots (zmm0..zmm31; xmm/ymm fold into the zmm root). */
REGID_VEC0,
REGID_VEC31 = REGID_VEC0 + 31,
/* AVX-512 mask k0..k7. */
REGID_K0,
REGID_K7 = REGID_K0 + 7,
/* FLAGS as ONE pseudo-register root (NOT per-bit CF/ZF/...): enough for
* def-use "cmp writes flags, jcc/cmov/setcc read flags"; per-bit is not v1. */
REGID_FLAGS,
/* rip / segment / other roots kept coarse (def-use rarely tracks them). */
REGID_OTHER,
REGID_COUNT
} reg_id;
/* Per-operand / per-register access, normalized (raw backend enums never leak). */
typedef enum {
ACC_NONE = 0,
ACC_READ, /* read only */
ACC_WRITE, /* write only */
ACC_READWRITE, /* read-modify-write (add, sub, ...) */
ACC_COND_READ, /* conditionally read (cmovcc src, ...) */
ACC_COND_WRITE /* conditionally written (cmovcc dst, setcc) */
} reg_access;
/* Explicit-operand kind (the readable, perception side of the decode). */
typedef enum {
OPND_NONE = 0,
OPND_REG, /* register operand */
OPND_MEM, /* memory operand (base/index/scale/disp) */
OPND_IMM /* immediate operand (value kept) */
} operand_kind;
/* One register reference, used both for explicit register operands and for the
* aggregate implicit-or-explicit read/write sets. Sub-register overlap is modeled
* EXPLICITLY and backend-neutrally:
* root - canonical full-register root (al/ax/eax/rax -> REGID_RAX)
* width_log2 - ACCESS width in log2 bytes (0=1B .. 4=16B), as
* width_log2_of already computes for semsig
* writes_zero_extend- on a WRITE, 1 if the upper part of the root is zeroed
* (a 32-bit GPR write, e.g. `eax`, zeroes the upper 32 bits
* of `rax`; a VEX/EVEX vector write zeroes the upper YMM/ZMM)
* and 0 if it is a PARTIAL write that preserves the upper
* part (8/16-bit GPR write, legacy-SSE vector write). This
* ONE flag is what def-use needs to decide overlap without
* carrying a raw bit mask.
* access - ACC_* for this reference */
typedef struct {
uint8_t root; /* reg_id */
uint8_t width_log2; /* access width, log2 bytes (0..4+) */
uint8_t writes_zero_extend;
uint8_t access; /* reg_access */
} reg_ref;
/* One explicit operand (readable / perception view). */
typedef struct {
uint8_t kind; /* operand_kind */
uint8_t access; /* reg_access for the operand as a whole */
uint8_t width_log2; /* access width, log2 bytes */
uint8_t mem_scale; /* MEM: index scale (1/2/4/8), 0 if no index */
uint8_t mem_is_riprel; /* MEM: 1 if RIP-relative */
uint8_t reg_root; /* REG: reg_id of the operand register */
uint8_t reg_zero_extend; /* REG: writes_zero_extend for a write operand */
uint8_t imm_width_log2; /* IMM: value width, log2 bytes */
uint8_t mem_base; /* MEM: reg_id of the base register (read) */
uint8_t mem_index; /* MEM: reg_id of the index register (read) */
int64_t mem_disp; /* MEM: displacement value (signed) */
int64_t imm_value; /* IMM: immediate value (sign-extended) */
} operand;
/* Mnemonic-class group, backend-neutral. Mirrors the semsig SEM_MN_* groups so a
* consumer can switch on the instruction kind without backend enums; def-use uses
* only INSN_CALL (call sites) but the rest aids readability of the decode. */
typedef enum {
INSN_OTHER = 0,
INSN_MOV, INSN_ARITH, INSN_LOGIC, INSN_CMP, INSN_TEST,
INSN_BRANCH, /* conditional jcc / setcc / cmovcc */
INSN_JMP, /* unconditional jmp */
INSN_CALL, INSN_RET, INSN_PUSH, INSN_POP, INSN_LEA, INSN_NOP,
INSN_FLOAT, INSN_VEC
} insn_class;
/* Caps on the rich decode. Four explicit operands cover x86-64; eight register
* refs per side cover the widest implicit sets (e.g. mul touching rax/rdx plus an
* explicit source) with margin. Anything beyond is truncated (count still grows
* past the cap so a caller can detect it), which never happens for real code. */
#define INSN_MAX_OPERANDS 4
#define INSN_MAX_REGREFS 8
/* A fully decoded instruction. Two complementary representations:
* op[] - explicit operands, in instruction operand order (readability);
* reads[] / writes[] - aggregate register sets over canonical roots, UNIONING
* explicit AND implicit effects (this is what def-use consumes). A
* memory operand's base/index registers appear in reads[] as address
* reads; FLAGS appears as REGID_FLAGS; rsp/rdx:rax appear via the
* backend's implicit operands. */
typedef struct {
uint8_t length; /* full instruction length in bytes (>= 1) */
uint8_t mnemonic_class; /* insn_class */
uint8_t is_control_flow; /* 1 if branch/call/ret */
uint8_t noperands; /* explicit operands in op[] (<= INSN_MAX_OPERANDS)*/
uint8_t nreads; /* register refs in reads[] */
uint8_t nwrites; /* register refs in writes[] */
operand op[INSN_MAX_OPERANDS];
reg_ref reads[INSN_MAX_REGREFS];
reg_ref writes[INSN_MAX_REGREFS];
} insn_decoded;
/* Decode ONE instruction at view[off .. ]. `view` is a flat byte span and `off`
* is the byte offset of the instruction within it; at most (view_size - off)
* bytes are available to the backend. Returns 1 on success (fills *out), 0 on
* undecodable / not enough bytes / a build with no backend (VMIE_HAVE_DISASM==0).
* out->length == 0 is also treated as a desync by callers. PURE: only the byte
* span and the decoder, no I/O, no allocation, reentrant.
*
* `view` / `view_size` are a flat buffer (not mem_view_t) so this primitive
* stays dependency-light; the caller owns the buffer.
*
* Example - read the registers an instruction writes:
* insn_decoded d;
* if (insn_decode(code, n, 0, &d))
* for (int i = 0; i < d.nwrites; i++)
* printf("writes root %u (zx=%u)\n",
* d.writes[i].root, d.writes[i].writes_zero_extend); */
int insn_decode(const uint8_t* view, size_t view_size, size_t off,
insn_decoded* out);
/* Stable identity of the compiled disassembler backend ("zydis"/"capstone", or
* "none" without a backend). Shares the backend with semsig; this name is the
* same backend. Static, never-NULL. */
const char* insndec_backend_name(void);
#endif /* VMIE_INSNDEC_H */
include/win32.h
+35
View File
@@ -23,6 +23,7 @@
#include "memmodel.h" /* vmie_mem, vregion/VR_*, task/range, gva_read/write/ptr/regions/sweep */
#include "sigscan.h" /* mem_view_t, sig_pattern_t */
#include "scan.h" /* scan_type, scan_ptr_path, generic scan surface */
#include "codetrace.h" /* insn_usedef, trace_abi (register def-use surface) */
/* Opaque introspection context. Completed in src/engine/win32/engine-win32.h;
* callers only ever hold a pointer. Created by vmie_win32_open(), populated by
@@ -492,6 +493,40 @@ int vmie_win32_func_imports(vmie_win32* v, uint64_t cr3, uint64_t module_base,
uint64_t vmie_win32_func_semsig(vmie_win32* v, uint64_t cr3,
uint64_t module_base, uint32_t func_rva);
/* Register def-use of ONE function, addressed by its func_rva (a .pdata function
* start, as from vmie_win32_functions or an export). The win32 convenience entry
* point over the generic func_usedef (codetrace.h): it locates the function
* extent from .pdata (like vmie_win32_func_semsig), gathers its body with
* gva_read, builds a SECTION_LOCAL mem_view_t, and DELEGATES to func_usedef -
* PASSING TRACE_ABI_WIN64, because choosing the Win64 calling convention is the
* OS-specific decision that belongs in this layer (which already owns .pdata /
* cr3 / module_base). It does NOT reimplement the analysis or apply the volatile
* clobber itself (func_usedef does), and pulls in no disassembler backend (the
* feature is inherited transitively via the same VMIE_HAVE_DISASM as the generic).
*
* v - engine handle.
* cr3 - the process address space the module is mapped in.
* module_base - image base VA (its PE headers must be resident).
* func_rva - function start RVA (must match a .pdata function start).
* out, max - caller array receiving up to `max` per-instruction insn_usedef
* records in ascending function-local offset order (NULL to count).
*
* Returns the TOTAL instruction count, 0 if the function is empty, or -1 on any
* of: func_rva is not a known function start, the body is unreadable (paged out),
* a decode desync, or a build with no disassembler backend (VMIE_HAVE_DISASM==0).
* The off field of each record is a function-local byte offset (SECTION_LOCAL).
*
* The Win64 volatile clobber applied at call sites - and its honest limits (only
* the standard Win64 convention; not __vectorcall/custom; the return value is not
* a separate def) - are documented at the clobber table in codetrace.c.
*
* Example - registers each instruction of a function kills:
* insn_usedef ud[512];
* int n = vmie_win32_func_usedef(v, pr->cr3, m.base, fn_rva, ud, 512);
* for (int i = 0; i < n && i < 512; i++) ... inspect ud[i].def ... */
int vmie_win32_func_usedef(vmie_win32* v, uint64_t cr3, uint64_t module_base,
uint32_t func_rva, insn_usedef* out, int max);
/* Devirtualization (C++ vtables) needs NO dedicated symbol - it is a
* COMPOSITION of primitives the engine already exposes:
* - a vtable at `vtable_va` is an array of code pointers, so its METHODS are