Detecting and evading behavioral analysis tools

There are multiple ways in which malware can detect and evade behavioral analysis tools, such as ProcMon, Wireshark, API Monitor, and more, even if they don’t directly debug the malware or interact with it. In this section, we will talk about two common examples of how it can be done.

Finding the tool process

One of the simplest and most common ways that malware can deal with malware-analysis tools (and antivirus tools, too) is to loop through all the running processes and detect any unwanted entries. Then, it is possible to either terminate or stop them to avoid further analysis.

In Chapter 5, Inspecting Process Injection and API Hooking, we covered how malware can loop through all running processes using the CreateToolhelp32Snapshot, Process32First, and Process32Next APIs. In this anti-reverse engineering trick, the malware uses these APIs in exactly the same way to check the process name against a list of unwanted process...