Upgrading legacy non-dynamic rules to the new syntax

[yelhamer]

With the rollout of the new dynamic feature extraction and consequently the changes made to the rules' syntax (replacing scope with scopes), we'll need a way to port the old rules into the new format; since many of them might be reusable anyways. In this issue, I'll be proposing a way to automatically port rules to the new format and make any necessary changes, all with a single script.

The way I propose we do this rule upgrade is via a script that would parse test each rule for validity in the dynamic scope, and if it is, it would give it a dynamic interpretation; else, it would restrict it to the static analysis context. The way this script would work for a single rule is that it would essentially construct the corresponding statement for each rule, and try to activate all the shared features that are contained in that statement; then, if the rule evaluates to True, then it can be ported, otherwise it would be labeled as static.

This script should also modify the rules' inner blocks to contain a dynamic parallel in the case that the said inner block can match in a dynamic context. Here's an example to illustrate what I mean:

Original rule:

rule:
  meta:
    name: Software Packing
    scope: file
  features:
    - and:
      - characteristic: embedded pe # assume we have added support for this characteristic
      - instruction:
        - and:
          - api: VirtualProtect
          - number: 0x40 = PAGE_EXECUTE_READWRITE

Would become:

rule:
  meta:
    name: Software Packing
    scopes:
      static: file
      dynamic: file
  features:
    - and:
      - characteristic: embedded pe # assume we have added support for this characteristic
      - or:
        - basic block:
          - and:
            - api: VirtualProtect
            - number: 0x40 = PAGE_EXECUTE_READWRITE
        - call:
          - api: VirtualProtect
          - number: 0x40 = PAGE_EXECUTE_READWRITE

In the previous example, the basic block node would be replaced by an or one which contains the original basic block node as well as a mirror node that contains only static features and that has the original scope's "parallel scope"

The parallel dynamic scopes I suggest for each static scope are:
instruction -> call: since I believe it's commonly used for detecting api calls.
basic block -> thread: since I believe it's commonly used for applying boolean logic to detected api calls.
function -> process: same reason as above, and since having a hierarchy would make things easier.

Another option for picking the parallel dynamic scopes is to do it relative to what types of features are within each one.

looking forward to hear your thoughts on this.

[williballenthin]

when it comes to picking the appropriate dynamic scope, could we just brute force it? that is, try call, and if that doesn't work, try thread, and if that doesn't work, fall back to file?

[williballenthin]

in any case, it makes sense to me to try to develop this script and see how the results look. lets try it out and review the changes together. if we need to make some tweaks, thats ok too, and we shouldn't be discouraged from iteratively developing the script until the results are perfect. the better the script is, the more hours we save of manual translation!

[yelhamer]

The legacy static-only capa rules have been migrated to the new syntax, and this script is no longer needed.

Closed with release v7.