Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Volatility can't match the memory dump file (MacOS Monterey 12.6 build 21G115) to the symbol table created #848

Open
Lowengrube opened this issue Oct 11, 2022 · 14 comments · May be fixed by #1116
Open

Volatility can't match the memory dump file (MacOS Monterey 12.6 build 21G115) to the symbol table created #848

Lowengrube opened this issue Oct 11, 2022 · 14 comments · May be fixed by #1116
Assignees
@atcuno

Comments

@Lowengrube
Copy link

Lowengrube commented Oct 11, 2022 •
edited
Loading

Hey there,
so currently i'm facing problem in using Volatility 3 to analyse the ram dump file from MacOS Monterey 12.6 build 21G115, I had successfully created the symbol table for that OS version:
./dwarf2json mac --macho /Library/Developer/KDKs/KDK_12.6_21G115.kdk/System/Library/Kernels/kernel.dSYM/Contents/Resources/DWARF/kernel --macho-symbols /Library/Developer/KDKs/KDK_12.6_21G115.kdk/System/Library/Kernels/kernel > 12.6.json

After that I copied 12.6.json to /path_to_volatility3/symbols/mac/ directory.

ISFinfo shows:
python3 ./volatility3/vol.py isfinfo
file:///Users/test/volatility3/volatility3/symbols/mac/allmacho.json Unknown 19 0 64681 392 b'Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64\x00'

Banners of image show:
python3 ./volatility3/vol.py -f raw_dump_only_osxpmem.dump banners

0x18d60273	Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x18d602d6	Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x19301fc2	Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x1f960273	Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x1f9602d6	Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x1ff01fc2	Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x47fae08e	Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x124601008	Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x12548e2b9	Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x12548e31c	Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x1f5db09c6	Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x30cd8be4e	Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x3f0865046	Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x3f2ee408e	Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x3f3ae38c6	Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x3f4af0d57	Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
0x400e3f88e	Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64

It looks like ok, but any operations with image (mac.bash/mac.lsof/mac.pstree and etc.) don't work:
python3 ./volatility3/vol.py -vvvvvv -f raw_dump_only_osxpmem.dump mac.bash

Volatility 3 Framework 2.3.0
INFO     volatility3.cli: Volatility plugins path: ['/Users/test/volatility3/volatility3/plugins', '/Users/test/volatility3/volatility3/framework/plugins']
INFO     volatility3.cli: Volatility symbols path: ['/Users/test/volatility3/volatility3/symbols', '/Users/test/volatility3/volatility3/framework/symbols']
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/plugins, /Users/test/volatility3/volatility3/framework/plugins
INFO     volatility3.plugins.yarascan: Python Yara (>3.8.0) module not found, plugin (and dependent plugins) not available
DEBUG    volatility3.framework: No module named 'yara'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.yarascan based on file: /Users/test/volatility3/volatility3/framework/plugins/yarascan.py
INFO     volatility3.plugins.yarascan: Python Yara (>3.8.0) module not found, plugin (and dependent plugins) not available
DEBUG    volatility3.framework: No module named 'yara'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.svcscan based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/svcscan.py
DEBUG    volatility3.framework: No module named 'Crypto'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.hashdump based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/hashdump.py
DEBUG    volatility3.framework: No module named 'pefile'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.skeleton_key_check based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/skeleton_key_check.py
DEBUG    volatility3.framework: No module named 'Crypto'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.cachedump based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/cachedump.py
INFO     volatility3.plugins.yarascan: Python Yara (>3.8.0) module not found, plugin (and dependent plugins) not available
DEBUG    volatility3.framework: No module named 'yara'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.vadyarascan based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/vadyarascan.py
INFO     volatility3.plugins.yarascan: Python Yara (>3.8.0) module not found, plugin (and dependent plugins) not available
DEBUG    volatility3.framework: No module named 'yara'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.mftscan based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/mftscan.py
INFO     volatility3.plugins.windows.verinfo: Python pefile module not found, plugin (and dependent plugins) not available
DEBUG    volatility3.framework: No module named 'pefile'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.netscan based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/netscan.py
INFO     volatility3.plugins.windows.verinfo: Python pefile module not found, plugin (and dependent plugins) not available
DEBUG    volatility3.framework: No module named 'pefile'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.netstat based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/netstat.py
DEBUG    volatility3.framework: No module named 'Crypto'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.lsadump based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/lsadump.py
INFO     volatility3.plugins.windows.verinfo: Python pefile module not found, plugin (and dependent plugins) not available
DEBUG    volatility3.framework: No module named 'pefile'
DEBUG    volatility3.framework: Failed to import module volatility3.plugins.windows.verinfo based on file: /Users/test/volatility3/volatility3/framework/plugins/windows/verinfo.py
INFO     volatility3.cli: The following plugins could not be loaded (use -vv to see why): volatility3.plugins.windows.cachedump, volatility3.plugins.windows.hashdump, volatility3.plugins.windows.lsadump, volatility3.plugins.windows.mftscan, volatility3.plugins.windows.netscan, volatility3.plugins.windows.netstat, volatility3.plugins.windows.skeleton_key_check, volatility3.plugins.windows.svcscan, volatility3.plugins.windows.vadyarascan, volatility3.plugins.windows.verinfo, volatility3.plugins.yarascan
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/automagic
Level 7  volatility3.cli: Cache directory used: /Users/test/.cache/volatility3
INFO     volatility3.framework.automagic: Detected a mac category plugin
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
INFO     volatility3.framework.automagic: Running automagic: ConstructionMagic
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Bash.kernel
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Bash.kernel
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Bash.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Bash.kernel
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name
Level 9  volatility3.framework.automagic.construct_layers: Failed on requirement: plugins.Bash
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 6  volatility3.framework.automagic.construct_layers: Construction Exception occurred: Unexpected config value found: None
INFO     volatility3.framework.automagic: Running automagic: SymbolCacheMagic
Level 6  volatility3.framework.symbols.intermed: Searching for symbols in /Users/test/volatility3/volatility3/symbols, /Users/test/volatility3/volatility3/framework/symbols
INFO     volatility3.framework.automagic: Running automagic: LayerStacker
Level 6  volatility3.framework: Importing from the following paths: /Users/test/volatility3/volatility3/framework/layers
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name
Level 7  volatility3.framework.layers.resources: Available URL handlers: HTTPErrorProcessor, HTTPDefaultErrorHandler, HTTPRedirectHandler, ProxyHandler, HTTPBasicAuthHandler, ProxyBasicAuthHandler, HTTPDigestAuthHandler, ProxyDigestAuthHandler, AbstractHTTPHandler, HTTPHandler, HTTPSHandler, HTTPCookieProcessor, UnknownHandler, FileHandler, FTPHandler, CacheFTPHandler, DataHandler, VolatilityHandler, JarHandler, OfflineHandler
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using AVMLStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using LimeStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using Elf64Stacker
Level 6  volatility3.framework.layers.elf: Exception: Bad magic 0x4034b50 at file offset 0x0
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using QemuStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using WindowsCrashDumpStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using VmwareStacker
Level 8  volatility3.framework.automagic.stacker: Attempting to stack using MacIntelStacker
DEBUG    volatility3.framework.automagic.mac: Identified banner: b'Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64\x00'
INFO     volatility3.schemas: Dependency for validation unavailable: jsonschema
DEBUG    volatility3.schemas: All validations will report success, even with malformed input
Level 7  volatility3.framework.automagic.stacker: Exception during stacking: Symbol type not in MacintelStacker1 SymbolTable: vm_map_object
Level 6  volatility3.framework.automagic.stacker: Traceback (most recent call last):

  File "/Users/test/volatility3/volatility3/framework/automagic/stacker.py", line 171, in stack_layer
    new_layer = stacker.stack(context, initial_layer, progress_callback)

  File "/Users/test/volatility3/volatility3/framework/automagic/mac.py", line 61, in stack
    table = mac.MacKernelIntermedSymbols(context = context,

  File "/Users/test/volatility3/volatility3/framework/symbols/mac/__init__.py", line 21, in __init__
    self.set_type_class('vm_map_object', extensions.vm_map_object)

  File "/Users/test/volatility3/volatility3/framework/symbols/intermed.py", line 54, in _delegate_function
    return getattr(self._delegate, name)(*args, **kwargs)

  File "/Users/test/volatility3/volatility3/framework/symbols/intermed.py", line 362, in set_type_class
    raise ValueError(f"Symbol type not in {self.name} SymbolTable: {name}")

ValueError: Symbol type not in MacintelStacker1 SymbolTable: vm_map_object

Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: TypeError - Layer is not the required Architecture: FileLayer
DEBUG    volatility3.framework.automagic.stacker: Stacked layers: ['FileLayer']
INFO     volatility3.framework.automagic: Running automagic: SymbolFinder  
INFO     volatility3.framework.automagic: Running automagic: MacSymbolFinder
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name
INFO     volatility3.framework.automagic: Running automagic: KernelModule
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: IndexError - No configuration provided: plugins.Bash.kernel.layer_name
Level 9  volatility3.framework.configuration.requirements: Symbol table requirement not yet fulfilled: plugins.Bash.kernel.symbol_table_name

Unsatisfied requirement plugins.Bash.kernel.layer_name: 
Unsatisfied requirement plugins.Bash.kernel.symbol_table_name: 

A translation layer requirement was not fulfilled.  Please verify that:
	A file was provided to create this layer (by -f, --single-location or by config)
	The file exists and is readable
	The file is a valid memory image and was acquired cleanly

A symbol table requirement was not fulfilled.  Please verify that:
	The associated translation layer requirement was fulfilled
	You have the correct symbol file for the requirement
	The symbol file is under the correct directory or zip file
	The symbol file is named appropriately or contains the correct banner

Unable to validate the plugin requirements: ['plugins.Bash.kernel.layer_name', 'plugins.Bash.kernel.symbol_table_name']
@ikelos
Copy link
Member

ikelos commented Oct 26, 2022

So it did identify the correct banner, but we then immediately try to use a structure that wasn't present:

volatility3.framework.automagic.mac: Identified banner: b'Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64\x00'
INFO     volatility3.schemas: Dependency for validation unavailable: jsonschema
DEBUG    volatility3.schemas: All validations will report success, even with malformed input
Level 7  volatility3.framework.automagic.stacker: Exception during stacking: Symbol type not in MacintelStacker1 SymbolTable: vm_map_object
Level 6  volatility3.framework.automagic.stacker: Traceback (most recent call last):

  File "/Users/test/volatility3/volatility3/framework/automagic/stacker.py", line 171, in stack_layer
    new_layer = stacker.stack(context, initial_layer, progress_callback)

  File "/Users/test/volatility3/volatility3/framework/automagic/mac.py", line 61, in stack
    table = mac.MacKernelIntermedSymbols(context = context,

  File "/Users/test/volatility3/volatility3/framework/symbols/mac/__init__.py", line 21, in __init__
    self.set_type_class('vm_map_object', extensions.vm_map_object)

  File "/Users/test/volatility3/volatility3/framework/symbols/intermed.py", line 54, in _delegate_function
    return getattr(self._delegate, name)(*args, **kwargs)

  File "/Users/test/volatility3/volatility3/framework/symbols/intermed.py", line 362, in set_type_class
    raise ValueError(f"Symbol type not in {self.name} SymbolTable: {name}")

ValueError: Symbol type not in MacintelStacker1 SymbolTable: vm_map_object

It's not clear if this structure's name has changed, or it was removed (or possibly if the symbol table was generated incorrectly, but it doesn't look like it). We'll need to do some investigation to figure out what the problem is. The vm_map_object type is one that we override with a custom handler, and it appears the custom handler isn't finding the original definition in the JSON. I've asked @atcuno to see whether the vm_map_object structure was renamed or removed from recent mac kernels...

@github-actions
Copy link

This issue is stale because it has been open for 200 days with no activity.

@github-actions github-actions bot added the stale label Aug 18, 2023
@ikelos
Copy link
Member

ikelos commented Aug 19, 2023

Ping @atcuno, before this times out in a couple months, could you please check about the vm_map_object in the mac symbol tables?

@github-actions github-actions bot removed the stale label Aug 20, 2023
@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 8, 2024

This issue is stale because it has been open for 200 days with no activity.

@github-actions github-actions bot added the stale label Mar 8, 2024
@ikelos
Copy link
Member

ikelos commented Mar 12, 2024

@atcuno The stale ticket just got added, which means it's been a couple of months since I asked. Have you had a chance to check out what's going on with the symbol tables and the vm_map_object symbol?

@github-actions github-actions bot removed the stale label Mar 12, 2024
@Abyss-W4tcher
Copy link
Contributor

This structure was removed from the kernel, as well as many related vm_map ones.

The old/new versions are here :

Here is an article brieflly talking about it :

I was planning to update the framework, but It's gonna need more time and analysis to fix it. It mostly impacts mac.malfind.

@ikelos
Copy link
Member

ikelos commented Mar 20, 2024

@Abyss-W4tcher thanks for the analysis! Perhaps @atcuno or @gcmoreira can help out now we know what it is?

@Abyss-W4tcher
Copy link
Contributor

Hi, I will propose a patch in a PR soon, it's only in my fork right now.

I inform any dev here, to avoid potentially duplicating the same work 😃

@Abyss-W4tcher Abyss-W4tcher linked a pull request Mar 23, 2024 that will close this issue
Draft
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 7, 2024

This issue is stale because it has been open for 200 days with no activity.

@github-actions github-actions bot added the stale label Oct 7, 2024
@ikelos ikelos removed the stale label Oct 7, 2024
@ikelos
Copy link
Member

ikelos commented Oct 7, 2024

Definitely not stale, just tricky to fix... 5:S Sorry I've been so slow on it, I'm finding it hard to get dedicated time to fix big issues like this. It is still on my list though.

@Abyss-W4tcher
Copy link
Contributor

Hi, this should have been fixed in the two macOS PRs that were released a few months ago. However due to the current calendar I completely understand the lack of time to allocate on this subject 👍.

@ikelos
Copy link
Member

ikelos commented Oct 7, 2024

Yeah, I thought I was producing a MacSymbolTable that was a facade for the two different mac tables and gave you the right table (with the right shift) based on which symbol you asked for? Did we still need that, I don't remember how we left it sadly... 5:S

@Abyss-W4tcher
Copy link
Contributor

Abyss-W4tcher commented Oct 7, 2024 •
edited
Loading

When you are ready, just comment on any of the PRs and I will provide you with a quick resume.

But basically, this issue is related to malfind (fixed by the "plugins" PR) and the "double module" aspect you mentioned relates to the automagic update PR supporting a new self-contained MACHO kernel in macOS.

@ikelos
Copy link
Member

ikelos commented Oct 7, 2024

Ok thanks, as I say it hasn't slipped off my list but it needs me to find some time to sit and concentrate on it. Thanks for your understanding, it's really appreciated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@atcuno atcuno

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

macOS: Plugin enhancements for latest kernel compatibility Abyss-W4tcher/volatility3
4 participants
@atcuno @ikelos @Lowengrube @Abyss-W4tcher