Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Usage of libv4l causes a crash during rr record #2929

Closed
georgewsinger opened this issue Jul 15, 2021 · 44 comments
Closed

Usage of libv4l causes a crash during rr record #2929

georgewsinger opened this issue Jul 15, 2021 · 44 comments

Comments

@georgewsinger
Copy link

libv4l2 found.
[FATAL /build/source/src/record_syscall.cc:5817:rec_process_syscall_arch()] 
 (task 36247 (rec:36247) at time 11843)
 -> Assertion `t->regs().syscall_result_signed() == -syscall_state.expect_errno' failed to hold. Expected EINVAL for 'ioctl' but got result 0 (errno SUCCESS); Unknown ioctl(0x80045626): type:0x56 nr:0x26 dir:0x2 size:4 addr:0x7ffc5be5f1f0
Tail of trace dump:
{
  real_time:48306.157920 global_time:11823, event:`SYSCALL: ioctl' (state:EXITING_SYSCALL) tid:36247, ticks:532405443
rax:0x0 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be60140 rsi:0xc02c564a rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0xffffffff r9:0x7ffc5be60390 r10:0x13 r11:0x246 r12:0x1 r13:0xc r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
  { tid:36247, addr:0x7ffc5be60140, length:0x2c }
}
{
  real_time:48306.157979 global_time:11824, event:`SYSCALL: ioctl' (state:ENTERING_SYSCALL) tid:36247, ticks:532405508
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be60140 rsi:0xc02c564a rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0xffffffff r9:0x7ffc5be60390 r10:0x13 r11:0x246 r12:0x1 r13:0xd r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
}
{
  real_time:48306.158027 global_time:11825, event:`SYSCALL: ioctl' (state:EXITING_SYSCALL) tid:36247, ticks:532405508
rax:0x0 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be60140 rsi:0xc02c564a rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0xffffffff r9:0x7ffc5be60390 r10:0x13 r11:0x246 r12:0x1 r13:0xd r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
  { tid:36247, addr:0x7ffc5be60140, length:0x2c }
}
{
  real_time:48306.158086 global_time:11826, event:`SYSCALL: ioctl' (state:ENTERING_SYSCALL) tid:36247, ticks:532405577
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be60140 rsi:0xc02c564a rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0xffffffff r9:0x7ffc5be60390 r10:0x13 r11:0x246 r12:0x1 r13:0xe r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
}
{
  real_time:48306.158134 global_time:11827, event:`SYSCALL: ioctl' (state:EXITING_SYSCALL) tid:36247, ticks:532405577
rax:0x0 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be60140 rsi:0xc02c564a rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0xffffffff r9:0x7ffc5be60390 r10:0x13 r11:0x246 r12:0x1 r13:0xe r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
  { tid:36247, addr:0x7ffc5be60140, length:0x2c }
}
{
  real_time:48306.158193 global_time:11828, event:`SYSCALL: ioctl' (state:ENTERING_SYSCALL) tid:36247, ticks:532405650
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be60140 rsi:0xc02c564a rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0xffffffff r9:0x7ffc5be60390 r10:0x13 r11:0x246 r12:0x1 r13:0xf r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
}
{
  real_time:48306.158241 global_time:11829, event:`SYSCALL: ioctl' (state:EXITING_SYSCALL) tid:36247, ticks:532405650
rax:0x0 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be60140 rsi:0xc02c564a rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0xffffffff r9:0x7ffc5be60390 r10:0x13 r11:0x246 r12:0x1 r13:0xf r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
  { tid:36247, addr:0x7ffc5be60140, length:0x2c }
}
{
  real_time:48306.158300 global_time:11830, event:`SYSCALL: ioctl' (state:ENTERING_SYSCALL) tid:36247, ticks:532405727
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be60140 rsi:0xc02c564a rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0xffffffff r9:0x7ffc5be60390 r10:0x13 r11:0x246 r12:0x1 r13:0x10 r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
}
{
  real_time:48306.158353 global_time:11831, event:`SYSCALL: ioctl' (state:EXITING_SYSCALL) tid:36247, ticks:532405727
rax:0x0 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be60140 rsi:0xc02c564a rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0xffffffff r9:0x7ffc5be60390 r10:0x13 r11:0x246 r12:0x1 r13:0x10 r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
  { tid:36247, addr:0x7ffc5be60140, length:0x2c }
}
{
  real_time:48306.158412 global_time:11832, event:`SYSCALL: ioctl' (state:ENTERING_SYSCALL) tid:36247, ticks:532405808
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be60140 rsi:0xc02c564a rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0xffffffff r9:0x7ffc5be60390 r10:0x13 r11:0x246 r12:0x1 r13:0x11 r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
}
{
  real_time:48306.158461 global_time:11833, event:`SYSCALL: ioctl' (state:EXITING_SYSCALL) tid:36247, ticks:532405808
rax:0x0 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be60140 rsi:0xc02c564a rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0xffffffff r9:0x7ffc5be60390 r10:0x13 r11:0x246 r12:0x1 r13:0x11 r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
  { tid:36247, addr:0x7ffc5be60140, length:0x2c }
}
{
  real_time:48306.158521 global_time:11834, event:`SYSCALL: ioctl' (state:ENTERING_SYSCALL) tid:36247, ticks:532405893
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be60140 rsi:0xc02c564a rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0xffffffff r9:0x7ffc5be60390 r10:0x13 r11:0x246 r12:0x1 r13:0x12 r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
}
{
  real_time:48306.158569 global_time:11835, event:`SYSCALL: ioctl' (state:EXITING_SYSCALL) tid:36247, ticks:532405893
rax:0x0 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be60140 rsi:0xc02c564a rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0xffffffff r9:0x7ffc5be60390 r10:0x13 r11:0x246 r12:0x1 r13:0x12 r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
  { tid:36247, addr:0x7ffc5be60140, length:0x2c }
}
{
  real_time:48306.158628 global_time:11836, event:`SYSCALL: ioctl' (state:ENTERING_SYSCALL) tid:36247, ticks:532405982
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be60140 rsi:0xc02c564a rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0xffffffff r9:0x7ffc5be60390 r10:0x13 r11:0x246 r12:0x1 r13:0x13 r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
}
{
  real_time:48306.158677 global_time:11837, event:`SYSCALL: ioctl' (state:EXITING_SYSCALL) tid:36247, ticks:532405982
rax:0xffffffffffffffea rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be60140 rsi:0xc02c564a rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0xffffffff r9:0x7ffc5be60390 r10:0x13 r11:0x246 r12:0x1 r13:0x13 r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
  { tid:36247, addr:0x7ffc5be60140, length:0x2c }
}
{
  real_time:48306.158735 global_time:11838, event:`SYSCALL: ioctl' (state:ENTERING_SYSCALL) tid:36247, ticks:532405996
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be60170 rsi:0xc0405602 rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0x7ffc5be60390 r9:0x7ffc5be60390 r10:0xffffffff r11:0x246 r12:0x2 r13:0x13 r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
}
{
  real_time:48306.158784 global_time:11839, event:`SYSCALL: ioctl' (state:EXITING_SYSCALL) tid:36247, ticks:532405996
rax:0xffffffffffffffea rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be60170 rsi:0xc0405602 rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0x7ffc5be60390 r9:0x7ffc5be60390 r10:0xffffffff r11:0x246 r12:0x2 r13:0x13 r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
  { tid:36247, addr:0x7ffc5be60170, length:0x40 }
}
{
  real_time:48306.158843 global_time:11840, event:`SYSCALL: ioctl' (state:ENTERING_SYSCALL) tid:36247, ticks:532406009
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be60170 rsi:0x80685600 rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0x7ffc5be60390 r9:0x7ffc5be60390 r10:0x7ffc5be60390 r11:0x246 r12:0x2 r13:0x13 r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
}
{
  real_time:48306.158892 global_time:11841, event:`SYSCALL: ioctl' (state:EXITING_SYSCALL) tid:36247, ticks:532406009
rax:0x0 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be60170 rsi:0x80685600 rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0x7ffc5be60390 r9:0x7ffc5be60390 r10:0x7ffc5be60390 r11:0x246 r12:0x2 r13:0x13 r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
  { tid:36247, addr:0x7ffc5be60170, length:0x68 }
}
{
  real_time:48306.158954 global_time:11842, event:`SYSCALL: ioctl' (state:ENTERING_SYSCALL) tid:36247, ticks:532406113
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x7ffc5be5f1f0 rsi:0x80045626 rdi:0x7 rbp:0x681ffea0 rsp:0x681ffe50 r8:0x1 r9:0x13 r10:0x7ec7ec0 r11:0x246 r12:0x7ec7ec0 r13:0x7ffc5be5f1f0 r14:0x7ffc5be60170 r15:0x7ec84c0 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x10 fs_base:0x7f062423ba80 gs_base:0x0
}
=== Start rr backtrace:
/nix/store/hjbjqsbv32y440c6vqp3kz1hrf1s2vzl-rr-unstable-2020-10-10/bin/rr(_ZN2rr13dump_rr_stackEv+0x44)[0x68eed8]
/nix/store/hjbjqsbv32y440c6vqp3kz1hrf1s2vzl-rr-unstable-2020-10-10/bin/rr(_ZN2rr9GdbServer15emergency_debugEPNS_4TaskE+0x1a2)[0x4ce81e]
/nix/store/hjbjqsbv32y440c6vqp3kz1hrf1s2vzl-rr-unstable-2020-10-10/bin/rr[0x4fda7c]
/nix/store/hjbjqsbv32y440c6vqp3kz1hrf1s2vzl-rr-unstable-2020-10-10/bin/rr(_ZN2rr21EmergencyDebugOstreamD1Ev+0x62)[0x4fdcd6]
/nix/store/hjbjqsbv32y440c6vqp3kz1hrf1s2vzl-rr-unstable-2020-10-10/bin/rr[0x577ce3]
/nix/store/hjbjqsbv32y440c6vqp3kz1hrf1s2vzl-rr-unstable-2020-10-10/bin/rr[0x560ac8]
/nix/store/hjbjqsbv32y440c6vqp3kz1hrf1s2vzl-rr-unstable-2020-10-10/bin/rr(_ZN2rr19rec_process_syscallEPNS_10RecordTaskE+0xe8)[0x560c45]
/nix/store/hjbjqsbv32y440c6vqp3kz1hrf1s2vzl-rr-unstable-2020-10-10/bin/rr(_ZN2rr13RecordSession21syscall_state_changedEPNS_10RecordTaskEPNS0_9StepStateE+0xd00)[0x541be4]
/nix/store/hjbjqsbv32y440c6vqp3kz1hrf1s2vzl-rr-unstable-2020-10-10/bin/rr(_ZN2rr13RecordSession11record_stepEv+0x404)[0x547754]
/nix/store/hjbjqsbv32y440c6vqp3kz1hrf1s2vzl-rr-unstable-2020-10-10/bin/rr[0x539817]
/nix/store/hjbjqsbv32y440c6vqp3kz1hrf1s2vzl-rr-unstable-2020-10-10/bin/rr(_ZN2rr13RecordCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x3dd)[0x53a447]
/nix/store/hjbjqsbv32y440c6vqp3kz1hrf1s2vzl-rr-unstable-2020-10-10/bin/rr(main+0x227)[0x6aa46f]
/nix/store/hp8wcylqr14hrrpqap4wdrwzq092wfln-glibc-2.32-37/lib/libc.so.6(__libc_start_main+0xed)[0x7fbf87134ded]
/nix/store/hjbjqsbv32y440c6vqp3kz1hrf1s2vzl-rr-unstable-2020-10-10/bin/rr(_start+0x2a)[0x43056a]
=== End rr backtrace
Launch gdb with
  gdb '-l' '10000' '-ex' 'set sysroot /' '-ex' 'target extended-remote 127.0.0.1:36247' /home/george/Simula/submodules/godot/bin/godot.x11.tools.64
@khuey
Copy link
Collaborator

khuey commented Jul 15, 2021

That looks like VIDIOC_G_INPUT.

khuey added a commit that referenced this issue Jul 15, 2021
@khuey
Copy link
Collaborator

khuey commented Jul 15, 2021

Try again with the latest tip?

@georgewsinger
Copy link
Author

We now get

[FATAL /build/source/src/RecordSession.cc:345:handle_seccomp_traced_syscall()] 
 (task 78878 (rec:78878) at time 91107)
 -> Assertion `patch_ok' failed to hold. The tracee issues a vsyscall, but we failed to moneypatch the
caller. Recording will not succeed. Exiting.
Tail of trace dump:
{
  real_time:66044.530806 global_time:91087, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:78878, ticks:2408613
rax:0x7fb6266ae000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x210808 rdi:0x0 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fb64d2ffa80 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7fb6266ae000, length:0x211000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x211000 }
}
{
  real_time:66044.530863 global_time:91088, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:78878, ticks:2408695
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x20000 rdi:0x7fb64e2ff000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fb64d2ffa80 gs_base:0x0
}
{
  real_time:66044.530915 global_time:91089, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:78878, ticks:2408695
rax:0x7fb64e2ff000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x20000 rdi:0x7fb64e2ff000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fb64d2ffa80 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7fb64e2ff000, length:0x20000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x20000 }
}
{
  real_time:66044.530967 global_time:91090, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:78878, ticks:2408730
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x7fb64d13f000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fb64d2ffa80 gs_base:0x0
}
{
  real_time:66044.531022 global_time:91091, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:78878, ticks:2408730
rax:0x7fb64d13f000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x7fb64d13f000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fb64d2ffa80 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7fb64d13f000, length:0x1000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x1000 }
}
{
  real_time:66044.531074 global_time:91092, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:78878, ticks:2408765
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x7fb64ccc5000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fb64d2ffa80 gs_base:0x0
}
{
  real_time:66044.531130 global_time:91093, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:78878, ticks:2408765
rax:0x7fb64ccc5000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x7fb64ccc5000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fb64d2ffa80 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7fb64ccc5000, length:0x1000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x1000 }
}
{
  real_time:66044.531181 global_time:91094, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:78878, ticks:2408800
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x7fb64a8ef000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fb64d2ffa80 gs_base:0x0
}
{
  real_time:66044.531236 global_time:91095, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:78878, ticks:2408800
rax:0x7fb64a8ef000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x7fb64a8ef000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fb64d2ffa80 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7fb64a8ef000, length:0x1000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x1000 }
}
{
  real_time:66044.531287 global_time:91096, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:78878, ticks:2408835
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x7fb638a3f000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fb64d2ffa80 gs_base:0x0
}
{
  real_time:66044.531342 global_time:91097, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:78878, ticks:2408835
rax:0x7fb638a3f000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x7fb638a3f000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fb64d2ffa80 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7fb638a3f000, length:0x1000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x1000 }
}
{
  real_time:66044.531394 global_time:91098, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:78878, ticks:2408882
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x100000 rdi:0x0 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fb64d2ffa80 gs_base:0x0
}
{
  real_time:66044.531445 global_time:91099, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:78878, ticks:2408882
rax:0x7fb6265ae000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x100000 rdi:0x0 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fb64d2ffa80 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7fb6265ae000, length:0x100000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x100000 }
}
{
  real_time:66044.531518 global_time:91100, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:78878, ticks:2409503
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x10000 rdi:0x0 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x55de8eb5dca0 r13:0x12 r14:0x80c000000000 r15:0x80c000001fff rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fb64d2ffa80 gs_base:0x0
}
{
  real_time:66044.531569 global_time:91101, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:78878, ticks:2409503
rax:0x7fb62659e000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x10000 rdi:0x0 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x55de8eb5dca0 r13:0x12 r14:0x80c000000000 r15:0x80c000001fff rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fb64d2ffa80 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7fb62659e000, length:0x10000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x10000 }
}
{
  real_time:66044.531627 global_time:91102, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:78878, ticks:2409567
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x10000 rdi:0x0 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x0 r13:0x12 r14:0x80c000000000 r15:0x80c000001fff rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fb64d2ffa80 gs_base:0x0
}
{
  real_time:66044.531679 global_time:91103, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:78878, ticks:2409567
rax:0x7fb62658e000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x10000 rdi:0x0 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x0 r13:0x12 r14:0x80c000000000 r15:0x80c000001fff rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fb64d2ffa80 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7fb62658e000, length:0x10000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x10000 }
}
{
  real_time:66044.531756 global_time:91104, event:`INSTRUCTION_TRAP' tid:78878, ticks:2412555
rax:0xd rbx:0x756e6547 rcx:0x6c65746e rdx:0x49656e69 rsi:0x55de8e63f6c0 rdi:0xc00000c1e0 rbp:0x7ffe0426d088 rsp:0x7ffe0426d050 r8:0x7fb6266ae60e r9:0x203000 r10:0x8 r11:0x75 r12:0xf5 r13:0x0 r14:0x55de8e4299c8 r15:0x0 rip:0x55de8db35f5a eflags:0x10246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xffffffffffffffff fs_base:0x7fb64d2ffa80 gs_base:0x0
}
{
  real_time:66044.531804 global_time:91105, event:`INSTRUCTION_TRAP' tid:78878, ticks:2412556
rax:0x306c3 rbx:0x100800 rcx:0x3ffafbbf rdx:0xbfebfbff rsi:0x55de8e63f6c0 rdi:0xc00000c1e0 rbp:0x7ffe0426d088 rsp:0x7ffe0426d050 r8:0x7fb6266ae60e r9:0x203000 r10:0x8 r11:0x75 r12:0xf5 r13:0x0 r14:0x55de8e4299c8 r15:0x0 rip:0x55de8db35f5a eflags:0x10206 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xffffffffffffffff fs_base:0x7fb64d2ffa80 gs_base:0x0
}
{
  real_time:66044.531852 global_time:91106, event:`INSTRUCTION_TRAP' tid:78878, ticks:2412559
rax:0x0 rbx:0x27ab rcx:0x0 rdx:0x9c000600 rsi:0x55de8e63f6c0 rdi:0xc00000c1e0 rbp:0x7ffe0426d088 rsp:0x7ffe0426d050 r8:0x7fb6266ae60e r9:0x203000 r10:0x8 r11:0x75 r12:0xf5 r13:0x0 r14:0x55de8e4299c8 r15:0x0 rip:0x55de8db35f5a eflags:0x10206 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xffffffffffffffff fs_base:0x7fb64d2ffa80 gs_base:0x0
}
=== Start rr backtrace:
/nix/store/6cglxyfwblnb1jpiqa2j008sm8j80q7z-rr-unstable-2020-10-10/bin/rr(_ZN2rr13dump_rr_stackEv+0x44)[0x69e873]
/nix/store/6cglxyfwblnb1jpiqa2j008sm8j80q7z-rr-unstable-2020-10-10/bin/rr(_ZN2rr9GdbServer15emergency_debugEPNS_4TaskE+0x1a2)[0x4d0896]
/nix/store/6cglxyfwblnb1jpiqa2j008sm8j80q7z-rr-unstable-2020-10-10/bin/rr[0x500834]
/nix/store/6cglxyfwblnb1jpiqa2j008sm8j80q7z-rr-unstable-2020-10-10/bin/rr(_ZN2rr21EmergencyDebugOstreamD1Ev+0x62)[0x500a8e]
/nix/store/6cglxyfwblnb1jpiqa2j008sm8j80q7z-rr-unstable-2020-10-10/bin/rr(_ZN2rr13RecordSession29handle_seccomp_traced_syscallEPNS_10RecordTaskEPNS0_9StepStateEPNS0_12RecordResultEPb+0x479)[0x5414b7]
/nix/store/6cglxyfwblnb1jpiqa2j008sm8j80q7z-rr-unstable-2020-10-10/bin/rr(_ZN2rr13RecordSession19handle_ptrace_eventEPPNS_10RecordTaskEPNS0_9StepStateEPNS0_12RecordResultEPb+0x31a)[0x54262a]
/nix/store/6cglxyfwblnb1jpiqa2j008sm8j80q7z-rr-unstable-2020-10-10/bin/rr(_ZN2rr13RecordSession11record_stepEv+0x31f)[0x54a97b]
/nix/store/6cglxyfwblnb1jpiqa2j008sm8j80q7z-rr-unstable-2020-10-10/bin/rr[0x53c807]
/nix/store/6cglxyfwblnb1jpiqa2j008sm8j80q7z-rr-unstable-2020-10-10/bin/rr(_ZN2rr13RecordCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x3dd)[0x53d44b]
/nix/store/6cglxyfwblnb1jpiqa2j008sm8j80q7z-rr-unstable-2020-10-10/bin/rr(main+0x227)[0x6b9e1f]
/nix/store/hp8wcylqr14hrrpqap4wdrwzq092wfln-glibc-2.32-37/lib/libc.so.6(__libc_start_main+0xed)[0x7fe5a0353ded]
/nix/store/6cglxyfwblnb1jpiqa2j008sm8j80q7z-rr-unstable-2020-10-10/bin/rr(_start+0x2a)[0x43056a]
=== End rr backtrace
Launch gdb with
  gdb '-l' '10000' '-ex' 'set sysroot /' '-ex' 'target extended-remote 127.0.0.1:13342' /usr/bin/snap

@khuey
Copy link
Collaborator

khuey commented Jul 15, 2021

Attach gdb to the tracee following the instructions at the bottom and get a backtrace?

@georgewsinger
Copy link
Author

When I run

gdb '-l' '10000' '-ex' 'set sysroot /' '-ex' 'target extended-remote 127.0.0.1:30174' /usr/bin/snap

I get taken to

Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.32.so...
0x000056040c49c20c in _start () from /lib64/ld-linux-x86-64.so.2
(gdb) bt
#0  0x000056040c49c20c in _start () from /lib64/ld-linux-x86-64.so.2
#1  0x000056040cdcee20 in ?? ()
#2  0x000000c000002001 in ?? ()
#3  0x000000c00000c000 in ?? ()
#4  0x000056040cb95ea0 in ?? ()
#5  0x00007ffe52a173a8 in ?? ()
#6  0x000056040c47b039 in ?? ()
#7  0x0000000000010000 in ?? ()
#8  0x000056040cb95ea0 in ?? ()
#9  0x0000000000000004 in ?? ()
#10 0x000056040d4cd3ab in ?? ()
#11 0x0000000000000010 in ?? ()
#12 0x00007ffe52a173f0 in ?? ()
#13 0x000056040c466cac in ?? ()
#14 0x000056040d4cd240 in ?? ()
#15 0x0000000000000080 in ?? ()
#16 0x0000000000000080 in ?? ()
#17 0x0000000000000010 in ?? ()
#18 0x0000000000000000 in ?? ()

Pressing c just results in termination of the program:

(gdb) r
The program being debugged has been started already.
Start it from the beginning? (y or n) n
Program not restarted.
(gdb) c
Continuing.
Warning:
Cannot insert breakpoint -2: Remote connection closed
Command aborted.

@rocallahan
Copy link
Collaborator

If you rerun that recording and attach the emergency debugger again, what's the value of $rip, what's the return address on the stack at [$rsp], and what's the disassembly of the code at that address?

@georgewsinger
Copy link
Author

Some output from gdb:

(gdb) p $rip
$1 = (void (*)()) 0x55e0662bf20c <_start>
(gdb) p $rsp
$2 = (void *) 0x7ffe96ad7210
(gdb) x/10i $rsp
   0x7ffe96ad7210:	and    %bl,(%rsi)
   0x7ffe96ad7212:	mov    $0x55e066,%edi
   0x7ffe96ad7217:	add    %al,(%rcx)
   0x7ffe96ad7219:	and    %al,(%rax)
   0x7ffe96ad721b:	add    %al,%al
   0x7ffe96ad721d:	add    %al,(%rax)
   0x7ffe96ad721f:	add    %al,(%rax)
   0x7ffe96ad7221:	rolb   $0x0,(%rax)
   0x7ffe96ad7224:	rolb   $0x0,(%rax)
   0x7ffe96ad7227:	add    %ah,-0x1f996472(%rax)
(gdb) x/10i $rip
=> 0x55e0662bf20c <_start>:	mov    (%rsp),%eax
   0x55e0662bf20f <_start+3>:	mov    0x8(%rsp),%edx
   0x55e0662bf213 <_start+7>:	mov    %rbp,%rsp
   0x55e0662bf216 <_dl_start_user+2>:	movq   $0x0,0x328(%rbx)
   0x55e0662bf221 <_dl_start_user+13>:	imul   $0x3e8,%rdx,%rdx
   0x55e0662bf228 <_dl_start_user+20>:	imul   $0x3b9aca00,%rax,%rax
   0x55e0662bf22f <_dl_start_user+27>:	add    %rdx,%rax
   0x55e0662bf232 <_dl_start_user+30>:	mov    %rax,0x18(%rsp)
   0x55e0662bf237 <_dl_start_user+35>:	mov    0x8(%rsp),%rbp
   0x55e0662bf23c <_dl_start_user+40>:	add    $0x10,%rsp

@rocallahan
Copy link
Collaborator

779040e adds some more debug logging, can you pull it and rerun your failing test?

@rocallahan
Copy link
Collaborator

It seems deeply wrong that the return address should be _start here...

@georgewsinger
Copy link
Author

Here's the updated log:

[FATAL /build/source/src/RecordSession.cc:347:handle_seccomp_traced_syscall()] 
 (task 30031 (rec:30031) at time 87012)
 -> Assertion `patch_ok' failed to hold. The tracee issues a vsyscall to 0xffffffffff600000 but we failed to moneypatch the caller (return address 0x5618f2a7920b, sp=0x7ffdf2fe34e8). Recording will not succeed. Exiting.
Tail of trace dump:
{
  real_time:6140.719522 global_time:86992, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:30031, ticks:629286
rax:0x7fa5d33e1000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x210808 rdi:0x0 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fa5fa033080 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7fa5d33e1000, length:0x211000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x211000 }
}
{
  real_time:6140.719591 global_time:86993, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:30031, ticks:629368
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x20000 rdi:0x7fa5fa246000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fa5fa033080 gs_base:0x0
}
{
  real_time:6140.719656 global_time:86994, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:30031, ticks:629368
rax:0x7fa5fa246000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x20000 rdi:0x7fa5fa246000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fa5fa033080 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7fa5fa246000, length:0x20000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x20000 }
}
{
  real_time:6140.719719 global_time:86995, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:30031, ticks:629403
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x7fa5f9e72000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fa5fa033080 gs_base:0x0
}
{
  real_time:6140.719787 global_time:86996, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:30031, ticks:629403
rax:0x7fa5f9e72000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x7fa5f9e72000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fa5fa033080 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7fa5f9e72000, length:0x1000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x1000 }
}
{
  real_time:6140.719851 global_time:86997, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:30031, ticks:629438
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x7fa5f99f8000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fa5fa033080 gs_base:0x0
}
{
  real_time:6140.719919 global_time:86998, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:30031, ticks:629438
rax:0x7fa5f99f8000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x7fa5f99f8000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fa5fa033080 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7fa5f99f8000, length:0x1000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x1000 }
}
{
  real_time:6140.719983 global_time:86999, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:30031, ticks:629473
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x7fa5f7622000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fa5fa033080 gs_base:0x0
}
{
  real_time:6140.720051 global_time:87000, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:30031, ticks:629473
rax:0x7fa5f7622000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x7fa5f7622000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fa5fa033080 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7fa5f7622000, length:0x1000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x1000 }
}
{
  real_time:6140.720115 global_time:87001, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:30031, ticks:629508
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x7fa5e5772000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fa5fa033080 gs_base:0x0
}
{
  real_time:6140.720183 global_time:87002, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:30031, ticks:629508
rax:0x7fa5e5772000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x7fa5e5772000 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x32 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fa5fa033080 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7fa5e5772000, length:0x1000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x1000 }
}
{
  real_time:6140.720247 global_time:87003, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:30031, ticks:629555
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x100000 rdi:0x0 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fa5fa033080 gs_base:0x0
}
{
  real_time:6140.720310 global_time:87004, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:30031, ticks:629555
rax:0x7fa5d32e1000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x100000 rdi:0x0 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x40 r13:0x40 r14:0x1 r15:0x6e43a318 rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fa5fa033080 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7fa5d32e1000, length:0x100000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x100000 }
}
{
  real_time:6140.720399 global_time:87005, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:30031, ticks:630176
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x10000 rdi:0x0 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x5618f3a3bca0 r13:0x12 r14:0x80c000000000 r15:0x80c000001fff rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fa5fa033080 gs_base:0x0
}
{
  real_time:6140.720462 global_time:87006, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:30031, ticks:630176
rax:0x7fa5d32d1000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x10000 rdi:0x0 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x5618f3a3bca0 r13:0x12 r14:0x80c000000000 r15:0x80c000001fff rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fa5fa033080 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7fa5d32d1000, length:0x10000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x10000 }
}
{
  real_time:6140.720533 global_time:87007, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:30031, ticks:630240
rax:0xffffffffffffffda rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x10000 rdi:0x0 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x0 r13:0x12 r14:0x80c000000000 r15:0x80c000001fff rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fa5fa033080 gs_base:0x0
}
{
  real_time:6140.720596 global_time:87008, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:30031, ticks:630240
rax:0x7fa5d32c1000 rbx:0x681fffa0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x10000 rdi:0x0 rbp:0x681ffec0 rsp:0x681ffe70 r8:0xffffffff r9:0x0 r10:0x22 r11:0x246 r12:0x0 r13:0x12 r14:0x80c000000000 r15:0x80c000001fff rip:0x70000002 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x7fa5fa033080 gs_base:0x0
  { map_file:"<ZERO>", addr:0x7fa5d32c1000, length:0x10000, prot_flags:"rw-p", file_offset:0x0, device:0, inode:0, data_file:"", data_offset:0x0, file_size:0x10000 }
}
{
  real_time:6140.720686 global_time:87009, event:`INSTRUCTION_TRAP' tid:30031, ticks:633222
rax:0xd rbx:0x756e6547 rcx:0x6c65746e rdx:0x49656e69 rsi:0x5618f351d6c0 rdi:0xc00000c1e0 rbp:0x7ffdf2fe3548 rsp:0x7ffdf2fe3510 r8:0x7fa5d33e160e r9:0x203000 r10:0x8 r11:0x75 r12:0xf5 r13:0x0 r14:0x5618f33079c8 r15:0x0 rip:0x5618f2a13f5a eflags:0x10246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xffffffffffffffff fs_base:0x7fa5fa033080 gs_base:0x0
}
{
  real_time:6140.720745 global_time:87010, event:`INSTRUCTION_TRAP' tid:30031, ticks:633223
rax:0x306c3 rbx:0x100800 rcx:0x3ffafbbf rdx:0xbfebfbff rsi:0x5618f351d6c0 rdi:0xc00000c1e0 rbp:0x7ffdf2fe3548 rsp:0x7ffdf2fe3510 r8:0x7fa5d33e160e r9:0x203000 r10:0x8 r11:0x75 r12:0xf5 r13:0x0 r14:0x5618f33079c8 r15:0x0 rip:0x5618f2a13f5a eflags:0x10206 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xffffffffffffffff fs_base:0x7fa5fa033080 gs_base:0x0
}
{
  real_time:6140.720803 global_time:87011, event:`INSTRUCTION_TRAP' tid:30031, ticks:633226
rax:0x0 rbx:0x27ab rcx:0x0 rdx:0x9c000600 rsi:0x5618f351d6c0 rdi:0xc00000c1e0 rbp:0x7ffdf2fe3548 rsp:0x7ffdf2fe3510 r8:0x7fa5d33e160e r9:0x203000 r10:0x8 r11:0x75 r12:0xf5 r13:0x0 r14:0x5618f33079c8 r15:0x0 rip:0x5618f2a13f5a eflags:0x10206 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xffffffffffffffff fs_base:0x7fa5fa033080 gs_base:0x0
}
=== Start rr backtrace:
/nix/store/r4riwfhnx9is1809kwwywgkjzhkpr1q3-rr-unstable-2020-10-10/bin/rr(_ZN2rr13dump_rr_stackEv+0x44)[0x69d39d]
/nix/store/r4riwfhnx9is1809kwwywgkjzhkpr1q3-rr-unstable-2020-10-10/bin/rr(_ZN2rr9GdbServer15emergency_debugEPNS_4TaskE+0x1a2)[0x4d080a]
/nix/store/r4riwfhnx9is1809kwwywgkjzhkpr1q3-rr-unstable-2020-10-10/bin/rr[0x50088c]
/nix/store/r4riwfhnx9is1809kwwywgkjzhkpr1q3-rr-unstable-2020-10-10/bin/rr(_ZN2rr21EmergencyDebugOstreamD1Ev+0x62)[0x500ae6]
/nix/store/r4riwfhnx9is1809kwwywgkjzhkpr1q3-rr-unstable-2020-10-10/bin/rr(_ZN2rr13RecordSession29handle_seccomp_traced_syscallEPNS_10RecordTaskEPNS0_9StepStateEPNS0_12RecordResultEPb+0x4d6)[0x5415c2]
/nix/store/r4riwfhnx9is1809kwwywgkjzhkpr1q3-rr-unstable-2020-10-10/bin/rr(_ZN2rr13RecordSession19handle_ptrace_eventEPPNS_10RecordTaskEPNS0_9StepStateEPNS0_12RecordResultEPb+0x31a)[0x542734]
/nix/store/r4riwfhnx9is1809kwwywgkjzhkpr1q3-rr-unstable-2020-10-10/bin/rr(_ZN2rr13RecordSession11record_stepEv+0x377)[0x54aa8b]
/nix/store/r4riwfhnx9is1809kwwywgkjzhkpr1q3-rr-unstable-2020-10-10/bin/rr[0x53c88e]
/nix/store/r4riwfhnx9is1809kwwywgkjzhkpr1q3-rr-unstable-2020-10-10/bin/rr(_ZN2rr13RecordCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x3dd)[0x53d4f9]
/nix/store/r4riwfhnx9is1809kwwywgkjzhkpr1q3-rr-unstable-2020-10-10/bin/rr(main+0x227)[0x6b894f]
/nix/store/hp8wcylqr14hrrpqap4wdrwzq092wfln-glibc-2.32-37/lib/libc.so.6(__libc_start_main+0xed)[0x7ff962546ded]
/nix/store/r4riwfhnx9is1809kwwywgkjzhkpr1q3-rr-unstable-2020-10-10/bin/rr(_start+0x2a)[0x43056a]
=== End rr backtrace
Launch gdb with
  gdb '-l' '10000' '-ex' 'set sysroot /' '-ex' 'target extended-remote 127.0.0.1:30031' /usr/bin/snap

@khuey
Copy link
Collaborator

khuey commented Jul 18, 2021

Can you disassemble the code at 0xffffffffff600000?

@georgewsinger
Copy link
Author

georgewsinger commented Jul 18, 2021

@khuey Re-running the trace I get

[FATAL /build/source/src/RecordSession.cc:347:handle_seccomp_traced_syscall()] 
 (task 4805 (rec:4805) at time 94687)
 -> Assertion `patch_ok' failed to hold. The tracee issues a vsyscall to 0xffffffffff600000 but we failed to moneypatch the caller (return address 0x55dc68b6020b, sp=0x7fffd94b0cc8). Recording will not succeed. Exiting.
# ...

and then

(gdb) x/10i 0xffffffffff600000
   0xffffffffff600000:	Cannot access memory at address 0xffffffffff600000

@khuey
Copy link
Collaborator

khuey commented Jul 18, 2021

Hmm, ok. The vsyscall page should be there ...

You could try booting your kernel with the parameter vsyscall=none and see if this goes away.

@georgewsinger
Copy link
Author

georgewsinger commented Jul 18, 2021

✔️ This worked. Thank you.

Testing now to see if Pernosco accepts the trace.

@georgewsinger
Copy link
Author

vsyscall=none allows rr record to run without immediate crash, but terminating the program (via Ctrl-c from the terminal) causes the terminal to hang. Then attempting to upload the latest-trace folder into Pernosco yields

Can't find rr trace in ./rr/latest-trace

The latest-trace folder contents are:

george@UbuntuBox:~/Simula/rr/godot.x11.tools.64-3$ ls -tlra
total 1288256
-r-xr-xr-x  5 george george     19000 Dec 31  1969 mmap_hardlink_9_librrpage.so
-r-xr-xr-x  5 george george    182416 Dec 31  1969 mmap_hardlink_996_libv4lconvert.so.0.0.0
-r-xr-xr-x  5 george george     64424 Dec 31  1969 mmap_hardlink_992_libv4l2.so.0.0.0
-r-xr-xr-x  3 george george     59432 Dec 31  1969 mmap_hardlink_9927_libbookmarks.so
-r-xr-xr-x  3 george george     83680 Dec 31  1969 mmap_hardlink_9923_libsession.so
-r-xr-xr-x  3 george george    104736 Dec 31  1969 mmap_hardlink_9904_libweb-extensions.so
-r-xr-xr-x 10 george george    103424 Dec 31  1969 mmap_hardlink_98_libwayland-server.so.0.1.0
-r-xr-xr-x  3 george george     12656 Dec 31  1969 mmap_hardlink_9347_WebKitNetworkProcess
-r-xr-xr-x 12 george george    288352 Dec 31  1969 mmap_hardlink_89_libxkbcommon.so.0.0.0
-r-xr-xr-x 12 george george     18432 Dec 31  1969 mmap_hardlink_85_libdl-2.32.so
-r-xr-xr-x  7 george george    163296 Dec 31  1969 mmap_hardlink_8170_libmcpp.so.0.3.0
-r-xr-xr-x  7 george george     16128 Dec 31  1969 mmap_hardlink_8148_mcpp
-r-xr-xr-x 12 george george    143608 Dec 31  1969 mmap_hardlink_80_libpthread-2.32.so
-r-xr-xr-x 11 george george    740928 Dec 31  1969 mmap_hardlink_75_libGL.so.1.7.0
-r-xr-xr-x 11 george george    405080 Dec 31  1969 mmap_hardlink_71_libpulse.so.0.23.0
-r-xr-xr-x 11 george george   1252240 Dec 31  1969 mmap_hardlink_67_libasound.so.2.0.0
-r-xr-xr-x 12 george george    120648 Dec 31  1969 mmap_hardlink_62_libz.so.1.2.11
-r-xr-xr-x 11 george george    472184 Dec 31  1969 mmap_hardlink_58_libGLU.so.1.3.1
-r-xr-xr-x  6 george george     37280 Dec 31  1969 mmap_hardlink_5764_libpixbufloader-png.so
-r-xr-xr-x  7 george george    173152 Dec 31  1969 mmap_hardlink_5505_libxkbfile.so.1.0.2
-r-xr-xr-x 11 george george     31128 Dec 31  1969 mmap_hardlink_54_libXfixes.so.3.1.0
-r-xr-xr-x  7 george george    233344 Dec 31  1969 mmap_hardlink_5466_xkbcomp
-r-xr-xr-x 11 george george     94880 Dec 31  1969 mmap_hardlink_50_libXext.so.6.4.0
-r-xr-xr-x  3 george george    161184 Dec 31  1969 mmap_hardlink_5030_libgiognutls.so
-r-xr-xr-x  3 george george     28104 Dec 31  1969 mmap_hardlink_5026_libgiognomeproxy.so
-r-xr-xr-x  3 george george  13557728 Dec 31  1969 mmap_hardlink_5020_libmozjs-60.so
-r-xr-xr-x  3 george george    209808 Dec 31  1969 mmap_hardlink_5016_libproxy.so.1.0.0
-r-xr-xr-x  3 george george     22624 Dec 31  1969 mmap_hardlink_5012_libgiolibproxy.so
-r-xr-xr-x  6 george george     80840 Dec 31  1969 mmap_hardlink_4951_libdconfsettings.so
-r-xr-xr-x 11 george george     26280 Dec 31  1969 mmap_hardlink_4884_libnss_dns-2.32.so
-r-xr-xr-x 11 george george     55840 Dec 31  1969 mmap_hardlink_4879_libnss_files-2.32.so
-r-xr-xr-x 11 george george     45968 Dec 31  1969 mmap_hardlink_477_libnss_compat-2.32.so
-r-xr-xr-x 11 george george     80272 Dec 31  1969 mmap_hardlink_46_libXi.so.6.1.0
-r-xr-xr-x 11 george george   1727472 Dec 31  1969 mmap_hardlink_468_libxml2.so.2.9.10
-r-xr-xr-x 11 george george    508000 Dec 31  1969 mmap_hardlink_464_libncursesw.so.6.2
-r-xr-xr-x 11 george george     44672 Dec 31  1969 mmap_hardlink_460_libdrm_nouveau.so.2.0.0
-r-xr-xr-x 11 george george     51632 Dec 31  1969 mmap_hardlink_456_libdrm_amdgpu.so.1.0.0
-r-xr-xr-x 11 george george    105544 Dec 31  1969 mmap_hardlink_452_libelf.so.0.8.13
-r-xr-xr-x 11 george george     61904 Dec 31  1969 mmap_hardlink_448_libdrm_radeon.so.1.0.1
-r-xr-xr-x 11 george george 107358768 Dec 31  1969 mmap_hardlink_443_libLLVM-11.so
-r-xr-xr-x 12 george george   1433656 Dec 31  1969 mmap_hardlink_42_libX11.so.6.4.0
-r-xr-xr-x 11 george george  23654344 Dec 31  1969 mmap_hardlink_429_swrast_dri.so
-r-xr-xr-x 11 george george     16592 Dec 31  1969 mmap_hardlink_421_libxshmfence.so.1.0.0
-r-xr-xr-x 11 george george     42984 Dec 31  1969 mmap_hardlink_417_libxcb-sync.so.1.0.0
-r-xr-xr-x 11 george george     17736 Dec 31  1969 mmap_hardlink_413_libxcb-present.so.0.0.0
-r-xr-xr-x 11 george george     23640 Dec 31  1969 mmap_hardlink_409_libxcb-dri3.so.0.0.0
-r-xr-xr-x 11 george george     17776 Dec 31  1969 mmap_hardlink_405_libxcb-shm.so.0.0.0
-r-xr-xr-x 11 george george     30432 Dec 31  1969 mmap_hardlink_401_libXxf86vm.so.1.0.0
-r-xr-xr-x 11 george george     16928 Dec 31  1969 mmap_hardlink_397_libXdamage.so.1.1.0
-r-xr-xr-x 11 george george     28496 Dec 31  1969 mmap_hardlink_393_libxcb-dri2.so.0.0.0
-r-xr-xr-x 11 george george    161920 Dec 31  1969 mmap_hardlink_389_libxcb-glx.so.0.0.0
-r-xr-xr-x 11 george george    312712 Dec 31  1969 mmap_hardlink_384_libglapi.so.0.0.0
-r-xr-xr-x 11 george george     53128 Dec 31  1969 mmap_hardlink_37_libXrender.so.1.3.0
-r-xr-xr-x 11 george george    642384 Dec 31  1969 mmap_hardlink_379_libGLX_mesa.so.0.0.0
-r-xr-xr-x  5 george george    118552 Dec 31  1969 mmap_hardlink_3529_libelf-0.182.so
-r-xr-xr-x  5 george george     31152 Dec 31  1969 mmap_hardlink_3525_libattr.so.1.1.2448
-r-xr-xr-x  5 george george    860304 Dec 31  1969 mmap_hardlink_3483_libaspell.so.15.3.1
-r-xr-xr-x 12 george george    375600 Dec 31  1969 mmap_hardlink_347_libblkid.so.1.1.0
-r-xr-xr-x  5 george george    616472 Dec 31  1969 mmap_hardlink_3474_liborc-0.4.so.0.32.0
-r-xr-xr-x  5 george george    699784 Dec 31  1969 mmap_hardlink_3470_libdw-0.182.so
-r-xr-xr-x  6 george george     67792 Dec 31  1969 mmap_hardlink_3465_libunwind.so.8.0.1
-r-xr-xr-x  5 george george     16944 Dec 31  1969 mmap_hardlink_3452_libwoff2common.so.1.0.2
-r-xr-xr-x 12 george george     96624 Dec 31  1969 mmap_hardlink_341_libresolv-2.32.so
-r-xr-xr-x  5 george george    777272 Dec 31  1969 mmap_hardlink_3413_libzstd.so.1.4.8
-r-xr-xr-x 11 george george     54256 Dec 31  1969 mmap_hardlink_33_libXrandr.so.2.2.0
-r-xr-xr-x  5 george george     45008 Dec 31  1969 mmap_hardlink_3399_libacl.so.1.1.2253
-r-xr-xr-x 12 george george    204320 Dec 31  1969 mmap_hardlink_336_libselinux.so.1
-r-xr-xr-x 12 george george    428664 Dec 31  1969 mmap_hardlink_332_libmount.so.1.1.0
-r-xr-xr-x 12 george george     26080 Dec 31  1969 mmap_hardlink_328_libgmodule-2.0.so.0.6600.4
-r-xr-xr-x  5 george george    480856 Dec 31  1969 mmap_hardlink_3279_libXt.so.6.0.0
-r-xr-xr-x  5 george george    487576 Dec 31  1969 mmap_hardlink_3254_libsecret-1.so.0.0.0
-r-xr-xr-x  5 george george    149512 Dec 31  1969 mmap_hardlink_3249_libseccomp.so.2.5.1
-r-xr-xr-x  5 george george     61448 Dec 31  1969 mmap_hardlink_3245_libenchant-2.so.2.2.15
-r-xr-xr-x  5 george george    484448 Dec 31  1969 mmap_hardlink_3240_libwebp.so.7.1.0
-r-xr-xr-x 12 george george    502136 Dec 31  1969 mmap_hardlink_323_libpcre.so.1.2.12
-r-xr-xr-x  5 george george     21632 Dec 31  1969 mmap_hardlink_3236_libwebpdemux.so.2.0.6
-r-xr-xr-x  5 george george    414920 Dec 31  1969 mmap_hardlink_3232_libopenjp2.so.2.4.0
-r-xr-xr-x  5 george george     51888 Dec 31  1969 mmap_hardlink_3219_libgstfft-1.0.so.0.1802.0
-r-xr-xr-x  5 george george    573888 Dec 31  1969 mmap_hardlink_3215_libgstgl-1.0.so.0.1802.0
-r-xr-xr-x  5 george george    896264 Dec 31  1969 mmap_hardlink_3210_libgstvideo-1.0.so.0.1802.0
-r-xr-xr-x  5 george george    298624 Dec 31  1969 mmap_hardlink_3205_libgsttag-1.0.so.0.1802.0
-r-xr-xr-x  5 george george    599600 Dec 31  1969 mmap_hardlink_3201_libgstaudio-1.0.so.0.1802.0
-r-xr-xr-x  5 george george    308920 Dec 31  1969 mmap_hardlink_3197_libgstpbutils-1.0.so.0.1802.0
-r-xr-xr-x  5 george george   1580152 Dec 31  1969 mmap_hardlink_3192_libgstreamer-1.0.so.0.1802.0
-r-xr-xr-x 12 george george   2345928 Dec 31  1969 mmap_hardlink_318_libgio-2.0.so.0.6600.4
-r-xr-xr-x  5 george george    606880 Dec 31  1969 mmap_hardlink_3188_libgstbase-1.0.so.0.1802.0
-r-xr-xr-x  5 george george     86888 Dec 31  1969 mmap_hardlink_3183_libgstapp-1.0.so.0.1802.0
-r-xr-xr-x  5 george george     17688 Dec 31  1969 mmap_hardlink_3174_libharfbuzz-icu.so.0.20704.0
-r-xr-xr-x  5 george george     59504 Dec 31  1969 mmap_hardlink_3161_libwoff2dec.so.1.0.2
-r-xr-xr-x  5 george george    279424 Dec 31  1969 mmap_hardlink_3157_libxslt.so.1.1.34
-r-xr-xr-x 12 george george   1397672 Dec 31  1969 mmap_hardlink_313_libglib-2.0.so.0.6600.4
-r-xr-xr-x  5 george george     43688 Dec 31  1969 mmap_hardlink_3135_libnotify.so.4.0.0
-r-xr-xr-x 12 george george    425848 Dec 31  1969 mmap_hardlink_309_libgobject-2.0.so.0.6600.4
-r-xr-xr-x  5 george george    906848 Dec 31  1969 mmap_hardlink_3062_libarchive.so.13.5.1
-r-xr-xr-x 12 george george     61360 Dec 31  1969 mmap_hardlink_305_libgudev-1.0.so.0.3.0
-r-xr-xr-x  5 george george    265192 Dec 31  1969 mmap_hardlink_3053_libgirepository-1.0.so.1.0.0
-r-xr-xr-x  5 george george    114472 Dec 31  1969 mmap_hardlink_3045_libpeas-1.0.so.0.2800.0
-r-xr-xr-x 12 george george    176608 Dec 31  1969 mmap_hardlink_300_libgpg-error.so.0.31.1
-r-xr-xr-x 11 george george     17144 Dec 31  1969 mmap_hardlink_29_libXinerama.so.1.0.0
-r-xr-xr-x  5 george george     76984 Dec 31  1969 mmap_hardlink_2996_libpeas-gtk-1.0.so.0.2800.0
-r-xr-xr-x  5 george george    273136 Dec 31  1969 mmap_hardlink_2988_libgck-1.so.0.0.0
-r-xr-xr-x  5 george george    785096 Dec 31  1969 mmap_hardlink_2983_libgcr-base-3.so.1.0.0
-r-xr-xr-x  5 george george    546608 Dec 31  1969 mmap_hardlink_2978_libgcr-ui-3.so.1.0.0
-r-xr-xr-x  5 george george  30715000 Dec 31  1969 mmap_hardlink_2975_libjavascriptcoregtk-4.0.so.18.17.11
-r-xr-xr-x  5 george george  78870104 Dec 31  1969 mmap_hardlink_2972_libwebkit2gtk-4.0.so.37.49.7
-r-xr-xr-x 11 george george    186144 Dec 31  1969 mmap_hardlink_296_libmvec-2.32.so
-r-xr-xr-x  5 george george    475584 Dec 31  1969 mmap_hardlink_2945_libmidori-core.so.0.6
-r-xr-xr-x 11 george george     65832 Dec 31  1969 mmap_hardlink_292_libwacom.so.2.6.1
-r-xr-xr-x  5 george george     16240 Dec 31  1969 mmap_hardlink_2925_.midori-wrapped
-r-xr-xr-x 11 george george    126096 Dec 31  1969 mmap_hardlink_287_libevdev.so.2.3.0
-r-xr-xr-x 11 george george     29824 Dec 31  1969 mmap_hardlink_283_libmtdev.so.1.0.0
-r-xr-xr-x 12 george george    203312 Dec 31  1969 mmap_hardlink_279_libexpat.so.1.6.12
-r-xr-xr-x 12 george george   1337512 Dec 31  1969 mmap_hardlink_274_libgcrypt.so.20.3.1
-r-xr-xr-x 12 george george    215272 Dec 31  1969 mmap_hardlink_269_liblz4.so.1.9.3
-r-xr-xr-x 12 george george    182568 Dec 31  1969 mmap_hardlink_265_liblzma.so.5.2.5
-r-xr-xr-x  6 george george    139360 Dec 31  1969 mmap_hardlink_2657_libbrotlicommon.so.1.0.9
-r-xr-xr-x  6 george george    139120 Dec 31  1969 mmap_hardlink_2653_libidn2.so.0.3.7
-r-xr-xr-x  6 george george     55552 Dec 31  1969 mmap_hardlink_2649_libbrotlidec.so.1.0.9
-r-xr-xr-x  6 george george     75752 Dec 31  1969 mmap_hardlink_2645_libpsl.so.5.3.2
-r-xr-xr-x  6 george george     41560 Dec 31  1969 mmap_hardlink_2620_libdatrie.so.1.3.5
-r-xr-xr-x 11 george george     43760 Dec 31  1969 mmap_hardlink_261_libogg.so.0.8.4
-r-xr-xr-x  6 george george   1389368 Dec 31  1969 mmap_hardlink_2616_libsqlite3.so.0.8.6
-r-xr-xr-x  6 george george   4315552 Dec 31  1969 mmap_hardlink_2611_libicui18n.so.68.2
-r-xr-xr-x  6 george george    804736 Dec 31  1969 mmap_hardlink_2601_libsoup-2.4.so.1.11.0
-r-xr-xr-x 11 george george     55864 Dec 31  1969 mmap_hardlink_25_libXcursor.so.1.0.2
-r-xr-xr-x  6 george george    210384 Dec 31  1969 mmap_hardlink_2597_libjson-glib-1.0.so.0.600.2
-r-xr-xr-x  6 george george    275576 Dec 31  1969 mmap_hardlink_2593_libatspi.so.0.0.1
-r-xr-xr-x 11 george george    410760 Dec 31  1969 mmap_hardlink_257_libopus.so.0.8.0
-r-xr-xr-x  6 george george     30984 Dec 31  1969 mmap_hardlink_2571_libxcb-util.so.1.0.0
-r-xr-xr-x  6 george george    189952 Dec 31  1969 mmap_hardlink_2567_libgraphite2.so.3.2.1
-r-xr-xr-x  6 george george     36408 Dec 31  1969 mmap_hardlink_2563_libuuid.so.1.3.0
-r-xr-xr-x  6 george george  28574744 Dec 31  1969 mmap_hardlink_2543_libicudata.so.68.2
-r--r--r--  6 george george    322880 Dec 31  1969 mmap_hardlink_2535_libhogweed.so.6.0
-r--r--r--  6 george george    292440 Dec 31  1969 mmap_hardlink_2531_libnettle.so.8.0
-r-xr-xr-x 11 george george    811552 Dec 31  1969 mmap_hardlink_252_libvorbisenc.so.2.0.12
-r-xr-xr-x  6 george george     92312 Dec 31  1969 mmap_hardlink_2526_libtasn1.so.6.6.0
-r-xr-xr-x  6 george george   1632264 Dec 31  1969 mmap_hardlink_2522_libunistring.so.2.1.0
-r-xr-xr-x  6 george george   1677536 Dec 31  1969 mmap_hardlink_2518_libp11-kit.so.0.3.0
-r-xr-xr-x 11 george george    233520 Dec 31  1969 mmap_hardlink_248_libvorbis.so.0.4.9
-r-xr-xr-x  6 george george     15728 Dec 31  1969 mmap_hardlink_2481_libgthread-2.0.so.0.6600.4
-r-xr-xr-x 11 george george    472848 Dec 31  1969 mmap_hardlink_244_libFLAC.so.8.3.0
-r-xr-xr-x  6 george george     51928 Dec 31  1969 mmap_hardlink_2447_libthai.so.0.3.1
-r-xr-xr-x  6 george george     17024 Dec 31  1969 mmap_hardlink_2434_libXcomposite.so.1.0.0
-r-xr-xr-x  6 george george     39232 Dec 31  1969 mmap_hardlink_2413_libwayland-cursor.so.0.0.0
-r-xr-xr-x 12 george george     30792 Dec 31  1969 mmap_hardlink_240_libXdmcp.so.6.0.0
-r-xr-xr-x  6 george george   1352144 Dec 31  1969 mmap_hardlink_2392_libtracker-sparql-3.0.so.0.1.0
-r-xr-xr-x  6 george george    266456 Dec 31  1969 mmap_hardlink_2386_libatk-bridge-2.0.so.0.0.0
-r-xr-xr-x  6 george george    336648 Dec 31  1969 mmap_hardlink_2370_libfontconfig.so.1.12.0
-r-xr-xr-x  6 george george    129352 Dec 31  1969 mmap_hardlink_2365_libpangoft2-1.0.so.0.4700.0
-r-xr-xr-x 12 george george     17656 Dec 31  1969 mmap_hardlink_235_libXau.so.6.0.0
-r-xr-xr-x  6 george george     56864 Dec 31  1969 mmap_hardlink_2357_libstartup-notification-1.so.0.0.0
-r-xr-xr-x  6 george george     49528 Dec 31  1969 mmap_hardlink_2352_libcairo-gobject.so.2.11600.0
-r-xr-xr-x  6 george george   1066568 Dec 31  1969 mmap_hardlink_2348_libharfbuzz.so.0.20704.0
-r-xr-xr-x  6 george george    116176 Dec 31  1969 mmap_hardlink_2343_libICE.so.6.3.0
-r-xr-xr-x  6 george george     44568 Dec 31  1969 mmap_hardlink_2339_libSM.so.6.0.1
-r-xr-xr-x  6 george george    202608 Dec 31  1969 mmap_hardlink_2321_libatk-1.0.so.0.23609.1
-r-xr-xr-x 12 george george     54432 Dec 31  1969 mmap_hardlink_231_libffi.so.7.1.0
-r-xr-xr-x  6 george george     80264 Dec 31  1969 mmap_hardlink_2317_libpangocairo-1.0.so.0.4700.0
-r-xr-xr-x  6 george george    637952 Dec 31  1969 mmap_hardlink_2303_libpcre2-8.so.0.10.1
-r-xr-xr-x  6 george george   2406232 Dec 31  1969 mmap_hardlink_2298_libicuuc.so.68.2
-r-xr-xr-x  6 george george   2100928 Dec 31  1969 mmap_hardlink_2293_libgnutls.so.30.28.1
-r-xr-xr-x  6 george george    126456 Dec 31  1969 mmap_hardlink_2289_libfribidi.so.0.4.0
-r-xr-xr-x 12 george george    194896 Dec 31  1969 mmap_hardlink_227_libxcb-xinput.so.0.1.0
-r-xr-xr-x  6 george george    151832 Dec 31  1969 mmap_hardlink_2251_libxfconf-0.so.3.0.0
-r-xr-xr-x  6 george george     79800 Dec 31  1969 mmap_hardlink_2246_libxfce4util.so.7.0.0
-r-xr-xr-x  6 george george    181808 Dec 31  1969 mmap_hardlink_2241_libgdk_pixbuf-2.0.so.0.4200.2
-r-xr-xr-x 12 george george     15648 Dec 31  1969 mmap_hardlink_223_libX11-xcb.so.1.0.0
-r-xr-xr-x  6 george george   1481192 Dec 31  1969 mmap_hardlink_2236_libcairo.so.2.11600.0
-r-xr-xr-x  6 george george    392200 Dec 31  1969 mmap_hardlink_2232_libpango-1.0.so.0.4700.0
-r-xr-xr-x  6 george george   1298864 Dec 31  1969 mmap_hardlink_2227_libgdk-3.so.0.2404.20
-r-xr-xr-x  6 george george   9275000 Dec 31  1969 mmap_hardlink_2222_libgtk-3.so.0.2404.20
-r-xr-xr-x  6 george george    120904 Dec 31  1969 mmap_hardlink_2218_libxfce4ui-2.so.0.0.0
-r-xr-xr-x  6 george george    706064 Dec 31  1969 mmap_hardlink_2213_libvte-2.91.so.0.6200.1
-r-xr-xr-x 11 george george     33304 Dec 31  1969 mmap_hardlink_219_libxcb-icccm.so.4.0.0
-r-xr-xr-x  6 george george    307832 Dec 31  1969 mmap_hardlink_2190_.xfce4-terminal-wrapped
-r-xr-xr-x 12 george george     45648 Dec 31  1969 mmap_hardlink_214_libxcb-xfixes.so.0.0.0
-r-xr-xr-x 12 george george     77528 Dec 31  1969 mmap_hardlink_210_libxcb-render.so.0.0.0
-r-xr-xr-x  5 george george     55568 Dec 31  1969 mmap_hardlink_20_librrpreload.so
-r-xr-xr-x 12 george george     17944 Dec 31  1969 mmap_hardlink_206_libxcb-composite.so.0.0.0
-r-xr-xr-x  6 george george    920472 Dec 31  1969 mmap_hardlink_2061_bash
-r-xr-xr-x 12 george george    261264 Dec 31  1969 mmap_hardlink_202_libudev.so.1.7.0
-r-xr-xr-x 11 george george    391656 Dec 31  1969 mmap_hardlink_198_libinput.so.10.13.0
-r-xr-xr-x 11 george george     21312 Dec 31  1969 mmap_hardlink_1979_libXmuu.so.1.0.0
-r-xr-xr-x 11 george george    319928 Dec 31  1969 mmap_hardlink_1944_libEGL_mesa.so.0.0.0
-r-xr-xr-x 12 george george     70048 Dec 31  1969 mmap_hardlink_193_libgbm.so.1.0.0
-r-xr-xr-x 11 george george     42096 Dec 31  1969 mmap_hardlink_1929_xrdb
-r-xr-xr-x 12 george george     93184 Dec 31  1969 mmap_hardlink_189_libdrm.so.2.4.0
-r-xr-xr-x  8 george george     27120 Dec 31  1969 mmap_hardlink_1862_libkeyutils.so.1.10
-r-xr-xr-x 12 george george     90696 Dec 31  1969 mmap_hardlink_185_libGLESv2.so.2.1.0
-r--r--r--  8 george george     71528 Dec 31  1969 mmap_hardlink_1859_libkrb5support.so.0.1
-r--r--r--  8 george george     22664 Dec 31  1969 mmap_hardlink_1835_libcom_err.so.3.0
-r--r--r--  8 george george    207320 Dec 31  1969 mmap_hardlink_1830_libk5crypto.so.3.1
-r--r--r--  8 george george   1066632 Dec 31  1969 mmap_hardlink_1826_libkrb5.so.3.3
-r--r--r--  8 george george    384768 Dec 31  1969 mmap_hardlink_1822_libgssapi_krb5.so.2.2
-r-xr-xr-x 12 george george    100912 Dec 31  1969 mmap_hardlink_181_libEGL.so.1.1.0
-r-xr-xr-x 12 george george    106288 Dec 31  1969 mmap_hardlink_1813_libwayland-server.so.0.1.0
-r-xr-xr-x  9 george george    257392 Dec 31  1969 mmap_hardlink_1800_libpng16.so.16.37.0
-r-xr-xr-x  9 george george     78984 Dec 31  1969 mmap_hardlink_1796_libbz2.so.1.0.6
-r-xr-xr-x  9 george george    883320 Dec 31  1969 mmap_hardlink_1792_libfreetype.so.6.17.4
-r-xr-xr-x  9 george george     40312 Dec 31  1969 mmap_hardlink_1786_libfontenc.so.1.0.0
-r-xr-xr-x 12 george george     15920 Dec 31  1969 mmap_hardlink_177_libwayland-egl.so.1.0.0
-r-xr-xr-x  8 george george    223792 Dec 31  1969 mmap_hardlink_1762_libtirpc.so.3.0.0
-r-xr-xr-x  9 george george   3374240 Dec 31  1969 mmap_hardlink_1757_libcrypto.so.1.1
-r-xr-xr-x 12 george george     80344 Dec 31  1969 mmap_hardlink_173_libwayland-client.so.0.3.0
-r-xr-xr-x  9 george george   1826472 Dec 31  1969 mmap_hardlink_1736_libepoxy.so.0.0.0
-r-xr-xr-x  9 george george    261808 Dec 31  1969 mmap_hardlink_1720_libXfont2.so.2.0.0
-r-xr-xr-x  8 george george  18956928 Dec 31  1969 mmap_hardlink_1695_Xwayland
-r-xr-xr-x 12 george george    965048 Dec 31  1969 mmap_hardlink_167_libsystemd.so.0.30.0
-r-xr-xr-x 12 george george   1030752 Dec 31  1969 mmap_hardlink_162_libGLdispatch.so.0.0.0
-r-xr-xr-x 11 george george    359536 Dec 31  1969 mmap_hardlink_155_libGLX.so.0.0.0
-r--r--r-- 12 george george     43896 Dec 31  1969 mmap_hardlink_152_libcap.so.2.47
-r-xr-xr-x 11 george george    414280 Dec 31  1969 mmap_hardlink_147_libdbus-1.so.3.19.13
-r-xr-xr-x 11 george george    575272 Dec 31  1969 mmap_hardlink_142_libsndfile.so.1.0.30
-r-xr-xr-x 11 george george    705008 Dec 31  1969 mmap_hardlink_1423_libgmp.so.10.4.1
-r-xr-xr-x 11 george george     17648 Dec 31  1969 mmap_hardlink_1419_libutil-2.32.so
-r-xr-xr-x  5 george george   6359664 Dec 31  1969 mmap_hardlink_1417_libHSghc-prim-0.6.1-ghc8.10.4.so
-r-xr-xr-x  5 george george    336696 Dec 31  1969 mmap_hardlink_1415_libHSinteger-gmp-1.0.3.0-ghc8.10.4.so
-r-xr-xr-x  5 george george  13159104 Dec 31  1969 mmap_hardlink_1412_libHSbase-4.14.1.0-ghc8.10.4.so
-r-xr-xr-x  5 george george    685648 Dec 31  1969 mmap_hardlink_1410_libHSarray-0.5.4.0-ghc8.10.4.so
-r-xr-xr-x  5 george george    177192 Dec 31  1969 mmap_hardlink_1408_libHSdeepseq-1.4.4.0-ghc8.10.4.so
-r-xr-xr-x  5 george george   4261272 Dec 31  1969 mmap_hardlink_1406_libHScontainers-0.6.2.1-ghc8.10.4.so
-r-xr-xr-x  5 george george   1963632 Dec 31  1969 mmap_hardlink_1404_libHStime-1.9.3-ghc8.10.4.so
-r-xr-xr-x  5 george george    351760 Dec 31  1969 mmap_hardlink_1402_libHSrandom-1.1-IcUwfYF3CGOHS82o6ktbLi-ghc8.10.4.so
-r-xr-xr-x  5 george george    105368 Dec 31  1969 mmap_hardlink_1400_libHSsplitmix-0.1.0.3-1YkuweqDYq8DBxer1Np43T-ghc8.10.4.so
-r-xr-xr-x  5 george george    371464 Dec 31  1969 mmap_hardlink_1398_libHSghc-boot-th-8.10.4-ghc8.10.4.so
-r-xr-xr-x  5 george george    599864 Dec 31  1969 mmap_hardlink_1396_libHSpretty-1.1.3.6-ghc8.10.4.so
-r-xr-xr-x  5 george george   4801896 Dec 31  1969 mmap_hardlink_1394_libHStemplate-haskell-2.16.0.0-ghc8.10.4.so
-r-xr-xr-x  5 george george   1846344 Dec 31  1969 mmap_hardlink_1392_libHStransformers-0.5.6.2-ghc8.10.4.so
-r-xr-xr-x  5 george george   1983184 Dec 31  1969 mmap_hardlink_1390_libHSQuickCheck-2.14.2-emyW1cBTLx2Wxf2sOiMGt-ghc8.10.4.so
-r-xr-xr-x 11 george george    637312 Dec 31  1969 mmap_hardlink_138_libpulsecommon-14.2.so
-r-xr-xr-x  5 george george   1279968 Dec 31  1969 mmap_hardlink_1388_libHSbytestring-0.10.12.0-ghc8.10.4.so
-r-xr-xr-x  5 george george    850352 Dec 31  1969 mmap_hardlink_1386_libHSbinary-0.8.8.0-ghc8.10.4.so
-r-xr-xr-x  5 george george   2663288 Dec 31  1969 mmap_hardlink_1384_libHStext-1.2.4.1-ghc8.10.4.so
-r-xr-xr-x  5 george george    239248 Dec 31  1969 mmap_hardlink_1382_libHShashable-1.3.0.0-114VjR6BLynK0LtEQSxPrz-ghc8.10.4.so
-r-xr-xr-x  5 george george     83024 Dec 31  1969 mmap_hardlink_1380_libHSinteger-logarithms-1.0.3.1-FNyUGQZIN2EkBxPD20xnp-ghc8.10.4.so
-r-xr-xr-x  5 george george    852424 Dec 31  1969 mmap_hardlink_1378_libHSprimitive-0.7.1.0-Jxsyd70oUttYiCXCa0HqV-ghc8.10.4.so
-r-xr-xr-x  5 george george    362496 Dec 31  1969 mmap_hardlink_1376_libHSscientific-0.3.6.2-Jfq1KDfx183TvepkXe9HB-ghc8.10.4.so
-r-xr-xr-x  5 george george   1822104 Dec 31  1969 mmap_hardlink_1374_libHSattoparsec-0.13.2.5-3pf622GQJByEVRvPPx3aMy-ghc8.10.4.so
-r-xr-xr-x  5 george george   1338664 Dec 31  1969 mmap_hardlink_1372_libHSunix-2.7.2.2-ghc8.10.4.so
-r-xr-xr-x  5 george george    124584 Dec 31  1969 mmap_hardlink_1370_libHSbase-compat-0.11.2-JggHr2fdYLe83DSFk1MYK3-ghc8.10.4.so
-r-xr-xr-x  5 george george    132624 Dec 31  1969 mmap_hardlink_1368_libHSbase-compat-batteries-0.11.2-CVOjUJpfJjlDnvMfaMHtvH-ghc8.10.4.so
-r-xr-xr-x  5 george george    100072 Dec 31  1969 mmap_hardlink_1366_libHSdata-fix-0.3.1-FcmoSBUpMIeFCPX7eYAHmQ-ghc8.10.4.so
-r-xr-xr-x  5 george george     79120 Dec 31  1969 mmap_hardlink_1364_libHSdlist-0.8.0.8-FJ4hncuTyhRHtpRjYcZTlQ-ghc8.10.4.so
-r-xr-xr-x  5 george george    257272 Dec 31  1969 mmap_hardlink_1362_libHSbase-orphans-0.8.4-LBjA9BA9X8o4iaOtW6aOUx-ghc8.10.4.so
-r-xr-xr-x  5 george george    177672 Dec 31  1969 mmap_hardlink_1360_libHStagged-0.8.6.1-HwwcL9gkCbnAPGOH1AhzJ5-ghc8.10.4.so
-r-xr-xr-x  5 george george    119304 Dec 31  1969 mmap_hardlink_1358_libHSdistributive-0.6.2.1-6pF78RnfUJsOC2ijESnbi-ghc8.10.4.so
-r-xr-xr-x  5 george george    361992 Dec 31  1969 mmap_hardlink_1356_libHSindexed-traversable-0.1.1-Dw8MIJyvCj8IZQDXOa8TBp-ghc8.10.4.so
-r-xr-xr-x  5 george george    230816 Dec 31  1969 mmap_hardlink_1354_libHStransformers-compat-0.6.6-5ZJJ9POl5DGLIWTn4Z8HjM-ghc8.10.4.so
-r-xr-xr-x  5 george george    299240 Dec 31  1969 mmap_hardlink_1352_libHScomonad-5.0.8-8kp9TBGsZVb5Aey7zxbqP0-ghc8.10.4.so
-r-xr-xr-x  5 george george    628272 Dec 31  1969 mmap_hardlink_1350_libHSth-abstraction-0.4.2.0-CPMLTlyMgmr6dbHxHL95CG-ghc8.10.4.so
-r-xr-xr-x  5 george george   1327048 Dec 31  1969 mmap_hardlink_1348_libHSbifunctors-5.5.10-CHJFsdhExY9A2muaJ0rpX6-ghc8.10.4.so
-r-xr-xr-x  5 george george     35864 Dec 31  1969 mmap_hardlink_1346_libHSassoc-1.0.2-1nKRaKtLcaUFScUJparWKG-ghc8.10.4.so
-r-xr-xr-x  5 george george    278744 Dec 31  1969 mmap_hardlink_1344_libHSthese-1.1.1.1-80pQUjXMbPe2nosshkpqyP-ghc8.10.4.so
-r-xr-xr-x  5 george george    514736 Dec 31  1969 mmap_hardlink_1342_libHSstrict-0.4.0.1-5Vmvm89RscDLq36yJxFYX7-ghc8.10.4.so
-r-xr-xr-x  5 george george    393880 Dec 31  1969 mmap_hardlink_1340_libHStime-compat-1.9.5-I1tHGrNfoIT6zPhkHvFW8y-ghc8.10.4.so
-r-xr-xr-x 12 george george     49408 Dec 31  1969 mmap_hardlink_133_librt-2.32.so
-r-xr-xr-x  5 george george    646824 Dec 31  1969 mmap_hardlink_1338_libHSunordered-containers-0.2.13.0-5Bv3z5ishlO9UjabQbJ5ZW-ghc8.10.4.so
-r-xr-xr-x  5 george george    189384 Dec 31  1969 mmap_hardlink_1336_libHSuuid-types-1.0.4-Jr6rU6iV0R74NWeYeRIW01-ghc8.10.4.so
-r-xr-xr-x  5 george george   4697984 Dec 31  1969 mmap_hardlink_1334_libHSvector-0.12.2.0-1MXlgbkUTwY2zc9kzwivVC-ghc8.10.4.so
-r-xr-xr-x  5 george george   6550464 Dec 31  1969 mmap_hardlink_1332_libHSaeson-1.5.6.0-H6ldsAOmUwm3S1sjDiy157-ghc8.10.4.so
-r-xr-xr-x  5 george george     98688 Dec 31  1969 mmap_hardlink_1330_libHSbase64-bytestring-1.1.0.0-5eIY92lnzMD3onHt2Rfy9M-ghc8.10.4.so
-r-xr-xr-x  5 george george    149424 Dec 31  1969 mmap_hardlink_1328_libHSclock-0.8-3vNRSIIFvuh5T6Bf3yeRJC-ghc8.10.4.so
-r-xr-xr-x  5 george george    532888 Dec 31  1969 mmap_hardlink_1326_libHScolour-2.3.5-C29WD8Qg69AL3qeQgayHy7-ghc8.10.4.so
-r-xr-xr-x  5 george george    207096 Dec 31  1969 mmap_hardlink_1324_libHSDiff-0.4.0-5OxCl91wLdqAVmXAelKfev-ghc8.10.4.so
-r-xr-xr-x  5 george george     75832 Dec 31  1969 mmap_hardlink_1322_libHSaeson-pretty-0.8.8-KgyR6dhcVdPHDpp3NheECp-ghc8.10.4.so
-r-xr-xr-x  5 george george    511712 Dec 31  1969 mmap_hardlink_1320_libHSansi-terminal-0.10.3-Bgh2DXQxNh2EIUXvXgvcLq-ghc8.10.4.so
-r-xr-xr-x  5 george george    187968 Dec 31  1969 mmap_hardlink_1318_libHSfilepath-1.4.2.1-ghc8.10.4.so
-r-xr-xr-x  5 george george    587760 Dec 31  1969 mmap_hardlink_1316_libHSdirectory-1.3.6.0-ghc8.10.4.so
-r-xr-xr-x  5 george george    311256 Dec 31  1969 mmap_hardlink_1314_libHSmtl-2.2.2-ghc8.10.4.so
-r-xr-xr-x  5 george george    191800 Dec 31  1969 mmap_hardlink_1312_libHSstm-2.5.0.0-ghc8.10.4.so
-r-xr-xr-x  5 george george    329584 Dec 31  1969 mmap_hardlink_1310_libHSexceptions-0.10.4-ghc8.10.4.so
-r-xr-xr-x  5 george george     44760 Dec 31  1969 mmap_hardlink_1308_libHStemporary-1.3-4G3dC9t4lqQLtgWPalhcva-ghc8.10.4.so
-r-xr-xr-x  5 george george     19144 Dec 31  1969 mmap_hardlink_1306_libHSunix-compat-0.5.3-E98n0JyMac7Bz478oqWJHX-ghc8.10.4.so
-r-xr-xr-x  5 george george     81544 Dec 31  1969 mmap_hardlink_1304_libHSatomic-write-0.2.0.7-L5OTDNItoVaBLpXFDFw3sd-ghc8.10.4.so
-r-xr-xr-x  5 george george     91376 Dec 31  1969 mmap_hardlink_1302_libHScase-insensitive-1.2.1.0-4CodBW8TUSU73yivWG0KSf-ghc8.10.4.so
-r-xr-xr-x  5 george george    129824 Dec 31  1969 mmap_hardlink_1300_libHShalf-0.3.1-Gt2dDugMbNl1KM8wYfphWP-ghc8.10.4.so
-r-xr-xr-x  5 george george   1416656 Dec 31  1969 mmap_hardlink_1298_libHScborg-0.2.4.0-BE0VjOk0rSe9xJOiRp5dr8-ghc8.10.4.so
-r-xr-xr-x  5 george george     56488 Dec 31  1969 mmap_hardlink_1296_libHScborg-json-0.2.2.0-IEjVIaJYAEHBHBsBdmyev8-ghc8.10.4.so
-r-xr-xr-x  5 george george     49128 Dec 31  1969 mmap_hardlink_1294_libHSStateVar-1.2.1-7vo3tV8mPssJqOV48TU4OF-ghc8.10.4.so
-r-xr-xr-x  5 george george    204552 Dec 31  1969 mmap_hardlink_1292_libHScontravariant-1.5.3-Ac4TAzGGDBLeGMWs4L1e-ghc8.10.4.so
-r-xr-xr-x  5 george george   3138704 Dec 31  1969 mmap_hardlink_1290_libHSbasement-0.0.11-CejmV4k3OtGGKPkDYOlSof-ghc8.10.4.so
-r-xr-xr-x 11 george george   2076736 Dec 31  1969 mmap_hardlink_128_libc-2.32.so
-r-xr-xr-x  5 george george    668992 Dec 31  1969 mmap_hardlink_1288_libHSmemory-0.15.0-83dn9b3rZzC25EOnb2sblF-ghc8.10.4.so
-r-xr-xr-x  5 george george   6805744 Dec 31  1969 mmap_hardlink_1285_libHScryptonite-0.27-6jiLzjhO6c8KkfOkJvcva-ghc8.10.4.so
-r-xr-xr-x  5 george george     71744 Dec 31  1969 mmap_hardlink_1283_libHSdotgen-0.4.3-5l59B5e0ZPN2MJb4kFEAyj-ghc8.10.4.so
-r-xr-xr-x  5 george george    906328 Dec 31  1969 mmap_hardlink_1281_libHSprofunctors-5.5.2-1XFqdCrn0zRCiPNqTRbjwt-ghc8.10.4.so
-r-xr-xr-x  5 george george   1059344 Dec 31  1969 mmap_hardlink_1279_libHSsemigroupoids-5.3.5-BbGqnG7leLdHUa74Rjgyw0-ghc8.10.4.so
-r-xr-xr-x  5 george george    103416 Dec 31  1969 mmap_hardlink_1277_libHSeither-5.0.1.1-8u8U5ZMZ4Db64ClCgDSplJ-ghc8.10.4.so
-r-xr-xr-x  5 george george    339256 Dec 31  1969 mmap_hardlink_1275_libHSprocess-1.6.9.0-ghc8.10.4.so
-r-xr-xr-x  5 george george    350320 Dec 31  1969 mmap_hardlink_1273_libHSterminfo-0.4.1.4-ghc8.10.4.so
-r-xr-xr-x  5 george george   2175320 Dec 31  1969 mmap_hardlink_1271_libHShaskeline-0.8.0.1-ghc8.10.4.so
-r-xr-xr-x  5 george george    167808 Dec 31  1969 mmap_hardlink_1269_libHSblaze-builder-0.4.2.1-8DyVSB3DbYmGwGeiny4STQ-ghc8.10.4.so
-r-xr-xr-x  5 george george     31880 Dec 31  1969 mmap_hardlink_1267_libHSdata-default-class-0.1.2.0-IIN1s3V8yfYEDHe5yjxXHV-ghc8.10.4.so
-r-xr-xr-x  5 george george    109792 Dec 31  1969 mmap_hardlink_1265_libHScookie-0.4.5-IZJUrKKrfQBLOA7wMhWWoP-ghc8.10.4.so
-r-xr-xr-x  5 george george    432072 Dec 31  1969 mmap_hardlink_1263_libHShttp-types-0.12.3-UYUXnsOx2u9EfNOs11ZUE-ghc8.10.4.so
-r-xr-xr-x  5 george george   1615960 Dec 31  1969 mmap_hardlink_1261_libHSmime-types-0.1.0.9-1XpBorrtO1G2nfqA8lHD1Y-ghc8.10.4.so
-r--r--r-- 11 george george    112336 Dec 31  1969 mmap_hardlink_125_libgcc_s.so.1
-r-xr-xr-x  5 george george    759440 Dec 31  1969 mmap_hardlink_1259_libHSnetwork-3.1.1.1-F91Y9KbLQh5FWIKWxm6vtC-ghc8.10.4.so
-r-xr-xr-x  5 george george    878920 Dec 31  1969 mmap_hardlink_1257_libHSparsec-3.1.14.0-ghc8.10.4.so
-r-xr-xr-x  5 george george     73176 Dec 31  1969 mmap_hardlink_1255_libHSth-compat-0.1.1-EyHbnTl15in4KAoAYMPZ6H-ghc8.10.4.so
-r-xr-xr-x  5 george george    453144 Dec 31  1969 mmap_hardlink_1253_libHSnetwork-uri-2.6.4.1-LYNZLvtPiw0Cxy5b0hjgxS-ghc8.10.4.so
-r-xr-xr-x  5 george george    158880 Dec 31  1969 mmap_hardlink_1251_libHSasync-2.2.3-4nMcpgUlhOW1VgbG4skDoI-ghc8.10.4.so
-r-xr-xr-x  5 george george    497912 Dec 31  1969 mmap_hardlink_1249_libHSzlib-0.6.2.3-4VZuTR0saG5KSYyb6UEWDm-ghc8.10.4.so
-r-xr-xr-x  5 george george    643216 Dec 31  1969 mmap_hardlink_1247_libHSstreaming-commons-0.2.2.1-EpoEy6Rn0h5szzcbmVpSF-ghc8.10.4.so
-r-xr-xr-x  5 george george   1234288 Dec 31  1969 mmap_hardlink_1245_libHShttp-client-0.6.4.1-Hz8iO8TfykY7YHJFqxLdJW-ghc8.10.4.so
-r-xr-xr-x  5 george george    597144 Dec 31  1969 mmap_hardlink_1243_libHScereal-0.5.8.1-DIyqTvyKKj09hmXOoBMIfC-ghc8.10.4.so
-r-xr-xr-x  5 george george    608496 Dec 31  1969 mmap_hardlink_1241_libHSsocks-0.6.1-CGydb7kaxlh1b8Jk10nfgR-ghc8.10.4.so
-r-xr-xr-x  5 george george   1137264 Dec 31  1969 mmap_hardlink_1239_libHShourglass-0.2.12-90fGM11gDtiIBFhNAjaLNG-ghc8.10.4.so
-r-xr-xr-x  5 george george    326472 Dec 31  1969 mmap_hardlink_1237_libHSasn1-types-0.3.4-Izb99Cgs3LLKvZws0WfSfb-ghc8.10.4.so
-r-xr-xr-x  5 george george    320688 Dec 31  1969 mmap_hardlink_1235_libHSasn1-encoding-0.9.6-8TkvjtyBeXrCT2Ys25UjUb-ghc8.10.4.so
-r-xr-xr-x  5 george george     44976 Dec 31  1969 mmap_hardlink_1233_libHSasn1-parse-0.9.5-4a3tTYQtr4MHnx2gx39Skn-ghc8.10.4.so
-r-xr-xr-x  5 george george     62632 Dec 31  1969 mmap_hardlink_1231_libHSpem-0.2.4-ItvYCWnhN09IqzoLULDpJb-ghc8.10.4.so
-r-xr-xr-x  5 george george   1084264 Dec 31  1969 mmap_hardlink_1229_libHSx509-1.7.5-8h6kp7UzmKn6rwX7Onr23q-ghc8.10.4.so
-r-xr-xr-x  5 george george     96416 Dec 31  1969 mmap_hardlink_1227_libHSx509-store-1.6.7-5c5f9KtWS7C7XZX7P6wY2p-ghc8.10.4.so
-r-xr-xr-x  5 george george    219040 Dec 31  1969 mmap_hardlink_1225_libHSx509-validation-1.6.11-7FVowmoaTWq8B5sOtbWuAJ-ghc8.10.4.so
-r-xr-xr-x  5 george george   4553992 Dec 31  1969 mmap_hardlink_1223_libHStls-1.5.5-I1iYLpHlstk9E1K5YGHdWb-ghc8.10.4.so
-r-xr-xr-x  5 george george     36840 Dec 31  1969 mmap_hardlink_1221_libHSx509-system-1.6.6-8ITfEq16fOi3pvwH5HYeA9-ghc8.10.4.so
-r-xr-xr-x  5 george george    200528 Dec 31  1969 mmap_hardlink_1219_libHSconnection-0.3.1-SfGQg1QGfe4KaGwufPhYw-ghc8.10.4.so
-r-xr-xr-x  5 george george    161472 Dec 31  1969 mmap_hardlink_1217_libHShttp-client-tls-0.3.5.3-4QD45g74oU2EOsLdsCA4ub-ghc8.10.4.so
-r-xr-xr-x  5 george george    211000 Dec 31  1969 mmap_hardlink_1215_libHSlens-family-core-2.0.0-EwMZyf7N3mnDrHrw1CJKwH-ghc8.10.4.so
-r-xr-xr-x  5 george george    135984 Dec 31  1969 mmap_hardlink_1213_libHSparser-combinators-1.2.1-3NaZnLrWosAAXRnVPLCGYf-ghc8.10.4.so
-r-xr-xr-x  5 george george   1311152 Dec 31  1969 mmap_hardlink_1211_libHSmegaparsec-9.0.1-GLa9z0qeLKCA5WNKOpq73s-ghc8.10.4.so
-r-xr-xr-x 12 george george   1422328 Dec 31  1969 mmap_hardlink_120_libm-2.32.so
-r-xr-xr-x  5 george george     99072 Dec 31  1969 mmap_hardlink_1209_libHSmmorph-1.1.5-35UKpZuDVOl8bcf8tICns3-ghc8.10.4.so
-r-xr-xr-x  5 george george    196088 Dec 31  1969 mmap_hardlink_1207_libHSansi-wl-pprint-0.6.9-6gLwvNUqviPFfWwbvEGsaw-ghc8.10.4.so
-r-xr-xr-x  5 george george    868912 Dec 31  1969 mmap_hardlink_1205_libHSoptparse-applicative-0.15.1.0-1NlWv7wr2aM4qnx27SYgil-ghc8.10.4.so
-r-xr-xr-x  5 george george    843784 Dec 31  1969 mmap_hardlink_1203_libHScharset-0.3.8-DHSncSnkf3V5uga6YVPFjt-ghc8.10.4.so
-r-xr-xr-x  5 george george     31784 Dec 31  1969 mmap_hardlink_1201_libHSsemigroups-0.19.1-FzieqrLdZI71C64LnuY3Qi-ghc8.10.4.so
-r-xr-xr-x  5 george george    944936 Dec 31  1969 mmap_hardlink_1199_libHSparsers-0.12.10-2FLMJBdccRjDMklqNrTtgj-ghc8.10.4.so
-r-xr-xr-x  5 george george    956552 Dec 31  1969 mmap_hardlink_1197_libHSprettyprinter-1.7.0-MyQaq6vEptLUwfImcZX6Q-ghc8.10.4.so
-r-xr-xr-x  5 george george    204880 Dec 31  1969 mmap_hardlink_1195_libHSprettyprinter-ansi-terminal-1.1.2-9Zcj3orVVEwAdJYn5kp49g-ghc8.10.4.so
-r-xr-xr-x  5 george george    399512 Dec 31  1969 mmap_hardlink_1193_libHSpretty-simple-4.0.0.0-EvXtxAKEv0QDXZGsg5jwXp-ghc8.10.4.so
-r-xr-xr-x  5 george george    127200 Dec 31  1969 mmap_hardlink_1191_libHSrepline-0.4.0.0-8lX7VaX9rLqKN9TPXtwf8C-ghc8.10.4.so
-r-xr-xr-x  5 george george    593464 Dec 31  1969 mmap_hardlink_1189_libHSserialise-0.2.3.0-8dOmcr7VSWF57bxH6k5YbB-ghc8.10.4.so
-r-xr-xr-x  5 george george    460336 Dec 31  1969 mmap_hardlink_1187_libHStext-manipulate-0.2.0.1-2Or7OalAr5jFBc4ourDtvp-ghc8.10.4.so
-r-xr-xr-x  5 george george    112048 Dec 31  1969 mmap_hardlink_1185_libHSth-lift-0.8.2-7d37mrJ37yC1VcDgqjDphG-ghc8.10.4.so
-r-xr-xr-x  5 george george     79416 Dec 31  1969 mmap_hardlink_1183_libHSth-lift-instances-0.1.18-FWPftYgQyLKL7QhxN2zylD-ghc8.10.4.so
-r-xr-xr-x  5 george george    222592 Dec 31  1969 mmap_hardlink_1181_libHSutf8-string-1.0.2-HmQGdKRfc81Auhtk5Kfhkd-ghc8.10.4.so
-r-xr-xr-x  5 george george     31648 Dec 31  1969 mmap_hardlink_1179_libHSuri-encode-1.5.0.7-CazIvSmolKl4Vb2Jn4apyV-ghc8.10.4.so
-r-xr-xr-x  5 george george   8831136 Dec 31  1969 mmap_hardlink_1177_libHSdhall-1.38.0-1u20xkXW4ct67VCLEMqVMt-ghc8.10.4.so
-r-xr-xr-x  5 george george    389040 Dec 31  1969 mmap_hardlink_1175_libHSextra-1.7.9-Ilu3azSsLG1LSO0ZqY6d5F-ghc8.10.4.so
-r-xr-xr-x  5 george george     41344 Dec 31  1969 mmap_hardlink_1169_libHScall-stack-0.2.0-1Bicsj1gQrKCmn9kZBvIe8-ghc8.10.4.so
-r-xr-xr-x  5 george george     93704 Dec 31  1969 mmap_hardlink_1167_libHStransformers-base-0.4.5.2-D02NtOQve6bLl3VgOBxG6p-ghc8.10.4.so
-r-xr-xr-x  5 george george   1905584 Dec 31  1969 mmap_hardlink_1165_libHSfree-5.1.5-CyQF3ogsgGjEGJzj0SmWOw-ghc8.10.4.so
-r-xr-xr-x  5 george george     10480 Dec 31  1969 mmap_hardlink_1163_libHSvoid-0.7.3-1OlEP2jDqdyKnlVWzSlxMT-ghc8.10.4.so
-r-xr-xr-x  5 george george    543632 Dec 31  1969 mmap_hardlink_1161_libHSadjunctions-4.4-Hqo4JsDpLZoIlVu5nB5jH8-ghc8.10.4.so
-r-xr-xr-x 11 george george   1903088 Dec 31  1969 mmap_hardlink_115_libstdc++.so.6.0.28
-r-xr-xr-x  5 george george    533032 Dec 31  1969 mmap_hardlink_1159_libHSinvariant-0.5.4-2ufhEHdSCAzJ8cPQGxk2uo-ghc8.10.4.so
-r-xr-xr-x  5 george george    554896 Dec 31  1969 mmap_hardlink_1157_libHSkan-extensions-5.2.2-FKKfnilRDhVK06cLSXtRSt-ghc8.10.4.so
-r-xr-xr-x  5 george george    123496 Dec 31  1969 mmap_hardlink_1155_libHSparallel-3.2.2.0-HTf2o2horULDwL7cXUB9uv-ghc8.10.4.so
-r-xr-xr-x  5 george george    840152 Dec 31  1969 mmap_hardlink_1153_libHSreflection-2.1.6-4ZV0dCLPUM9IqXJWa3PlSJ-ghc8.10.4.so
-r-xr-xr-x  5 george george   5650656 Dec 31  1969 mmap_hardlink_1151_libHSlens-4.19.2-3h3gR4O7JSzKicKcuduw5z-ghc8.10.4.so
-r-xr-xr-x  5 george george     14544 Dec 31  1969 mmap_hardlink_1147_libHSbinary-orphans-1.0.1-DUKQmLqQOFJ7KnulXAFPc1-ghc8.10.4.so
-r-xr-xr-x  5 george george    832656 Dec 31  1969 mmap_hardlink_1145_libHSbytes-0.17.1-9SIDKDtJ2QD8FLGVV24LYJ-ghc8.10.4.so
-r-xr-xr-x  5 george george   3516624 Dec 31  1969 mmap_hardlink_1143_libHSlinear-1.21.5-2xg0tsDoKuX869LaC7tM4n-ghc8.10.4.so
-r-xr-xr-x  5 george george    245424 Dec 31  1969 mmap_hardlink_1139_libHSHUnit-1.6.1.0-1oUFApdrOSJDWeVrB91EiJ-ghc8.10.4.so
-r-xr-xr-x  5 george george     94992 Dec 31  1969 mmap_hardlink_1137_libHShspec-expectations-0.8.2-7ypHAbEJFB7JWzAUHDgNlW-ghc8.10.4.so
-r-xr-xr-x  5 george george     45440 Dec 31  1969 mmap_hardlink_1135_libHSquickcheck-io-0.2.0-ECEEhYvmIL17Wow2sGtsox-ghc8.10.4.so
-r-xr-xr-x  5 george george     10448 Dec 31  1969 mmap_hardlink_1133_libHSsetenv-0.1.1.3-FjFLzlxhySgkTKHvS1WwL-ghc8.10.4.so
-r-xr-xr-x  5 george george    199048 Dec 31  1969 mmap_hardlink_1131_libHStf-random-0.5-JO7vR7ZhK2c7n3X147ieRZ-ghc8.10.4.so
-r-xr-xr-x  5 george george   1312384 Dec 31  1969 mmap_hardlink_1129_libHShspec-core-2.7.8-E2BQzUTkm7a6YV84KnFgsS-ghc8.10.4.so
-r-xr-xr-x  5 george george    154120 Dec 31  1969 mmap_hardlink_1127_libHShspec-discover-2.7.8-2d5Xuqc1BEgLYK1dWswix-ghc8.10.4.so
-r-xr-xr-x  5 george george     54456 Dec 31  1969 mmap_hardlink_1125_libHShspec-2.7.8-23t5Hrhoz3e5QuA7ofzTn6-ghc8.10.4.so
-r-xr-xr-x  5 george george   1655584 Dec 31  1969 mmap_hardlink_1123_libHSinline-c-0.9.1.4-1EWTMzEiJuG6UBnGFTnTWQ-ghc8.10.4.so
-r-xr-xr-x  5 george george     23256 Dec 31  1969 mmap_hardlink_1121_libHSiso8601-time-0.1.5-KVOgncc5Jzi1K4Digk9WvB-ghc8.10.4.so
-r-xr-xr-x  5 george george    369824 Dec 31  1969 mmap_hardlink_1119_libHSordered-containers-0.2.2-GyU6swLq6k0DcCVw8rbPro-ghc8.10.4.so
-r-xr-xr-x  5 george george     92256 Dec 31  1969 mmap_hardlink_1117_libHSfmlist-0.9.4-Ehec1h15Yr5Gl7rT4eaULy-ghc8.10.4.so
-r-xr-xr-x  5 george george   2682544 Dec 31  1969 mmap_hardlink_1115_libHSListLike-4.7.4-J6J3vcdgxMS2fvtjKDnSaO-ghc8.10.4.so
-r-xr-xr-x  5 george george     10504 Dec 31  1969 mmap_hardlink_1113_libHSdata-default-instances-containers-0.0.1-GmmlqVT2wS4CIAeicFX9e0-ghc8.10.4.so
-r-xr-xr-x  5 george george     10456 Dec 31  1969 mmap_hardlink_1111_libHSdata-default-instances-dlist-0.0.1-ChO1ufxrWFt8egDPNFcveI-ghc8.10.4.so
-r-xr-xr-x 12 george george    223808 Dec 31  1969 mmap_hardlink_110_libxcb.so.1.1.0
-r-xr-xr-x  5 george george     93432 Dec 31  1969 mmap_hardlink_1109_libHSold-locale-1.0.0.7-LpTuchyagHAP7STZdm71P-ghc8.10.4.so
-r-xr-xr-x  5 george george     10480 Dec 31  1969 mmap_hardlink_1107_libHSdata-default-instances-old-locale-0.0.1-LkrhLZM06G59SMNGVzL0si-ghc8.10.4.so
-r-xr-xr-x  5 george george     10448 Dec 31  1969 mmap_hardlink_1105_libHSdata-default-0.7.1.1-CYtyoeRTyO02nIfJy16t2P-ghc8.10.4.so
-r-xr-xr-x  5 george george   2502648 Dec 31  1969 mmap_hardlink_1103_libHSgeneric-deriving-1.13.1-3lR3Zks7vLTHV1nFff8Ahj-ghc8.10.4.so
-r-xr-xr-x  5 george george    249576 Dec 31  1969 mmap_hardlink_1101_libHSprocess-extras-0.7.4-BOJGsanlHyi8AfZidBO5OS-ghc8.10.4.so
-r-xr-xr-x 11 george george    196896 Dec 31  1969 mmap_hardlink_10_ld-2.32.so
-r-xr-xr-x  5 george george     23376 Dec 31  1969 mmap_hardlink_1099_libHSraw-strings-qq-1.1-IJ5OGfWCmgO8CVbT9sVxHs-ghc8.10.4.so
-r-xr-xr-x  5 george george     91744 Dec 31  1969 mmap_hardlink_1097_libHSsafe-exceptions-0.1.7.1-4wWuKhvUCOjJDBwPuYEdDO-ghc8.10.4.so
-r-xr-xr-x  5 george george     40608 Dec 31  1969 mmap_hardlink_1095_libHScryptohash-md5-0.11.100.1-3QTHLlHPsllFYNWOyKQV4e-ghc8.10.4.so
-r-xr-xr-x  5 george george     40608 Dec 31  1969 mmap_hardlink_1093_libHScryptohash-sha1-0.11.100.1-9nHpanqCLA6G0Yq990k2zW-ghc8.10.4.so
-r-xr-xr-x  5 george george     41112 Dec 31  1969 mmap_hardlink_1091_libHSentropy-0.4.1.6-6Z0sejCw1fy7sJ1AU1oQ6y-ghc8.10.4.so
-r-xr-xr-x  5 george george     96680 Dec 31  1969 mmap_hardlink_1089_libHSnetwork-info-0.2.0.10-LJXZu231xIy8KJ0QI8k9yv-ghc8.10.4.so
-r-xr-xr-x  5 george george     79608 Dec 31  1969 mmap_hardlink_1087_libHSuuid-1.3.14-Fq2BikRFoAO7ihHMyG5SaL-ghc8.10.4.so
-r-xr-xr-x  5 george george    666280 Dec 31  1969 mmap_hardlink_1084_libHSrts_thr-ghc8.10.4.so
-r-xr-xr-x  5 george george     46632 Dec 31  1969 mmap_hardlink_106_libxcb-errors.so.0.0.0
-r-xr-xr-x  2 george george    515088 Dec 31  1969 mmap_hardlink_10548_libzeitgeist-2.0.so.0.0.0
-r-xr-xr-x  2 george george     22120 Dec 31  1969 mmap_hardlink_10487_libkeybinder-3.0.so.0.0.0
-r-xr-xr-x  2 george george   1113912 Dec 31  1969 mmap_hardlink_10479_libgee-0.8.so.2.6.1
-r-xr-xr-x  2 george george   1476824 Dec 31  1969 mmap_hardlink_10454_synapse
-r-xr-xr-x 12 george george    744424 Dec 31  1969 mmap_hardlink_102_libpixman-1.so.0.38.4
-r-xr-xr-x  3 george george   1905416 Dec 31  1969 mmap_hardlink_10255_libnm.so.0.1.0
-r-xr-xr-x  3 george george     22432 Dec 31  1969 mmap_hardlink_10251_network_networkmanager.so
-r-xr-xr-x  3 george george     51792 Dec 31  1969 mmap_hardlink_10247_config_kde.so
-r-xr-xr-x  3 george george     55336 Dec 31  1969 mmap_hardlink_10243_config_gnome3.so
-r-xr-xr-x  3 george george  16671176 Dec 31  1969 mmap_hardlink_10143_librsvg-2.so.2.47.0
-r-xr-xr-x  3 george george     16784 Dec 31  1969 mmap_hardlink_10139_libpixbufloader-svg.so
-r-xr-xr-x  5 george george     16320 Dec 31  1969 mmap_hardlink_1005_libv4l-mplane.so
-r-xr-xr-x  5 george george    740040 Dec 31  1969 mmap_hardlink_1001_libjpeg.so.62.3.0
-rwxr-xr-x 11 george george    175136 Mar 24 17:27 mmap_hardlink_1173_libHSsplit-0.2.3.4-997190f34aef1304ebada20bb2a2711270f05fc32b9bc98936eedab3b12578f9-ghc8.10.4.so
-rwxr-xr-x 11 george george     82976 Mar 24 17:27 mmap_hardlink_1171_libHScasing-0.1.4.1-18d6eff0f4633f5e2228abb92fc0f25c9667e26c48e47ba1a20213955e4de945-ghc8.10.4.so
-rwxrwxr-x  5 george george   1889912 Jun 28 21:47 mmap_hardlink_978_libgodot_openhmd.so
-rwxrwxr-x  5 george george  74916392 Jun 28 21:47 mmap_hardlink_981_libgodot_openvr.so
-rwxrwxr-x  5 george george    341416 Jun 28 21:47 mmap_hardlink_988_libgodot_openxr.so
-rw-rw-r--  5 george george   2694032 Jun 28 21:47 mmap_hardlink_986_libopenvr_api.so
-rwxr-xr-x  5 george george   3888160 Jun 29 11:19 mmap_hardlink_94_libwlroots.so.0.0.0
-rwxr-xr-x  5 george george   1745784 Jun 30 22:23 mmap_hardlink_1149_libHSgodot-haskell-3.1.0.0-inplace-generate-ghc8.10.4.so
-rwxr-xr-x  5 george george  49380472 Jul  2 18:42 mmap_hardlink_1141_libHSgodot-haskell-3.1.0.0-inplace-ghc8.10.4.so
-rwxr-xr-x  5 george george   5314192 Jul 16 18:14 mmap_hardlink_1082_libgodot-haskell-plugin.so
-rwxr-xr-x  5 george george 353920648 Jul 17 16:10 mmap_hardlink_3_godot.x11.tools.64
-r--------  1 george george         0 Jul 17 22:39 tasks
-rw-------  1 george george         3 Jul 17 22:39 incomplete
drwx------  8 george george      4096 Jul 17 22:39 ..
drwxrwx---  2 george george     36864 Jul 17 22:40 .
-r--------  1 george george    217921 Jul 17 22:41 mmaps
-r--------  1 george george   3933023 Jul 17 22:41 events
-r--------  1 george george 258194122 Jul 17 22:41 data

@rocallahan
Copy link
Collaborator

When we get the output return address 0x5618f2a7920b what's the disassembly of the code around 0x5618f2a7920b?

@rocallahan
Copy link
Collaborator

terminating the program (via Ctrl-c from the terminal) causes the terminal to hang

Try SIGKILLing the tracee (rr's child) instead.

@rocallahan
Copy link
Collaborator

Hmm, ok. The vsyscall page should be there ...

No, there should be nothing there.

The kernel's vsyscall emulation relies on that page being unmapped. Jumping there triggers a fault with RIP in the vsyscall page, and the kernel emulates everything from there.

@georgewsinger
Copy link
Author

Sending kill -9 <pid_of_rr_tracee> still fails to create a sendable trace afterwards.

I'll remove the vsyscall kernel paramater line, restart, and send those disassemblies now.

@georgewsinger
Copy link
Author

With

[FATAL /build/source/src/RecordSession.cc:347:handle_seccomp_traced_syscall()] 
 (task 3528 (rec:3528) at time 94016)
 -> Assertion `patch_ok' failed to hold. The tracee issues a vsyscall to 0xffffffffff600000 but we failed to moneypatch the caller (return address 0x56168997920b, sp=0x7fff74cc67a8). Recording will not succeed. Exiting.

I got

(gdb) x/20i 0x56168997920b
   0x56168997920b:	mov    (%rsp),%rax
   0x56168997920f <_start+3>:	mov    0x8(%rsp),%edx
   0x561689979213 <_start+7>:	mov    %rbp,%rsp
   0x561689979216 <_dl_start_user+2>:	movq   $0x0,0x328(%rbx)
   0x561689979221 <_dl_start_user+13>:	imul   $0x3e8,%rdx,%rdx
   0x561689979228 <_dl_start_user+20>:	imul   $0x3b9aca00,%rax,%rax
   0x56168997922f <_dl_start_user+27>:	add    %rdx,%rax
   0x561689979232 <_dl_start_user+30>:	mov    %rax,0x18(%rsp)
   0x561689979237 <_dl_start_user+35>:	mov    0x8(%rsp),%rbp
   0x56168997923c <_dl_start_user+40>:	add    $0x10,%rsp
   0x561689979240 <_dl_start_user+44>:	retq   
   0x561689979241 <_dl_start_user+45>:	int3   
   0x561689979242 <_dl_start_user+46>:	int3   
   0x561689979243 <_dl_start_user+47>:	int3   
   0x561689979244 <_dl_start_user+48>:	int3   
   0x561689979245 <_dl_start_user+49>:	int3   
   0x561689979246 <_dl_start_user+50>:	int3   
   0x561689979247 <_dl_start_user+51>:	int3   
   0x561689979248 <_dl_start_user+52>:	int3   
   0x561689979249 <_dl_start_user+53>:	int3  

@khuey
Copy link
Collaborator

khuey commented Jul 18, 2021

I think the tracee execed through a vsyscall and now we can't "patch the caller" because the caller is completely gone.

@rocallahan
Copy link
Collaborator

I don't think you can execve through a vsyscall.

@rocallahan
Copy link
Collaborator

Yeah, vsyscall can only do gettimeofday, time and getcpu.

@rocallahan
Copy link
Collaborator

0xffffffffff600000 is gettimeofday in particular.

(gdb) x/20i 0x56168997920b
   0x56168997920b:	mov    (%rsp),%rax
   0x56168997920f <_start+3>:	mov    0x8(%rsp),%edx

This return address we pulled from the stack is clearly is clearly _start, which makes no sense.

@khuey
Copy link
Collaborator

khuey commented Jul 18, 2021

@rocallahan
Copy link
Collaborator

Actually though ... the code in the disassembly here is clearly NOT _start. So I don't trust the symbols loaded here, they're not matching the code.

@rocallahan
Copy link
Collaborator

In fact it looks like we're pulling values from the stack, doing arithmetic on them and returning the result, just like you'd expect from a wrapper function calling gettimeofday to a struct timeval on the stack and then returning the result. Note that 0x3b9aca00/0x3e8 is 1,000,000 and the fields of struct timeval are seconds and microseconds.

@rocallahan
Copy link
Collaborator

I think the problem is simply that this code doesn't match the single very limited template we know how to patch for vsyscalls.

@georgewsinger can you disassemble the 256 bytes before and after that return address, with /r please?

@georgewsinger
Copy link
Author

@rocallahan What do you mean? x/10r <return address> seems to generate exactly the same result as x/10i <return address>, so I assume it's not that.

@khuey
Copy link
Collaborator

khuey commented Jul 19, 2021

Use <return_address> - 256 or similar.

@khuey
Copy link
Collaborator

khuey commented Jul 22, 2021

Over in #2939 this appears to be a PLT thunk trampolining directly into the vsyscall (where the GNU IFUNC stuff was used to select an implementation at runtime). That's definitely not going to fit the X64VSyscallEntry template.

@GitMensch
Copy link
Contributor

@georgewsinger: Can you please drop the output of uname -a?

@rocallahan
Copy link
Collaborator

That PLT thunk stuff is probably glibc only and probably not happening here according to the parts of the disassembly we've seen so far.

Most likely all that is required here is to get the disassembly of the code leading up to the vsyscall as well as after it, and use that to create a patch template that matches this call site.

@georgewsinger
Copy link
Author

@khuey

Screenshot_20210723-132908_JuiceSSH

@GitMensch

$ uname -a
Linux UbuntuBox 5.8.0-50-generic #56-Ubuntu SMP Mon Apr 12 17:18:36 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

@khuey
Copy link
Collaborator

khuey commented Jul 23, 2021

Can you keep going on the last one until you get back to 0x55b39d8a720b?

@georgewsinger
Copy link
Author

With

[FATAL /build/source/src/RecordSession.cc:347:handle_seccomp_traced_syscall()] 
 (task 588943 (rec:588943) at time 79573)
 -> Assertion `patch_ok' failed to hold. The tracee issues a vsyscall to 0xffffffffff600000 but we failed to moneypatch the caller (return address 0x55ba7e56520b, sp=0x7ffdfad84678). Recording will not succeed. Exiting.

I'm now getting

image

...which looks like mangled/unreadable output?

@khuey
Copy link
Collaborator

khuey commented Jul 24, 2021

Can you do x/100i for that?

@georgewsinger
Copy link
Author

I ran it again and got a similar result, but it seems to be more readable assembly if I use /i instead of /r?

image

@khuey
Copy link
Collaborator

khuey commented Jul 24, 2021

idk what /r does, idk why roc told you to use that :)

@khuey
Copy link
Collaborator

khuey commented Jul 24, 2021

Out of curiosity, what version of libc does this system have?

@georgewsinger
Copy link
Author

2.32:

$ ldd --version ldd
ldd (Ubuntu GLIBC 2.32-0ubuntu3) 2.32
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

@rocallahan
Copy link
Collaborator

Have you updated to 8a15f2a ?

@GitMensch
Copy link
Contributor

GitMensch commented Jul 24, 2021 via email

@georgewsinger
Copy link
Author

Have you updated to 8a15f2a ?

@rocallahan Just tried updating to latest tip, and the issue has gone away. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants