First function call takes significantly longer with OS_ENABLE_HW_BOUND_CHECK

[sjamesr]

One of our engineers is observing that the first call into WAMR takes significantly longer than subsequent calls when OS_ENABLE_HW_BOUND_CHECK is set.

invoke_native_with_hw_bound_check eventually calls into os_thread_get_stack_boundary, where we have

#ifdef __linux__
    if (pthread_getattr_np(self, &attr) == 0) {
        pthread_attr_getstack(&attr, (void **)&addr, &stack_size);
        pthread_attr_getguardsize(&attr, &guard_size);
        pthread_attr_destroy(&attr);
        if (stack_size > max_stack_size)
            addr = addr + stack_size - max_stack_size;
        addr += guard_size;
    }
    (void)stack_size;

pthread_getattr_np may be a culprit because (at least in glibc) it does a read of /proc/self/maps, scanf, etc. It has been identified as a potential culprit of slowdowns in other projects, e.g. golang/go#68587.

The environment is Android ARM. Would it be safe to disable OS_ENABLE_HW_BOUND_CHECK?

[wenyongh]

Hi, disabling OS_ENABLE_HW_BOUND_CHECK will disable the hardware boundary check and may impact the performance. But os_thread_get_stack_boundary is also called in aot_call_function when OS_ENABLE_HW_BOUND_CHECK is disabled:

wasm-micro-runtime/core/iwasm/aot/aot_runtime.c

Lines 2245 to 2248 in e8c2952

	#ifndef OS_ENABLE_HW_BOUND_CHECK
	/* Set thread handle and stack boundary */
	wasm_exec_env_set_thread_info(exec_env);
	#else

Do you think is it better to make the pthread_attr_t attr a thread local storage and get it previously? e.g., (1) in the main thread, runtime can get it in wasm_runtime_init/wasm_runtime_full_init (which calls bh_platform_init and bh_platform_init calls pthread_getattr_np), (2) in the child thread created, runtime gets pthread_attr_t attr in os_thread_wrapper. By this way, the first function call doesn't need to call pthread_getattr_np again, but it makes runtime init and thread creating slower.