Fix ub and shrink unsafe blocks #541

Job	Run time
build bundled (macos-latest)	2m 18s
build linux pkg-config	1m 15s
build bundled (ubuntu-latest)	2m 6s
build bundled (windows-latest)	5m 49s
build bundled (--release, macos-latest)	1m 59s
build vcpkg (macos-latest)	3m 18s
build bundled (--release, ubuntu-latest)	2m 10s
build vcpkg (ubuntu-latest)	6m 36s
build bundled (--release, windows-latest)	7m 28s
build vcpkg (windows-latest)	9m 33s
build bundled (static-link, macos-latest)	3m 21s
build bundled (static-link, ubuntu-latest)	2m 9s
build bundled (static-link, windows-latest)	5m 12s
build bundled (static-link, --release, macos-latest)	2m 4s
build bundled (static-link, --release, ubuntu-latest)	2m 9s
build bundled (static-link, --release, windows-latest)	7m 8s
	1h 4m 35s

Provide feedback