Dfuutil missing for arm64

[dolfandringa]

The ststm32 core for platformio defines a dependency on tool-dfuutil, but that package does not seem to be available for linux_aarch64 so installing the platform fails. I am not actually using dfu for uploading code (I am using the stlinkv2 programmer), so for me a workaround is to manually remove the dependency from ~/.platformio/platforms/ststm32/platform.json. For me that works, but it is definitely not an ideal workaround, and others might actually use dfu. So would it be possible to release an arm64 linux package for it?

[thinkyhead]

Also, we need a Darwin arm_64 version of tool-dfuutil for the M1 Mac system.
If a set of binaries is needed, maybe I can cobble them together.

[jhaand]

It would be nice if the latest dfu-util would be added to the tool-dfuutil package. At this moment only an old revision is used and the latest 0.11 only seems available for Macos Arm64.
The dfu-util project has released 0.11 and it's not integrated in platformio for other platforms.
As you can see here:
https://registry.platformio.org/tools/platformio/tool-dfuutil/compatibility

See also this issue.
sipeed/platform-gd32v#17
Although this points to a Sipeed archive, this tool is owned by Platformio and in the top 100 of used packages.

[jhaand]

Today a new version of dfu-util 0.11 was introduced in the registry. Unfortunately not for linux_aarch64.

[thinkyhead]

It's not helping me either. The chart on their page says arm64 is supported, but we're still not picking up a universal binary.

> pio --version
PlatformIO Core, version 6.0.2

> pio pkg install -g --tool "platformio/tool-dfuutil@^1.11.0"

> pio run -e Opulo_Lumen_REV3

dyld[91452]: Library not loaded: '/usr/local/opt/libusb/lib/libusb-1.0.0.dylib'
  Referenced from: '/u/t/.platformio/packages/tool-dfuutil/bin/dfu-suffix'
  Reason: tried: '/usr/local/opt/libusb/lib/libusb-1.0.0.dylib' (no such file), '/usr/local/lib/libusb-1.0.0.dylib' (mach-o file, but is an incompatible architecture (have (arm64), need (x86_64))), '/usr/lib/libusb-1.0.0.dylib' (no such file)

> find ~/.platformio -name "dfu-*" -type f
/u/t/.platformio/packages/tool-dfuutil/bin/dfu-util
/u/t/.platformio/packages/tool-dfuutil/bin/dfu-prefix
/u/t/.platformio/packages/tool-dfuutil/bin/dfu-suffix

> file ~/.platformio/packages/tool-dfuutil/bin/*
/u/t/.platformio/packages/tool-dfuutil/bin/dfu-prefix: Mach-O 64-bit executable x86_64
/u/t/.platformio/packages/tool-dfuutil/bin/dfu-suffix: Mach-O 64-bit executable x86_64
/u/t/.platformio/packages/tool-dfuutil/bin/dfu-util:   Mach-O 64-bit executable x86_64

[thijstriemstra]

Same on issue on Raspberry Pi 3b+:

$ pio --version
PlatformIO Core, version 6.1.1

$ pio pkg install -g --tool "platformio/tool-dfuutil@^1.11.0"
Tool Manager: Installing platformio/tool-dfuutil @ ^1.11.0
Error: Could not find the package with 'platformio/tool-dfuutil @ ^1.11.0' requirements for your system 'linux_armv7l'

$ uname -a
Linux mpcnc 5.10.103-v7+ #1529 SMP Tue Mar 8 12:21:37 GMT 2022 armv7l GNU/Linux

Unable to compile Marlin 2.1.x, sigh.

Update: Marlin fixed this literally 2 hrs ago and all seems to work again 🥳 when applying this patch: MarlinFirmware/Marlin@068624a

[loozhengyuan]

I am still encountering this issue for macOS ARM64. Based on the registry, it looks like darwin_arm64 is available the wrong binary is downloaded?

$ pio pkg uninstall -g --tool "platformio/tool-dfuutil@^1.11.0"         
Tool Manager: Removing tool-dfuutil @ 1.11.0
Tool Manager: [email protected] has been removed!
$ pio pkg install -g --tool "platformio/tool-dfuutil@^1.11.0"           
Tool Manager: Installing platformio/tool-dfuutil @ ^1.11.0
Unpacking  [####################################]  100%
Tool Manager: [email protected] has been installed!
$ file ~/.platformio/packages/tool-dfuutil/bin/*
~/.platformio/packages/tool-dfuutil/bin/dfu-prefix: Mach-O 64-bit executable x86_64
~/.platformio/packages/tool-dfuutil/bin/dfu-suffix: Mach-O 64-bit executable x86_64
~/.platformio/packages/tool-dfuutil/bin/dfu-util:   Mach-O 64-bit executable x86_64

Is it a case of wrong binary is fetched? Or the binary for darwin_arm64 does not exist? Or the wrong binary was uploaded to the registry?

[valeros]

Hi @loozhengyuan, could you run pio pkg update -g -t platformio/tool-dfuutil and try again?

[loozhengyuan]

@valeros Thanks for looking into this! 🙇

I was able to confirm that the arm64 binaries are installed now:

$ pio pkg update -g -t platformio/tool-dfuutil
Tool Manager: Updating tool-dfuutil @ 1.11.0
Tool Manager: Removing tool-dfuutil @ 1.11.0
Tool Manager: [email protected] has been removed!
Tool Manager: Installing platformio/tool-dfuutil @ 1.11.241024
Downloading  [####################################]  100%
Unpacking  [####################################]  100%
$ ~/.platformio/packages/tool-dfuutil/bin/*
~/.platformio/packages/tool-dfuutil/bin/dfu-prefix: Mach-O 64-bit executable arm64
~/.platformio/packages/tool-dfuutil/bin/dfu-suffix: Mach-O 64-bit executable arm64
~/.platformio/packages/tool-dfuutil/bin/dfu-util:   Mach-O 64-bit executable arm64

However, it does not seem to work correctly:

$ pio run --target upload
...
Adding dfu suffix to firmware.bin
dyld[29406]: Library not loaded: /tmp/dfu-util-0.11-Darwin-arm64/lib/libusb-1.0.0.dylib
  Referenced from: <98506EAE-D454-32F5-917E-8BC573B4FB07> ~/packages/tool-dfuutil/bin/dfu-suffix
  Reason: tried: '/tmp/dfu-util-0.11-Darwin-arm64/lib/libusb-1.0.0.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/tmp/dfu-util-0.11-Darwin-arm64/lib/libusb-1.0.0.dylib' (no such file), '/tmp/dfu-util-0.11-Darwin-arm64/lib/libusb-1.0.0.dylib' (no such file)

[valeros]

@thijstriemstra I've uploaded a new package, please run pio pkg update -g -t platformio/tool-dfuutil again.

[thinkyhead]

@loozhengyuan , @valeros —

At the moment, for many of our Marlin builds PlatformIO is downloading and running [email protected]. Is this because of the version spec in our environment? We're currently sticking to STSTM32 version 12.1 because a newer version would require us to update all our boards and variants.

[common_stm32]
platform = ststm32@~12.1

I added platform_packages for testing…

[[env:FYSETC_S6]
extends                     = stm32_variant
board                       = marlin_fysetc_s6
platform_packages           = [email protected]

…and I'm getting back a rather sad error message:

FileNotFoundError: [Errno 2] No such file or directory: 
.../.platformio/packages/[email protected]/variants/MARLIN_FYSETC_S6':

Someone please explain why PlatformIO is looking for a folder named [email protected]/variants/MARLIN_FYSETC_S6 – appending the path to the board variants folder to the path of the dfu tool ??

Anyway, since PlatformIO was insisting on being difficult, or using [email protected] I tried copying the contents of tool-dfuutil-1.11.0 into [email protected]. The results were the same, as it was unable to find libusb-1.0.0.dylib in various fixed paths (including /System/Volumes/Preboot/Cryptexes/OS/tmp which I'm sure we all have).

I then tried version 1.11.23411026 and got the same results. However, when I used the dfu-prefix, dfu-suffix, and dfu-util files installed by sudo port install dfu-util the build finally succeeded. I followed up with sudo port uninstall usblib to see if it would cause dfu-util to fail, and it did not cause any problems.

Here are the binaries that finally worked for me:

dfuutil-darwin-arm64.zip

Now, how can we get these reliable and non-weirdly-linked binaries to download when building Marlin on macOS?

[thinkyhead]

Note that proper macOS binaries will not only need proper linkage, but must also be Universal binaries (not just arm64), presuming that PlatformIO does not distinguish between Darwin on ARM versus Intel for these tools.

Universal binaries from sudo port install dfu-util +universal:

dfuutil-darwin-universal.zip

Edit: These do appear to rely on /opt/local/lib/libusb-1.0.0.dylib being present, after all. So ultimately we need a set of binaries that include /opt/local/lib/libusb-1.0.0.dylib in the dynamic linker search paths. The 1.11.241026 binaries do not seem to look here.

[thinkyhead]

That rather weird error message still deserves a look. But the download was at least made to work by adding platformio/ in front of the tool name.

platform_packages = platformio/tool-dfuutil@^1.11.241026

However, as previously mentioned, the October 26 binaries still produce the same dynamic linking error, so the updated Universal binaries are definitely needed, along with a new "1.11.241028" version.

dyld[16354]: Library not loaded: /tmp/dfu-util-0.11-Darwin-arm64/lib/libusb-1.0.0.dylib
  Referenced from: <98506EAE-D454-32F5-917E-8BC573B4FB07> /Users/thinkyhead/.platformio/packages/tool-dfuutil/bin/dfu-suffix
  Reason: tried: '/tmp/dfu-util-0.11-Darwin-arm64/lib/libusb-1.0.0.dylib' (no such file),
  '/System/Volumes/Preboot/Cryptexes/OS/tmp/dfu-util-0.11-Darwin-arm64/lib/libusb-1.0.0.dylib' (no such file),
  '/tmp/dfu-util-0.11-Darwin-arm64/lib/libusb-1.0.0.dylib' (no such file)
*** [.pio/build/FYSETC_S6/firmware.bin] Error -6

[valeros]

Hi @thinkyhead,

At the moment, for many of our Marlin builds PlatformIO is downloading and running [email protected]. Is this because of the version spec in our environment?

That's right, tool-dfuutil v1.9.211020 is the default choice for the ststm32 dev-platform v12.1.

Someone please explain why PlatformIO is looking for a folder named [email protected]/variants/MARLIN_FYSETC_S6 – appending the path to the board variants folder to the path of the dfu tool ??

Not sure where's this error coming from. Maybe you have a custom build command added via an extra script or something?

However, as previously mentioned, the October 26 binaries still produce the same dynamic linking error, so the updated Universal binaries are definitely needed

I've uploaded a new package for MacOS ARM64, so the missing libusb-1.0.0.dylib shouldn't be a problem now. You can pull the latest version via pio pkg update -g -t platformio/tool-dfuutil.

[thinkyhead]

You can pull the latest version via pio pkg update -g -t platformio/tool-dfuutil.

Thanks for the update! Of course, when we do a build with ststm32 12.1 we still get v1.9.211020. How can I ensure that we're pulling down the latest and greatest tool-dfuutil with all our STM32 builds going forward — in a way that is compatible with Linux, Windows, and macOS? I will be sure to modify all our legacy / LTS versions with the necessary changes as well.

[valeros]

How can I ensure that we're pulling down the latest and greatest tool-dfuutil with all our STM32 builds going forward — in a way that is compatible with Linux, Windows, and macOS?

Your previous attempt with platform_packages is the correct approach. Even though you're using a quite outdated version of the ststm32 dev-platform, the platform_packages option will force PlatformIO to download the latest dfu-util package v0.11 for each OS and architecture (if available).

Although, for your project, I'd recommend a bit stricter version range that will install only patch updates:

platform_packages = platformio/tool-dfuutil@~1.11.0

[thinkyhead]

@valeros — We have an intermediate proposal to assign a "latest" version and bump that as we gain confidence. I did a bunch of work to update the 40 custom boards and variants to fit the new platform version, but some APIs have also changed and so STM32 HAL code depending on those also need updates.

Anyway, I went ahead and added platform_packages = platformio/tool-dfuutil@~1.11.0 to the environments and 1.11.241029 seems to be working much better, thanks!