bitsandbytes 0.39.0?

[ewof]

do u have plans for it or is it not possible rn

[evshiron]

The BitsAndBytes is built from https://github.com/agrocylo/bitsandbytes-rocm, which is 0.37.2 currently. It's like doing hipify and fix, but I personally don't have the knowledge nor the time to maintain an up-to-date version, so I have no plan for it at the moment.

[shermdog]

I dug into this a little - the hurdle may be porting the 4-bit update over to rocm. It's also over my head but it seems like rocm can support it.

[ewof]

i have most of it ported but the .hip and .hiph files generated by hipify-clang don't work there's a lot of manual editing needed but i think it's because i don't have hipBLASLt which i have been trying to compile for a bit (their install script doesnt support arch based distros)

[shermdog]

I've made some solid progress on porting over 0.39.0 to rocm. It currently compiles and will load models in 4-bit but generation returns gibberish. There's still two major things to port - the wmma matrix bits and fixing bfloat16

https://github.com/TimDettmers/bitsandbytes/compare/main...shermdog:rocm_039?expand=1

[evshiron]

@shermdog

I cloned your repo and tried locally. When running examples/int8_inference_huggingface.py with load_in_4bit=True, it always decoded into the same token, but load_in_8bit=True did worked.

Nice work here, and I am looking forward to your future updates!

[evshiron]

Here is another fork:

• https://github.com/arlo-phoenix/bitsandbytes-rocm-5.6

Which looks quite promising but I haven't tested it.

[ewof]

got load-in-4bit to work with this and textgen-webui by setting the kQuantizeBlockwise to how it is in the main repo