[BOUNTY] Add Qwen1.5 0.5B

[JushBJJ]

This PR is with the help of @marty1885 and @JonathanALevine.
Bounty: #20

The following issues need to be addressed:

• [ x ] Reduce time complexity of node replacment in PyTorch frontend tt-tvm#1
• [ x ] Support triu function for tvm.relay.expr.Call Inputs tt-tvm#2
• [ x ] No proper support for aten::triu OP inputs when inputs are CallNodes [Bug]  tt-tvm#3
• [ x ] Decompose tril ops with dynamic diagonal argument tt-tvm#5
• [ x ] Allow masked_fill to handle non-boolean/hot tensor masks tt-tvm#4
• [ x ] DynamicCache for models is not supported tt-buda#42

---

Video of it working: https://www.youtube.com/watch?v=Tlb1ElBHqVk

Requirements

• Transformers >=4.42.0
• ~12 GB of RAM (Host)

[JushBJJ]

With all these patches, we internally have Qwen1.5 0.5B working on a Grayskull e150 (thanks @JonathanALevine). We're waiting for @marty1885 to test all of this in his e75. Should we convert this out of a draft PR?

@nvukobratTT @milank94

[JushBJJ]

A lot of optimizations will need to be made to speed this model up, stay tuned. I gotta do my uni assignments which are due tonight 😭

[nvukobratTT]

Great work @JushBJJ @marty1885 @JonathanALevine! Let us know once you get silicon results out of E75 :)))

We're working out some internal checking, as soon as we finalize those on our side we're going to complete code reviews and start with PR approvals.

Once again great progress and thanks for hunting these bounties! :)))

[marty1885]

I think there's some problems with the e75 - I ran inference for an hour but it never finished. Hope this doesn't affect the bounty as it works on the e150. It feels more like a low level compilation bug.

NVM. IT WORKS!! Just it's so slow that running it with the default settings takes forever. I can get it finish in a reasonable time if I set max_length to something short. Like 2.

[JushBJJ]

@nvukobratTT @milank94 Update :)

[milank94]

@JushBJJ way to go!

As we don't have automated CI setup yet for this project, we'll manually test the model.

Before that, there are a few miscellaneous components that should be added in such as update support table, add test case, add license header, etc. You can see a breakdown here: https://github.com/tenstorrent/tt-buda-demos/blob/main/CONTRIBUTING.md#adding-models

Mind adding these details in and we'll continue the review process?

[JushBJJ]

Sure I'll do it too with the Phi 2 bounty as well, let me know if you guys had any issue running it.
Also before manually testing it, add these workarounds:
JushBJJ/tt-buda@f765838

I raised an issue (tenstorrent/tt-buda#42) because I'm not quite sure how a proper implementation would look like for DynamicCaching so it would be great if anyone in the TT team familiar with it can make a proper fix :)

[milank94]

Sure I'll do it too with the Phi 2 bounty as well, let me know if you guys had any issue running it. Also before manually testing it, add these workarounds: JushBJJ/tt-buda@f765838

I raised an issue (tenstorrent/tt-buda#42) because I'm not quite sure how a proper implementation would look like for DynamicCaching so it would be great if anyone in the TT team familiar with it can make a proper fix :)

Thanks for the update. We'll get someone to review and integrate the work arounds. We can look to package this as an alpha release @staylorTT

FYI: @Shubhamsaboo for bounty tracking

[JushBJJ]

Made some changes:

• Cleaned up code and added batching like the other examples
• Renamed filename for qwen
• Adjusted parameters to reduce weird token generation
• Added chat version of qwen1.5
• Added test case for qwen1.5

I also tested on previous transformers versions, seems like v4.42.0 is some magic number??? Any version below that Qwen does not work at all.

[milank94]

Made some changes:

• Cleaned up code and added batching like the other examples
• Renamed filename for qwen
• Adjusted parameters to reduce weird token generation
• Added chat version of qwen1.5
• Added test case for qwen1.5

I also tested on previous transformers versions, seems like v4.42.0 is some magic number??? Any version below that Qwen does not work at all.

Thanks @JushBJJ . I will have a look.

That's good to note. The latest release of Buda is on transformers==4.41.0, so will need to uplift it in tt-buda or add it here as a temporary requirement.

[JushBJJ]

See tenstorrent/tt-buda#42 (comment)

I'll rewrite Qwen's file with a custom wrapper/generate forward function

[JushBJJ]

This should be good now for final review and testing.

See tenstorrent/tt-buda#42 (comment) for my comment on disabling DynamicCache on new models

[JushBJJ]

Any updates on this on TT's side? @milank94

[milank94]

Any updates on this on TT's side? @milank94

Hey @JushBJJ -- we're currently working on integrating all of necessary changes into a new Buda release. This should add the support for this model and the Phi-2, along with some additional features.

Once we have that RC ready, we'll do the test and merging of these models.

Thanks for the patience.

[milank94]

Hey @JushBJJ can you switch the target branch to be: mkordic/rc_20240830?

[JushBJJ]

Done

[milank94]

Looks great @JushBJJ - can you address the merge conflict around model_demos/pyproject.toml?

I merged the changes for #117, I think this branch just needs a rebase onto mkordic/rc_20240830.

The plan is to merge these two models into a RC branch (mkordic/rc_20240830), that we are using to updated the changes for the next TT-Buda release, then merge into main and complete the bounties.

[JushBJJ]

@milank94 Rebased it, i normally dont do it so I hope i did it right 😅