Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Switch to TRT-LLM LLM (High Level) API over trtllm-build CLI workflow #87

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

rmccorm4
Copy link
Collaborator

@rmccorm4 rmccorm4 commented Oct 25, 2024

Changes:

  • Switch to TRT-LLM LLM (High Level) API over trt-llm build workflow
    • Remove unnecessary helper scripts
  • Refer to framework model support sections in README
  • Add some better checks and error messages around --source arg
  • Add 'local:' source prefix for parity
  • Bump version to 0.1.0dev so it's easier to have 'patch' versions such as 0.1.1, 0.1.2, etc.
  • Disable genai-perf (triton profile) based tests due to bug in genai-perf r24.09:

Note:

  • Disable genai-perf (triton profile) based tests due to bug in genai-perf r24.09:
$ triton profile -m gpt2
triton - INFO - Running: 'genai-perf profile -m gpt2'
2024-10-25 12:40 [INFO] genai_perf.parser:83 - Profiling these models: gpt2
Traceback (most recent call last):
  File "/home/rmccormick/.local/lib/python3.10/site-packages/genai_perf/main.py", line 198, in run
    generate_inputs(config_options)
  File "/home/rmccormick/.local/lib/python3.10/site-packages/genai_perf/main.py", line 125, in generate_inputs
    inputs.create_inputs()
  File "/home/rmccormick/.local/lib/python3.10/site-packages/genai_perf/inputs/inputs.py", line 57, in create_inputs
    json_in_pa_format = self._convert_generic_json_to_output_format(
  File "/home/rmccormick/.local/lib/python3.10/site-packages/genai_perf/inputs/inputs.py", line 77, in _convert_generic_json_to_output_format
    return converter.convert(generic_dataset, self.config)
  File "/home/rmccormick/.local/lib/python3.10/site-packages/genai_perf/inputs/converters/tensorrtllm_converter.py", line 52, in convert
    pa_json = self._populate_trtllm_output_json(
  File "/home/rmccormick/.local/lib/python3.10/site-packages/genai_perf/inputs/converters/tensorrtllm_converter.py", line 96, in _populate_trtllm_output_json
    pa_json = self._add_optional_tags_to_trtllm_json(
  File "/home/rmccormick/.local/lib/python3.10/site-packages/genai_perf/inputs/converters/tensorrtllm_converter.py", line 125, in _add_optional_tags_to_trtllm_json
    if config.add_stream:
AttributeError: 'str' object has no attribute 'add_stream'

…el support sections in README, add some better checks and error messages around --source arg, add 'local:' source prefix for parity, bump version to 0.1.0dev so it's easier to have 'patch' versions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant