[#43] add "query_image" service

[jekalmin]

Objective

• Add "query_image" service to upload an image to gpt
• Although it's not ideal, let gpt know via function response content returned from service call (extended_openai_conversation.query_image).

Service

Example

• I tested with following image searched from website

  

Function

- spec:
    name: get_refrigerator_items
    description: Get description of items in refrigerator
    parameters:
      type: object
      properties:
        url:
          type: string
          description: image url of refrigerator
          enum:
            - https://i.pinimg.com/originals/8b/cc/f1/8bccf14daf77ce887fc162934335cb21.jpg # needs to change
      required:
      - url
  function:
    type: composite
    sequence:
      - type: script
        sequence:
          - service: extended_openai_conversation.query_image
            data:
              prompt: What alcohol and brands do you see in this picture?
              images:
                - url: "{{url}}"
              max_tokens: 300
              config_entry: YOUR_CONFIG_ENTRY_KEY # needs to change
            response_variable: _function_result
        response_variable: image_result
      - type: template
        value_template: "{{image_result.choices[0].message.content}}"

[TBTHEDBA]

How did I add this service?

[jekalmin]

How did I add this service?

I haven't released it yet. Let me release this in beta.

[jekalmin]

@TBTHEDBA
I just released this in 1.0.1-beta2.
Please try this and give a feedback :D

[TBTHEDBA]

I received this error in debugging. Should I be using this as my "your_config_entry_key". I got it from hacs.repositories "700780425": { "repository_manifest": { 2024-01-17 09:53:53.608 ERROR (MainThread) [custom_components.extended_openai_conversation.helpers] extended_openai_conversation: Error executing script. Invalid data for call_service at pos 1: expected str for dictionary value @ data['config_entry'] 2024-01-17 09:53:53.609 ERROR (MainThread) [homeassistant.components.assist_pipeline.pipeline] Unexpected error during intent recognition Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/assist_pipeline/pipeline.py", line 988, in recognize_intent conversation_result = await conversation.async_converse( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

…

On Wed, Jan 17, 2024 at 8:22 AM Min Jekal ***@***.***> wrote: @TBTHEDBA <https://github.com/TBTHEDBA> I just released this in 1.0.1-beta2. Please try this and give a feedback :D — Reply to this email directly, view it on GitHub <#60 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJXIGPX3CHV3TUJO45QCUOTYO7NBHAVCNFSM6AAAAABBBT4U4OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJVHEYTKMBUGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jekalmin]

Try to call service from developers tools.
You can get entry key from there.

ui mode	yaml mode

[TBTHEDBA]

[image: image.png] That did it! Thank you so much for your help. Now I get to shame my children for not cleaning their room.

…

On Wed, Jan 17, 2024 at 10:28 AM Min Jekal ***@***.***> wrote: Try to call service from developers tools. You can get entry key from there. Screenshot_20240118_012737_Home.Assistant.jpg (view on web) <https://github.com/jekalmin/extended_openai_conversation/assets/2917984/bc5de19d-44b9-42be-9411-33d550436bcd> — Reply to this email directly, view it on GitHub <#60 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJXIGPUGWDWFYKZAVOFR4J3YO733HAVCNFSM6AAAAABBBT4U4OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJWGE3DKNBSHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>