Added Ctranslate2 #1253

aryan1165 · 2024-09-28T06:41:04Z

I have tried to build ctranslate2. As mentioned in #1239 i have to create seperate recipe for c++ build of it which is named as ctranslate2-lib and for python part it is a pre-requisite.

Fixes We need Ctranslate2 plz #1239

aryan1165 · 2024-09-28T07:13:22Z

In second commit, loading c++ library locally instead of git repo, to modify CMakeLists.txt, so that it could find libiomp5.so, As i had to build it locally, It wasn't present for my system.

mhsmith · 2024-10-20T14:16:49Z

Originally posted by @aryan1165 in #1239 (comment)

Hi @mhsmith, The wheels are building for ctranslate2-lib and ctranslate2-python seperately, but when i unzip them they only have dist-info. I dont know where i went wrong? I know it must be a stupid thing, but as i am new to all this. Please consider having a look at the PR and guide me where i am going wrong?

Hi @mhsmith, i rebuilt the c++ part, and the logs are showing that it is building successfully, but the .whl doesn't have the actual package inside it. Also i am seeing that ctranslate2-lib directory is being created at the root of my machine.

Logs -

[ 97%] Built target ctranslate2
[ 98%] Building CXX object cli/CMakeFiles/translator.dir/translator.cc.o
[100%] Linking CXX executable ct2-translator
[100%] Built target translator
[sudo] password for aryan: 
[ 97%] Built target ctranslate2
[100%] Built target translator
Install the project...
-- Install configuration: "Release"
-- Installing: /usr/local/lib/libctranslate2.so
-- Up-to-date: /usr/local/include
-- Up-to-date: /usr/local/include/nlohmann
-- Up-to-date: /usr/local/include/nlohmann/json.hpp
-- Up-to-date: /usr/local/include/ctranslate2
-- Up-to-date: /usr/local/include/ctranslate2/primitives.h
-- Up-to-date: /usr/local/include/ctranslate2/sampling.h
-- Up-to-date: /usr/local/include/ctranslate2/storage_view.h
-- Up-to-date: /usr/local/include/ctranslate2/generation.h
-- Up-to-date: /usr/local/include/ctranslate2/logging.h
-- Up-to-date: /usr/local/include/ctranslate2/decoding_utils.h
-- Up-to-date: /usr/local/include/ctranslate2/filesystem.h
-- Up-to-date: /usr/local/include/ctranslate2/batch_reader.h
-- Up-to-date: /usr/local/include/ctranslate2/encoder.h
-- Up-to-date: /usr/local/include/ctranslate2/devices.h
-- Up-to-date: /usr/local/include/ctranslate2/random.h
-- Up-to-date: /usr/local/include/ctranslate2/ops
-- Up-to-date: /usr/local/include/ctranslate2/ops/tile.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/softmax.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/rotary.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/bias_add.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/conv1d.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/topk.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/nccl_ops.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/min_max.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/multinomial.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/sub.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/ops.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/mean.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/gemm.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/concat.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/transpose.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/log.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/layer_norm.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/identity.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/flash-attention
-- Up-to-date: /usr/local/include/ctranslate2/ops/flash-attention/softmax.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/flash-attention/rotary.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/flash-attention/flash_fwd_launch_template.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/flash-attention/mask.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/flash-attention/flash.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/flash-attention/kernel_traits.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/flash-attention/utils.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/flash-attention/static_switch.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/flash-attention/alibi.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/flash-attention/flash_fwd_kernel.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/flash-attention/block_info.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/awq
-- Up-to-date: /usr/local/include/ctranslate2/ops/awq/gemm.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/awq/gemv.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/awq/dequantize_awq.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/split.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/sin.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/dequantize.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/tanh.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/gumbel_max.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/sigmoid.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/quantize.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/matmul.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/sum.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/activation.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/topp_mask.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/rms_norm.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/alibi_add.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/cos.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/flash_attention.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/swish.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/gelu.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/gather.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/relu.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/add.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/unsqueeze.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/op.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/squeeze.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/mul.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/median_filter.h
-- Up-to-date: /usr/local/include/ctranslate2/ops/slide.h
-- Up-to-date: /usr/local/include/ctranslate2/scoring.h
-- Up-to-date: /usr/local/include/ctranslate2/vocabulary_map.h
-- Up-to-date: /usr/local/include/ctranslate2/bfloat16.h
-- Up-to-date: /usr/local/include/ctranslate2/replica_pool.h
-- Up-to-date: /usr/local/include/ctranslate2/profiler.h
-- Up-to-date: /usr/local/include/ctranslate2/translation.h
-- Up-to-date: /usr/local/include/ctranslate2/utils.h
-- Up-to-date: /usr/local/include/ctranslate2/thread_pool.h
-- Up-to-date: /usr/local/include/ctranslate2/buffered_translation_wrapper.h
-- Up-to-date: /usr/local/include/ctranslate2/models
-- Up-to-date: /usr/local/include/ctranslate2/models/whisper.h
-- Up-to-date: /usr/local/include/ctranslate2/models/wav2vec2.h
-- Up-to-date: /usr/local/include/ctranslate2/models/model_factory.h
-- Up-to-date: /usr/local/include/ctranslate2/models/sequence_to_sequence.h
-- Up-to-date: /usr/local/include/ctranslate2/models/model_reader.h
-- Up-to-date: /usr/local/include/ctranslate2/models/wav2vec2bert.h
-- Up-to-date: /usr/local/include/ctranslate2/models/language_model.h
-- Up-to-date: /usr/local/include/ctranslate2/models/transformer.h
-- Up-to-date: /usr/local/include/ctranslate2/models/model.h
-- Up-to-date: /usr/local/include/ctranslate2/vocabulary.h
-- Up-to-date: /usr/local/include/ctranslate2/decoding.h
-- Up-to-date: /usr/local/include/ctranslate2/allocator.h
-- Up-to-date: /usr/local/include/ctranslate2/encoding.h
-- Up-to-date: /usr/local/include/ctranslate2/translator.h
-- Up-to-date: /usr/local/include/ctranslate2/types.h
-- Up-to-date: /usr/local/include/ctranslate2/generator.h
-- Up-to-date: /usr/local/include/ctranslate2/padder.h
-- Up-to-date: /usr/local/include/ctranslate2/layers
-- Up-to-date: /usr/local/include/ctranslate2/layers/whisper.h
-- Up-to-date: /usr/local/include/ctranslate2/layers/wav2vec2.h
-- Up-to-date: /usr/local/include/ctranslate2/layers/encoder.h
-- Up-to-date: /usr/local/include/ctranslate2/layers/attention_layer.h
-- Up-to-date: /usr/local/include/ctranslate2/layers/attention.h
-- Up-to-date: /usr/local/include/ctranslate2/layers/common.h
-- Up-to-date: /usr/local/include/ctranslate2/layers/layers.h
-- Up-to-date: /usr/local/include/ctranslate2/layers/flash_attention.h
-- Up-to-date: /usr/local/include/ctranslate2/layers/wav2vec2bert.h
-- Up-to-date: /usr/local/include/ctranslate2/layers/transformer.h
-- Up-to-date: /usr/local/include/ctranslate2/layers/decoder.h
-- Up-to-date: /usr/local/include/half_float
-- Up-to-date: /usr/local/include/half_float/half.hpp
-- Up-to-date: /usr/local/lib/cmake/ctranslate2/ctranslate2Targets.cmake
-- Installing: /usr/local/lib/cmake/ctranslate2/ctranslate2Targets-release.cmake
-- Up-to-date: /usr/local/lib/cmake/ctranslate2/ctranslate2Config.cmake
-- Up-to-date: /usr/local/lib/cmake/ctranslate2/FindNCCL.cmake
-- Up-to-date: /usr/local/lib/cmake/ctranslate2/ctranslate2ConfigVersion.cmake
-- Installing: /usr/local/bin/ct2-translator
build-wheel: mkdir -p /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/prefix/ctranslate2_lib-4.1.0.dist-info
build-wheel: wheel pack /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/prefix --dest-dir /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/src --build-number 0
Repacking wheel as /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/src/ctranslate2_lib-4.1.0-0-py3-none-android_21_arm64_v8a.whl...OK
build-wheel: mkdir -p /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/fix_wheel
build-wheel: unzip -d /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/fix_wheel -q /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/src/ctranslate2_lib-4.1.0-0-py3-none-android_21_arm64_v8a.whl
build-wheel: cp -a /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/src/LICENSE /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/fix_wheel/ctranslate2_lib-4.1.0.dist-info
build-wheel: Processing native binaries
build-wheel: wheel pack /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/fix_wheel --dest-dir /home/aryan/aryan/chaquopy/server/pypi/dist/ctranslate2-lib --build-number 0
Repacking wheel as /home/aryan/aryan/chaquopy/server/pypi/dist/ctranslate2-lib/ctranslate2_lib-4.1.0-0-py3-none-android_21_arm64_v8a.whl...OK```

Thanks.

----

Hi, I was able to resolve it by giving -DCMAKE_INSTALL_PREFIX = /home/user_name/chaqoupy/server/pip/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/prefix/

Now wheel is of 12.6MB and has everything related to this package.

Please can i know how to build the python part. Currently, This is my meta.yaml

```package:
  name: "ctranslate2-python"
  version: "4.1.0"

source:
  path: /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/CTranslate2/python

build:
  number: 0
  

requirements:
  build:
    - pybind11 2.11.1
  host:
    - ctranslate2-lib 4.1.0 # Add dependency on the C++ library
    - python

And build.sh

python setup.py bdist_wheel
pip install dist/*.whl

Error -

(build-3.9) aryan@NPCI-Model-deploy:~/aryan/chaquopy/server/pypi$ ./build-wheel.py --python 3.9 --abi arm64-v8a ctranslate2-python
build-wheel: mkdir -p /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0
build-wheel: cd /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0
build-wheel: mkdir -p /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/py3-none-android_21_arm64_v8a
build-wheel: cp -a /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/CTranslate2/python /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/py3-none-android_21_arm64_v8a/src
build-wheel: mkdir -p /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/py3-none-android_21_arm64_v8a/requirements
build-wheel: mkdir -p /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/py3-none-android_21_arm64_v8a/requirements/chaquopy/include
build-wheel: mkdir -p /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/py3-none-android_21_arm64_v8a/requirements/chaquopy/lib
build-wheel: unzip -d /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/py3-none-android_21_arm64_v8a/requirements -q /home/aryan/aryan/chaquopy/server/pypi/dist/ctranslate2-lib/ctranslate2_lib-4.1.0-0-py3-none-android_21_arm64_v8a.whl
build-wheel: /home/aryan/aryan/chaquopy/server/pypi/build/_bootstrap/3.9/bin/pip --version
build-wheel: mkdir -p /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/py3-none-android_21_arm64_v8a/env
build-wheel: python3.9 -m venv --without-pip /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/py3-none-android_21_arm64_v8a/env
build-wheel: /home/aryan/aryan/chaquopy/server/pypi/build/_bootstrap/3.9/bin/pip --python /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/py3-none-android_21_arm64_v8a/env/bin/python install pybind11==2.11.1
Collecting pybind11==2.11.1
  Obtaining dependency information for pybind11==2.11.1 from https://files.pythonhosted.org/packages/06/55/9f73c32dda93fa4f539fafa268f9504e83c489f460c380371d94296126cd/pybind11-2.11.1-py3-none-any.whl.metadata
  Using cached pybind11-2.11.1-py3-none-any.whl.metadata (9.5 kB)
Using cached pybind11-2.11.1-py3-none-any.whl (227 kB)
Installing collected packages: pybind11
Successfully installed pybind11-2.11.1
build-wheel: abi=arm64-v8a; api_level=21; prefix=/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/py3-none-android_21_arm64_v8a/requirements/chaquopy; . /home/aryan/aryan/chaquopy/server/pypi/../../target/build-common.sh; export
build-wheel: mkdir -p /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/py3-none-android_21_arm64_v8a/wrappers
build-wheel: /home/aryan/android-sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ar rc /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/py3-none-android_21_arm64_v8a/requirements/chaquopy/lib/libpthread.a
build-wheel: /home/aryan/android-sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ar rc /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/py3-none-android_21_arm64_v8a/requirements/chaquopy/lib/librt.a
build-wheel: cd /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/py3-none-android_21_arm64_v8a/src
build-wheel: mkdir -p /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/py3-none-android_21_arm64_v8a/prefix
build-wheel: mkdir -p /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/py3-none-android_21_arm64_v8a/prefix/chaquopy
build-wheel: /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build.sh
Requirement already satisfied: pybind11==2.11.1 in /home/aryan/anaconda3/envs/build-3.9/lib/python3.9/site-packages (from -r install_requirements.txt (line 1)) (2.11.1)
Requirement already satisfied: setuptools in /home/aryan/anaconda3/envs/build-3.9/lib/python3.9/site-packages (from -r install_requirements.txt (line 2)) (69.1.1)
Requirement already satisfied: wheel in /home/aryan/anaconda3/envs/build-3.9/lib/python3.9/site-packages (from -r install_requirements.txt (line 3)) (0.33.6)
Requirement already satisfied: setuptools in /home/aryan/anaconda3/envs/build-3.9/lib/python3.9/site-packages (69.1.1)
Traceback (most recent call last):
  File "/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/py3-none-android_21_arm64_v8a/src/setup.py", line 8, in <module>
    from setuptools import Extension, find_packages, setup
ModuleNotFoundError: No module named 'setuptools'
ERROR: ctranslate2-4.4.0-cp311-cp311-linux_x86_64.whl is not a supported wheel on this platform.
build-wheel: Error: Command '['/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build.sh']' returned non-zero exit status 1.

Hi @mhsmith , After resolving some issues, I am getting stuck here, I think issue is with the pybind11 as for python build ctranslate2 uses it.

Log -

build-wheel: cd /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0
build-wheel: rm -rf /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a
build-wheel: mkdir -p /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a
build-wheel: cp -a /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/CTranslate2/python /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/src
build-wheel: mkdir -p /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/requirements
build-wheel: mkdir -p /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/requirements/chaquopy/include
build-wheel: mkdir -p /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/requirements/chaquopy/lib
build-wheel: unzip -q -d /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/requirements/chaquopy /home/aryan/aryan/chaquopy/maven/com/chaquo/python/target/3.8.18-0/target-3.8.18-0-arm64-v8a.zip include/* jniLibs/*
build-wheel: mv /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/requirements/chaquopy/jniLibs/arm64-v8a/* /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/requirements/chaquopy/lib
build-wheel: rm -r /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/requirements/chaquopy/jniLibs
build-wheel: unzip -d /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/requirements -q /home/aryan/aryan/chaquopy/server/pypi/dist/ctranslate2-lib/ctranslate2_lib-4.1.0-0-py3-none-android_21_arm64_v8a.whl
build-wheel: ln -s libcrypto_chaquopy.so /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/requirements/chaquopy/lib/libcrypto.so
build-wheel: ln -s libpython3.8.so /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/requirements/chaquopy/lib/libpython3..so
build-wheel: ln -s libssl_chaquopy.so /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/requirements/chaquopy/lib/libssl.so
build-wheel: ln -s libsqlite3_chaquopy.so /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/requirements/chaquopy/lib/libsqlite3.so
build-wheel: /home/aryan/aryan/chaquopy/server/pypi/build/_bootstrap/3.8/bin/pip --version
build-wheel: mkdir -p /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env
build-wheel: python3.8 -m venv --without-pip /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env
build-wheel: /home/aryan/aryan/chaquopy/server/pypi/build/_bootstrap/3.8/bin/pip --python /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/bin/python install pybind11==2.11.1 wheel 'setuptools >= 40.8.0'
Collecting pybind11==2.11.1
  Obtaining dependency information for pybind11==2.11.1 from https://files.pythonhosted.org/packages/06/55/9f73c32dda93fa4f539fafa268f9504e83c489f460c380371d94296126cd/pybind11-2.11.1-py3-none-any.whl.metadata
  Using cached pybind11-2.11.1-py3-none-any.whl.metadata (9.5 kB)
Collecting wheel
  Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/1b/d1/9babe2ccaecff775992753d8686970b1e2755d21c8a63be73aba7a4e7d77/wheel-0.44.0-py3-none-any.whl.metadata
  Using cached wheel-0.44.0-py3-none-any.whl.metadata (2.3 kB)
Collecting setuptools>=40.8.0
  Obtaining dependency information for setuptools>=40.8.0 from https://files.pythonhosted.org/packages/ff/ae/f19306b5a221f6a436d8f2238d5b80925004093fa3edea59835b514d9057/setuptools-75.1.0-py3-none-any.whl.metadata
  Using cached setuptools-75.1.0-py3-none-any.whl.metadata (6.9 kB)
Using cached pybind11-2.11.1-py3-none-any.whl (227 kB)
Using cached wheel-0.44.0-py3-none-any.whl (67 kB)
Using cached setuptools-75.1.0-py3-none-any.whl (1.2 MB)
Installing collected packages: wheel, setuptools, pybind11
Successfully installed pybind11-2.11.1 setuptools-75.1.0 wheel-0.44.0
build-wheel: abi=arm64-v8a; api_level=21; prefix=/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/requirements/chaquopy; . /home/aryan/aryan/chaquopy/server/pypi/../../target/build-common.sh; export
build-wheel: mkdir -p /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/wrappers
running egg_info
writing ctranslate2.egg-info/PKG-INFO
writing dependency_links to ctranslate2.egg-info/dependency_links.txt
writing entry points to ctranslate2.egg-info/entry_points.txt
writing requirements to ctranslate2.egg-info/requires.txt
writing top-level names to ctranslate2.egg-info/top_level.txt
reading manifest file 'ctranslate2.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'ctranslate2.egg-info/SOURCES.txt'
build-wheel: abi=arm64-v8a; api_level=21; prefix=/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/requirements/chaquopy; . /home/aryan/aryan/chaquopy/server/pypi/../../target/build-common.sh; export
build-wheel: /home/aryan/android-sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ar rc /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/requirements/chaquopy/lib/libpthread.a
build-wheel: /home/aryan/android-sdk/ndk/22.1.7171670/toolchains/llvm/prebuilt/linux-x86_64/bin/llvm-ar rc /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/requirements/chaquopy/lib/librt.a
build-wheel: cd /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/src
build-wheel: mkdir -p /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/prefix
build-wheel: mkdir -p /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/prefix/chaquopy
build-wheel: . /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/bin/activate; /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build.sh
Installing Python dependencies from install_requirements.txt...
/home/aryan/anaconda3/envs/build-3.8/lib/python3.8/site-packages/_distutils_hack/__init__.py:26: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Requirement already satisfied: pybind11==2.11.1 in /home/aryan/anaconda3/envs/build-3.8/lib/python3.8/site-packages (from -r install_requirements.txt (line 1)) (2.11.1)
Requirement already satisfied: setuptools in /home/aryan/anaconda3/envs/build-3.8/lib/python3.8/site-packages (from -r install_requirements.txt (line 2)) (69.1.1)
Requirement already satisfied: wheel in /home/aryan/anaconda3/envs/build-3.8/lib/python3.8/site-packages (from -r install_requirements.txt (line 3)) (0.33.6)
Using CTranslate2 root: /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/prefix
Building the Python wheel...
running bdist_wheel
running build
running build_py
creating build/lib.linux_aarch64-cpython-38/ctranslate2
copying ctranslate2/extensions.py -> build/lib.linux_aarch64-cpython-38/ctranslate2
copying ctranslate2/logging.py -> build/lib.linux_aarch64-cpython-38/ctranslate2
copying ctranslate2/__init__.py -> build/lib.linux_aarch64-cpython-38/ctranslate2
copying ctranslate2/version.py -> build/lib.linux_aarch64-cpython-38/ctranslate2
creating build/lib.linux_aarch64-cpython-38/ctranslate2/converters
copying ctranslate2/converters/converter.py -> build/lib.linux_aarch64-cpython-38/ctranslate2/converters
copying ctranslate2/converters/utils.py -> build/lib.linux_aarch64-cpython-38/ctranslate2/converters
copying ctranslate2/converters/openai_gpt2.py -> build/lib.linux_aarch64-cpython-38/ctranslate2/converters
copying ctranslate2/converters/opus_mt.py -> build/lib.linux_aarch64-cpython-38/ctranslate2/converters
copying ctranslate2/converters/__init__.py -> build/lib.linux_aarch64-cpython-38/ctranslate2/converters
copying ctranslate2/converters/fairseq.py -> build/lib.linux_aarch64-cpython-38/ctranslate2/converters
copying ctranslate2/converters/opennmt_py.py -> build/lib.linux_aarch64-cpython-38/ctranslate2/converters
copying ctranslate2/converters/transformers.py -> build/lib.linux_aarch64-cpython-38/ctranslate2/converters
copying ctranslate2/converters/opennmt_tf.py -> build/lib.linux_aarch64-cpython-38/ctranslate2/converters
copying ctranslate2/converters/marian.py -> build/lib.linux_aarch64-cpython-38/ctranslate2/converters
creating build/lib.linux_aarch64-cpython-38/ctranslate2/specs
copying ctranslate2/specs/whisper_spec.py -> build/lib.linux_aarch64-cpython-38/ctranslate2/specs
copying ctranslate2/specs/model_spec.py -> build/lib.linux_aarch64-cpython-38/ctranslate2/specs
copying ctranslate2/specs/attention_spec.py -> build/lib.linux_aarch64-cpython-38/ctranslate2/specs
copying ctranslate2/specs/wav2vec2_spec.py -> build/lib.linux_aarch64-cpython-38/ctranslate2/specs
copying ctranslate2/specs/common_spec.py -> build/lib.linux_aarch64-cpython-38/ctranslate2/specs
copying ctranslate2/specs/__init__.py -> build/lib.linux_aarch64-cpython-38/ctranslate2/specs
copying ctranslate2/specs/wav2vec2bert_spec.py -> build/lib.linux_aarch64-cpython-38/ctranslate2/specs
copying ctranslate2/specs/transformer_spec.py -> build/lib.linux_aarch64-cpython-38/ctranslate2/specs
creating build/lib.linux_aarch64-cpython-38/ctranslate2/models
copying ctranslate2/models/__init__.py -> build/lib.linux_aarch64-cpython-38/ctranslate2/models
running build_ext
building 'ctranslate2._ext' extension
creating build/temp.linux_aarch64-cpython-38/cpp
/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/wrappers/aarch64-linux-android21-clang++ -fPIC -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/prefix/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/include -I/home/aryan/anaconda3/envs/build-3.8/include/python3.8 -c cpp/encoder.cc -o build/temp.linux_aarch64-cpython-38/cpp/encoder.o -std=c++17 -fvisibility=hidden
/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/wrappers/aarch64-linux-android21-clang++ -fPIC -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/prefix/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/include -I/home/aryan/anaconda3/envs/build-3.8/include/python3.8 -c cpp/execution_stats.cc -o build/temp.linux_aarch64-cpython-38/cpp/execution_stats.o -std=c++17 -fvisibility=hidden
/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/wrappers/aarch64-linux-android21-clang++ -fPIC -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/prefix/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/include -I/home/aryan/anaconda3/envs/build-3.8/include/python3.8 -c cpp/generation_result.cc -o build/temp.linux_aarch64-cpython-38/cpp/generation_result.o -std=c++17 -fvisibility=hidden
/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/wrappers/aarch64-linux-android21-clang++ -fPIC -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/prefix/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/include -I/home/aryan/anaconda3/envs/build-3.8/include/python3.8 -c cpp/generator.cc -o build/temp.linux_aarch64-cpython-38/cpp/generator.o -std=c++17 -fvisibility=hidden
Chaquopy: ignored invalid -I directory: '/home/aryan/anaconda3/envs/build-3.8/include/python3.8'
In file included from cpp/execution_stats.cc:1:
In file included from cpp/module.h:3:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/pybind11.h:13:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/class.h:12:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/../attr.h:13:
/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/common.h:266:10: fatal error: 'Python.h' file not found
#include <Python.h>
         ^~~~~~~~~~
Chaquopy: ignored invalid -I directory: '/home/aryan/anaconda3/envs/build-3.8/include/python3.8'
Chaquopy: ignored invalid -I directory: '/home/aryan/anaconda3/envs/build-3.8/include/python3.8'
In file included from cpp/generation_result.cc:1:
In file included from cpp/module.h:3:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/pybind11.h:13:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/class.h:12:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/../attr.h:13:
/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/common.h:266:10: fatal error: 'Python.h' file not found
#include <Python.h>
         ^~~~~~~~~~
In file included from cpp/generator.cc:1:
In file included from cpp/module.h:3:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/pybind11.h:13:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/class.h:12:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/../attr.h:13:
/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/common.h:266:10: fatal error: 'Python.h' file not found
#include <Python.h>
         ^~~~~~~~~~
Chaquopy: ignored invalid -I directory: '/home/aryan/anaconda3/envs/build-3.8/include/python3.8'
In file included from cpp/encoder.cc:1:
In file included from cpp/module.h:3:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/pybind11.h:13:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/class.h:12:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/../attr.h:13:
/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/common.h:266:10: fatal error: 'Python.h' file not found
#include <Python.h>
         ^~~~~~~~~~
1 error generated.
/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/wrappers/aarch64-linux-android21-clang++ -fPIC -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/prefix/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/include -I/home/aryan/anaconda3/envs/build-3.8/include/python3.8 -c cpp/logging.cc -o build/temp.linux_aarch64-cpython-38/cpp/logging.o -std=c++17 -fvisibility=hidden
error: command '/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/wrappers/aarch64-linux-android21-clang++' failed with exit code 1
build-wheel: Error: Command '. /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/bin/activate; /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build.sh' returned non-zero exit status 1.
1 error generated.
(build-3.8) aryan@NPCI-Model-deploy:~/aryan/chaquopy/server/pypi$ Chaquopy: ignored invalid -I directory: '/home/aryan/anaconda3/envs/build-3.8/include/python3.8'
In file included from cpp/logging.cc:1:
In file included from cpp/module.h:3:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/pybind11.h:13:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/class.h:12:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/../attr.h:13:
/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/common.h:266:10: fatal error: 'Python.h' file not found
#include <Python.h>
         ^~~~~~~~~~
1 error generated.
1 error generated.
(build-3.8) aryan@NPCI-Model-deploy:~/aryan/chaquopy/server/pypi$ 1 error generated.

Link for build instructions - https://opennmt.net/CTranslate2/installation.html

Also my c++ lib is getting build at prefix directory so my CTRANSLATE2_ROOT=/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/prefix

Screenshot.2024-10-13.at.5.46.44.PM.png (view on web)
Here c++ is built. Please help me with this. Thanks

Hi @mhsmith , Can you please look into this, Python.h header file is there located in my anaconda envs.

Hi @mhsmith , I also tried to build the wheel without conda and using system's python 3.8, Still same error Cant find Python.h. It is because chaquopy is printing this Chaquopy: ignored invalid -I directory: '/usr/include/python3.8'

hi @mhsmith , After including manual path to valid_dir in compiler_wrapper.py , I am getting this error

creating build/temp.linux_aarch64-cpython-38/cpp
/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/wrappers/aarch64-linux-android21-clang++ -fPIC -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/prefix/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/include -I/usr/include/python3.8 -c cpp/generator.cc -o build/temp.linux_aarch64-cpython-38/cpp/generator.o -std=c++17 -fvisibility=hidden
/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/wrappers/aarch64-linux-android21-clang++ -fPIC -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/prefix/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/include -I/usr/include/python3.8 -c cpp/generation_result.cc -o build/temp.linux_aarch64-cpython-38/cpp/generation_result.o -std=c++17 -fvisibility=hidden
/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/wrappers/aarch64-linux-android21-clang++ -fPIC -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/prefix/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/include -I/usr/include/python3.8 -c cpp/execution_stats.cc -o build/temp.linux_aarch64-cpython-38/cpp/execution_stats.o -std=c++17 -fvisibility=hidden
/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/wrappers/aarch64-linux-android21-clang++ -fPIC -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/prefix/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/include -I/usr/include/python3.8 -c cpp/encoder.cc -o build/temp.linux_aarch64-cpython-38/cpp/encoder.o -std=c++17 -fvisibility=hidden
In file included from cpp/generator.cc:1:
In file included from cpp/module.h:3:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/pybind11.h:13:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/class.h:12:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/../attr.h:13:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/common.h:266:
In file included from /usr/include/python3.8/Python.h:8:
/usr/include/python3.8/pyconfig.h:9:12: fatal error: 'aarch64-linux-gnu/python3.8/pyconfig.h' file not found
#  include <aarch64-linux-gnu/python3.8/pyconfig.h>
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from cpp/generation_result.cc:1:
In file included from cpp/module.h:3:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/pybind11.h:13:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/class.h:12:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/../attr.h:13:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/common.h:266:
In file included from /usr/include/python3.8/Python.h:8:
/usr/include/python3.8/pyconfig.h:9:12: fatal error: 'aarch64-linux-gnu/python3.8/pyconfig.h' file not found
#  include <aarch64-linux-gnu/python3.8/pyconfig.h>
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from cpp/encoder.cc:1:
In file included from cpp/module.h:3:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/pybind11.h:13:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/class.h:12:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/../attr.h:13:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/common.h:266:
In file included from /usr/include/python3.8/Python.h:8:
/usr/include/python3.8/pyconfig.h:9:12: fatal error: 'aarch64-linux-gnu/python3.8/pyconfig.h' file not found
#  include <aarch64-linux-gnu/python3.8/pyconfig.h>
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from cpp/execution_stats.cc:1:
In file included from cpp/module.h:3:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/pybind11.h:13:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/class.h:12:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/../attr.h:13:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/common.h:266:
In file included from /usr/include/python3.8/Python.h:8:
/usr/include/python3.8/pyconfig.h:9:12: fatal error: 'aarch64-linux-gnu/python3.8/pyconfig.h' file not found
#  include <aarch64-linux-gnu/python3.8/pyconfig.h>
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/wrappers/aarch64-linux-android21-clang++ -fPIC -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-lib/build/4.1.0/py3-none-android_21_arm64_v8a/prefix/include -I/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/include -I/usr/include/python3.8 -c cpp/logging.cc -o build/temp.linux_aarch64-cpython-38/cpp/logging.o -std=c++17 -fvisibility=hidden
error: command '/home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/wrappers/aarch64-linux-android21-clang++' failed with exit code 1
1 error generated.
1 error generated.
build-wheel: Error: Command '. /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/bin/activate; /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build.sh' returned non-zero exit status 1.
1 error generated.
aryan@NPCI-Model-deploy:~/aryan/chaquopy/server/pypi$ In file included from cpp/logging.cc:1:
In file included from cpp/module.h:3:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/pybind11.h:13:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/class.h:12:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/../attr.h:13:
In file included from /home/aryan/aryan/chaquopy/server/pypi/packages/ctranslate2-python/build/4.1.0/cp38-cp38-android_21_arm64_v8a/env/lib/python3.8/site-packages/pybind11/include/pybind11/detail/common.h:266:
In file included from /usr/include/python3.8/Python.h:8:
/usr/include/python3.8/pyconfig.h:9:12: fatal error: 'aarch64-linux-gnu/python3.8/pyconfig.h' file not found
#  include <aarch64-linux-gnu/python3.8/pyconfig.h>
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.

Please can you help me around this? It is soem cross-compiling issue

mhsmith · 2024-10-20T14:55:35Z

It's important to make a distinction between the Linux headers and libraries of the "build" platform, and the Android headers and libraries of the "host" platform. Including LInux components in an Android build will never work – that's why compiler-wrapper removes them from the command line.

Instead, you need to work out why the Android components aren't being found. In this case, Python.h wasn't being found because when the source code is downloaded from anywhere except the default of PyPI, build-wheel doesn't know that this is a Python package and won't add the necessary entry to CFLAGS. You can fix this by adding a host requirement of python, as in the cmake-example recipe.

Also, unless there's something very unusual about this package, you should not provide a build.sh script for the Python recipe. Remove it, and the package will be built using a standard PEP 517 process.

There are several other issues with this recipe:

The name of this package on PyPI is simply ctranslate2. You should match this, and not add a -python suffix.
For the non-Python library, you should follow the convention of the other recipes and call it chaquopy-ctranslate2, not ctranslate2-lib.
The path settings in meta.yaml refer to your local copies of the source code. This may be necessary for the Python component because of its directory structure, as mentioned here, but it won't be necessary for the C++ component, since the build script can simply change directories before doing anything. So the C++ recipe should get its source from a url or git_url – see the other recipes for examples.
If any recipe does require you to download the source code manually, then the meta.yaml file should contain a comment giving instructions about where to get it from and where to unpack it. The unpacked location should be within the recipe directory, and listed in a .gitignore file.
If any edits need to be made to the source code, these should not be made to the original location, but the copy in the build directory, and then saved as patch files. See "If any changes are needed to make the build work" in the README.
The variable CHAQUOPY_LIB is used but never defined.

aryansharma added 2 commits September 28, 2024 06:38

Added Ctranslate2-lib and Ctranslate2-python. Fixes chaquo#1239.

6c5d3a7

Added libiomp.so to CMakeLists.txt to solve intel backend error

523b73a

aryan1165 mentioned this pull request Oct 4, 2024

We need Ctranslate2 plz #1239

Open

mhsmith added the packages label Oct 9, 2024

mhsmith force-pushed the master branch from 6182abc to 7875131 Compare October 15, 2024 22:08

mhsmith changed the title ~~Added Ctranslate2-lib and Ctranslate2-python. Fixes #1239.~~ Added Ctranslate2 Oct 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added Ctranslate2 #1253

Added Ctranslate2 #1253

aryan1165 commented Sep 28, 2024 •

edited by mhsmith

Loading

aryan1165 commented Sep 28, 2024

mhsmith commented Oct 20, 2024 •

edited

Loading

mhsmith commented Oct 20, 2024

Added Ctranslate2 #1253

Are you sure you want to change the base?

Added Ctranslate2 #1253

Conversation

aryan1165 commented Sep 28, 2024 • edited by mhsmith Loading

aryan1165 commented Sep 28, 2024

mhsmith commented Oct 20, 2024 • edited Loading

mhsmith commented Oct 20, 2024

aryan1165 commented Sep 28, 2024 •

edited by mhsmith

Loading

mhsmith commented Oct 20, 2024 •

edited

Loading