Installation procedure has chronic crashes due to bulky files. Crash recovery fails. Manual procedure needed.

[bruceleerabbit]

The upgrade from Debian Bullseye to Bookworm caused an upgrade from one subversion of python 3 to another. The python packaging system is extremely fragile and cannot handle upgrades of itself. This is what happens after upgrading Bullseye to Bookworm:

  $ /usr/local/bin/argos-translate -v
  Traceback (most recent call last):
    File "/usr/local/bin/argos-translate", line 3, in <module>
      from argostranslate import cli
  ModuleNotFoundError: No module named 'argostranslate'

Debian users were not warned that argostranslate would be broken in the upgrade. The app was silently removed and even after the fact the user receives no indication that argostranslate was ruined. Users should be able to plan their upgrades. This entails informing them of the consequences. It was only known that argostranslate was destroyed at the time a user attempts to run it. Failing that, the reinstallation was a disaster because pip tries to manage some enormous file downloads, and it’s not well suited for this because it’s incapable of crash recovery. I tried installing from 4 different libraries. At each library, I tried clearnet, Tor, and VPNs. Every attempt made a different amount of progress before falling over. My first attempt looked like this:

  $ rename 's/$/.old/' /usr/local/bin/argos*
  $ cd /usr/local/src/
  $ git clone --depth 1 --shallow-submodules --config http.proxy=http://127.0.0.1:8118 https://github.com/argosopentech/argos-translate.git
  $ cd argos-translate
  $ virtualenv env
  $ source env/bin/activate
  $ pip install --proxy 127.0.0.1:8118 --log-file $logdir/pip-argostranslate_instal_$(date +%F).err --log $logdir/pip-argostranslate_instal_$(date +%F).log -e .

  Obtaining file:///usr/local/src/argos-translate
    Preparing metadata (setup.py) ... done
  Collecting ctranslate2<5,>=4.0
    Downloading ctranslate2-4.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (179.6 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━ 143.9/179.6 MB 41.0 kB/s eta 0:10:25
  ERROR: Exception:
  Traceback (most recent call last):
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_vendor/urllib3/response.py", line 438, in _error_catcher
      yield
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_vendor/urllib3/response.py", line 561, in read
      data = self._fp_read(amt) if not fp_closed else b""
             ^^^^^^^^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_vendor/urllib3/response.py", line 527, in _fp_read
      return self._fp.read(amt) if amt is not None else self._fp.read()
             ^^^^^^^^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 90, in read
      data = self.__fp.read(amt)
             ^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3.11/http/client.py", line 465, in read
      s = self.fp.read(amt)
          ^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3.11/socket.py", line 706, in readinto
      return self._sock.recv_into(b)
             ^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3.11/ssl.py", line 1278, in recv_into
      return self.read(nbytes, buffer)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3.11/ssl.py", line 1134, in read
      return self._sslobj.read(len, buffer)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  TimeoutError: The read operation timed out
  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
      status = run_func(*args)
               ^^^^^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
      return func(self, options, args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_internal/commands/install.py", line 419, in run
      requirement_set = resolver.resolve(
                        ^^^^^^^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
      result = self._result = resolver.resolve(
                              ^^^^^^^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
      state = resolution.resolve(requirements, max_rounds=max_rounds)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 373, in resolve
      failure_causes = self._attempt_to_pin_criterion(name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
      criteria = self._get_updated_criteria(candidate)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 204, in _get_updated_criteria
      self._add_to_criteria(criteria, requirement, parent=candidate)
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
      if not criterion.candidates:
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
      return bool(self._sequence)
             ^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
      return any(self)
             ^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
      return (c for c in iterator if id(c) not in self._incompatible_ids)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
      candidate = func()
                  ^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link
      self._link_candidate_cache[link] = LinkCandidate(
                                         ^^^^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 297, in __init__
      super().__init__(
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 162, in __init__
      self.dist = self._prepare()
                  ^^^^^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 231, in _prepare
      dist = self._prepare_distribution()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 308, in _prepare_distribution
      return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 491, in prepare_linked_requirement
      return self._prepare_linked_requirement(req, parallel_builds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 536, in _prepare_linked_requirement
      local_file = unpack_url(
                   ^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 166, in unpack_url
      file = get_http_url(
             ^^^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 107, in get_http_url
      from_path, content_type = download(link, temp_dir.path)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_internal/network/download.py", line 147, in __call__
      for chunk in chunks:
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_internal/cli/progress_bars.py", line 53, in _rich_progress_bar
      for chunk in iterable:
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_internal/network/utils.py", line 63, in response_chunks
      for chunk in response.raw.stream(
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_vendor/urllib3/response.py", line 622, in stream
      data = self.read(amt=amt, decode_content=decode_content)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_vendor/urllib3/response.py", line 560, in read
      with self._error_catcher():
    File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__
      self.gen.throw(typ, value, traceback)
    File "/usr/local/src/argos-translate/env/lib/python3.11/site-packages/pip/_vendor/urllib3/response.py", line 443, in _error_catcher
      raise ReadTimeoutError(self._pool, None, "Read timed out.")
  pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.
  (env) cypher:/usr/local/src/argos-translate#

The pipx tool is better than pip in some ways but it loses the proxy capability and the log file options (but at least it has a hard-coded log file). That attempt looked like this:

PIPX_HOME=${prefix:-/opt/}/pipx PIPX_BIN_DIR=${prefix:-/usr/local}/bin pipx install --verbose argostranslate
  pipx >(setup:757): pipx version is 1.1.0
  pipx >(setup:758): Default python interpreter is '/usr/bin/python3'
  pipx >(package_name_from_spec:323): Determined package name: argostranslate
  pipx >(package_name_from_spec:324): Package name determined in 0.0s
  creating virtual environment...
  pipx >(run_subprocess:173): running /usr/bin/python3 -m venv --without-pip /opt/pipx/venvs/argostranslate
  pipx >(run_subprocess:173): running /opt/pipx/venvs/argostranslate/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
  pipx >(run_subprocess:173): running /opt/pipx/shared/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
  pipx >(run_subprocess:173): running /opt/pipx/venvs/argostranslate/bin/python --version
  pipx >(_parsed_package_to_package_or_url:128): cleaned package spec: argostranslate
  installing argostranslate...
  pipx >(run_subprocess:173): running /opt/pipx/venvs/argostranslate/bin/python -m pip install argostranslate
  pipx >(subprocess_post_check_handle_pip_error:335): '/opt/pipx/venvs/argostranslate/bin/python -m pip install argostranslate' failed
  pipx >(subprocess_post_check_handle_pip_error:352): Fatal error from pip prevented installation. Full pip output in file:
      /opt/pipx/logs/cmd_$timestamp_pip_errors.log

  pipx >(analyze_pip_output:310): pip seemed to fail to build package:
      mpmath>=0.19

  Some possibly relevant errors from pip install:
      TimeoutError: The read operation timed out
      pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.

  pipx >(rmdir:55): removing directory /opt/pipx/venvs/argostranslate
  Error installing argostranslate.

Log file looked like this:

   153.1ms (setup:756): /usr/bin/pipx install --verbose argostranslate
   153.2ms (setup:757): pipx version is 1.1.0
   153.3ms (setup:758): Default python interpreter is '/usr/bin/python3'
   154.5ms (package_name_from_spec:323): Determined package name: argostranslate
   154.6ms (package_name_from_spec:324): Package name determined in 0.0s
   155.0ms (run_subprocess:173): running /usr/bin/python3 -m venv --without-pip /opt/pipx/venvs/argostranslate
   227.7ms (run_subprocess:186): stdout:
   227.8ms (run_subprocess:188): stderr:
   227.9ms (run_subprocess:189): returncode: 0
   228.1ms (run_subprocess:173): running /opt/pipx/venvs/argostranslate/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
   250.1ms (run_subprocess:186): stdout: /opt/pipx/venvs/argostranslate/lib/python3.11/site-packages
   250.2ms (run_subprocess:189): returncode: 0
   250.5ms (run_subprocess:173): running /opt/pipx/shared/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
   279.3ms (run_subprocess:186): stdout: /opt/pipx/shared/lib/python3.11/site-packages
   279.4ms (run_subprocess:189): returncode: 0
   279.8ms (run_subprocess:173): running /opt/pipx/venvs/argostranslate/bin/python --version
   281.9ms (run_subprocess:186): stdout: Python 3.11.2
   282.0ms (run_subprocess:188): stderr:
   282.0ms (run_subprocess:189): returncode: 0
   282.3ms (_parsed_package_to_package_or_url:128): cleaned package spec: argostranslate
   282.7ms (run_subprocess:173): running /opt/pipx/venvs/argostranslate/bin/python -m pip install argostranslate
169222.9ms (run_subprocess:189): returncode: 2
169223.1ms (subprocess_post_check_handle_pip_error:335): '/opt/pipx/venvs/argostranslate/bin/python -m pip install argostranslate' failed
169223.7ms (subprocess_post_check_handle_pip_error:352): Fatal error from pip prevented installation. Full pip output in file:
    /opt/pipx/logs/cmd_2024-05-02_17.28.57_pip_errors.log
169225.9ms (analyze_pip_output:310): pip seemed to fail to build package:
    mpmath>=0.19
169226.4ms (rmdir:55): removing directory /opt/pipx/venvs/argostranslate
169227.8ms (cli:822): PipxError: Error installing argostranslate.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pipx/main.py", line 819, in cli
    return run_pipx_command(parsed_pipx_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pipx/main.py", line 202, in run_pipx_command
    return commands.install(
           ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pipx/commands/install.py", line 60, in install
    venv.install_package(
  File "/usr/lib/python3/dist-packages/pipx/venv.py", line 240, in install_package
    raise PipxError(
pipx.util.PipxError: Error installing argostranslate.
169229.0ms (cli:830): pipx finished.

The error log file looked like this:

PIP STDOUT
----------
Collecting argostranslate
  Using cached argostranslate-1.9.6-py3-none-any.whl.metadata (10 kB)
Collecting ctranslate2<5,>=4.0 (from argostranslate)
  Using cached ctranslate2-4.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (10 kB)
Collecting packaging (from argostranslate)
  Using cached packaging-24.0-py3-none-any.whl.metadata (3.2 kB)
Collecting sacremoses==0.0.53 (from argostranslate)
  Using cached sacremoses-0.0.53.tar.gz (880 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting sentencepiece==0.2.0 (from argostranslate)
  Using cached sentencepiece-0.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.7 kB)
Collecting stanza==1.1.1 (from argostranslate)
  Using cached stanza-1.1.1-py3-none-any.whl.metadata (10 kB)
Collecting regex (from sacremoses==0.0.53->argostranslate)
  Using cached regex-2024.4.28-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (40 kB)
Collecting six (from sacremoses==0.0.53->argostranslate)
  Using cached six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Collecting click (from sacremoses==0.0.53->argostranslate)
  Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting joblib (from sacremoses==0.0.53->argostranslate)
  Using cached joblib-1.4.2-py3-none-any.whl.metadata (5.4 kB)
Collecting tqdm (from sacremoses==0.0.53->argostranslate)
  Using cached tqdm-4.66.2-py3-none-any.whl.metadata (57 kB)
Collecting numpy (from stanza==1.1.1->argostranslate)
  Using cached numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)
Collecting protobuf (from stanza==1.1.1->argostranslate)
  Using cached protobuf-5.26.1-cp37-abi3-manylinux2014_x86_64.whl.metadata (592 bytes)
Collecting requests (from stanza==1.1.1->argostranslate)
  Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting torch>=1.3.0 (from stanza==1.1.1->argostranslate)
  Using cached torch-2.3.0-cp311-cp311-manylinux1_x86_64.whl.metadata (26 kB)
Requirement already satisfied: setuptools in /opt/pipx/shared/lib/python3.11/site-packages (from ctranslate2<5,>=4.0->argostranslate) (69.5.1)
Collecting pyyaml<7,>=5.3 (from ctranslate2<5,>=4.0->argostranslate)
  Using cached PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
Collecting filelock (from torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached filelock-3.14.0-py3-none-any.whl.metadata (2.8 kB)
Collecting typing-extensions>=4.8.0 (from torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached typing_extensions-4.11.0-py3-none-any.whl.metadata (3.0 kB)
Collecting sympy (from torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached sympy-1.12-py3-none-any.whl.metadata (12 kB)
Collecting networkx (from torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached networkx-3.3-py3-none-any.whl.metadata (5.1 kB)
Collecting jinja2 (from torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached Jinja2-3.1.3-py3-none-any.whl.metadata (3.3 kB)
Collecting fsspec (from torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached fsspec-2024.3.1-py3-none-any.whl.metadata (6.8 kB)
Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cublas-cu12==12.1.3.1 (from torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cufft-cu12==11.0.2.54 (from torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-curand-cu12==10.3.2.106 (from torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-nccl-cu12==2.20.5 (from torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl.metadata (1.8 kB)
Collecting nvidia-nvtx-cu12==12.1.105 (from torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB)
Collecting triton==2.3.0 (from torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached triton-2.3.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.4 kB)
Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached nvidia_nvjitlink_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
Collecting charset-normalizer<4,>=2 (from requests->stanza==1.1.1->argostranslate)
  Using cached charset_normalizer-3.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (33 kB)
Collecting idna<4,>=2.5 (from requests->stanza==1.1.1->argostranslate)
  Using cached idna-3.7-py3-none-any.whl.metadata (9.9 kB)
Collecting urllib3<3,>=1.21.1 (from requests->stanza==1.1.1->argostranslate)
  Using cached urllib3-2.2.1-py3-none-any.whl.metadata (6.4 kB)
Collecting certifi>=2017.4.17 (from requests->stanza==1.1.1->argostranslate)
  Using cached certifi-2024.2.2-py3-none-any.whl.metadata (2.2 kB)
Collecting MarkupSafe>=2.0 (from jinja2->torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached MarkupSafe-2.1.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Collecting mpmath>=0.19 (from sympy->torch>=1.3.0->stanza==1.1.1->argostranslate)
  Using cached mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)
Using cached argostranslate-1.9.6-py3-none-any.whl (34 kB)
Using cached sentencepiece-0.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
Using cached stanza-1.1.1-py3-none-any.whl (227 kB)
Using cached ctranslate2-4.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (179.6 MB)
Using cached packaging-24.0-py3-none-any.whl (53 kB)
Using cached PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (757 kB)
Downloading torch-2.3.0-cp311-cp311-manylinux1_x86_64.whl (779.2 MB)
   ━━━━━━━━━                                160.2/779.2 MB 1.4 MB/s eta 0:06:55

PIP STDERR
----------
ERROR: Exception:
Traceback (most recent call last):
  File "/opt/pipx/shared/lib/python3.11/site-packages/pip/_vendor/urllib3/response.py", line 438, in _error_catcher
    yield
  File "/opt/pipx/shared/lib/python3.11/site-packages/pip/_vendor/urllib3/response.py", line 561, in read
    data = self._fp_read(amt) if not fp_closed else b""
           ^^^^^^^^^^^^^^^^^^
  File "/opt/pipx/shared/lib/python3.11/site-packages/pip/_vendor/urllib3/response.py", line 527, in _fp_read
    return self._fp.read(amt) if amt is not None else self._fp.read()
           ^^^^^^^^^^^^^^^^^^
  File "/opt/pipx/shared/lib/python3.11/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 98, in read
    data: bytes = self.__fp.read(amt)
                  ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/http/client.py", line 465, in read
    s = self.fp.read(amt)
        ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/socket.py", line 706, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 1278, in recv_into
    return self.read(nbytes, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 1134, in read
    return self._sslobj.read(len, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/pipx/shared/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/opt/pipx/shared/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pipx/shared/lib/python3.11/site-packages/pip/_internal/commands/install.py", line 377, in run
    requirement_set = resolver.resolve(
                      ^^^^^^^^^^^^^^^^^
  File "/opt/pipx/shared/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 179, in resolve
    self.factory.preparer.prepare_linked_requirements_more(reqs)
  File "/opt/pipx/shared/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 552, in prepare_linked_requirements_more
    self._complete_partial_requirements(
  File "/opt/pipx/shared/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 467, in _complete_partial_requirements
    for link, (filepath, _) in batch_download:
  File "/opt/pipx/shared/lib/python3.11/site-packages/pip/_internal/network/download.py", line 183, in __call__
    for chunk in chunks:
  File "/opt/pipx/shared/lib/python3.11/site-packages/pip/_internal/cli/progress_bars.py", line 53, in _rich_progress_bar
    for chunk in iterable:
  File "/opt/pipx/shared/lib/python3.11/site-packages/pip/_internal/network/utils.py", line 63, in response_chunks
    for chunk in response.raw.stream(
  File "/opt/pipx/shared/lib/python3.11/site-packages/pip/_vendor/urllib3/response.py", line 622, in stream
    data = self.read(amt=amt, decode_content=decode_content)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pipx/shared/lib/python3.11/site-packages/pip/_vendor/urllib3/response.py", line 560, in read
    with self._error_catcher():
  File "/usr/lib/python3.11/contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/opt/pipx/shared/lib/python3.11/site-packages/pip/_vendor/urllib3/response.py", line 443, in _error_catcher
    raise ReadTimeoutError(self._pool, None, "Read timed out.")
pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.

The pip* tools are able to recover whole files that completed, but they cannot recover from partial files. So it took several attempts at a couple libraries to fetch ctranslate2-4.2.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl, everytime starting back at the beginning of that file. This means the servers were having to do a lot of wasted work by resending data that was already previously sent. Even though that whl file is ~180mb, I probably pulled several hundred megs until it was fetched in one complete session from beginning to end. Then came torch-2.3.0-cp311-cp311-manylinux1_x86_64.whl which is 779.2 MB. I visited several libraries over several days making reattempts to fetch that monster. Three libraries seemed to have no hope. They struggled to grab even 200mb of it. The only reason I was able to complete the install was because I happened to come accross a library that could handle it, incidentally. But it required some hackery to bypass the library’s QoS mechanism.

This installation procedure is unsustainable. A normal user would not have been able to install argos-translate without having some quite high quality unlimited bandwidth internet service that exceeds that of public libraries.

We need a procedure for using wget to download the big files so that when the fetch dies we can use wget -c to recover. Then we need a way to feed the files to pipx.

Even better would be a proper debian package, because debian’s pkg manager is immune to the fragility of pip. There are many ways to fetch a deb file especially if it’s in the official debian repos.