feat: replace Elasticsearch by Meilisearch

changelog.d/20241017_125209_regis_meilisearch.md

@@ -0,0 +1 @@
+- 💥[Feature] Replace Elasticsearch by Meilisearch. Elasticsearch was both a source of complexity and high resource usage. With this change, we no longer run Elasticsearch to perform common search queries across Open edX. This includes: course discovery, courseware search and studio search. Instead, we index all these documents in a Meilisearch instance, which is much more lightweight in terms of memory consumption. (by @regisb)

docs/configuration.rst

@@ -40,7 +40,7 @@ With an up-to-date environment, Tutor is ready to launch an Open edX platform an
 Individual service activation
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-- ``RUN_ELASTICSEARCH`` (default: ``true``)
+- ``RUN_MEILISEARCH`` (default: ``true``)
 - ``RUN_MONGODB`` (default: ``true``)
 - ``RUN_MYSQL`` (default: ``true``)
 - ``RUN_REDIS`` (default: ``true``)
@@ -71,9 +71,9 @@ This configuration parameter defines the name of the Docker image to run the dev
 
 This configuration parameter defines which Caddy Docker image to use.
 
-- ``DOCKER_IMAGE_ELASTICSEARCH`` (default: ``"docker.io/elasticsearch:7.17.9"``)
+- ``DOCKER_IMAGE_MEILISEARCH`` (default: ``"docker.io/getmeili/meilisearch:v1.8.4"``)
 
-This configuration parameter defines which Elasticsearch Docker image to use.
+This configuration parameter defines which Meilisearch Docker image to use.
 
 - ``DOCKER_IMAGE_MONGODB`` (default: ``"docker.io/mongo:7.0.7"``)
 
@@ -228,13 +228,19 @@ By default, a running Open edX platform deployed with Tutor includes all necessa
 .. note::
     When configuring an external MySQL database, please make sure it is using version 8.4.
 
-Elasticsearch
-*************
+Meilisearch
+***********
+
+- ``MEILISEARCH_URL`` (default: ``"http://meilisearch:7700"``): internal URL used for backend-to-backend communication.
+- ``MEILISEARCH_PUBLIC_URL`` (default: ``"{% if ENABLE_HTTPS %}https{% else %}http{% endif %}://meilisearch.{{ LMS_HOST }}"``): external URL from which the frontend will access the Meilisearch instance.
+- ``MEILISEARCH_INDEX_PREFIX`` (default: ``"tutor_"``)
+- ``MEILISEARCH_MASTER_KEY`` (default: ``"{{ 24|random_string }}"``)
+- ``MEILISEARCH_API_KEY_UID`` (default: ``"{{ 4|uuid }}"``): UID used to sign the API key.
+- ``MEILISEARCH_API_KEY`` (default: ``"{{ MEILISEARCH_MASTER_KEY|uid_master_hash(MEILISEARCH_API_KEY_UID) }}"``)
+
+To reset the Meilisearch API key, make sure to unset both the API key and it's UID:
 
-- ``ELASTICSEARCH_SCHEME`` (default: ``"http"``)
-- ``ELASTICSEARCH_HOST`` (default: ``"elasticsearch"``)
-- ``ELASTICSEARCH_PORT`` (default: ``9200``)
-- ``ELASTICSEARCH_HEAP_SIZE`` (default: ``"1g"``)
+    tutor config save --unset MEILISEARCH_API_KEY_UID MEILISEARCH_API_KEY
 
 MongoDB
 *******

docs/tutorials/nightly.rst

@@ -58,7 +58,7 @@ When running Tutor Nightly, you usually do not want to override your existing Tu
 Making changes to Tutor Nightly
 -------------------------------
 
-In general pull requests should be open on the "master" branch of Tutor: the "master" branch is automatically merged on the "nightly" branch at every commit, such that changes made to Tutor releases find their way to Tutor Nightly as soon as they are merged. However, sometimes you want to make changes to Tutor Nightly exclusively, and not to the Tutor releases. This might be the case for instance when upgrading the running version of a third-party service (for instance: Elasticsearch, MySQL), or when the master branch requires specific changes. In that case, you should follow the instructions from the :ref:`contributing` section of the docs, with the following differences:
+In general pull requests should be open on the "master" branch of Tutor: the "master" branch is automatically merged on the "nightly" branch at every commit, such that changes made to Tutor releases find their way to Tutor Nightly as soon as they are merged. However, sometimes you want to make changes to Tutor Nightly exclusively, and not to the Tutor releases. This might be the case for instance when upgrading the running version of a third-party service (for instance: Meilisearch, MySQL), or when the master branch requires specific changes. In that case, you should follow the instructions from the :ref:`contributing` section of the docs, with the following differences:
 
 - Open your pull request on top of the "nightly" branch instead of "master".
 - Add a description of your changes by creating a changelog entry with `make changelog-entry`, as in the master branch.

docs/tutorials/scale.rst

@@ -37,11 +37,11 @@ Offloading data storage
 
 Aside from web workers, the most resource-intensive services are in the data persistence layer. They are, by decreasing resource usage:
 
-- `Elasticsearch <https://www.elastic.co/elasticsearch/>`__: indexing of course contents and forum topics, mostly for search. Elasticsearch is never a source of truth in Open edX, and the data can thus be trashed and re-created safely.
 - `MySQL <https://www.mysql.com>`__: structured, consistent data storage which is the default destination of all data.
 - `MongoDB <https://www.mongodb.com>`__: structured storage of course data.
 - `Redis <https://redis.io/>`__: caching and asynchronous task management.
 - `MinIO <https://min.io>`__: S3-like object storage for user-uploaded files, which is enabled by the `tutor-minio <https://github.com/overhangio/tutor-minio>`__ plugin. It is possible to replace MinIO by direct filesystem storage (the default), but scaling will then become much more difficult down the road.
+- `Meilisearch <https://www.meilisearch.com>`__: indexing of course contents and forum topics, mostly for search. Meilisearch is never a source of truth in Open edX, and the data can thus be trashed and re-created safely.
 
 When attempting to scale a single-server deployment, we recommend starting by offloading some of these stateful data storage components, in the same order of priority. There are multiple benefits:
 

tutor/commands/images.py

@@ -63,7 +63,7 @@ def _add_images_to_pull(
     """
     vendor_images = [
         ("caddy", "DOCKER_IMAGE_CADDY"),
-        ("elasticsearch", "DOCKER_IMAGE_ELASTICSEARCH"),
+        ("meilisearch", "DOCKER_IMAGE_MEILISEARCH"),
         ("mongodb", "DOCKER_IMAGE_MONGODB"),
         ("mysql", "DOCKER_IMAGE_MYSQL"),
         ("redis", "DOCKER_IMAGE_REDIS"),

tutor/commands/jobs.py

@@ -42,6 +42,10 @@ def _add_core_init_tasks() -> None:
         hooks.Filters.CLI_DO_INIT_TASKS.add_item(
             ("mysql", env.read_core_template_file("jobs", "init", "mysql.sh"))
         )
+    with hooks.Contexts.app("meilisearch").enter():
+        hooks.Filters.CLI_DO_INIT_TASKS.add_item(
+            ("lms", env.read_core_template_file("jobs", "init", "meilisearch.sh"))
+        )
     with hooks.Contexts.app("lms").enter():
         hooks.Filters.CLI_DO_INIT_TASKS.add_item(
             (

tutor/commands/k8s.py

@@ -390,7 +390,7 @@ def _start_base_deployments(_job_name: str, *_args: Any, **_kwargs: Any) -> None
         """
         config = tutor_config.load(context.root)
         wait_for_deployment_ready(config, "caddy")
-        for name in ["elasticsearch", "mysql", "mongodb"]:
+        for name in ["meilisearch", "mysql", "mongodb"]:
             if tutor_config.is_service_activated(config, name):
                 wait_for_deployment_ready(config, name)
 

tutor/config.py

@@ -235,7 +235,6 @@ def upgrade_obsolete(config: Config) -> None:
     for name in [
         "ACTIVATE_LMS",
         "ACTIVATE_CMS",
-        "ACTIVATE_ELASTICSEARCH",
         "ACTIVATE_MONGODB",
         "ACTIVATE_MYSQL",
         "ACTIVATE_REDIS",

tutor/env.py

@@ -1,10 +1,10 @@
 from __future__ import annotations
 
+from copy import deepcopy
 import os
 import re
 import shutil
 import typing as t
-from copy import deepcopy
 
 import importlib_resources
 import jinja2
@@ -56,6 +56,8 @@ def _prepare_environment() -> None:
             ("reverse_host", utils.reverse_host),
             ("rsa_import_key", utils.rsa_import_key),
             ("rsa_private_key", utils.rsa_private_key),
+            ("uuid", utils.uuid),
+            ("uid_master_hash", utils.uid_master_hash),
         ],
     )
     # Template variables

tutor/plugins/openedx.py

@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+
 import os
 import re
 import typing as t
@@ -28,6 +29,17 @@ def _edx_platform_public_hosts(
     return hosts
 
 
+@hooks.Filters.APP_PUBLIC_HOSTS.add()
+def _meilisearch_public_hosts(
+    hosts: list[str], context_name: t.Literal["local", "dev"]
+) -> list[str]:
+    if context_name == "dev":
+        hosts.append("{{ MEILISEARCH_PUBLIC_URL.split('://')[1] }}:7700")
+    else:
+        hosts.append("{{ MEILISEARCH_PUBLIC_URL.split('://')[1] }}")
+    return hosts
+
+
 @hooks.Filters.IMAGES_BUILD_MOUNTS.add()
 def _mount_edx_platform_build(
     volumes: list[tuple[str, str]], path: str

tutor/templates/apps/caddy/Caddyfile

@@ -82,4 +82,10 @@
     }
 }
 
+{% if RUN_MEILISEARCH %}
+{{ MEILISEARCH_PUBLIC_URL.split("://")[1] }}{$default_site_port} {
+    import proxy "meilisearch:7700"
+}
+{% endif %}
+
 {{ patch("caddyfile") }}

tutor/templates/apps/openedx/settings/cms/development.py

@@ -8,6 +8,8 @@
 CMS_BASE = "{{ CMS_HOST }}:8001"
 CMS_ROOT_URL = "http://" + CMS_BASE
 
+MEILISEARCH_PUBLIC_URL = "{{ MEILISEARCH_PUBLIC_URL }}:7700"
+
 # Authentication
 SOCIAL_AUTH_EDX_OAUTH2_KEY = "{{ CMS_OAUTH2_KEY_SSO_DEV }}"
 SOCIAL_AUTH_EDX_OAUTH2_PUBLIC_URL_ROOT = LMS_ROOT_URL

tutor/templates/apps/openedx/settings/lms/development.py

@@ -15,6 +15,8 @@
 CMS_ROOT_URL = "http://{}".format(CMS_BASE)
 LOGIN_REDIRECT_WHITELIST.append(CMS_BASE)
 
+MEILISEARCH_PUBLIC_URL = "{{ MEILISEARCH_PUBLIC_URL }}:7700"
+
 # Session cookie
 SESSION_COOKIE_DOMAIN = "{{ LMS_HOST }}"
 SESSION_COOKIE_SECURE = False

tutor/templates/apps/openedx/settings/partials/common_all.py

@@ -34,12 +34,14 @@
 # Behave like memcache when it comes to connection errors
 DJANGO_REDIS_IGNORE_EXCEPTIONS = True
 
-# Elasticsearch connection parameters
-ELASTIC_SEARCH_CONFIG = [{
-  {% if ELASTICSEARCH_SCHEME == "https" %}"use_ssl": True,{% endif %}
-  "host": "{{ ELASTICSEARCH_HOST }}",
-  "port": {{ ELASTICSEARCH_PORT }},
-}]
+# Meilisearch connection parameters
+MEILISEARCH_ENABLED = True
+MEILISEARCH_URL = "{{ MEILISEARCH_URL }}"
+MEILISEARCH_PUBLIC_URL = "{{ MEILISEARCH_PUBLIC_URL }}"
+MEILISEARCH_INDEX_PREFIX = "{{ MEILISEARCH_INDEX_PREFIX }}"
+MEILISEARCH_API_KEY = "{{ MEILISEARCH_API_KEY }}"
+MEILISEARCH_MASTER_KEY = "{{ MEILISEARCH_MASTER_KEY }}"
+SEARCH_ENGINE = "search.meilisearch.MeilisearchEngine"
 
 # Common cache config
 CACHES = {

tutor/templates/apps/openedx/settings/partials/common_cms.py

@@ -20,6 +20,9 @@
 FRONTEND_LOGIN_URL = LMS_ROOT_URL + '/login'
 FRONTEND_REGISTER_URL = LMS_ROOT_URL + '/register'
 
+# Enable "reindex" button
+FEATURES["ENABLE_COURSEWARE_INDEX"] = True
+
 # Create folders if necessary
 for folder in [LOG_DIR, MEDIA_ROOT, STATIC_ROOT, ORA2_FILEUPLOAD_ROOT]:
     if not os.path.exists(folder):

tutor/templates/apps/openedx/settings/partials/common_lms.py

@@ -37,6 +37,10 @@
     "LOCATION": "staticfiles_lms",
 }
 
+# Enable search features
+FEATURES["ENABLE_COURSE_DISCOVERY"] = True
+FEATURES["ENABLE_COURSEWARE_SEARCH"] = True
+
 # Create folders if necessary
 for folder in [DATA_DIR, LOG_DIR, MEDIA_ROOT, STATIC_ROOT, ORA2_FILEUPLOAD_ROOT]:
     if not os.path.exists(folder):

tutor/templates/apps/permissions/setowners.sh

@@ -1,6 +1,6 @@
 #! /bin/sh
 setowner $OPENEDX_USER_ID /mounts/lms /mounts/cms /mounts/openedx
-{% if RUN_ELASTICSEARCH %}setowner 1000 /mounts/elasticsearch{% endif %}
+{% if RUN_MEILISEARCH %}setowner 1000 /mounts/meilisearch{% endif %}
 {% if RUN_MONGODB %}setowner 999 /mounts/mongodb{% endif %}
 {% if RUN_MYSQL %}setowner 999 /mounts/mysql{% endif %}
 {% if RUN_REDIS %}setowner 1000 /mounts/redis{% endif %}

tutor/templates/build/openedx/Dockerfile

@@ -106,6 +106,12 @@ RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \
 # Install scorm xblock
 RUN pip install "openedx-scorm-xblock>=18.0.0,<19.0.0"
 
+# Install edx-search fork with meilisearch engine
+# TODO this should be a temporary change for the duration of the Sumac branch testing.
+# After that, the corresponding change should have been merged in edx-search and the
+# package upgraded in edx-platform (sumac release branch).
+RUN pip install git+https://github.com/regisb/edx-search@regisb/meilisearch#edx-search
+
 {% for extra_requirements in OPENEDX_EXTRA_PIP_REQUIREMENTS %}
 RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \
     pip install '{{ extra_requirements }}'
@@ -265,6 +271,8 @@ RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \
 # https://pypi.org/project/ipython (>=Python 3.10 started with 8.20)
 RUN --mount=type=cache,target=/openedx/.cache/pip,sharing=shared \
     pip install ipdb==0.13.13 ipython==8.24.0
+# Re-install forked edx-search which is otherwise overridden by development.txt
+RUN pip install git+https://github.com/regisb/edx-search@regisb/meilisearch#edx-search
 
 {# Re-install mounted requirements, otherwise they will be superseded by upstream reqs #}
 {% for name in iter_mounted_directories(MOUNTS, "openedx") %}

tutor/templates/config/base.yml

@@ -2,6 +2,9 @@
 CMS_OAUTH2_SECRET: "{{ 24|random_string }}"
 ID: "{{ 24|random_string }}"
 JWT_RSA_PRIVATE_KEY: "{{ 2048|rsa_private_key }}"
+MEILISEARCH_MASTER_KEY: "{{ 24|random_string }}"
+MEILISEARCH_API_KEY_UID: "{{ 4|uuid }}"
+MEILISEARCH_API_KEY: "{{ MEILISEARCH_MASTER_KEY|uid_master_hash(MEILISEARCH_API_KEY_UID) }}"
 MYSQL_ROOT_PASSWORD: "{{ 8|random_string }}"
 OPENEDX_MYSQL_PASSWORD: "{{ 8|random_string }}"
 OPENEDX_SECRET_KEY: "{{ 24|random_string }}"

tutor/templates/config/defaults.yml

@@ -16,8 +16,8 @@ DOCKER_IMAGE_OPENEDX: "{{ DOCKER_REGISTRY }}overhangio/openedx:{{ TUTOR_VERSION
 DOCKER_IMAGE_OPENEDX_DEV: "openedx-dev:{{ TUTOR_VERSION }}"
 # https://hub.docker.com/_/caddy/tags
 DOCKER_IMAGE_CADDY: "docker.io/caddy:2.7.4"
-# https://hub.docker.com/_/elasticsearch/tags
-DOCKER_IMAGE_ELASTICSEARCH: "docker.io/elasticsearch:7.17.13"
+# https://hub.docker.com/r/getmeili/meilisearch/tags
+DOCKER_IMAGE_MEILISEARCH: "docker.io/getmeili/meilisearch:v1.8.4"
 # https://hub.docker.com/_/mongo/tags
 DOCKER_IMAGE_MONGODB: "docker.io/mongo:7.0.7"
 # https://hub.docker.com/_/mysql/tags
@@ -29,10 +29,6 @@ DOCKER_IMAGE_REDIS: "docker.io/redis:7.2.4"
 DOCKER_IMAGE_SMTP: "docker.io/devture/exim-relay:4.96-r1-0"
 EDX_PLATFORM_REPOSITORY: "https://github.com/openedx/edx-platform.git"
 EDX_PLATFORM_VERSION: "{{ OPENEDX_COMMON_VERSION }}"
-ELASTICSEARCH_HOST: "elasticsearch"
-ELASTICSEARCH_PORT: 9200
-ELASTICSEARCH_SCHEME: "http"
-ELASTICSEARCH_HEAP_SIZE: 1g
 ENABLE_HTTPS: false
 ENABLE_WEB_PROXY: true
 JWT_COMMON_AUDIENCE: "openedx"
@@ -42,6 +38,9 @@ K8S_NAMESPACE: "openedx"
 LANGUAGE_CODE: "en"
 LMS_HOST: "www.myopenedx.com"
 LOCAL_PROJECT_NAME: "{{ TUTOR_APP }}_local"
+MEILISEARCH_URL: "http://meilisearch:7700"
+MEILISEARCH_PUBLIC_URL: "{% if ENABLE_HTTPS %}https{% else %}http{% endif %}://meilisearch.{{ LMS_HOST }}"
+MEILISEARCH_INDEX_PREFIX: "tutor_"
 MONGODB_AUTH_MECHANISM: ""
 MONGODB_AUTH_SOURCE: "admin"
 MONGODB_HOST: "mongodb"
@@ -73,7 +72,7 @@ REDIS_HOST: "redis"
 REDIS_PORT: 6379
 REDIS_USERNAME: ""
 REDIS_PASSWORD: ""
-RUN_ELASTICSEARCH: true
+RUN_MEILISEARCH: true
 RUN_MONGODB: true
 RUN_MYSQL: true
 RUN_REDIS: true

tutor/templates/dev/docker-compose.yml

@@ -32,19 +32,18 @@ services:
     ports:
         - "8001:8000"
 
+  meilisearch:
+    ports:
+      - "127.0.0.1:7700:7700"
+    networks:
+      default:
+        aliases:
+          - "{{ MEILISEARCH_PUBLIC_URL.split('://')[1] }}"
+
   # Additional service for watching theme changes
   watchthemes:
     <<: *openedx-service
     command: npm run watch-sass
     restart: unless-stopped
 
-  {% if RUN_ELASTICSEARCH and is_docker_rootless() %}
-  elasticsearch:
-    ulimits:
-      memlock:
-        # Fixes error setting rlimits for ready process in rootless docker
-        soft: 0  # zero means "unset" in the memlock context
-        hard: 0
-  {% endif %}
-
   {{ patch("local-docker-compose-dev-services")|indent(2) }}

tutor/templates/jobs/init/cms.sh

@@ -16,3 +16,7 @@ fi
 # Create waffle switches to enable some features, if they have not been explicitly defined before
 # Copy-paste of units in Studio (highly requested new feature, but defaults to off in Quince)
 (./manage.py cms waffle_flag --list | grep contentstore.enable_copy_paste_units) || ./manage.py lms waffle_flag --create contentstore.enable_copy_paste_units --everyone
+
+# Re-index studio and courseware content
+./manage.py cms reindex_studio --experimental
+./manage.py cms reindex_course --active

tutor/templates/jobs/init/meilisearch.sh

@@ -0,0 +1,18 @@
+# Get or create Meilisearch API key
+python -c "
+import meilisearch
+client = meilisearch.Client('{{ MEILISEARCH_URL }}', '{{ MEILISEARCH_MASTER_KEY }}')
+try:
+    client.get_key('{{ MEILISEARCH_API_KEY_UID }}')
+    print('Key already exists')
+except meilisearch.errors.MeilisearchApiError:
+    print('Key does not exist: creating...')
+    client.create_key({
+        'name': 'Open edX backend API key',
+        'uid': '{{ MEILISEARCH_API_KEY_UID }}',
+        'actions': ['*'],
+        'indexes': ['{{ MEILISEARCH_INDEX_PREFIX }}*'],
+        'expiresAt': None,
+        'description': 'Use it for backend API calls -- Created by Tutor',
+    })
+"