Starting OpenSearch Docker containers fails when using a custom config #768
Comments
|
Thanks for bringing this issue. Since this docs are still in beta, we do expect some gaps. Let us reproduce this from our end and fix the issue no 1 and more documentation to fix issue 2 and 3. We encourage you to raise a PR for fixing issue 1. Please see our developer guide on how you can contribute. |
|
I tried reproducing. Case 1: Running the sample docker compose file The container Case 2: Tried running with modified docker compose file removing container This time both I am yet to triage issue 1 with custom config files. For issue 3, the custom config files are modification of and/or addition of other OpenSearch and plugin settings. In the absence of default settings in opensearch.yml and opensearch-dashboards.yml documented, I agree it presents a barrier for users to tinker with settings. I will post a PR which mentions default config settings in docker and a example setting change like a different cluster name. |
|
Thanks abbashus! For clarification: What would help is if the custom config files (custom-opensearch_dashboards.yml and custom-opensearch.yml) are automatically created and filled with default settings, but only if they don't exist. This would fix issue one and two, which I reported earlier. As for your testing cases; case two works fine since you're not using custom.yml files in your Docker Compose file. |
|
Unfortunately, the links in this thread are not working anymore. I followed this guide https://opensearch.org/docs/latest/opensearch/install/docker-security/ and stumbled into the same issue where the mounts (certificates and config files) are created as folders instead of files. The errors are a bunch of this kind The guide does not mention it is required to create the certificates and config files before starting the containers, but I assume not doing that is what causes the issue. On a side note though: the creation of folders vs files is not related to opensearch but to docker itself. I removed all the certificate and most of the config mounts and only left the The first thing I notice is that Any idea how to get this solved? |
|
I suspect that the "containers fail to start properly" and "sed cannot do its thing" are the same root cause. See the following block in opensearch-docker-entrypoint.sh: The "sed -i" is an attempt to modify the opensearch.yml file "in place" -- But according to the GNU sed documentation (https://www.gnu.org/software/sed/manual/sed.html#Command_002dLine-Options), "in-place" actually "does this by creating a temporary file and sending output to this file rather than to the standard output. ... the temporary file is renamed to the output file’s original name". I believe this rename would require changing the inode of the original file -- something that Docker volume mounts don't permit. If this is correct, then to allow Dockerized OpenSearch to support a volume-mounted opensearch.yml, opensearch-docker-entrypoint.sh probably needs to use a method of updating the file which doesn't change the file's inode number, like "cat"ing the contents of the modified file over top of the original. |
|
Im curious about this issue. I am wondering if this bug was already on some older versions ?! |
|
Haven't checked how this might have been enabled in earlier incarnations of the software (e.g. OpenDistro, Elasticsearch), but if you check the history, this particular block was only added to OpenSearch 2 months ago. |
|
I believe this is the same as issue #1579 |
Using `sed -i` was causing an issue when a custom opensearch.yml file was mounted as a volume. ``` sed: cannot rename /usr/share/opensearch/config/sedqdMb0d: Device or resource busy ``` The reason for the issue was found by @unhipzippo opensearch-project/OpenSearch#768 (comment) ❤️ > The "sed -i" is an attempt to modify the opensearch.yml file "in place" -- But according to the GNU sed documentation (https://www.gnu.org/software/sed/manual/sed.html#Command_002dLine-Options), "in-place" actually "does this by creating a temporary file and sending output to this file rather than to the standard output. ... the temporary file is renamed to the output file’s original name". > > I believe this rename would require changing the inode of the original file -- something that Docker volume mounts don't permit.
Thanks for the investigation. It makes a lot of sense to me now. I just jumped the gun and created a PR, avoiding the |
|
It might be worth looking at an application called homeassistant, to see how they handle this. I have a Docker container running homeassistant, which exposes a configuration.yaml file, which can be edited from outside the container. |
Using `sed -i` was causing an issue when a custom opensearch.yml file was mounted as a volume. ``` sed: cannot rename /usr/share/opensearch/config/sedqdMb0d: Device or resource busy ``` The reason for the issue was found by @unhipzippo opensearch-project/OpenSearch#768 (comment) ❤️ > The "sed -i" is an attempt to modify the opensearch.yml file "in place" -- But according to the GNU sed documentation (https://www.gnu.org/software/sed/manual/sed.html#Command_002dLine-Options), "in-place" actually "does this by creating a temporary file and sending output to this file rather than to the standard output. ... the temporary file is renamed to the output file’s original name". > > I believe this rename would require changing the inode of the original file -- something that Docker volume mounts don't permit. Signed-off-by: Robin Böning <[email protected]>
Using `sed -i` was causing an issue when a custom opensearch.yml file was mounted as a volume. ``` sed: cannot rename /usr/share/opensearch/config/sedqdMb0d: Device or resource busy ``` The reason for the issue was found by @unhipzippo opensearch-project/OpenSearch#768 (comment) ❤️ > The "sed -i" is an attempt to modify the opensearch.yml file "in place" -- But according to the GNU sed documentation (https://www.gnu.org/software/sed/manual/sed.html#Command_002dLine-Options), "in-place" actually "does this by creating a temporary file and sending output to this file rather than to the standard output. ... the temporary file is renamed to the output file’s original name". > > I believe this rename would require changing the inode of the original file -- something that Docker volume mounts don't permit. Signed-off-by: Robin Böning <[email protected]>
Using `sed -i` was causing an issue when a custom opensearch.yml file was mounted as a volume. ``` sed: cannot rename /usr/share/opensearch/config/sedqdMb0d: Device or resource busy ``` The reason for the issue was found by @unhipzippo opensearch-project/OpenSearch#768 (comment) ❤️ > The "sed -i" is an attempt to modify the opensearch.yml file "in place" -- But according to the GNU sed documentation (https://www.gnu.org/software/sed/manual/sed.html#Command_002dLine-Options), "in-place" actually "does this by creating a temporary file and sending output to this file rather than to the standard output. ... the temporary file is renamed to the output file’s original name". > > I believe this rename would require changing the inode of the original file -- something that Docker volume mounts don't permit. Signed-off-by: Robin Böning <[email protected]>
Using `sed -i` was causing an issue when a custom opensearch.yml file was mounted as a volume. ``` sed: cannot rename /usr/share/opensearch/config/sedqdMb0d: Device or resource busy ``` The reason for the issue was found by @unhipzippo opensearch-project/OpenSearch#768 (comment) ❤️ > The "sed -i" is an attempt to modify the opensearch.yml file "in place" -- But according to the GNU sed documentation (https://www.gnu.org/software/sed/manual/sed.html#Command_002dLine-Options), "in-place" actually "does this by creating a temporary file and sending output to this file rather than to the standard output. ... the temporary file is renamed to the output file’s original name". > > I believe this rename would require changing the inode of the original file -- something that Docker volume mounts don't permit. Signed-off-by: Robin Böning <[email protected]>
|
Fixed in opensearch-project/opensearch-build#1130. |
|
Hi @peterzhuamazon , Is it possible to update the docker image in docker hub repository ? |
|
Saw this issue in Full error log: https://github.com/opensearch-project/helm-charts/actions/runs/7809681228/job/21301876373 But this passed in next run https://github.com/opensearch-project/helm-charts/actions/runs/7809681228/job/21302994506?pr=516. Is this flaky ? do we know where is this message in code |
|
@prudhvigodithi This message appears when opensearch.yml already contains at-least 1 setting related to security, i.e. |
|
Thanks @DarshitChanpura, what If I want to start the cluster (a brand new) with a pre-created |
|
Since helm charts use docker images which internally uses docker-entrypoint.sh script to start the container, there is way to skip demo config install completely if you want to use your own custom setup. You can do so by setting the env var If you try to do both, the demo setup will quit with "already configured" message. |
I'm attempting to setup OpenSearch and OpenSearch Dashboard, with Docker.
According to the documentation we can use a custom config: https://docs-beta.opensearch.org/docs/opensearch/install/docker/
However when following the steps in the documentation, the containers won't start properly. This happens with a custom config for OpenSearch or with a custom config for the OpenSearch Dashboard.
There are probably several issues here:
When running docker-compose for the first time (as per the documentation) two directories are created: custom-opensearch_dashboards.yml and custom-opensearch.yml
Clearly these should be files, instead of directories.
Even when manually creating the above .yml files, before starting Docker compose, the containers fail to start properly.
Probably due to the fact that the config files created with "touch" are empty. I'm not sure what to fill them with, which brings me to the 3th issue.
The documentation is unclear about the required contents of the custom config files.
I'd be looking for an example, including guidance on required and optional parameters (Ideally this should include a description of the various options, and guidance on recommended settings for production usage).
The text was updated successfully, but these errors were encountered: