-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Zipkin support of OpenSearch storage (add Docker images) #3766
Merged
Merged
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,70 @@ | ||
# | ||
# Copyright The OpenZipkin Authors | ||
# SPDX-License-Identifier: Apache-2.0 | ||
# | ||
|
||
# java_version is used for install and runtime layers of zipkin-opensearch2 | ||
# | ||
# Use latest version here: https://github.com/orgs/openzipkin/packages/container/package/java | ||
# This is defined in many places because Docker has no "env" script functionality unless you use | ||
# docker-compose: When updating, update everywhere. | ||
ARG java_version=21.0.2_p13 | ||
|
||
# We copy files from the context into a scratch container first to avoid a problem where docker and | ||
# docker-compose don't share layer hashes https://github.com/docker/compose/issues/883 normally. | ||
# COPY --from= works around the issue. | ||
FROM scratch as scratch | ||
|
||
COPY build-bin/docker/docker-healthcheck /docker-bin/ | ||
COPY docker/test-images/zipkin-opensearch2/start-opensearch /docker-bin/ | ||
COPY docker/test-images/zipkin-opensearch2/config/ /config/ | ||
|
||
FROM ghcr.io/openzipkin/java:${java_version} as install | ||
|
||
WORKDIR /install | ||
|
||
# Use latest 2.x version from https://opensearch.org/downloads.html | ||
# This is defined in many places because Docker has no "env" script functionality unless you use | ||
# docker-compose: When updating, update everywhere. | ||
ARG opensearch2_version=2.13.0 | ||
|
||
# Download only the OSS distribution (lacks X-Pack) | ||
RUN \ | ||
# Connection resets are frequent in GitHub Actions workflows \ | ||
wget --random-wait --tries=5 -qO- \ | ||
# We don't download bin scripts as we customize for reasons including BusyBox problems | ||
https://artifacts.opensearch.org/releases/bundle/opensearch/${opensearch2_version}/opensearch-${opensearch2_version}-linux-x64.tar.gz| tar xz \ | ||
--wildcards --strip=1 --exclude=jdk | ||
COPY --from=scratch /config/ ./config/ | ||
|
||
# Use a full Java distribution rather than adding test modules to the | ||
# production -jre base layer used by zipkin and zipkin-slim. | ||
FROM ghcr.io/openzipkin/java:${java_version} as zipkin-opensearch2 | ||
LABEL org.opencontainers.image.description="OpenSearch distribution on OpenJDK and Alpine Linux" | ||
ARG opensearch2_version=2.13.0 | ||
LABEL opensearch-version=$opensearch2_version | ||
|
||
# The full license is also included in the image at /opensearch/LICENSE.txt. | ||
LABEL org.opencontainers.image.licenses="Apache-2.0" | ||
|
||
# Add HEALTHCHECK and ENTRYPOINT scripts into the default search path | ||
COPY --from=scratch /docker-bin/* /usr/local/bin/ | ||
# We use start period of 30s to avoid marking the container unhealthy on slow or contended CI hosts | ||
HEALTHCHECK --interval=1s --start-period=30s --timeout=5s CMD ["docker-healthcheck"] | ||
ENTRYPOINT ["sh", "/usr/local/bin/start-opensearch"] | ||
|
||
# All content including binaries and logs write under WORKDIR | ||
ARG USER=opensearch | ||
WORKDIR /${USER} | ||
|
||
# Ensure the process doesn't run as root | ||
RUN adduser -g '' -h ${PWD} -D ${USER} | ||
USER ${USER} | ||
|
||
# Copy binaries and config we installed earlier | ||
COPY --from=install --chown=${USER} /install . | ||
|
||
# Use to set heap, trust store or other system properties. | ||
ENV OPENSEARCH_JAVA_OPTS="-Xms512m -Xmx512m -XX:+ExitOnOutOfMemoryError" | ||
ENV LIBFFI_TMPDIR=/tmp | ||
EXPOSE 9200 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
## zipkin-opensearch2 Docker image | ||
|
||
The `zipkin-opensearch2` testing image runs OpenSearch 2.x for [Elasticsearch storage](../../../zipkin-storage/elasticsearch) | ||
integration. | ||
|
||
To build `openzipkin/zipkin-opensearch2:test`, from the top-level of the repository, run: | ||
```bash | ||
$ DOCKER_FILE=docker/test-images/zipkin-opensearch2/Dockerfile build-bin/docker/docker_build openzipkin/zipkin-opensearch2:test | ||
``` | ||
|
||
You can use the env variable `OPENSEARCH_JAVA_OPTS` to change settings such as heap size for OpenSearch. | ||
|
||
#### Host setup | ||
|
||
OpenSearch is [strict](https://github.com/docker-library/docs/tree/master/elasticsearch#host-setup) | ||
about virtual memory. You will need to adjust accordingly (especially if you notice OpenSearch crash!) | ||
|
||
```bash | ||
# If docker is running on your host machine, adjust the kernel setting directly | ||
$ sudo sysctl -w vm.max_map_count=262144 | ||
|
||
# If using docker-machine/Docker Toolbox/Boot2Docker, remotely adjust the same | ||
$ docker-machine ssh default "sudo sysctl -w vm.max_map_count=262144" | ||
|
||
# If using colima, it is similar as well | ||
$ colima ssh "sudo sysctl -w vm.max_map_count=262144" | ||
``` |
14 changes: 14 additions & 0 deletions
14
docker/test-images/zipkin-opensearch2/config/log4j2.properties
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
# | ||
# Copyright The OpenZipkin Authors | ||
# SPDX-License-Identifier: Apache-2.0 | ||
# | ||
|
||
status = error | ||
|
||
appender.console.type = Console | ||
appender.console.name = console | ||
appender.console.layout.type = PatternLayout | ||
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n | ||
|
||
rootLogger.level = info | ||
rootLogger.appenderRef.console.ref = console |
21 changes: 21 additions & 0 deletions
21
docker/test-images/zipkin-opensearch2/config/opensearch.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
# | ||
# Copyright The OpenZipkin Authors | ||
# SPDX-License-Identifier: Apache-2.0 | ||
# | ||
|
||
cluster.name: "docker-cluster" | ||
network.host: 0.0.0.0 | ||
node.name: zipkin-opensearch | ||
|
||
# Enable development mode and disable bootstrap checks | ||
# See https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html | ||
discovery.type: single-node | ||
# Avoid deprecation errors: as of 2.x the only accepted value is true. | ||
cluster.routing.allocation.disk.watermark.enable_for_single_data_node: true | ||
|
||
# This is a test image, so we are not afraid to delete all indices. Avoids: | ||
# Wildcard expressions or all indices are not allowed | ||
action.destructive_requires_name: false | ||
|
||
# Disable security | ||
plugins.security.disabled: true |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
#!/bin/sh | ||
# | ||
# Copyright The OpenZipkin Authors | ||
# SPDX-License-Identifier: Apache-2.0 | ||
# | ||
|
||
# ENTRYPOINT script that starts OpenSearch | ||
# | ||
# This intentionally locates config using the current working directory, in order to consolidate | ||
# Dockerfile instructions to WORKDIR | ||
set -eu | ||
|
||
# Configure the Docker HEALTHCHECK | ||
export HEALTHCHECK_PORT=9200 | ||
export HEALTHCHECK_PATH=/_cluster/health | ||
|
||
# Set tmp folder | ||
OPENSEARCH_TMPDIR=`java -cp 'classes:lib/*' org.opensearch.tools.launchers.TempDirectory` | ||
reta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
# The JVM options parser produces the final JVM options to start OpenSearch. | ||
# It does this by incorporating JVM options in the following way: | ||
# - first, system JVM options are applied (these are hardcoded options in the | ||
# parser) | ||
# - second, JVM options are read from jvm.options and jvm.options.d/*.options | ||
# - third, JVM options from OPENSEARCH_JAVA_OPTS are applied | ||
# - fourth, ergonomic JVM options are applied | ||
OPENSEARCH_JAVA_OPTS=`export OPENSEARCH_TMPDIR; java -cp 'classes:lib/*' org.opensearch.tools.launchers.JvmOptionsParser "$PWD/config"` | ||
reta marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
# -cp 'classes:lib/*' allows layers to patch the image without packaging or overwriting jars | ||
# We allow security manager (via flag to prevent JRE 21 crash) as OpenSearch.main needs it. | ||
exec java -cp 'classes:lib/*' ${OPENSEARCH_JAVA_OPTS} \ | ||
-Djava.security.manager=allow \ | ||
-Djava.io.tmpdir=/tmp \ | ||
-Dlog4j2.disable.jmx=true \ | ||
-Dopensearch.path.home=$PWD \ | ||
-Dopensearch.path.conf=$PWD/config \ | ||
-Dopensearch.distribution.type=docker \ | ||
-Dopensearch.bundled_jdk=false \ | ||
org.opensearch.bootstrap.OpenSearch "$@" |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we are setting this to /tmp as an env variable in dockerfile to /tmp. do you want to always overwrite it like this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
or does TempDirectory consider an existing OPENSEARCH_TMPDIR.. if so, maybe mention in the comment like below on OPENSEARCH_JAVA_OPTS
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good question, so
TempDirectory
usesjava.io.tmpdir
and pre-creates theopensearch
folder there