accessing ImageStream from private repository with auth delegated to a different server requires creating two secrets instead of one #9584
Comments
|
@jperville have you checked our docs at https://docs.openshift.org/latest/dev_guide/managing_images.html#allowing-pods-to-reference-images-from-other-secured-registries I think that topic fits your use case. If not lemme know where you're having problems, I'll help. |
|
Yes @soltysh , I know how to reference docker images on other registries in my pod definitions, however I would really like my pods to be automatically redeployed if their docker images are updated; to me this is the point of the ImageStream feature. If there is another method, how to use use it with |
|
@jperville my bad, I wanted to point you to https://docs.openshift.org/latest/dev_guide/managing_images.html#private-registries apologies |
|
Thanks @soltysh . I have created the secret, however I see no way with ImageStream object or with the Assuming I am in the 'project' project, here are my secrets: $ oc secrets new-dockercfg registry-login --docker-server=registry.example.com \
--docker-username=someuser --docker-password=somepassword [email protected]
$ oc secrets add serviceaccount/default secrets/registry-login --for=pull
$ oc secrets add serviceaccount/builder secrets/registry-login
$ oc get secrets/registry-login
NAME TYPE DATA AGE
registry-login kubernetes.io/dockercfg 1 26sI want to create the following image Stream: $ cat <<EOF | tee is.yaml
apiVersion: v1
kind: ImageStream
metadata:
name: myapp
spec:
tags:
- from:
kind: DockerImage
name: registry.internal.example.com/myapp:latest
name: latest
EOF
$ oc create -f is.yaml
$ oc get is
NAME DOCKER REPO TAGS UPDATED
myapp 10.10.87.191:5000/default/myapp latest If I $ oc import-image myapp
oc describe is myapp
Name: myapp
Created: About a minute ago
Labels: <none>
Annotations: openshift.io/image.dockerRepositoryCheck=2016-06-29T12:56:33Z
Docker Pull Spec: 10.10.87.191:5000/default/myapp
Tag Spec Created PullSpec Image
latest registry.si.perfect-memory.com/core/exchange-manager:latest About a minute ago import failed: Internal error occurred: Get https://registry.si.per...
$ oc get is/myapp -o yaml
apiVersion: v1
kind: ImageStream
metadata:
annotations:
openshift.io/image.dockerRepositoryCheck: 2016-06-29T12:56:33Z
creationTimestamp: 2016-06-29T12:56:32Z
generation: 2
name: myapp
namespace: default
resourceVersion: "639"
selfLink: /oapi/v1/namespaces/default/imagestreams/myapp
uid: e75ff849-3df8-11e6-8434-080027c23ce5
spec:
tags:
- annotations: null
from:
kind: DockerImage
name: registry.internal.example.com/myapp:latest
generation: 2
importPolicy: {}
name: latest
status:
dockerImageRepository: 10.10.87.191:5000/default/myapp
tags:
- conditions:
- generation: 2
lastTransitionTime: 2016-06-29T12:56:33Z
message: 'Internal error occurred: Get https://registry.internal.example.com/myapp/manifests/latest:
token auth attempt for registry: https://gitlab.example.com/jwt/auth?scope=repository%3Amyapp%3Apull&service=container_registry
request failed with status: 403 Forbidden'
reason: InternalError
status: "False"
type: ImportSuccess
items: null
tag: latestNote that in the error above, the registry public endpoint is https://registry.internal.example.com which delegates authentication by redirecting to https://gitlab.example.com (a different domain name but still part of the same environment); that authentication fails with 403 . From reading https://docs.openshift.org/latest/dev_guide/managing_images.html#private-registries I was expecting the secret to be automatically used to authenticate to the external registry, but it seems it is not used. If the secret is to be associated with a specific serviceaccount so that all image stream pulls use that secret automatically, can it be documented somewhere? |
|
Your use-case is different from what I've been testing/working on so far. Generally all this works like this: during importing image the importer will match against all secrets, in the same project the ImageStream is defined in, based on the hostname (see here). Having said that you start with changing the secret name so that it matches the registry uri, or rather auth endpoint uri in your case. If either won't help attach me log from import operation at level 8, I'll have a look, it might be that your use-case requires more work. |
|
I have tried again naming the secret after the registry name (eg. `oc get secrets/registry.internal.example.com) without success. You will find below a trace of |
|
Sorry for not being clear, I need logs from openshift master, since the |
|
Have no special logs appear on the master after I run the Here is how processes look on the master VM: [root@master ~]# ps aux | grep origin
root 20908 0.3 2.4 578588 46580 ? Ssl 12:37 0:25 /usr/bin/openshift start node --config=/etc/origin/node/node-config.yaml --loglevel=2
root 44545 3.4 9.3 834148 176644 ? Ssl 14:48 0:13 /usr/bin/openshift start master --config=/etc/origin/master/master-config.yaml --loglevel=8Notice the loglevel=8 on the master. I get some verbose messages in |
|
You should see the logs like this one in the log. Without them I won't be able to dig what's going on. |
|
I'm sorry but I really don't see anything in the origin-master log after the booting messages (the last ones being about listing service account tokens). In neither the origin-master nor on the origin-node, both running at loglevel=8, anything shows when I try my |
|
@jperville Assuming those processes were started by systemd units the journalctl command used should be fine. |
|
@sdodson yes I have set the loglevel in /etc/sysconfig, as shown by the output of Still nothing interesting in the origin-master log when trying to |
|
@jperville thanks for confirming, @soltysh had asked me on irc how to properly configure loglevel for systemd units, sorry if that comment was a bit out of context. However I have an idea, is the certificate for registry.si.perfect-memory.com trusted by the master host? If not you'll need to add the insecure annotation to the image stream, I think the error message for that condition is more clear than a 500 error, but I may be mistaken. See https://docs.openshift.org/latest/dev_guide/managing_images.html#insecure-registries |
|
I will try tomorrow the insecure annotation; for information the private registry is using a regular domain-validated certificate which poses no problem with |
|
After trying all combinations of insecureRegistry and naming the secret after the registry URL and the registry authentication endpoint, I am proud to announce success with the following combination:
Thank you very much for the help @soltysh and @sdodson. Could you add a note in the documentation about the way secrets have to be named when working with authenticated image streams? This was not easy to figure out, I didn't even know about having to name the secret in a specific manner in the first place. Sample working image stream definition: apiVersion: v1
kind: ImageStream
metadata:
name: myapp
spec:
tags:
- from:
kind: DockerImage
name: registry.internal.example.com/myapp:latest
name: latestCommand-line to make a suitably named secret and add it to the default service accounts: for name in gitlab.internal.example.com ; do
oc secrets new-dockercfg ${name } --docker-server=${name } \
--docker-username=someuser --docker-password=somepassword [email protected]
oc secrets add serviceaccount/default secrets/${name } --for=pull
oc secrets add serviceaccount/builder secrets/${name }
donePS: found nothing relevant in the master log about image streams, even at loglevel8. |
|
After successfully importing my authenticated ImageStream, I finally can Surprisingly, after In the end, I had to duplicate my secret and add it to the default and deployer service accounts twice to get my application working. For now, this is OK, I have a workaround, but it would be nice to be able to have a single place to declare which secret to use to pull which docker image, which would work both for ImageStream and for instanciating pods. |
Actually after re-testing and re-checking the code the key is created from the value passed to Secondly, I don't understand why you need to update both deployer and default serviceaccount. Since your image is used only for deployment, default should be sufficient. Again, I just verified that. The only thing that comes to my mind and would be worth checking would be the fact that your auth is done against different server, but that shouldn't be a problem, imho. The logs I was looking into for look like that: And these should be available at level 5, already. I'm not sure why you're not seeing those :( Finally, I'd really appreciate if you could give it a try once again with above details. |
|
I have done more testing @soltysh and indeed the name of secret can be anything, as long as there is two secrets:
Yes, my authentication is done versus a different server (using the new integrated container registry from gitlab), hence the need for two secrets because of the two endpoints. I just wish there was a way to DRY that but for the moment duplicating the secrets is acceptable. Also, I have tried adding the secrets only to the default service account (and not to the deployer), and it still works. Thanks for pointing that to me. |
soltysh
changed the title
|
@jperville thanks for the detailed info, I've changed the tile of the bug and recategorized this a bit, since it seems like a bug that we should address. I'll leave this open and will try to tackle that. Thanks a lot for your report and patience! |
|
I wonder if this is another flavor of https://bugzilla.redhat.com/show_bug.cgi?id=1476330 in which the secret k8s creates is not the right format for artifactory. @zonArt Can you try the workaround suggested in https://bugzilla.redhat.com/show_bug.cgi?id=1476330#c1 ? |
|
Still the same (but not exactly). Is it a problem if different kind of secrets are linked to a serviceaccount ? Notice that the |
|
@miminar maybe you're right somehow, I just tried your suggestion and here is what I got: Is that expected ? Should I add anything special in my secret to handle that ? |
|
One more thing I wanted to share is that with my existing "artifactory" secret, I have no issue pulling images within a DeploymentConfig referencing the full path to the registry (without going through an ImageStream) |
|
you definitely need to specify the --from as "registry.hostname.com/namespace/repositoryname" (the same way you'd specify the image if you were doing a "docker pull", but leave off the tag. is |
|
yes it is the value, here is a DC which is working fine: |
|
@miminar well i'm out of ideas... any known issues in v1.5 (the server level @zonArt is running) w/ respect to imagestream import? Any logging we can enable to ensure the secret is being used during import? @zonArt regarding why your deploymentconfig works, is there any chance the image already existed on the node? (in which case it didn't actually pull it using the secret) |
|
@bparees not at all it has always been working this way and as I said, when the registry is accessed/required from let say "directly" (with spec.containers.image for instance), I never had any problem pulling it. It's just recently that I tried to used I really think that the secret is absolutely not taken into account as when I check our Artifactory server's logs, the GET request is made with Anonymous user. If you can point me to other kind of logs I can look at, I'll provide you details |
|
@zonArt From your prior comment:
Looks inauthentic :-)
I believe the port part prevents the keyring from matching the credentials. Can you please add the entry with the port to your dockerconfig secret? Or modify your proxy/lb in front of your registry to send port-less location? |
|
@miminar I added port and protocol and nothing changed |
|
One more question: |
|
I'm not exactly sure what you're asking, but the secret that you use to import the image should also be usable to push to the source registry (ie if you reference the secret in a buildconfig that has your source registry as the output image target). |
|
Hi, Regarding the the possibility to push to an Regarding the pull secret, I'm still not able to get the tags from the |
|
@bparees yes it is, having this BC: makes the build success without any issue (thanks to the |
|
Here is the info: (I see nothing really relevant though) Even with loglevel 8: |
|
@zonArt I'm sorry, the log from the master service needs to be captured from the time of import with |
|
@miminar Mmmmh ok, where are these logs located ? I'm having quite a hard time to figure it out, looks like everything gets spread in daemon.log, is that possible ? And there is nothing in daemon.log related to import-image |
|
@zonArt do you run the cluster containerized? If yes, you need to |
|
@miminar thanks, here is the linked log output: Still not sure to find any interesting info, except on the fact it tries to reach the inexistent internal registry (which doesn't really help by the way) |
|
It case it may interest anybody: I found the solution in RedHat KB: Thanks again for your support |
This is actually working, thanks |
|
@jperville jperville thanks for posting the solution! This works for me also (y) |
|
Thanks @zonArt , this is the only solution that works for me! |
|
For the convenience of people finding this through a search engine, here is a condensed answer. Thank you to everyone above for the excellent guidance. Suppose you want an ImageStream that references a secured external registry such as GitLab, and the auth for said registry is done through a different hostname e.g. gitlab.com rather than registry.gitlab.com. The following has been tested in OpenShift Online 4.2. First we find out the exact URI of the auth server with: We will therefore use Create secretHave a dockerconfig.json file with contents like: Then do: Important: It seems necessary to name it Set default service account to use this secret: Verify: Create the image streamYou should now have a working ImageStream. |
|
confirmed that the solution from @3dbrows works for registry.digitalocean.com as well:
Then modify ~/.docker/config.json to look like this: Then run: Setup your image stream as shown above, and it should work |
I want to declare an ImageStream that tracks an external docker image located on an authenticated docker registry.
My use-case is:
So far, I have found no way to declare an ImageStream which uses a secret to pull the image.
I have found 2 workarounds:
I have also tried without success to run
docker loginon the Openshift hosts, in the hope that the image puller would run as same user and implicitly use my dockercfg but no success.Version
Steps To Reproduce
Run the following commands (after adjusting docker image source to point to an authenticated docker registry).
Current Result
Image fails to import because the image puller does not use any secret.
Expected Result
ImageStream should provide a way to specify a secret to use when pulling the image (with a mechanism similar to pullSecret in BuildConfig dockerStrategy).
The text was updated successfully, but these errors were encountered: