Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE REQ] promoting operation-level policies in SOAP APIs #399

Closed
AmitAnajwala opened this issue Oct 13, 2023 · 21 comments
Closed

[FEATURE REQ] promoting operation-level policies in SOAP APIs #399

AmitAnajwala opened this issue Oct 13, 2023 · 21 comments

Comments

@AmitAnajwala
Copy link

Please describe the feature.

Hi,
When we extract an API having policy at operation level, it creates a folder with OperationId.
It should create folder with Operation name.

Issue:
I extracted API having operation level policy using extractor pipeline.
Then I deleted that API from API Management.
I tried to create API with publisher pipeline in same API Management instance.

I got below error:
System.Net.Http.HttpRequestException: HTTP request to URI
https://management.azure.com/subscriptions/nn/resourceGroups/rg-001/providers/Microsoft.ApiManagement/service/apim-dev-int001-pro/apis/FC-AdminFeeService-Interface/operations/64f70d4e46346101d4e97723/policies/policy?api-version=2022-04-01-preview&format=rawxml
failed with status code 400. Content is '{"error":{"code":"ValidationError","message":"Entity with specified identifier not found","details":null}}'.

Let me know if any additional information is required.

Thanks,
Amit Anajwala.
@github-actions
Copy link 

  Thank you for opening this issue! Please be patient while we will look into it and get back to you as this is an open source project. In the meantime make sure you take a look at the [closed issues](https://github.com/Azure/apiops/issues?q=is%3Aissue+is%3Aclosed) in case your question has already been answered. Don't forget to provide any additional information if needed (e.g. scrubbed logs, detailed feature requests,etc.).
  Whenever it's feasible, please don't hesitate to send a Pull Request (PR) our way. We'd greatly appreciate it, and we'll gladly assess and incorporate your changes.

@waelkdouh
Copy link
Contributor

Again seems like you need to override using the configuration file.

@AmitAnajwala
Copy link
Author

Hi Wael,
I tried to override by adding configuration.dev.yaml file with following content and referred configuration.dev.yaml file from publisher pipeline.

apimServiceName:
apis:

  • name: proxy-dp-thdochandler-soap

If I delete the proxy-dp-thdochandler-soap from above APIM instance and then try to create it using publisher pipeline, I am getting below error:
'{"error":{"code":"ValidationError","message":"Entity with specified identifier not found","details":null}}'.

Once API is re-created, operations will have new operation ids.
And it won’t be able to apply Operation policy with old operation id.

@waelkdouh
Copy link
Contributor

@guythetechie thoughts on this?

@guythetechie
Copy link
Contributor

We use IDs instead of display names to better align with the APIM REST API. The REST API expects IDs in the URL for APIs, operations, backends, etc.

Older versions of ApiOps used names, but it created all sorts of issues. For example:

  • Every time we read a folder name, we had to look up what ID corresponded to the folder name before running operations on it. This made the code much more complex and brittle.
  • Many customers had characters in display names that were not supported folder names. For instance, d#s?382<> is a perfectly fine display name, but you cannot create a folder with that name.

@guythetechie
Copy link
Contributor

Once API is re-created, operations will have new operation ids.
And it won’t be able to apply Operation policy with old operation id.

Why will it have new operation ids? If you're just publishing what was extracted, the publisher should publish using the original operation IDs.

  1. Run extractor. You now have the following structure in Git:
    • artifacts/apis/myapi/operations/operationId1/policy.xml: policy file for operationId1
    • artifacts/apis/myapi/specification.xxx: specification file that contains operationId1
  2. Delete myapi from APIM instance in Azure portal.
  3. Run publisher. It should recreate the API, its operations (including operationId1), and a policy for operationId1 based on the files extracted in step 1.

@AmitAnajwala
Copy link
Author

AmitAnajwala commented Oct 17, 2023 via email

@TKProlifics
Copy link

I work with Amit and I would like to ask a related question.

I guessed the reasons (both of them) for using ids in the operation folder names. However, it seems to me that APIOps is limited to working with a single instance of APIM. We would like to do this:

  • Extract the APIs from our DEV APIM instance and commit them to our Git repo
  • Use APIOps for future changes in the DEV environment
  • Promote our API definitions in the next environment by publishing the same definitions from the same repo into a new (empty) APIM instance.

I do understand that this was not the main goal of the APIOps tool. APIOps is a great time-saver for many teams. However, promoting code between environments is a really important activity and we are almost completely blocked on this currently.

Can the APIOps team offer any advice on how to promote an entire set of APIs, Backends, Named Values etc into a new empty APIM instance? We really liked the idea of using APIOps, and we thought we had a way forward until we hit the issue with the operation-level policies.

If the answer is 'feel free to submit a pull request' then we might be open to that, but only if it can done in a way that is consistent with the project roadmap.

@waelkdouh
Copy link
Contributor

waelkdouh commented Oct 19, 2023 •
edited
Loading

I work with Amit and I would like to ask a related question.

I guessed the reasons (both of them) for using ids in the operation folder names. However, it seems to me that APIOps is limited to working with a single instance of APIM. We would like to do this:

  • Extract the APIs from our DEV APIM instance and commit them to our Git repo

  • Use APIOps for future changes in the DEV environment

  • Promote our API definitions in the next environment by publishing the same definitions from the same repo into a new (empty) APIM instance.

I do understand that this was not the main goal of the APIOps tool. APIOps is a great time-saver for many teams. However, promoting code between environments is a really important activity and we are almost completely blocked on this currently.

Can the APIOps team offer any advice on how to promote an entire set of APIs, Backends, Named Values etc into a new empty APIM instance? We really liked the idea of using APIOps, and we thought we had a way forward until we hit the issue with the operation-level policies.

If the answer is 'feel free to submit a pull request' then we might be open to that, but only if it can done in a way that is consistent with the project roadmap.

The intent of APIops is to actually allow you to promote across different APIM instances. We offer a configuration system to override values that you would like to change in the target APIM instance (e.g different backend url for an operation). Seems to me that our tool already offers what you're asking for here.

@guythetechie
Copy link
Contributor

@AmitAnajwala - can you zip and attach here the contents of the artifacts/apis folder after running the extractor? Please scrub any sensitive information. You can also create a private GitHub repo with your artifacts and invite me to it.

@TKProlifics
Copy link

@guythetechie: Thanks for the very prompt response.

It's good to hear that we're not so far off the map as I feared. However, I still don't know how to use APIOps to do this:

  • Create an API which has multiple operations, each operation having its own set of policies
  • Publish that API into an APIM instance that has never hosted any version of that API

Right now we have a set of 23 APIs each with up to 4 operations and the only way to promote them into a new environment is to manually copy each operation's policy XML via the APIM portal. This is time-consuming and error-prone, as I'm sure you can imagine.

Can you advise?

@guythetechie
Copy link
Contributor

@TKProlifics - I assume you're creating your initial APIs using the Azure portal. Here is the happy path I would expect:

  1. You create your APIs in the dev portal, assign policies to operations, etc.
  2. Once you're happy with the dev state, you run the extractor.
  3. The extractor generates this file structure:
  • artifacts/apis/api1/apiInformation.json
  • artifacts/apis/api1/specification.xxx. This specification file should contain all your operations, including the APIM-generated operation IDs. For REST APIs, the operation ID should be fined in paths.somePath.httpMethod.operationId.
openapi: 3.0.1
info:
  title: Swagger API Team2
  description: A sample API that uses a petstore as an example to demonstrate features in the OpenAPI 3.0 specification
  termsOfService: http://swagger.io/terms/
  contact:
    name: Swagger API Team
    url: http://swagger.io/
    email: [email protected]
  license:
    name: Apache 2.0
    url: https://www.apache.org/licenses/LICENSE-2.0.html
  version: '1.0'
servers:
  - url: https://apiopskdalfddev.azure-api.net/3223
paths:
  /pets:
    get:
      summary: findPets
      description: "Returns all pets from the system that the user has access to\nNam sed condimentum est. Maecenas tempor sagittis sapien, nec rhoncus sem sagittis sit amet. Aenean at gravida augue, ac iaculis sem. Curabitur odio lorem, ornare eget elementum nec, cursus id lectus. Duis mi turpis, pulvinar ac eros ac, tincidunt varius justo. In hac habitasse platea dictumst. Integer at adipiscing ante, a sagittis ligula. Aenean pharetra tempor ante molestie imperdiet. Vivamus id aliquam diam. Cras quis velit non tortor eleifend sagittis. Praesent at enim pharetra urna volutpat venenatis eget eget mauris. In eleifend fermentum facilisis. Praesent enim enim, gravida ac sodales sed, placerat id erat. Suspendisse lacus dolor, consectetur non augue vel, vehicula interdum libero. Morbi euismod sagittis libero sed lacinia.\n\nSed tempus felis lobortis leo pulvinar rutrum. Nam mattis velit nisl, eu condimentum ligula luctus nec. Phasellus semper velit eget aliquet faucibus. In a mattis elit. Phasellus vel urna viverra, condimentum lorem id, rhoncus nibh. Ut pellentesque posuere elementum. Sed a varius odio. Morbi rhoncus ligula libero, vel eleifend nunc tristique vitae. Fusce et sem dui. Aenean nec scelerisque tortor. Fusce malesuada accumsan magna vel tempus. Quisque mollis felis eu dolor tristique, sit amet auctor felis gravida. Sed libero lorem, molestie sed nisl in, accumsan tempor nisi. Fusce sollicitudin massa ut lacinia mattis. Sed vel eleifend lorem. Pellentesque vitae felis pretium, pulvinar elit eu, euismod sapien.\n"
      operationId: findPets0 // Here is your operation ID
  • artifacts/apis/api1/policy.xml. This should be your API-level policy, if you use one.
  • artifacts/apis/api1/operations/operationId1/policy.xml. This policy is scoped to operationId1. Again, operationId1 should be defined in your extracted specification.xxx.
  1. You run the publisher on your empty PROD instance (having updated your configuration accordingly).
  2. The publisher creates the extracted APIs with the operation IDs specified in your specification file. It then assigns operation policies.

I can think of two reasons why things wouldn't work this way:

  • A bug with the publisher (certainly possible).
  • The operation IDs in your folder path don't match the IDs in the specification file.

Seeing the files generated by the extractor would help me narrow down the issue.

@AmitAnajwala
Copy link
Author

Hi @waelkdouh and @guythetechie,
I am attaching APIs extracted using APIOps extractor pipeline after removing sensitive information from it.
APIs_Extract.zip

Thanks,
Amit.

@AmitAnajwala
Copy link
Author

@TKProlifics - I assume you're creating your initial APIs using the Azure portal. Here is the happy path I would expect:

  1. You create your APIs in the dev portal, assign policies to operations, etc.
  2. Once you're happy with the dev state, you run the extractor.
  3. The extractor generates this file structure:
  • artifacts/apis/api1/apiInformation.json
  • artifacts/apis/api1/specification.xxx. This specification file should contain all your operations, including the APIM-generated operation IDs. For REST APIs, the operation ID should be fined in paths.somePath.httpMethod.operationId.
openapi: 3.0.1
info:
  title: Swagger API Team2
  description: A sample API that uses a petstore as an example to demonstrate features in the OpenAPI 3.0 specification
  termsOfService: http://swagger.io/terms/
  contact:
    name: Swagger API Team
    url: http://swagger.io/
    email: [email protected]
  license:
    name: Apache 2.0
    url: https://www.apache.org/licenses/LICENSE-2.0.html
  version: '1.0'
servers:
  - url: https://apiopskdalfddev.azure-api.net/3223
paths:
  /pets:
    get:
      summary: findPets
      description: "Returns all pets from the system that the user has access to\nNam sed condimentum est. Maecenas tempor sagittis sapien, nec rhoncus sem sagittis sit amet. Aenean at gravida augue, ac iaculis sem. Curabitur odio lorem, ornare eget elementum nec, cursus id lectus. Duis mi turpis, pulvinar ac eros ac, tincidunt varius justo. In hac habitasse platea dictumst. Integer at adipiscing ante, a sagittis ligula. Aenean pharetra tempor ante molestie imperdiet. Vivamus id aliquam diam. Cras quis velit non tortor eleifend sagittis. Praesent at enim pharetra urna volutpat venenatis eget eget mauris. In eleifend fermentum facilisis. Praesent enim enim, gravida ac sodales sed, placerat id erat. Suspendisse lacus dolor, consectetur non augue vel, vehicula interdum libero. Morbi euismod sagittis libero sed lacinia.\n\nSed tempus felis lobortis leo pulvinar rutrum. Nam mattis velit nisl, eu condimentum ligula luctus nec. Phasellus semper velit eget aliquet faucibus. In a mattis elit. Phasellus vel urna viverra, condimentum lorem id, rhoncus nibh. Ut pellentesque posuere elementum. Sed a varius odio. Morbi rhoncus ligula libero, vel eleifend nunc tristique vitae. Fusce et sem dui. Aenean nec scelerisque tortor. Fusce malesuada accumsan magna vel tempus. Quisque mollis felis eu dolor tristique, sit amet auctor felis gravida. Sed libero lorem, molestie sed nisl in, accumsan tempor nisi. Fusce sollicitudin massa ut lacinia mattis. Sed vel eleifend lorem. Pellentesque vitae felis pretium, pulvinar elit eu, euismod sapien.\n"
      operationId: findPets0 // Here is your operation ID
  • artifacts/apis/api1/policy.xml. This should be your API-level policy, if you use one.
  • artifacts/apis/api1/operations/operationId1/policy.xml. This policy is scoped to operationId1. Again, operationId1 should be defined in your extracted specification.xxx.
  1. You run the publisher on your empty PROD instance (having updated your configuration accordingly).
  2. The publisher creates the extracted APIs with the operation IDs specified in your specification file. It then assigns operation policies.

I can think of two reasons why things wouldn't work this way:

  • A bug with the publisher (certainly possible).
  • The operation IDs in your folder path don't match the IDs in the specification file.

Seeing the files generated by the extractor would help me narrow down the issue.

Hi,
I think issue is due to second reason in above reply. I checked extracted specification.wsdl file and operation ids are not present in that file.

If I manually export the API with OpenAPI v3(YAML) or OpenAPI v3(JSON) format, then Operation Ids are present in those files.

Regards,
Amit,

@guythetechie
Copy link
Contributor

@AmitAnajwala - you're correct. APIM doesn't export operation IDs in WSDL specification files (unlike OpenAPI specification files).

Since the specification files have no operation IDs, APIM generates new ones when the publisher runs. These don't match the original IDs, so it fails when applying operation-level policies.

I guess you found an unsupported ApiOps scenario: promoting operation-level policies in SOAP APIs. I don't have any clean solutions off the top of my head, unfortunately. It would work if you converted to OpenAPI specs, but I certainly understand why you would want to keep using SOAP.

@waelkdouh waelkdouh changed the title [FEATURE REQ] [FEATURE REQ] promoting operation-level policies in SOAP APIs Oct 20, 2023
@TKProlifics
Copy link

Thanks @guythetechie - that makes sense, although it sounds like bad news for our project. You're correct that we have no choice about using SOAP/WSDL APIs.

If I understand correctly, the set of data written by the exporter does not contain enough information to recreate the original APIs. So this is not (only) a limitation in the publisher. The publisher uses the exported operation ids (held as folder names in the filesystem), but there is no way to link those operation ids to the correct operation names in the WSDL specification.

Questions:

  • Is there any way to resolve this by changing only the publisher? I don't think there is but I just want to check my understanding.
  • If the exporter was modified to put the operation name somewhere in the operation folder, would that offer a way forward? I'm thinking that the publisher could then query the newly-minted API and find out its new operation ids. Then the publisher could use the operation names in the exported data to find the correct operation id for each operation. Not sure whether that would introduce new sequencing requirements in the publisher though.
  • Is there any value in using operation names as folder names when exporting WSDL APIs? I know there were good reasons for moving away from this approach. Just wondering whether a limited use of this idea could offer a solution for this problem?

@guythetechie
Copy link
Contributor

If I understand correctly, the set of data written by the exporter does not contain enough information to recreate the original APIs. So this is not (only) a limitation in the publisher. The publisher uses the exported operation ids (held as folder names in the filesystem), but there is no way to link those operation ids to the correct operation names in the WSDL specification.

That's correct. Using operation names in the folder will not help you though, because the REST API calls rely on IDs. To create an operation policy, you make PUT request to https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.ApiManagement/service/{serviceName}/apis/{apiId}/operations/{operationId}/policies/policy?api-version=2022-08-01. Note that the URL expects the operation ID, not its name. Any solution will have to include the ID.

Your second bullet point might be viable, except that we really want to make sure that IDs are aligned across environments. This is a key tenet for many things to work, including configuration overrides. We could make an exception for SOAP operations, but things will get messy...

@guythetechie
Copy link
Contributor

I also noticed that APIM allows you to export SOAP APIs into OpenAPI specification files, which will include the correct operation IDs. Not sure if that helps, but good to know.

@TKProlifics
Copy link

TKProlifics commented Oct 21, 2023 •
edited
Loading

Summary of what we know so far...

  • APIOps assumes that the API specification file will contain operation ids.
  • Extract / publish of an API using OpenAPIv3 preserves the operation ids.
  • Extract / publish of an API using WSDL does not preserve the operation ids unless the API already exists in the target APIM instance.

Because of the above, APIOps cannot promote a SOAP API into a new instance of APIM if it has operation-level policies.

The operation ids are considered an integral part of the API specification and must be preserved when an API is extracted/re-published. This rules out any solution that allows APIOps to cope with environment-specific operation ids (so no point in pursuing the idea of using operation names as folder names).

We need a short-term workaround for our project. I wonder whether this would work:

  • Export SOAP APIs as OpenAPIv3 and store the exported YAML in our repo alongside the extracted API definitions.
  • Import the API from the OpenAPI spec before publishing the API using APIOps. This will definitely create an API with the correct operation ids. Hopefully it will also preserve the essential SOAP-ness of the API and will not interfere with the APIOps publish (which will attempt to import the API from the WSDL).

We will test out this approach, but we would appreciate any advice/feedback on potential issues.

@guythetechie guythetechie mentioned this issue Oct 31, 2023
Closed
@mirzos
Copy link

mirzos commented Dec 6, 2023

I am having the same issue. Looks like I will have to do the workaround which Devs may not appreciate :(

@guythetechie guythetechie mentioned this issue Feb 7, 2024
Closed
@mmmbai
Copy link

mmmbai commented Feb 7, 2024

This is very critical feature. Half of our APIs are SOAP based and have operation level policy, Template, Query, Headers, Request and Response. Without this feature we can not use APIOPS tools to extract/publish to another environment. @waelkdouh @guythetechie Do you have any plan to implement this feature?

@guythetechie guythetechie mentioned this issue Apr 30, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@waelkdouh @guythetechie @mmmbai @AmitAnajwala @TKProlifics @mirzos