Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Custom Backend for APIs #3007

Closed
BLasan opened this issue Jul 15, 2024 · 10 comments · May be fixed by wso2/product-apim#13574
Closed

Add Custom Backend for APIs #3007

BLasan opened this issue Jul 15, 2024 · 10 comments · May be fixed by wso2/product-apim#13574

Comments

@BLasan
Copy link

BLasan commented Jul 15, 2024

Problem

Some API developers want to use connectors within the api manager to implement mediation logic as they do not want to use an additional MI deployment if their use cases are simple. Within the API Manager, currently, we do not have a construct that will allow us to facilitate this requirement.

Solution

Introduce a way to define a Custom Backend for an API which gets added to the api.xml file of the API.

Affected Component

APIM

Version

No response

Implementation

No response

Related Issues

No response

Suggested Labels

No response

@BLasan BLasan self-assigned this Jul 15, 2024
@RakhithaRR RakhithaRR added this to the APIM 4.4.0 milestone Jul 22, 2024
@BLasan BLasan changed the title Support Broker APIS through Gateway Add Custom Backend for APIs Sep 3, 2024
@BLasan
Copy link
Author

BLasan commented Sep 3, 2024

Please find the below attachment related to the issue.
https://docs.google.com/document/d/1DxjI4f-XIwvr414PkT0Mo7C4vuJT_BCGzCidEKwSdF0/edit

@BLasan
Copy link
Author

BLasan commented Sep 3, 2024

[Update 1]

As per the initial discussion, following was the proposed solution.

  1. Introduce a new tab under Endpoints called Custom Sequence to upload a sequence.
  2. Implement a new REST API to update the API with Custom Sequence provided. Following is the REST API format.
      requestBody:
        content:
          multipart/form-data:
            schema:
              properties:
                sequence:
                  type: string
                  description: The sequence that needs to be uploaded.
                  format: binary
                type:
                  type: string
                  description: Type of the Endpoint
                apiData:
                  type: string
                  description: API Data to be updated
  1. Update velocity template of the API to override the existing configurations with Custom Backend provided.
  2. This will update the api.xml provided with the custom backend without storing the provided sequence in sequences directory of the APIM pack. This might affect the reeadabiility of the api.xml.
  3. On update, the endpointConfiguration object will look as follows.
	endpointConfig:
		endpoint_type: custom_sequence
		sequence: FILE_NAME

@BLasan
Copy link
Author

BLasan commented Sep 3, 2024

[Update 2]

As per the second suggestion, changed the velocity template logic to refer the sequence file provided. Following is the approached followed.

  1. Upload the file via the REST API and store under sequences directory which is used to load all sequences by synapse.
  2. Velocity template refers to the Sequence key generated by the code which looks as below.
                #if( $endpoint_config.get("endpoint_type") == 'custom_sequence' )
                    #set( $sequenceKey = $endpoint_config.get("sequence_key") )
                    <sequence key="$sequenceKey"/>
  1. This approach does not contain any tables to store uploaded custom backends. Only the API Artifact contains all necessary configurations of the endpoint.

@BLasan
Copy link
Author

BLasan commented Sep 3, 2024

[Update 3]

Changed the REST API to support add, delete, get operations of the custom sequence as the UI requirement. Followings are the changes introduced.

  1. Changed the REST API to only support Custom Backend upload. Following is the REST API format.
      requestBody:
        content:
          multipart/form-data:
            schema:
              properties:
                sequence:
                  type: string
                  description: The sequence that needs to be uploaded.
                  format: binary
                type:
                  type: string
                  description: Type of the Endpoint
  1. Introduced a new table AM_API_CUSTOM_BACKEND to store sequence related data. Following is the proposed schema for the table.
CREATE TABLE IF NOT EXISTS AM_API_CUSTOM_BACKEND (
  ID INTEGER NOT NULL AUTO_INCREMENT,
  API_UUID VARCHAR(256) NOT NULL,
  REVISION_UUID VARCHAR(256) DEFAULT NULL,
  SEQUENCE MEDIUMBLOB,
  NAME VARCHAR(256) NOT NULL,
  TYPE VARCHAR(120) NOT NULL,
  PRIMARY KEY (ID),
  FOREIGN KEY (API_UUID) REFERENCES AM_API(API_UUID) ON DELETE CASCADE
)ENGINE INNODB;
  1. New REST API supports the following operations related to the Custom Backend.
  • get
  • delete
  • update
  1. When retrieving the API Artifact, we need to retrieve the corresponding sequence from the AM_API_CUSTOM_BACKEND table using API_UUID and REVISION_ID.

@BLasan
Copy link
Author

BLasan commented Sep 4, 2024

[Update 4]

  1. Changed the schema of the new table as follows.
CREATE TABLE IF NOT EXISTS AM_API_CUSTOM_BACKEND (
  ID VARCHAR(60) NOT NULL,
  API_UUID VARCHAR(256) NOT NULL,
  REVISION_UUID VARCHAR(256) DEFAULT NULL,
  SEQUENCE MEDIUMBLOB,
  NAME VARCHAR(256) NOT NULL,
  TYPE VARCHAR(120) NOT NULL,
  PRIMARY KEY (ID),
  FOREIGN KEY (API_UUID) REFERENCES AM_API(API_UUID) ON DELETE CASCADE
)ENGINE INNODB;
  1. Name of the Custom Backend should be in the following format.
    <API_UUID>-<ENDPOINT_TYPE>

  2. Issue with the current table schema.

  • Cannot define a PRIMARY KEY as same policy ID will be replicated in the table on a revision deployment.
  • Cannot define a composite primary key as REVISION_UUID can be a null
  1. Solutions for the schema issue.
  • Make REVISION_UUID not null and make REVISION_UUID = "1" for the undeployed version of the API and have a composite primary key (REVISION_UUID, ID, API_UUID)
  • Introduce two tables. One to store sequence data (this will make a Custom Backend globally available) another one to map Custom Backend to API & Revision.

@BLasan
Copy link
Author

BLasan commented Sep 4, 2024

[Update 5]

  1. Changed the endpointConfig object structure to the following as Custom Backends can be added to both Production and Sandbox endpoints.
endpointConfig: {
   "endpoint_type": "custom_backend",
   "production": {
         "sequence_id": xxxxx,
         "sequence_name": xxxxxx,
         "type": "PRODUCTION"
   },
  "sandbox": {
         "sequence_id": xxxxx,
         "sequence_name": xxxxxx,
         "type": "SANDBOX"
  }
}

2 Implemented get, delete REST APIs which fetches and deletes the Custom Backends from the API_UUID, BACKEND_UUID, TYPE and REVISION_UUID
3. Made default value as "0" for the REVISION_UUID. Also, new composite primary key is (REVISION_UUID, API_UUID, TYPE and ID)

@BLasan
Copy link
Author

BLasan commented Sep 5, 2024

[Update 6]:

  1. Tested Custom Backend upload, Get operations
  2. Working on revision deployment. Fixing minor database issues.

Draft PR - wso2/carbon-apimgt#12524
Updated REST API - https://github.com/wso2/carbon-apimgt/pull/12524/files#diff-d8623bfc39bc03ceb819b7dff3e7cb16254f9eb97238c795f40d3b025a4a3747R491

@BLasan
Copy link
Author

BLasan commented Sep 9, 2024

[Update 7]:

  1. Tested the following flows against the new feature and working without an issue. (Tested for MySQL 8)
  • API creation
  • API deletion
  • Revision Deployment
  • Revision Restore
  • Revision Deletion
  • Changing Endpoint Configurations
  • API Invocation
  • API Import via APICTL
  1. New REST API changes can be found via the following links
  1. Tasks completed
  • REST APIs for Custom Backend
  • Revision related operations
  • API Velocity Template support for Custom Backend
  • API related operations (API Creation, Deletion)
  • APICTL support for Custom Backend (Import & Export)
  1. Tasks remaining
  • UI for Custom Backend upload and Endpoint Configuration change
  • Test APICTL export API
  • Analytics related changes

@BLasan
Copy link
Author

BLasan commented Sep 9, 2024

All scenarios to be tested

  1. API Creation
  2. Changing API Endpoint
  3. Add Custom Backend, Delete Custom Backend, Change Endpoint Config, Download Sequence
  4. API Revision Deployment, Undeploy revision, Delete revision, Restore revision
  5. API Delete
  6. API Invocation
  7. APICTL Import, Export, Update
  8. APIProduct creation, deploy, invoke, import & export

@BLasan
Copy link
Author

BLasan commented Sep 22, 2024

Tasks Completed -

  1. REST API support to define custom mediation logic by selecting endpoint type as "Sequence Backend".
  2. Upload, Delete, Download sequence backend
  3. Revision deployment, Revision deletion, Revision restore and API invocation with sequence backend
  4. APICTL import/export operations for REST APIs with sequence backend

Future Work -

  • Improve validations in both UI and backend
  • Add support for API Product with sequence backend
  • Check analytics related operations

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants