Archived in favor of https://github.com/magento/async-import
Magento uses Web API to cover the persistence operations of the business entities from the external clients. Some integrations require massive invocation of the Web API while persisting entities from the external systems. Usually it causes performance and scalability issues both on the Magento system and on the external clients. This project is intended to provide a way for persisting big amounts of data in the Magento in efficient scalable way for the external system.
Bulk API track consists of 3 repositories:
- current, https://github.com/magento-engcom/bulk-api is for documentation and tasks tracking
- https://github.com/magento/bulk-api-ce is for contributions to the CE part of the Bulk API scope
- https://github.com/magento/bulk-api-ee is for contribution to the EE part of the scope
- Implement support of asynchronous invocation of the magento Web API endpoints: requested operation should not be performed immediately, but rather saved to the queue for the later processing.
- Support invocation of the CRUD APIs with the multiple entities in one request
- Support of the status tracking API for asynchronous operations
- Improve performance of persistence operations for the key entities like catalog in the Magento business logic. Resolve deadlocks.
Technical vision for the Bulk API: https://github.com/magento-engcom/bulk-api/wiki High Level design for asynchronous operations: https://github.com/magento-engcom/bulk-api/wiki/Asynchronous-Web-API
We are using ZenHub board to manage stories and tasks and build burndown chart for them. Please install browser plugin to get all the features of this application.
The Kanban board: https://app.zenhub.com/workspace/o/magento-engcom/bulk-api
Currently, all the functionality of the Message Queue and RabbitMQ integration is part of the Magento EE edition. Most stories as a part of the Bulk API track relies on the Message Queue interfaces. There is already a product decision made of moving the interfaces from the EE to CE edition of Magento.
All the Solution Partners of Magento and developers affiliated with them should have access to the EE repositories for this project. If you don't have it yet, please write us at [email protected]
If you are not the Solution Partner but still want to contribute, completion of moving of the Message Queue interfaces to CE should be a prerequisite for this task.
git clone https://github.com/magento/bulk-api-ce.git
cd bulk-api-ce
git clone https://github.com/magento/bulk-api-ee.git
php -f bulk-api-ee/dev/tools/build-ee.php -- --ce-source=. --ee-source=bulk-api-ee --command=link
This will clone the repositories and create symlinks from the EE to CE. PLease note that EE is cloned to the subdirectory of CE. This is needed to resolve the templates files references.
- Start Docker container for RabbitMQ 3 with the management plugin and expose AMQP port and Admin UI port:
docker run --hostname localhost -p 5672:5672 -p 15672:15672 rabbitmq:3-management
- Go to http://localhost:15672/ login:guest, password:guest to check the statuses of the exchanges and queues
bin/magento setup:install
--backend-frontname="admin"
--amqp-host="localhost"
--amqp-port="5672"
--amqp-user="guest"
--amqp-password="guest"
--db-host="localhost"
--db-name="bulk_api_ce"
--db-user="root"
--db-password="root"
--admin-user="admin"
--admin-password="admin123"
--admin-email="[email protected]"
--admin-firstname="Eugene"
--admin-lastname="Tulika"
--base-url="magento.url"
--cleanup-database
- Setup Postman
- Create Integration in the Magento backend
- Send Web API request to Magento and ensure that it works
http://magento.loc/rest/V1/products POST
{
"product": {
"sku": "simple-attempt-1-request-1",
"type_id": "simple",
"attribute_set_id": "4",
"price": "12.22",
"name": "new_name_simple-attempt-1-request-1",
"extension_attributes": {
"stock_item": {
"qty": 0
}
},
"custom_attributes": [
{
"attribute_code": "ap21_size_code",
"value": "123"
},
{
"attribute_code": "ap21_style_code",
"value": "abc"
},
{
"attribute_code": "special_price",
"value": "12.11"
},
{
"attribute_code": "color",
"value": "4121"
},
{
"attribute_code": "size",
"value": "3242"
}
]
}
}
- add
/async
prefix to the Web API endpointrest/V1/products
=>rest/async/V1/products
- Send request. Navigate to RabbitMQ and see message in the Queue
- Navigate to the System => Bulk Actions to see the UI of bulk operations and new operation there