Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support transparent mapping of Scrapy requests to Zyte Data API requests #41

Merged
merged 85 commits into from
Oct 19, 2022
Merged

Support transparent mapping of Scrapy requests to Zyte Data API requests #41

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
ce6f9ba
Add ZYTE_API_ENABLED
Gallaecio Aug 12, 2022
6348e81
Split ZYTE_API_ALL off of ZYTE_API_ENABLED
Gallaecio Aug 12, 2022
5f78c55
ZYTE_API_ALL → ZYTE_API_ON_ALL_REQUESTS; mention setting default valu…
Gallaecio Aug 12, 2022
68f43dc
Initial proposal for transparent mapping
Gallaecio Aug 12, 2022
a2ad009
Update test_zyte_api_request_meta
Gallaecio Aug 12, 2022
4c44ebe
Run pre-commit hooks
Gallaecio Aug 12, 2022
78fb734
Merge remote-tracking branch 'upstream/main' into enable-setting
Gallaecio Aug 24, 2022
26b2ef0
README: move ZYTE_API_ENABLED under Configuration
Gallaecio Aug 24, 2022
e78a499
test_zyte_api_request_meta: update misleading values
Gallaecio Aug 24, 2022
29fdbec
Merge remote-tracking branch 'origin/enable-setting' into transparent…
Gallaecio Aug 24, 2022
d9fbbf1
Clarify value precedence in warnings
Gallaecio Aug 24, 2022
4442ac9
Include an unsupported header in the test for ZYTE_API_UNSUPPORTED_HE…
Gallaecio Aug 24, 2022
a80b045
Implement the zyte_api_automap request meta key
Gallaecio Aug 24, 2022
9864273
Add a missing test case
Gallaecio Aug 24, 2022
afd9b6d
Restore Python 3.7 support
Gallaecio Aug 24, 2022
c49e1b3
Refactor parse_api_params and start refactoring tests
Gallaecio Aug 24, 2022
400a4c2
Complete test refactoring
Gallaecio Aug 25, 2022
a19c004
Revert change no longer needed
Gallaecio Aug 29, 2022
c59fb8b
Refactor response mapping tests
Gallaecio Aug 29, 2022
0840fa8
Fix tests
Gallaecio Aug 29, 2022
08adc14
Make ZYTE_API_AUTOMAP False by default
Gallaecio Aug 30, 2022
a649756
Complete testing scenarios for ZYTE_API_ENABLED
Gallaecio Aug 30, 2022
f95e61d
Refactor test_job_id
Gallaecio Aug 30, 2022
e0ac089
Fix test_higher_concurrency
Gallaecio Aug 30, 2022
3508bee
Move test_automap next to related tests
Gallaecio Aug 30, 2022
4f36bda
Test the side effects of the output of _get_api_params
Gallaecio Aug 30, 2022
3bfb787
test_use_api_by_default: add a docstring
Gallaecio Aug 30, 2022
3cf94b3
test_api_disabling_deprecated: add a docstring
Gallaecio Aug 30, 2022
13bcde6
test_job_id: add a docstring
Gallaecio Aug 30, 2022
871f178
test_default_params_none: add a docstring
Gallaecio Aug 30, 2022
558a32b
test_default_params_merging: add a docstring
Gallaecio Aug 30, 2022
c661d38
Move test_default_params_automap after default params and automap tests
Gallaecio Aug 30, 2022
a934bee
test_default_params_immutability: add a docstring
Gallaecio Aug 30, 2022
0bdf54e
test_bad_meta_type: reorder and add a docstring
Gallaecio Aug 30, 2022
4288101
test_automap_toggling: add a docstring
Gallaecio Aug 30, 2022
8aa623a
test_default_params_immutability: fix, finish its refactoring
Gallaecio Aug 30, 2022
7191ae2
test_automap: finish refactoring
Gallaecio Aug 30, 2022
efd61d0
test_automap_main_outputs: add comments
Gallaecio Aug 30, 2022
5fd6a5f
test_automap_header_output: add comments
Gallaecio Aug 30, 2022
dca7e4b
test_automap_method: add comments
Gallaecio Aug 30, 2022
bfee2ba
test_automap_headers: add comments
Gallaecio Aug 30, 2022
30df7c8
test_automap_default_parameter_cleanup: add comments
Gallaecio Aug 30, 2022
1bad96b
Extend test_automap_headers
Gallaecio Aug 30, 2022
15cfea7
Merge remote-tracking branch 'upstream/main' into transparent-mapping
Gallaecio Aug 30, 2022
325c105
setup.py: revert unneeded change
Gallaecio Aug 30, 2022
5d888f4
Merge remote-tracking branch 'upstream/main' into transparent-mapping
Gallaecio Sep 20, 2022
a56c1f0
Update the documentation
Gallaecio Sep 20, 2022
46f498e
Zyte Data API → Zyte API
Gallaecio Sep 20, 2022
2c28cac
README: do not use Sphinx references
Gallaecio Sep 20, 2022
3241a4c
Fix typo: Rerquest → Request
Gallaecio Sep 20, 2022
a89efbb
Remove unneeded detail
Gallaecio Sep 20, 2022
75918bc
Clarify when to use header settings
Gallaecio Sep 20, 2022
15d89d2
test_api_toggling → test_transparent_mode_toggling
Gallaecio Sep 20, 2022
2492036
Update some tests
Gallaecio Sep 20, 2022
8e9113f
Complete test update
Gallaecio Sep 20, 2022
ebe3b4f
Refactor _update_api_params_from_request_headers
Gallaecio Sep 20, 2022
e0ed243
Refactor _update_api_params_from_request
Gallaecio Sep 20, 2022
2afc104
Refactor _get_raw_params and _get_automap_params
Gallaecio Sep 20, 2022
414ccc2
_merge_params: do not modify meta_params
Gallaecio Sep 20, 2022
41d6430
Refactor downloader __init__
Gallaecio Sep 20, 2022
d7a02c7
When automapping, always set httpRequestMethod for non-GET values
Gallaecio Sep 20, 2022
1b0a292
When automapping, always set httpRequestBody for non-empty values
Gallaecio Sep 20, 2022
6889a25
Allow disabling or forcing header mapping
Gallaecio Sep 20, 2022
db6c3ae
Test that browserHtml takes precedence over httpResponseBody for Resp…
Gallaecio Sep 20, 2022
550cff3
cannot known → cannot know
Gallaecio Sep 22, 2022
d454c9b
Update tests/test_api_requests.py
Gallaecio Sep 27, 2022
1c46849
README: sort usage approaches by relevance
Gallaecio Oct 13, 2022
987ef16
README: include more code examples
Gallaecio Oct 14, 2022
a4fa769
README: provide more specific links to the Zyte API documentation
Gallaecio Oct 14, 2022
6098b69
unsupported headers → skip headers
Gallaecio Oct 14, 2022
41d9a70
README: add code examples of automated parameter mapping
Gallaecio Oct 14, 2022
8e93e00
Clarify how default parameter settings affect unrelated requests
Gallaecio Oct 14, 2022
0019ea2
_iter_headers: parameter → header_parameter
Gallaecio Oct 14, 2022
82cbc54
Move parameter handling to its own module
Gallaecio Oct 14, 2022
109ed87
make_handler: clarify when handler is set to None
Gallaecio Oct 14, 2022
33a0501
tests: fix typo (Request.meta → Request.method)
Gallaecio Oct 14, 2022
9f7fb4b
tests: Fix example Content-Length value
Gallaecio Oct 14, 2022
6a1a496
Clarify message about httpResponseHeaders=False being unnecessary
Gallaecio Oct 14, 2022
dcfc5b7
Remove commented out code
Gallaecio Oct 14, 2022
40e708e
tests: fix Content-Length expectations
Gallaecio Oct 14, 2022
329ba58
Pin pyopenssl==22.0.0 on the pinned Tox environment
Gallaecio Oct 14, 2022
79ecfd1
Document how httpResponseBody should be accompanied by httpResponseHe…
Gallaecio Oct 17, 2022
dd67e12
Add a section about response mapping
Gallaecio Oct 17, 2022
c386036
Discourage using certain parameters on ZYTE_API_AUTOMAP_PARAMS
Gallaecio Oct 17, 2022
80f7b46
Do not warn about explicit httpResponseBody or httpResponseHeaders, a…
Gallaecio Oct 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ repos:
hooks:
- id: isort
- repo: https://github.com/ambv/black
rev: 20.8b1
rev: 22.3.0
hooks:
- id: black
language_version: python3.8
Expand Down
Loading