Add support for URLs in DUD_LOAD_RULE_PATHS. #15
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Gallaecio
approved these changes
May 28, 2024
BurnzZ
reviewed
May 29, 2024
Comment on lines
+35
to
+36
| response = await maybe_deferred_to_future(treq.get(rule_path)) | ||
| data = await response.text() |
There was a problem hiding this comment.
Let's handle the case when the rules were not successfully retrieved due to things like connection issues, timeouts, etc:
- Logging the error
- Terminate the crawl - I think this would be a good behavior since if the spider proceeds without any rules, the user would accumulate a lot of requests due to unfiltered requests.
| rules: Set[UrlRule] = set() | ||
| full_rule_count = 0 | ||
| for rule_path in rule_paths: | ||
| data = Path(rule_path).read_text() | ||
| data: str | ||
| if isinstance(rule_path, str) and self._is_url(rule_path): |
There was a problem hiding this comment.
minor: isinstance(rule_path, str) can be placed inside _is_url().
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1.
This uses
treqfor downloading, but there are many other options so this is open for discussion.requests). Pros: synchronous requests simplify the code. Cons: synchronous may be bad?UrlCanonicalizer()is currently decoupled from Scrapy.treq. Pros: straightforward. Cons: additional dep.aiohttp. Pros: just a more modern thing. Cons: additional dep, requires the asyncio reactor which is a blocker.Also this doesn't have tests for URLs, should we add mockserver, or wait until we publish the rules and use them in the tests?