New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

extend the four-component to every request #808

Open

Henior wants to merge 1 commit into code4craft:master from Henior:master

Henior commented Jul 9, 2018

对每一个请求，进行可定制化配置，即每个请求包含自己的downloader,processor,scheduler和pipeline,
如果没有设置则走默认的
举个例子，当你在爬虫的时候，有的请求只需要httpclient进行简单的页面请求，但同时有的请求却要进行phantomjs的本地js操作，这种情形下为了拥有js操作的功能，我们只能选后者的downloader，即使逻辑复杂化又降低了性能（仿浏览器操作比简单的网络请求消耗更多资源），代码耦合性也很高


          extend the four-component to every request

e6b0ef0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet