Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatic decentralised update function using the dht #157

Open
chinwendu opened this issue Feb 7, 2015 · 4 comments
Open

Automatic decentralised update function using the dht #157

chinwendu opened this issue Feb 7, 2015 · 4 comments

Comments

@chinwendu
Copy link

https://github.com/chinwendu/openbay-crawler

I finally released the first 100% independent decentralised automatic update function for the openbay projet. I do not see it as a replacement for an RSS update, but more like a complement. Once you launch it, it will listen to the dht for torrents requests. On torrent request (announce or peer request), it will download the torrent file from the peers found in the dht or try to download it from torcache if none of the peers have responded.
It will then feed the database based on the metadata inside the torrent file (name, list of files, size, creation date ...).

I made a function to distribute the info_hash and torrent url to others running the program to distribute the crawling effort of the dht between the participant.

I personally feed my own database with almost 3.000.000 torrents (starting from scratch) in less than a month.

I love to hear some feedback from openbay users, so feel free to comment.

edit: I forgot to say it can also scrape torrents againts udp://open.demonii.com:1337 and update to db with seeder and leecher number.

@TPBT-OFFICIAL
Copy link

Unfortunatly i can't seem to make it work i keep getting some weird python errors if you could help me i would be glad to show you what kind of errors i get.

@chinwendu
Copy link
Author

sure no problem, if you can give me a traceback

edit: and the OS you are using

@TPBT-OFFICIAL
Copy link

OS: CentOS 6
Errors: When i run "./crawler.py" i get
Traceback (most recent call last):
File "./crawler.py", line 14, in
from btdht import DHT, ID, RoutingTable
File "/usr/lib64/python2.6/site-packages/btdht/init.py", line 13, in
from .dht import DHT, DHT_BASE, ID, Node, Bucket, RoutingTable, NotFound, BucketFull, NoTokenError
File "btdht/krcp.pxd", line 3, in init btdht.dht (btdht/dht.c:48774)
File "btdht/krcp.pyx", line 19, in init btdht.krcp (btdht/krcp.c:19498)
File "btdht/utils.pyx", line 15, in init btdht.utils (btdht/utils.c:12432)
ImportError: cannot import name total_ordering

@chinwendu
Copy link
Author

I think you need python 2.7 https://docs.python.org/2/library/functools.html#functools.total_ordering New in version 2.7

edit: I added it as a dependacy in README.md, did think to do it as python 2.7 is the version distributed with the last debian stable (wheezy)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants