Extract Readwise API into its own sdk project

[renehernandez]

This would allow to more easily:

• Implement retry mechanism when reaching rate limits against Readwise
• Auto-paginate, when pulling more documents/highlights than the page_size value
• Test the implementation
• Mock the API interface in obsidian-readwise for better testing

[technicalpickles]

I'm taking a Building a Second Brain course, and there was a session with the folks from Readwise. I asked:

Do you have any resources you'd recommend for developers wanting to integrate with Readwise? I'm using Obsidian, and there is a community plugin, but it only lets you pull in highlights once you install the plugin. I've read the API docs which look super helpful to get started, but looking for patterns like... pulling down existing highlights initially

And they answered:

If you need some assistance with our API, just email us [email protected] and we’ll be happy to help with your workflow.

[technicalpickles]

Is the thought to make this a separate package on like NPM? Or extract in a different sense?

[technicalpickles]

For testing, I think we could use a combination of:

• https://www.npmjs.com/package/node-fetch for being able to run the tests outside of running obsidian
• https://www.npmjs.com/package/nock for mocking http requests

I'm going to start a branch to start playing with those ideas.

[renehernandez]

If you need some assistance with our API, just email us [email protected] and we’ll be happy to help with your workflow.

Most likely, this is due to them working on their own Readwise plugin

Is the thought to make this a separate package on like NPM? Or extract in a different sense?

That is something I am considering. The idea of extracting to a separate API would be the best route IMO. There is another Readwise plugin, that it has an API that already supports this features, see here.

[technicalpickles]

There is another Readwise plugin, that it has an API that already supports this features, see here.

Would you be able to use that client? There is not a license in the repository unfortunately. @jsonMartin any chance of adding a license to your repository?

[jsonMartin]

Hey @technicalpickles , sure thing. Just went ahead and added a MIT license to the repo 👍

[technicalpickles]

That's great, thanks! I took a closer look, and the only thing that would need to change in the library is that it uses Notify for updating the status bar. This plugin has its own way to use that, so I think it would just need a more general hook for when there is a status update.

[technicalpickles]

Trying to drop the API in as is within minimal modifcation was probably overly optimistic 🤔 What I'm thinking now is to:

• start with the API client in
• refactor out a parent class that is more generalized (ie handling things like auth + 429)
• import that parent class into obsidian-readwise
• refactor obsidian-readwise's API client to use the generalized API client

I don't know how much time I'm going to be put towards it and on what time scale, but I will keep this issue updated.

[renehernandez]

@technicalpickles Sounds good. If you start working on a separate repo for the Readwise API, share it here. I can contribute to it after finishing with the improvements left for v0.1