Support queues other than RabbitMQ

[pawelbaranski]

Hi,

Would you consider adding different queues than RabbitMQ ? Is there a way to easily replace RabbitMQ with Kafka, ZeroMQ, etc... ? I am having an issue where I'd like to use servicebus, bus our technology stack may not use RabbitMQ in a long run

[mateodelnorte]

I started a refactor to do just that, which is why there are two bus.js
files and a rabbitmq folder, but ran out of time. More than happy to accept
pull requests. ;)
On Jan 14, 2016 7:09 AM, "Paweł Barański" [email protected] wrote:

Hi,

Would you consider adding different queues than RabbitMQ ? Is there a way
to easily replace RabbitMQ with Kafka, ZeroMQ, etc... ? I am having an
issue where I'd like to use servicebus, bus our technology stack may not
use RabbitMQ in a long run

—
Reply to this email directly or view it on GitHub
#61.

[pawelbaranski]

I would be happy to help, but I am just starting to improve my NodeJS skills. Maybe later :)

[emadb]

@mateodelnorte do you have an idea on how to implement this functionality (decoupling bus from transport)? I mean which refactoring steps are needed to have it I could try to help you.

[mateodelnorte]

Hi @emadb. Happy to have the help!

The first step will be modifying https://github.com/mateodelnorte/servicebus/blob/master/bus/index.js#L3 to accept a transport type and creating an implementation for redis (or whatever the provider will be) that will implement the same features that https://github.com/mateodelnorte/servicebus/blob/master/bus/rabbitmq/bus.js implements. The bus() function will then instantiate the appropriate transport and functionality should continue from there.

To provide a standard interface to implement, send+listen+publish+subscribe+close from https://github.com/mateodelnorte/servicebus/blob/master/bus/rabbitmq/bus.js should probably all have stubs implemented in https://github.com/mateodelnorte/servicebus/blob/master/bus/bus.js which all providers must implement, else they throw a new Error('bla is not implemented)' error.

That should get us to the point where basic functionality works, then we can hack through getting any additional things working like servicebus-retry, etc.

Would be happy to help with all of this.

[emadb]

Ok @mateodelnorte.
I will give a try, I can't assure you to complete all the task but as soon as I have some free time I would like to help.
One thing that I need is an in-memory bus, primarily for testing purpose and for dev scenario where you don't need (yet) a real bus.
Having a redis implementation is also useful!

[mateodelnorte]

In memory could be performed either by zeromq or a combination of UDP and
TCP messages. Zeromq would be the easier easier way to go, but introduces
the single dependency.

If what you want to do is unit test your handlers / modules that use
service bus, look at using something like mockrequire
On Apr 12, 2016 9:42 AM, "Emanuele DelBono" [email protected]
wrote:

Ok @mateodelnorte https://github.com/mateodelnorte.
I will give a try, I can't assure you to complete all the task but as soon
as I have some free time I would like to help.
One thing that I need is an in-memory bus, primarily for testing purpose
and for dev scenario where you don't need (yet) a real bus.
Having a redis implementation is also useful!

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#61 (comment)

[emadb]

I implemented a first implementation of a FakeBus. Most methods throw an error (I just implemented pub-sub for testing).
By now I pushed my code here: https://github.com/emadb/servicebus/commits/master

I tried to keep the old tests green and the default implementation is still on Rabbit, I simply added an optional argument on the constructor.
Let me know if it is a good starting point or not :-)

[archa347]

Are similar modifications needed to work with other AMQP compliant systems, or is this module truly RabbitMQ specific?

[mateodelnorte]

I haven't tried it on any vendors other than rabbitmq. Happy to merge PR's for additional brokers or transports.

[mateodelnorte]

@emadb I responded in your repo, but I'll parrot here.

A fake bus isn't beneficial here, as an external mocking framework can provide that - and having a fake bus internal to the servicebus library would only help in mocking out single process scenarios, where sb is most widely used in creating distributed systems.

[archa347]

Alright, my company has decided to invest its time in ActiveMQ, which supports AMQP as well. I'll have to try it out and I'll let you know what comes up

[emadb]

@mateodelnorte Yes. I saw your response. I replied here:

Hi @mateodelnorte I totally agree with you. I was just trying to create a fake bus for "playing" with it and see if everything works fine. FakeBus is not to meant to stay there.
The point on which I would like to have some feedback is the "factory method" to which you can pass the bus implementation to use: https://github.com/emadb/servicebus/blob/master/bus/index.js#L4

Is this the idea that you have in mind? If it's ok I go on on cleaning up the code, write some docs add some tests and..of course removing the FakeBus :-)

[mateodelnorte]

Hi @archa347 what were your results? I'm guessing we're far too tightly coupled to RabbiqMQ's AMQP specification to be simple plug-and-play. The AMQP standard is, in and of itself, all over the place with vendor implementations being much tighter than the standard.

[bradoyler]

@mateodelnorte Foresee any major issues with swapping in this package? https://github.com/markbirbeck/amqp-sqs

[mateodelnorte]

Interesting. I think that wrapper goes half way, and covers send/listen. It looks like for publish/subscribe and fanout you'd need to also implement a wrapper on top of SNS. Search for 'fanout' here: https://aws.amazon.com/sqs/details/.

Theoretically, though, I think it could be done.

I've started doing some experimental work with making servicebus use a provider model for transports: https://github.com/mateodelnorte/servicebus-rabbitmq I'm still finicking with the best way to separate the required modules into different packages without leaving servicebus, itself, as a mostly empty shell.

I think what would be required is making something like servicebus-rabbitmq (or whatever interface it ends up boiling down to) that implements amqp-sqs and a corollary to fanout/topic routing, perhaps being a wrapper around SNS. Something like https://www.npmjs.com/package/aws-sns-publish might help do the trick.

[patrickleet]

this is in progress in servicebus org - kafkabus on it's way