Commit d09b758c authored by Torstein Nicolaysen's avatar Torstein Nicolaysen
Browse files

DEICH-5663 update RabbitMQ doc

parent 7f47e515
# RabbitMQ
## Design choices
- [RabbitMQ](#rabbitmq)
- [Why RabbitMQ](#why-rabbitmq)
- [Design choices](#design-choices)
- [Configuration](#configuration)
- [Centralized configuration](#centralized-configuration)
- [Pre-configured resources](#pre-configured-resources)
- [VHost](#vhost)
- [Users](#users)
- [Exchanges](#exchanges)
- [Queues](#queues)
- [Security](#security)
- [Considerations](#considerations)
- [Operations](#operations)
- [Shovel plugin](#shovel-plugin)
- [Clients](#clients)
- [Koha](#koha)
- [MailChimp synchronizer](#mailchimp-synchronizer)
- [Resources and links](#resources-and-links)
## Why RabbitMQ
- Open Source
- Mature
- Working community
- High adoption rate
- Tried and tested
- Built on solid technology (Erlang)
- Works well with Docker
- Libraries for all the technologies used by Deichman
### What is it optimized for?
## Design choices
- Not loosing messages
- At least once delivery
- Not for speed and throughput
- Simple setup
- Use sane defaults as much as possible
- Declarative configuration
- Avoid complex setup and/or custom code for setting up RabbitMQ
### Configuration
## Configuration
Deichman's needs are fairly simple. The volume and frequency of messages will be low for the first use of RabbitMQ.
### Centralized configuration
The `definitions.json` file contains the _declarative_ configuration of the setup that should be in place regardless
of the clients.
The following files compose the (centralized) configuration of RabbitMQ, and are mounted into the Docker container:
- `/etc/rabbitmq.conf`
- Main configuration file. Minimal configuration. Defaults are good, and little need to be overridden.
- `/etc/enabled_plugins`
- Used to enabled plugins like _shovel_ and define explicitly which plugins to load.
- `/etc/definitions.json`
- Defines _VHosts_, _users_ (including credentials), _permissions_, common _exchanges_, initial _queues_ and _bindings_.
The clients of RabbitMQ are responsible for setting up everything needed, except users and access control.
The trade-off is that the user has to have more access then strictly needed, but the access control mechanisms in RabbitMQ was not very suited to the needs here.
#### Pre-configured resources
> This explains the contents of `definitions.json`
#### VHost
`deichman` is the only VHost in the first use case of RabbitMQ. It's recommended to not used the default `/`. New VHosts can be introduced to isolate uses. E.g. could it be useful to have a dedicated VHost for use outside Deichman's internal systems, like an integration with Oslo Kommune.
#### Users
- `admin`
- `mailchimp_synchronizer`
- `koha`
#### Exchanges
- `service`: Shared exchange for all services within the vhost
- `delay`: To delay failed messages before a retry
- `retry`: To retry failed messages up to maximum number of times
- `dead_letters`: When retrying fails, messages end up here
`service` is the primary exchange and the routing key is used to decide which queue to route the message to. The exchange is set up to be a _topic_, which gives the benefit of being able to add a new queue without much effort to process certain messages. E.g. it could be used to intercept the "borrower deleted" message and initiate a GDPR cleanup.
#### Queues
- `delay:1m`: Used to delay messages 1 minute. Messages put here will have a header which is the queue to route it back to, and the built in TTL will make sure it happens after 1 minute.
- There is also a binding to the `delay` exchange
There is support for making more variations of this.
### Security
- No way to connect to AMQP externally in production
- Management UI (over HTTPS) will only be accessible on Intranet
- One user per service
- Because of the design choice to let clients set up the resources, they need to have a bit more access then needed
- Can also improve/restrict access over time
- Encryption is not used
- Avoid sensitive data in messages
- SSL is not used
- LDAP is not enabled, but an option if needed
- OAuth is also an option, maybe better
#### Considerations
......@@ -32,6 +111,10 @@ RabbitMQ comes with multiple mechanisms for monitoring.
**TODO** DETAIL THIS SECTION
### Shovel plugin
The clients can put messages on a dead-letter queue. In a scenario where e.g. MailChimp is unavailable for a long time, a lot of messages would be put on a dead-letter queue. The _shovel_ plugin enables an administrator to put all those messages back on a queue so that they can be retried.
## Clients
### Koha
......@@ -50,26 +133,7 @@ Future use cases can be to use the deleted event to trigger clean up in other sy
**Consumes** events related to borrowers. This service is responsible of synchronizing the subscription status and email address of borrowers with MailChimp.
## Configuration and Docker setup
The following files contain declarative configuration and are mounted into the container:
- `/etc/rabbitmq.conf`
- Main configuration file. Minimal configuration. Most defaults are OK.
- `/etc/enabled_plugins`
- Used to enabled things like _shovel_ and define explicitly which plugins to load.
- `/etc/definitions.json`
- Defines _VHosts_, _users_ (including credentials), _permissions_, common _exchanges_, initial _queues_ and _bindings_.
The clients of RabbitMQ are responsible for setting up everything needed, except users and access control.
The trade-off is that the user has to have more access then strictly needed, but the access control mechanisms in RabbitMQ was not very suited to the needs here.
### Shovel plugin
The clients can put messages on a dead-letter queue. In a scenario where e.g. MailChimp is unavailable for a long time, a lot of messages
would be put on a dead-letter queue. The _shovel_ plugin enables an administrator to put all those messages back on a queue so that they
can be retried.
## Resources and links
- <https://www.rabbitmq.com/>
- <https://github.com/guidesmiths/rascal/tree/master/examples/advanced/> Several thing are based on this example
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment