Rabbit MQ

  • Why we need a messaging system.
    • When we implement rest based micro services, which require asynchronous architecture but the HTTP protocol, we use in the services is synchronous.
    • A messaging system may have a message broker that push his messages to client or may allow clients to pull messages from the queue.
    • To develop an architecture Using HTTP Rest services We use messaging system.
    • A message broker allows applications to communicate with each other And exchange messages.
    • A Message broker Allows engineer to create a loosely coupled application. Allows new parts of the system to be modified without affecting system.
    • Message brokers operate asynchronously thus making system more scalable.
    • Message brokers are mostly used in micro service based Applications.
  • AMQP
    • AMQP uses its service call pattern, which allows one computer to execute methods or programmes on another computer.
    • The communication is two way.
    • Rabbit MQ uses commands to communicate between client and broker.
    • The date of command is declared in Data structure called frames.
      • Frames have a standard structure, which is of following types
        • Method frame
          • Method frame Has the class and method information.
            • Number of bytes that represent the class.
            • Number of bytes that represent the method.
            • Number of arguments to be passed to the method.
        • Content header frame
        • Body frame
        • Heartbeat frame
  • AQMP Frames
    • Structure of frame is as follows
      • One byte Which defines the frame type.
      • Two bytes that define channel frame is being sent on.
      • Four bytes represents size of the message.
      • Frame Specific content.
      • One byte To represent end of frame.
  • Rabbit MQ components
    • Producer
      • A producer is an application that sends messages.
      • It does not send messages to consumer directly. It sent messages only to be rabbit MQ broker.
    • Consumer
      • A consumer is an application that read messages from the rabbit MQ Broker.
      • Multiple consumers can subscribe to a rabbit MQ Broker.
    • Queue
      • Queue is a buffer or a storage in rabbit MQ broker to store the messages.
      • The messages are put into queue by a producer and read from it By a consumer. 
      • Once a message is read, It is consumed and removed from the queue.
      • A message can thus only be Processed exactly once.
    • Exchange
      • Acts as an intermediate between The producer and a queue.
      • Instead of sending messages directly to a queue, a producer can send them to an exchange instead. The exchange then send those messages to one or more Queues Following a specified set of rules.
      • Thus the producer does not know the queues that eventually receive those messages.
    • Binding
      • A binding is a link between Queue and exchange.
      • Binding is done by routing key.
    • Routing key
      • Routing is a key that the exchange looks at to decide how to route the message to queues.
      • The routing key is like an address for the message.
    • Message
      • Information that is sent from the producer to a consumer through rabbit, MQ.
      • Message can be a string, byte array, Json, plain text or HTML.
  • RabbitMQ Architecture
    • Producer send message to rabbit MQ broker.
    • Consumer receive message from rabbit MQ broker.

  • Rabbit MQ is integrated in spring boot project using spring AMQP library.
    • Advanced message, queueing protocol.
    • To connect to our spring boot application with rabbit MQ server, we use spring boot auto configuration as follows.
    • When we use auto configuration of rabbit MQ in spring boot, it automatically creates following beans
      • Connection Factory
      • Rabbit Template
      • Rabbit Admin
    • We just need to inject above beans To use it.
  • We then configure rabbit MQ Components in spring boot application.
    • Binding between Queue and Exchange
  • Then producer and consumer communicate in string or json Messages using exchange and queues.
  • We can have multiple queues in a Rabbit MQ architecture.
  • Message Queue
    • Message Queuing Allows application to communicate by sending messages to each other. The message Queue Provides temporary message storage when the Destination program is busy or not connected.
    • A message Queue Is made up of a producer, a broker, and a consumer.
    • A message queue Provides an asynchronous communication between applications.
      • Producer sends message to message broker.
      • Consumer consumes the message from message broker.
  • Rabbit MQ
    • Rabbit MQ is a message Queue software (message broker/queue manager) that act as an intermediate platform where different applications can send and receive messages.
    • Rabbit MQ originally implements, the advanced message, queueing protocol(AMQP).But now Rabbit MQ Also, Supports Several other API protocols such as STOMP, MQTT and HTTP.
    • Producer is an application that sends messages to the rabbit, MQ, broker and consumers is an application that reads messages from the rabbit. MQ broker.
    • Use of rabbit MQ in micro services 
      • Rabbit MQ is one of the simplest freely available Options for implementing message queues In your micro-service architecture.
      • It helps us to create an asynchronous model of micro services.
  • Rabbit MQ connection Establishment
    • Client sends Protocol header to the broker.
    • Server, accept or reject the protocol header.
      • If it rejects, it adds a valid header to the TCP header.
      • If it accepts it, sends, a connection start method Frame, which includes
        • Protocol version server purposes.
        • Security mechanisms.
        • List of server properties.
    • Client, then sends Connection start okay frame After analysing security parameters using SSL security mechanism.
    • Server sends a connection secure method To client. The SSL protocol works by exchanging challenges and responses until both server and client have received sufficient information to pair with each other.
    • Client sends connection, secure, okay, method frame.
    • Server and connection tune method frame to the client.
      • It contains information like maximum channel supported
      • The maximum frame size supported
      • The desired heartbeat delay.
    • Client responds with method tune, okay frame.
    • Client sends connection open method frame.
    • Server responds with connection open okay frame.
    • The communication between client and broker is mostly synchronous For connection negotiation and opening process.
      • Follows a request response pattern.
    • To create a queue, client, sends queue declare method frame to the server. The server responds with Queue declare ok frame.
    • To create exchange A client sends exchange declare method frame to the server. The server responds with Server. declare. Okay frame.
    • To publish a message, the client send a basic.publish method. It then sends content header frame Which contains details such as size of message to be published.
      • This involves a method frame, content header frame, And one or more body frames.
      • It will then Send one or more body frames that will make up the message.
      • The number of body frames required depends on the size of message and maximum size of message supported by rabbit MQ server.
  • Rabbit MQ message Exchange using basic get method
    • To receive or consume messages, the client sends the basic get method.
      • This provides a direct access to the messages in the queue by using a Synchronous dialogue That is designed for specific type of application where synchronous functionality is more important than performance.
    • The server responds with basic get okay method or basic get Empty method. This is then followed by message itself, which contains content had a frame and number of body frames, depending on the size of message.
    • The client responds with basic. Ack All acknowledgement frame To acknowledge the receipt of message.
    • Then again for next message, the Client sends the basic.get method And the process continues For each individual Message.
  • Rabbit MQ message exchange using basic.consume method frame
    • Besides basic get method, we have another way to consume messages using basic consume method frame.
    • This method asked the server to start a consumer, which is transient request for messages from a specific Queue. Consumers, last as long as the channel they were declared on exists or until client, cancel them.
    • Broker then responds with a basic.consume method, okay frame. 
    • As and when messages arrive for that consumer on the broker. The broker will send Basic delivery method to the client, which will be followed by content, header frame or number of body frames. 
    • The client acknowledges each message With acknowledgment frame.
    • When another message is received, the broker will send another basic. Deliver  message and this process Continues until either channel is closed or client cancel the consume.
  • Rabbit MQ Management
    • Connections
      • We see all the TCP connections here.
      • When we connect our application with rabbit, MQ server That entry is shown in connection tab.
    • Channels
      • Producer and consumer exchange a message over a channel.
      • Whenever we establish a connection between producer and consumer, we see it in channel.
  • Competing Consumer Pattern
    • It is also known as Work Queue pattern.
    • It is often used to distribute time consuming tasks among multiple workers.
    • Resource incentive tasks, make us wait for a while to complete. We add the task to a work queue And work process in the background, pops the task out of the Queue once it is completed.
    • This helps us to improve scalability and reliability of the architecture.
    • By default, Rabbit MQ Send message to each consumer in round Robin fashion. This has a disadvantage if two consumers have different speeds to process a message As it may lead to one of the consumers, be piled up with messages while other is free.
    • To overcome above problem, we set the prefetch value to one. This, let’s rabbit MQ not give one more message to a worker at a time.
    • Whenever we see messages backlog with consumer or slowdown, just add one more consumer to consume messages From Queue.
    • Example
  • Publish Subscribe pattern(Pub Sub Pattern)
    • It is about delivering messages to the multiple consumers.
    • It is opposite of competing consumers pattern.
      • In competing consumer, we share one message across many consumers. In publish, subscribe, we take a message, duplicate the message and send it across multiple consumer.
    • Multiple micro services Might be interested in processing the same message. For example, a new Supplier added to a system may inform all the downstream system of the new supplier being added.
    • There are many types of exchanges in rabbit MQ?
      • Direct Exchange
      • Topic exchange
      • Headers exchange
      • Fan out exchange
    • Exchange received messages from producers and push them on queues. An exchange knows, what needs to be done with message , Should it be discarded or sent to one or more queues.
    • A fanout exchange publish a message to multiple queues
      • It publishes the message to all the queues that are interested in the message
    • Rabbit MQ does not store multiple copies of messages in the memory. It just stores the message once And each Queue stores reference to the message.
      • So it is also memory efficient as we are not duplicating message and consuming a ton of memory.
    • The producer does not have knowledge of what consumers are interested in its message.
      • Producer is the decoupled from downstream consumer services.
    • Service can create a temporary queue to get messages from exchange And then destroy the queue when not required.
    • Example
  • Routing Pattern
  • Request Reply Pattern
    • Allows us to send a request into rabbit MQ Specifying a queue that we would like our reply to be sent to.
    • When the consumer consumes the message, it is able to send a reply on the queue we specified in request.
    • We call the consumer sending request as client and consumer processing the request as server.
    • Client sends message on a, request queue bound to exchange.
    • Server sends message On reply Queue And uses default exchange.
    • Client specifies the reply Queue In “reply-to” Property of the request.
    • The client may send multiple requests which may be processed by different servers It will receive multiple corresponding responses. In order to verify, which response correspond to which request. It tags metadata on to Request. The server then tags same metadata also on reply too. Thus client can coordinate which response belongs to which request.
    • The correlation ID or message ID properties Are used to convey this information.
      • Client sends a unique message_id to server With correlation ID.
      • The server populates the correlation ID with same message. It received in request on its reply.
      • The server may populate Only the correlation ID Received in request, depending on the requirements.
    • Example
  • Exchange to exchange routing
    • Exchanges can also be Bound to other exchanges.
    • Messages can be routed Flexible in most complex way possible.
    • A message may need to be routed to queue or a Fan out exchange Based on conditions.
    • We can link a type of exchange to another type of exchange, for example, we can link a direct exchange to a fanout exchange.
    • We can chain exchanges Together.
  • Headers exchange
    • The header exchange Like direct exchange, uses a parameter to decide where to route the message.
    • It uses the contents of the headers table to decide where to route the message rather than routing key.
    • The message is published with a number of headers. These headers are sent in the basic properties. Data structure.
    • Headers is basically a list of key value pairs strings.
    • The headers exchange is then bound to one or more queues.
    • We use binding arguments which are set when binding headers exchange to the queue.
    • The binding argument is a list of key value pairs
      • The binding argument ‘x-match’ decides Weather the queue connected to the exchange should receive the message or not.
        • For example, if the value of ‘x-match’ is “any” then if the message Contains a key value pair, which matches any of the key value pair in binding arguments, then messages routed to the queue.
        • If the value of ‘x-match’ Key is ‘all’ Then it will only route if the message contains all the key value pairs in headers as defined in binding arguments. The headers may contain more Arguments, but it should have all the key values as defined in the binding arguments.
  • Consistent hashing exchange
    • The consistent hashing exchange comes bundled as part of the rabbit MQ Default installation, but as a separate Plug-in that needs to be installed.
    • The goal of the consistent hashing exchange is to distribute possible equally messages that are sent to the exchange across the queues That are connected to it.
    • Used in cases where we do not want rabbit MQ to decide which Queue The message goes to, we want to route message instead based on some property Of the message itself.
    • We may want to weigh our services, or queues So that one Queue Receive more messages than the other queues.
    • There may be a case that some services have higher level of hardware, provision, then others and task can handle more message load comparatively.
    • We may want all the messages that are pertaining to a certain object or instance Sent to same Queue or same service.
    • The binding key, which binds queues to exchange In case of consistent hashing exchange is a simple numeric value.
    • The numeric value indicates what proportion of messages should be sent to the queue in question from the hashing exchange.
    • For example, a numeric value 1 Across all 3 queues treats the queues Equal. However, if we change the value of one of the queues to 2 Means that Queue will receive Twice the messages as compared to other Queue.
    • Rabbit MQ doubles the hash space Of the queue with respect to other queues in above case.
    • When a message comes in it Sets its routing key, this routing key is then hashed by the consistent hashing exchange. The hash is then assigned to one or more queues based on where it falls in the hash Space.
    • Two Messages With same routing key will be routed to same queue.
    • Changing the routing key Slightly could Completely alter the hash and thus could be sent to a completely different queue.
    • In our example, messages are sent with very high entropy Of routing key, overtime, we may end up with an even distribution of messages across our queues Best on the outing key, they are created with.
    • If we add a new queue to our system, then it may screw up our hash space. There will be no guarantee that a message sent with a specific routing key Will be routed To same queue.
  • Publishing options
    • These options allow us to improve speed of our system, increase system Reliability, help us to describe messages in a better way.
    • Some of these options are available in AMQP specification, other have been enhanced by rabbit. MQ team based on specifications available in AMQP.
    • Some of basic AMQP properties, data structure which allow us to set basic properties or as follows
      • Content type
        • Allows our consumers to determine media type of the message body.
        • Like application/json, application/pdf.
      • Content encoding
        • What Encoding standard, our message is using like gzip, compress, deflate etc.
      • Timestamp
        • When message was published
      • AppId or UserId
        • Determine who has published the message
        • If user ID is same as of logged in user, then rabbit MQ Will reject the message.
      • Delivery mode
        • Setting to 1 indicates that message must persist to disk before being sent to consumers.
          • This may improve reliability and ensure published messages are not lost before they are consumed, but this can lead to more latency in our system and maybe consume by more than one Resources.
        • Setting to 0, indicates that it does not need to persist to disk
        • So we must choose wisely delivery mode option.
      • Expiration
        • Message is discarded if not consumed after A certain period of time.
        • Messages set to unix epoch stored in as a string.
        • Using this can help free up memory in our system as out of date or relevant messages are automatically removed and don’t consume resources.
  • Maintaining a balance of speed of our system and reliability.
    • The more fast, the system is the least reliable system is.
    • The more slow, the system is the more reliable it is.
    • Also depends on our implementation and hardware, we are using.
    • Following list states a system which can be defined from fastest with least reliability to slowest with more reliability.
      • No confirmations
        • This is the fastest and least reliable system where we do not have any guarantee of delivery.
        • This may be used where the delivery acknowledgement is least important.
      • Mandatory messages
        • Use of mandatory flag simply tells rabbit to notify if the message has been failed to route.
      • Publisher confirms
        • This is an option at the channel level. All messages sent on the channel with this enabled would be published using this setting. Rabbit MQ respond with an acknowledgement that message is successfully consumed.
      • Transactions
        • Guarantee that a batch of messages have been committed to a queue or else rolled back.
        • Start transaction, publish messages, and commit transaction.
        • If a transaction affects more than one queue Then it will only complete once all messages are pushed to all queues.
      • Persisted messages
        • We set delivery mode to 1.
        • Provides highest reliability and slowest speed with durable queues.
        • We can use alternate exchanges and highly reliable and available queues to create reliable system.
        • We should use well, provisioned hardware While persisting messages.
      • Example how properies can be used.
  • RabbitMQ uses a custom DB to store the messages, the db is usually located at 
    • /var/lib/rabbitmq/mnesia/rabbit@hostname/queues
  • Alternative exchange
    • Alternate exchanges are an extension to AMQP model created by rabbit MQ to route unreadable messages.
    • It involves specifying already existing exchange that the new exchange will route messages too If they are not currently routable.
    • We can Bind an alternate exchange to the main exchange.
    • The alternate exchange Can be of any type. The most common type is fanout exchange.
    • The alternate exchange can then be consumed by a logging or learning service.
    • Example
  • Dead letter exchange
    • A dead letter exchange Is an extension to AMQP specification and like alternate exchange is a normal exchange.
    • hen we declare a queue, we can declare that Queue has an associated dead letter exchange with it.
    • Any message that is routed to that queue, but cannot be delivered to a consumer or expires for some reason can be then sent to the dead letter Exchange.
    • Once the message has been routed To the dead letter exchange, again, we can do whatever we want with it.
    • The dead letter exchange can then be Hooked To a logging or learning service to make our developers, aware that messages are being rejected from Queue and not being delivered to the consumers.
    • The difference between alternate exchange and dead letter exchange is Expired or rejected messages are delivered to a dead letter exchange while unroutable messages are delivered to an alternate exchange.
    • Example
  • Message acknowledgements
    • We consume messages from a consumer with Auto acknowledgement enabled.
    • As soon as the consumer read the message off the queue, it acknowledges to broker that it has read the message. The broker and then remove that message from the queue.So Even if the consumer fails to process the message, the producer will think that message is successfully processed.
    • In many cases, we may want to remove Auto acknowledgement functionality from consumer and explicitly implement it ourselves.
      • We manually send acknowledgement to broker once we have finished consuming a message.
        • We do this using basic acknowledgement
          • It accepts a parameter called as delivery tag. The delivery tag is an auto incrementing number For that consumer, that indicates what message it has now received.
          • The first message consumer received will have a delivery tag zero Followed by one, two, three and so on.
          • The delivery tag can be used when using the basic acknowledgement method frame to tell rabbit MQ, what message we are acknowledging.
          • Delivery tags are scoped per consumer, So When we are sending a basic acknowledgement, it needs to be sent on the same channel that the message is received.
          • Basic acknowledgement takes a Boolean argument Called as multiple, which allows us to acknowledge multiple messages at once.
            • If we receive five messages with delivery tags, 0–4, and we acknowledge fourth message with multiple flag set to true This means we have acknowledged all messages up to and, including four.
        • An alternative to Basic acknowledgement. We also have basic reject method frame which is similar to basic acknowledgement except that it tells rabbit MQ that the message failed to process successfully or we cannot process it.
          • Basic rejects shouldn’t be used as a routing mechanism in our system.
          • They take a delivery tag to tell rabbit MQ, which message we are rejecting.
          • They also have an option to requeue the message.
            • This is a boolean option That tells rabbit MQ that we have rejected the message and we want it to be requeued On the queue It came from.
            • If we have a single consumer of that queue, this can lead to a loop where a message rejected and requeued multiple times to same consumer.
            • If there are multiple consumers, rabbit MQ will make an effort to deliver it to a different consumer.
          • The basic reject does not have a multiple flag.
        • Another acknowledgement called as basic nack Takes a delivery tag But and has both the options of requeue And multiple, which work same way As in basic nack and basic reject. This allows us to reject multiple messages at once the same way we can Acknowledge multiple messages at once.
    • Example
  • Queue Control options
    • Auto delete
      • This allows us to delete a queue automatically if it is No longer needed.
      • A Queue is deleted When there are no consumers consuming from it.
      • Such a Queue Is known as temporary Queue And is often used in applications such as chat application or application that use a request reply pattern where we create queues For time, but is no longer needed once session is closed.
    • Auto expire
      • Expires a Queue Or delete a Queue When a certain amount of time has elapsed or Expired when Queue Is not used.
      • We declare the queue with the X-Expires Argument, we also give the queue a TTL or a time to LIVE, which tells rabbit MQ broker how long we want a queue to stay around after it’s no longer being used.
      • The Queue Expires, after certain condition have passed, including the Queue having no more consumers.
      • Auto delete is set with a flag while auto expire is usually set with X – expires argument.
    • Auto expire message
      • We also have ability to auto expire The messages.
      • This prevents the messages from hanging around on a queue. They haven’t been consumed for too long.
      • If we have a dead letter, exchange, set up and a message on a Queue expires It will be sent to that letter exchange.
      • When we are setting up queue In order to enable auto expire, Messages, we send the argument X message TTL along with the TTL of how long we want the message to survive on the queue for.
    • Max length queue
      • We also have the ability to keep the limited number of messages on a queue at any one time.
      • This allows us to create queues That have a known max length.
      • If more messages or published to the queue, then the max length configured while creating the queue will drop messages from the front of the queue as more messages are added. These removed messages can be dead lettered to enable this.
      • We need to send the argument X max length while creating the queue which configure the maximum message count for that queue.
    • Exclusive
      • We can also create an exclusive Queue Which allows only one consumer at a time. It can only be consumed on same connection and channel that it was declared on and is set usually by a flag while creating the queue.
    • Durable
      • A durable queue is also set using a flag while creating a queue.
      • A durable queue Persist across server restarts.
      • This makes our rarebit MQ deployment more reliable.
    • Other arguments include setting up dead letter exchange We want to use using X-dead-letter Exchange argument.
    • Queues may Span more than one nodes.
  • Examples
  • No comments:

    Post a Comment

    Spring Boot

    What is circular/cyclic dependency in spring boot? When two services are interdependent on each other, that is to start one service, we requ...