MuleSoft Surat Virtual Meetup#18 - Persistent Queue, Object Store and Persistent Queue
0 likes1,329 views
This document summarizes an agenda for a MuleSoft meetup event on persistent queues, VM connectors, and object stores. It introduces the two organizers, Jitendra Bafna and Nitish Jain, and speaker Jitendra Bafna. The agenda includes an introduction, presentations on persistent queues, VM connectors, and object stores, followed by a live demonstration and trivia quiz. It then provides details on persistent queues, their use cases, recommendations, and limitations. It continues with explanations of VM connectors and their operations. It also outlines the features of object store V1 and V2 and how long data can persist in each.
![[19th June 2021]
16:00 IST (GMT+05:30)
[Surat] MuleSoft Meetup Group
Persistent Queue, VM Connector and
Object Store](https://image.slidesharecdn.com/suratmulesoftmeetup18-210619180243/85/MuleSoft-Surat-Virtual-Meetup-18-Persistent-Queue-Object-Store-and-Persistent-Queue-1-320.jpg)
MuleSoft Surat Virtual Meetup#18 - Persistent Queue, Object Store and Persistent Queue
- 1. [19th June 2021] 16:00 IST (GMT+05:30) [Surat] MuleSoft Meetup Group Persistent Queue, VM Connector and Object Store
- 2. 2 Organizers Jitendra Bafna Senior Solution Architect Capgemini About the organizer: ➢ Working as Senior Solution Architect at Capgemini. ➢ Surat MuleSoft Meetup Leader and MuleSoft Ambassador. ➢ 12.5+ Years of Experience in Integrations and API Technologies. ➢ Certified MuleSoft Integration Architect and platform Architect.
- 3. 3 Organizers Nitish Jain Consultant IBM About the organizer : ➢ Working as Consultant at IBM. ➢ Surat MuleSoft Meetup Leader. ➢ 2.5+ Years of Experience in Integrations and API Technologies. ➢ Certified MuleSoft Developer and Platform Architect.
- 4. 4 Speakers Jitendra Bafna Senior Solution Architect Capgemini About the speaker: ➢ Working as Senior Solution Architect at Capgemini. ➢ Surat MuleSoft Meetup Leader and MuleSoft Ambassador. ➢ 12.5+ Years of Experience in Integrations and API Technologies. ➢ Certified MuleSoft Integration Architect and platform Architect.
- 5. Agenda 5 ❖ Introduction ❖Persistent Queue ❖VM Connector – Request Reply and Reliability Pattern ❖ Object Store ❖Live Demonstration ❖Trivia Quiz
- 6. Persistent Queue CloudHub Persistent Queues is a Cloud Service that allows messages that are published to VM queues to be stored externally to the application. This feature is directly tied into the CloudHub Fabric (Multi- worker) feature and is part of CloudHub's high availability infrastructure. When CloudHub Persistent Queues is enabled, it allows a developer to design VM queues in such a way that payloads can be shared across different workers, for the purposes of load distribution. In the case of a failure of one of the workers in a multi-worker deployment, messages on the queues are able to be consumed by other working workers, ensuring continuity of service. 6
- 7. Persistent Queue – Use Cases CloudHub Persistent Queues is a specific feature for a specific task. ● It's design is solely to enable the capability to share payloads across multi-worker application deployments. It should only be used for this purpose. ● Enabled for this use-case, CloudHub Persistent Queues will improve HA resiliency and performance due to the load distribution and sharing capabilities. 7
- 8. Persistent Queue – Recommendations Reliability Patterns. CloudHub Persistent Queues should not be confused with traditional Mule Persistent VM queues. While Mule Persistent VM queues were intended to be part of a reliability pattern solution, this was never the intention for CloudHub Persistent Queues. The strong recommendation is that CloudHub Persistent Queues should not be considered as the feature to use when designing a Reliability pattern in CloudHub. With that said, CloudHub Persistent queues should never be enabled for single worker applications. For a true reliability pattern, the queues used should be an enterprise grade message broker, examples include Anypoint MQ, ActiveMQ, IBM MQ, or other equivalent JMS or AMQP broker. 8
- 9. Persistent Queue – Limitations ● Performance. Applications with CloudHub Persistent Queues will process messages up-to or more than 10x slower than not using CloudHub Persistent Queues. The performance is further significantly decreased if using the Encryption option. The reason for this is because CloudHub Persistent Queues uses a network attached store, which has both network latency and physical limitations in terms of read / write speed. When disabled, VM queues will be stored in-memory, and has almost instantaneous latency. ● Message retention. CloudHub Persistent Queues will retain messages for 4 days. This retention period is not customisable. Because of this CloudHub Persistent Queues should not be used for any long term storage. ● Batch Jobs. Mule Batch Jobs internally make use of queues. When CloudHub Persistent Queues is enabled for an application with Batch processors, severe performance degradation will occur. Batch also has its own set of limitations incurred if using CloudHub Persistent Queues. The significant one being that message reliability is not guaranteed. Message loss can still occur in the case of application restart (intentional or unintentional restarts). This limitation means there is no benefit from having CloudHub Persistent Queues enabled for a batch application. 9
- 10. VM Connector VM Connector manages the inter or intra communication via persistent or transient asynchronous queue. ● Transient Queue is faster than Persistent Queue but not reliable. ● Persistent Queue is reliable but slower than Transient Queue. VM Connector- Use Cases ● One of the use cases for using VM connector when you want to pass message between flows. ● Distribute work across a cluster. ● Communicate with different apps that are running in the same Mule domain ● Perform simple queueing that does not justify the use of a full JMS broker 10
- 11. VM Connector VM Queue Operations ● Listener ● Publish ● Consume ● Publish Consumer 11
- 12. Live Demonstration • Publish and Consumer messages Request Reply Pattern • Reliability and Transactions • Persistent Queue
- 13. Object Store Object Store enables you ● Retrieve a list of object stores and keys associated with an application. ● Store and retrieve key-value pairs in an object store. ● Delete key-value pairs from an object store. Object Store Connector is a Mule component that allows for simple key-value storage. Although it can serve a wide variety of use cases, it is mainly design for: ● Storing synchronization information, such as watermarks. ● Storing temporary data ● Storing User Information 13
- 14. Object Store V2 Object Store v2 lets CloudHub applications store data and states across batch processes, Mule components and applications, from within an application or by using the Object Store REST API. ● On-premises Mule applications can access Object Store v2 only via the Object Store REST API. ● You can configure a Mule app to use the Object Store REST API to store and retrieve values from an object store in another Mule app. 14
- 15. Object Store V2 - Features ● Supports both static and rolling TTL (time to live) for data persistence. ● Allows for an unlimited number of entries. There is no limit on the total size of v2 object-stores. ● Stores values up to 10 MB (when Base64 encoded) in size. You can estimate base64 size of a payload as follows: CEILING(base10size * 1024/3) * 4, where base10size is object size in bytes. + Example: A base10 payload size of 7.5 MB converts to 10.24 MB base64. ● Is available in all supported regions and availability zones within each region. ● Is co-located in the same region as your workers. For example, workers hosted in Singapore would use Object Store v2 hosted in Singapore. ● Provides a Mule connector and REST interface for access by external applications. ● Provides end-to-end secure TLS-transport. ● Encrypts persistent storage to FIPS 140-2 compliant standards. 15
- 16. Object Store V2 - Features ● The Object Store Connector works for both Object Store version 1 and 2. ● Object Store v2 enforces a maximum data persistence of 30 days from time of creation. ● The free Object Store v2 limits usage to 10 transactions per second (TPS) per application. ● Premium add-on customers are allowed up to 100 TPS per app. ● To increase your TPS limit, contact the MuleSoft account teams 16
- 17. How long can data persist in Object Store v2? The maximum TTL (time to live) is 2592000 seconds (30 days). The entryTtl value, specified in the global configuration parameters for the connector, determines when to evict key-value pairs from the store. The TTL can be either rolling or static. Rolling Accessing the data extends the TTL for 30 days. The minimum rolling TTL is 1 second. The maximum rolling TTL is unlimited as long as the data continues to be accessed. Static The value of entryTtl determines when the data is evicted. The minimum static TTL is 1 second. The maximum (and default) static TTL is 2592000 seconds (30 days). If the entryTtl value is greater than 30 days, TTL is set to the maximum value of 2592000 seconds (30 days) without returning an error. Updating the data (for example, by overwriting the key) resets the TTL. 17
- 18. Object Store V1 V/S Object Store V2 18 Object Store V1 Object Store V2 Maximum of 100,000 key:value pairs per application 768 byte key size 1 MB value size 1 GB of data per application Supported by ObjectStore Connector, but attributes like entryTTL, Expiration Interval, Max Entries are _not_ configurable. No API Rate limit. • Key time to live is 30 days, and extended by 30 days if accessed. • Unlimited number of entries, unlimited total size of object stores. • Max value size 10 MB. • Max key length 256 characters. • Supported in each deployment region, and is co-located with your CloudHub application • Secured end-to-end with TLS-transport • Encrypts persistent storage to FIPS 140-2 compliant standards • Supported by Object Store Connector, but attributes like Expiration Interval, Max Entries are _not_ configurable. • Standard version provides a maximum API request rate limit of 10TPS. • Premium OSv2 provides a maximum API request rate limit of 100TPS (upgradable through Object Store Premium subscription purchase).
- 19. 19
- 20. Get ready to WIN a Special Gift from MuleSoft Community Quiz Time
- 21. Introduce yourself to your neighbor Networking time
- 22. Thank you