Stream multi-tenant information with Amazon MSK
Actual-time information streaming has develop into distinguished in as we speak’s world of instantaneous digital experiences. Fashionable software program as a service (SaaS) functions throughout all industries rely increasingly on repeatedly generated information from completely different information sources similar to net and cell functions, Web of Issues (IoT) units, social media platforms, and ecommerce websites. Processing these information streams in actual time is vital to delivering responsive and personalised options, and maximizes the worth of information by processing it as near the occasion time as doable.
AWS helps SaaS distributors by offering the constructing blocks wanted to implement a streaming software with Amazon Kinesis Information Streams and Amazon Managed Streaming for Apache Kafka (Amazon MSK), and real-time processing functions with Amazon Managed Service for Apache Flink.
On this submit, we have a look at implementation patterns a SaaS vendor can undertake when utilizing a streaming platform as a way of integration between inner elements, the place streaming information will not be instantly uncovered to 3rd events. Specifically, we deal with Amazon MSK.
Streaming multi-tenancy patterns
When constructing streaming functions, you need to take the next dimensions under consideration:
- Information partitioning – Occasion streaming and storage must be remoted on the applicable stage, bodily or logical, primarily based on tenant possession
- Efficiency equity – The efficiency coupling of functions processing streaming information for various tenants should be managed and restricted
- Tenant isolation – A strong authorization technique must be put in place to ensure tenants can entry solely their information
Underpinning all interactions with a multi-tenant system is the idea of SaaS id. For extra data, discuss with SaaS Structure Fundamentals.
SaaS deployment fashions
Tenant isolation will not be elective for SaaS suppliers, and tenant isolation approaches will differ relying in your deployment mannequin. The mannequin is influenced by enterprise necessities, and the fashions aren’t mutually unique. Commerce-offs should be weighed throughout particular person providers to attain a correct steadiness of isolation, complexity, and price. There is no such thing as a common resolution, and a SaaS vendor must rigorously weigh their enterprise and buyer wants towards three isolation methods: silo, pool and bridge (or mixtures thereof).
Within the following sections, we discover these deployment fashions throughout information isolation, efficiency equity, and tenant isolation dimensions.
Silo mannequin
The silo mannequin represents the best stage of information segregation, but in addition the best operating value. Having a devoted MSK cluster per tenant will increase the chance of overprovisioning and requires duplication of administration and monitoring tooling.
Having a devoted MSK cluster per tenant makes positive tenant information partitioning happens on the disk stage when utilizing an Amazon MSK Provisioned mannequin. Each Amazon MSK Provisioned and Serverless clusters assist server-side encryption at relaxation. Amazon MSK Provisioned additional permits you to use a buyer managed AWS Key Administration Service (AWS KMS) key (see Amazon MSK encryption).
In a silo mannequin, Kafka ACL and quotas will not be strictly required except your small business necessities require them. Efficiency equity is assured as a result of solely a single tenant shall be utilizing the assets of the whole MSK cluster and are devoted to functions producing and consuming occasions of a single tenant. This implies spikes of visitors on a selected tenant can’t impression different tenants, and there’s no threat of cross-tenant information entry. As a disadvantage, having a provisioned cluster per tenant requires a right-sizing train per tenant, with a better threat of overprovisioning than within the pool or bridge fashions.
You possibly can implement tenant isolation the MSK cluster stage with AWS Id and Entry Administration (IAM) insurance policies, creating per-cluster credentials, relying on the authentication scheme in use.
Pool mannequin
The pool mannequin is the best mannequin the place tenants share assets. A single MSK cluster is used for all tenants with information cut up into matters primarily based on the occasion kind (for instance, all occasions associated to orders go to the subject orders), and all tenant’s occasions are despatched to the identical subject. The next diagram illustrates this structure.
This mannequin maximizes operational simplicity, however reduces the tenant isolation choices accessible as a result of the SaaS supplier gained’t be capable of differentiate per-tenant operational parameters and all obligations of isolation are delegated to the functions producing and consuming information from Kafka. The pool mannequin additionally doesn’t present any mechanism of bodily information partitioning, nor efficiency equity. A SaaS supplier with these necessities ought to contemplate both a bridge or silo mannequin. For those who don’t have necessities to account for parameters similar to per-tenant encryption keys or tenant-specific information operations, a pool mannequin affords decreased complexity and could be a viable choice. Let’s dig deeper into the trade-offs.
A typical technique to implement client isolation is to establish the tenant inside every occasion utilizing a tenant ID. The choices accessible with Kafka are passing the tenant ID both as occasion metadata (header) or a part of the payload itself as an express area. With this method, the tenant ID shall be used as a standardized area throughout all functions inside each the message payload and the occasion header. This method can cut back the chance of semantic divergence when elements course of and ahead messages as a result of occasion headers are dealt with otherwise by completely different processing frameworks and might be stripped when forwarded. Conversely, the occasion physique is commonly forwarded as a single object and no contained data is misplaced except the occasion is explicitly remodeled. Together with the tenant ID within the occasion header as effectively could simplify the implementation of providers permitting you to specify tenants that should be recovered or migrated with out requiring the supplier to deserialize the message payload to filter by tenant.
When specifying the tenant ID utilizing both a header or as a area within the occasion, client functions will be unable to selectively subscribe to the occasions of a selected tenant. With Kafka, a client subscribes to a subject and receives all occasions despatched to that subject of all tenants. Solely after receiving an occasion will the patron will be capable of examine the tenant ID to filter the tenant of curiosity, making entry segregation nearly unattainable. This implies delicate information should be encrypted to ensure a tenant can’t learn one other tenant’s information when viewing these occasions. In Kafka, server-side encryption can solely be set on the cluster stage, the place all tenants sharing a cluster will share the identical server-side encryption key.
In Kafka, information retention can solely be set on the subject. Within the pool mannequin, occasions belonging to all tenants are despatched to the identical subject, so tenant-specific operations like deleting all information for a tenant won’t be doable. The immutable, append-only nature of Kafka solely permits a whole subject to be deleted, not selective occasions belonging to a selected tenant. If particular buyer information within the stream requires the precise to be forgotten, similar to for GDPR, a pool mannequin won’t work for that information and silo needs to be thought-about for that particular information stream.
Bridge mannequin
Within the bridge mannequin, a single Kafka cluster is used throughout all tenants, however occasions from completely different tenants are segregated into completely different matters. With this mannequin, there’s a subject for every group of associated occasions per tenant. You possibly can simplify operations by adopting a subject naming conference similar to together with the tenant ID within the subject identify. This may virtually create a namespace per tenant, and in addition permits completely different directors to handle completely different tenants, setting permissions with a prefix ACL, and avoiding naming clashes (for instance, occasions associated to orders for tenant 1 go to tenant1.orders and orders of tenant 2 go to tenant2.orders). The next diagram illustrates this structure.
With the bridge mannequin, server-side encryption utilizing a per-tenant key will not be doable. Information from completely different tenants is saved in the identical MSK cluster, and server-side encryption keys will be specified per cluster solely. For a similar purpose, information segregation can solely be achieved at file stage, as a result of separate matters are saved in separate recordsdata. Amazon MSK shops all matters inside the identical Amazon Elastic Block Retailer (Amazon EBS) quantity.
The bridge mannequin affords per-tenant customization, similar to retention coverage or max message measurement, as a result of Kafka permits you to set these parameters per subject. The bridge mannequin additionally simplifies segregating and decoupling occasion processing per tenant, permitting a stronger isolation between separate functions that course of information of separate tenants.
To summarize, the bridge mannequin affords the next capabilities:
- Tenant processing segregation – A client software can selectively subscribe to the matters belonging to particular tenants and solely obtain occasions for these tenants. A SaaS supplier will be capable of delete information for particular tenants, selectively deleting the matters belonging to that tenant.
- Selective scaling of the processing – With Kafka, the utmost variety of parallel customers is set by the variety of partitions of a subject, and the variety of partitions will be set per subject, and subsequently per tenant.
- Efficiency equity – You possibly can implement efficiency equity utilizing Kafka quotas, supported by Amazon MSK, stopping the providers processing a very busy tenant to devour too many cluster assets, on the expense of different tenants. Consult with the next two-part sequence for extra particulars on Kafka quotas in Amazon MSK, and an instance implementation for IAM authentication.
- Tenant isolation – You possibly can implement tenant isolation utilizing IAM entry management or Apache Kafka ACLs, relying on the authentication scheme that’s used with Amazon MSK. Each IAM and Kafka ACLs permit you to management entry per subject. You possibly can authorize an software to entry solely the matters belonging to the tenant it’s imagined to course of.
Commerce-offs in a SaaS setting
Though every mannequin gives completely different capabilities for information partitioning, efficiency equity, and tenant isolation, in addition they include completely different prices and complexities. Throughout planning, it’s essential to establish what trade-offs you’re prepared to make for typical clients, and present a tier construction to your consumer subscriptions.
The next desk summarizes the supported capabilities of the three fashions in a streaming software.
| . | Pool | Bridge | Silo |
| Per-tenant encryption at relaxation | No | No | Sure |
| Can implement proper to be forgotten for single tenant | No | Sure | Sure |
| Per-tenant retention insurance policies | No | Sure | Sure |
| Per-tenant occasion measurement restrict | No | Sure | Sure |
| Per-tenant replayability | Sure (should implement with logic in customers) | Sure | Sure |
Anti-patterns
Within the bridge mannequin, we mentioned tenant segregation by subject. An alternate could be segregating by partition, the place all messages of a given kind are despatched to the identical subject (for instance, orders), however every tenant has a devoted partition. This method has many disadvantages and we strongly discourage it. In Kafka, partitions are the unit of horizontal scaling and balancing of brokers and customers. Assigning partitions per tenants can introduce unbalancing of the cluster, and operational and efficiency points that shall be laborious to beat.
Some stage of information isolation, similar to per-tenant encryption keys, might be achieved utilizing client-side encryption, delegating any encryption or description to the producer and client functions. This method would permit you to use a separate encryption key per tenant. We don’t advocate this method as a result of it introduces a better stage of complexity in each the patron and producer functions. It could additionally stop you from utilizing many of the commonplace programming libraries, Kafka tooling, and most Kafka ecosystem providers, like Kafka Join or MSK Join.
Conclusion
On this submit, we explored three patterns that SaaS distributors can use when architecting multi-tenant streaming functions with Amazon MSK: the pool, bridge, and silo fashions. Every mannequin presents completely different trade-offs between operational simplicity, tenant isolation stage, and price effectivity.
The silo mannequin dedicates full MSK clusters per tenant, providing a simple tenant isolation method however incurring a better upkeep and price per tenant. The pool mannequin affords elevated operational and cost-efficiencies by sharing all assets throughout tenants, however gives restricted information partitioning, efficiency equity, and tenant isolation capabilities. Lastly, the bridge mannequin affords a very good compromise between operational and cost-efficiencies whereas offering a very good vary of choices to create strong tenant isolation and efficiency equity methods.
When architecting your multi-tenant streaming resolution, rigorously consider your necessities round tenant isolation, information privateness, per-tenant customization, and efficiency ensures to find out the suitable mannequin. Mix fashions if wanted to seek out the precise steadiness for your small business. As you scale your software, reassess isolation wants and migrate throughout fashions accordingly.
As you’ve seen on this submit, there is no such thing as a one-size-fits-all sample for streaming information in a multi-tenant structure. Fastidiously weighing your streaming outcomes and buyer wants will assist decide the proper trade-offs you may make whereas ensuring your buyer information is safe and auditable. Proceed your studying journey on SkillBuilder with our SaaS curriculum, get hands-on with an AWS Serverless SaaS workshop or Amazon EKS SaaS workshop, or dive deep with Amazon MSK Labs.
Concerning the Authors
Emmanuele Levi is a Options Architect within the Enterprise Software program and SaaS staff, primarily based in London. Emanuele helps UK clients on their journey to refactor monolithic functions into fashionable microservices SaaS architectures. Emanuele is principally occupied with event-driven patterns and designs, particularly when utilized to analytics and AI, the place he has experience within the fraud-detection business.
Lorenzo Nicora is a Senior Streaming Answer Architect serving to clients throughout EMEA. He has been constructing cloud-native, data-intensive methods for over 25 years, working throughout industries, in consultancies and product firms. He has leveraged open-source applied sciences extensively and contributed to a number of initiatives, together with Apache Flink.
Nicholas Tunney is a Senior Associate Options Architect for Worldwide Public Sector at AWS. He works with World SI companions to develop architectures on AWS for shoppers within the authorities, nonprofit healthcare, utility, and schooling sectors. He’s additionally a core member of the SaaS Technical Subject Neighborhood the place he will get to satisfy shoppers from everywhere in the world who’re constructing SaaS on AWS.
