DynamoDB Secondary Indexes | Rockset

Introduction

Indexes are a vital a part of correct information modeling for all databases, and DynamoDB isn’t any exception. DynamoDB’s secondary indexes are a strong instrument for enabling new entry patterns on your information.

On this put up, we’ll take a look at DynamoDB secondary indexes. First, we’ll begin with some conceptual factors about how to consider DynamoDB and the issues that secondary indexes clear up. Then, we’ll take a look at some sensible ideas for utilizing secondary indexes successfully. Lastly, we’ll shut with some ideas on when it’s best to use secondary indexes and when it’s best to search for different options.

Let’s get began.

What’s DynamoDB, and what are DynamoDB secondary indexes?

Earlier than we get into use circumstances and greatest practices for secondary indexes, we must always first perceive what DynamoDB secondary indexes are. And to do this, we must always perceive a bit about how DynamoDB works.

This assumes some fundamental understanding of DynamoDB. We’ll cowl the essential factors it is advisable know to know secondary indexes, however in case you’re new to DynamoDB, you might need to begin with a extra fundamental introduction.

The Naked Minimal you Must Learn about DynamoDB

DynamoDB is a novel database. It is designed for OLTP workloads, which means it is nice for dealing with a excessive quantity of small operations — consider issues like including an merchandise to a procuring cart, liking a video, or including a touch upon Reddit. In that method, it will possibly deal with comparable purposes as different databases you may need used, like MySQL, PostgreSQL, MongoDB, or Cassandra.

DynamoDB’s key promise is its assure of constant efficiency at any scale. Whether or not your desk has 1 megabyte of information or 1 petabyte of information, DynamoDB desires to have the identical latency on your OLTP-like requests. This can be a massive deal — many databases will see diminished efficiency as you enhance the quantity of information or the variety of concurrent requests. Nonetheless, offering these ensures requires some tradeoffs, and DynamoDB has some distinctive traits that it is advisable perceive to make use of it successfully.

First, DynamoDB horizontally scales your databases by spreading your information throughout a number of partitions below the hood. These partitions should not seen to you as a person, however they’re on the core of how DynamoDB works. You’ll specify a major key on your desk (both a single aspect, known as a ‘partition key’, or a mix of a partition key and a form key), and DynamoDB will use that major key to find out which partition your information lives on. Any request you make will undergo a request router that can decide which partition ought to deal with the request. These partitions are small — typically 10GB or much less — to allow them to be moved, cut up, replicated, and in any other case managed independently.

Horizontal scalability through sharding is fascinating however is on no account distinctive to DynamoDB. Many different databases — each relational and non-relational — use sharding to horizontally scale. Nonetheless, what is distinctive to DynamoDB is the way it forces you to make use of your major key to entry your information. Relatively than utilizing a question planner that interprets your requests right into a collection of queries, DynamoDB forces you to make use of your major key to entry your information. You might be basically getting a instantly addressable index on your information.

The API for DynamoDB displays this. There are a collection of operations on particular person objects (GetItem, PutItem, UpdateItem, DeleteItem) that let you learn, write, and delete particular person objects. Moreover, there’s a Question operation that permits you to retrieve a number of objects with the identical partition key. In case you have a desk with a composite major key, objects with the identical partition key will probably be grouped collectively on the identical partition. They are going to be ordered in keeping with the type key, permitting you to deal with patterns like “Fetch the latest Orders for a Person” or “Fetch the final 10 Sensor Readings for an IoT Machine”.

For instance, lets say a SaaS software that has a desk of Customers. All Customers belong to a single Group. We would have a desk that appears as follows:

We’re utilizing a composite major key with a partition key of ‘Group’ and a form key of ‘Username’. This permits us to do operations to fetch or replace a person Person by offering their Group and Username. We are able to additionally fetch the entire Customers for a single Group by offering simply the Group to a Question operation.

What are secondary indexes, and the way do they work

With some fundamentals in thoughts, let’s now take a look at secondary indexes. One of the best ways to know the necessity for secondary indexes is to know the issue they clear up. We have seen how DynamoDB partitions your information in keeping with your major key and the way it pushes you to make use of the first key to entry your information. That is all nicely and good for some entry patterns, however what if it is advisable entry your information another way?

In our instance above, we had a desk of customers that we accessed by their group and username. Nonetheless, we might also have to fetch a single person by their e-mail deal with. This sample would not match with the first key entry sample that DynamoDB pushes us in the direction of. As a result of our desk is partitioned by totally different attributes, there’s not a transparent method to entry our information in the way in which we would like. We might do a full desk scan, however that is sluggish and inefficient. We might duplicate our information right into a separate desk with a distinct major key, however that provides complexity.

That is the place secondary indexes are available. A secondary index is principally a completely managed copy of your information with a distinct major key. You’ll specify a secondary index in your desk by declaring the first key for the index. As writes come into your desk, DynamoDB will mechanically replicate the info to your secondary index.

Notice: Every little thing on this part applies to world secondary indexes. DynamoDB additionally supplies native secondary indexes, that are a bit totally different. In virtually all circumstances, you want a worldwide secondary index. For extra particulars on the variations, try this text on selecting a worldwide or native secondary index.

On this case, we’ll add a secondary index to our desk with a partition key of “E mail”. The secondary index will look as follows:

Discover that this is identical information, it has simply been reorganized with a distinct major key. Now, we will effectively lookup a person by their e-mail deal with.

In some methods, that is similar to an index in different databases. Each present an information construction that’s optimized for lookups on a selected attribute. However DynamoDB’s secondary indexes are totally different in a couple of key methods.

First, and most significantly, DynamoDB’s indexes dwell on completely totally different partitions than your predominant desk. DynamoDB desires each lookup to be environment friendly and predictable, and it desires to supply linear horizontal scaling. To do that, it must reshard your information by the attributes you may use to question it.

In different distributed databases, they typically do not reshard your information for the secondary index. They’re going to often simply keep the secondary index for all information on the shard. Nonetheless, in case your indexes do not use the shard key, you are shedding a number of the advantages of horizontally scaling your information as a question with out the shard key might want to do a scatter-gather operation throughout all shards to search out the info you are searching for.

A second method that DynamoDB’s secondary indexes are totally different is that they (usually) copy your entire merchandise to the secondary index. For indexes on a relational database, the index will usually comprise a pointer to the first key of the merchandise being listed. After finding a related document within the index, the database will then have to go fetch the complete merchandise. As a result of DynamoDB’s secondary indexes are on totally different nodes than the principle desk, they need to keep away from a community hop again to the unique merchandise. As a substitute, you may copy as a lot information as you want into the secondary index to deal with your learn.

Secondary indexes in DynamoDB are highly effective, however they’ve some limitations. First off, they’re read-only — you possibly can’t write on to a secondary index. Relatively, you’ll write to your predominant desk, and DynamoDB will deal with the replication to your secondary index. Second, you’re charged for the write operations to your secondary indexes. Thus, including a secondary index to your desk will usually double the full write prices on your desk.

Ideas for utilizing secondary indexes

Now that we perceive what secondary indexes are and the way they work, let’s speak about the best way to use them successfully. Secondary indexes are a strong instrument, however they are often misused. Listed below are some ideas for utilizing secondary indexes successfully.

Attempt to have read-only patterns on secondary indexes

The primary tip appears apparent — secondary indexes can solely be used for reads, so it’s best to purpose to have read-only patterns in your secondary indexes! And but, I see this error on a regular basis. Builders will first learn from a secondary index, then write to the principle desk. This ends in further value and further latency, and you’ll usually keep away from it with some upfront planning.

In the event you’ve learn something about DynamoDB information modeling, you in all probability know that it’s best to consider your entry patterns first. It is not like a relational database the place you first design normalized tables after which write queries to affix them collectively. In DynamoDB, it’s best to take into consideration the actions your software will take, after which design your tables and indexes to help these actions.

When designing my desk, I like to start out with the write-based entry patterns first. With my writes, I am usually sustaining some kind of constraint — uniqueness on a username or a most variety of members in a gaggle. I need to design my desk in a method that makes this simple, ideally with out utilizing DynamoDB Transactions or utilizing a read-modify-write sample that could possibly be topic to race situations.

As you’re employed by means of these, you may typically discover that there is a ‘major’ method to determine your merchandise that matches up along with your write patterns. This can find yourself being your major key. Then, including in further, secondary learn patterns is simple with secondary indexes.

In our Customers instance earlier than, each Person request will seemingly embrace the Group and the Username. This can permit me to lookup the person Person document in addition to authorize particular actions by the Person. The e-mail deal with lookup could also be for much less distinguished entry patterns, like a ‘forgot password’ movement or a ‘seek for a person’ movement. These are read-only patterns, and so they match nicely with a secondary index.

Use secondary indexes when your keys are mutable

A second tip for utilizing secondary indexes is to make use of them for mutable values in your entry patterns. Let’s first perceive the reasoning behind it, after which take a look at conditions the place it applies.

DynamoDB permits you to replace an current merchandise with the UpdateItem
operation. Nonetheless, you can’t change the first key of an merchandise in an replace. The first secret is the distinctive identifier for an merchandise, and altering the first secret is principally creating a brand new merchandise. If you wish to change the first key of an current merchandise, you may have to delete the previous merchandise and create a brand new one. This two-step course of is slower and dear. Typically you may have to learn the unique merchandise first, then use a transaction to delete the unique merchandise and create a brand new one in the identical request.

However, when you’ve got this mutable worth within the major key of a secondary index, then DynamoDB will deal with this delete + create course of for you throughout replication. You possibly can subject a easy UpdateItem request to alter the worth, and DynamoDB will deal with the remaining.

I see this sample come up in two predominant conditions. The primary, and most typical, is when you have got a mutable attribute that you just need to type on. The canonical examples listed here are a leaderboard for a recreation the place individuals are frequently racking up factors, or for a frequently updating checklist of things the place you need to show essentially the most not too long ago up to date objects first. Consider one thing like Google Drive, the place you possibly can type your information by ‘final modified’.

A second sample the place this comes up is when you have got a mutable attribute that you just need to filter on. Right here, you possibly can consider an ecommerce retailer with a historical past of orders for a person. You could need to permit the person to filter their orders by standing — present me all my orders which are ‘shipped’ or ‘delivered’. You possibly can construct this into your partition key or the start of your type key to permit exact-match filtering. Because the merchandise modifications standing, you possibly can replace the standing attribute and lean on DynamoDB to group the objects accurately in your secondary index.

In each of those conditions, shifting this mutable attribute to your secondary index will prevent money and time. You may save time by avoiding the read-modify-write sample, and you may lower your expenses by avoiding the additional write prices of the transaction.

Moreover, observe that this sample matches nicely with the earlier tip. It is unlikely you’ll determine an merchandise for writing based mostly on the mutable attribute like their earlier rating, their earlier standing, or the final time they had been up to date. Relatively, you may replace by a extra persistent worth, just like the person’s ID, the order ID, or the file’s ID. Then, you may use the secondary index to type and filter based mostly on the mutable attribute.

Keep away from the ‘fats’ partition

We noticed above that DynamoDB divides your information into partitions based mostly on the first key. DynamoDB goals to maintain these partitions small — 10GB or much less — and it’s best to purpose to unfold requests throughout your partitions to get the advantages of DynamoDB’s scalability.

This typically means it’s best to use a high-cardinality worth in your partition key. Consider one thing like a username, an order ID, or a sensor ID. There are massive numbers of values for these attributes, and DynamoDB can unfold the visitors throughout your partitions.

Typically, I see individuals perceive this precept of their predominant desk, however then utterly neglect about it of their secondary indexes. Typically, they need ordering throughout your entire desk for a sort of merchandise. In the event that they need to retrieve customers alphabetically, they will use a secondary index the place all customers have USERS because the partition key and the username as the type key. Or, if they need ordering of the latest orders in an ecommerce retailer, they will use a secondary index the place all orders have ORDERS because the partition key and the timestamp as the type key.

This sample can work for small-traffic purposes the place you will not come near the DynamoDB partition throughput limits, nevertheless it’s a harmful sample for a heavy-traffic software. All your visitors could also be funneled to a single bodily partition, and you’ll shortly hit the write throughput limits for that partition.

Additional, and most dangerously, this may trigger issues on your predominant desk. In case your secondary index is getting write throttled throughout replication, the replication queue will again up. If this queue backs up an excessive amount of, DynamoDB will begin rejecting writes in your predominant desk.

That is designed that can assist you — DynamoDB desires to restrict the staleness of your secondary index, so it’ll stop you from a secondary index with a considerable amount of lag. Nonetheless, it may be a shocking state of affairs that pops up whenever you’re least anticipating it.

Use sparse indexes as a worldwide filter

Folks usually consider secondary indexes as a method to replicate all of their information with a brand new major key. Nonetheless, you do not want your entire information to finish up in a secondary index. In case you have an merchandise that does not match the index’s key schema, it will not be replicated to the index.

This may be actually helpful for offering a worldwide filter in your information. The canonical instance I exploit for this can be a message inbox. In your predominant desk, you may retailer all of the messages for a selected person ordered by the point they had been created.

However in case you’re like me, you have got a number of messages in your inbox. Additional, you may deal with unread messages as a ‘todo’ checklist, like little reminders to get again to somebody. Accordingly, I often solely need to see the unread messages in my inbox.

You could possibly use your secondary index to supply this world filter the place unread == true. Maybe your secondary index partition secret is one thing like ${userId}#UNREAD, and the type secret is the timestamp of the message. While you create the message initially, it’ll embrace the secondary index partition key worth and thus will probably be replicated to the unread messages secondary index. Later, when a person reads the message, you possibly can change the standing to READ and delete the secondary index partition key worth. DynamoDB will then take away it out of your secondary index.

I exploit this trick on a regular basis, and it is remarkably efficient. Additional, a sparse index will prevent cash. Any updates to learn messages won’t be replicated to the secondary index, and you may save on write prices.

Slender your secondary index projections to cut back index dimension and/or writes

For our final tip, let’s take the earlier level a bit additional. We simply noticed that DynamoDB will not embrace an merchandise in your secondary index if the merchandise would not have the first key parts for the index. This trick can be utilized for not solely major key parts but in addition for non-key attributes within the information!

While you create a secondary index, you possibly can specify which attributes from the principle desk you need to embrace within the secondary index. That is known as the projection of the index. You possibly can select to incorporate all attributes from the principle desk, solely the first key attributes, or a subset of the attributes.

Whereas it is tempting to incorporate all attributes in your secondary index, this could be a pricey mistake. Keep in mind that each write to your predominant desk that modifications the worth of a projected attribute will probably be replicated to your secondary index. A single secondary index with full projection successfully doubles the write prices on your desk. Every further secondary index will increase your write prices by 1/N + 1, the place N is the variety of secondary indexes earlier than the brand new one.

Moreover, your write prices are calculated based mostly on the scale of your merchandise. Every 1KB of information written to your desk makes use of a WCU. In the event you’re copying a 4KB merchandise to your secondary index, you may be paying the complete 4 WCUs on each your predominant desk and your secondary index.

Thus, there are two methods that you could lower your expenses by narrowing your secondary index projections. First, you possibly can keep away from sure writes altogether. In case you have an replace operation that does not contact any attributes in your secondary index projection, DynamoDB will skip the write to your secondary index. Second, for these writes that do replicate to your secondary index, it can save you cash by lowering the scale of the merchandise that’s replicated.

This could be a difficult steadiness to get proper. Secondary index projections should not alterable after the index is created. In the event you discover that you just want further attributes in your secondary index, you may have to create a brand new index with the brand new projection after which delete the previous index.

Do you have to use a secondary index?

Now that we have explored some sensible recommendation round secondary indexes, let’s take a step again and ask a extra elementary query — must you use a secondary index in any respect?

As we have seen, secondary indexes assist you entry your information another way. Nonetheless, this comes at the price of the extra writes. Thus, my rule of thumb for secondary indexes is:

Use secondary indexes when the diminished learn prices outweigh the elevated write prices.

This appears apparent whenever you say it, however it may be counterintuitive as you are modeling. It appears really easy to say “Throw it in a secondary index” with out desirous about different approaches.

To deliver this residence, let us take a look at two conditions the place secondary indexes won’t make sense.

Numerous filterable attributes in small merchandise collections

With DynamoDB, you typically need your major keys to do your filtering for you. It irks me a bit every time I exploit a Question in DynamoDB however then carry out my very own filtering in my software — why could not I simply construct that into the first key?

Regardless of my visceral response, there are some conditions the place you may need to over-read your information after which filter in your software.

The commonest place you may see that is whenever you need to present a number of totally different filters in your information on your customers, however the related information set is bounded.

Consider a exercise tracker. You may need to permit customers to filter on a number of attributes, corresponding to kind of exercise, depth, length, date, and so forth. Nonetheless, the variety of exercises a person has goes to be manageable — even an influence person will take some time to exceed 1000 exercises. Relatively than placing indexes on all of those attributes, you possibly can simply fetch all of the person’s exercises after which filter in your software.

That is the place I like to recommend doing the mathematics. DynamoDB makes it straightforward to calculate these two choices and get a way of which one will work higher on your software.

Numerous filterable attributes in massive merchandise collections

Let’s change our state of affairs a bit — what if our merchandise assortment is massive? What if we’re constructing a exercise tracker for a fitness center, and we need to permit the fitness center proprietor to filter on the entire attributes we talked about above for all of the customers within the fitness center?

This modifications the state of affairs. Now we’re speaking about lots of and even hundreds of customers, every with lots of or hundreds of exercises. It will not make sense to over-read your entire merchandise assortment and do post-hoc filtering on the outcomes.

However secondary indexes do not actually make sense right here both. Secondary indexes are good for recognized entry patterns the place you possibly can rely on the related filters being current. If we would like our fitness center proprietor to have the ability to filter on a wide range of attributes, all of that are non-compulsory, we would have to create a lot of indexes to make this work.

We talked in regards to the potential downsides of question planners earlier than, however question planners have an upside too. Along with permitting for extra versatile queries, they’ll additionally do issues like index intersections to take a look at partial outcomes from a number of indexes in composing these queries. You are able to do the identical factor with DynamoDB, however it will lead to a number of forwards and backwards along with your software, together with some complicated software logic to determine it out.

When I’ve most of these issues, I typically search for a instrument higher fitted to this use case. Rockset and Elasticsearch are my go-to suggestions right here for offering versatile, secondary-index-like filtering throughout your dataset.

Conclusion

On this put up, we discovered about DynamoDB secondary indexes. First, we checked out some conceptual bits to know how DynamoDB works and why secondary indexes are wanted. Then, we reviewed some sensible tricks to perceive the best way to use secondary indexes successfully and to study their particular quirks. Lastly, we checked out how to consider secondary indexes to see when it’s best to use different approaches.

Secondary indexes are a strong instrument in your DynamoDB toolbox, however they don’t seem to be a silver bullet. As with all DynamoDB information modeling, ensure you fastidiously think about your entry patterns and rely the prices earlier than you bounce in.

Study extra about how you need to use Rockset for secondary-index-like filtering in Alex DeBrie’s weblog DynamoDB Filtering and Aggregation Queries Utilizing SQL on Rockset.