Week 39, 2019 - Step Functions Dynamic Parallelism; S3 Same-Region Replication

By (3 minutes read)

This week AWS released several useful features for existing services. Step Functions now support Dynamic Parallelism, and you can now replicate your S3 buckets within the same region.

Step Functions Dynamic Parallelism

The Step Functions team has been on a roll lately, last month they added nested step functions, and now they’ve added Dynamic Parallelism as well. Now, dynamic parallelism may not be the clearest explanation1 of what this new feature does, so I’ll try to break it down.

What it lets you do is configure parts of your state machines to potentially run in parallel. It was already possible to run various parts of your state machine in parallel, for example sending a notification to Slack and an email at the same time. Those were hardcoded parallelisations though, and generally different functionalities. What dynamic parallelism gives you, however, is the ability to run the same functions in parallel.

The example given in the announcement is an online store, where for each item in an order the Step Function can check the stock and prepare it for shipping simultaneously. While a reasonably simplified example, it does illustrate how you can speed up some processes.

Another effect of it is that it can potentially clean up your configuration as well since you don’t need to build your own loop to iterate over a set of items. The dynamic parallelism, which happily uses the Iterator keyword takes care of that. And when you then combine that with nested step functions, you can end up with a pretty clean and concise state machine2.

S3 Same-Region Replication

Four-and-a-half years ago3, AWS released cross-region replication for S3. And now it’s possible to replicate S3 within a region as well. As a side-effect, they now refer to both functionalities together as simply S3 Replication.

There are a couple of clear use cases for using same-region replication instead of cross-region replication. Possibly the most important one for anyone living in a country without multiple regions is data sovereignty. Sometimes you may store data that for legal reasons isn’t allowed to cross borders. This is a quite common restriction here in Australia, for example, and not having to build my own solution for replication is definitely a good thing.

And no, replicating data to a different account isn’t quite as disaster-proof4 as replicating it to a different region. It will help protect against accidental (or wilful) deletion, however, which is far more likely to happen. Some other use cases are collating logs from different accounts, and replicating production assets to your development environments.

Ambassador Corner

  1. Or it could be me. [return]
  2. Maybe try to avoid too many levels of nesting though. [return]
  3. Yes, I deliberately spelled that out. [return]
  4. Although, let’s be serious, if all 3 Availability Zones in Sydney go down at the same time there is probably a far bigger issue you should focus on than your data being available right away. [return]
comments powered by Disqus