{ "version": "https://jsonfeed.org/version/1", "title": "ig.nore.me", "home_page_url": "https://ig.nore.me/", "feed_url": "https://ig.nore.me/feed.json", "favicon": "https://ig.nore.me/favicon-192.png", "author": { "url": "https://www.twitter.com/ArjenSchwarz", "name": "Arjen Schwarz", "avatar": "https://ig.nore.me/img/profilepicsmall.jpg" }, "items": [ { "id": "https://ig.nore.me/weekly-notes/week-6-2019/", "title": "Week 6, 2019 - Facebook and Google's VPNs; FaceTime Bug; Small Things", "url": "https://ig.nore.me/weekly-notes/week-6-2019/", "content_html": "\n\n

A few words about the whole deal regarding Facebook’s (and Google’s) datamining VPNs, the terrible FaceTime bug, and a couple small items of interest.

\n\n

Facebook and Google’s VPNs

\n\n

I had something written about the whole Facebook and Google VPN debacles last week1. In the end though, while it had some snarky comments, I don’t feel like it added much to what has been written elsewhere. My feelings, unsurprisingly, come down to: Datamining VPNs are bad and punishing that is good2, but Apple’s ability to do so is worrying.

\n\n

So, that’s clearly not really worth spending a lot of words on. Data on what people do on their phones is extremely valuable to these companies so it’s not surprising3 that they want to collect it any way they can.

\n\n

FaceTime Bug

\n\n

Apple however also had a different issue last week. A very severe bug in FaceTime allowed the use of its group functionality to listen in on, and even see, people before they answered the call. Not exactly what I’d call a happy situation.

\n\n

Within a couple hours of this being reported, Apple shut down the group call part of the service4 while working on a fix, which hasn’t been finished yet. This sounds like a good thing, and it is. Except for the little part where someone tried to report this a week earlier and Apple’s processes prevented that report from reaching the right people. Oops.

\n\n

Small Things

\n\n

Let’s round this up with a couple of small things I found interesting.

\n\n

Homebrew, the package manager for macOS, released version 2. If you’re not familiar with Homebrew, it basically allows you to easily install a lot of tools a developer5 might want to use on their Mac. Except, the big thing about version 2 is that it now also runs on Linux and even Windows Subsystem for Linux. This is pretty cool, and certainly worth a mention.

\n\n

Next, AWS released a new icon set for their architecture diagrams. This one invalidates the one they released several months ago, and which I was planning to write about because it was well… terrible. I don’t know who came up with the idea to make all icons the same shape with a thin white figure on a black background, but it made diagrams hard to quickly understand. This new release mostly fixes that by reintroducing this amazing thing called colour6. Of course it also includes all the newly released resources, so I can finally put a proper Transit Gateway into my diagrams.

\n\n

Lastly, I just want to point out that, as one of the organisers, I wrote a summary of the last AWS User Group meetup here in Melbourne which includes the videos of the speakers. If you want an idea of what’s going on at these meetups, read that, watch the videos, and then join us for the next one7.

\n
\n\n
\n\n
    \n
  1. In case you missed it, please look at the linked article. It’s not a pretty situation.\n [return]
  2. \n
  3. In case you’re not aware, as it was misused for this, Apple revoked their enterprise certificates for iOS which basically broke all internal applications their employees use. These have now been restored though.\n [return]
  4. \n
  5. Although frustrating.\n [return]
  6. \n
  7. Which obviously they also have sole control over. But then, that’s the case with most of these sort of tools.\n [return]
  8. \n
  9. Or anyone who wants to use those tools.\n [return]
  10. \n
  11. Unfortunately, everything is still he same shape but I guess this is as good as it’s gonna get right now.\n [return]
  12. \n
  13. If you’re in Melbourne that is. While it would be cool if you traveled thousands of kilometres to come to the meetup, it probably isn’t worth coming here for that by itself.\n [return]
  14. \n
\n
\n", "summary": "A few words about the whole deal regarding Facebook’s (and Google’s) datamining VPNs, the terrible FaceTime bug, and a couple small items of interest.\nFacebook and Google’s VPNs I had something written about the whole Facebook and Google VPN debacles last week1. In the end though, while it had some snarky comments, I don’t feel like it added much to what has been written elsewhere. My feelings, unsurprisingly, come down to: Datamining VPNs are bad and punishing that is good2, but Apple’s ability to do so is worrying.", "banner_image": "https://ig.nore.me/img/categories/category-weekly-notes-full.jpg", "date_published": "2019-02-06T22:12:46+11:00", "date_modified": "2019-02-06T22:12:46+11:00" }, { "id": "https://ig.nore.me/weekly-notes/week-5-2019/", "title": "Week 5, 2019 - TLS Termination for NLBs; Amazon WorkLink", "url": "https://ig.nore.me/weekly-notes/week-5-2019/", "content_html": "\n\n

TLS Termination for Network Load Balancers might make Classic Load Balancers unnecessary and Amazon WorkLink seems to indicate an even greater focus on the enterprise market.

\n\n

TLS Termination for NLBs

\n\n
\nUpdate January 28, 2019: I orginally made a mistake here concerning the ports an ALB can listen to, apologies if you got the wrong information. This has been corrected in the version below.\n
\n\n

Since AWS introduced first the Application Load Balancers and then the Network Load Balancers, they have been a bit passive-aggressive about moving people away from what are now called Classic Load Balancers1. You can see this for example in the Console where they suggest you upgrade your Classic Load Balancers to something new.

\n\n

Don’t get me wrong here, there are many advantages to the newer types such as faster scaling, so no need to prescale for big events) and the ease with which you can use a single Load Balancer to point at very different backends. However, there was also a limitation2. Which is that until the just released TLS Termination for Network Load Balancers, you couldn’t use SSL/TLS termination on any protocol other than HTTPS.

\n\n

Application Load Balancers only accept traffic on two possible protocols: HTTP and HTTPS3. However, often you might want to use different protocols as well. Usually for communication that is not HTTP based like a communication protocol. The downside with Network Load Balancers, due to their layer 4 nature, was that you couldn’t do TLS termination on them, and therefore you had to manage the certificates on all of your servers or containers.

\n\n

Here you might say that all of that is a solved problem! You can just use something like LetsEncrypt or even install the certificates manually. And true, you can do that. It also makes it harder than when you only need to manage the certificate in a single place so in those cases it was easier to stick to the Classic Load Balancer.

\n\n

With this new release this is now no longer needed and it even has some improvements on the Classic Load Balancer4. One of these is that anything passed through an NLB, whether it’s TLS terminated or not, still has the same origin header. Which means that your application has direct access to the originating IP of the request, instead of needing to check the added X-Forwarded-For header.

\n\n

If you want more information on how this is achieved, there is a Twitter thread describing the under the hood parts of this.

\n\n

My main question now becomes one of time. I believe that this was the main reason to still use a Classic Load Balancer, and if that’s the case I suspect AWS will start being more aggressive about pushing solutions to the newer types. As development on Classic Load Balancers stopped or was deprioritized5 several years ago, the newer ones are probably better for AWS as well.

\n\n

I could be wrong of course, and if you have a use case for Classic Load Balancers that isn’t covered by one of the newer types, I’d love to hear about it.

\n\n

Amazon WorkLink

\n\n

Another thing released this week is Amazon WorkLink. From my understanding, this is a bit of a managed VPN solution for mobile devices. You can set it up so that your corporate users can access your network securely without the need to go through a VPN.

\n\n

How it works is that AWS will render the requested site within a VPC and then forward the rendered result to an app on the user’s mobile device6. This is all secure and presumably quite nice7, but the more interesting part to me is that this follows so closely to the release of Client VPN.

\n\n

These are not the only solutions that seem to be aimed mostly at the bigger enterprises and government agencies either. There have been a lot of SLA announcements8 lately, and even receiving things like the recent PROTECTED certification in Australia and PCI compliance for EKS all help with boosting their status in this regard.

\n\n

I’m sure the certifications and compliances would have happened anyway, but it looks to me that AWS has significantly increased their focus on this part of the market recently9. Even at re:Invent this was a big focus, including the various hybrid cloud solutions culminating in AWS Outposts.

\n
\n\n
\n\n
    \n
  1. Originally these were named Elastic Load Balancers, but that has since become the term in use for all the different types.\n [return]
  2. \n
  3. Not counting it being more complex to configure.\n [return]
  4. \n
  5. In the original version, I incorrectly mentioned that this was limited to ports 80 and 443 as well. As @aneesh pointed out in the comments, this was wrong.\n [return]
  6. \n
  7. And even Application Load Balancer for that matter.\n [return]
  8. \n
  9. I have no information this is the case, but based on the lack of changes it seems logical.\n [return]
  10. \n
  11. iOS 12 only for now, soon they’ll add modern Android support.\n [return]
  12. \n
  13. Or as nice as a first version of something like this can be.\n [return]
  14. \n
  15. I count 10 so far in 2019.\n [return]
  16. \n
  17. Remember, just because it gets announced now doesn’t mean it hasn’t been in development for a long time.\n [return]
  18. \n
\n
\n", "summary": "TLS Termination for Network Load Balancers might make Classic Load Balancers unnecessary and Amazon WorkLink seems to indicate an even greater focus on the enterprise market.\nTLS Termination for NLBs Update January 28, 2019: I orginally made a mistake here concerning the ports an ALB can listen to, apologies if you got the wrong information. This has been corrected in the version below. Since AWS introduced first the Application Load Balancers and then the Network Load Balancers, they have been a bit passive-aggressive about moving people away from what are now called Classic Load Balancers1.", "banner_image": "https://ig.nore.me/img/categories/category-weekly-notes-full.jpg", "date_published": "2019-01-28T19:59:07+11:00", "date_modified": "2019-01-28T22:50:11+11:00" }, { "id": "https://ig.nore.me/2019/01/github-actions-awesome-automation/", "title": "GitHub Actions - Awesome Automation", "url": "https://ig.nore.me/2019/01/github-actions-awesome-automation/", "content_html": "\n\n

In October, GitHub announced GitHub Actions, their upcoming integrated solution for running automation triggered by things you do on GitHub. I would call it CI/CD, but that’s only part of what it can do. I’ve had beta access for about 2 months now, but didn’t really play around with it until around the Christmas period.

\n\n

Background

\n\n

You may already know all of this, but let’s start with a bit of background anyway. GitHub Actions lets you create workflows that consist of actions1, and they can be triggered by various events such as opening or closing of pull requests, creating or deleting branches, forking a repository, or even creating a wiki page. The number of events that can cause a workflow to trigger is a bit insane2 and it’s the main reason I don’t want to call it a CI/CD tool. That just sounds too limiting.

\n\n

A note on the naming as this can get a bit confusing. The feature itself is called GitHub Actions, but what you actually build is called a workflow that in turns consists of actions. I’ve noticed that people tend to mix and match the naming a bit, but in this post I’ll try to adhere to the above3.

\n\n

Jessie Frazelle wrote an excellent and entertaining blogpost about how a GitHub Action behaves, and I highly recommend that you read that for a concise description of how it works. I’ll cover some of the same ground, but you should really read that first.

\n\n

During the beta period most of the functionality is only available for private repos. On 19 December, GitHub opened up the push event to public repos and until the recent announcement of free private repos that is what I was playing around with. As I don’t have any personal repos at the moment4, I ended up duplicating one of my public repos so I could use the full set of features.

\n\n

Building a workflow

\n\n

So, let’s start with building a workflow. While I generally like to use code or the CLI to configure things, the visual editor for building GitHub Actions is really good5. A nice and simple drag and drop interface that takes care of the underlying complexity for you. It starts with choosing a name for the workflow and defining what event should trigger it6. Then you can drag from this starting block to one below it to create a new action and choose what that should be.

\n\n

\"\"

\n\n

Before going into more detail on the actions, I just want to point out the section on the left in the editor. There you can see multiple workflows. These are all managed from the same main.workflow file in your .github directory, but in the visual editor they’re nicely separated. Personally, I would have appreciated the ability to have multiple workflow files for separating functionalities, but hopefully that will come in the future7.

\n\n

One small issue with the drag and drop interface is the way it adds actions to the main.workflow file. In particular, it always adds them at the bottom instead of grouping it by workflow. The order doesn’t actually matter so in that regard it’s not an issue, and it makes diffs a bit more readable, but it also means that your workflows can be completely spread out across a file. This is mainly an issue if you mix and match how you edit them, and of course there’s nothing stopping you from organising it again afterwards.

\n\n

Going back to the actions, when you create it through the visual editor you get the option to choose one of the featured ones8 or provide a URL. This URL allows one of three possible types:

\n\n
    \n
  1. An action in a separate GitHub repo.
  2. \n
  3. An action that is in the same repo as the project.
  4. \n
  5. A public Docker image, for example on Docker Hub9.
  6. \n
\n\n

Of these, the first 2 are basically the same except for the path. Personally I think that unless you have a very good reason, you should put your actions in a separate repository so you can easily reuse it. Using a public Docker image is certainly an interesting option as well, but playing around with it I got mixed results and just ended up using actions.

\n\n

Once we’ve picked our source, we can then provide secrets and environment variables or even override the runs and args of the action. These two are the values of the endpoint and command in Docker respectively.

\n\n

After this you just connect the next action in your chain (or a parallel one) and continue building your workflow until you’re done.

\n\n

Now, even if you decide to use drag and drop to build all of your workflows, I recommend looking through the source code as it reveals a couple of things and how they work.

\n\n

Diving into the workflow

\n\n

In the below example I’m using the same file I was editing above. You may notice a couple of interesting things in here.

\n\n

\"\"

\n\n

First, while I didn’t mention it above, you need to define the version of the action you wish to run. Usually you would pick @master, but this is GitHub so you can also pick a branch, tag, or even a specific commit. Looking closely you can see that I generally used @master, but for the filter action10 I dragged it in from the featured actions so GitHub gave me a commit specific one. I completely agree with doing so for featured actions as it prevents a malicious actor from changing the action. In fact, in general it’s probably a good idea to use something that’s not master unless you trust the source.

\n\n

The syntax itself is simple and clear, but takes some getting used to as it flows backwards. The workflow resolves the final nodes11 and each of these nodes will then say what it depends on, or needs. It’s a clean way of dealing with parallel paths and means that the position in the file isn’t important. But let’s look a bit closer at the actions themselves now.

\n\n

The power of actions

\n\n

In essence an action is a Docker image, but you don’t have to push it up to a repository. All you need is a Dockerfile. And I mean that literally, the smallest action I’ve created consists of only a Dockerfile that installs what I need in the container12. Generally you will need more than that though, and the recommended way is to have an entrypoint.sh file that is used as the default command.

\n\n

While actions are expected to have a default entrypoint and command, you can always override that. So you may build or use an action that by default will run a certain command, but if you have a Makefile in your project that has the command ready to go it might be easier to use that instead of typing it out again in your workflow.

\n\n

So, if you don’t need to provide built Docker images you’ll likely realise that this gets built at the time an action runs. This makes it a bit slower, although the builds are pretty fast, but if that’s not fast enough you can of course just use images. Having it compile only on run does mean you might want to be a bit more careful about using :latest in your FROM statements or exactly what version of a dependency you’re installing.

\n\n

Unsurprisingly, one of the biggest strengths of GitHub Actions is its integration with GitHub. By default you can provide every action with a GITHUB_TOKEN secret that the action can then use for any interaction with GitHub that may require authentication. I’m sure you can think of ways this can be useful, but as an example I use it in several of my actions to use the GitHub API to find the latest version of an application and pull that in. Yes, you can do that without authentication, but then you quickly run into API rate limits which is never fun.

\n\n

And all of this also means that you can easily fork or copy every action you find and make your own version of it. Of the actions I’ve made, most are inspired in one way or another by another action. Some are straight copies with a small change, while others may only use similar structures.

\n\n

The full experience

\n\n

As I said, I’ve mostly used GitHub Actions for my public projects but that doesn’t mean I’m not looking forward to everything else it can do. Especially when you get to projects with a lot of collaboration in your organisation, there are good opportunities there. For example, I can easily see a vulnerability alert in a Ruby project triggering the following workflow:

\n\n
    \n
  1. Create a new branch
  2. \n
  3. Run bundle update
  4. \n
  5. Create a pull request with the new Gemfile.lock
  6. \n
  7. Send a message to the company’s #security Slack channel to ask for a review.
  8. \n
\n\n

Keep in mind here that this PR probably won’t trigger GitHub Actions to run as GitHub Actions can’t trigger other GitHub Actions.

\n\n

Or if you’d like another idea, how about a workflow that triggers a wiki page to be updated when you release a new version of your project? Or the other way around? In all honesty, I wouldn’t be surprised at all if someone has already written actions that do some of these things.

\n\n

If you don’t have access yet, you might not be aware that right now it’s not possible to see any GitHub Actions in other accounts. Obviously you can see their main.workflow file, but not the history of the runs. I don’t know if this is a beta limitation or a security feature13, but it means you can’t see what is being run. They do show up in the status checks of Pull Requests, but other than that it’s invisible.

\n\n

Some useful things to get started

\n\n

Usually you’ll have different paths you want your workflow to take based on the branch that triggered it. A common case is that you want to test your code in every branch but only do a deployment from master. This can be achieved with the filter action I referred to earlier. The filter is part of a set of actions provided by GitHub in their actions organisation, specifically it’s part of the actions/bin repository.

\n\n

The filter action uses a third state14 that is defined as neutral in GitHub Actions. This is triggered by exit code 78 so you can use it to implement your own filters as well. When this state is triggered it will cancel all following actions in the chain, marking them as Neutral. This is different from marking them as failed15, and is therefore useful to ensure actions are only triggered in some cases. I recommend reading the README file to see all the ways you can use the filter.

\n\n

The actions/action-builder repo holds several tools aimed at helping you build and test your own actions. Use those. They will make your life easier.

\n\n

Pick fun colours and icons for your actions. You have some control over what it looks like in the UI so there’s no need to limit it to the default. Aside from being more fun, it also helps in being more recognisable.

\n\n

Learn from others. Many excellent (and less so) actions have already been written and they’re worth having a look at to see how it all works. Someone created a list already and while it hasn’t been updated recently, the pull requests have more. Play with these, use them, or copy them as a basis for your own actions.

\n\n

How does it hold up?

\n\n

It’s been almost 3 years since I wrote a similar piece about Bitbucket Pipelines and I ended up with a wish list of things I wanted to see improved16. Where it comes to GitHub Actions I don’t have a list like that. All the things I generally want are available and work really well. Oh, I can say that I would really like the full capabilities in public repos, but I suspect that’s currently because its still in beta.

\n\n

One small downside I can mention is that there’s no way to trigger a build manually. In practice that’s not really a concern though. GitHub Actions is super flexible and you can turn it into anything you want17, and personally I have moved everything over from Wercker18.

\n\n

The biggest questions I’ve got are around availability. Will this become available to everyone for free? Will there be limits based on your account type19? Questions like that. I’m sure that answers to these will come in time though, so for now I recommend to have fun when playing this new cool toy.

\n\n

Let me know if I missed anything obvious, or if you have any questions, and I’ll do my best to answer them.

\n
\n\n
\n\n
    \n
  1. That must have come as a shock.\n [return]
  2. \n
  3. In a very good way.\n [return]
  4. \n
  5. Which has caused me a lot of editing.\n [return]
  6. \n
  7. Some time ago I realised that nobody cares about my unfinished projects, so why go through the trouble of hiding them?\n [return]
  8. \n
  9. And besides, the article above already did it all by code.\n [return]
  10. \n
  11. Except, as mentioned before, with public repos.\n [return]
  12. \n
  13. Because of how polished GitHub Actions looks, I sometimes have to remind myself this is still only a beta.\n [return]
  14. \n
  15. Currently all provided by GitHub, but I assume that might expand after release.\n [return]
  16. \n
  17. I think you can use a private docker repository, but I don’t think there’s a secure way to provide authentication for those so probably not a good idea right now.\n [return]
  18. \n
  19. I’ll come back to this one in a bit, as it’s very useful.\n [return]
  20. \n
  21. Yes, you can have multiple end nodes.\n [return]
  22. \n
  23. It’s a pretty silly one, but I needed a lightweight way to zip things.\n [return]
  24. \n
  25. A case can be made both ways.\n [return]
  26. \n
  27. In addition to the regular success (exit code 0) and fail (every other exit code).\n [return]
  28. \n
  29. Although for a while these would show up as failed in pull requests. That’s now been fixed though.\n [return]
  30. \n
  31. I believe all of those have been implemented since.\n [return]
  32. \n
  33. Unless you’re stuck with Windows jobs.\n [return]
  34. \n
  35. I was looking for something new since their acquisition by Oracle anyway.\n [return]
  36. \n
  37. It seems fair to me that paying users get more.\n [return]
  38. \n
\n
\n", "summary": "In October, GitHub announced GitHub Actions, their upcoming integrated solution for running automation triggered by things you do on GitHub. I would call it CI/CD, but that’s only part of what it can do. I’ve had beta access for about 2 months now, but didn’t really play around with it until around the Christmas period.\nBackground You may already know all of this, but let’s start with a bit of background anyway.", "banner_image": "https://ig.nore.me/img/categories/category-ci-cd-full.jpg", "date_published": "2019-01-24T22:05:34+11:00", "date_modified": "2019-01-24T22:05:34+11:00" }, { "id": "https://ig.nore.me/weekly-notes/week-4-2019/", "title": "Week 4, 2019 - AWS Backup; Go in Cloud Functions", "url": "https://ig.nore.me/weekly-notes/week-4-2019/", "content_html": "\n\n

A single place to configure your backups is available with AWS Backup and Google’s Cloud Functions now supports Go.

\n\n

AWS Backup

\n\n

As AWS Auto Scaling did for auto scaling functionalities, AWS Backup is a convenience wrapper around (mostly) existing backup functionalities. It allows you to create a single backup configuration for a variety of services, ranging from EBS to several database types, and even Storage Gateways.

\n\n

It works by creating a plan, which is really just a schedule, and then assigning resources to it. One thing I noticed though is that apparently this was released without CloudFormation support. My understanding is that new services from AWS shouldn’t be launched anymore without CloudFormation support, but that definitely seems to be the case here and I find that very disappointing. Hopefully, this oversight will be rectified soon.

\n\n

Most interestingly perhaps is that AWS Backup1 introduces support for backing up EFS. Until this introduction, EFS backups were… well… unpleasant… to deal with? There was no native solution, and the while the recommended way according to AWS answers has improved over time2 it wasn’t what I would call a good solution. So it’s great to see that this is included.

\n\n

Of course, while it’s not mentioned in the announcement post, it is limited to the usual short list of regions at the moment. Hopefully, this will expand soon to other regions like Sydney. I just checked, however, and the aforementioned AWS Auto Scaling was released one year ago3 and didn’t become available in Sydney until late December. Let’s hope that’s not a precedent.

\n\n

Go in Cloud Functions

\n\n

Speaking of news from exactly a year ago, it’s been a year since Lambda received support for Go and now GCP’s Cloud Functions supports it as well. Admittedly, it’s only in beta right now but I’m very happy to see that it’s available.

\n\n

It’s only the third language supported in Cloud Functions, after Node.js and Python, and considering Go originated at Google it’s no surprise that it comes before some of the other languages that Lambda supports4. The implementations between Lambda and Cloud Functions are quite different though.

\n\n

First, Cloud Functions specifically uses Go 1.115 whereas Lambda has a go1.x catch-all type. Presumably, sometime after the February release of Go 1.12, Cloud Functions will have a version for this as well. While I don’t have any special insight, I suspect that one reason for the explicit version here6 is that with Cloud Functions you only need to upload your code.

\n\n

With Lambda, Go is different from most other languages as you are expected to upload a binary file instead of only your source code. In Cloud Functions, you don’t need to do this as Google will compile it for you and even install any modules you have configured. It will be interesting to see how this difference works out in practice.

\n
\n\n
\n\n
    \n
  1. And I can already tell you that an overly generic name like this is going to be wonderful to search for in your search engine of choice. After all, who has ever written anything about backing up AWS?\n [return]
  2. \n
  3. The early versions especially were very fiddly and a lot of work.\n [return]
  4. \n
  5. If we go by Weekly Notes, it was exactly a year.\n [return]
  6. \n
  7. Of course, technically speaking Lambda now supports all languages.\n [return]
  8. \n
  9. Which is indeed currently the latest version.\n [return]
  10. \n
  11. Which is likely to be more work by the way, considering a new version of Go is released every 6 months.\n [return]
  12. \n
\n
\n", "summary": "A single place to configure your backups is available with AWS Backup and Google’s Cloud Functions now supports Go.\nAWS Backup As AWS Auto Scaling did for auto scaling functionalities, AWS Backup is a convenience wrapper around (mostly) existing backup functionalities. It allows you to create a single backup configuration for a variety of services, ranging from EBS to several database types, and even Storage Gateways.\nIt works by creating a plan, which is really just a schedule, and then assigning resources to it.", "banner_image": "https://ig.nore.me/img/categories/category-weekly-notes-full.jpg", "date_published": "2019-01-21T20:17:48+11:00", "date_modified": "2019-01-21T20:17:48+11:00" }, { "id": "https://ig.nore.me/weekly-notes/week-3-2019/", "title": "Week 3, 2019 - GitHub Free Private Repos; Amazon DocumentDB; Fargate Pricing", "url": "https://ig.nore.me/weekly-notes/week-3-2019/", "content_html": "\n\n

The holiday period is over and there were a couple of very interesting announcements this week. GitHub now offers free private repositories, AWS has a MongoDB compatible database, and Fargate became a lot cheaper to use.

\n\n

GitHub Free Private Repos

\n\n

There are many things to like about GitHub, but one of the main complaints over the year has been that you had to pay for private repositories. Many of us work on a personal project that isn’t quite ready for public scrutiny but still needed to be stored somewhere that’s not our own laptop. However, we also prefer not to spend money that can otherwise go to a worthy cause like coffee or beer1.

\n\n

This is an obvious conflict and it’s often solved by pushing your code somewhere else2. So I think we can all say that it’s a good thing GitHub now offers free private repositories, that can have 3 collaborators. It’s also an interesting move as this likely costs them money in the short term. I wonder if that means they’ll tie GitHub Actions to paid accounts3, or if they’re willing to take the hit due to the new ownership of the company.

\n\n

Amazon DocumentDB

\n\n

At re:Invent, AWS announced several new databases, but it turns out that they kept one in reserve. Amazon DocumentDB (with MongoDB Compatibility)4 is a MongoDB compatible hosted solution. It is API compatible with MongoDB 3.6, which is a bit5 behind the current MongoDB version. It also won’t come as a surprise that some people, understandably, aren’t happy with AWS releasing this service.

\n\n

But instead of talking about that, let’s have a look at what is actually going on here. Because in all honesty, I’m interested in a lot of similarities here with Aurora. In fact, if you look at what we know of the underlying structure it shows that, like Aurora, it uses a separation of data and compute with the data replicated 6 times across 3 Availability Zones. A lot of the wording, including the “(with MongoDB Compatibility)” part of the name, seems to be similar to what they say about Aurora as well.

\n\n
\n

Internally, Amazon DocumentDB implements the MongoDB 3.6 API by emulating the responses that a MongoDB client expects from a MongoDB server.

\n
\n\n

Reading between the lines here, but it sounds to me very much like AWS developed their own document database and is building compatibility modes on top of it. Which means that I wouldn’t be surprised if other compatibility layers are going to show up sooner or later, similar to how Postgres followed MySQL on Aurora.

\n\n

Fargate Pricing

\n\n

When AWS announced Firecracker, it was clear that one of the major advantages for it is a purely internal one: it allows AWS to more economically handle spinning up Fargate and Lambda functions. It looks like they have now reached a point where this became impactful enough that they can lower the pricing for Fargate.

\n\n

A quick calculation shows that while it’s still more expensive than EC2 instances6. It’s now “only” 50% more when comparing an m5.large to its equivalent Fargate option, where that used to be about double. The price changes are based on what you spin up, with the bigger options getting the most benefit as RAM got the biggest discount, but either way it makes me happy to see it become cheaper to use one of my favourite services.

\n\n

AWS Client VPN

\n\n

Lastly, I was reminded that AWS released their own VPN client. This is not the VPN solution they already had that works similar7 to Direct Connect, but a direct competitor to running your own VPN servers using something like Pritunl or OpenVPN in order to access your environment. However, as you can see in a couple of blog posts from my colleague Greg Cockburn, it still seems a very fiddly process, especially configuring the client on your computer8.

\n
\n\n
\n\n
    \n
  1. Insert your preferred beverage here.\n [return]
  2. \n
  3. Many of my own repos started on BitBucket.\n [return]
  4. \n
  5. In which case I’ll sign up right away.\n [return]
  6. \n
  7. Yes, the full name sounds silly like that.\n [return]
  8. \n
  9. 3.6 was originally released 2 years ago.\n [return]
  10. \n
  11. Obviously, and of course you won’t run these anywhere close to 247.\n [return]
  12. \n
  13. Yes, there are quite some differences.\n [return]
  14. \n
  15. I wouldn’t care much if the one-time setup part is fiddly, but when everybody using it has to manually work through settings that’s a deal breaker.\n [return]
  16. \n
\n
\n", "summary": "The holiday period is over and there were a couple of very interesting announcements this week. GitHub now offers free private repositories, AWS has a MongoDB compatible database, and Fargate became a lot cheaper to use.\nGitHub Free Private Repos There are many things to like about GitHub, but one of the main complaints over the year has been that you had to pay for private repositories. Many of us work on a personal project that isn’t quite ready for public scrutiny but still needed to be stored somewhere that’s not our own laptop.", "banner_image": "https://ig.nore.me/img/categories/category-weekly-notes-full.jpg", "date_published": "2019-01-14T19:29:18+11:00", "date_modified": "2019-01-14T19:29:18+11:00" }, { "id": "https://ig.nore.me/weekly-notes/week-2-2019/", "title": "Week 2, 2019 - Secrets in ECS; EKS Updates; Serverless Updates", "url": "https://ig.nore.me/weekly-notes/week-2-2019/", "content_html": "\n\n

The first week of the year isn’t exactly exciting when it comes to news, so I’ll discuss some things from last year. As is so often the case when I’m looking back at a longer period, that means containers and serverless.

\n\n

Secrets in ECS

\n\n

I never got around to talking about it, but in November AWS released support for secrets in ECS. What this means is that you can define secrets from the Parameter Store in your task definitions and these then become available as environment variables. This means that you now have an easy way to securely make these secrets available to your Docker applications.

\n\n

When this was released, there was a major shortcoming however. Secrets only worked in ECS on EC2, but not on the superior Fargate1. In good news however, this was rectified in December with platform version 1.32.

\n\n

EKS Updates

\n\n

While EKS didn’t get a lot of love during re:Invent, it did get some important updates after. First, a solution for upgrading your cluster is finally available. You can now update both Kubernetes and EKS platform versions on running clusters. EKS Platform versions are basically the extra glue that AWS puts on top of Kubernetes to make it work in EKS. That mostly means some patches and things like the API interface.

\n\n

Obviously this is an important step in making EKS usable, but personally I liked the availability of EKS in new regions more. After all, one of those regions is Sydney and it was something I’d been waiting for. It happened just before Christmas though, so I haven’t actually gotten around to spinning a cluster up in Sydney3.

\n\n

Serverless Updates

\n\n

Today, I’m not going into the wonderful toys we got at re:Invent. I’ve got a dedicated post for that coming soon4.

\n\n

When I wrote my review of the Serverless Application Repository you could only release applications and updates to them by going through the Console. This wasn’t a great process and obviously it also means that releasing those updates couldn’t be an automated process. With the new publish command from the SAM CLI this has now changed and you can finally do this from the command line and automate the process.

\n\n

The last thing I should mention is WebSocket support in API Gateway. In case you’re not familiar with WebSockets, in short WebSockets API are a standardised way to enable two-way communication between the server5 and client. This means that the API Gateway will keep a connection in place, which in turn allows for more responsive applications as well as actions initiated by the application.

\n
\n\n
\n\n
    \n
  1. Well, superior in my opinion at least which should be obvious after all my talks.\n [return]
  2. \n
  3. Although the rollout of that sounded a bit messy according to the note at the end, it all worked fine.\n [return]
  4. \n
  5. I know, seems a bit unappreciative after asking for it so long.\n [return]
  6. \n
  7. I’ve got a whole bunch of articles in various stages of completion, but I need to actually finish them so you can read them as well. I’ve heard that is more useful.\n [return]
  8. \n
  9. Not exactly the right term when used with a serverless solution, but hopefully you understand what I mean.\n [return]
  10. \n
\n
\n", "summary": "The first week of the year isn’t exactly exciting when it comes to news, so I’ll discuss some things from last year. As is so often the case when I’m looking back at a longer period, that means containers and serverless.\nSecrets in ECS I never got around to talking about it, but in November AWS released support for secrets in ECS. What this means is that you can define secrets from the Parameter Store in your task definitions and these then become available as environment variables.", "banner_image": "https://ig.nore.me/img/categories/category-weekly-notes-full.jpg", "date_published": "2019-01-10T22:06:36+11:00", "date_modified": "2019-01-10T22:06:36+11:00" }, { "id": "https://ig.nore.me/weekly-notes/week-1-2019/", "title": "Week 1, 2019 - Looking Back at 2018; Looking Forward to 2019", "url": "https://ig.nore.me/weekly-notes/week-1-2019/", "content_html": "\n\n

My annual look back at the past year, and forward at the coming year. Also, it’s weird writing something called Week 1 of 2019 when it’s still 2018, but that’s how calendars work.

\n\n

Looking Back at 2018

\n\n

A lot has happened in 2018. Plenty of good things, and of course not so good things. On the tech side of things that impact me it ranges from fun new toys to play with to politicians being stupid about encryption, and that was just in the last month! However, I don’t have a lot of interest in doing a recap of all the news of 2018. That can be found everywhere and on most of the interesting things I’ve already given my opinion. So, let’s look at what I did instead.

\n\n

Writing

\n\n

One year ago, I wrote a similar article and in there I made some statements about my writing plans for 2018. To save you a click, my plans were this:

\n\n\n\n

Well, I can say I failed at this quite a bit. I only created 2 videos, both in January, wrote 38 weekly notes2, and 17 other non-video posts3. There was also one guest post by Chris Coombs.

\n\n

Clearly the goals I set were too ambitious, or I was too lazy. Most likely it was a mix of things, for one I missed a lot of writing time around my wedding and there was a time earlier in the year where I started to feel burned out4 and so stopped writing for a bit.

\n\n

It also turned out that even just making 5 minutes videos was a LOT of work. I estimate that combined those first couple of videos took me about 40 hours to make. A big part of it was learning the tools and trying different ways of doing things, and no doubt this would have shortened over time, but everything that impacted my writing did even more so for the videos.

\n\n

Regardless, this means that over the past year 58 new pieces of content appeared on this site. That’s still not bad and it is definitely more than last year.

\n\n

As a stats junkie5, that means I’m interested in looking how this impacted visitors6. Well, it’s not a 50% increase like last year, as it’s only about 14 to 17% based on what you measure. Still, that brought me at 26.500 page views, which is a big number, and unlike last year the last couple of months actually increased traffic instead of going down. This period is also where most of the increased traffic came from as the rest of the year was flat compared to 2017.

\n\n

\"\"

\n\n

More interesting perhaps are the pages that were visited the most.

\n\n

\"\"

\n\n

The number 1 is still my original Babysteps with CloudFormation article. In second and third place however are two new articles. Conditionals in CloudFormation was actually my most visited page each month from September and both it and AutoScaling ECS Containers using CloudFormation have been in the top 3 since May.

\n\n

After that, most of the visits went to last year’s remaining top articles. One other item of note here is my Building and Testing CloudFormation Macros which landed in the top 5 for both November7 and December.

\n\n

As for the videos, combined those two have now had 1241 views while my most popular video (Introduction to CloudFormation Designer) is currently at 8374.

\n\n

Other

\n\n

Aside from writing on this site, I did a lot of other things as well that were related to this. First, I became an AWS APN Cloud Warrior, which was recently rebranded into the global APN Ambassador program. This was a lot of fun, and let me meet others in similar positions to me who are highly knowledgeable about AWS. It also ensured I got a ticket to re:Invent, which was obviously great.

\n\n

Then I gave quite a few talks this year. I presented twice at the Cloud Warrior/Ambassador meetups, one of which was at re:Invent which I therefore count as presenting at re:Invent. I also gave presentations at the AWS User Group Conference in Melbourne, and the AWS Community Day in Sydney. Then there were several repeats of these same talks at Meetups as well as running a workshop. You may notice that all of these talks were about Fargate, which is because that was my favourite toy in 20188.

\n\n

And lastly, I was volunteered9 to become a co-organiser of the Melbourne AWS User Group, starting with the next meetup in January.

\n\n

Looking Forward to 2019

\n\n

Having learned from my overambitious goals from last year, I’ll lower my estimates but will still set a challenge. As I have periods where I write a lot more than other times, I’m not going to focus on having something every week10 but will instead try to put numbers against them that will sound like a schedule, but could happen more irregularly.

\n\n\n\n

I’ve already got some good content lined up in my head, especially after having been able to play with some of the re:Invent goodies. Also, have I mentioned I got access to the GitHub Actions beta a while back? Expect something on that soon.

\n\n

Of course, I’ll be doing something for the AWS Meetup every month11 but I’m also hoping to talk at some more conferences. While I was quite nervous for them, it’s still a lot of fun and that is mostly what I want to do with this site. A place where I can have fun while hopefully helping people understand things.

\n
\n\n
\n\n
    \n
  1. Well, obviously I knew what that was going in.\n [return]
  2. \n
  3. Which is less than last year’s 41.\n [return]
  4. \n
  5. Not counting a couple of posts I wrote somewhere else.\n [return]
  6. \n
  7. A mix of being very busy both at and outside of work.\n [return]
  8. \n
  9. Feel free to skip this if you think stats are boring.\n [return]
  10. \n
  11. The year isn’t completely over yet, but I doubt that I’ll suddenly have a lot of visitors in these last hours.\n [return]
  12. \n
  13. When it was written.\n [return]
  14. \n
  15. That said, following re:Invent Lambda has closed some of the gaps that could be filled in with Fargate. More on that soon.\n [return]
  16. \n
  17. Yes Chris, a special footnote to thank you for that.\n [return]
  18. \n
  19. Other than the Weekly Notes obviously.\n [return]
  20. \n
  21. We’re planning to hold DeepRacer races and do other fun things! Come along!\n [return]
  22. \n
\n
\n", "summary": "My annual look back at the past year, and forward at the coming year. Also, it’s weird writing something called Week 1 of 2019 when it’s still 2018, but that’s how calendars work.\nLooking Back at 2018 A lot has happened in 2018. Plenty of good things, and of course not so good things. On the tech side of things that impact me it ranges from fun new toys to play with to politicians being stupid about encryption, and that was just in the last month!", "banner_image": "https://ig.nore.me/img/categories/category-weekly-notes-full.jpg", "date_published": "2018-12-31T21:49:03+11:00", "date_modified": "2018-12-31T21:49:03+11:00" }, { "id": "https://ig.nore.me/weekly-notes/week-50-2018/", "title": "Week 50, 2018 - Australian Anti-Encryption Law; Chromium in Edge", "url": "https://ig.nore.me/weekly-notes/week-50-2018/", "content_html": "\n\n

I’m still writing more about re:Invent, but not everything is about AWS so here are a couple of those. Australia implements a stupid law regarding encryption and Microsoft announces they’ll switch to using Chromium for their Edge browser.

\n\n

Australian Anti-Encryption Law

\n\n

Not that long after they banned Huawei from being involved in building their upcoming 5G network, the Australian government decided to shoot themselves in the foot by enacting an anti-encryption law. If you’re not in Australia, and maybe don’t follow this kind of news closely, it might shock you that something this stupid can happen in an otherwise reasonably sane country1.

\n\n

First though, let’s discuss what this law actually is: It allows the Australian government to force companies operating in Australia to hack, implant malware, undermine encryption, or build a backdoor into their product or website. Really? What could possibly go wrong with something as nice as that? Surely there will be proper checks and balances on such a system, right? Well, obviously they promise that it will only be used against severe criminals. But there doesn’t seem to be a lot of protection against mission creep or misuse2.

\n\n

This law has been threatened for a while, famously with the previous3 prime minister stating that the laws of Australia beat the laws of maths. A sure sign of the intelligence behind laws like this4.

\n\n

But instead of moaning about it, let’s have a quick look at possible repercussions from this. Imagine a fictional trillion dollar company that sells devices and has declined exactly this kind of thing in the past. Now this company is faced with a choice: don’t sell anything in Australia, losing a market of 24 million, or implement something that will cost them their reputation and probably a lot more customers. I’m sure that’ll be a hard choice.

\n\n

Alternatively, imagine you’re an international software company with a team in Australia. Suddenly you have a team that is a security risk as they might be forced to compromise your software. Obviously you’ll be so invested in this Australian team that you’ll keep them around despite a major downside like that, right? Right? And that’s not even considering how easy it’ll be to comply with both this law and GDPR which is aimed5 at preventing this sort of thing.

\n\n

Apologies for the rant here. This kind of stupidity just annoys me and then I get fired up when typing it up.

\n\n

Chromium in Edge

\n\n

Something more interesting to think about is Microsoft’s move to change their rendering engine to Chromium. To be honest, I’m in two minds about this. On the one hand there is the part of me6 that rejoices that Microsoft “lost” this battle and will now have to use a different engine. On the other hand, regardless of how small the Edge user base was, it was still an alternative to the overwhelming usage of Chrome. To be clear, this is the opinion that Mozilla holds and they want you to believe this is a big danger.

\n\n

Frankly, I think it’ll be fine, and likely give a better, more secure, browser to Windows users. Yes, it means even more people will be using Chromium, that’s true. But that’s also still an open source project so it will also mean more developers on a shared codebase to improve the experience for everyone. And if worst comes to worst, it’ll be reasonably easy for Microsoft7 to fork it and have their own engine.

\n
\n\n
\n\n
    \n
  1. Ok, they’ve made a fair number of stupid decisions over the years, work with me here.\n [return]
  2. \n
  3. To be fair, I haven’t read the whole thing myself. But I don’t have to be fair on this site and I think this whole thing is stupid.\n [return]
  4. \n
  5. Presumably, I might have missed a change of prime minister during a holiday as they happen quite frequently.\n [return]
  6. \n
  7. Also a reason why I don’t like that I can’t vote here, but not enough to give up my Dutch citizenship to do so.\n [return]
  8. \n
  9. Yes, that one has its own flaws that can be misused.\n [return]
  10. \n
  11. Grown from all the pent up frustration of trying to load IE6 only sites in a decent browser back when that was king.\n [return]
  12. \n
  13. Or anyone else.\n [return]
  14. \n
\n
\n", "summary": "I’m still writing more about re:Invent, but not everything is about AWS so here are a couple of those. Australia implements a stupid law regarding encryption and Microsoft announces they’ll switch to using Chromium for their Edge browser.\nAustralian Anti-Encryption Law Not that long after they banned Huawei from being involved in building their upcoming 5G network, the Australian government decided to shoot themselves in the foot by enacting an anti-encryption law.", "banner_image": "https://ig.nore.me/img/categories/category-weekly-notes-full.jpg", "date_published": "2018-12-10T21:09:14+11:00", "date_modified": "2018-12-10T21:09:14+11:00" }, { "id": "https://ig.nore.me/2018/12/deepracer/", "title": "DeepRacer", "url": "https://ig.nore.me/2018/12/deepracer/", "content_html": "\n\n

In lieu of a single weekly note, I will be writing several articles to catch up with some of the events from re:Invent. Chris helped out last week with his post about the DynamoDB changes, and today I will start my write-ups with the coolest new toy: DeepRacer.

\n\n

DeepRacer is a self-driving car, or rather a self-driving toy car. It’s small, and you can buy it yourself on Amazon for an introductory price of $250 USD. At re:Invent attending the DeepRacer workshop ensured you would get one, and it will come as no surprise that these workshops then became super popular1.

\n\n

Reinforcement Learning

\n\n

But before going into DeepRacer itself, let’s look at the reason why it exists in the first place. One of the new functionalities introduced for SageMaker is reinforcement learning. This is a specific type of training a machine learning model, where the model gets taught by rewarding or punishing certain behaviours. In other words, it’s very close to how you would for example train a dog2: when it does something right you give it a treat, and when it does something wrong you don’t give it one.

\n\n

Except this will all take place into simulated environments and you will be able to have more granular control over your rewards. And hopefully your model is paying more attention than your pet3.

\n\n

DeepRacer

\n\n

So, where does DeepRacer come in here? In essence, it’s a toy to get you hooked on reinforcement learning. The idea is simple, you have your car and you need to train a model that carries it around the track in the shortest amount of time. Let’s have a look at how that works.

\n\n

Unfortunately I start here with bad news, right now DeepRacer is not enabled for every account4. And there is no link in the Console to it either. What you need to do is to manually change the URL to go to where the link will point to, which is basically /deepracer (or even easier, just click this link). There you will be redirected to a sign-up page.

\n\n

Luckily, the temporary account that we got during the workshop still seems to work5, so I’m using that to demonstrate this.

\n\n

\"\"

\n\n

The goal is to train your model. You do this by writing the reward and punishment system I described above. The example code, which works pretty well, trains the car by teaching it to stay in the middle of the road.

\n
def reward_function(on_track, x, y, distance_from_center, car_orientation, progress, steps, throttle, steering, track_width, waypoints, closest_waypoint):\n\n    marker_1 = 0.1 * track_width\n    marker_2 = 0.25 * track_width\n    marker_3 = 0.5 * track_width\n\n    reward = 1e-3\n    if distance_from_center >= 0.0 and distance_from_center <= marker_1:\n        reward = 1\n    elif distance_from_center <= marker_2:\n        reward = 0.5\n    elif distance_from_center <= marker_3:\n        reward = 0.1\n    else:\n        reward = 1e-3  # likely crashed/ close to off track\n\n    return float(reward)
\n

I’m not going to take you through the rest of the details here, they are clearly explained in the workshop documentation. But, once you have created your training model, the fun actually starts.

\n\n

\"\"

\n\n

Well, it will start after about 6 minutes. Apparently training machine learning models doesn’t quite work with my usual way of working where I expect a result within a couple of seconds to see if it was written correctly.

\n\n

\"\"

\n\n

As you define the time a training takes, it will keep running the model over and over again until that time has been reached. It also shows you the results of your reward, which you return at the end of your function as shown above.

\n\n

The best part however, happens while you’re training. Using a combination of RoboMaker and Sumerian, you can follow live what is happening during the training.

\n\n
\n\n
\n\n

Yes, this can be quite a lot of fun. Especially when your car is not behaving as you want it to and you can make fun of it with friends.

\n\n

\"\"

\n\n

Moving on though. Once your training is complete (for a good model you’ll probably want about 2 hours of training), you can then do an evaluation to see how well the model performs. What this means is that it’s once again put on the track, but without the reinforcement training and you’ll get to see how well it performed.

\n\n

\"\"

\n\n

Personally, I one day hope to get good results out of this.

\n\n

After this you can export your model and run it on your actual car where you can make it run on your track. Unfortunately, right now my car is6 somewhere between Las Vegas and my house.

\n\n

Time to ship my #DeepRacer home from #reInvent pic.twitter.com/KHfZ29cd9W

— Arjen Schwarz (@ArjenSchwarz) November 29, 2018
\n\n\n\n

DeepRacer League

\n\n

So, one other thing that AWS announced is that they will be running DeepRacer championships at all of their Summits in 2019. While there was a competition at re:Invent itself, this was obviously not very intensive as there wasn’t a lot of time to train the models. I think that will be a lot of fun in itself.

\n\n

However, as they also released instructions on how to build your own physical tracks I have spoken to a number of people who are planning to build these and have some additional competitions. Unfortunately at this stage it’s unclear if you can design your own tracks and upload those to the DeepRacer UI. Right now only the re:Invent 2018 track is available for training, but it would be really nice to build our own.

\n
\n\n
\n\n
    \n
  1. I myself waited 1.5 hours in line to ensure I got a spot. Well worth the wait.\n [return]
  2. \n
  3. Please note, I have no actual experience with training pets. I only had fish while growing up and they never paid any attention to me.\n [return]
  4. \n
  5. Unless you want to train a simulated fish I guess.\n [return]
  6. \n
  7. I really hope at least those of us who got a DeepRacer will get access soon.\n [return]
  8. \n
  9. Please don’t tell anyone at AWS about this…\n [return]
  10. \n
  11. hopefully\n [return]
  12. \n
\n
\n", "summary": "In lieu of a single weekly note, I will be writing several articles to catch up with some of the events from re:Invent. Chris helped out last week with his post about the DynamoDB changes, and today I will start my write-ups with the coolest new toy: DeepRacer.\nDeepRacer is a self-driving car, or rather a self-driving toy car. It’s small, and you can buy it yourself on Amazon for an introductory price of $250 USD.", "banner_image": "https://ig.nore.me/img/categories/category-aws-full.jpg", "date_published": "2018-12-03T21:53:33+11:00", "date_modified": "2018-12-03T21:53:33+11:00" }, { "id": "https://ig.nore.me/2018/11/serverless-databases-after-re-invent/", "title": "Serverless Databases after re:Invent", "url": "https://ig.nore.me/2018/11/serverless-databases-after-re-invent/", "content_html": "

That’s a wrap for re:Invent 2018! Whilst Arjen will no doubt be providing us with the low down on all things containers1 in due course, I’ve hijacked his excellent blog to bring you a short guest post on serverless database news.

\n\n

Until re:Invent this week, AWS had only one serverless database offering - Aurora Serverless (MySQL compatible). It’s true that there was some disagreement as to whether it was truly serverless,

\n\n

I refuse to call Aurora serverless until it has an HTTP query API to play well with Lambda. A DB that requires persistent connections implies non-serverless compute. #AWSSummit https://t.co/dzyl3WUvCS

— Ben Kehoe (@ben11kehoe) July 17, 2018
\n\n\n\n

but in the flurry of announcements leading up to re:Invent, the Aurora Serverless Data API was released in beta (allowing Lambda to call Aurora Serverless directly over HTTP); making Aurora Serverless irrefutably 100% approved serverless! “Chris, but what about DynamoDB?”; well sure. it’s fantastic. It’s one of my favourite AWS services, but sadly it wasn’t serverless… until now!

\n\n

Whilst there aren’t any serverless police, there are 4 generally agreed upon rules for something to be considered serverless, namely:

\n\n\n\n

Whilst Aurora Serverless always satisfied all of these requirements, with DynamoDB you paid for read/write capacity regardless of whether or not you used it, violating the ’never pay for idle’ rule. With DynamoDB On-Demand you don’t need to specify read/write capacity or bother with auto-scaling rules, rather you pay per million write/read requests consumed (about $1.42 USD and $0.28 USD respectively in ap-southeast-2). Yes, you read that right, it’s available in Sydney at launch! Sadly, there isn’t CloudFormation support yet - but you can’t have everything. For an in depth look at DynamoDB On-Demand in action, please check out the AWS announcement post*.

\n\n

I hope you’re all as excited about DynamoDB becoming truly serverless as I am. No more over/under provisioning, it will just work, like all serverless things should. Thanks for reading, Arjen back over to you!

\n
\n\n
\n\n
    \n
  1. and YAML… and Python\n [return]
  2. \n
\n
\n About the author: Chris Coombs is an AWS Ambassador and has the questionable honour of being the first guest author on this blog. He believes that containers are overrated, and for some reason thinks that JSON is more readable than YAML. Follow him on Twitter to get regular updates on his Switch usage. ", "summary": "That’s a wrap for re:Invent 2018! Whilst Arjen will no doubt be providing us with the low down on all things containers1 in due course, I’ve hijacked his excellent blog to bring you a short guest post on serverless database news.\nUntil re:Invent this week, AWS had only one serverless database offering - Aurora Serverless (MySQL compatible). It’s true that there was some disagreement as to whether it was truly serverless,", "banner_image": "https://ig.nore.me/img/categories/category-aws-full.jpg", "date_published": "2018-11-30T11:04:17-08:00", "date_modified": "2018-11-30T11:04:17-08:00" }, { "id": "https://ig.nore.me/presentations/2018/11/containers-in-deep-space/", "title": "Containers in Deep Space", "url": "https://ig.nore.me/presentations/2018/11/containers-in-deep-space/", "content_html": "\n\n

At the AWS Ambassador day during re:Invent I gave a lightning talk about Fargate that went a bit deeper than my previous talks. If you’re not yet very familiar with Fargate, it might make sense to have a look at the contents of those other talks first.

\n\n

\"\"

\n\n

Black Magic

\n\n

Fargate is a lot of fun to use, but one of the drawbacks of it being a serverless solution is that we don’t have any insight into what our containers run on. Presumably it’s on EC2 instances that are managed by AWS, but for all we really know it could be running on black magic. While that would be cool, I do prefer to believe they are just regular instances floating somewhere above us in space1. So, what I want to show today is some ways to get the information you need and what you can use it for.

\n\n

Regardless of how much things are hidden, there are always ways to find out more information and we have been provided some of that in the way of metadata endpoints. Similar to the EC2 metadata endpoint, these two endpoints can be called with a cURL2 call to give us either statistical information or pure metadata.

\n
curl 169.254.170.2/v2/stats\ncurl 169.254.170.2/v2/metadata
\n

The Stats Endpoint

\n\n

When calling the stats endpoint you get a JSON blob back, and when you parse that into something readable with a tool like jq you will find that you get over 300 lines of JSON per running container, and you will get the information for all the containers running in the task. Below is a partial sample of one such call.

\n
  "4195092261f726c026b0a86820060d2560085d7b158efb4721910feb4354d1c8": {\n\t    "read": "2018-11-23T00:23:49.431731172Z",\n\t    "preread": "2018-11-23T00:23:48.427533626Z",\n\t    "num_procs": 0,\n\t    "pids_stats": {},\n\t    "network": {},\n\t    "memory_stats": {\n\t      "stats": {\n\t        "cache": 6127616,\n\t        "mapped_file": 2072576,\n\t        "total_inactive_file": 1556480,\n\t        "active_anon": 3911680,\n\t        "total_active_anon": 3911680,\n\t        "total_pgpgout": 3209,\n\t        "total_cache": 6127616,\n\t        "active_file": 4571136,\n\t        "pgfault": 6250,\n\t        "inactive_file": 1556480,\n\t        "total_pgpgin": 5660,\n\t        "hierarchical_memsw_limit": 9223372036854772000\n\t      },\n\t      "max_usage": 12476416,\n\t      "usage": 11476992,\n\t      "limit": 3940294656\n\t    },
\n

This gives a lot of information about everything you might want to know for your system, ranging from CPU, to RAM, and disk I/O. However, the big question then is what can you do with this?

\n\n

The information can be useful for a couple of reasons. Ranging from optimisation, where you use it to find the right size for your tasks3, to debugging where you can see what your container was using when something happened.

\n\n

So, how do we get that information? The easiest way might just be using a sidecar pattern. In microservices a sidecar is basically a container that runs alongside the main container for a specific purpose. So we can have a single container that periodically pulls the data we want and dumps it into stdout where it will be picked up by CloudWatch logs.

\n\n

The Metadata Endpoint

\n\n

The other endpoint is the metadata endpoint. This gives us the information about our running containers, where it comes from, when it was started, all of this. Below is a sample of this data.

\n
{\n  "Cluster": "arn:aws:ecs:ap-southeast-2:1234567890:cluster/Demo",\n  "TaskARN": "arn:aws:ecs:ap-southeast-2:1234567890:task/7c1aa905-5bc8-4582-8ee6-91c717699340",\n  "Family": "bastion-arjen",\n  "Revision": "7",\n  "DesiredStatus": "RUNNING",\n  "KnownStatus": "RUNNING",\n  "Containers": [\n    {\n      "DockerId": "4195092261f726c026b0a86820060d2560085d7b158efb4721910feb4354d1c8",\n      "Name": "~internal~ecs~pause",\n      "DockerName": "ecs-bastion-arjen-7-internalecspause-f8cabda787c9c2a8dd01",\n      "Image": "fg-proxy:tinyproxy",\n      "ImageID": "",
\n

There is some task level data, but the rest is at the container level. And you might notice that the DockerId here is the same as the key for the data in the stats endpoint.

\n\n

Before seeing everything we can do with the metadata, let’s have a look at what the container shown here is. Because this is not a container that I created for my task.

\n\n

Internal ECS Pause Container

\n\n

The internal ECS pause container serves as the glue that holds some of the black magic together. Fargate uses the awsvpc networking mode, which spins up an ENI and connects that to our container. To make this all work, AWS created several CNI plugins which do the connecting part here. These plugins ensure that the connection to the ENI is set up, and that various network endpoints are accessible, such as our metadata endpoints.

\n\n

As it takes a bit of time for an ENI to become available, all of this happens with the pause container only while the containers we actually want are not started yet. This is to ensure these don’t fail while the container is starting before there is a networking connection. Then, once the networking for the pause container has successfully been set up, that connection is shared with the other containers.

\n\n

Looking at the metadata we can see this in practice.

\n
"Containers": [\n    {\n      "DockerId": "4195092261f726c026b0a86820060d2560085d7b158efb4721910feb4354d1c8",\n      "Name": "~internal~ecs~pause",\n      "CreatedAt": "2018-11-23T00:17:20.908301885Z",\n      "StartedAt": "2018-11-23T00:17:21.74755447Z",\n    },\n    {\n      "DockerId": "0d83c4706372444f4a3e5748c3f167440db2eda77539b270a30114b6f4b9123c",\n      "Name": "bastion",\n      "CreatedAt": "2018-11-23T00:17:34.676860362Z",\n      "StartedAt": "2018-11-23T00:17:35.32407093Z",
\n

The bastion container was created and started about 14 seconds after the internal ECS pause container, which means that in this case it took about 14 seconds for all the networking to be ready. These numbers are dependent on various factors, but it’s good to be able to see them.

\n\n

Health Checks

\n\n

So, the metadata gives us a way to identify what stats belong to which container. It also shows us how long it took before the containers started and other bits like the task’s IP address. But the other thing you can see in there is the results of the health check.

\n
      "Type": "NORMAL",\n      "Networks": [\n        {\n          "NetworkMode": "awsvpc",\n          "IPv4Addresses": [\n            "172.31.8.21"\n          ]\n        }\n      ],\n      "Health": {\n        "status": "UNHEALTHY",\n        "statusSince": "2018-11-23T00:18:05.910856476Z",
\n

Health checks in ECS are mostly used in services. Where if one of your essential containers is marked as unhealthy, the task will be seen as unhealthy in which case the service will replace it. You can even use this with your deployment strategies. But in the end it’s really a task runner.

\n\n

Which means we can use it for our stats collecting sidecar! The big advantage here is that you can configure the health checks to be run as granularly as every 5 seconds, which is a lot better than you’d get with a cronjob, and you don’t have to worry about ensuring a service keeps running.

\n\n

Lastly, more as food for thought, when you run a task by itself, failing the health check won’t result in automatic replacements. You can still enable a health check though and the results show up in the metadata. And you can have fun with it as well.

\n\n

For example, if you have a bastion task running, you can build in a self-destruct timer based on the StartedAt value of the metadata. You can compare that time to the current time and once the difference is 30 minutes you call an API Gateway endpoint that terminates the task.

\n
\n\n
\n\n
    \n
  1. Or possibly at the other end of a wormhole.\n [return]
  2. \n
  3. Or any other way you wish to access it of course.\n [return]
  4. \n
  5. Remember, Fargate only has certain allowed CPU/RAM combinations and picking the right one can save money.\n [return]
  6. \n
\n
\n", "summary": "At the AWS Ambassador day during re:Invent I gave a lightning talk about Fargate that went a bit deeper than my previous talks. If you’re not yet very familiar with Fargate, it might make sense to have a look at the contents of those other talks first.\nBlack Magic Fargate is a lot of fun to use, but one of the drawbacks of it being a serverless solution is that we don’t have any insight into what our containers run on.", "banner_image": "https://ig.nore.me/img/categories/category-aws-full.jpg", "image": "https://ig.nore.me/presentations/2018/11/containers-in-deep-space/me_at_ambassador_reinvent.jpg", "date_published": "2018-11-27T02:49:48-08:00", "date_modified": "2018-11-27T02:49:48-08:00" }, { "id": "https://ig.nore.me/weekly-notes/week-48-2018/", "title": "Week 48, 2018 - AWS Route 53 Resolver; Resource Access Manager; Predictive Scaling", "url": "https://ig.nore.me/weekly-notes/week-48-2018/", "content_html": "\n\n

I’m writing this during a 14 hour flight to re:Invent, so once again this is very AWS focused. Several new releases this week include the Route 53 Resolver for Hybris Clouds, the Resource Access Manager, and Predictive Scaling for EC2.

\n\n

Route 53 Resolver

\n\n

Route 53 has been around for a long time, as has its ability for private zones. These are zones that allow you to have a DNS zone specific to your VPC, and it’s invisible to the outside. What the new Route 53 Resolver allows however is the ability to share this DNS with your internal DNS, and also to access that DNS.

\n\n

What this means is that you can have a name server in your internal network, and a private Route 53 zone for your VPC and the two can talk and stay in sync. The service is not cheap though, and as always it’s possible to build your own solution for something like this that might be cheaper to run. However, there was another announcement that potentially makes this a more interesting option after all.

\n\n

Resource Access Manager

\n\n

The Route 53 Resolver is the first service that can be used with the Resource Access Manager. The Resource Access Manager allows you to literally share resources across accounts. This can be within an Organisation, or it can be with a specific account.

\n\n

In the case of the Resolver this means that you can share the ENIs that get created for this, and thereby allow all of your accounts to have access to this new DNS. As you don’t pay anything for the sharing itself, although obviously still for the usage, that means the costs for using the Resolver can be spread out over multiple accounts while you get the benefit for all of those accounts it is shared with.

\n\n

Predictive Scaling for EC2

\n\n

Possibly the coolest thing to come out this week however is predictive scaling. As the name implies, predictive scaling literally predicts the scaling actions your autoscaling group should take and makes that happen. It bases this on historical data, thrown over a bunch of machine learning.

\n\n

So, if for example you have always scaled down your instances during the night because of less traffic, it will see this and do the scaling for you. However, as it uses all of the data available to you, it should1 do this at better times. So instead of your usual scale down at 9pm from 8 to 2 instances it might scale down to 6 by 7 o’clock, 4 by 7:30, and eventually down to 2 by 8:42.

\n\n

Yes, you can do a lot of this by configuring your autoscaling groups based on load, but similarly it will be able to predict more load. So if for example it detects that around lunch time you usually have a spike in traffic, it can scale up your instances before the spike happens, thereby ensuring the best possible experience for your visitors.

\n\n

I have to admit though, and I haven’t found an answer to this yet, I wonder how this will work with daylight saving time. The predictions are still based on time, and if the time changes for the people, but not the predictive software, how will that impact those predictions.

\n\n

re:Invent 2018

\n\n

As I said at the top, I’m on my way to re:Invent. I’m also quite confident that there will be a lot of new things announced that I’ll want to write about. Right now I don’t know how or when I will be doing that. There is a chance that I’ll be writing a lot2, but equally I could be so busy that I won’t write anything3. Either way, expect that I’ll have a lot to say afterwards.

\n
\n\n
\n\n
    \n
  1. Because nothing is perfect.\n [return]
  2. \n
  3. Whether you think that’s good or bad is up to you.\n [return]
  4. \n
  5. Highly unlikely though.\n [return]
  6. \n
\n
\n", "summary": "I’m writing this during a 14 hour flight to re:Invent, so once again this is very AWS focused. Several new releases this week include the Route 53 Resolver for Hybris Clouds, the Resource Access Manager, and Predictive Scaling for EC2.\nRoute 53 Resolver Route 53 has been around for a long time, as has its ability for private zones. These are zones that allow you to have a DNS zone specific to your VPC, and it’s invisible to the outside.", "banner_image": "https://ig.nore.me/img/categories/category-weekly-notes-full.jpg", "date_published": "2018-11-26T12:18:12+11:00", "date_modified": "2018-11-26T12:18:12+11:00" }, { "id": "https://ig.nore.me/2018/11/building-and-testing-cloudformation-macros/", "title": "Building and Testing CloudFormation Macros", "url": "https://ig.nore.me/2018/11/building-and-testing-cloudformation-macros/", "content_html": "\n\n

CloudFormation Macros were introduced recently, and they add a lot of power. In this article I want to take a look at how you can build and test one of these.

\n\n

What are Macros?

\n\n

Macros are a way to transform your CloudFormation template. You can write Lambda functions that change the template you’ve written (or a part of it) in order to get your desired result. This is the exact same functionality that is present in SAM templates, but with the difference that you are in control of the transformation that happens.

\n\n

There are two different ways to use a macro, at the template level and snippet level. Template level transformations can change anything in the template, including adding and deleting resources, while snippet transformations are limited to just the snippet they are invoked in and its siblings. For these snippets your macro will still get the entire Resource provided, but you’re limited in your changes to that single Resource. These two have a different syntax for getting invoked.

\n\n

Template level transforms are defined in the Transform section of your template.

\n
# Template level transform\nAWSTemplateFormatVersion: "2010-09-09"\nDescription: The dev environment VPC and subnets\nResources:\n  VPC:\n    Type: AWS::EC2::VPC\n    Properties:\n      CidrBlock: 10.42.0.0/16\nTransform:\n  - NamingStandard
\n

While resource level ones use the Fn::Transform function.

\n
# Resource level transform\nAWSTemplateFormatVersion: 2010-09-09\n Resources:\n    MyBucket:\n      Type: 'AWS::S3::Bucket'\n      Properties:\n        BucketName: MyBucket\n        Tags: [{"key":"value"}]\n        'Fn::Transform':\n          - Name: PolicyAdder\n        CorsConfiguration:[]
\n

All macros in the template are managed in order. The more specific a macro, the earlier it is handled. If the above transforms are used in a single template, the PolicyAdder will be handled before NamingStandard. Aside from this they are handled in the order of the template. If you have 2 template level macros they are parsed in the order you put them in.

\n\n

Building a Macro

\n\n

Let’s build our own Macro then, but let’s keep it simple1. Now, one of the advantages of CloudFormation is that you can reuse templates. However, one of the things that annoys me personally is that when you reuse templates the description is always the same2. As this has to be a string, we can’t fill it with our parameters to distinguish between our dev and prod environments or between different instances.

\n\n

Let’s fix this with our new macro! The desired result is that if we deploy the following template with “DEV” as the value for the Identifier parameter:

\n
AWSTemplateFormatVersion: "2010-09-09"\nDescription: The %s environment VPC and subnets\nParameters:\n  Identifier:\n    Type: String\n    Description: The unique identifier of the template\nResources:\n  VPC:\n    Type: AWS::EC2::VPC\n    Properties:\n      CidrBlock: 10.42.0.0/16\nTransform:\n  - DescriptionFixer
\n

That it will generate the following actual template3:

\n
AWSTemplateFormatVersion: "2010-09-09"\nDescription: The DEV environment VPC and subnets\nParameters:\n  Identifier:\n    Type: String\n    Description: The unique identifier of the template\nResources:\n  VPC:\n    Type: AWS::EC2::VPC\n    Properties:\n      CidrBlock: 10.42.0.0/16
\n

The Lambda Function

\n\n

Usually I prefer to write my Lambda functions in Go, but in order to make it easier to deploy this one I’ll use a language we can inline into a CloudFormation template: Python.

\n\n

As any Lambda function, our macro requires a handler function that serves as the endpoint and is responsible for returning the result. The event that gets passed to this contains all the parts that we need to fix the description of our templates. In this case we only need the fragment, which contains the template already parsed for use, and template parameters. The other information stored in event can be found in the documentation.

\n
def handler(event, context):\n    # Globals\n    fragment = event['fragment']\n    templateParameterValues = event['templateParameterValues']
\n

From the template parameters we then collect the Identifier and check if it contains a %s. This is only needed for error checking, because obviously we shouldn’t be adding the transform when this isn’t the case. If a %s is found, we override the value of fragment['Description'] with one where we replace %s with the identifier.

\n
    identifier = templateParameterValues['Identifier'].upper()\n\n    if '%s' in fragment['Description']:\n        fragment['Description'] = fragment['Description'] % identifier
\n

Then finally we respond by returning the required values.

\n
\tmacro_response = {\n        "requestId": event["requestId"],\n        "status": "success"\n\t\t"fragment": fragment\n    }\n    return macro_response
\n

So, the next step would be to deploy this. But let’s assume we’re actually doing this the right way and build a test for this first.

\n\n

Testing our Function

\n\n

With the function written in Python, that makes the built-in unittest framework the obvious choice4.

\n
import unittest\nimport macro\n\nclass TestStringMethods(unittest.TestCase):\n    identifier = "TEST"\n    event = {}\n\n    def setUp(self):\n        self.event = {"requestId": "testRequest",\n            "templateParameterValues": {"Identifier": self.identifier},\n            "region": "ap-southeast-2"}\n\n    def test_replacement(self):\n        self.event["fragment"] = {"Description": "%s template"}\n        result = macro.handler(self.event, None)\n        fragment = result["fragment"]\n        self.assertEqual(fragment['Description'], "TEST template")\n\n    def test_no_replacement(self):\n        self.event["fragment"] = {"Description": "static template"}\n        result = macro.handler(self.event, None)\n        fragment = result["fragment"]\n        self.assertEqual(fragment['Description'], "static template")\n\nif __name__ == '__main__':\n    unittest.main()
\n

The unit test is pretty self-explanatory. I import both the unittest module and the macro itself (which is stored in the file macro.py) to ensure I can call the handler function. I define the main part of the event in the setUp function and then in the actual test functions I can just supply the template fragment directly as a Python object5.

\n\n

\"\"

\n\n

Running the tests gives me the expected result and now we know our function is working correctly! Hurray!

\n\n

The CloudFormation Template

\n\n

Now all we need to do is create a CloudFormation template for the macro and deploy it to my account. This part is easy actually, and is not much different from any other Lambda function. First, we define the execution role. This needs to be able to write the logs

\n
AWSTemplateFormatVersion: 2010-09-09\nDescription: "CloudFormation Macro for fixing the description of templates"\nResources:\n  TransformExecutionRole:\n    Type: AWS::IAM::Role\n    Properties:\n      AssumeRolePolicyDocument:\n        Version: 2012-10-17\n        Statement:\n          - Effect: Allow\n            Principal:\n              Service: [lambda.amazonaws.com]\n            Action: ['sts:AssumeRole']\n      Path: /\n      Policies:\n        - PolicyName: root\n          PolicyDocument:\n            Version: 2012-10-17\n            Statement:\n              - Effect: Allow\n                Action: ['logs:*']\n                Resource: 'arn:aws:logs:*:*:*'
\n

Then we need the function itself. As I mentioned earlier, to make it easier to distribute I’ve inlined the Python code. This means some copy/paste work, but is worth it6.

\n
  TransformFunction:\n    Type: AWS::Lambda::Function\n    Properties:\n      Code:\n        ZipFile: |\n            import traceback\n\n\n            def handler(event, context):\n                macro_response = {\n                    "requestId": event["requestId"],\n                    "status": "success"\n                }\n                # Globals\n                fragment = event['fragment']\n                result = fragment\n                templateParameterValues = event['templateParameterValues']\n\n                identifier = templateParameterValues['Identifier'].upper()\n\n                if '%s' in fragment['Description']:\n                    result['Description'] = fragment['Description'] % identifier\n\n                macro_response['fragment'] = result\n                return macro_response\n\n      Handler: index.handler\n      Runtime: python3.6\n      Role: !GetAtt TransformExecutionRole.Arn
\n

But it’s after this that we finally get to the interesting bits. First, we need to grant the function permission to be invoked from CloudFormation. And then we have to use the new AWS::CloudFormation::Macro Resource Type to set the name for our macro.

\n
  TransformFunctionPermissions:\n    Type: AWS::Lambda::Permission\n    Properties:\n      Action: 'lambda:InvokeFunction'\n      FunctionName: !GetAtt TransformFunction.Arn\n      Principal: 'cloudformation.amazonaws.com'\n  Transform:\n    Type: AWS::CloudFormation::Macro\n    Properties:\n      Name: 'DescriptionFixer'\n      Description: Fixes CloudFormation stack descriptions\n      FunctionName: !GetAtt TransformFunction.Arn
\n

With the template written, we can deploy it. Obviously, we’ll use the command line for this.

\n
aws cloudformation deploy --template-file DescriptionFixer/macro.yml --stack-name IGN-CFN-DESCRIPTION-FIXER --capabilities CAPABILITY_IAM
\n

\"\"

\n\n

Let’s try it

\n\n

Now we can try out the sample template from earlier and see the transformation in action7.

\n
aws cloudformation deploy --template-file DescriptionFixer/example.yml --stack-name IGN-CFN-DEV-VPC --capabilities CAPABILITY_IAM --parameter-overrides Identifier=DEV
\n

\"\"

\n\n

And with that we’ve got our unique description. As usual, all the code used in this article can be found on GitHub.

\n\n

As Complex as You Want It

\n\n

While updating the description is a simple use case, you can do anything you want with the template, as long as it results in a valid CloudFormation template. Examples of other macros that I’ve written are a naming standard that enforces and applies a particular naming convention to all resources in a template8 and the expansion of short form Network ACL rules9.

\n\n

The main takeaway though is that CloudFormation Macros allow you to do a lot more with CloudFormation and even allow you to customise how you want to write your templates. After all, there are many places where it seems extremely verbose and you just want to write a single line instead of 4 separate resources.

\n\n

DSLs are Redundant

\n\n

Let’s end this with a little side note. Four years ago10 I wrote about using CFNDSL to build CloudFormation templates. While DSLs have their own quirks, at the time using a tool like this was the best way to make your templates more flexible and it allowed you to do more complex things such as loops. Even conditionals, have existed for a long time in CloudFormation, but they don’t give the flexibility and power you can get with an actual programming language. And of course CFNDSL meant you didn’t have to use JSON directly, always a good thing.

\n\n

Since that time however, there have been many improvements to CloudFormation. The two with the biggest impact are the support for YAML as a way to write the templates and the Macros I’ve just been talking about. Between these two I’m happy to make the case that tools like CFNDSL are no longer needed. Using YAML solves the mess that JSON can be11, and as we’ve seen the Macros give you the same power to make changes to your templates using a programming language.

\n\n

This doesn’t mean that I think you should change your existing projects, but I would recommend against using a DSL for anything new and instead embrace the power of Macros.

\n
\n\n
\n\n
    \n
  1. Yes, you can do super complex things, but for demonstration purposes simple is best.\n [return]
  2. \n
  3. Yes, it’s a minor thing and not something people usually pay attention to, but it’s there for a reason.\n [return]
  4. \n
  5. Ok, under the hood it actually generates a JSON template, but for readability I’m displaying it as YAML.\n [return]
  6. \n
  7. I’m not familiar enough with Python to know if there’s a better testing framework. If there is, I’m happy to hear about it.\n [return]
  8. \n
  9. Remember, it automatically gets translated into this before reaching the Lambda function.\n [return]
  10. \n
  11. Of course, you can automate this by using a main template with a placeholder in the ZipFile property that you then replace with the contents of the Python file.\n [return]
  12. \n
  13. Don’t use deploy for anything but testing, make sure you can verify the Changeset before rolling it out.\n [return]
  14. \n
  15. Using Name tags and resource specific name fields.\n [return]
  16. \n
  17. Based on the great work by Kablamo in this VPC builder (updated on 2018-11-25 with link to newer version).\n [return]
  18. \n
  19. I really need to do a new version of that post.\n [return]
  20. \n
  21. Yes, I know some people believe JSON is human readable. They are wrong. Unless you compare it to XML of course, then JSON wins.\n [return]
  22. \n
\n
\n", "summary": "CloudFormation Macros were introduced recently, and they add a lot of power. In this article I want to take a look at how you can build and test one of these.\nWhat are Macros? Macros are a way to transform your CloudFormation template. You can write Lambda functions that change the template you’ve written (or a part of it) in order to get your desired result. This is the exact same functionality that is present in SAM templates, but with the difference that you are in control of the transformation that happens.", "banner_image": "https://ig.nore.me/img/categories/category-aws-full.jpg", "date_published": "2018-11-22T19:43:35+11:00", "date_modified": "2018-11-22T19:43:35+11:00" }, { "id": "https://ig.nore.me/presentations/2018/11/space-containers/", "title": "Space Containers", "url": "https://ig.nore.me/presentations/2018/11/space-containers/", "content_html": "\n\n

In October I gave a Serverless Containers Deep Dive talk at the AWS Community Day and AWS User Group Conference, which focused on AWS Fargate. Below is a written version of these presentations, as well as the recording of when I repeated this talk at the Melbourne AWS User Group1. The slides by themselves are available on Speaker Deck.

\n\n
\n\n
\n\n

What Are Serverless Containers?

\n\n

In order to talk about serverless containers, let’s take a quick step back and look at what containers really are.

\n\n

Once upon a time we had to deal with hardware. Hardware can be annoying, it doesn’t scale well2 and it ends up giving you a lot of issues you might need to solve by going to a data centre somewhere. In addition it means that to have good utilisation you’ll end up running multiple applications together and all the headaches that might cause.

\n\n

This was improved with the introduction of virtual machines. Instead of having to deal with the hardware directly, and a separate piece of hardware for anything that needed to be separate3, you can run many virtual machines on the same hardware. This means that you have both better utilisation as you can run multiple operating systems that are purpose fit, and it makes it easier to manage and scale.

\n\n

But then we got containers. Made popular by Docker, containers offer a way to combine your application and all of its dependencies into a single package. These can then be run on different machines, virtual or otherwise, where these lightweight4 containers provide an additional layer of separation between the various applications. Containers will use parts of the underlying operating system such as the kernel, I/O, and networking, but these are all abstracted away by the runtime so the container doesn’t know what it’s running on.

\n\n

\"\"

\n\n

But in the end this still leaves you with several layers that need to be maintained. After all, the container runs on top of your virtual machine, that you need to patch and update, and this in turn runs on top of the hardware that has the same requirements5. So how about we just get rid of that? Let’s remove that virtual machine, and the underlying hardware, and make it all serverless6. Or as I like to think of it: Serverless Containers IN SPACE!

\n\n

\"\"

\n\n

As with all serverless technologies, this means that the containers run on (virtual) machines managed by someone else. However, as we don’t have insight into where these machines are I prefer to think of them floating high above us in space as that makes it more fun7.

\n\n

How Do You Use Space Containers?

\n\n

While there are multiple providers of serverless containers, I will focus only on the AWS solution for this: AWS Fargate.

\n\n

Fargate was introduced at re:Invent 2017 and is a deployment method for ECS, the Elastic Container Service. Originally ECS only allowed you to deploy containers on instances you spin up and prepare yourself, but with Fargate you can now deploy it straight into “space”. Since its introduction there have been various improvements to Fargate, most of which are a result of improvements to ECS such as the just released support for resource tagging8, and most importantly it was rolled out to more regions.

\n\n

How do we use ECS then to create a Fargate container? The below 5-minute video is one I recorded in January that shows me spinning up Fargate containers.

\n\n
\n\n
\n\n

In essence though, it boils down to 2 steps:

\n\n
    \n
  1. Create an ECS Task Definition
  2. \n
  3. Create a Task or Service using that Task Definition.
  4. \n
\n\n

If you’re not familiar with the actual terminology used here you might want to use an analogy for the task, service, and task definition. If you compare it to EC2 Instances, you can think of the ECS Task Definition as an AMI9, a Task as the Instance, and a Service as an Autoscaling Group.

\n\n

Both of these steps can be done through the Console, CLI, or something like CloudFormation.

\n
aws ecs register-task-definition --family demo --container-definitions "[{\"name\":\"nginx\",\"image\":\"nginx\",\"cpu\":256,\"memory\":512,\"essential\":true}]" --network-mode awsvpc --requires-compatibilities FARGATE --execution-role-arn arn:aws:iam::123456:role/ecsTaskExecutionRole --cpu 256 --memory 512\n\naws ecs run-task --task-definition demo --network-configuration="{\"awsvpcConfiguration\":{\"subnets\":[\"subnet-123456\"],\"securityGroups\":[\"sg-123456\"],\"assignPublicIp\":\"ENABLED\"}}" --launch-type FARGATE
\n

Looking at the above, you can see it’s fairly straightforward. A couple things to pay attention to are that you have to say a container is meant for Fargate both when you create the task definition and when you run the task. I’m not sure why, as it seems redundant, but it’s not a big deal either. Aside from that, you need to define the amount of memory and CPU your task requires when you create the task definition, and there is a limited number of allowed combinations here.

\n\n

The other thing you need to pay attention to is the networking. In the task definition you have to specify that you wish to use the awsvpc networking mode, and then when you run the task you have to tell it which subnets and security groups to use while also deciding if the containers should get a public IP or not.

\n\n

How Do Space Containers Work?

\n\n

Now that we know how to spin up a Fargate container, let’s have a look at how this all actually works. After all, having a container up in space makes it a bit hard to use.

\n\n

This is where the magic of the awsvpc networking mode comes in. When spinning up a container, it will create an ENI in the subnet you specified. An ENI is an Elastic Network Interface, which comes down to it being the thing that connects your compute power to the network. It is used for every type of compute that requires access to a VPC, and Fargate containers are therefore no exception.

\n\n

Each container will have its own ENI, which means there are some limitations you need to keep in mind. First of, it means that if you spin up a lot of Fargate containers, you have a good chance to run out of private IPs in your subnets or run into service limits. In addition, a side effect of working this way is that your containers can only expose unique ports. In contrast, with EC2-based ECS you can have multiple containers in the same task listening to, for example, port 80, which can then be translated to different ephemeral ports on the instance.

\n\n

As mentioned earlier, you can also give your tasks a public IP. In this case, you will be randomly assigned an Elastic IP to the ENI. It’s not possible to manage this EIP, so don’t expect a consistent public IP.

\n\n

\"\"

\n\n

Of course, you can always attach the service to a load balancer to ensure it has a consistent endpoint. If your service is stateless and only needs to be available internally you can also consider using ECS service discovery which uses a multi value Route53 record and is therefore comparable to a round robin DNS.

\n\n

Why Use Space Containers?

\n\n

This is the biggest and probably most important question when considering a technology. Why would I use this? What are the benefits, and how do they weigh against the downsides? Especially when you consider the rest of the ecosystem that already contains several compute options such as EC2 instances, Lambda functions, and of course ECS on EC2.

\n\n

In this case, let’s compare the different options using the below (non-scientific) diagram.

\n\n

\"\"

\n\n

One the y-axis we have the average runtime of the compute option, while the x-axis shows how much time you spend on maintaining them10. Unsurprisingly, Lambda functions run for the shortest amount of time. Even with the recent increase to a 15-minute runtime, most of these functions will run for seconds at a time. Similarly, there is very little maintenance as AWS handles most of that for you, with the occasional reminder to please update your NodeJS runtime once the version you use is no longer supported.

\n\n

On the other hand of the spectrum are EC2 instances. These will usually run for a very long time, ranging from days to years. And of course, these also require a lot of maintenance work even just to keep up with all the security patching. While you may (and should!) have automated this into your CI/CD pipeline, there is still work involved.

\n\n

ECS on EC2 instances is similar, with the big difference that as you don’t care as much about the actual underlying host you will likely use the ECS-optimised images provided by AWS. In that case you probably only need to include a yum update in your user data and once in a while refresh the AMI11. You’ll still need to update your containers though, but that will usually happen as part of the application development.

\n\n

Fargate on the other hand, doesn’t care about the underlying host at all, which makes it fall more on the Lambda side of the spectrum. You’ll still deal with the container side of things though, so there is more work than dealing with Lambda functions.

\n\n

Based on this you can draw some conclusions where Fargate might be useful, but also remember that it’s very easy to draw diagrams like this and get any outcome you like. For example, if we focus on the price per second you’ll likely find that Fargate doesn’t come out nearly as positive.

\n\n

In the end, what it comes down to is that you need to use the right tool for the job. So, let’s have a look at what potential jobs are where Fargate is the right tool.

\n\n

1. Standard Workloads

\n\n

Hopefully the section on how to run space containers has given you a feeling for how easy it is to run Fargate containers as services. You pay a bit more for using Fargate, but on the other hand you (or your team) can focus on more interesting matters than dealing with the underlying hosts.

\n\n

In fact, once you’ve done it a couple of times you will find that it is quite boring. Don’t get me wrong though, when it comes to infrastructure boring is a good thing. It means that it’s predictable, stable, and won’t cause an alert at 3am in the morning.

\n\n

2. On Demand

\n\n

As with every AWS service, we’re not limited to just using the service by itself and we can build nice solutions. One example of this is the Serverless Bastion on Demand that I wrote and spoke about earlier in the year. This allows you to spin up a Fargate container into your environment that you can then SSH into to get access to instances you may have running in there.

\n\n

\"\"

\n\n

With the introduction of the AWS Systems Manager Session Manager12, this particular use case has become less useful but the idea itself still stands. Using a Lambda function or something else, you can start a container when you need it and tear it down shortly after.

\n\n

3. Scheduled Tasks

\n\n

Of course, we can automate this as well. With the introduction of scheduled tasks for Fargate, it is possible to run jobs on a regular basis. One potential use case for this is to run cronjobs. While cronjobs have served the computing world well, they’re not quite as useful in a modern architecture where your application servers are autoscaling13. One workaround for this was to have a separate server that runs the cronjobs, but instead you can set up a task that does this. Aside from regular cron syntax, you can even set it up to run every x amount of time.

\n\n

\"\"

\n\n

Limitations of Space Containers

\n\n

As mentioned before, you need to have a look at the limitations as well if you want to see if something is useful to you. So let’s have a look at some of these14.

\n\n

First, there is no support for IPv6. Even if you deploy it into an IPv6 enabled subnet, your Fargate ENI will not get an IPv6 IP. Speaking of IPs, I already mentioned that you have one IP per running task, and all the limitations that brings along.

\n\n

Being a serverless technology you have less control over your runtime, so for example if you’re using Docker plugins this is not the solution for you. Similarly, keep in mind that you have to take into account that the local storage is not under your control so if you have persistent storage needs you will have to find a solution. There is support for volumes, but that doesn’t necessarily fulfil the needs you have for it.

\n\n

The last limitation I want to mention is the lack of support for Windows containers. While I’m not a big fan of these, this obviously means that if you use Windows containers you can’t run these.

\n\n

That said, the downsides of serverless technologies are countered by the upsides of it: you have less maintenance, don’t need to think about security as much, and you have a pay as you go model. And when you compare Fargate to Lambda you have the full capabilities of containers in your toolset. There is no need to limit yourself to what Lambda allows, or to switch to one of the blessed languages, you can run anything you want in your container15 and for however long you want to.

\n\n

Go Build (in space)

\n\n

To summarise this all, Fargate is a powerful tool that you can use. Between standard workloads, on demand tasks, and scheduled tasks I’ve given several use cases that might work, but in the end I don’t know your environment and what may or may not work.

\n\n

Hopefully your take away from all this that Fargate allows you to do some things in a different way than you did before and that it gave you some ideas of how to use it. So, as our friends at AWS like to say: Go Build!

\n\n

But do it in space.

\n\n

\"\"

\n
\n\n
\n\n
    \n
  1. The user group has recently started recording all presentations, check out their channel!\n [return]
  2. \n
  3. Scaling down was easy, but scaling up could literally takes weeks.\n [return]
  4. \n
  5. For example, Windows and Linux workloads.\n [return]
  6. \n
  7. Containers can be as small as the application you run in it.\n [return]
  8. \n
  9. Usually taken care of by your hosting provider/cloud environment.\n [return]
  10. \n
  11. Which can be defined as: making everything below the application somebody else’s problem\n [return]
  12. \n
  13. Yes, I probably read and watch too much science fiction.\n [return]
  14. \n
  15. Although some ECS features aren’t available immediately to Fargate, like the new secrets.\n [return]
  16. \n
  17. Or perhaps a Launch Template.\n [return]
  18. \n
  19. Not taking into account any time spent on the application itself.\n [return]
  20. \n
  21. Did you know you can automatically get the latest image from the Parameter Store?\n [return]
  22. \n
  23. I still believe that name is silly.\n [return]
  24. \n
  25. Which potentially means your job runs more than you want it, or gets cancelled due to a scaling action.\n [return]
  26. \n
  27. Just in case of big announcements, this is posted before re:Invent 2018.\n [return]
  28. \n
  29. Except Windows obviously.\n [return]
  30. \n
\n
\n", "summary": "In October I gave a Serverless Containers Deep Dive talk at the AWS Community Day and AWS User Group Conference, which focused on AWS Fargate. Below is a written version of these presentations, as well as the recording of when I repeated this talk at the Melbourne AWS User Group1. The slides by themselves are available on Speaker Deck.\n What Are Serverless Containers? In order to talk about serverless containers, let’s take a quick step back and look at what containers really are.", "banner_image": "https://ig.nore.me/img/categories/category-aws-full.jpg", "date_published": "2018-11-20T20:24:34+11:00", "date_modified": "2018-11-20T20:24:34+11:00" }, { "id": "https://ig.nore.me/weekly-notes/week-47-2018/", "title": "Week 47, 2018 - CloudFormation Drift Detection; Multiple Instance Types in ASGs; Amazon Corretto", "url": "https://ig.nore.me/weekly-notes/week-47-2018/", "content_html": "\n\n

Clearly we’re in the lead up to re:Invent as AWS has started releasing the big features that didn’t make the cut. Today I’ll focus on CloudFormation Drift Detection, Multiple Instance Types in AutoScaling Groups, and Amazon Corretto.

\n\n

CloudFormation Drift Detection

\n\n

A long awaited feature for CloudFormation has arrived. Drift Detection was originally announced at re:Invent 2017, and it spent a lot of time in beta. But now it is available for all of us to use.

\n\n

But what is it, you might ask1. The concept is straightforward, drift detection allows you to see if changes have been made to your infrastructure outside of CloudFormation. For example, if you deployed a security group through CloudFormation but then manually added a rule to it, drift detection can show you this. It’s also pretty easy to use. You select your stack, kick off a drift detection task and once it’s finished you can see if it’s done.

\n\n

The fact that it’s a manual two-step job is slightly annoying, but it can still be automated. Whether you build a Lambda function for it, or make it part of your CI/CD pipeline, it’s all possible. There are however some signs that it’s a bit early still as not everything is as great as we’d like it to be. Take the below detected “drift” for example.

\n\n

\"\"

\n\n

This IPv6 address is assigned and managed by AWS. I have no control over it at all, but because CloudFormation stores the information slightly different than the VPC itself does, it’s shown as having drifted. There are a number of these false positives, and while I’m sure these bugs will be worked out soon enough it does mean it’s hard to set up automation for this at the moment.

\n\n

Multiple Instance Types in ASGs

\n\n

Up until last week, you could only define a single type of instance in an AutoScaling Group. For example, you could say that you wanted to deploy it using on-demand M5 instances or perhaps R4 spot instances. If you wanted multiple types, a mix of on-demand and spot for example, you had to create multiple groups and make them all work together. This could be a lot of work, and increased the chances of errors.

\n\n

Now, you can mix and match all of these in a single ASG, which will make life a lot easier. As you can still define a preference, this means you can be more sure that you’ll have instances available when you need them. One example, specific to the Sydney region, is that this allows you to define M5 instances as your primary type, with a fallback for M4s for the availability zone that doesn’t have (many) of these.

\n\n

Amazon Corretto

\n\n

In case you missed it, there has been some bad blood between AWS and Oracle. With Larry Ellison making statements during his Oracle cloud keynote2, and a bit of a back and forth afterwards. I don’t know if that had any impact on the sudden release of Amazon Corretto, which is a replacement for Oracle’s Java, but it’s certainly convenient timing3.

\n\n

For me there isn’t much to say about Corretto. I don’t like Java as a programming language, and prefer if it doesn’t need to be installed. That said, an alternative free version is better than the ever more license encumbered “official” version provided by Oracle. Also, Corretto sounds like a drink I want to try.

\n
\n\n
\n\n
    \n
  1. Unless you actually read the announcement.\n [return]
  2. \n
  3. I personally didn’t think anything in the event was worth mentioning here.\n [return]
  4. \n
  5. And the documentation page is a lot more bare bones than usual for an AWS product.\n [return]
  6. \n
\n
\n", "summary": "Clearly we’re in the lead up to re:Invent as AWS has started releasing the big features that didn’t make the cut. Today I’ll focus on CloudFormation Drift Detection, Multiple Instance Types in AutoScaling Groups, and Amazon Corretto.\nCloudFormation Drift Detection A long awaited feature for CloudFormation has arrived. Drift Detection was originally announced at re:Invent 2017, and it spent a lot of time in beta. But now it is available for all of us to use.", "banner_image": "https://ig.nore.me/img/categories/category-weekly-notes-full.jpg", "date_published": "2018-11-19T21:20:08+11:00", "date_modified": "2018-11-19T21:20:08+11:00" } ] }