Go forth and Lambda…

Over the holiday period I finally settled down to spend some time building a photo archiving service for personal use — you’ll hear more about that incoming days. This also meant that I buckled down to start writing some serious Go code, rather than playing around with initial tutorials and lessons. Here’s how it unfolded.

So, first, some caveats. This is not a how to article. It’s not even a this is how to do AWS Lambda with Go. It’s a quick survey of some of the things I learned, and the journey I went on. Part of that journey was some smashing small books from John Arundel (bitfieldconsulting.com/books) that were an excellent introduction to Go that focus on writing good code. As a final caveat, I’m not yet at the stage of writing good idiomatic Go code. That’s going to take some practice.

Alright, so lets set some context. My photo archiving solution is a data pipeline — I will throw JPEGs at an ingestion S3 bucket on AWS, and some AWS Lambda code will notice that, file them somewhere else organised by year / month / day, and then create a smaller thumbnail version of the image. I’ve built this sort of pipeline before, previously using Java and Python, so I was comfortable with all the bits and pieces needed to make this work, allowing me to concentrate on the Lambda code.

Knowing the landscape, I headed off down a logical route: I knew the Lambda would be invoked through some method hook, and that the method would be handed a chunk of JSON that described the event to be handled. Fair enough, first step, let’s go find a way to parse JSON.

My research led to two possibilities. First, the supplied Go libraries are pretty good at unmarshalling JSON to a map. I grabbed an example of the JSON message from AWS, and started there. Hmm. Yes, I can deserialise that way, but the code is a bit ugly. A cleaner solution is the GJson library from Josh Baker. A bit of a steeper learning curve, but the code looks nicer, and it has the sorts of semantics you may be used to from jq — win!

Right, that’s the JSON bit sorted out, now how do I go about talking to S3 to read and write objects… enter the AWS SDK for Go. This is a really nice SDK. It’s well documented, well aligned with the ever-evolving API, and bears the hallmarks have having been thoughtfully put together with the goal to making the developers’ lives easier. Um, and yeah, completely throws the day-and-a-bit of fiddling with JSON parsing out the window.

The first thing we see is that there’s a piece of boiler plate in the main() function:

// main function is invoked when the lambda is launched
func main() {
    lambda.Start(HandleLambdaEvent)
}

Oh, yeah, and there’s some imports needed to be able to use the SDK — you can see that the modules in the SDK are intelligently bundled, so you can easily pick and choose the services needed.

import (
   "github.com/aws/aws-lambda-go/events"
   "github.com/aws/aws-lambda-go/lambda"
   "github.com/aws/aws-sdk-go/aws"
   "github.com/aws/aws-sdk-go/aws/session"
   "github.com/aws/aws-sdk-go/service/s3"
   .
   .
   .
)

So, that main() gets called when an instance of the function is run up by the surrounding Lambda infrastructure. It’s only purpose is to register the actual event handler, although it can be used for doing any setup your code needs. I opted to pop most of the setup into an init() function, because that seems a more Go kind of thing to do, but it’s functionally the same:

func init() {
   params = &runtimeParameters{
      SourcePrefix:      validatePrefix(os.Getenv("SOURCE_PREFIX"), DefaultSrcPrefix),
      DestinationPrefix: validatePrefix(os.Getenv("DESTINATION_PREFIX"), DefaultDestPrefix),
      DestinationBucket: validateDestination(os.Getenv("DESTINATION_BUCKET")),
      Region:            validateRegion(os.Getenv("AWS_REGION")),
   }

   sess, err := session.NewSession(&aws.Config{
      Region: aws.String(params.Region),
   })
   if err != nil {
      log.Fatal("Error starting session", err)
   }

   params.S3service = s3.New(sess)
}

Nothing fancy there — I grab some variables from the environment, coerce them to sensible defaults, and start up an AWS session. You might notice that the session is used to create a service that is our facade for S3. Looking through the documentation, that’s very much the pattern of the SDK: each AWS service has a dedicated proxy service, allowing operations against the AWS account to map neatly onto the same organisation of services that AWS provides.

Right, the juicy bit is the handler itself:

func HandleLambdaEvent(request events.S3Event) (int, error) {

Oops. Should have read the documentation first. The surrounding infrastructure in Lambda for Go means that the handler is given a completely well formed struct. No JSON parsing needed. What have we learned? Read the (fine) manual.

I would also suggest that a decent IDE is a help, which sounds rather obvious, but sometimes it needs to be said. I’m currently using IntelliJ, and it’s support for Go is quite nice, although the interface for configuring test runs is currently a little clunky. For me the key part is that the IDE makes if trivially easy to jump into the SDK code, and easily jump across to any available documentation.

That dedication to visibility in the Go community is an utter joy. In the case of the AWS SDK that means we have the code itself, and:

• The Go documentation in standard form at https://pkg.go.dev/github.com/aws/aws-sdk-go-v2
• The source code at https://github.com/aws/aws-sdk-go-v2

Go is proving to be an ideal match for AWS Lambda so far. It removes the drawback of Python, where it’s quite a pain to bundle all the required dependencies, and it removes the bloat and large memory footprint of Java. The code compiles down into a single binary (albeit one larger than I would desire from my old C days) which you zip, and there’s your deployable. It’s also quite lovely that building for Lambda deploy can be done on the developer’s machine just by specifying the target run time:

$ GOOS=linux go build main.go
$ zip function.zip main
$ aws lambda create-function \
    --function-name my-function \
    --runtime --zipfile files://function.zip \
    --handler main \
    --role arn:aws:iam::1234567890:role/my-lambda-role

Although that’s not how I’m actually doing the build…

The key take aways for me so far:

1. Go makes it really easy to write Lambda functions
2. The AWS SDK is our friend
3. Read the documentation!