Serverless framework: a small example

Use case

I've been willing to give the serverless framework a try for a while, and recently I came up with a small side project that was a potential good fit for it.

I am a Monzo user, and I use their card to pay for things like my weekday lunches around the office or my travel expenses like Oyster Card top ups. I also happen to be a user of Toshl, which helps me with my personal finances and budgets.

Typically, whenever my Oyster card gets topped up or I pay for a lunch in a restaurant, I then input that expense into Toshl. But turns out both Monzo and Toshl are developer friendly, so I thought I could easily automate that process by having a system that would get certain Monzo expenses directly into Toshl for me, saving me a few seconds per day.

A serverless approach was ideal for this, especially since AWS offers a free tier for its services, meaning this would also not cost me a penny.

High level design

Monzo provides web hooks that will make an HTTP request to an endpoint of your choice whenever a transaction happens in your account. Toshl also provides an API to create expenses. The way we can tie them together is through a couple of AWS Lambdas, one of them being behind an API Gateway, talking to each other through SNS notifications. A diagram of the architecture can be found below.

Architecture of the solution

In a nutshell the journey will be as follows:

Something gets paid with your Monzo card.
Monzo sends a request to an endpoint we have previously configured.
This endpoint goes to an AWS API Gateway, which will map the request to an AWS Lambda named monzo.
The monzo lambda will inspect the transaction that's been created, and determine via very basic rules if it's an expense we'd like to track in Toshl.
If the transaction is of interest, it will then publish a message to an SNS topic with the details of the expense.

In parallel to this process, the following will also be happening:

Another AWS Lamnda named toshl will be subscribed to the SNS topics where expenses get sent to.
Whenever a message gets published to that topic, the lambda will read the message and make an appropriate HTTP request to the Toshl API so the expense appears in our timeline.

We achieve a few things with this approach, amongst them:

We decouple Monzo from Toshl. Monzo requests will be handled by a lambda, and if needed that lambda will just publish a message to an SNS topic indicating a new expense has been created.
Similarly, the Toshl lambda knows nothing about Monzo either, it just knows that expenses get published to an SNS topic and it needs to transform them into an HTTP request to be made to the Toshl API. If in the future any other system was interested in knowing about new expenses, we'd only have to plug new lambdas (or any other type of supported receiver) to the SNS topic (e.g. Kafka, DynamoDB, SQS, another SNS topic...).
We only pay for what we need. The API Gateway needs to be always up and running to listen to requests, but we don't pay anything for being idle in any other case. Only when a Monzo transaction happens we'll run a lambda, and only when that transaction is actually relevant to us, we'll then pay for sending a message to SNS, and later on retrieving it and running the toshl lambda.

This also means that for the low traffic nature of the system and the AWS Free Tier, we will end up not paying a single penny for all of the above.

All the code explained in this post is available at https://github.com/brafales/expenses and should be ready to run, with the appropriate environment variables set up.

Creating our infrastructure with the Serverless Framework

The serverless framework allows you to define all the moving pieces you need fairly easily. I'll show you the final serverless.yml file of the project while explaining all the sections individually. While most of it is standard, I used a couple of third party plugins to enable the use of my own domain in API Gateway and to configure an SQS dead letter queue for the toshl lambda (more on dead letter queues later).

serverless.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
service: ${self:custom.environment.service}
app: ${self:custom.environment.app}
tenant: ${self:custom.environment.tenant}

frameworkVersion: ">=1.28.0 <2.0.0"

provider:
  name: aws
  runtime: go1.x

  stage: dev

  # Allow IAM role to publish to the SNS topic
  iamRoleStatements:
    - Effect: Allow
      Action:
        - SNS:Publish
      Resource: ${self:custom.snsTopicArn}
    - Effect: Allow
      Action:
        - SQS:SendMessage
      Resource: ${self:custom.sqsToshlDLQArn}

plugins:
  - serverless-domain-manager
  - serverless-plugin-lambda-dead-letter

custom:
  environment: ${file(env.yml)}
  customDomain:
    domainName: ${self:custom.environment.domainName}
    basePath: ''
    stage: ${self:provider.stage}
    createRoute53Record: true

  # SNS/SQS configuration
  snsTopic: "${self:service}-${self:provider.stage}-expense-created"
  snsTopicArn: { "Fn::Join" : ["", ["arn:aws:sns:${self:provider.region}:", { "Ref" : "AWS::AccountId" }, ":${self:custom.snsTopic}" ] ]  }
  sqsToshlDLQ: "${self:custom.snsTopic}-toshl-dlq"
  sqsToshlDLQArn: { "Fn::Join" : ["", ["arn:aws:sqs:${self:provider.region}:", { "Ref" : "AWS::AccountId" }, ":${self:custom.sqsToshlDLQ}" ] ]  }

package:
 exclude:
   - "./**"
 include:
   - "./bin/**"

functions:
  monzo:
    handler: "bin/monzo"
    events:
      - http:
          path: "api/v1/create"
          method: "post"
    environment:
      snsTopicArn: ${self:custom.snsTopicArn}
      categoryData: ${ssm:/${self:service}/CategoryData}

  toshl:
    handler: "bin/toshl"
    events:
      - sns: ${self:custom.snsTopic}
    deadLetter:
      sqs: ${self:custom.sqsToshlDLQ}
    environment:
      snsTopicArn: ${self:custom.snsTopicArn}
      token: ${ssm:/${self:service}/ToshlToken~true}
      accountId: ${ssm:/${self:service}/ToshlAccountId}
      categoryData: ${ssm:/${self:service}/CategoryData}

The first three lines are serverless parameters. You can have different applications in your account, and an application can be composed of one or more services. For this example, we will just have one application and one service. The tenant parameter is linked to your username in the serverless dashboard.

As you can see the values for these parameters are not hardcoded, but instead have the form ${self:custom.environment.variable_name}. This is the pattern the serverless framework uses to interpolate environment variables into your yml file. On line 29, you can see that under the custom key we have environment: ${file(env.yml)}. What this will do is read the env.yml file from the current directory, and make its contents available under the self:custom.environment key for you to use throughout the rest of the serverless.yml file. This way you can have your env.yml on the .gitignore file and ensure your sensitive data does not get leaked. This means than in order for this example to work, you'll have to create your own env.yml file. Find below a sample one:

service: expenses
app: expenses
tenant: your_serverless_user
domainName: your-custom-domain.com

Moving on to line 5, we can tell serverless what framework versions our service is compatible with.

Lines 7 to 26 set the provider configuration. The serverless framework supports a few of them. For our example, we'll use aws, and also for our lambdas we will use the go1.x runtime (at the time of writing this post, AWS Lambda supported Go, Node.js, Java, Python and .NET Core).

Serverless also allows you to have different stages for deployment. These are like development environments, so you could have a development stage, a staging stage, and a production stage, for example. Again, in our case we'll just stick to one stage and we'll call it dev.

Lines 14 to 22 are interesting. By default, the serverless framework will manage IAM Roles for you. These roles will control what permissions each of the created artifacts will have. In our use case, we need to ensure that the API Gateway has permissions to invoke the monzo lambda, and so under the hood, serverless will ensure that the permission lambda:InvokeFunction is set to Allow for the Arn of our monzo lambda. However, sometimes you may be interested in adding permissions outside of what is automatically managed by the framework. If we look back at our architecture, we'll see that the monzo lambda will need to publish messages to an SNS topic. And the toshl lambda will need to publish to SQS for its dead letter queue (where messages we failed to process will go for inspection). Serverless doesn't yet provide this functionality out of the box, so the extra permissions need to be set here.

iamRoleStatements:
  - Effect: Allow
    Action:
      - SNS:Publish
    Resource: ${self:custom.snsTopicArn}
  - Effect: Allow
    Action:
      - SQS:SendMessage
    Resource: ${self:custom.sqsToshlDLQArn}

We are adding SNS:Publish permissions to the AWS resource with the Arn ${self:custom.snsTopicArn}, and SQS:SendMessage permissions to the AWS resource with Arn ${self:custom.sqsToshlDLQArn}. Again, the resource Arns are not being hardcoded, but are references to another part of the yml file, which are configured under the custom key. Let's have a look at them.

custom:
  environment: ${file(env.yml)}
  customDomain:
    domainName: ${self:custom.environment.domainName}
    basePath: ''
    stage: ${self:provider.stage}
    createRoute53Record: true

  # SNS/SQS configuration
  snsTopic: "${self:service}-${self:provider.stage}-expense-created"
  snsTopicArn: { "Fn::Join" : ["", ["arn:aws:sns:${self:provider.region}:", { "Ref" : "AWS::AccountId" }, ":${self:custom.snsTopic}" ] ]  }
  sqsToshlDLQ: "${self:custom.snsTopic}-toshl-dlq"
  sqsToshlDLQArn: { "Fn::Join" : ["", ["arn:aws:sqs:${self:provider.region}:", { "Ref" : "AWS::AccountId" }, ":${self:custom.sqsToshlDLQ}" ] ]  }

We've covered the environment bit before, so we're going to skip it. We'll also skip the customDomain bit as it's out of the scope of this post, but you can use that alongside the plugin serverless-domain-manager to point your own domain to the API Gateway endpoint. The lines below are where we can configure our own variables to be used somewhere else. For our use case we need two variables: an SNS topic and an SQS queue. With snsTopic and sqlToshlDLQ we set human readable names by interpolating a few other variables. In this case the values would end up being expenses-dev-expense-created and expenses-dev-expense-created-toshl-dlq. We then use these human readable values to figure out what the AWS Arn will be, by using some of the provided AWS CloudFormation functions like Fn::Join (to join a list of values with a delimiter) or Ref (to get references to existing AWS resources). We use Ref to get access to our AWS Account ID, as it's required to build the Arn string, and it's a value AWS has access to through the credentials we use to execute serverless. So, if your account ID was 1234, the snsTopicArn would end up having the value arn:aws:sns:us-east-1:1234:expenses-dev-expense-created.

Let's move to the package key now, where the packaging of our code happens. In here we can decide what files to include and exclude in our package. Serverless will create a zip file when run, and upload it to an S3 bucket, where AWS Lambda will go to find out what to execute when certain lambdas are triggered.

In order to understand the exclude and include parameters, we need to look at our Makefile first:

GOPACKAGES = $(shell go list ./...  | grep -v /vendor/)

build:
    dep ensure -v
    env GOOS=linux go build -ldflags="-s -w" -o bin/monzo monzo/main.go
    env GOOS=linux go build -ldflags="-s -w" -o bin/toshl toshl/main.go

.PHONY: clean
clean:
    rm -rf ./bin ./vendor Gopkg.lock

.PHONY: deploy
deploy: clean build
    sls deploy --verbose

silent_deploy: clean build
    sls deploy > /dev/null 2>&1

test:
@go test -v $(GOPACKAGES)

When building our go binaries, we will output them into the ./bin folder. The monzo lambda will go to ./bin/monzo and the toshl lambda will go to ./bin/toshl.

This means that what we really want to package and upload to S3 is the ./bin folder only. By excluding ./** we exclude from packaging essentially everything except what's on the inclusion list, and so by then including ./bin/** we are effectively only packaging the ./bin folder which is exactly what we need.

The functions key is the one that actually sets the serverless artifacts for us. It has two keys, monzo and toshl, as we have 2 lambdas.

The monzo lambda

The monzo lambda has one job: process an incoming Monzo transaction request, filter it if we are interested in it, and then publish a message to an SNS topic with the information of the expense we want to record. The definition of the function in serverless is quite straightforward:

monzo:
  handler: "bin/monzo"
  events:
    - http:
        path: "api/v1/create"
        method: "post"
  environment:
    snsTopicArn: ${self:custom.snsTopicArn}
    categoryData: ${ssm:/${self:service}/CategoryData}

We tell AWS lambda that the binary in bin/monzo should be the one handling the request through the handler key. In events we tell API Gateway to map POST requests to the URL api/v1/create to the lambda. Finally, via the environment block, we pass two environment variables to the running process or handler: snsTopicArn and categoryData. snsTopicArn is the SNS topic where we want to publish messages, and the categoryData is a JSON string with information that will let us filter the transactions we are interested in. We get the SNS topic from a variable set up in the same serverless.yml file, and the category data we get from AWS Parameter Store, more specifically from the key expenses/CategoryData. You will have to manually populate this to your needs. For example, if you're only interested in transactions labelled as entertainment, and map them to a Toshl category named Entertainment, the string will look like this: {"entertainment":"Entertainment"}

To summarise, whenever a POST request is made to the api/v1/create URL, API Gateway will work its magic, transform that request into a suitable object, and invoke the executable in bin/monzo, also providing the necessary environment variables as a way to configure our lambda.

Let's have a look at the Go code that will execute as a lambda:

monzo/main.go

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
package main

import (
    "encoding/json"
    "os"

    "github.com/aws/aws-lambda-go/lambda"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/sns"
    "github.com/brafales/expenses/monzo/handler"
)

func main() {
    categories, err := getCategories()
    if err != nil {
        panic(err)
    }
    h := handler.Handler{
        SnsTopicArn: os.Getenv("snsTopicArn"),
        Categories:  categories,
        SNSClient:   sns.New(session.New()),
    }
    lambda.Start(h.Handle)
}

func getCategories() ([]string, error) {
    rawData := os.Getenv("categoryData")
    var categoryMap map[string]string
    err := json.Unmarshal([]byte(rawData), &categoryMap)
    if err != nil {
        return []string{}, err
    }
    categories := make([]string, 0, len(categoryMap))
    for k := range categoryMap {
        categories = append(categories, k)
    }

    return categories, nil
}

AWS provides a Lambda SDK for Go. The lambda is a Go binary that needs to define a main function, and that main function needs to eventually call a lambda handler function. The signature of the lambda handler function depends on the type of event we are handling. In our case we are handling an API Gateway event, and so our handler looks like this:

monzo/hanler/handler.go

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
package handler
package handler

import (
	"context"
	"encoding/json"
	"time"

	"github.com/aws/aws-lambda-go/events"
	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/service/sns"
	"github.com/aws/aws-sdk-go/service/sns/snsiface"
)

// Response is of type APIGatewayProxyResponse since we're leveraging the
// AWS Lambda Proxy Request functionality (default behavior)
//
// https://www.serverless.com/framework/docs/providers/aws/events/apigateway/#lambda-proxy-integration
type Response events.APIGatewayProxyResponse

// {
//     "type": "transaction.created",
//     "data": {
//         "account_id": "acc_00008gju41AHyfLUzBUk8A",
//         "amount": -350,
//         "created": "2015-09-04T14:28:40Z",
//         "currency": "GBP",
//         "description": "Ozone Coffee Roasters",
//         "id": "tx_00008zjky19HyFLAzlUk7t",
//         "category": "eating_out",
//         "is_load": false,
//         "settled": true,
//         "merchant": {
//             "address": {
//                 "address": "98 Southgate Road",
//                 "city": "London",
//                 "country": "GB",
//                 "latitude": 51.54151,
//                 "longitude": -0.08482400000002599,
//                 "postcode": "N1 3JD",
//                 "region": "Greater London"
//             },
//             "created": "2015-08-22T12:20:18Z",
//             "group_id": "grp_00008zIcpbBOaAr7TTP3sv",
//             "id": "merch_00008zIcpbAKe8shBxXUtl",
//             "logo": "https://pbs.twimg.com/profile_images/527043602623389696/68_SgUWJ.jpeg",
//             "emoji": "🍞",
//             "name": "The De Beauvoir Deli Co.",
//             "category": "eating_out"
//         }
//     }
// }
type monzoEvent struct {
	Type string `json:"type"`
	Data struct {
		AccountID   string    `json:"account_id"`
		Amount      int       `json:"amount"`
		Created     time.Time `json:"created"`
		Currency    string    `json:"currency"`
		Description string    `json:"description"`
		ID          string    `json:"id"`
		Category    string    `json:"category"`
		IsLoad      bool      `json:"is_load"`
		Merchant    struct {
			Address struct {
				Address   string  `json:"address"`
				City      string  `json:"city"`
				Country   string  `json:"country"`
				Latitude  float64 `json:"latitude"`
				Longitude float64 `json:"longitude"`
				Postcode  string  `json:"postcode"`
				Region    string  `json:"region"`
			} `json:"address"`
			Created  time.Time `json:"created"`
			GroupID  string    `json:"group_id"`
			ID       string    `json:"id"`
			Logo     string    `json:"logo"`
			Emoji    string    `json:"emoji"`
			Name     string    `json:"name"`
			Category string    `json:"category"`
		} `json:"merchant"`
	} `json:"data"`
}

type expenseEvent struct {
	Amount      int       `json:"amount"`
	Created     time.Time `json:"created"`
	Currency    string    `json:"currency"`
	Description string    `json:"description"`
	Category    string    `json:"category"`
}

// Handler is a type that will handle a Monzo request coming through an API Gateway Proxy Request
type Handler struct {
	SnsTopicArn string
	Categories  []string
	SNSClient   snsiface.SNSAPI
}

// Handle handles a Monzo request coming through an API Gateway Proxy Request
func (h *Handler) Handle(ctx context.Context, request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
	event := &monzoEvent{}

	if err := json.Unmarshal([]byte(request.Body), event); err != nil {
		return events.APIGatewayProxyResponse{Body: err.Error(), StatusCode: 500}, err
	}

	if h.interested(*event) {
		err := h.publishEvent(*event)
		if err != nil {
			return events.APIGatewayProxyResponse{Body: err.Error(), StatusCode: 500}, err
		}
	}

	return events.APIGatewayProxyResponse{Body: "OK", StatusCode: 200}, nil
}

func (h *Handler) publishEvent(event monzoEvent) error {
	expense := expenseEvent{
		Amount:      event.Data.Amount,
		Created:     event.Data.Created,
		Description: event.Data.Description,
		Category:    event.Data.Category,
		Currency:    event.Data.Currency,
	}
	expenseBytes, err := json.Marshal(expense)
	if err != nil {
		return err
	}

	params := &sns.PublishInput{
		Message:  aws.String(string(expenseBytes)),
		TopicArn: aws.String(h.SnsTopicArn),
	}
	_, err = h.SNSClient.Publish(params)

	if err != nil {
		return err
	}
	return nil
}

func (h *Handler) interested(event monzoEvent) bool {
	for _, v := range h.Categories {
		if v == event.Data.Category {
			return true
		}
	}
	return false
}

Let's quickly go through the code to understand what it does. The main function starts by reading the categoryData environment variable (which AWS will set up for us) and fills a string array with it, which will contain the list of transaction categories we are interested in.

After this, it instantiates a new handler, passing in some configuration options (the SNS topic, the list of categories we just retrieved and an SNS client to talk to SNS), and uses this handler's Handle function as an argument of lambda.Start.

Once in the handler, we get the API Gateway request body and unmarshal it into our own struct type monzoEvent. We check if the category of the monzo event is in the list of categories we care about, and if that's the case, we build an expenseEvent payload with the information we require, and we publish an SNS message.

If all goes well, our handler returns the return events.APIGatewayProxyResponse{Body: "OK", StatusCode: 200}, nil tuple. The AWS Lambda SDK will interpret that as a success and return the appropriate 200 HTTP response code back. If something fails, we let the Handle method return an error and an appropriate response code (in our case it'd be a 500 error), with the error string as the body.

The toshl lambda

The toshl lambda works the opposite way: picks up a message from an SNS topic, and as a result ends up making an HTTP request to the Toshl API.

toshl:
  handler: bin/toshl
  events:
    - sns: ${self:custom.snsTopic}
  deadLetter:
    sqs: ${self:custom.sqsToshlDLQ}
  environment:
    snsTopicArn: ${self:custom.snsTopicArn}
    token: ${ssm:/${self:service}/ToshlToken~true}
    accountId: ${ssm:/${self:service}/ToshlAccountId}
    categoryData: ${ssm:/${self:service}/CategoryData}

In our serverless config, we link the event to the SNS topic we have configured. This will instruct AWS to create a subscription to the SNS topic piped to our lambda. We also configure a deadLetter parameter. This comes from the plugin serverless-plugin-lambda-dead-letter and allows us to use the Dead Letter Queue pattern for our lambda. The way the Dead Letter Queue will work is as follows:

A message is sent to the SNS topic, and processed by the lambda.
If the lambda fails to process the message by returning an error code, AWS will retry a few times, applying exponential backoff between retries.
If the lambda fails to process the message after all retries, the message will get queued in the configured Dead Letter Queue. We can then inspect the failed messages in that queue for analysis if we need to. This way we ensure that we never lose important information, and that we can always replay those events if they were caused by a bug in our software or by a transient error (e.g. Toshl had an outage).

This is what the Go code looks like for the lambda itself:

toshl/main.go

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
package main

import (
    "context"
    "encoding/json"
    "net/http"
    "os"

    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
    "github.com/brafales/expenses/toshl/client"
)

// Handler will handle the lambda function
func Handler(ctx context.Context, snsEvent events.SNSEvent) {
    categories, err := categories()
    if err != nil {
        panic(err)
    }
    toshlClient := client.Client{
        AuthToken:    os.Getenv("token"),
        AccountID:    os.Getenv("accountId"),
        HTTPClient:   http.Client{},
        CategoryMap:  categories,
        ToshlBaseURL: "https://api.toshl.com",
    }

    for _, record := range snsEvent.Records {
        snsRecord := record.SNS

        expense, err := createExpense(snsRecord.Message)
        if err != nil {
            panic(err)
        }
        err = toshlClient.CreateEntry(&expense)
        if err != nil {
            panic(err)
        }
    }
}

func main() {
    lambda.Start(Handler)
}

func categories() (map[string]string, error) {
    rawData := os.Getenv("categoryData")
    var categoryMap map[string]string
    err := json.Unmarshal([]byte(rawData), &categoryMap)
    if err != nil {
        return map[string]string{}, err
    }

    return categoryMap, nil
}

func createExpense(message string) (client.Expense, error) {
    var expense client.Expense
    err := json.Unmarshal([]byte(message), &expense)
    return expense, err
}

toshl/client/client.go

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
package client

import (
    "bytes"
    "encoding/json"
    "fmt"
    "net/http"
    "time"
)

// Client is a client to talk to Toshl's REST API
type Client struct {
    AuthToken    string
    AccountID    string
    HTTPClient   http.Client
    CategoryMap  map[string]string
    ToshlBaseURL string
}

// Expense defines a Toshl Entry
type Expense struct {
    Amount      int       `json:"amount"`
    Created     time.Time `json:"created"`
    Currency    string    `json:"currency"`
    Description string    `json:"description"`
    Category    string    `json:"category"`
}

type toshlCurrency struct {
    Code string `json:"code"`
}

type toshlEntry struct {
    Amount   float64       `json:"amount"`
    Currency toshlCurrency `json:"currency"`
    Date     string        `json:"date"`
    Desc     string        `json:"desc"`
    Account  string        `json:"account"`
    Category string        `json:"category"`
}

type toshlCategory struct {
    ID   string `json:"id"`
    Name string `json:"name"`
}

// CreateEntry entry will create a new entry in Toshl based on the Expense data
func (c *Client) CreateEntry(expense *Expense) error {
    data, err := c.newToshlEntry(expense)
    if err != nil {
        return err
    }

    dataJSON, err := json.Marshal(data)
    if err != nil {
        return err
    }

    req, err := http.NewRequest("POST", fmt.Sprintf("%s/entries", c.ToshlBaseURL), bytes.NewBuffer(dataJSON))
    if err != nil {
        return err
    }
    req.SetBasicAuth(c.AuthToken, "")
    req.Header.Set("Content-Type", "application/json")
    resp, err := c.HTTPClient.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    respCode := resp.StatusCode
    if respCode != 201 {
        return fmt.Errorf("Something went wrong, response code was %d", respCode)
    }

    return nil
}

func (c *Client) newToshlEntry(expense *Expense) (toshlEntry, error) {
    category, err := c.categoryID(expense.Category)
    if err != nil {
        return toshlEntry{}, err
    }

    entry := toshlEntry{
        Amount: float64(expense.Amount) / 100,
        Currency: toshlCurrency{
            Code: expense.Currency,
        },
        Date:     expense.Created.Format("2006-01-02"),
        Desc:     expense.Description,
        Account:  c.AccountID,
        Category: category,
    }

    return entry, nil
}

func (c *Client) categoryID(categoryName string) (string, error) {
    mappedCategoryName := c.CategoryMap[categoryName]
    if mappedCategoryName == "" {
        return "", fmt.Errorf("Category %s not in the mapping", categoryName)
    }
    req, err := http.NewRequest("GET", fmt.Sprintf("%s/categories", c.ToshlBaseURL), nil)
    if err != nil {
        return "", err
    }
    req.SetBasicAuth(c.AuthToken, "")
    req.Header.Set("Content-Type", "application/json")
    resp, err := c.HTTPClient.Do(req)
    if err != nil {
        return "", err
    }
    defer resp.Body.Close()
    var categories []toshlCategory
    err = json.NewDecoder(resp.Body).Decode(&categories)
    if err != nil {
        return "", err
    }
    var category string
    for _, v := range categories {
        if v.Name == mappedCategoryName {
            category = v.ID
        }
    }
    if category == "" {
        return "", fmt.Errorf("Category not found in Toshl for name %s", mappedCategoryName)
    }
    return category, nil
}

In the main entry point we call a handling function. This handler function will do a few things:

It'll read the categoryData environment variable to build a category mapping. This will allow us to map incoming expenses categories into Toshl categories.
It'll then create an instance of a client struct, which we will use to communicate with Toshl. For this client, we need some other secrets we read from environment variables too.
Finally, it'll loop through the records in the SNS event, and will process them by mapping them into an expense and then telling the Toshl client to create an entry. An SNS event can have multiple records, if a few messages have been published to the topic in quick succession.

The Toshl client struct is a pretty straightforward Go HTTP client talking to a REST JSON API.

Putting it all together

Assuming you have set up your serverless account properly and have configured your AWS credentials, you are ready to go. Check out the code, and in the root folder, type in make deploy, and you're good to go. This is what should typically happen:

The binaries for your two lambdas will be built in the ./bin folder.
serverless will then package your binaries in a zip file and upload them into an S3 bucket.
serverless will then start creating all the AWS artifacts: the API Gateway, the 2 lambdas, an SNS topic, an SQS queue, all the subscriptions between SNS topics, SQS queues, API Gateway requests and Lambdas.

As an output, and assuming everything went ok, you'll get a URL. This is the URL of your API Gateway endpoint. You can then pick that up and make a call to the Monzo API so their webhooks point at that same URL. After that, your Monzo expenses will be linked and will magically appear in Toshl.