Creating CakeBalance

So I used to be a pretty big fan of budgeting apps like Emma and Money Dashboard but they got annoying with their notifications and I realised after a few months the only thing I was really interested in was keeping track of my balances over time. It's nice to see my savings going up as I say no to a new game or whatever.

I've recently got into Grafana and thought it would be cool if I could keep track of my balances there instead. I'd (partially) created by own budgeting app, like the ones above, so I knew how to get the data and I'm starting to get the hang of Grafana and Prometheus so why not create a simple app that links them together and lets me learn some more about these technologies?

This post is more about the steps I took to do this rather than a tutorial on how to set it up. You can find that on the github page here.

You can also find the docker image here.

Talking to the banks

Open Banking is the current best way to talk to banks. However, to be a provider you have to comply to some strict rules and I don't have the time or the resources to go through that whole process. Luckily a bunch of other people have. The only one I'm familiar with is TrueLayer. I've used it before to build my own budgeting app so I'm going to use it again for this.

TrueLayer acts as a middleman. For a fee. Luckily they provide a free developers sandbox which you can use to "try things out".

The plan

So to get this working I needed to make the following;

  • A way to interact with TrueLayer
  • A way for users to add accounts
  • An endpoint for prometheus to use which would supply some data
  • Configuration for prometheus
  • A Grafana dashboard

Interacting with TrueLayer

Their documentation is pretty good, and I've done this bit before, so nothing here is an issue. I did decide to keep this seperate to the main application just in case I wanted to use it again later or I wanted to swap it out. So it's in it's own library CakeBudget.TrueLayer.

You can see the code on github so I wont go into anything specific here. The overall summary is that they use an authorization code flow and you can make requests to their API to get information about your bank like balances and transactions.

The only "gotcha" here is something that is only going to be a "gotcha" if you've never dealt with refresh tokens before.

So basically you have an access token that you send with each request that says who are you (read more about JWTs here) but that token expires after a certain time. No biggie - we get a refresh token that lets us ask for a new access token. But what if we get 2 threads that notice it's invalid and BOTH try and get a new one? well one is going to fail because a refresh token can only be used once. So you need to manage it somehow.

Also, to make things a bit more hectic, each "connection" to a bank will have its own authentication process and set of tokens. So one token to talk to BankA and another to talk to BankB.

Configuration

I've always found apps that I run locally are easier to configure when I have to edit some sort of file. On the flip side I've found environment variables are better when configuring something in Kubernetes. So I wanted to make sure that this app worked with both.

It was actually pretty easy to do.

public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureAppConfiguration(config =>
                {
                    config.AddJsonFile(
                    		"config.json",
                            optional: true,
                            reloadOnChange: true);
                    config.AddEnvironmentVariables();
                })
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder
                        .UseUrls("http://0.0.0.0:5100")
                        .UseStartup<Startup>();
                });

This let me either configure things in json, e.g;

{
  "TrueLayerSettings": {
    "TokenSaveLocation": "tokens.json",
    "TrueLayerClientId": "your_truelayer_clientId",
    "TrueLayerClientSecret": "your_truelayer_clientSecret",
    "TrueLayerRedirectUri": "https://cakebalance/admin/callback",
    "Scope": "info%20accounts%20balance%20cards%20transactions%20direct_debits%20standing_orders%20offline_access",
    "Providers": "uk-ob-all%20uk-oauth-all%20uk-cs-mock"
  }
}

or with environment variables like;

Windows Linux
TrueLayerSettings:TokenSaveLocation TrueLayerSettings__TokenSaveLocation
TrueLayerSettings:TrueLayerClientId TrueLayerSettings__TrueLayerClientId
TrueLayerSettings:TrueLayerClientSecret TrueLayerSettings__TrueLayerClientSecret
TrueLayerSettings:TrueLayerRedirectUri TrueLayerSettings__TrueLayerRedirectUri
TrueLayerSettings:Scope TrueLayerSettings__Scope
TrueLayerSettings:Providers TrueLayerSettings__Providers
TrueLayerSettings:NonceGenerator TrueLayerSettings__NonceGenerator

Adding accounts

I wanted to make this as easy as possible because adding and keeping accounts "valid" is, in my opinion, the most annoying thing in those other apps I tried and also a problem for me in my own app. There's only so much I can do when the actual work is done by TrueLayer but I wanted to make sure the part between me and them was easy peasy.

To do this I added a single endpoint that displays the status of the current connections simply as "Valid" or "Invalid". Connections should only go invalid if I screw up the refreshing of the tokens, the refresh token becomes invalid over time, or TrueLayer have an issue.

I figured this was probably also a good place to spit out some of the settings the user will have entered to help with debugging.

In the end it looks something like this;

{
  "connectorStatuses": [
    {
      "connectorId": "oauth-bankA",
      "isValid": true
    },
    {
      "connectorId": "ob-bankB",
      "isValid": true
    }
  ],
  "settings": {
    "trueLayerClientId": "......",
    "trueLayerClientSecret": "**********",
    "trueLayerRedirectUri": "......",
    "scope": "info%20accounts%20balance%20cards%20transactions%20direct_debits%20standing_orders%20offline_access",
    "providers": "uk-ob-all%20uk-oauth-all%20uk-cs-mock",
    "nonceGenerator": 0,
    "authenticationLink": "https://auth.truelayer.com/?response_type=code&client_id......&nonce=1793378647&scope=info%20accounts%20balance%20cards%20transactions%20direct_debits%20standing_orders%20offline_access&redirect_uri=......&providers=uk-ob-all%20uk-oauth-all%20uk-cs-mock"
  },
  "gitHub": "https://github.com/AdamAwan/CakeBalance/",
  "blogPost": "https://blog.wastedcake.com/"
}

So the flow for the user is basically;

  • go to https://somesite.com/admin
  • click the authentication link
  • follow the steps on TrueLayers side
  • callback url automatically finishes my side of the auth flow
  • redirect them back to https://somesite.com/admin

This will let them know straight away if it was added and the status of the other ones. To fix an invalid connection they just need to do the same thing again.

Prometheus Endpoint

I have a tiny bit of experience with prometheus and by that I mean I've hooked up endpoints made by others following tutorials :)

From previous "hooking up" I knew what I wanted was something I could add to the prometheus.yml that looked a little something like a node-exporter one;

  - job_name: node-exporter
    scheme: http
    static_configs:
      - targets:
          - *.*.*.*:9100

and I also knew that I needed the endpoint to spit out some very confusing data like this;

# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 915968
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 3033
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 658
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 2.240512e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge

AND MORE STUFF

I didn't get it at all. I don't even know what I really want at this stage.

I did know prometheus stores data so I don't need this endpoint to give anything historical - it just needs the current state. So something like this?

balance_available_account_1 100
balance_available_account_2 300

I tried to build this stuff manually but I didn't understand the differences. Then I stumbled upon this fantastic library prometheus-net that seemed overkill for what I wanted but actually in hindsight it's perfect.

So after a bunch of trial and error I learned what I wanted was called a "Gauge". A Gauge is a numeric value which changes arbitrarily. Different to a counter which starts at zero and only increases. In this context I guess my bank balance is arbitrary... not sure how I feel about that.

Then I learned about "labels" which are basically just labels but they let you add some metadata on to the metrics. It's recommended to minimize the different amount of label values. I decided my labels for my data were going to be

  • Provider because I wanted to know the bank and figured this was a pretty limited set of words
  • Display Name... not limited at all in the world but for my accounts it is
  • Currency which is super limited for me as I only have GBP but thought it'd be good to include it anyway as others will likely have multiple.

In code this ended up looking like this;

First you have to setup the metrics.

_totalAccounts = Metrics.CreateGauge(
	"cakebalance_total_accounts",
    "Number of accounts");

_accountBalancesCurrent = Metrics.CreateGauge(
	"cakebalance_account_balances_current", "",
    new GaugeConfiguration(){
		LabelNames = new[] { "Provider", "DisplayName", "Currency" }
    });
            
_accountBalancesAvailable = Metrics.CreateGauge(
	"cakebalance_account_balances_available", "",
    new GaugeConfiguration(){
        LabelNames = new[] { "Provider", "DisplayName", "Currency" },
	});

Then, in this case, you have to add a "Before Collect Callback" so we can find the current values when the endpoint is called. This is just done by looping through each connector, and then each account, and set the metrics with different labels for each account.

_totalAccounts.Set(accountCounter);

_accountBalancesCurrent.WithLabels(
	account.Provider.DisplayName,
    account.DisplayName,
    account.Currency)
    .Set(balance.Current);
 
_accountBalancesAvailable.WithLabels(
	account.Provider.DisplayName,
    account.DisplayName,
    account.Currency)
    .Set(balance.Available);

Doing all that produces this output when you hit the /metrics endpoint;

# HELP cakebalance_account_balances_current 
# TYPE cakebalance_account_balances_current gauge
cakebalance_account_balances_current{Provider="SomeBank",DisplayName="Bank Account",Currency="GBP"} 12.34
cakebalance_account_balances_current{Provider="SomeOtherBank",DisplayName="Other Bank Account",Currency="GBP"} 567.89
# HELP cakebalance_total_accounts Number of accounts
# TYPE cakebalance_total_accounts gauge
cakebalance_total_accounts 2
# HELP cakebalance_account_balances_available 
# TYPE cakebalance_account_balances_available gauge
cakebalance_account_balances_available{Provider="SomeBank",DisplayName="Bank Account",Currency="GBP"} 12.34
cakebalance_account_balances_available{Provider="SomeOtherBank",DisplayName="Other Bank Account",Currency="GBP"} 567.89

Getting the data into prometheus

Prometheus works by having a series of jobs to do on a timer. We just need to add a task so it hits our endpoint every so often.

  - job_name: cakebalance
    scheme: https
    scrape_interval: 60m
    metrics_path: /metrics
    static_configs:
      - targets:
          - url.orip.com

A key part here is the scrape_interval. I don't have enough activity or interest to see balance changes every minute. As well as that, API providers tend to have usage limits. I'm sure 1 request a minute isn't going to hurt them but why push it.

So after waiting for a while you can see if this is working by heading to the Prometheus UI and going to status > targets and you'll see your endpoint. Hopefully it has the state "UP" and no error!

And then if you go into "Graph" you can do a quick query on it and see some data;

At this point everything is looking good. My 2 accounts are there and the amount looks pretty accurate - showing my billions of moneys.

Showing it off in Grafana

I already had prometheus as a data source so I didn't have to do anything there. I wasn't sure how to get the specific data I wanted out though as so far everything I've done in Grafana was with dashboards from the community.

Clicking on "new panel" landed me with a chance for me to pick my metric. Luckily this is a pre populated drop-down so I picked the one I recognized cakebalance_account_balances_current. The moment I did that I got some sensible data being shown to me in maybe a not so unsensible way;

Turns out this combination of grafana and prometheus is pretty smart.

So to get this looking how I wanted I just needed to do a couple of thing;

  • I set to Legend Format to be {{DisplayName}} which means it pulls the DisplayName label from the name. I figured this out by just checking out some other dashboards I had installed.
  • In "Draw Modes" I had to set the null value option to connected which basically meant it would assume the missing data points just connected up
  • Set a title and some other style options

And ended up with this!

When this is a little bit more interesting I'll come back and update it :)

Extra fun stuff

Docker builds!

My codes in github and I want it to be running in Kubernetes. I don't want to be doing this all manually.

Docker hub has this great service that will automatically build your docker images from your github repo and serve them like normal. It's free as well which is awesome.

There's a couple of ways to get it working - either on a commits to a certain branch or with tagging. I personally opted for tagging. There's a bunch of configurations you could do to get this just how you want it but mines really simple, just any tag will make a release. Here's a screenshot which includes their examples as well;

Kubernetes

I moved 90% of the stuff I run in my homelab over to K8s a few months ago so naturally I wanted this sitting in there. The setup was pretty simple, nothing fancy, but if you want to do it as well here's my deployment.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cakebalance
  labels:
    app: cakebalance
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cakebalance
  template:
    metadata:
      labels:
        app: cakebalance
    spec:
      containers:
        - name: cakebalance
          image: adamawan/cakebalance:release-1.0.3
          ports:
            - containerPort: 5100
          env:
            - name: TrueLayerSettings__TokenSaveLocation
              value: "/etc/cakebalance/tokens.json"
            - name: TrueLayerSettings__TrueLayerClientId
              value: "MyClientId
            - name: TrueLayerSettings__TrueLayerClientSecret
              value: "MyClientSecret"
            - name: TrueLayerSettings__TrueLayerRedirectUri
              value: "https://somewhere/admin/callback"
            - name: TrueLayerSettings__Scope
              value: "info%20accounts%20balance%20cards%20transactions%20direct_debits%20standing_orders%20offline_access"
            - name: TrueLayerSettings__Providers
              value: "uk-ob-all%20uk-oauth-all%20uk-cs-mock"
            - name: TrueLayerSettings__NonceGenerator
              value: "1234546789"
          volumeMounts:
            - name: config
              mountPath: /etc/cakebalance/
      volumes:
        - name: config
          persistentVolumeClaim:
            claimName: cakebalance-data-claim
---
apiVersion: v1
kind: Service
metadata:
  name: cakebalance
spec:
  ports:
    - protocol: TCP
      name: cakebalance
      port: 5100
  selector:
    app: cakebalance
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: cakebalance-volume
  labels:
    type: local
    app: cakebalance
spec:
  storageClassName: manual
  capacity:
    storage: 100Mi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/cakebalance"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cakebalance-data-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Mi
  selector:
    matchLabels:
      app: "cakebalance"
---