Setting up Ghost behind a different reverse proxy (træfik)

Whilst installing Ghost on Ubuntu using the recommended setup I encountered a step that required installing nginx.

I like nginx, I've used it a bunch in the past but recently whilst experimenting with Kubernetes I've moved my reverse proxying needs to træfik. So I didn't want to install it if it wasn't needed.

I continued the setup without it but quickly discovered that ghost install is not a fan of that and will not start without it. So I gave in, installed it, and continued with the ghost installation; setting the website url to be https://blog.wastedcake.com and ignoring everything about SSL because I've set that up with træfik.

Setup completed, with træfik pointing to my server on port 80, I tried to access my new blog only to be met with a too many redirects error.

I didn't setup any redirects so that was a surprise. I checked what the issue was with curl -svILk https://blog.wastedcake.com and found that it was basically just stuck in a loop; redirecting to itself.


The Adventure

Disable nginx? nope.

Ok let's go back to my original plan of just not involving nginx. sudo service nginx stop.

I found the Ghost configuration file /var/www/ghost/config.production.json which showed me that Ghost was running on port 2368 and hosted on 127.0.0.1 so was not accessible externally (hence nginx as the reverse proxy) so I just changed  with ghost config server.host 0.0.0.0 and tried to access it locally with a simple curl curl localhost:2368 and got Moved Permanently. Redirecting to https://localhost/. Huh? how? where? Nginx isn't running (and it's config had no redirects anyway).

I tried again with 127.0.0.1:2368 and got redirected to 127.0.0.1. So something is just stripping out the port...

I found this test which makes sure if the blogs host is different to the requested url it just redirects the user to the correct url but that would mean I should be getting redirected to https://blog.wastedcake.com.

So I set the host headers to make sure nothing like that was triggered;

curl --verbose --header 'Host: blog.wastedcake.com' 'localhost:2368'
Moved Permanently. Redirecting to https://blog.wastedcake.com/

This looks better but it's still redirecting me around in circles.

Http instead of Https? nope.

I decided to just change the url that I configured the site with to just be http and see what happened;

ghost config url http://blog.wastedcake.com
ghost restart

And it worked!

I could access my site and it seemed fine. Until I noticed now all the links on my page were to absolute pages `http://blog.wastedcake.com/whatever` and not relative links. I don't really want that... http->https redirection will fix it but it's not nice. There has to be  a better way. So I changed the url back to https.

Reconfigure nginx? yes!

Here's the default Ghost nginx configuration found here /var/www/ghost/system/files/blog.wastedcake.com.conf.

server {
    listen 80;
    listen [::]:80;

    server_name blog.wastedcake.com;
    root /var/www/ghost/system/nginx-root; # Used for acme.sh SSL verification (https://acme.sh)

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host https;
        proxy_pass http://127.0.0.1:2368;

    }

    location ~ /.well-known {
        allow all;
    }

    client_max_body_size 50m;
}

Something has to be causing this redirect loop. Http works so it's related to https and we know Ghost has its own redirect rules. It's doing odd stuff there so it's not irrational to assume it's also doing a http->https redirect internally.

  1. Go to https://blog.wastedcake.com
  2. Traefik forwards request to http://x.x.x.x
  3. nginx picks that up and proxies the request to Ghost using the same protocol
    - proxy_set_header X-Forwarded-Proto $scheme;
  4. Ghost isn't happy with the non-https request
  5. Ghost then redirects the user to https://blog.wastedcake.com
  6. Bam, back at #2

Taking a look at Ghosts source code I found a test which ensures this behavior

it('blog is https, request is http', function (done) {
    urlRedirects.__set__('urlUtils', urlUtils.getInstance({
    url: 'https://default.com:2368/'
    }));

    host = 'default.com:2368';

    req.originalUrl = '/';
    redirect(req, res, next, getBlogRedirectUrl);
    next.called.should.be.false();
    res.redirect.called.should.be.true();
    res.redirect.calledWith(301, 'https://default.com:2368/').should.be.true();
    res.set.called.should.be.true();
    done();
 });

So we have to make sure the final proxy to Ghost is https or else it will just redirect us.


The Solution

The problem is ultimately that Ghost redirects http to https if you've setup your blog to work on https. I'm not a fan of this, it's better to leave this stuff up to the reverse proxy.

With nginx

So to fix this issue and keep nginx running it's as simple as saying the protocol used is always https by changing configuration found here; /var/www/ghost/system/files/blog.wastedcake.com.conf

By replacing

proxy_set_header X-Forwarded-Proto $scheme;

with

proxy_set_header X-Forwarded-Proto https;

You can then restarted ghost and nginx;

ghost restart
sudo service nginx restart

And everything should work as expected. Calls to http://x.x.x.x should now be treated by ghost as https and it wont try to redirect you.

Without nginx

Now we know what the problem is we can go back and fix the redirect loop we got back when we turned off nginx.

Summary of previous steps;

  1. Stop/remove nginx
  2. Set Ghost to expose itself externally ghost config server.host 0.0.0.0 and restart it ghost restart
  3. In your chosen reverse proxy set it send traffic to http://x.x.x.x:2368 and set the header X-Forwarded-Proto https

Example in træfik;

[http.services]
   [http.services.blogwastedcake.loadBalancer]
      [[http.services.blogwastedcake.loadBalancer.servers]]
         url = "http://x.x.x.x:2368"
[http.routers]      
   [http.routers.blogwastedcake]
      entryPoints = ["websecure"]
      rule = "Host(`blog.wastedcake.com`)"
      service = "blogwastedcake"
      middlewares = ["default-https-protocol-header@kubernetescrd"]
      [http.routers.blogwastedcake.tls]
         certResolver = "default"
Traefik Dynamic Config
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: https-protocol-header
spec:
  headers:
    customRequestHeaders:
      x-forwarded-proto: "https"
Kubernetes Træfik middleware