Serverless Reverse Proxies for Service to Service calls with Cloud Run
Because sometimes, you need a bit more control with your load balancing than what you get from the managed service.
Introduction
One of the nice things about about Cloud Run is that, out of the box, you get a public routable URL which is load balanced and secured with SSL, however that is also a not-so-nice thing if you need that service to be protected. Cloud Run gives us an interesting feature which disables public invocations, and only allows authenticated calls to be able to invoke the service. How can we do this though if we want to use NGINX as our proxy? Let’s find out.
Overview
The use case I’ll be referring to in this process will be the situation where you have a web app on one route which is where authenticated users want to go, and maybe another route with some public information, like a landing page and then other rules based on whatever conditions you might need in your config.
If you want all your traffic to go through the NGINX “front door”, then you’ll need to configure Load Balancing, as well as your NEGs (network endpoint groups) and so on, so that the Front Door is reachable from the public. I’ll leave that exercise to the reader as these are well-documented and solved problems.
Consider the below diagram. We’ve got our traffic from the Internet coming through, then we route to the protected service as required. That protected service then reaches into the VPC for certain things like databases, other compute services or in-memory data stores.
Project Setup
Let’s setup our project so that we can get this working! For this project we probably won’t bother with a Global Load Balancer and NEGs since it’s not necessary, but I would recommend that if you’re going to be running prod traffic for a variety of reasons.
First thing's first, create some custom roles for the jobs that you need the protected service to do, ie. cloudSQL.client and so on. For the front-door service, create a service account that has the Cloud Run Invoker role. The invoker role is needed because the front door service will be “invoking” the service on behalf of the user.
With the service accounts created, we’ll then need to have a protected cloud run service that has public invocations turned off, i.e. you need to be both authenticated and authorised to invoke the protected service.
Note!
Whilst the service will be protected it will still have a public URL which is addressable by the public internet. However, if a request comes to it without authorisation, then nothing happens, and the requestor will simply get a http 401 or 403 depending on if they provide a token.
So, create the service and then in the security tab, select “Require Authentication”. It’s worth pointing out, that this requirement cannot apply to users of your application, it is specifically designed for Cloud IAM users only. Once the service is invoked, you can then authenticate your application users if you need to.
We then need to setup the public facing service, the front door. That will have the NGINX service running in it.
Configuring NGINX
Once your Cloud Run services are deployed, it’s now time to configure NGINX. I’d like to point you to a repo which has all the code you need for this bit: https://github.com/jgunnink/nginx-cloudrun-reverse-proxy
If you don’t want to visit the link or just want to read here, that’s fine too. I’ll put the relevant bits of code as we go along.
Dockerfile and Script
Here’s the Dockerfile for the NGINX service:
FROM nginx
COPY run.sh /run.sh
# Copy Nginx configuration to a template file
COPY nginx.conf /etc/nginx/nginx.conf.template
CMD ["./run.sh"]For the purposes of the demo, we just use the NGINX default tag and copy over the template file and script. The entrypoint of the container is just to execute the script.
The job of the script is to go to the internal metadata server and using the IAM service get a token which it can use in a subsequent call, it then replaces the keyword inside the NGINX config with the value of the token, and sleeps for (almost) an hour. The expiry time of these tokens is an hour so if the service is always running or has been running for long enough, it will refresh the token.
One of the key things to look for in this process is the “audience” claim in the request to the server. Specifically, at the end of the URL in the curl call.
service-accounts/default/identity?audience=[protected_service_url]
That protected service URL is the URL of the cloud run service you deployed. It means that when you pass the token through it can authorise against the right service.
NGINX Conf
The NGINX conf is pretty straightforward and simple for this example, but let’s quickly discuss.
The important part of this process is the injection of the header, the X-Serverless-Authorization header. This is a bearer token, which is setup by the script in the previous step. We also set a proxy_pass of all traffic (denoted by the “/” in the location block) to the destination cloud run service.
Once you’ve got this setup, then all you need to do is deploy your NGINX service again and you should be good to go.
You can deploy it with this command:
gcloud run deploy nginx-frontend \
--platform managed \
--region us-central1 \
--allow-unauthenticated \
--source .Done
With that deployment, you’re now ready to do service to service calls with your cloud run services & use NGINX as a front door to your protected environment. Here’s some links in case you want to do further reading.