Configuring SSL for Hasura GraphQL on DigitalOcean Kubernetes

I recently had some trouble configuring SSL on a Hasura GraphQL instance deployed on DigitalOcean Kubernetes. After sifting through official documentation from the Hasura team and DigitalOcean and getting nowhere, I put together this guide after I got things to work as expected. Although I am specifically using Hasura as my service of choice, this guide can be used to secure any internal K8s service meant to be public-facing via SSL.

Special thanks to Hanif Jetha for writing the original guide on DO! 🙏

Prerequisites

Make sure you have configured kubectl and doctl. Sign in to DigitalOcean using the command doctl auth init.

  • If you have a pre-existing cluster you want to use, you can set up that cluster for use by running doctl kubernetes cluster kubeconfig save <your_cluster_name>
  • If not, create a new K8s cluster by running doctl kubernetes cluster create <your_cluster_name>. Once it is done provisioning, doctl should automatically set up your kubectl config to use the new cluster
  • Verify your kubectl setup works by running kubectl cluster-info
  • Have a PostgreSQL instance ready with a connection string noted down

Setting up Hasura

Official Hasura K8s deployment files I used as a starting point are available here:

https://github.com/hasura/graphql-engine/tree/master/install-manifests/kubernetes

We will first configure secrets for Hasura. You can add as many as you like here, and reference them as needed but for now lets include the admin access key and database URL. Configure these secrets in a YAML file.

apiVersion: v1
kind: Secret
metadata:
  name: hasura
stringData:
  accessKey: <your_admin_access_key>
  dburl: <your_pg_database_url>

Take care not to commit this file to version control! This file will be deployed on your cluster first. Use the command kubectl apply -f secrets.yaml and verify using kubectl get secret hasura.

Now we will configure Hasura deployment and service configurations.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: hasura
    hasuraService: custom
  name: hasura
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hasura
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: hasura
    spec:
      containers:
        - image: hasura/graphql-engine:v1.0.0
          imagePullPolicy: IfNotPresent
          name: hasura
          env:
            - name: HASURA_GRAPHQL_DATABASE_URL
              valueFrom:
                secretKeyRef:
                  key: dburl
                  name: hasura
            - name: HASURA_GRAPHQL_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  key: accessKey
                  name: hasura
            # in case you plan to use JWT auth
            #  - name: HASURA_GRAPHQL_JWT_SECRET
            #    value: 'your_jwt_secret'
            - name: HASURA_GRAPHQL_UNAUTHORIZED_ROLE
              value: anonymous
            - name: HASURA_GRAPHQL_ENABLE_CONSOLE
              value: 'true'
          ports:
            - containerPort: 8080
              protocol: TCP

kubectl apply -f hasura_deployment.yaml

apiVersion: v1
kind: Service
metadata:
  labels:
    app: hasura
  name: hasura
  namespace: default
spec:
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  selector:
    app: hasura

kubectl apply -f hasura_service.yaml

This will set up a Hasura instance which will connect to a Postgres instance via the connection string we set as secret. You will not be able to access it publicly because the service is not set up with public access. We will fix that once we set up our ingress configuration.

Verify that the service was created by running kubectl get svc. You should see something similar to this:

Setting Up the Kubernetes Nginx Ingress Controller

We will use nginx-ingress controller to set up SSL. Further documentation can be found here.

⚠️ Make sure to save the cloud_generic.yaml file using wget as we will need it later. It can also be found here.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/mandatory.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/cloud-generic.yaml

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/cloud-generic.yaml

After running the commands, verify that the controller pod that was created is up and running by running kubectl get pods --all-namespaces -l app.kubernetes.io/name=ingress-nginx.

Next, check the load balancer service by running kubectl get svc --namespace=ingress-nginx.

When you get an IP address assigned in the EXTERNAL-IP field, note it down. This address will be used to update DNS records.

From the official documentation:

This load balancer receives traffic on HTTP and HTTPS ports 80 and 443, and forwards it to the Ingress Controller Pod. The Ingress Controller will then route the traffic to the appropriate backend Service.

Setting up an ingress rule for our Hasura service

Next up, we need to create a rule which forwards incoming traffic to a certain domain to the designated service. For now, let’s forget about SSL. An example of such a rule is:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: echo-ingress
spec:
  rules:
    - host: subdomain.yourdomain.com
      http:
        paths:
          - backend:
              serviceName: hasura
              servicePort: 80

This config will create an Ingress Resource called echo-ingress, and route traffic based on the Host header.

An HTTP request Host header specifies the domain name of the target server.

Requests with host subdomain.yourdomain.com will be directed to the hasura backend we set up in the first step. Create the rule by running kubectl apply -f hasura_ingress_without_ssl.yaml.

Now we need to update DNS records to point to the load balancer’s IP address that ingress-nginx created. Navigate to your DNS management service control panel and create A records for subdomain.yourdomain.com pointing to the external IP we noted down.

If you are using DigitalOcean to manage your domain’s DNS records, consult How to Manage DNS Records to learn how to create A records.

Once the necessary DNS records are created and have propagated, you can test what we have done so far by going to subdomain.yourdomain.com in a web browser. It should load the Hasura console.

Troubleshooting

  • Make sure the IP you get after running wget --save-headers -O- subdomain.yourdomain.com is the same IP you noted down before. If not, DNS changes might not have propagated or you configured the wrong IP address as A record in your DNS provider control panel.
  • If you get HTTP 500 errors, run kubectl get pods to get a list of pods running on your cluster. Copy the name of the pod.
  • Check the logs generated by hasura by running the command kubectl logs <hasura-pod-name-you-copied>. If the pod is not running and shows some other status, check what went wrong by running kubectl describe pod <hasura-pod-name-you-copied>. This name may change between deployments so make sure you get the current one.

Installing and Configuring Cert-Manager

I installed cert-manager by following instructions here.

After cert-manager is done installing, verify the install works by following instructions here.

Before we begin issuing certificates for our Ingress hosts, we need to create an Issuer, which specifies the certificate authority from which signed x509 certificates can be obtained. In this guide, we’ll use the Let’s Encrypt certificate authority, which provides free TLS certificates and offers both a staging server for testing your certificate configuration, and a production server for rolling out verifiable TLS certificates.

Let’s create and deploy a staging certificate issuer to test if the functionality works.

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
  namespace: cert-manager
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: <your_email_address_here>
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging
    # Enable the HTTP-01 challenge provider
    solvers:
      - http01:
          ingress:
            class: nginx

Run kubectl create -f staging_issuer.yaml to create the issuer. Now we are ready to enable TLS encryption for subdomain.yourdomain.com. Edit hasura_ingress_without_ssl.yaml and make the following additions:

Save this new file as hasura_ingress_with_staging_ssl.yaml. Then, update the existing resource by running kubectl apply -f hasura_ingress_with_staging_ssl.yaml.

You can then use the commands kubectl describe ingress and kubectl describe certificate to track certificate creation.

Troubleshooting

  • If you do not see the certificate request being fulfilled and are stuck at ‘Waiting for CertificateRequest’, it may be due to this issue. In short, we will need to edit the default ingress-nginx cloud_generic.yaml file we saved when we installed ingress-nginx and add one annotation to it as described here. This will result in a file which looks like this:
kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-hostname: 'domain.com'
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  externalTrafficPolicy: Local
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: https
      port: 443
      targetPort: https

After editing, deploy it by running kubectl apply -f modified_cloud_generic.yaml. After this one change, the certificate should be issued successfully.

Now, running wget --save-headers -O- subdomain.yourdomain.com should result in this:

This means the staging server is running properly and HTTPS has successfully been enabled, but the certificate cannot be verified as it’s a fake temporary certificate issued by the Let’s Encrypt staging server.

Now we simply need to create a production certificate issuer and deploy it.

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: <your_email_address_here>
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the HTTP-01 challenge provider
    solvers:
      - http01:
          ingress:
            class: nginx

kubectl create -f prod_issuer.yaml

After that, change the issuer annotation in the ingress config from staging to prod, save the file as hasura_ingress_with_prod_ssl.yaml, deploy the change and everything should work!

kubectl apply -f hasura_ingress_with_prod_ssl.yaml

Wait a couple of minutes for the Let’s Encrypt production server to issue the certificate. You can track its progress by running kubectl describe on the certificate object.

We’ll now perform a test using curl to verify that HTTPS is working correctly:

curl subdomain.yourdomain.com

This command should show a permanent 308 redirect for all HTTP requests to use HTTPS. You can now visit subdomain.yourdomain.com in your browser and verify that the connection is secure. ✔️

I hope you found this guide helpful, and feel free to contact me if you have any queries. Thank you for reading, and thanks to the amazing people over at Hasura and DigitalOcean for creating such awesome tools for us developers to play with. 🚀