Gitpod Cleanup Workspaces - Selfhosted KOTS Install

Check this of first - Everything you copy and paste without understanding what it does is your responsibility:

Running Gitpod with the KOTS install
Full access to the K8S Server
WORKING BACKUPS?!

Start by connecting to the System and becoming root:

ssh [email protected]

Identify the Minio storage

kubectl get persistentvolumes --all-namespaces

Example:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                    STORAGECLASS   REASON   AGE
pvc-02cb47c4-b3e9-44e8-bbd3-547a2ceb47e9   8Gi        RWO            Delete           Bound    gitpod/data-mysql-0                      local-path              214d
pvc-21485831-7c7c-423d-a00b-22bdd27b8197   1Gi        RWO            Delete           Bound    gitpod/kotsadm-rqlite-kotsadm-rqlite-0   local-path              214d
pvc-23ec99f0-d40b-4c0f-9148-caa41f474cdc   4Gi        RWO            Delete           Bound    gitpod/kotsadm-minio-kotsadm-minio-0     local-path              214d
pvc-4796259e-18ba-49bd-a38e-b35c5479dec7   8Gi        RWO            Delete           Bound    gitpod/redis-data-openvsx-proxy-0        local-path              214d
pvc-65254c19-4c19-425c-a43c-9bf79bdc0982   10Gi       RWO            Delete           Bound    gitpod/registry                          local-path              214d
pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e   8Gi        RWO            Delete           Bound    gitpod/minio                             local-path              214d

Target => pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e

How much storage is beeing used

du -h -d 1 /var/lib/rancher/k3s/storage/pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e_gitpod_minio/

Example:

108K    /var/lib/rancher/k3s/storage/pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e_gitpod_minio/gitpod-user-REDACTED
280K    /var/lib/rancher/k3s/storage/pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e_gitpod_minio/gitpod-user-REDACTED
12G     /var/lib/rancher/k3s/storage/pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e_gitpod_minio/gitpod-user-REDACTED
260K    /var/lib/rancher/k3s/storage/pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e_gitpod_minio/gitpod-user-REDACTED
4.5G    /var/lib/rancher/k3s/storage/pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e_gitpod_minio/gitpod-user-REDACTED
196K    /var/lib/rancher/k3s/storage/pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e_gitpod_minio/gitpod-user-REDACTED
7.8G    /var/lib/rancher/k3s/storage/pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e_gitpod_minio/gitpod-user-REDACTED
4.0G    /var/lib/rancher/k3s/storage/pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e_gitpod_minio/gitpod-user-REDACTED
4.4M    /var/lib/rancher/k3s/storage/pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e_gitpod_minio/.minio.sys
136K    /var/lib/rancher/k3s/storage/pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e_gitpod_minio/gitpod-user-REDACTED
1.7G    /var/lib/rancher/k3s/storage/pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e_gitpod_minio/gitpod-user-4b18d07e-deea-4141-848e-81743db80578
30G     /var/lib/rancher/k3s/storage/pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e_gitpod_minio/

Get the UserId

Now we want to clean all workspaces for user Elias Hackradt. But who is who?

We could look into the Database, but for now just use the Gitpod Admin UI like this:

In the URL you can now find the UserId:

Example: https://your.awesome.host/admin/users/4b18d07e-deea-4141-848e-81743db80578

Now we can scan the users data usage:

du -h -d 1 /var/lib/rancher/k3s/storage/pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e_gitpod_minio/gitpod-user-4b18d07e-deea-4141-848e-81743db80578

Example:

1.7G    /var/lib/rancher/k3s/storage/pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e_gitpod_minio/gitpod-user-4b18d07e-deea-4141-848e-81743db80578/workspaces
412K    /var/lib/rancher/k3s/storage/pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e_gitpod_minio/gitpod-user-4b18d07e-deea-4141-848e-81743db80578/blobs
1.7G    /var/lib/rancher/k3s/storage/pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e_gitpod_minio/gitpod-user-4b18d07e-deea-4141-848e-81743db80578

☢ Nuke the data ☢

Warning

This will delete data!

Info

This step might not be necessary - I should test this 🤔

Lets nuke the workspaces of Elias Hackradt aka. 4b18d07e-deea-4141-848e-81743db80578

rm -rf /var/lib/rancher/k3s/storage/pvc-8d499b12-5810-44dd-8159-f3d1b9c12c3e_gitpod_minio/gitpod-user-4b18d07e-deea-4141-848e-81743db80578/workspaces/*

Cleanup the Database

Ok that is done. Now the User still has his old, but deleted, workspaces in the Web UI. We need to fix that!

We will need the Database root access.

If you installed gitpod in the default gitpod namespace:

Get the mysql root password:

kubectl -n gitpod get secrets db-password -o json | jq -r '.data["mysql-root-password"]' | base64 -d

Attach to the database pod:

kubectl -n gitpod exec --stdin --tty mysql-0 -- /bin/bash

mysql -uroot -p

Warning

Editing the database can be destructive - maybe use TRANSACTION - But you have a backup anyway, right? 😏

USE gitpod;
UPDATE d_b_workspace SET deleted = 1 WHERE ownerId='4b18d07e-deea-4141-848e-81743db80578';

Now after a few seconds to a minute all workspaces from the user will also be deleted from the web view.

Thanks for reading! - I do not take responsibility for any spelling mistakes. It was my keyboard not me! 😵

If this helped you, maybe buy me Beer? 🍻

https://www.paypal.me/brutalbirdie