refactor: Replace minio/mc with garage/rclone

[NickLarsenNZ]

• Replace minio with garage
  • I use region-1 as the S3 region to keep things vendor agnostic.
• replace mc commands with rclone
  • I chose rclone over aws-cli to make it vendor agnostic. There were other options but they were less maintained.
• Update docs
• Update storage requirements for the big demo/stack.
• todo: Only request public parts for S3 clients

Testing

I ran parts of the demos to verify S3 read/write paths (except the big demo).

• airflow-scheduled-job
• data-lakehouse-iceberg-trino-spark
• jupyterhub-keycloak
• nifi-kafka-druid-earthquake-data (hard to test since Druid writes to S3 after 1 hour)
• nifi-kafka-druid-water-level-data (hard to test since Druid writes to S3 after 1 hour)
• spark-k8s-anomaly-detection-taxi-data
• trino-iceberg
• trino-taxi-data
• monitoring (stack)

[NickLarsenNZ]

We could look at putting the Garage helm chart somewhere so we can helm install it.
https://garagehq.deuxfleurs.fr/documentation/cookbook/kubernetes/

[sbernauer]

May thanks for this PR, it surely was a big effort.
I also like all of the small cleanups. The only "real" problems are related to missing icons in images

[sbernauer]

Images now look good, thanks!

[NickLarsenNZ]

I merged too soon. @adwk67 has some suggestions

[adwk67]

Some comments about running garage (and thereby nginx) on Openshift (will be relevant for tests too):

• Nginx sets up temporary folders which all users need to be able to write to: on Openshift these will be non-root and the default location will not work. Fix: define these folder as being under /tmp:

@@ -43,8 +43,17 @@ data:
   # Garage terminates no TLS, so nginx does it and forwards to Garage on
   # loopback. Mounted over /etc/nginx/nginx.conf, hence the full file.
   nginx.conf: |
+    # The image defaults for these are under /run and /var/cache/nginx, which
+    # only root can write. OpenShift runs the container as an arbitrary UID.
+    pid /tmp/nginx.pid;
     events {}
     http {
+      client_body_temp_path /tmp/client_temp;
+      proxy_temp_path /tmp/proxy_temp;
+      fastcgi_temp_path /tmp/fastcgi_temp;
+      uwsgi_temp_path /tmp/uwsgi_temp;
+      scgi_temp_path /tmp/scgi_temp;
+
       server {
         listen 9000 ssl;
         ssl_certificate /stackable/tls/tls.crt;

• N.B. for nginx in the Airflow ca-cert test there is an alternative approach, using an nginxinc/nginx-unprivileged and extra volume mounts

      containers:
        - name: nginx
          image: nginxinc/nginx-unprivileged:alpine
          ports:
            - containerPort: 8443
          volumeMounts:
            - name: config
              mountPath: /etc/nginx/nginx.conf
              subPath: nginx.conf
            - name: tls
              mountPath: /etc/nginx/tls
              readOnly: true
            - name: cache
              mountPath: /var/cache/nginx
            - name: run
              mountPath: /var/run
      volumes:
        - name: config
          configMap:
            name: git-proxy-config
        - name: tls
          secret:
            secretName: git-proxy-tls
        - name: cache
          emptyDir: {}
        - name: run
          emptyDir: {}

• also, running on our Openshift/OKD clusters requires the demos to be started with the --listener-class-preset stable-nodes option, but that's because the replicated OKD doesn't seem to have a load-balancer controller (nothing to do with garage per se).