Need technical guidance how to make Alfresco ACS containerized deployment work on OpenShift cluster

[dzilberman]

Hello,

We are working on a prototype of containerized Alfresco ACS Enterprise suite deployment for our corporae customer on RedHat OpenShift cluster platform. 

Prototype cluster environment is deployed in AWS so we are basically following Helm deployment example: https://github.com/Alfresco/acs-deployment/blob/master/docs/helm/examples/with-aws-services.md except that deploying ActiveMQ cotainer rather than AWS ActiveMQ service. 

We have followed instruction carefully (except that K8s cluster is OpenShift vs AWS EKS) and noticed that in 2 different clusters out of  about 21 pods, 6 cocnsitently CrashLoopBackoff. Upon closer examination, it looks like most of those pods crashed on their "initContainer" stage where that helper containers run in limited security context and try to change ownership of mounted directory like:

acs-alfresco-filestore-…..

..

Pod YAML (initContainer section):

..

spec:

  restartPolicy: Always

  initContainers:

    - resources: {}

      terminationMessagePath: /dev/termination-log

      name: init-fs

      command:

        - sh

        - '-c'

        - 'chown -R 33030:1000 /tmp/Alfresco'

      securityContext:

        capabilities:

          drop:

            - KILL

            - MKNOD

            - SETGID

            - SETUID

        runAsUser: 1000850000

      imagePullPolicy: Always

      volumeMounts:

        - name: data

          mountPath: /tmp/Alfresco

          subPath: alfresco-content-services/filestore-data

        - name: default-token-zgvf6

          readOnly: true

          mountPath: /var/run/secrets/kubernetes.io/serviceaccount

——

Console output

chown: /tmp/Alfresco: Operation not permitted

chown: /tmp/Alfresco: Operation not permitted

———

Othr pods (acs-activemq-..., acs-alfresco-cs-repository-..., acs-alfresco-search-solr-..) have identical problems - lack of permissions to run 'chown' commands on a mounted directory or someth similar.

We are looking for quidance to solve this issue - either by forking repo https://github.com/Alfresco/acs-deployment.git and making necessary changes to pods' deployment YAMLs ourselves (once we know what parameters in securityContext to change) or possible getting help from Alfresco engineering team making those changes so that ASC containerized deployment would work on OpenShift platform.

[dzilberman]

An update: looks like we have pintpointed the issue with underlying mounted NFS file system (which is mounted to underlying AWS EFS file store (fs-c8605ccf.efs.us-west-2.amazonaws.com) is somehow read-only.

The Volume is mounted via PVC:

..

volumes:

- name: data

persistentVolumeClaim:

claimName: alfresco-volume-claim

..

which is in "ReadWriteMany" mode:

alfresco-volume-claim            Bound    pvc-63023764-d877-48ba-b90d-f629b2501c44   20Gi       RWX            nfs-client     

...

but when initContainer attempts to change ownership using command:

chown -R 33031:1000 /opt/activemq/data

it fails. and prevents main container from initialization.

Assuming that instructions on https://github.com/Alfresco/acs-deployment/blob/master/docs/helm/eks-deployment.md for provisioning of 

storageClass.name="nfs-client"

 were verified against AWS EFS instances, would like to understand where our problem may be.

thanks,

Daniel Zilberman

RedHat

[EddieMay]

Hi @dzilberman and welcome to Alfresco!

As an enterprise customer I would also suggest raising a support ticket.

Have a good weekend.

[dzilberman]

Hi @EddieMay 

Thank you for the response. As of this moment, Red Hat is not an enterprise customer of Alfresco AFAIK, but our common customer is (Sony Pictures Entertainment). We are just working on Proof of Concept deployment of containerized ACS on OpenShift platform following instructions on GitHub: https://github.com/Alfresco/acs-deployment/tree/master/docs/helm

Given the above, can I still go ahead and file a support ticket at some level or could you suggest other avenues to get our technical issues addressed, please? Getting this PoC to work is very important for us and the customer.

best regards,

Daniel Zilberman

RedHat

[EddieMay]

Hi @dzilberman 

OK, I'll ask internally but as it's late Friday evening here, it probably won't be dealt with until next week.

[dzilberman]

Hi @EddieMay and Alfresco support team,

After some research, I have arrived at conslcusion that permissions issues we are experiencing depoying Alfresco ACS container images to OpenShift K8s platform are related to specific permissions settings that OpenShift restricts the ability for a pod to choose their UserID (UID) and GroupID (GID), instead opting to provide the Pod with a pair allocated for it. Following workaround in our support article https://access.redhat.com/solutions/5220551, I was able to update ownership of mounted directories in containers and get them to start up, at least ost of them. 

Thanks for your attention. For future questions related to specific issues with containerized ACS deployments, what channel would be the most efficient to reach out to?

thanks,

Daniel Zilberman

Solutions Architect

RedHat 

[EddieMay · Accepted solution]

Hi @dzilberman 

Thanks for reporting back. I would be interested in following your progress in getting this to work. For container stuff our Discord channel might be a good place to talk things through.

Cheers,

[dzilberman]

Thanks @EddieMay .

We are keenly intrested to get cotainerized Alfresco ACS running on OpenShift platform. With a full confession that I am not a current Discord user, can you please point me to that specific Discord channel or perhaps an alternative like Slack etc.? I was unable to join the text channel following posted link to Discord https://discord.com/channels/451644531323174912/451644531323174914
I realize that https://github.com/Alfresco/acs-deployment is an open spource project and contributors likely use popular OSS comm channels...

Daniel Zilberman

Red Hat

[EddieMay]

Hi @dzilberman 

I've sent Discord link to you via email - not sure why that one didn't work? Hopefully this will work for you - I'll keep an eye out for you on Discord.

I'm afraid we don't have a public Slack channel. 

HTH,