[go: up one dir, main page]

Sidekiq and unicorn pods fails when deploying gitlab helm chart 1.6.1

Summary

Sidekiq and unicorn pods fail to start because there's no psql-password. I guess the reason for this behavior lies in the configure script executed by the init-containers. The init-containers have all needed secrets mounted as a projected volume. This leads to the fact, that inside the volume the secret-file is a symlink:

$ ls -la /var/lib/kubelet/pods/c2072074-44c1-11e9-9c1e-0050568fe3c2/volumes/kubernetes.io~projected/init-sidekiq-secrets/postgres/
total 0
drwxr-sr-x 2 root sysadmin  60 Mar 12 13:24 .
drwxrwsrwt 9 root sysadmin 200 Mar 12 13:24 ..
lrwxrwxrwx 1 root root      32 Mar 12 13:24 psql-password -> ../..data/postgres/psql-password

$ cat /var/lib/kubelet/pods/c2072074-44c1-11e9-9c1e-0050568fe3c2/volumes/kubernetes.io~projected/init-sidekiq-secrets/postgres/psql-password 
REDACTED

But since the configure script uses cp -r that symlink is copied as-is (as a symlink) instead of copying its content:

$ ls -la /var/lib/kubelet/pods/c2072074-44c1-11e9-9c1e-0050568fe3c2/volumes/kubernetes.io~empty-dir/sidekiq-secrets/postgres/
total 0
drwxr-sr-x 2 sysadmin sysadmin  60 Mar 12 13:24 .
drwxrwsrwt 8 root     sysadmin 160 Mar 12 13:24 ..
lrwxrwxrwx 1 sysadmin sysadmin  32 Mar 12 13:24 psql-password -> ../..data/postgres/psql-password

$ cat /var/lib/kubelet/pods/c2072074-44c1-11e9-9c1e-0050568fe3c2/volumes/kubernetes.io~empty-dir/sidekiq-secrets/postgres/psql-password
cat: /var/lib/kubelet/pods/c2072074-44c1-11e9-9c1e-0050568fe3c2/volumes/kubernetes.io~empty-dir/sidekiq-secrets/postgres/psql-password: No such file or directory

Steps to reproduce

Install gitlab chart 1.6.1

Configuration used

certmanager:
  install: false
certmanager-issuer:
  email: REDACTED
gitlab:
  migrations:
    image:
      repository: registry.gitlab.com/gitlab-org/build/cng/gitlab-rails-ce
  sidekiq:
    image:
      repository: registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce
  task-runner:
    image:
      repository: registry.gitlab.com/gitlab-org/build/cng/gitlab-task-runner-ce
  unicorn:
    image:
      repository: registry.gitlab.com/gitlab-org/build/cng/gitlab-unicorn-ce
    workhorse:
      image: registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ce
global:
  hosts:
    domain: REDACTED
    externalIP: REDACTED
    https: false
  ingress:
    class: ""
    configureCertmanager: false
    tls:
      secretName: REDACTED
nginx-ingress:
  enabled: false
registry:
  enabled: false

Current behavior

Pods fail to start

Expected behavior

Pods don't fail to start

Versions

  • Chart: 1.6.1
  • Platform:
    • Self-hosted: bare metal
  • Kubernetes: (kubectl version)
    • Client: v1.10.0
    • Server: v1.8.4
  • Helm: (helm version)
    • Client: v2.13.0
    • Server: v2.13.0

Relevant logs

from both containers (sidekiq and unicorn)

+ /scripts/set-config /var/opt/gitlab/templates /srv/gitlab/config
(erb):7:in `read': No such file or directory @ rb_sysopen - /etc/gitlab/postgres/psql-password (Errno::ENOENT)
	from (erb):7:in `<main>'
	from /usr/lib/ruby/2.5.0/erb.rb:876:in `eval'
	from /usr/lib/ruby/2.5.0/erb.rb:876:in `result'
	from /scripts/set-config:22:in `block in <main>'
	from /scripts/set-config:18:in `each'
	from /scripts/set-config:18:in `<main>'
Begin parsing .erb files from /var/opt/gitlab/templates
Writing /srv/gitlab/config/database.yml