"Cannot handle hot-plugged CPUs" running rake backup or rake check
Summary
GitLab omnibus 11.2.3, same errors existed with 10.8.4.
Creating a backup with /opt/gitlab/bin/gitlab-rake gitlab:backup:create shows tens of thousands of lines like:
E0903 15:11:24.291009647 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs Even when trying to redirect the output to a file, for example /opt/gitlab/bin/gitlab-rake gitlab:backup:create | tee /tmp/output 2>&1 the "E0903 ..." is still displayed on console.
Furthermore, while creating this bug report and running /opt/gitlab/bin/gitlab-rake gitlab:check SANITIZE=true, same errors are displayed but in lesser counts.
Steps to reproduce
In /etc/gitlab/gitlab.rb the backup directory is set with
gitlab_rails['backup_path'] = "/mnt/backup/gitlab"
Then the backup is run manually with
/opt/gitlab/bin/gitlab-rake gitlab:backup:create
Run /opt/gitlab/bin/gitlab-rake gitlab:check SANITIZE=true
Example Project
There are several dozen repositories in the GitLab.
What is the current bug behavior?
Thousands of lines with "Cannot handle hot-plugged CPUs" are shown.
What is the expected correct behavior?
This is a VMware VM, just rebooted (does not make a difference in behavior), and CPUs are not hot-plugged. There should be no errors about "hot-plugged CPUs".
Relevant logs and/or screenshots
Excerpt for /opt/gitlab/bin/gitlab-rake gitlab:backup:create taken from middle of the run:
[...]
E0903 15:25:04.640568028 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.640715151 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.640814838 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.640910938 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.641006296 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.641104662 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.641201256 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.641296248 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.641392130 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.643862944 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.644004914 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.644116665 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.644215785 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.644332632 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.644466517 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.644577852 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.644713039 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.644836754 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.644954144 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.645064571 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:25:04.645182440 2883 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
* docker/apache2.4 ... [DONE][...]
Excerpts for /opt/gitlab/bin/gitlab-rake gitlab:check SANITIZE=true:
[...]
ok
5/150 ... E0903 15:20:12.576216701 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:12.577349147 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:12.578018243 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:12.578477645 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:12.579727601 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:12.580925348 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:12.581950247 5712 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:12.582144875 5712 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:12.582280415 5712 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:12.583034797 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:12.603359013 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:12.624492667 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:12.645669092 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:12.666835920 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:12.688001323 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
repository is empty
102/152 ... E0903 15:20:12.993270007 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
repository is empty
73/154 ... E0903 15:20:13.241183076 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:13.242560546 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:13.244775042 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:13.248482538 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:13.249786481 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:13.251212994 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:13.252352373 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:13.275172195 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:13.348247248 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:13.375281167 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:13.587001393 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:20:13.608204915 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
ok
39/155 ... ok
102/156 ... ok[...]
Git user has default SSH configuration? ... yes
Active users: ... 63
Checking GitLab ... Finished
E0903 15:21:08.513939619 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:21:08.517885985 5712 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:21:08.523139274 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:21:08.528043101 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:21:08.529285534 5712 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:21:08.529320630 5712 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:21:08.529343858 5712 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:21:08.529425999 5712 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:21:08.751798638 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
E0903 15:21:08.758256510 5514 cpu_linux.cc:75] Cannot handle hot-plugged CPUs
#Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.4.4p296 Gem Version: 2.7.6 Bundler Version:1.16.2 Rake Version: 12.3.1 Redis Version: 3.2.11 Git Version: 2.18.0 Sidekiq Version:5.1.3 Go Version: unknownGitLab information Version: 11.2.3 Revision: 06cbee3 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://git.example.com HTTP Clone URL: https://git.example.com/some-group/some-project.git SSH Clone URL: git@git.example.com:some-group/some-project.git Using LDAP: yes Using Omniauth: no
GitLab Shell Version: 8.1.1 Repository storage paths: default: /mnt/git/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...GitLab Shell version >= 8.1.1 ? ... OK (8.1.1) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 2/1 ... ok 3/2 ... ok 5/3 ... ok 5/4 ... ok 4/5 ... ok 48/6 ... ok 2/13 ... ok 15/15 ... ok 35/16 ... ok 35/17 ... ok 33/18 ... ok 33/19 ... ok 33/20 ... ok 33/21 ... ok 4/22 ... ok 34/23 ... ok 34/24 ... ok 34/25 ... ok 34/26 ... ok 35/27 ... ok 15/28 ... ok 15/29 ... ok 15/30 ... ok 15/31 ... ok 34/32 ... ok 6/35 ... ok 4/36 ... ok 14/37 ... ok 14/38 ... ok 14/39 ... ok 14/40 ... ok 14/41 ... ok 14/42 ... ok 39/43 ... ok 39/44 ... repository is empty 34/45 ... ok 40/46 ... repository is empty 6/47 ... ok 40/48 ... ok 45/49 ... ok 45/50 ... ok 45/51 ... ok 45/52 ... ok 45/53 ... ok 45/54 ... ok 33/55 ... ok 33/56 ... ok 48/57 ... ok 2/58 ... ok 18/59 ... ok 18/60 ... ok 18/61 ... ok 18/62 ... ok 2/63 ... ok 6/64 ... ok 2/65 ... ok 46/66 ... repository is empty 54/67 ... ok 3/68 ... ok 17/69 ... ok 40/70 ... ok 15/71 ... ok 3/72 ... ok 3/73 ... ok 3/74 ... ok 3/75 ... ok 17/76 ... ok 17/77 ... ok 17/78 ... ok 17/79 ... ok 17/80 ... ok 3/81 ... ok 3/82 ... ok 3/83 ... ok 3/84 ... ok 3/85 ... ok 39/86 ... ok 3/87 ... ok 14/88 ... ok 14/89 ... ok 14/90 ... ok 3/91 ... ok 17/92 ... ok 39/93 ... ok 39/94 ... ok 17/95 ... ok 9/96 ... ok 3/97 ... ok 14/98 ... ok 14/100 ... ok 65/101 ... ok 44/103 ... repository is empty 27/106 ... ok 27/107 ... ok 3/108 ... ok 6/109 ... ok 72/110 ... ok 72/111 ... ok 3/112 ... ok 3/114 ... ok 3/115 ... ok 73/116 ... ok 2/118 ... ok 4/119 ... ok 15/120 ... ok 39/121 ... ok 5/122 ... ok 15/123 ... ok 3/124 ... repository is empty 14/125 ... ok 14/126 ... ok 3/127 ... ok 27/128 ... ok 90/129 ... repository is empty 88/130 ... repository is empty 93/132 ... ok 5/133 ... ok 93/134 ... ok 5/135 ... ok 5/136 ... ok 5/137 ... ok 92/138 ... ok 5/139 ... ok 5/140 ... ok 15/141 ... repository is empty 20/142 ... ok 93/143 ... ok 5/144 ... ok 20/146 ... ok 3/147 ... ok 94/148 ... repository is empty 5/149 ... ok 5/150 ... repository is empty 102/152 ... repository is empty 73/154 ... ok 39/155 ... ok 102/156 ... ok 102/157 ... ok 102/158 ... ok 18/159 ... ok 93/160 ... ok 103/161 ... ok 5/162 ... ok 102/163 ... ok 5/164 ... ok 107/165 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /srv/git-data/home/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) DN: cn=example,ou=example,dc=example,dc=com userPrincipalName: example@example.com [...] Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 2/1 ... yes 3/2 ... yes 5/3 ... yes 5/4 ... yes 4/5 ... yes 48/6 ... yes 2/13 ... yes 15/15 ... yes 35/16 ... yes 35/17 ... yes 33/18 ... yes 33/19 ... yes 33/20 ... yes 33/21 ... yes 4/22 ... yes 34/23 ... yes 34/24 ... yes 34/25 ... yes 34/26 ... yes 35/27 ... yes 15/28 ... yes 15/29 ... yes 15/30 ... yes 15/31 ... yes 34/32 ... yes 6/35 ... yes 4/36 ... yes 14/37 ... yes 14/38 ... yes 14/39 ... yes 14/40 ... yes 14/41 ... yes 14/42 ... yes 39/43 ... yes 39/44 ... yes 34/45 ... yes 40/46 ... yes 6/47 ... yes 40/48 ... yes 45/49 ... yes 45/50 ... yes 45/51 ... yes 45/52 ... yes 45/53 ... yes 45/54 ... yes 33/55 ... yes 33/56 ... yes 48/57 ... yes 2/58 ... yes 18/59 ... yes 18/60 ... yes 18/61 ... yes 18/62 ... yes 2/63 ... yes 6/64 ... yes 2/65 ... yes 46/66 ... yes 54/67 ... yes 3/68 ... yes 17/69 ... yes 40/70 ... yes 15/71 ... yes 3/72 ... yes 3/73 ... yes 3/74 ... yes 3/75 ... yes 17/76 ... yes 17/77 ... yes 17/78 ... yes 17/79 ... yes 17/80 ... yes 3/81 ... yes 3/82 ... yes 3/83 ... yes 3/84 ... yes 3/85 ... yes 39/86 ... yes 3/87 ... yes 14/88 ... yes 14/89 ... yes 14/90 ... yes 3/91 ... yes 17/92 ... yes 39/93 ... yes 39/94 ... yes 17/95 ... yes 9/96 ... yes 3/97 ... yes 14/98 ... yes 14/100 ... yes 65/101 ... yes 44/103 ... yes 27/106 ... yes 27/107 ... yes 3/108 ... yes 6/109 ... yes 72/110 ... yes 72/111 ... yes 3/112 ... yes 3/114 ... yes 3/115 ... yes 73/116 ... yes 2/118 ... yes 4/119 ... yes 15/120 ... yes 39/121 ... yes 5/122 ... yes 15/123 ... yes 3/124 ... yes 14/125 ... yes 14/126 ... yes 3/127 ... yes 27/128 ... yes 90/129 ... yes 88/130 ... yes 93/132 ... yes 5/133 ... yes 93/134 ... yes 5/135 ... yes 5/136 ... yes 5/137 ... yes 92/138 ... yes 5/139 ... yes 5/140 ... yes 15/141 ... yes 20/142 ... yes 93/143 ... yes 5/144 ... yes 20/146 ... yes 3/147 ... yes 94/148 ... yes 5/149 ... yes 5/150 ... yes 102/152 ... yes 73/154 ... yes 39/155 ... yes 102/156 ... yes 102/157 ... yes 102/158 ... yes 18/159 ... yes 93/160 ... yes 103/161 ... yes 5/162 ... yes 102/163 ... yes 5/164 ... yes 107/165 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.4.4) Git version >= 2.9.5 ? ... yes (2.18.0) Git user has default SSH configuration? ... yes Active users: ... 63
Checking GitLab ... Finished
Possible fixes
No ideas, seems to be problem with some third-party component?