[go: up one dir, main page]
Skip to content

Do not generate too long file names for uploads

Summary

File uploaded via UI got too long file name generated which breaks backups. There is an issue on the same topic (about long file name generated for uploaded file) which was closed without any fix: #9352 (closed). In our case file with long name was leading to backup script error (backup was not created). This behavior is obviously not acceptable and thus has to be fixed. Message from gitlab-task-runner-backup pod log after which backup script exits:

Problem: <class 'OSError: [Errno 36] File name too long: '/srv/gitlab/tmp/uploads/@hashed/db/ae/dbae772db29058a88f9bd830e957c695347c41b6162a7eb9a9ea13def34be56b/e7d19ea9ee6b824bdc4be345f0367602/.s3cmd.0_bfta_v.tmp' -> b'/srv/gitlab/tmp/uploads/@hashed/db/ae/dbae772db29058a88f9bd830e957c695347c41b6162a7eb9a9ea13def34be56b/e7d19ea9ee6b824bdc4be345f0367602/Der_Block_dar\xc3\xbcber__LegalStructure__c__behandelt_ebenfalls_den_Account._Beide_Felder_sollten_in_einem_if__configurationRecords_i_.Entity__c_____Account__c__-_Block_behandelt_werden_und_die_Helfer-Methode_entsprechend_in_transformAccountRecordType_umbenannt_werden__um_zu_zeigen__dass_es_nur_um_den_Account_geht._'

Steps to reproduce

Not sure, probably paste file from OneNote like described in the closed ticket linked above.

What is the current bug behavior?

File name generated for a file uploaded via Gitlab UI is too long and backups script can't proceed with backup.

What is the expected correct behavior?

Gitlab generates sane file names for files uploaded via UI (for example, with timestamps like Jira does) and not something weird...

Relevant logs and/or screenshots

	Sep 18, 2020 @ 02:49:36.831	[34mDumping uploads ...[0m
	Sep 18, 2020 @ 02:49:50.426	[0m
	Sep 18, 2020 @ 02:49:50.426
	Sep 18, 2020 @ 02:49:50.426	!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
	Sep 18, 2020 @ 02:49:50.426	   s3tools-bugs@lists.sourceforge.net
	Sep 18, 2020 @ 02:49:50.426	  info as necessary) to:
	Sep 18, 2020 @ 02:49:50.426	  above lines (removing any private
	Sep 18, 2020 @ 02:49:50.426	  If the error persists, please report the
	Sep 18, 2020 @ 02:49:50.426	    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions
	Sep 18, 2020 @ 02:49:50.426	  and have a look at the known issues list:
	Sep 18, 2020 @ 02:49:50.426	    https://github.com/s3tools/s3cmd
	Sep 18, 2020 @ 02:49:50.426	  branch found at:
	Sep 18, 2020 @ 02:49:50.426	  the latest s3cmd code from the git master
	Sep 18, 2020 @ 02:49:50.426	  Please try reproducing the error using
	Sep 18, 2020 @ 02:49:50.426	    An unexpected error has occurred.
	Sep 18, 2020 @ 02:49:50.426	!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
	Sep 18, 2020 @ 02:49:50.426
	Sep 18, 2020 @ 02:49:50.426	OSError: [Errno 36] File name too long: '/srv/gitlab/tmp/uploads/@hashed/db/ae/dbae772db29058a88f9bd830e957c695347c41b6162a7eb9a9ea13def34be56b/e7d19ea9ee6b824bdc4be345f0367602/.s3cmd.0_bfta_v.tmp' -> b'/srv/gitlab/tmp/uploads/@hashed/db/ae/dbae772db29058a88f9bd830e957c695347c41b6162a7eb9a9ea13def34be56b/e7d19ea9ee6b824bdc4be345f0367602/Der_Block_dar\xc3\xbcber__LegalStructure__c__behandelt_ebenfalls_den_Account._Beide_Felder_sollten_in_einem_if__configurationRecords_i_.Entity__c_____Account__c__-_Block_behandelt_werden_und_die_Helfer-Methode_entsprechend_in_transformAccountRecordType_umbenannt_werden__um_zu_zeigen__dass_es_nur_um_den_Account_geht._'
	Sep 18, 2020 @ 02:49:50.426	    os.rename(chkptfname_b, deunicodise(dst_file))
	Sep 18, 2020 @ 02:49:50.426	  File "/usr/local/bin/s3cmd", line 1343, in _download
	Sep 18, 2020 @ 02:49:50.426	    ret, seq, size_transferred = _download(remote_list, seq, remote_count + update_count, size_transferred, dir_cache)
	Sep 18, 2020 @ 02:49:50.426	  File "/usr/local/bin/s3cmd", line 1488, in cmd_sync_remote2local
	Sep 18, 2020 @ 02:49:50.426	    return cmd_sync_remote2local(args)
	Sep 18, 2020 @ 02:49:50.426	  File "/usr/local/bin/s3cmd", line 1900, in cmd_sync
	Sep 18, 2020 @ 02:49:50.426	    rc = cmd_func(args)
	Sep 18, 2020 @ 02:49:50.426	  File "/usr/local/bin/s3cmd", line 3030, in main
	Sep 18, 2020 @ 02:49:50.426	    rc = main()
	Sep 18, 2020 @ 02:49:50.426	  File "/usr/local/bin/s3cmd", line 3121, in <module>
	Sep 18, 2020 @ 02:49:50.426	Traceback (most recent call last):
	Sep 18, 2020 @ 02:49:50.426
	Sep 18, 2020 @ 02:49:50.426	environment LANG=C.UTF-8
	Sep 18, 2020 @ 02:49:50.426	[GCC 6.3.0 20170516]
	Sep 18, 2020 @ 02:49:50.426	python:   3.7.3 (default, Aug 13 2020, 21:08:12)
	Sep 18, 2020 @ 02:49:50.426	S3cmd:   2.1.0
	Sep 18, 2020 @ 02:49:50.426	Problem: <class 'OSError: [Errno 36] File name too long: '/srv/gitlab/tmp/uploads/@hashed/db/ae/dbae772db29058a88f9bd830e957c695347c41b6162a7eb9a9ea13def34be56b/e7d19ea9ee6b824bdc4be345f0367602/.s3cmd.0_bfta_v.tmp' -> b'/srv/gitlab/tmp/uploads/@hashed/db/ae/dbae772db29058a88f9bd830e957c695347c41b6162a7eb9a9ea13def34be56b/e7d19ea9ee6b824bdc4be345f0367602/Der_Block_dar\xc3\xbcber__LegalStructure__c__behandelt_ebenfalls_den_Account._Beide_Felder_sollten_in_einem_if__configurationRecords_i_.Entity__c_____Account__c__-_Block_behandelt_werden_und_die_Helfer-Methode_entsprechend_in_transformAccountRecordType_umbenannt_werden__um_zu_zeigen__dass_es_nur_um_den_Account_geht._'
	Sep 18, 2020 @ 02:49:50.426	Invoked as: /usr/local/bin/s3cmd --stop-on-error --delete-removed sync s3://<REMOVED BUCKET NAME>/ /srv/gitlab/tmp/uploads/
	Sep 18, 2020 @ 02:49:50.426
	Sep 18, 2020 @ 02:49:50.426	!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
	Sep 18, 2020 @ 02:49:50.426
	Sep 18, 2020 @ 02:49:50.426
	Sep 18, 2020 @ 02:49:50.426	   s3tools-bugs@lists.sourceforge.net
	Sep 18, 2020 @ 02:49:50.426	  info as necessary) to:
	Sep 18, 2020 @ 02:49:50.426	  following lines (removing any private
	Sep 18, 2020 @ 02:49:50.426	  If the error persists, please report the
	Sep 18, 2020 @ 02:49:50.426	    https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions
	Sep 18, 2020 @ 02:49:50.426	  and have a look at the known issues list:
	Sep 18, 2020 @ 02:49:50.426	    https://github.com/s3tools/s3cmd
	Sep 18, 2020 @ 02:49:50.426	  branch found at:
	Sep 18, 2020 @ 02:49:50.426	  the latest s3cmd code from the git master
	Sep 18, 2020 @ 02:49:50.426	  Please try reproducing the error using
	Sep 18, 2020 @ 02:49:50.426	    An unexpected error has occurred.
	Sep 18, 2020 @ 02:49:50.426	!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
	Sep 18, 2020 @ 02:49:50.426
	Sep 18, 2020 @ 02:49:50.426	ERROR: Exiting now because of --stop-on-error
	Sep 18, 2020 @ 02:49:50.426	ERROR: Download of '@hashed/db/ae/dbae772db29058a88f9bd830e957c695347c41b6162a7eb9a9ea13def34be56b/e7d19ea9ee6b824bdc4be345f0367602/Der_Block_darüber__LegalStructure__c__behandelt_ebenfalls_den_Account._Beide_Felder_sollten_in_einem_if__configurationRecords_i_.Entity__c_____Account__c__-_Block_behandelt_werden_und_die_Helfer-Methode_entsprechend_in_transformAccountRecordType_umbenannt_werden__um_zu_zeigen__dass_es_nur_um_den_Account_geht._' failed (Reason: File Name is too long)
	Sep 18, 2020 @ 02:49:50.426	WARNING: Waiting 3 sec...
	Sep 18, 2020 @ 02:49:50.426	[31m[Error] WARNING: Retrying failed request: /%40hashed/db/ae/dbae772db29058a88f9bd830e957c695347c41b6162a7eb9a9ea13def34be56b/e7d19ea9ee6b824bdc4be345f0367602/Der_Block_dar%C3%BCber__LegalStructure__c__behandelt_ebenfalls_den_Account._Beide_Felder_sollten_in_einem_if__configurationRecords_i_.Entity__c_____Account__c__-_Block_behandelt_werden_und_die_Helfer-Methode_entsprechend_in_transformAccountRecordType_umbenannt_werden__um_zu_zeigen__dass_es_nur_um_den_Account_geht._ (Remote end closed connection without response)
	Sep 18, 2020 @ 02:49:50.426	Restore uploads failed

Output of checks

Results of GitLab environment info

git@gitlab-task-runner-7c768578b5-ndhmc:/$ gitlab-rake gitlab:env:info

System information
System:
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.6.6p146
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	unknown
Git Version:	unknown
Sidekiq Version:5.2.9
Go Version:	unknown

GitLab information
Version:	13.3.1-ee
Revision:	f0a10848821
Directory:	/srv/gitlab
DB Adapter:	PostgreSQL
DB Version:	11.7
URL:		<REMOVED URL>
HTTP Clone URL:	<REMOVED URL>
SSH Clone URL:	<REMOVED URL>
Elasticsearch:	no
Geo:		no
Using LDAP:	yes
Using Omniauth:	yes
Omniauth Providers: saml

GitLab Shell
Version:	13.6.0
Repository storage paths:
- default: 	/var/opt/gitlab/repo
GitLab Shell path:		/home/git/gitlab-shell
Git:		/usr/bin/git

Possible fixes

Generate a sane file name (for example, with timestamps like Jira does) for a fil uploaded via UI

Edited by Dmytro Zavalkin