From 0914e73f50da0c98e707c0041f6df01683d71a8d Mon Sep 17 00:00:00 2001 From: Karthik Nayak Date: Wed, 5 Nov 2025 16:04:48 +0100 Subject: [PATCH 1/3] refs: allow setting the reference directory MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit While Git allows users to select different reference backends, unlike with objects, there is no flexibility in selecting the reference directory. Currently, the reference format is obtained from the config of the repository and the reference directory is set to the $GIT_DIR. This patch extends the config 'extensions.refStorage' to take in multiple forms of inputs: - A format name alone (e.g., `reftable` or `files`) which uses the default location (the repository's common directory). - A URI format `://` which explicitly specifies both the format and location (e.g., `reftable:///foo/bar`). We also add in a new ENV variable GIT_REFERENCE_BACKEND which can be used to override the config. One use case for this is migration between different backends. On the server side, migrating from the files backend to the newly introduced reftable backend can be achieved by running 'git refs migrate'. However, for large repositories with millions of references, this migration can take from seconds to minutes. For some background, at GitLab, the criteria for our migration was to reduce the downtime of the migrate ideally to zero. So running 'git refs migrate --ref-format=reftable' by itself wouldn't work, since it scales with the number of references and we have repos with millions of references, so we need to migrate without loosing any information. We came up with the following plan: 1. Run git-pack-refs(1) and note timestamp of the generated packed-refs file. 2. Run git refs migrate –dry-run. 3. If there are no ongoing reference requests (read/write) a. Lock the repository by blocking incoming requests (done on a layer above git, in Gitaly [1]). b. If the timestamp of the packed-refs file has changed, unlock the repo and repeat from step 1. c. Apply all the loose refs to the dry-run reftable folder (this requires support in Git to write refs to arbitrary folder). d. Move the reftable dry-run folder into the GIT_DIR. e. Swap the repo config f. Unlock repo access Using such a route, scales much better since we only have to worry about blocking the repository by O(ref written between #1 and #3a) and not O(refs in repo). But for doing so, we need to be able to write to a arbitrary reference backend + path. This is to add the missing references to the dry-run reftable folder. This series, achieves that. The only crux is that this doesn't work with workspaces, as the files backend currently extracts the worktree directly from the repo's commondir, without consulting the directory passed during initialization of the backend. This is based on top of 9a2fb147f2 (Git 2.52, 2025-11-17). [1]: https://gitlab.com/gitlab-org/gitaly --- Changes in v4: - EDITME: describe what is new in this series revision. - EDITME: use bulletpoints and terse descriptions. - Link to v3: https://patch.msgid.link/20251201-kn-alternate-ref-dir-v3-0-c11b946bc2fa@gmail.com Changes in v3: - Cleanup some stale code which wasn't removed. - Localize strings which will be output to the user. - Remove additional defensive checks which are not needed. - Link to v2: https://patch.msgid.link/20251126-kn-alternate-ref-dir-v2-0-8b9f6f18f635@gmail.com Changes in v2: - Added more clarification and proper intent in the cover message. - Changed the format from '://' to `://` as it much clearer. - Added logic to check for the '//' in the provided URI and a test for the same. - In the tests: - Use test_must_fail() instead of ! git - Fix looped tests not using the variables correctly and ensure that the test description is correct. - Link to v1: https://patch.msgid.link/20251119-kn-alternate-ref-dir-v1-0-4cf4a94c8bed@gmail.com --- b4-submit-tracking --- { "series": { "revision": 4, "change-id": "20251105-kn-alternate-ref-dir-3e572e8cd0ef", "prefixes": [], "history": { "v1": [ "20251119-kn-alternate-ref-dir-v1-0-4cf4a94c8bed@gmail.com" ], "v2": [ "20251126-kn-alternate-ref-dir-v2-0-8b9f6f18f635@gmail.com" ], "v3": [ "20251201-kn-alternate-ref-dir-v3-0-c11b946bc2fa@gmail.com" ] } } } -- GitLab From af0996bc2e933d60deaab0d27822721423f9fe3a Mon Sep 17 00:00:00 2001 From: Karthik Nayak Date: Tue, 2 Dec 2025 23:24:50 +0100 Subject: [PATCH 2/3] refs: allow reference location in refstorage config The 'extensions.refStorage' config is used to specify the reference backend for a given repository. Both the 'files' and 'reftable' backends utilize the $GIT_DIR as the reference folder by default in `get_main_ref_store()`. Since the reference backends are pluggable, this means that they should work with out-of-tree reference directories too. Extend the 'refStorage' config to also support taking an URI input, where users can specify the reference backend and the location. When no location is provided, we simply default to $GIT_DIR as before. When location is provided, we pass on the information to the reference backend. While this commit adds support for interacting with out-of-tree reference directories, we do not add support for creating such directories. Currently only the 'git refs migrate --dry-run' command creates an out-of-tree reference directory. In the future, we can potentially modify the '--ref-format' of 'git-clone(1)' and 'git-init(1)' to also support arbitrary reference locations. Helped-by: Patrick Steinhardt Signed-off-by: Karthik Nayak --- Documentation/config/extensions.adoc | 17 +++- refs.c | 10 +- repository.c | 9 +- repository.h | 5 +- setup.c | 37 +++++++- setup.h | 1 + t/meson.build | 1 + t/t1423-ref-backend.sh | 131 +++++++++++++++++++++++++++ 8 files changed, 200 insertions(+), 11 deletions(-) create mode 100755 t/t1423-ref-backend.sh diff --git a/Documentation/config/extensions.adoc b/Documentation/config/extensions.adoc index 532456644b..ab4d336ad9 100644 --- a/Documentation/config/extensions.adoc +++ b/Documentation/config/extensions.adoc @@ -57,10 +57,25 @@ For historical reasons, this extension is respected regardless of the `core.repositoryFormatVersion` setting. refStorage::: - Specify the ref storage format to use. The acceptable values are: + Specify the ref storage format and location to use. The value can be + either a format name or a URI: + -- +* A format name alone (e.g., `reftable` or `files`) uses the default + location (the repository's common directory). + +* A URI format `://` explicitly specifies both the + format and location (e.g., `reftable:///foo/bar`). + +Supported format names are: ++ include::../ref-storage-format.adoc[] ++ +The location part of the URI is treated as an opaque string and interpreted by +the reference backend itself. For the `files` and `reftable` backends, this is +a filesystem path. Future backends may support other location schemes (e.g., +`postgres://127.0.0.1:5432?database=myrepo`). Currently custom locations do not +work with worktrees. -- + Note that this setting should only be set by linkgit:git-init[1] or diff --git a/refs.c b/refs.c index 5583f6e09d..a7978bbc93 100644 --- a/refs.c +++ b/refs.c @@ -2193,15 +2193,21 @@ void ref_store_release(struct ref_store *ref_store) struct ref_store *get_main_ref_store(struct repository *r) { + const char *location; + if (r->refs_private) return r->refs_private; if (!r->gitdir) BUG("attempting to get main_ref_store outside of repository"); + location = r->ref_storage_location; + if (!location) + location = r->gitdir; + r->refs_private = ref_store_init(r, r->ref_storage_format, - r->gitdir, REF_STORE_ALL_CAPS); - r->refs_private = maybe_debug_wrap_ref_store(r->gitdir, r->refs_private); + location, REF_STORE_ALL_CAPS); + r->refs_private = maybe_debug_wrap_ref_store(location, r->refs_private); return r->refs_private; } diff --git a/repository.c b/repository.c index 6aaa7ba008..7a3d97d35b 100644 --- a/repository.c +++ b/repository.c @@ -204,9 +204,12 @@ void repo_set_compat_hash_algo(struct repository *repo, int algo) } void repo_set_ref_storage_format(struct repository *repo, - enum ref_storage_format format) + enum ref_storage_format format, + const char *location) { repo->ref_storage_format = format; + if (location) + repo->ref_storage_location = xstrdup(location); } /* @@ -288,7 +291,8 @@ int repo_init(struct repository *repo, repo_set_hash_algo(repo, format.hash_algo); repo_set_compat_hash_algo(repo, format.compat_hash_algo); - repo_set_ref_storage_format(repo, format.ref_storage_format); + repo_set_ref_storage_format(repo, format.ref_storage_format, + format.ref_storage_location); repo->repository_format_worktree_config = format.worktree_config; repo->repository_format_relative_worktrees = format.relative_worktrees; repo->repository_format_precious_objects = format.precious_objects; @@ -381,6 +385,7 @@ void repo_clear(struct repository *repo) FREE_AND_NULL(repo->index_file); FREE_AND_NULL(repo->worktree); FREE_AND_NULL(repo->submodule_prefix); + FREE_AND_NULL(repo->ref_storage_location); odb_clear(repo->objects); FREE_AND_NULL(repo->objects); diff --git a/repository.h b/repository.h index 5808a5d610..430ffc4b03 100644 --- a/repository.h +++ b/repository.h @@ -143,6 +143,8 @@ struct repository { /* Repository's reference storage format, as serialized on disk. */ enum ref_storage_format ref_storage_format; + /* Reference storage location information as needed for the backend. */ + char *ref_storage_location; /* A unique-id for tracing purposes. */ int trace2_repo_id; @@ -196,7 +198,8 @@ void repo_set_worktree(struct repository *repo, const char *path); void repo_set_hash_algo(struct repository *repo, int algo); void repo_set_compat_hash_algo(struct repository *repo, int compat_algo); void repo_set_ref_storage_format(struct repository *repo, - enum ref_storage_format format); + enum ref_storage_format format, + const char *location); void initialize_repository(struct repository *repo); RESULT_MUST_BE_USED int repo_init(struct repository *r, const char *gitdir, const char *worktree); diff --git a/setup.c b/setup.c index 7086741e6c..4e3b216b88 100644 --- a/setup.c +++ b/setup.c @@ -633,6 +633,21 @@ static enum extension_result handle_extension_v0(const char *var, return EXTENSION_UNKNOWN; } +static void parse_reference_uri(const char *value, char **format, + char **location) +{ + char *schema_end; + + schema_end = strstr(value, "://"); + if (!schema_end) { + *format = xstrdup(value); + *location = NULL; + } else { + *format = xstrndup(value, schema_end - value); + *location = xstrdup(schema_end + 3); + } +} + /* * Record any new extensions in this function. */ @@ -675,14 +690,22 @@ static enum extension_result handle_extension(const char *var, return EXTENSION_OK; } else if (!strcmp(ext, "refstorage")) { unsigned int format; + char *backend; if (!value) return config_error_nonbool(var); - format = ref_storage_format_by_name(value); + + parse_reference_uri(value, &backend, + &data->ref_storage_location); + + format = ref_storage_format_by_name(backend); + free(backend); + if (format == REF_STORAGE_FORMAT_UNKNOWN) return error(_("invalid value for '%s': '%s'"), "extensions.refstorage", value); data->ref_storage_format = format; + return EXTENSION_OK; } else if (!strcmp(ext, "relativeworktrees")) { data->relative_worktrees = git_config_bool(var, value); @@ -851,6 +874,7 @@ void clear_repository_format(struct repository_format *format) string_list_clear(&format->v1_only_extensions, 0); free(format->work_tree); free(format->partial_clone); + free(format->ref_storage_location); init_repository_format(format); } @@ -1860,7 +1884,8 @@ const char *setup_git_directory_gently(int *nongit_ok) repo_set_compat_hash_algo(the_repository, repo_fmt.compat_hash_algo); repo_set_ref_storage_format(the_repository, - repo_fmt.ref_storage_format); + repo_fmt.ref_storage_format, + repo_fmt.ref_storage_location); the_repository->repository_format_worktree_config = repo_fmt.worktree_config; the_repository->repository_format_relative_worktrees = @@ -1960,7 +1985,8 @@ void check_repository_format(struct repository_format *fmt) repo_set_hash_algo(the_repository, fmt->hash_algo); repo_set_compat_hash_algo(the_repository, fmt->compat_hash_algo); repo_set_ref_storage_format(the_repository, - fmt->ref_storage_format); + fmt->ref_storage_format, + fmt->ref_storage_location); the_repository->repository_format_worktree_config = fmt->worktree_config; the_repository->repository_format_relative_worktrees = @@ -2284,7 +2310,7 @@ void create_reference_database(enum ref_storage_format ref_storage_format, char *to_free = NULL; int reinit = is_reinit(); - repo_set_ref_storage_format(the_repository, ref_storage_format); + repo_set_ref_storage_format(the_repository, ref_storage_format, NULL); if (ref_store_create_on_disk(get_main_ref_store(the_repository), 0, &err)) die("failed to set up refs db: %s", err.buf); @@ -2563,7 +2589,8 @@ static void repository_format_configure(struct repository_format *repo_fmt, } else { repo_fmt->ref_storage_format = REF_STORAGE_FORMAT_DEFAULT; } - repo_set_ref_storage_format(the_repository, repo_fmt->ref_storage_format); + repo_set_ref_storage_format(the_repository, repo_fmt->ref_storage_format, + repo_fmt->ref_storage_location); } int init_db(const char *git_dir, const char *real_git_dir, diff --git a/setup.h b/setup.h index 8522fa8575..918d99ae2a 100644 --- a/setup.h +++ b/setup.h @@ -134,6 +134,7 @@ struct repository_format { int hash_algo; int compat_hash_algo; enum ref_storage_format ref_storage_format; + char *ref_storage_location; int sparse_index; char *work_tree; struct string_list unknown_extensions; diff --git a/t/meson.build b/t/meson.build index 7c994d4643..fd47a69b06 100644 --- a/t/meson.build +++ b/t/meson.build @@ -209,6 +209,7 @@ integration_tests = [ 't1420-lost-found.sh', 't1421-reflog-write.sh', 't1422-show-ref-exists.sh', + 't1423-ref-backend.sh', 't1430-bad-ref-name.sh', 't1450-fsck.sh', 't1451-fsck-buffer.sh', diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh new file mode 100755 index 0000000000..a64a189db9 --- /dev/null +++ b/t/t1423-ref-backend.sh @@ -0,0 +1,131 @@ +#!/bin/sh + +test_description='Test reference backend URIs' + +. ./test-lib.sh + +# Run a git command with the provided reference storage. Reset the backend +# post running the command. +# Usage: run_with_uri +# is the relative path to the repo to run the command in. +# is the original ref storage of the repo. +# is the new URI to be set for the ref storage. +# is the git subcommand to be run in the repository. +run_with_uri() { + repo=$1 && + backend=$2 && + uri=$3 && + cmd=$4 && + + git -C "$repo" config set extensions.refStorage "$uri" && + git -C "$repo" $cmd && + git -C "$repo" config set extensions.refStorage "$backend" +} + +# Test a repository with a given reference storage by running and comparing +# 'git refs list' before and after setting the new reference backend. If +# err_msg is set, expect the command to fail and grep for the provided err_msg. +# Usage: run_with_uri +# is the relative path to the repo to run the command in. +# is the original ref storage of the repo. +# is the new URI to be set for the ref storage. +# (optional) if set, check if 'git-refs(1)' failed with the provided msg. +test_refs_backend() { + repo=$1 && + backend=$2 && + uri=$3 && + err_msg=$4 && + + git -C "$repo" config set core.repositoryformatversion 1 && + if test -n "$err_msg"; + then + git -C "$repo" config set extensions.refStorage "$uri" && + test_must_fail git -C "$repo" refs list 2>err && + test_grep "$err_msg" err + else + git -C "$repo" refs list >expect && + run_with_uri "$repo" "$backend" "$uri" "refs list" >actual && + test_cmp expect actual + fi +} + +test_expect_success 'URI is invalid' ' + test_when_finished "rm -rf repo" && + git init --ref-format=files repo && + test_refs_backend repo files "reftable@/home/reftable" \ + "invalid value for ${SQ}extensions.refstorage${SQ}" +' + +test_expect_success 'URI ends with colon' ' + test_when_finished "rm -rf repo" && + git init --ref-format=files repo && + test_refs_backend repo files "reftable:" \ + "invalid value for ${SQ}extensions.refstorage${SQ}" +' + +test_expect_success 'unknown reference backend' ' + test_when_finished "rm -rf repo" && + git init --ref-format=reftable repo && + test_refs_backend repo files "db://.git" \ + "invalid value for ${SQ}extensions.refstorage${SQ}" +' + +test_expect_success 'URI with empty path' ' + test_when_finished "rm -rf repo" && + git init --ref-format=reftable repo && + test_refs_backend repo files "reftable://" +' + +ref_formats="files reftable" +for from_format in $ref_formats +do + for to_format in $ref_formats + do + if test "$from_format" = "$to_format" + then + continue + fi + + test_expect_success "read from $to_format backend" ' + test_when_finished "rm -rf repo" && + git init --ref-format=$from_format repo && + ( + cd repo && + test_commit 1 && + test_commit 2 && + test_commit 3 && + + git refs migrate --dry-run --ref-format=$to_format >out && + BACKEND_PATH=$(sed "s/.* ${SQ}\(.*\)${SQ}/\1/" out) && + test_refs_backend . $from_format "$to_format://$BACKEND_PATH" + ) + ' + + test_expect_success "write to $to_format backend" ' + test_when_finished "rm -rf repo" && + git init --ref-format=$from_format repo && + ( + cd repo && + test_commit 1 && + test_commit 2 && + test_commit 3 && + + git refs migrate --dry-run --ref-format=$to_format >out && + BACKEND_PATH=$(sed "s/.* ${SQ}\(.*\)${SQ}/\1/" out) && + + test_refs_backend . $from_format "$to_format://$BACKEND_PATH" && + + git refs list >expect && + run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "tag -d 1" && + git refs list >actual && + test_cmp expect actual && + + git refs list | grep -v "refs/tags/1" >expect && + run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "refs list" >actual && + test_cmp expect actual + ) + ' + done +done + +test_done -- GitLab From 3a83e79b57bbc5007d8d878bc158994f95a71109 Mon Sep 17 00:00:00 2001 From: Karthik Nayak Date: Thu, 4 Dec 2025 09:44:53 +0100 Subject: [PATCH 3/3] refs: add GIT_REFERENCE_BACKEND to specify reference backend MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Git allows setting a different object directory via 'GIT_OBJECT_DIRECTORY', but provides no equivalent for references. In the previous commit we extended the 'extensions.refStorage' config to also support an URI input for reference backend with location. Let's also add a new environment variable 'GIT_REFERENCE_BACKEND' that takes in the same input as the config variable. Having an environment variable allows us to modify the reference backend and location on the fly for individual git commands. Helped-by: default avatarJean-Noël Avila Signed-off-by: default avatarKarthik Nayak --- Documentation/git.adoc | 5 ++ environment.h | 1 + setup.c | 190 +++++++++++++++++++++-------------------- t/t1423-ref-backend.sh | 75 ++++++++++------ 4 files changed, 153 insertions(+), 118 deletions(-) diff --git a/Documentation/git.adoc b/Documentation/git.adoc index ce099e78b8..ed3191e8f6 100644 --- a/Documentation/git.adoc +++ b/Documentation/git.adoc @@ -584,6 +584,11 @@ double-quotes and respecting backslash escapes. E.g., the value repositories will be set to this value. The default is "files". See `--ref-format` in linkgit:git-init[1]. +`GIT_REFERENCE_BACKEND`:: + Specify which reference backend to be used along with its URI. + See `extensions.refStorage` option in linkgit:git-config[1] for more + description. Overrides the config variable when used. + Git Commits ~~~~~~~~~~~ `GIT_AUTHOR_NAME`:: diff --git a/environment.h b/environment.h index 51898c99cd..73c3137fe1 100644 --- a/environment.h +++ b/environment.h @@ -42,6 +42,7 @@ #define GIT_OPTIONAL_LOCKS_ENVIRONMENT "GIT_OPTIONAL_LOCKS" #define GIT_TEXT_DOMAIN_DIR_ENVIRONMENT "GIT_TEXTDOMAINDIR" #define GIT_ATTR_SOURCE_ENVIRONMENT "GIT_ATTR_SOURCE" +#define GIT_REFERENCE_BACKEND_ENVIRONMENT "GIT_REFERENCE_BACKEND" /* * Environment variable used to propagate the --no-advice global option to the diff --git a/setup.c b/setup.c index 4e3b216b88..00251bf0de 100644 --- a/setup.c +++ b/setup.c @@ -224,8 +224,8 @@ static void NORETURN die_verify_filename(struct repository *r, /* ... or fall back the most general message. */ die(_("ambiguous argument '%s': unknown revision or path not in the working tree.\n" "Use '--' to separate paths from revisions, like this:\n" - "'git [...] -- [...]'"), arg); - + "'git [...] -- [...]'"), + arg); } /* @@ -309,7 +309,8 @@ void verify_non_filename(const char *prefix, const char *arg) return; die(_("ambiguous argument '%s': both revision and filename\n" "Use '--' to separate paths from revisions, like this:\n" - "'git [...] -- [...]'"), arg); + "'git [...] -- [...]'"), + arg); } int get_common_dir(struct strbuf *sb, const char *gitdir) @@ -366,7 +367,7 @@ static int validate_headref(const char *path) /* Make sure it is a "refs/.." symlink */ if (S_ISLNK(st.st_mode)) { - len = readlink(path, buffer, sizeof(buffer)-1); + len = readlink(path, buffer, sizeof(buffer) - 1); if (len >= 5 && !memcmp("refs/", buffer, 5)) return 0; return -1; @@ -378,7 +379,7 @@ static int validate_headref(const char *path) fd = open(path, O_RDONLY); if (fd < 0) return -1; - len = read_in_full(fd, buffer, sizeof(buffer)-1); + len = read_in_full(fd, buffer, sizeof(buffer) - 1); close(fd); if (len < 0) @@ -436,8 +437,7 @@ int is_git_directory(const char *suspect) if (getenv(DB_ENVIRONMENT)) { if (access(getenv(DB_ENVIRONMENT), X_OK)) goto done; - } - else { + } else { strbuf_setlen(&path, len); strbuf_addstr(&path, "/objects"); if (access(path.buf, X_OK)) @@ -541,12 +541,12 @@ static void setup_original_cwd(void) "realpath-path", tmp_original_cwd); trace2_data_string("setup", the_repository, "realpath-failure", strerror(errno)); - free((char*)tmp_original_cwd); + free((char *)tmp_original_cwd); tmp_original_cwd = NULL; return; } - free((char*)tmp_original_cwd); + free((char *)tmp_original_cwd); tmp_original_cwd = NULL; startup_info->original_cwd = strbuf_detach(&tmp, NULL); @@ -572,14 +572,13 @@ static void setup_original_cwd(void) * original_cwd was inside worktree; precompose it just as * we do prefix so that built up paths will match */ - startup_info->original_cwd = \ - precompose_string_if_needed(startup_info->original_cwd - + offset); + startup_info->original_cwd = + precompose_string_if_needed(startup_info->original_cwd + offset); return; } no_prevention_needed: - free((char*)startup_info->original_cwd); + free((char *)startup_info->original_cwd); startup_info->original_cwd = NULL; } @@ -615,22 +614,22 @@ static enum extension_result handle_extension_v0(const char *var, const char *ext, struct repository_format *data) { - if (!strcmp(ext, "noop")) { - return EXTENSION_OK; - } else if (!strcmp(ext, "preciousobjects")) { - data->precious_objects = git_config_bool(var, value); - return EXTENSION_OK; - } else if (!strcmp(ext, "partialclone")) { - if (!value) - return config_error_nonbool(var); - data->partial_clone = xstrdup(value); - return EXTENSION_OK; - } else if (!strcmp(ext, "worktreeconfig")) { - data->worktree_config = git_config_bool(var, value); - return EXTENSION_OK; - } + if (!strcmp(ext, "noop")) { + return EXTENSION_OK; + } else if (!strcmp(ext, "preciousobjects")) { + data->precious_objects = git_config_bool(var, value); + return EXTENSION_OK; + } else if (!strcmp(ext, "partialclone")) { + if (!value) + return config_error_nonbool(var); + data->partial_clone = xstrdup(value); + return EXTENSION_OK; + } else if (!strcmp(ext, "worktreeconfig")) { + data->worktree_config = git_config_bool(var, value); + return EXTENSION_OK; + } - return EXTENSION_UNKNOWN; + return EXTENSION_UNKNOWN; } static void parse_reference_uri(const char *value, char **format, @@ -683,8 +682,8 @@ static enum extension_result handle_extension(const char *var, for_each_string_list_item(item, &data->v1_only_extensions) { if (!strcmp(item->string, "compatobjectformat")) return error(_("'%s' already specified as '%s'"), - "extensions.compatobjectformat", - hash_algos[data->compat_hash_algo].name); + "extensions.compatobjectformat", + hash_algos[data->compat_hash_algo].name); } data->compat_hash_algo = format; return EXTENSION_OK; @@ -953,7 +952,7 @@ void read_gitfile_error_die(int error_code, const char *path, const char *dir) */ const char *read_gitfile_gently(const char *path, int *return_error_code) { - const int max_file_size = 1 << 20; /* 1MB */ + const int max_file_size = 1 << 20; /* 1MB */ int error_code = 0; char *buf = NULL; char *dir = NULL; @@ -1002,7 +1001,7 @@ const char *read_gitfile_gently(const char *path, int *return_error_code) dir = buf + 8; if (!is_absolute_path(dir) && (slash = strrchr(path, '/'))) { - size_t pathlen = slash+1 - path; + size_t pathlen = slash + 1 - path; dir = xstrfmt("%.*s%.*s", (int)pathlen, path, (int)(len - 8), buf + 8); free(buf); @@ -1039,7 +1038,7 @@ static const char *setup_explicit_git_dir(const char *gitdirenv, if (PATH_MAX - 40 < strlen(gitdirenv)) die(_("'$%s' too big"), GIT_DIR_ENVIRONMENT); - gitfile = (char*)read_gitfile(gitdirenv); + gitfile = (char *)read_gitfile(gitdirenv); if (gitfile) { gitfile = xstrdup(gitfile); gitdirenv = gitfile; @@ -1073,8 +1072,7 @@ static const char *setup_explicit_git_dir(const char *gitdirenv, set_git_dir(gitdirenv, 0); free(gitfile); return NULL; - } - else if (git_work_tree_cfg) { /* #6, #14 */ + } else if (git_work_tree_cfg) { /* #6, #14 */ if (is_absolute_path(git_work_tree_cfg)) set_git_work_tree(git_work_tree_cfg); else { @@ -1089,14 +1087,12 @@ static const char *setup_explicit_git_dir(const char *gitdirenv, set_git_work_tree(core_worktree); free(core_worktree); } - } - else if (!git_env_bool(GIT_IMPLICIT_WORK_TREE_ENVIRONMENT, 1)) { + } else if (!git_env_bool(GIT_IMPLICIT_WORK_TREE_ENVIRONMENT, 1)) { /* #16d */ set_git_dir(gitdirenv, 0); free(gitfile); return NULL; - } - else /* #2, #10 */ + } else /* #2, #10 */ set_git_work_tree("."); /* set_git_work_tree() must have been called by now */ @@ -1110,7 +1106,7 @@ static const char *setup_explicit_git_dir(const char *gitdirenv, } offset = dir_inside_of(cwd->buf, worktree); - if (offset >= 0) { /* cwd inside worktree? */ + if (offset >= 0) { /* cwd inside worktree? */ set_git_dir(gitdirenv, 1); if (chdir(worktree)) die_errno(_("cannot chdir to '%s'"), worktree); @@ -1202,8 +1198,7 @@ static const char *setup_bare_git_dir(struct strbuf *cwd, int offset, root_len = offset_1st_component(cwd->buf); strbuf_setlen(cwd, offset > root_len ? offset : root_len); set_git_dir(cwd->buf, 0); - } - else + } else set_git_dir(".", 0); return NULL; } @@ -1211,12 +1206,11 @@ static const char *setup_bare_git_dir(struct strbuf *cwd, int offset, static dev_t get_device_or_die(const char *path, const char *prefix, int prefix_len) { struct stat buf; - if (stat(path, &buf)) { + if (stat(path, &buf)) die_errno(_("failed to stat '%*s%s%s'"), - prefix_len, - prefix ? prefix : "", - prefix ? "/" : "", path); - } + prefix_len, + prefix ? prefix : "", + prefix ? "/" : "", path); return buf.st_dev; } @@ -1243,9 +1237,8 @@ static int canonicalize_ceiling_entry(struct string_list_item *item, return 1; } else { char *real_path = real_pathdup(ceil, 0); - if (!real_path) { + if (!real_path) return 0; - } free(item->string); item->string = real_path; return 1; @@ -1525,7 +1518,8 @@ static enum discovery_result setup_git_directory_gently_1(struct strbuf *dir, strbuf_addch(dir, '/'); strbuf_addstr(dir, DEFAULT_GIT_DIR_ENVIRONMENT); gitdirenv = read_gitfile_gently(dir->buf, die_on_error ? - NULL : &error_code); + NULL : + &error_code); if (!gitdirenv) { if (die_on_error || error_code == READ_GITFILE_ERR_NOT_A_FILE) { @@ -1592,7 +1586,7 @@ static enum discovery_result setup_git_directory_gently_1(struct strbuf *dir, if (offset <= ceil_offset) return GIT_DIR_HIT_CEILING; - strbuf_setlen(dir, offset > min_offset ? offset : min_offset); + strbuf_setlen(dir, offset > min_offset ? offset : min_offset); if (one_filesystem && current_device != get_device_or_die(dir->buf, NULL, offset)) return GIT_DIR_HIT_MOUNT_POINT; @@ -1665,9 +1659,8 @@ void setup_git_env(const char *git_dir) args.graft_file = getenv_safe(&to_free, GRAFT_ENVIRONMENT); args.index_file = getenv_safe(&to_free, INDEX_ENVIRONMENT); args.alternate_db = getenv_safe(&to_free, ALTERNATE_DB_ENVIRONMENT); - if (getenv(GIT_QUARANTINE_ENVIRONMENT)) { + if (getenv(GIT_QUARANTINE_ENVIRONMENT)) args.disable_ref_updates = 1; - } repo_set_gitdir(the_repository, git_dir, &args); strvec_clear(&to_free); @@ -1675,8 +1668,7 @@ void setup_git_env(const char *git_dir) if (getenv(NO_REPLACE_OBJECTS_ENVIRONMENT)) disable_replace_refs(); replace_ref_base = getenv(GIT_REPLACE_REF_BASE_ENVIRONMENT); - git_replace_ref_base = xstrdup(replace_ref_base ? replace_ref_base - : "refs/replace/"); + git_replace_ref_base = xstrdup(replace_ref_base ? replace_ref_base : "refs/replace/"); update_ref_namespace(NAMESPACE_REPLACE, git_replace_ref_base); shallow_file = getenv(GIT_SHALLOW_FILE_ENVIRONMENT); @@ -1757,6 +1749,7 @@ const char *setup_git_directory_gently(int *nongit_ok) static struct strbuf cwd = STRBUF_INIT; struct strbuf dir = STRBUF_INIT, gitdir = STRBUF_INIT, report = STRBUF_INIT; const char *prefix = NULL; + const char *ref_backend_uri; struct repository_format repo_fmt = REPOSITORY_FORMAT_INIT; /* @@ -1824,11 +1817,10 @@ const char *setup_git_directory_gently(int *nongit_ok) *nongit_ok = 1; break; case GIT_DIR_DISALLOWED_BARE: - if (!nongit_ok) { + if (!nongit_ok) die(_("cannot use bare repository '%s' (safe.bareRepository is '%s')"), dir.buf, allowed_bare_repo_to_string(get_allowed_bare_repo())); - } *nongit_ok = 1; break; case GIT_DIR_CWD_FAILURE: @@ -1914,6 +1906,25 @@ const char *setup_git_directory_gently(int *nongit_ok) setenv(GIT_PREFIX_ENVIRONMENT, "", 1); } + /* + * The env variable should override the repository config + * for 'extensions.refStorage'. + */ + ref_backend_uri = getenv(GIT_REFERENCE_BACKEND_ENVIRONMENT); + if (ref_backend_uri) { + char *backend, *location; + enum ref_storage_format format; + + parse_reference_uri(ref_backend_uri, &backend, &location); + format = ref_storage_format_by_name(backend); + if (format == REF_STORAGE_FORMAT_UNKNOWN) + die(_("unknown ref storage format: '%s'"), backend); + repo_set_ref_storage_format(the_repository, format, location); + + free(backend); + free(location); + } + setup_original_cwd(); strbuf_release(&dir); @@ -1953,11 +1964,11 @@ int git_config_perm(const char *var, const char *value) * a chmod value to restrict to. */ switch (i) { - case PERM_UMASK: /* 0 */ + case PERM_UMASK: /* 0 */ return PERM_UMASK; - case OLD_PERM_GROUP: /* 1 */ + case OLD_PERM_GROUP: /* 1 */ return PERM_GROUP; - case OLD_PERM_EVERYBODY: /* 2 */ + case OLD_PERM_EVERYBODY: /* 2 */ return PERM_EVERYBODY; } @@ -1965,8 +1976,9 @@ int git_config_perm(const char *var, const char *value) if ((i & 0600) != 0600) die(_("problem with core.sharedRepository filemode value " - "(0%.3o).\nThe owner of files must always have " - "read and write permissions."), i); + "(0%.3o).\nThe owner of files must always have " + "read and write permissions."), + i); /* * Mask filemode value. Others can not get write permission. @@ -2031,12 +2043,12 @@ int daemonize(void) return -1; #else switch (fork()) { - case 0: - break; - case -1: - die_errno(_("fork failed")); - default: - exit(0); + case 0: + break; + case -1: + die_errno(_("fork failed")); + default: + exit(0); } if (setsid() == -1) die_errno(_("setsid failed")); @@ -2100,9 +2112,9 @@ const char *get_template_dir(const char *option_template) } #ifdef NO_TRUSTABLE_FILEMODE -#define TEST_FILEMODE 0 +# define TEST_FILEMODE 0 #else -#define TEST_FILEMODE 1 +# define TEST_FILEMODE 1 #endif #define GIT_DEFAULT_HASH_ENVIRONMENT "GIT_DEFAULT_HASH" @@ -2136,8 +2148,7 @@ static void copy_templates_1(struct strbuf *path, struct strbuf *template_path, if (lstat(path->buf, &st_git)) { if (errno != ENOENT) die_errno(_("cannot stat '%s'"), path->buf); - } - else + } else exists = 1; if (lstat(template_path->buf, &st_template)) @@ -2151,8 +2162,7 @@ static void copy_templates_1(struct strbuf *path, struct strbuf *template_path, strbuf_addch(template_path, '/'); copy_templates_1(path, template_path, subdir); closedir(subdir); - } - else if (exists) + } else if (exists) continue; else if (S_ISLNK(st_template.st_mode)) { struct strbuf lnk = STRBUF_INIT; @@ -2163,13 +2173,11 @@ static void copy_templates_1(struct strbuf *path, struct strbuf *template_path, die_errno(_("cannot symlink '%s' '%s'"), lnk.buf, path->buf); strbuf_release(&lnk); - } - else if (S_ISREG(st_template.st_mode)) { + } else if (S_ISREG(st_template.st_mode)) { if (copy_file(path->buf, template_path->buf, st_template.st_mode)) die_errno(_("cannot copy '%s' to '%s'"), template_path->buf, path->buf); - } - else + } else error(_("ignoring template %s"), template_path->buf); } } @@ -2210,7 +2218,7 @@ static void copy_templates(const char *option_template) if (template_format.version >= 0 && verify_repository_format(&template_format, &err) < 0) { warning(_("not copying templates from '%s': %s"), - template_dir, err.buf); + template_dir, err.buf); strbuf_release(&err); goto close_free_return; } @@ -2384,9 +2392,8 @@ static int create_default_files(const char *template_path, * We would have created the above under user's umask -- under * shared-repository settings, we would need to fix them up. */ - if (repo_settings_get_shared_repository(the_repository)) { + if (repo_settings_get_shared_repository(the_repository)) adjust_shared_perm(the_repository, repo_get_git_dir(the_repository)); - } initialize_repository_version(fmt->hash_algo, fmt->ref_storage_format, reinit); @@ -2396,9 +2403,9 @@ static int create_default_files(const char *template_path, if (TEST_FILEMODE && !lstat(path.buf, &st1)) { struct stat st2; filemode = (!chmod(path.buf, st1.st_mode ^ S_IXUSR) && - !lstat(path.buf, &st2) && - st1.st_mode != st2.st_mode && - !chmod(path.buf, st1.st_mode)); + !lstat(path.buf, &st2) && + st1.st_mode != st2.st_mode && + !chmod(path.buf, st1.st_mode)); if (filemode && !reinit && (st1.st_mode & S_IXUSR)) filemode = 0; } @@ -2616,8 +2623,7 @@ int init_db(const char *git_dir, const char *real_git_dir, set_git_dir(real_git_dir, 1); git_dir = repo_get_git_dir(the_repository); separate_git_dir(git_dir, original_git_dir); - } - else { + } else { set_git_dir(git_dir, 1); git_dir = repo_get_git_dir(the_repository); } @@ -2675,15 +2681,11 @@ int init_db(const char *git_dir, const char *real_git_dir, int len = strlen(git_dir); if (reinit) - printf(repo_settings_get_shared_repository(the_repository) - ? _("Reinitialized existing shared Git repository in %s%s\n") - : _("Reinitialized existing Git repository in %s%s\n"), - git_dir, len && git_dir[len-1] != '/' ? "/" : ""); + printf(repo_settings_get_shared_repository(the_repository) ? _("Reinitialized existing shared Git repository in %s%s\n") : _("Reinitialized existing Git repository in %s%s\n"), + git_dir, len && git_dir[len - 1] != '/' ? "/" : ""); else - printf(repo_settings_get_shared_repository(the_repository) - ? _("Initialized empty shared Git repository in %s%s\n") - : _("Initialized empty Git repository in %s%s\n"), - git_dir, len && git_dir[len-1] != '/' ? "/" : ""); + printf(repo_settings_get_shared_repository(the_repository) ? _("Initialized empty shared Git repository in %s%s\n") : _("Initialized empty Git repository in %s%s\n"), + git_dir, len && git_dir[len - 1] != '/' ? "/" : ""); } clear_repository_format(&repo_fmt); diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh index a64a189db9..11ed8eba13 100755 --- a/t/t1423-ref-backend.sh +++ b/t/t1423-ref-backend.sh @@ -11,15 +11,24 @@ test_description='Test reference backend URIs' # is the original ref storage of the repo. # is the new URI to be set for the ref storage. # is the git subcommand to be run in the repository. +# if 'config', set the backend via the 'extensions.refStorage' config. +# if 'env', set the backend via the 'GIT_REFERENCE_BACKEND' env. run_with_uri() { repo=$1 && backend=$2 && uri=$3 && cmd=$4 && + via=$5 && - git -C "$repo" config set extensions.refStorage "$uri" && - git -C "$repo" $cmd && - git -C "$repo" config set extensions.refStorage "$backend" + if test "$via" = "env" + then + test_env GIT_REFERENCE_BACKEND="$uri" git -C "$repo" $cmd + elif test "$via" = "config" + then + git -C "$repo" config set extensions.refStorage "$uri" && + git -C "$repo" $cmd && + git -C "$repo" config set extensions.refStorage "$backend" + fi } # Test a repository with a given reference storage by running and comparing @@ -29,51 +38,65 @@ run_with_uri() { # is the relative path to the repo to run the command in. # is the original ref storage of the repo. # is the new URI to be set for the ref storage. +# if 'config', set the backend via the 'extensions.refStorage' config. +# if 'env', set the backend via the 'GIT_REFERENCE_BACKEND' env. # (optional) if set, check if 'git-refs(1)' failed with the provided msg. test_refs_backend() { repo=$1 && backend=$2 && uri=$3 && - err_msg=$4 && + via=$4 && + err_msg=$5 && git -C "$repo" config set core.repositoryformatversion 1 && if test -n "$err_msg"; then - git -C "$repo" config set extensions.refStorage "$uri" && - test_must_fail git -C "$repo" refs list 2>err && - test_grep "$err_msg" err + if test "$via" = "env" + then + test_env GIT_REFERENCE_BACKEND="$uri" test_must_fail git -C "$repo" refs list 2>err && + test_grep "unknown ref storage format" err + elif test "$via" = "config" + then + git -C "$repo" config set extensions.refStorage "$uri" && + test_must_fail git -C "$repo" refs list 2>err && + test_grep "$err_msg" err + fi else git -C "$repo" refs list >expect && - run_with_uri "$repo" "$backend" "$uri" "refs list" >actual && + run_with_uri "$repo" "$backend" "$uri" "refs list" "$via" >actual && test_cmp expect actual fi } -test_expect_success 'URI is invalid' ' +methods="env config" +for method in $methods +do + +test_expect_success "$method: URI is invalid" ' test_when_finished "rm -rf repo" && git init --ref-format=files repo && - test_refs_backend repo files "reftable@/home/reftable" \ + test_refs_backend repo files "reftable@/home/reftable" "$method" \ "invalid value for ${SQ}extensions.refstorage${SQ}" ' -test_expect_success 'URI ends with colon' ' +test_expect_success "$method: URI ends with colon" ' test_when_finished "rm -rf repo" && git init --ref-format=files repo && - test_refs_backend repo files "reftable:" \ + test_refs_backend repo files "reftable:" "$method" \ "invalid value for ${SQ}extensions.refstorage${SQ}" ' -test_expect_success 'unknown reference backend' ' +test_expect_success "$method: unknown reference backend" ' test_when_finished "rm -rf repo" && git init --ref-format=reftable repo && - test_refs_backend repo files "db://.git" \ + test_refs_backend repo files "db://.git" "$method" \ "invalid value for ${SQ}extensions.refstorage${SQ}" ' -test_expect_success 'URI with empty path' ' +test_expect_success "$method: URI with empty path" ' test_when_finished "rm -rf repo" && git init --ref-format=reftable repo && - test_refs_backend repo files "reftable://" + test_refs_backend repo files "reftable://" "$method" ' ref_formats="files reftable" @@ -86,7 +109,7 @@ do continue fi - test_expect_success "read from $to_format backend" ' + test_expect_success "$method: read from $to_format backend" ' test_when_finished "rm -rf repo" && git init --ref-format=$from_format repo && ( @@ -97,11 +120,11 @@ do git refs migrate --dry-run --ref-format=$to_format >out && BACKEND_PATH=$(sed "s/.* ${SQ}\(.*\)${SQ}/\1/" out) && - test_refs_backend . $from_format "$to_format://$BACKEND_PATH" + test_refs_backend . $from_format "$to_format://$BACKEND_PATH" "$method" ) ' - test_expect_success "write to $to_format backend" ' + test_expect_success "$method: write to $to_format backend" ' test_when_finished "rm -rf repo" && git init --ref-format=$from_format repo && ( @@ -113,19 +136,23 @@ do git refs migrate --dry-run --ref-format=$to_format >out && BACKEND_PATH=$(sed "s/.* ${SQ}\(.*\)${SQ}/\1/" out) && - test_refs_backend . $from_format "$to_format://$BACKEND_PATH" && + test_refs_backend . $from_format "$to_format://$BACKEND_PATH" "$method" && git refs list >expect && - run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "tag -d 1" && + run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \ + "tag -d 1" "$method" && git refs list >actual && test_cmp expect actual && git refs list | grep -v "refs/tags/1" >expect && - run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "refs list" >actual && + run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \ + "refs list" "$method" >actual && test_cmp expect actual ) ' - done -done + done # closes to_format +done # closes from_format + +done # closes method test_done -- GitLab