[go: up one dir, main page]

Project updates to last_activity_at can be lost

The following thing happened:

  • A user pushed a tag to a repository
  • A configured system webhook correctly triggered and sent a correct notification of the tag push event
  • Nevertheless, the last_activity_at field in the project as seen via the gitlab API was not updated
  • And, probably as a consequence, the project does not appear as expected in the /api/v4/projects?order_by=last_activity_at listing

I think such a scenario should be impossible. That it happened is disturbing.

As for consequences: monitoring systems (malware detection, data warehousing etc.) could miss updates. There are other possibilities. In our context, this could mean that software updates would not be propagated.

Background

This happend on Debian's gitlab instance, salsa.debian.org, which is running GitLab Community Edition v18.2.8.

Our tag2upload service receives information about tags via a global system webhook. It wants to process all (public) tags in any (public) repository - or at least, it needs to be aware of them so that it can consider whether they are tag2upload instructions.

After we discovered that sometimes webhook events are not sent https://salsa.debian.org/salsa/support/-/issues/516, we wrote and deployed a scraping system. The scraper polls, using the gitlab API, to try to separately discover any tags that the webhook arrangements missed.

We use the /projects API for this, sorting by last_activity at. That ought to allow us to detect all tag pushes, separately from the webhook.

Our tag2upload service is capable of reporting on tags that were seen by only one of the two methods. This was intended to provide confirmation that the system is functioning as expected.

The specific lost API call event

The tag in question was https://salsa.debian.org/debian/libdevel-symdump-perl/-/tags/debian%2F2.18-6 which is a normal signed git tag, but it is also an instruction to Debian's tag2upload service.

It was pushed to the gitlab at 2025-11-23T12:07:23Z.

The configured system webhook duly notified the tag to the tag2upload service, as intended. The tag was then processed by the tag2upload service.

However, the backup polling scraper did not detect this tag. I investigated, and:

  • The repository's last_activity_at remained as it was before the tag push. Ie the tag push didn't update last_activity_at.
  • Probably as a consequence, the /projects? API endpoing used by the scraper also failed to report this repository as being changed.

API behaviour

To check whether I have understood the API intent correctly, and to investigate whether the behaviour was instance-specific, I experimentally created a test tag, in a little-used repository, on the another gitlab instance. That correctly updated the last_activity_at.

If tag pushes didn't count, then there would be no reliable way to detect all tags, since webhooks aren't reliable either. But it appears that the intent is that they should. That also aligns with the documentation, which says that it is updated by all "events" which ought to include everything that can generate a project webhook event.

Possible repro

However, after that, deleting and recreating tags in the same repository seems to not update the last_activity_at any more!

Specficially:

So far so good. Then:

  • I deleted that tag from the remote (git push origin :refs/tags/test-tag) and re-pushed it.
  • I re-made the tag locally, and pushed the updated version.
  • I made two other tags, test-tag-2 and test-tag-3 and pushed and deleted them a few times.
  • None of these subsequent activities updated the last_activity_at

Note that the last_activity_at is still 2025-11-24T11:29:49.818Z, which is the date of the now-deleted first instance of test-tag (perhaps the contained date, or perhaps the time of the push).

This is despite the fact that there is a more recent tag test-tag-3.

Observations

I'm pretty sure I tested deleting and recreating a tag earlier, before I wrote the scraping code, since the correctness of the scraper algorithm depends on tag pushes "counting" for the purposes of last_activity_at.

It looks like tag pushes update last_activity_at but only if there hasn't been other activity recently. I conjecture that some kind of cacheing is at work.

Edited by 🤖 GitLab Bot 🤖