From 56134313355015918bfb03dc7f39dc4d2747e8b9 Mon Sep 17 00:00:00 2001 From: Olivier El Mekki Date: Wed, 19 Jul 2023 17:17:06 +0200 Subject: [PATCH 01/10] Add ActivityPub Releases profile and outbox MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This MR is the first part in the implementation of ActivityPub in Gitlab, as specified in [my design documents](https://gitlab.com/oelmekki/gitlab-activitypub-design), in the context of [epic 260](https://gitlab.com/groups/gitlab-org/-/epics/260) , and more specifically [issue 21582](https://gitlab.com/gitlab-org/gitlab/-/issues/21582). Summary of those sources: we want to implement the ActivityPub protocol in Gitlab to allow it to federate instances and join the fediverse, so that: * the fediverse (like Mastodon, Lemmy, and apparently soon Threads) can subscribe to content in Gitlab * various instances of Gitlab can communicate * ultimately, cross instances merge requests become possible This is the very humble beginning. As a first step of the first step of… we're implementing a way for the fediverse to read content from Gitlab. We're starting by exposing project releases, as it will be immediately useful: when this MR and the next one (allowing to subscribe to the feed and receive push notifications) will be merged, fediverse users will be able to be notified on their favorite fediverse app when their favorite software releases a new version (provided it's on Gitlab, obviously :) ). This MR alone does not allow that yet, but it allows to display the profile of the resource (called an actor, in ActivityPub terminology). More importantly, it creates the fundations on which the rest of ActivityPub features will be built. --- .../projects/releases_controller.rb | 17 ++ app/serializers/activity_pub/README.md | 90 ++++++++++ .../activity_streams_serializer.rb | 84 +++++++++ .../activity_pub/project_entity.rb | 23 +++ .../activity_pub/release_entity.rb | 40 +++++ .../activity_pub/releases_actor_entity.rb | 29 ++++ .../activity_pub/releases_actor_serializer.rb | 7 + .../releases_outbox_serializer.rb | 9 + app/serializers/activity_pub/user_entity.rb | 22 +++ config/initializers/mime_types.rb | 2 + config/routes/project.rb | 4 + .../projects/releases_controller_spec.rb | 70 +++++++- .../activity_streams_serializer_spec.rb | 163 ++++++++++++++++++ .../activity_pub/project_entity_spec.rb | 32 ++++ .../activity_pub/release_entity_spec.rb | 48 ++++++ .../releases_actor_entity_spec.rb | 39 +++++ .../releases_actor_serializer_spec.rb | 16 ++ .../releases_outbox_serializer_spec.rb | 38 ++++ .../activity_pub/user_entity_spec.rb | 28 +++ 19 files changed, 760 insertions(+), 1 deletion(-) create mode 100644 app/serializers/activity_pub/README.md create mode 100644 app/serializers/activity_pub/activity_streams_serializer.rb create mode 100644 app/serializers/activity_pub/project_entity.rb create mode 100644 app/serializers/activity_pub/release_entity.rb create mode 100644 app/serializers/activity_pub/releases_actor_entity.rb create mode 100644 app/serializers/activity_pub/releases_actor_serializer.rb create mode 100644 app/serializers/activity_pub/releases_outbox_serializer.rb create mode 100644 app/serializers/activity_pub/user_entity.rb create mode 100644 spec/serializers/activity_pub/activity_streams_serializer_spec.rb create mode 100644 spec/serializers/activity_pub/project_entity_spec.rb create mode 100644 spec/serializers/activity_pub/release_entity_spec.rb create mode 100644 spec/serializers/activity_pub/releases_actor_entity_spec.rb create mode 100644 spec/serializers/activity_pub/releases_actor_serializer_spec.rb create mode 100644 spec/serializers/activity_pub/releases_outbox_serializer_spec.rb create mode 100644 spec/serializers/activity_pub/user_entity_spec.rb diff --git a/app/controllers/projects/releases_controller.rb b/app/controllers/projects/releases_controller.rb index 6a6a47bc33d948..115fd31db15f89 100644 --- a/app/controllers/projects/releases_controller.rb +++ b/app/controllers/projects/releases_controller.rb @@ -24,6 +24,23 @@ def index format.json do render json: ReleaseSerializer.new.represent(releases) end + format.apub do + opts = { + inbox: nil, + outbox: outbox_project_releases_url(@project) + } + + render json: ActivityPub::ReleasesActorSerializer.new.represent(@project, opts) + end + end + end + + def outbox + respond_to do |format| + format.apub do + serializer = ActivityPub::ReleasesOutboxSerializer.new.with_pagination(request, response) + render json: serializer.represent(releases, url: outbox_project_releases_url(@project)) + end end end diff --git a/app/serializers/activity_pub/README.md b/app/serializers/activity_pub/README.md new file mode 100644 index 00000000000000..289893183bd52a --- /dev/null +++ b/app/serializers/activity_pub/README.md @@ -0,0 +1,90 @@ +# ActivityPub serializers + +Those serializers implement half the core of ActivityPub support: they're +all about ActivityStreams, the message format used by ActivityPub. + +To sum it up: + +* [ActivityPub](https://www.w3.org/TR/activitypub/) defines the HTTP + requests happening to implement federation. +* [ActivityStreams](https://www.w3.org/TR/activitystreams-core/) defines the + format of the json messages exchanged by the users of the protocol. +* [Activity Vocabulary](https://www.w3.org/TR/activitystreams-vocabulary/) + defines the various messages recognized by default. + +The last two are what we're interested in serializers: the controllers and +workers handles requests, the serializers format the ActivityStreams +messages. + +We're not going to repeat the whole specification documents here (you're +encouraged to read them, they're actually quite concise and informative), +instead this document is your quick how-to to implement a new feature. + +## Making your serializer ActivityPub ready + +To leverage the features doing most of the formatting for you, your +serializer should inherit from `ActivityPub::ActivityStreamsSerializer`. + +It's used as usual by calling the `#represent` method, although it will +require you provide some options. + +There are two types of serializer you will implement for ActivityPub: + +* Actor serializers : the "profile page" of an actor, listing its endpoints + (like the inbox and outbox adresses) and basic information. +* Outbox serializers : the feed of activities for this actor, which is a + paginated list (you must mix in `WithPagination`). + +### Actor serializers + +When you call the represent method of an actor serializer, you should +provide its outbox and inbox addresses, eg: + +```ruby +opts = { + inbox: inbox_project_releases_url(project), + outbox: outbox_project_releases_url(project) +} + +render json: ActivityPub::ReleasesActorSerializer.new.represent(project, opts) +``` + +`outbox` is the endpoint when to find the activities feed for this +actor, `inbox` is the one where to POST to subscribe to the feed. + +### Outbox serializer + +When you call the represent method of an outbox serializer, you should +provide the outbox url, eg: + +```ruby +opts = { + url: outbox_project_releases_url(releases) +} + +serializer = ActivityPub::ReleasesOutboxSerializer.new.with_pagination(request, response) +render json: serializer.represent(releases, opts) +``` + +The reason for that is that the default document for an ActivityPub +pagination is the index of the collection, linking to other pages by +offering the same url with a `page` parameter, so we need that base url to +be provided. + +## Required fields + +Each resource serialized (included other resources embedded) must provide +an `id` and a `type` field. + +`id` is an url. It's meant to be a unique identifier for the resource, and +it must point to an existing page - ideally an actor, otherwise you can +just reference the closest actor and use an anchor, like this: + +``` +https://gitlab.com/user/project/-/releases#release-1 +``` + +`type` should be taken from ActivityStream core vocabulary ([here](https://www.w3.org/TR/activitystreams-vocabulary/#activity-types), [here](https://www.w3.org/TR/activitystreams-vocabulary/#actor-types) and [here](https://www.w3.org/TR/activitystreams-vocabulary/#object-types)). + +Each type type will have its own required and allowed fields, which are +documented when following type links on those pages. diff --git a/app/serializers/activity_pub/activity_streams_serializer.rb b/app/serializers/activity_pub/activity_streams_serializer.rb new file mode 100644 index 00000000000000..b1e6a60e2233ee --- /dev/null +++ b/app/serializers/activity_pub/activity_streams_serializer.rb @@ -0,0 +1,84 @@ +# frozen_string_literal: true + +module ActivityPub + class ActivityStreamsSerializer < ::BaseSerializer + MissingIdentifierError = Class.new(StandardError) + MissingTypeError = Class.new(StandardError) + MissingOutboxError = Class.new(StandardError) + MissingCollectionUrlError = Class.new(StandardError) + + alias_method :base_represent, :represent + + def represent(resource, opts = {}, entity_class = nil) + response = if respond_to?(:paginated?) && paginated? + represent_paginated(resource, opts, entity_class) + else + represent_whole(resource, opts, entity_class) + end + + validate_response(HashWithIndifferentAccess.new(response)) + end + + private + + def validate_response(response) + unless response[:id].present? + raise MissingIdentifierError, "The serializer does not provide the mandatory 'id' field." + end + + unless response[:type].present? + raise MissingTypeError, "The serializer does not provide the mandatory 'type' field." + end + + response + end + + def represent_whole(resource, opts, entity_class) + raise MissingOutboxError, 'Please provide an :outbox option for this actor' unless opts[:outbox].present? + + serialized = base_represent(resource, opts, entity_class) + + { + :@context => "https://www.w3.org/ns/activitystreams", + inbox: opts[:inbox], + outbox: opts[:outbox] + }.merge(serialized) + end + + def represent_paginated(resources, opts, entity_class) + raise MissingCollectionUrlError, 'Please provide a :url option for this collection' unless opts[:url].present? + + if paginator.params['page'].present? + represent_page(resources, resources.current_page, opts, entity_class) + else + represent_pagination_index(resources, opts[:url]) + end + end + + def represent_page(resources, page, opts, entity_class) + opts[:page] = page + serialized = base_represent(resources, opts, entity_class) + + { + :@context => 'https://www.w3.org/ns/activitystreams', + type: 'OrderedCollectionPage', + id: "#{opts[:url]}?page=#{page}", + prev: page > 1 ? "#{opts[:url]}?page=#{page - 1}" : nil, + next: page < resources.total_pages ? "#{opts[:url]}?page=#{page + 1}" : nil, + partOf: opts[:url], + orderedItems: serialized + } + end + + def represent_pagination_index(resources, url) + { + :@context => 'https://www.w3.org/ns/activitystreams', + type: 'OrderedCollection', + id: url, + totalItems: resources.total_count, + first: "#{url}?page=1", + last: "#{url}?page=#{resources.total_pages}" + } + end + end +end diff --git a/app/serializers/activity_pub/project_entity.rb b/app/serializers/activity_pub/project_entity.rb new file mode 100644 index 00000000000000..02ed0cdc0479e0 --- /dev/null +++ b/app/serializers/activity_pub/project_entity.rb @@ -0,0 +1,23 @@ +# frozen_string_literal: true + +module ActivityPub + class ProjectEntity < Grape::Entity + include RequestAwareEntity + + expose :id do |project| + project_url(project) + end + + expose :type do |*| + "Application" + end + + expose :name + + expose :description, as: :summary + + expose :url do |project| + project_url(project) + end + end +end diff --git a/app/serializers/activity_pub/release_entity.rb b/app/serializers/activity_pub/release_entity.rb new file mode 100644 index 00000000000000..9e3e5397034c42 --- /dev/null +++ b/app/serializers/activity_pub/release_entity.rb @@ -0,0 +1,40 @@ +# frozen_string_literal: true + +module ActivityPub + class ReleaseEntity < Grape::Entity + include RequestAwareEntity + + expose :id do |release, opts| + "#{opts[:url]}##{release.tag}" + end + + expose :type do |*| + "Create" + end + + expose :to do |*| + 'https://www.w3.org/ns/activitystreams#Public' + end + + expose :author, as: :actor, using: UserEntity + + expose :object do + expose :id do |release| + project_release_url(release.project, release) + end + + expose :type do |*| + "Application" + end + + expose :name + + expose :url do |release| + project_release_url(release.project, release) + end + + expose :description, as: :content + expose :project, as: :context, using: ProjectEntity + end + end +end diff --git a/app/serializers/activity_pub/releases_actor_entity.rb b/app/serializers/activity_pub/releases_actor_entity.rb new file mode 100644 index 00000000000000..c52741c73a5757 --- /dev/null +++ b/app/serializers/activity_pub/releases_actor_entity.rb @@ -0,0 +1,29 @@ +# frozen_string_literal: true + +module ActivityPub + class ReleasesActorEntity < Grape::Entity + include RequestAwareEntity + + expose :id do |project| + project_releases_url(project) + end + + expose :type do |*| + "Application" + end + + expose :path, as: :preferredUsername do |project| + "#{project.path}-releases" + end + + expose :name do |project| + "#{_('Releases')} - #{project.name}" + end + + expose :description, as: :content + + expose nil, using: ProjectEntity, as: :context do |project| + project + end + end +end diff --git a/app/serializers/activity_pub/releases_actor_serializer.rb b/app/serializers/activity_pub/releases_actor_serializer.rb new file mode 100644 index 00000000000000..5bae83f2dc7136 --- /dev/null +++ b/app/serializers/activity_pub/releases_actor_serializer.rb @@ -0,0 +1,7 @@ +# frozen_string_literal: true + +module ActivityPub + class ReleasesActorSerializer < ActivityStreamsSerializer + entity ReleasesActorEntity + end +end diff --git a/app/serializers/activity_pub/releases_outbox_serializer.rb b/app/serializers/activity_pub/releases_outbox_serializer.rb new file mode 100644 index 00000000000000..b6d4e633fb01de --- /dev/null +++ b/app/serializers/activity_pub/releases_outbox_serializer.rb @@ -0,0 +1,9 @@ +# frozen_string_literal: true + +module ActivityPub + class ReleasesOutboxSerializer < ActivityStreamsSerializer + include WithPagination + + entity ReleaseEntity + end +end diff --git a/app/serializers/activity_pub/user_entity.rb b/app/serializers/activity_pub/user_entity.rb new file mode 100644 index 00000000000000..bd0886db5b2f43 --- /dev/null +++ b/app/serializers/activity_pub/user_entity.rb @@ -0,0 +1,22 @@ +# frozen_string_literal: true + +module ActivityPub + class UserEntity < Grape::Entity + include RequestAwareEntity + + expose :id do |user| + user_url(user) + end + + expose :type do |*| + 'Person' + end + + expose :name + expose :username, as: :preferredUsername + + expose :url do |user| + user_url(user) + end + end +end diff --git a/config/initializers/mime_types.rb b/config/initializers/mime_types.rb index 5329232cc927eb..6c5ca3b0ee3273 100644 --- a/config/initializers/mime_types.rb +++ b/config/initializers/mime_types.rb @@ -18,4 +18,6 @@ Mime::Type.unregister :json Mime::Type.register 'application/json', :json, [LfsRequest::CONTENT_TYPE, 'application/json'] +Mime::Type.register "application/activity+json", :apub, ["application/ld+json"] + Mime::Type.register 'image/x-icon', :ico diff --git a/config/routes/project.rb b/config/routes/project.rb index a4b513fc33ea45..79e23002bb341d 100644 --- a/config/routes/project.rb +++ b/config/routes/project.rb @@ -231,6 +231,10 @@ resources :evidences, only: [:show] end end + + collection do + get :outbox, constraints: { format: :apub } + end end resources :starrers, only: [:index] diff --git a/spec/controllers/projects/releases_controller_spec.rb b/spec/controllers/projects/releases_controller_spec.rb index 35ac7ed0aa48e3..76d52f72dfbf94 100644 --- a/spec/controllers/projects/releases_controller_spec.rb +++ b/spec/controllers/projects/releases_controller_spec.rb @@ -2,7 +2,7 @@ require 'spec_helper' -RSpec.describe Projects::ReleasesController do +RSpec.describe Projects::ReleasesController, feature_category: :groups_and_projects do include AccessMatchersForController let!(:project) { create(:project, :repository, :public) } @@ -77,6 +77,16 @@ end end + shared_examples_for 'ActivityPub response' do + it 'returns an application/activity+json content_type' do + expect(response.media_type).to eq 'application/activity+json' + end + + it 'is formated as an ActivityStream document' do + expect(json_response['@context']).to eq 'https://www.w3.org/ns/activitystreams' + end + end + describe 'GET #index' do context 'as html' do let(:format) { :html } @@ -127,6 +137,30 @@ end end end + + context 'as ActivityPub' do + before do + get_index + end + + let(:format) { :apub } + + it_behaves_like 'ActivityPub response' + + it "returns the project's releases actor profile data" do + expect(json_response['id']).to include project_releases_path(project) + end + + it_behaves_like 'common access controls' + + context 'when the project is private and the user is not logged in' do + let(:project) { private_project } + + it 'returns a redirect' do + expect(response).to have_gitlab_http_status(:redirect) + end + end + end end describe 'GET #new' do @@ -228,6 +262,40 @@ end end + describe 'GET #outbox' do + before do + get :outbox, params: { namespace_id: project.namespace, project_id: project, format: format, page: page } + end + + context 'as ActivityPub' do + let(:format) { :apub } + + context 'with no page parameter' do + let(:page) { nil } + + it_behaves_like 'ActivityPub response' + + it "returns the project's releases collection index" do + expect(json_response['id']).to include outbox_project_releases_path(project) + expect(json_response['totalItems']).to eq 2 + end + end + + context 'with a page parameter' do + let(:page) { 1 } + + it_behaves_like 'ActivityPub response' + + it "returns the project's releases list" do + expect(json_response['id']).to include outbox_project_releases_path(project, page: 1) + + names = json_response['orderedItems'].map { |release| release['object']['name'] } + expect(names).to match_array([release_2.name, release_1.name]) + end + end + end + end + describe 'GET #latest_permalink' do # Uses default order_by=released_at parameter. subject do diff --git a/spec/serializers/activity_pub/activity_streams_serializer_spec.rb b/spec/serializers/activity_pub/activity_streams_serializer_spec.rb new file mode 100644 index 00000000000000..fd10851b96ec9d --- /dev/null +++ b/spec/serializers/activity_pub/activity_streams_serializer_spec.rb @@ -0,0 +1,163 @@ +# frozen_string_literal: true + +require 'spec_helper' + +RSpec.describe ActivityPub::ActivityStreamsSerializer, feature_category: :integrations do + let(:implementer_class) do + Class.new(described_class) do + include WithPagination + end + end + + let(:entity_class) do + Class.new(Grape::Entity) do + expose :id do |*| + 'https://example.com/unique/url' + end + + expose :type do |*| + 'Person' + end + + expose :name do |*| + 'Alice' + end + end + end + + shared_examples_for 'ActivityStreams document' do + it 'belongs to the ActivityStreams namespace' do + expect(subject['@context']).to eq 'https://www.w3.org/ns/activitystreams' + end + + it 'has a unique identifier' do + expect(subject).to have_key 'id' + end + + it 'has a type' do + expect(subject).to have_key 'type' + end + end + + before do + implementer_class.entity entity_class + end + + context 'when the serializer is not paginated' do + let(:resource) { build_stubbed(:release) } + let(:outbox_url) { 'https://example.com/unique/url/outbox' } + + context 'with a valid represented entity' do + subject { implementer_class.new.represent(resource, outbox: outbox_url) } + + it_behaves_like 'ActivityStreams document' + + it 'exposes an outbox' do + expect(subject['outbox']).to eq 'https://example.com/unique/url/outbox' + end + + it 'includes serialized data' do + expect(subject['name']).to eq 'Alice' + end + end + + context 'when the represented entity provides no identifier' do + subject { implementer_class.new.represent(resource, outbox: outbox_url) } + + before do + allow(entity_class).to receive(:represent).and_return({ type: 'Person' }) + end + + it 'raises an exception' do + expect { subject }.to raise_error(ActivityPub::ActivityStreamsSerializer::MissingIdentifierError) + end + end + + context 'when the represented entity provides no type' do + subject { implementer_class.new.represent(resource, outbox: outbox_url) } + + before do + allow(entity_class).to receive(:represent).and_return({ id: 'https://example.com/unique/url' }) + end + + it 'raises an exception' do + expect { subject }.to raise_error(ActivityPub::ActivityStreamsSerializer::MissingTypeError) + end + end + + context 'when the caller provides no outbox parameter' do + subject { implementer_class.new.represent(resource) } + + it 'raises an exception' do + expect { subject }.to raise_error(ActivityPub::ActivityStreamsSerializer::MissingOutboxError) + end + end + end + + context 'when the serializer is paginated' do + let(:resources) { build_stubbed_list(:release, 3) } + let(:request) { ActionDispatch::Request.new({}) } + let(:response) { ActionDispatch::Response.new } + let(:url) { 'https://example.com/resource/url' } + let(:decorated) { implementer_class.new.with_pagination(request, response) } + + before do + allow(resources).to receive(:page).and_return(resources) + allow(resources).to receive(:per).and_return(resources) + allow(resources).to receive(:current_page).and_return(2) + allow(resources).to receive(:total_pages).and_return(3) + allow(resources).to receive(:total_count).and_return(10) + allow(decorated.paginator).to receive(:paginate).and_return(resources) + end + + context 'with a valid call' do + context 'when no page parameter is provided' do + subject { decorated.represent(resources, url: url) } + + it_behaves_like 'ActivityStreams document' + + it 'is an index document for the pagination' do + expect(subject['type']).to eq 'OrderedCollection' + end + + it 'contains the total amount of items' do + expect(subject['totalItems']).to eq 10 + end + + it 'contains links to first and last page' do + expect(subject['first']).to be_present + expect(subject['last']).to be_present + end + end + + context 'when a page parameter is provided' do + subject { decorated.represent(resources, url: url) } + + before do + request.query_parameters['page'] = 2 + request.query_parameters['per_page'] = 1 + end + + it_behaves_like 'ActivityStreams document' + + it 'is a page document' do + expect(subject['type']).to eq 'OrderedCollectionPage' + end + + it 'contains navigation links' do + expect(subject['prev']).to be_present + expect(subject['next']).to be_present + expect(subject['partOf']).to be_present + end + end + end + + context 'when the caller provides no url' do + subject { decorated.represent(resources) } + + it 'raises an exception' do + expect { subject }.to raise_error(ActivityPub::ActivityStreamsSerializer::MissingCollectionUrlError) + end + end + end +end diff --git a/spec/serializers/activity_pub/project_entity_spec.rb b/spec/serializers/activity_pub/project_entity_spec.rb new file mode 100644 index 00000000000000..f273acace737f0 --- /dev/null +++ b/spec/serializers/activity_pub/project_entity_spec.rb @@ -0,0 +1,32 @@ +# frozen_string_literal: true + +require 'spec_helper' + +RSpec.describe ActivityPub::ProjectEntity, feature_category: :groups_and_projects do + let(:project) { build_stubbed(:project, name: 'Fooify', path: 'fooify') } + let(:entity) { described_class.new(project) } + + context 'as json' do + subject { entity.as_json } + + it 'has releases page as id' do + expect(subject[:id]).to match(%r{/fooify$}) + end + + it 'is an Application actor' do + expect(subject[:type]).to eq 'Application' + end + + it 'provides project name' do + expect(subject[:name]).to eq project.name + end + + it 'provides a description of the project' do + expect(subject[:summary]).to eq project.description + end + + it 'provides an url for web content' do + expect(subject[:url]).to match(%r{/fooify$}) + end + end +end diff --git a/spec/serializers/activity_pub/release_entity_spec.rb b/spec/serializers/activity_pub/release_entity_spec.rb new file mode 100644 index 00000000000000..a473fbcc2bdbfe --- /dev/null +++ b/spec/serializers/activity_pub/release_entity_spec.rb @@ -0,0 +1,48 @@ +# frozen_string_literal: true + +require 'spec_helper' + +RSpec.describe ActivityPub::ReleaseEntity, feature_category: :groups_and_projects do + let(:release) { build_stubbed(:release) } + let(:entity) { described_class.new(release, url: '/outbox') } + + context 'as json' do + subject { entity.as_json } + + it 'has tag as id' do + expect(subject[:id]).to match(/##{release.tag}$/) + end + + it 'is a Create activity' do + expect(subject[:type]).to eq 'Create' + end + + it 'is addressed to public' do + expect(subject[:to]).to eq 'https://www.w3.org/ns/activitystreams#Public' + end + + it 'has an author' do + expect(subject[:actor]).to include(:id, :type, :name, :preferredUsername, :url) + end + + it 'embeds the release as an Application actor' do + expect(subject[:object][:type]).to eq 'Application' + end + + it 'provides release name' do + expect(subject[:object][:name]).to eq release.name + end + + it 'provides release description' do + expect(subject[:object][:content]).to eq release.description + end + + it 'provides an url for web content' do + expect(subject[:object][:url]).to include release.tag + end + + it 'provides project data as context' do + expect(subject[:object][:context]).to include(:id, :type, :name, :summary, :url) + end + end +end diff --git a/spec/serializers/activity_pub/releases_actor_entity_spec.rb b/spec/serializers/activity_pub/releases_actor_entity_spec.rb new file mode 100644 index 00000000000000..fe3889688671eb --- /dev/null +++ b/spec/serializers/activity_pub/releases_actor_entity_spec.rb @@ -0,0 +1,39 @@ +# frozen_string_literal: true + +require 'spec_helper' + +RSpec.describe ActivityPub::ReleasesActorEntity, feature_category: :groups_and_projects do + let(:project) { build_stubbed(:project, name: 'Fooify', path: 'fooify') } + let(:releases) { build_stubbed_list(:release, 3, project: project) } + + let(:entity) { described_class.new(project) } + + context 'as json' do + subject { entity.as_json } + + it 'has releases page as id' do + expect(subject[:id]).to include "/fooify/-/releases" + end + + it 'is an Application actor' do + expect(subject[:type]).to eq 'Application' + end + + it 'has a recognizable username' do + expect(subject[:preferredUsername]).to include 'releases' + end + + it 'has a recognizable full name' do + expect(subject[:name]).to eq 'Releases - Fooify' + end + + it 'provides a description of the project' do + expect(subject[:content]).to eq project.description + end + + it 'provides project data as context' do + expect(subject[:context]).to include(:id, :type, :name, :summary, :url) + expect(subject[:context][:id]).to match(%r{/fooify$}) + end + end +end diff --git a/spec/serializers/activity_pub/releases_actor_serializer_spec.rb b/spec/serializers/activity_pub/releases_actor_serializer_spec.rb new file mode 100644 index 00000000000000..bc754eabe5c884 --- /dev/null +++ b/spec/serializers/activity_pub/releases_actor_serializer_spec.rb @@ -0,0 +1,16 @@ +# frozen_string_literal: true + +require 'spec_helper' + +RSpec.describe ActivityPub::ReleasesActorSerializer, feature_category: :groups_and_projects do + let(:project) { build_stubbed(:project, name: 'Fooify', path: 'fooify') } + let(:releases) { build_stubbed_list(:release, 3, project: project) } + + context 'when there is a single object provided' do + subject { described_class.new.represent(project, outbox: '/outbox') } + + it 'serializes the actor attributes' do + expect(subject).to include(:id, :type, :preferredUsername, :name, :content, :context) + end + end +end diff --git a/spec/serializers/activity_pub/releases_outbox_serializer_spec.rb b/spec/serializers/activity_pub/releases_outbox_serializer_spec.rb new file mode 100644 index 00000000000000..57328876df0a96 --- /dev/null +++ b/spec/serializers/activity_pub/releases_outbox_serializer_spec.rb @@ -0,0 +1,38 @@ +# frozen_string_literal: true + +require 'spec_helper' + +RSpec.describe ActivityPub::ReleasesOutboxSerializer, feature_category: :groups_and_projects do + let(:decorated) { described_class.new.with_pagination(request, response) } + + let(:project) { build_stubbed(:project, name: 'Fooify', path: 'fooify') } + let(:releases) { build_stubbed_list(:release, 3, project: project) } + + before do + allow(releases).to receive(:page).and_return(releases) + allow(releases).to receive(:per).and_return(releases) + allow(releases).to receive(:current_page).and_return(1) + allow(releases).to receive(:total_pages).and_return(1) + allow(decorated.paginator).to receive(:paginate).and_return(releases) + end + + context 'when there is a list of objects provided' do + subject { decorated.represent(releases, url: '/outbox') } + + let(:request) { ActionDispatch::Request.new({}) } + let(:response) { ActionDispatch::Response.new } + + before do + request.query_parameters['page'] = 1 + end + + it 'is a OrderedCollection document' do + expect(subject[:type]).to eq 'OrderedCollectionPage' + end + + it 'serializes the releases' do + expect(subject[:orderedItems].count).to eq 3 + expect(subject[:orderedItems][0]).to include(:id, :type, :to, :actor, :object) + end + end +end diff --git a/spec/serializers/activity_pub/user_entity_spec.rb b/spec/serializers/activity_pub/user_entity_spec.rb new file mode 100644 index 00000000000000..d9ab7a11ecffcb --- /dev/null +++ b/spec/serializers/activity_pub/user_entity_spec.rb @@ -0,0 +1,28 @@ +# frozen_string_literal: true + +require 'spec_helper' + +RSpec.describe ActivityPub::UserEntity, feature_category: :user_profile do + let(:user) { build_stubbed(:user, name: 'Alice', username: 'alice') } + let(:entity) { described_class.new(user) } + + context 'as json' do + subject { entity.as_json } + + it 'has releases page as id' do + expect(subject[:id]).to match(%r{/alice$}) + end + + it 'is a Person actor' do + expect(subject[:type]).to eq 'Person' + end + + it 'provides project name' do + expect(subject[:name]).to eq 'Alice' + end + + it 'provides an url for web content' do + expect(subject[:url]).to match(%r{/alice$}) + end + end +end -- GitLab From 38fcd96246d5c69eec5155d8220b9005afba1924 Mon Sep 17 00:00:00 2001 From: Olivier El Mekki Date: Wed, 2 Aug 2023 21:36:56 +0200 Subject: [PATCH 02/10] REFACTOR remove the need for `url` option parameter We can do fast computations to remove the need for that option, which may confuse implementers wondering what they must provide. --- .../projects/releases_controller.rb | 2 +- app/serializers/activity_pub/README.md | 14 ++-- .../activity_streams_serializer.rb | 30 +++++---- .../projects/releases_controller_spec.rb | 2 +- .../activity_streams_serializer_spec.rb | 66 +++++++++---------- .../releases_outbox_serializer_spec.rb | 6 +- 6 files changed, 55 insertions(+), 65 deletions(-) diff --git a/app/controllers/projects/releases_controller.rb b/app/controllers/projects/releases_controller.rb index 115fd31db15f89..331a937feb106a 100644 --- a/app/controllers/projects/releases_controller.rb +++ b/app/controllers/projects/releases_controller.rb @@ -39,7 +39,7 @@ def outbox respond_to do |format| format.apub do serializer = ActivityPub::ReleasesOutboxSerializer.new.with_pagination(request, response) - render json: serializer.represent(releases, url: outbox_project_releases_url(@project)) + render json: serializer.represent(releases) end end end diff --git a/app/serializers/activity_pub/README.md b/app/serializers/activity_pub/README.md index 289893183bd52a..c862a3fb54548d 100644 --- a/app/serializers/activity_pub/README.md +++ b/app/serializers/activity_pub/README.md @@ -55,21 +55,15 @@ actor, `inbox` is the one where to POST to subscribe to the feed. ### Outbox serializer When you call the represent method of an outbox serializer, you should -provide the outbox url, eg: +use the `WithPagination` mixin, eg: ```ruby -opts = { - url: outbox_project_releases_url(releases) -} - serializer = ActivityPub::ReleasesOutboxSerializer.new.with_pagination(request, response) -render json: serializer.represent(releases, opts) +render json: serializer.represent(releases) ``` -The reason for that is that the default document for an ActivityPub -pagination is the index of the collection, linking to other pages by -offering the same url with a `page` parameter, so we need that base url to -be provided. +This will automatically convert the response to an `OrderedCollection` +ActivityPub type, with all the correct fields. ## Required fields diff --git a/app/serializers/activity_pub/activity_streams_serializer.rb b/app/serializers/activity_pub/activity_streams_serializer.rb index b1e6a60e2233ee..39caa4a6d10e43 100644 --- a/app/serializers/activity_pub/activity_streams_serializer.rb +++ b/app/serializers/activity_pub/activity_streams_serializer.rb @@ -5,7 +5,6 @@ class ActivityStreamsSerializer < ::BaseSerializer MissingIdentifierError = Class.new(StandardError) MissingTypeError = Class.new(StandardError) MissingOutboxError = Class.new(StandardError) - MissingCollectionUrlError = Class.new(StandardError) alias_method :base_represent, :represent @@ -46,12 +45,10 @@ def represent_whole(resource, opts, entity_class) end def represent_paginated(resources, opts, entity_class) - raise MissingCollectionUrlError, 'Please provide a :url option for this collection' unless opts[:url].present? - if paginator.params['page'].present? represent_page(resources, resources.current_page, opts, entity_class) else - represent_pagination_index(resources, opts[:url]) + represent_pagination_index(resources) end end @@ -62,23 +59,32 @@ def represent_page(resources, page, opts, entity_class) { :@context => 'https://www.w3.org/ns/activitystreams', type: 'OrderedCollectionPage', - id: "#{opts[:url]}?page=#{page}", - prev: page > 1 ? "#{opts[:url]}?page=#{page - 1}" : nil, - next: page < resources.total_pages ? "#{opts[:url]}?page=#{page + 1}" : nil, - partOf: opts[:url], + id: collection_url(page), + prev: page > 1 ? collection_url(page - 1) : nil, + next: page < resources.total_pages ? collection_url(page + 1) : nil, + partOf: collection_url, orderedItems: serialized } end - def represent_pagination_index(resources, url) + def represent_pagination_index(resources) { :@context => 'https://www.w3.org/ns/activitystreams', type: 'OrderedCollection', - id: url, + id: collection_url, totalItems: resources.total_count, - first: "#{url}?page=1", - last: "#{url}?page=#{resources.total_pages}" + first: collection_url(1), + last: collection_url(resources.total_pages) } end + + def collection_url(page = nil) + uri = URI.parse(paginator.request.url) + uri.query ||= "" + parts = uri.query.split('&').reject { |part| part =~ /^page=/ } + parts << "page=#{page}" if page + uri.query = parts.join('&') + uri.to_s.sub(/\?$/, '') + end end end diff --git a/spec/controllers/projects/releases_controller_spec.rb b/spec/controllers/projects/releases_controller_spec.rb index 76d52f72dfbf94..2fb13075b4f2e0 100644 --- a/spec/controllers/projects/releases_controller_spec.rb +++ b/spec/controllers/projects/releases_controller_spec.rb @@ -287,7 +287,7 @@ it_behaves_like 'ActivityPub response' it "returns the project's releases list" do - expect(json_response['id']).to include outbox_project_releases_path(project, page: 1) + expect(json_response['id']).to include outbox_project_releases_path(project, page: 1, format: :apub) names = json_response['orderedItems'].map { |release| release['object']['name'] } expect(names).to match_array([release_2.name, release_1.name]) diff --git a/spec/serializers/activity_pub/activity_streams_serializer_spec.rb b/spec/serializers/activity_pub/activity_streams_serializer_spec.rb index fd10851b96ec9d..c74beba7a81265 100644 --- a/spec/serializers/activity_pub/activity_streams_serializer_spec.rb +++ b/spec/serializers/activity_pub/activity_streams_serializer_spec.rb @@ -96,7 +96,7 @@ context 'when the serializer is paginated' do let(:resources) { build_stubbed_list(:release, 3) } - let(:request) { ActionDispatch::Request.new({}) } + let(:request) { ActionDispatch::Request.new(request_data) } let(:response) { ActionDispatch::Response.new } let(:url) { 'https://example.com/resource/url' } let(:decorated) { implementer_class.new.with_pagination(request, response) } @@ -110,53 +110,47 @@ allow(decorated.paginator).to receive(:paginate).and_return(resources) end - context 'with a valid call' do - context 'when no page parameter is provided' do - subject { decorated.represent(resources, url: url) } + context 'when no page parameter is provided' do + subject { decorated.represent(resources) } - it_behaves_like 'ActivityStreams document' + let(:request_data) do + { "rack.url_scheme" => "https", "HTTP_HOST" => "example.com", "PATH_INFO" => '/resource/url' } + end - it 'is an index document for the pagination' do - expect(subject['type']).to eq 'OrderedCollection' - end + it_behaves_like 'ActivityStreams document' - it 'contains the total amount of items' do - expect(subject['totalItems']).to eq 10 - end + it 'is an index document for the pagination' do + expect(subject['type']).to eq 'OrderedCollection' + end - it 'contains links to first and last page' do - expect(subject['first']).to be_present - expect(subject['last']).to be_present - end + it 'contains the total amount of items' do + expect(subject['totalItems']).to eq 10 end - context 'when a page parameter is provided' do - subject { decorated.represent(resources, url: url) } + it 'contains links to first and last page' do + expect(subject['first']).to eq "#{url}?page=1" + expect(subject['last']).to eq "#{url}?page=3" + end + end - before do - request.query_parameters['page'] = 2 - request.query_parameters['per_page'] = 1 - end + context 'when a page parameter is provided' do + subject { decorated.represent(resources) } - it_behaves_like 'ActivityStreams document' + let(:request_data) do + { 'rack.url_scheme' => 'https', 'HTTP_HOST' => 'example.com', 'PATH_INFO' => '/resource/url', + 'QUERY_STRING' => 'page=2&per_page=1' } + end - it 'is a page document' do - expect(subject['type']).to eq 'OrderedCollectionPage' - end + it_behaves_like 'ActivityStreams document' - it 'contains navigation links' do - expect(subject['prev']).to be_present - expect(subject['next']).to be_present - expect(subject['partOf']).to be_present - end + it 'is a page document' do + expect(subject['type']).to eq 'OrderedCollectionPage' end - end - context 'when the caller provides no url' do - subject { decorated.represent(resources) } - - it 'raises an exception' do - expect { subject }.to raise_error(ActivityPub::ActivityStreamsSerializer::MissingCollectionUrlError) + it 'contains navigation links' do + expect(subject['prev']).to be_present + expect(subject['next']).to be_present + expect(subject['partOf']).to be_present end end end diff --git a/spec/serializers/activity_pub/releases_outbox_serializer_spec.rb b/spec/serializers/activity_pub/releases_outbox_serializer_spec.rb index 57328876df0a96..606b0130e0fb90 100644 --- a/spec/serializers/activity_pub/releases_outbox_serializer_spec.rb +++ b/spec/serializers/activity_pub/releases_outbox_serializer_spec.rb @@ -19,13 +19,9 @@ context 'when there is a list of objects provided' do subject { decorated.represent(releases, url: '/outbox') } - let(:request) { ActionDispatch::Request.new({}) } + let(:request) { ActionDispatch::Request.new({ 'QUERY_STRING' => 'page=1' }) } let(:response) { ActionDispatch::Response.new } - before do - request.query_parameters['page'] = 1 - end - it 'is a OrderedCollection document' do expect(subject[:type]).to eq 'OrderedCollectionPage' end -- GitLab From de65aa4b154276e26c0d6c9470bd12f0646adad1 Mon Sep 17 00:00:00 2001 From: Olivier El Mekki Date: Thu, 3 Aug 2023 18:04:36 +0200 Subject: [PATCH 03/10] ADD documentation --- app/serializers/activity_pub/README.md | 84 -------- doc/api/activity_pub/implementing-actor.md | 120 ++++++++++++ doc/api/activity_pub/index.md | 213 +++++++++++++++++++++ 3 files changed, 333 insertions(+), 84 deletions(-) delete mode 100644 app/serializers/activity_pub/README.md create mode 100644 doc/api/activity_pub/implementing-actor.md create mode 100644 doc/api/activity_pub/index.md diff --git a/app/serializers/activity_pub/README.md b/app/serializers/activity_pub/README.md deleted file mode 100644 index c862a3fb54548d..00000000000000 --- a/app/serializers/activity_pub/README.md +++ /dev/null @@ -1,84 +0,0 @@ -# ActivityPub serializers - -Those serializers implement half the core of ActivityPub support: they're -all about ActivityStreams, the message format used by ActivityPub. - -To sum it up: - -* [ActivityPub](https://www.w3.org/TR/activitypub/) defines the HTTP - requests happening to implement federation. -* [ActivityStreams](https://www.w3.org/TR/activitystreams-core/) defines the - format of the json messages exchanged by the users of the protocol. -* [Activity Vocabulary](https://www.w3.org/TR/activitystreams-vocabulary/) - defines the various messages recognized by default. - -The last two are what we're interested in serializers: the controllers and -workers handles requests, the serializers format the ActivityStreams -messages. - -We're not going to repeat the whole specification documents here (you're -encouraged to read them, they're actually quite concise and informative), -instead this document is your quick how-to to implement a new feature. - -## Making your serializer ActivityPub ready - -To leverage the features doing most of the formatting for you, your -serializer should inherit from `ActivityPub::ActivityStreamsSerializer`. - -It's used as usual by calling the `#represent` method, although it will -require you provide some options. - -There are two types of serializer you will implement for ActivityPub: - -* Actor serializers : the "profile page" of an actor, listing its endpoints - (like the inbox and outbox adresses) and basic information. -* Outbox serializers : the feed of activities for this actor, which is a - paginated list (you must mix in `WithPagination`). - -### Actor serializers - -When you call the represent method of an actor serializer, you should -provide its outbox and inbox addresses, eg: - -```ruby -opts = { - inbox: inbox_project_releases_url(project), - outbox: outbox_project_releases_url(project) -} - -render json: ActivityPub::ReleasesActorSerializer.new.represent(project, opts) -``` - -`outbox` is the endpoint when to find the activities feed for this -actor, `inbox` is the one where to POST to subscribe to the feed. - -### Outbox serializer - -When you call the represent method of an outbox serializer, you should -use the `WithPagination` mixin, eg: - -```ruby -serializer = ActivityPub::ReleasesOutboxSerializer.new.with_pagination(request, response) -render json: serializer.represent(releases) -``` - -This will automatically convert the response to an `OrderedCollection` -ActivityPub type, with all the correct fields. - -## Required fields - -Each resource serialized (included other resources embedded) must provide -an `id` and a `type` field. - -`id` is an url. It's meant to be a unique identifier for the resource, and -it must point to an existing page - ideally an actor, otherwise you can -just reference the closest actor and use an anchor, like this: - -``` -https://gitlab.com/user/project/-/releases#release-1 -``` - -`type` should be taken from ActivityStream core vocabulary ([here](https://www.w3.org/TR/activitystreams-vocabulary/#activity-types), [here](https://www.w3.org/TR/activitystreams-vocabulary/#actor-types) and [here](https://www.w3.org/TR/activitystreams-vocabulary/#object-types)). - -Each type type will have its own required and allowed fields, which are -documented when following type links on those pages. diff --git a/doc/api/activity_pub/implementing-actor.md b/doc/api/activity_pub/implementing-actor.md new file mode 100644 index 00000000000000..569b645f72c23b --- /dev/null +++ b/doc/api/activity_pub/implementing-actor.md @@ -0,0 +1,120 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" +--- + +# Implementing an ActivityPub actor + +ActivityPub is based on three standard documents: + +- [ActivityPub](https://www.w3.org/TR/activitypub/) defines the HTTP + requests happening to implement federation. +- [ActivityStreams](https://www.w3.org/TR/activitystreams-core/) defines the + format of the JSON messages exchanged by the users of the protocol. +- [Activity Vocabulary](https://www.w3.org/TR/activitystreams-vocabulary/) + defines the various messages recognized by default. + +The first one is typically handled by controllers, while the two other are +related to what happen in serializers. + +To implement an ActivityPub actor, you must: + +- implement the profile page of the resource +- implement the outbox page +- handle incoming requests on the inbox + +All requests are made using the `application/ld+json; +profile="https://www.w3.org/ns/activitystreams"` mime-type. + +## Profile page + +Querying the profile page typically is useful for retrieving general info +about it (name, description, etc) and the urls for the inbox and the +outbox. + +To implement a profile page, you need to create an ActivityStreams +serializer in `app/serializers/activity_pub/`, making your serializer +inherit from `ActivityStreamsSerializer`. See below in the serializers +section about the mandatory fields. + +You call your serializer in your controller like this: + +```ruby +opts = { + inbox: nil, + outbox: outbox_project_releases_url(project) +} + +render json: ActivityPub::ReleasesActorSerializer.new.represent(project, opts) +``` + +`outbox` is the endpoint when to find the activities feed for this +actor, `inbox` is the one where to POST to subscribe to the feed (not +implemented yet, so you can just pass `nil`). + +## Outbox page + +The outbox is the list of activities for the resource. It's a feed for the +resource, and it allows ActivityPub clients to show public activity for +this actor without having yet subscribed to it. + +To implement an outbox page, you need to create an ActivityStreams +serializer in `app/serializers/activity_pub/`, making your serializer +inherit from `ActivityStreamsSerializer`. See below in the serializers +section about the mandatory fields. + +You call your serializer in your controller like this: + +```ruby +serializer = ActivityPub::ReleasesOutboxSerializer.new.with_pagination(request, response) +render json: serializer.represent(releases) +``` + +This will automatically convert the response to an `OrderedCollection` +ActivityPub type, with all the correct fields. + +## Inbox + +> Not yet implemented + +The inbox is where the ActivityPub compatible third-parties makes there +requests, to subscribe to the actor or send it messages. + +## ActivityStreams serializers + +The serializers implement half the core of ActivityPub support: they're all +about [ActivityStreams](https://www.w3.org/TR/activitystreams-core/), the +message format used by ActivityPub. + +To leverage the features doing most of the formatting for you, your +serializer should inherit from `ActivityPub::ActivityStreamsSerializer`. + +It's used as usual by calling the `#represent` method, although it will +require you provide `inbox` and `outbox` options as mentioned above if it +is an actor profile page. + +Each resource serialized (included other resources embedded in your +resource) must provide an `id` and a `type` field. + +`id` is a URL. It's meant to be a unique identifier for the resource, and +it must point to an existing page - ideally an actor, otherwise you can +just reference the closest actor and use an anchor, like this: + +```plaintext +https://gitlab.com/user/project/-/releases#release-1 +``` + +`type` should be taken from ActivityStreams core vocabulary +([here](https://www.w3.org/TR/activitystreams-vocabulary/#activity-types), +[here](https://www.w3.org/TR/activitystreams-vocabulary/#actor-types) and +[here](https://www.w3.org/TR/activitystreams-vocabulary/#object-types)). + +The properties you can use are all documented in [the ActivityStreams vocabulary document](https://www.w3.org/TR/activitystreams-vocabulary). +Given the type you have chosen for your resource, you will find a +`properties` list, telling you all available properties, direct or +inherited. + +It's worth noting that Mastodon adds one more property, `preferredName`, +and expects it to be set on any actor, or that actor won't be recognized by +Mastodon. diff --git a/doc/api/activity_pub/index.md b/doc/api/activity_pub/index.md new file mode 100644 index 00000000000000..534c27c971511c --- /dev/null +++ b/doc/api/activity_pub/index.md @@ -0,0 +1,213 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" +--- + +# ActivityPub + +The goal of those documents is to provide an implementation path for adding +Fediverse capabilities to GitLab. + +This page will describe the conceptual and high level point of view, while +subpages will discuss implementation in more technical depth (as in, how to +implement this in the actual rails codebase of GitLab). + +- [What](#what) + - [The Fediverse](#the-fediverse) + - [ActivityPub](#activitypub) +- [Why](#why) +- [How](#how) + +## What + +Feel free to jump to [the Why section](#why) if you already know what +ActivityPub and the Fediverse are. + +Among the push for [decentralization of the web](https://en.wikipedia.org/wiki/Decentralized_web), +several projects tried different protocols with different ideals behind their reasoning +(some examples : [Secure Scuttlebutt](https://en.wikipedia.org/wiki/Secure_Scuttlebutt) or ssb for short, +[Dat](https://en.wikipedia.org/wiki/Dat_%28software%29), +[IPFS](https://en.wikipedia.org/wiki/InterPlanetary_File_System), +[Solid](https://en.wikipedia.org/wiki/Solid_%28web_decentralization_project%29)). +But one gained traction recently : what is known as +[ActivityPub](https://en.wikipedia.org/wiki/ActivityPub), better known for +the colloquial [Fediverse](https://en.wikipedia.org/wiki/Fediverse) built +on top of it, through applications like +[Mastodon](https://en.wikipedia.org/wiki/Mastodon_%28social_network%29) +(which could be described as some sort of decentralized Facebook) or +[Lemmy](https://en.wikipedia.org/wiki/Lemmy_%28software%29) (which could be +described as some sort of decentralized Reddit). + +We think that ActivityPub has several advantages that makes it attractive +to implementers and could explain its current success: + +- **It's built on top of HTTP**. You don't need to install new software or + to tinker with TCP/UDP to implement ActivityPub, if you have a webserver + or an application that provides an HTTP API (like a rails application), + you already have everything you need. +- **It's built on top of JSON**. All communications are basically JSON + objects, which webdevelopers are already used to. This makes adoption + really easy. +- **It's a w3c standard and already have multiple implementations**. Being + piloted by the w3c is a guarantee of stability and quality work. They + have profusely demonstrated in the past through their work on HTML, CSS + or other web standards that we can build on top of their work without + the fear of it becoming deprecated or irrelevant after a few years. + +### The Fediverse + +The core ideas behind Mastodon and Lemmy is called the Fediverse. Rather +than full decentralization, those applications rely on federation, in the +sense that there still are servers and clients (so it's not p2p like ssb, +Dat and IPFS), but there is a galaxy of servers chatting with each other +instead of having central servers controlled by a single entity (like +Facebook or Reddit). + +The user signs up to one of those servers (called **instances**), and they +can then interact with users either on this instance, or on other ones. +From the perspective of the user, they access a global network, and not +only their instance. They see the articles posted on other instances, they +can comment on them, upvote them, etc. What happens behind the scene is +that their instance know where the user they reply to is hosted, and it +contacts that other instance to let them know there is a message for them - +somewhat similar to what SMTP is doing. Similarly, when an user subscribes +to any sort of feed, their instance informs the instance where the feed is +hosted of this subscription, and then that target instance will post back +messages when new activities are created (allowing for a push model rather +than a constant poll model like RSS). Of course, what was just described is +the happy path, there is moderation, validation and fault tolerance +happening all the way. + +### ActivityPub + +Behind the Fediverse is the ActivityPub protocol. It's a simple HTTP API +attempting to be as general a social network implementation as possible, +while giving options to be extendable. + +The basic idea is that an `actor` will send and receive `activities`, which +are structured JSON messages with well defined properties, but extensible +to cover any need. An actor is defined by four endpoints, which are +contacted with the `application/ld+json; +profile="https://www.w3.org/ns/activitystreams"` Accept header: + +- `GET /inbox` : used by the actor to find new activities intended for them. +- `POST /inbox` : used by instances to push new activities intended for the + actor. +- `GET /outbox` : used by anyone to read the activities created by the + actor. +- `POST /outbox` : used by the actor to publish new activities. + +Among those, Mastodon and Lemmy currently only use `POST +/inbox` and `GET /outbox`, which is the minimum needed to implement +federation : instances push new activities for the actor on the inbox, and +reading the outbox allow to read the feed of an actor. + +Additionally, Mastodon and Lemmy implement a `GET /` endpoint (with the +mentioned Accept header) which responds with general information about the +actor, like name and URL of inbox and outbox. This is not required by the +standard, but it makes discovery easier. + +It's worth mentioning that while it's the main use case, an actor does not +necessarily map to a person. Anything can be an actor : a topic, a +subreddit, a group, an event, etc. It's easy to understand for GitLab : +anything that have activities (in the sense of what GitLab means by +"activity") can be an ActivityPub actor. So this includes projects, groups, +releases, etc. In those more abstract examples, an actor can be thought of +as an actionable feed. + +And that's it. Sounds too simple to be true? That's because it is. :) +ActivityPub by itself does not cover everything that is needed to implement +the Fediverse. Most notably, are left for the implementers to figure out: + +- finding a way to deal with spam, currently covered by authorizing or + blocking other instances (colloquially referred to as "defederating") +- discovering new instances +- performing network wide searches + +## Why + +Why would a social media protocol be useful for GitLab? + +There already has been several very popular discussions around this (see +[here](https://gitlab.com/gitlab-org/gitlab/-/issues/21582), +[here](https://gitlab.com/gitlab-org/gitlab/-/issues/14116) and the epic +[here](https://gitlab.com/groups/gitlab-org/-/epics/260)). The gist of it +is: what people really want is to have one global "GitLab network" to be +able to interact between various projects without having to register on +each of their hosts. + +The ideal workflow for this would be: + +- Alice registers to her favorite GitLab instance, like + `gitlab.example.org` +- She looks for a project on a given topic, and sees Bob's project popping + up, despite Bob being on `gitlab.com`. +- She clicks the "fork" button, and the `gitlab.com/Bob/project.git` is + forked to `gitlab.example.org/Alice/project.git` +- She makes her edits, and opens a merge request, which appears in Bob's + project on `gitlab.com`. +- Alice and Bob discuss the merge request, each one from their own GitLab + instance. +- Bob can send additional commits, which are picked up by Alice's instance +- When Bob accepts the merge request, his instance picks up the code from + Alice's instance. + +In this process, ActivityPub would help in: + +- letting Bob know a fork happened +- sending the MR to Bob +- allowing Alice and Bob to discuss the MR +- letting Alice know the code has been merged + +It will _not_ help in (please open an issue if I'm wrong): + +- implementing a network wide search +- implementing cross-instance forks (don't need it anyway, Git is kind of + good at that :) ) + +Those will need to get specific implementations. + +One may wonder : why use ActivityPub here rather than "just" implementing +cross-instance merge requests in a custom way? + +There are two reasons for that: + +- **Building on top of a standard makes it easy to reach beyond GitLab**. + While the workflow presented above only mentions GitLab, building on top + of a w3c standard means it will be easy for other forges to follow GitLab + there, and build a massive Fediverse of code sharing. +- **This is an opportunity to make GitLab more social**. To prepare the + architecture for the workflow above, smaller steps can be taken, allowing + people to subscribe to activity feeds from their Fediverse social + network. Basically, anything that has a RSS feed currently could become + an ActivityPub feed. This would mean that people on Mastodon could follow + their favorite developer, project or topic from GitLab and see the news + in their feed on Mastodon, hopefully raising engagement with GitLab. + +## How + +The idea of this implementation path is not to take the fastest route to +the feature with the most value added (cross-instance merge requests), but +to go on with the smallest useful step at each iteration, making sure each step +brings something actually useful immediately. + +1. implement ActivityPub for social following + After this, the Fediverse can follow activities on GitLab instances. + 1. ActivityPub to subscribe to project releases + 1. ActivityPub to subscribe to project creation in topics + 1. ActivityPub to subscribe to project activities + 1. ActivityPub to subscribe to group activities + 1. ActivityPub to subscribe to user activities +1. **implement cross-instance search**. After this, it's possible to + discover projects on other instances. +1. **implement cross-instance fork**. After this, it's possible to fork a + project from an other instance. +1. **implement ActivityPub for cross instance discussions** . After this, + it's possible to discuss on issues and MR from an other instance. + 1. in issues + 1. in merge requests +1. **implement ActivityPub to submit cross-instance merge request**. After + this, it's possible to submit merge requests to other instances. + +For now, you can see [how to implement an ActivityPub actor](implementing-actor.md). -- GitLab From 1bab6c99852ecb4f12f1f7a8d23c603c6d658070 Mon Sep 17 00:00:00 2001 From: Olivier El Mekki Date: Tue, 29 Aug 2023 18:27:42 +0200 Subject: [PATCH 04/10] REFACTOR puts controller in ActivityPub namespace --- .../activity_pub/application_controller.rb | 28 +++++ .../projects/application_controller.rb | 23 ++++ .../projects/releases_controller.rb | 27 +++++ .../projects/releases_controller.rb | 17 --- config/initializers/mime_types.rb | 2 - config/routes.rb | 1 + config/routes/activity_pub.rb | 31 +++++ config/routes/project.rb | 4 - lib/constraints/activity_pub_constrainer.rb | 15 +++ .../projects/releases_controller_spec.rb | 110 ++++++++++++++++++ .../projects/releases_controller_spec.rb | 70 +---------- 11 files changed, 236 insertions(+), 92 deletions(-) create mode 100644 app/controllers/activity_pub/application_controller.rb create mode 100644 app/controllers/activity_pub/projects/application_controller.rb create mode 100644 app/controllers/activity_pub/projects/releases_controller.rb create mode 100644 config/routes/activity_pub.rb create mode 100644 lib/constraints/activity_pub_constrainer.rb create mode 100644 spec/controllers/activity_pub/projects/releases_controller_spec.rb diff --git a/app/controllers/activity_pub/application_controller.rb b/app/controllers/activity_pub/application_controller.rb new file mode 100644 index 00000000000000..5697eadbcdc4be --- /dev/null +++ b/app/controllers/activity_pub/application_controller.rb @@ -0,0 +1,28 @@ +# frozen_string_literal: true + +module ActivityPub + class ApplicationController < ActionController::Base + include RoutableActions + + after_action :set_content_type + + # When we send activities out, we don't know if servers we federate with + # will keep them private. So we don't publish any private data, even if + # cookies for authorized user are passed somehow. + def current_user + nil + end + + def can?(object, action, subject = :global) + Ability.allowed?(object, action, subject) + end + + def route_not_found + head :not_found + end + + def set_content_type + self.content_type = "application/activity+json" + end + end +end diff --git a/app/controllers/activity_pub/projects/application_controller.rb b/app/controllers/activity_pub/projects/application_controller.rb new file mode 100644 index 00000000000000..9d0a6bc5562319 --- /dev/null +++ b/app/controllers/activity_pub/projects/application_controller.rb @@ -0,0 +1,23 @@ +# frozen_string_literal: true + +module ActivityPub + module Projects + class ApplicationController < ::ActivityPub::ApplicationController + before_action :project + + private + + def project + return unless params[:project_id] || params[:id] + + path = File.join(params[:namespace_id], params[:project_id] || params[:id]) + + @project = find_routable!(Project, path, request.fullpath, extra_authorization_proc: auth_proc) + end + + def auth_proc + ->(project) { !project.pending_delete? } + end + end + end +end diff --git a/app/controllers/activity_pub/projects/releases_controller.rb b/app/controllers/activity_pub/projects/releases_controller.rb new file mode 100644 index 00000000000000..c36f0029ba05c5 --- /dev/null +++ b/app/controllers/activity_pub/projects/releases_controller.rb @@ -0,0 +1,27 @@ +# frozen_string_literal: true + +module ActivityPub + module Projects + class ReleasesController < ApplicationController + def index + opts = { + inbox: nil, + outbox: outbox_project_releases_url(@project) + } + + render json: ActivityPub::ReleasesActorSerializer.new.represent(@project, opts) + end + + def outbox + serializer = ActivityPub::ReleasesOutboxSerializer.new.with_pagination(request, response) + render json: serializer.represent(releases) + end + + private + + def releases(params = {}) + ReleasesFinder.new(@project, current_user, params).execute + end + end + end +end diff --git a/app/controllers/projects/releases_controller.rb b/app/controllers/projects/releases_controller.rb index 331a937feb106a..6a6a47bc33d948 100644 --- a/app/controllers/projects/releases_controller.rb +++ b/app/controllers/projects/releases_controller.rb @@ -24,23 +24,6 @@ def index format.json do render json: ReleaseSerializer.new.represent(releases) end - format.apub do - opts = { - inbox: nil, - outbox: outbox_project_releases_url(@project) - } - - render json: ActivityPub::ReleasesActorSerializer.new.represent(@project, opts) - end - end - end - - def outbox - respond_to do |format| - format.apub do - serializer = ActivityPub::ReleasesOutboxSerializer.new.with_pagination(request, response) - render json: serializer.represent(releases) - end end end diff --git a/config/initializers/mime_types.rb b/config/initializers/mime_types.rb index 6c5ca3b0ee3273..5329232cc927eb 100644 --- a/config/initializers/mime_types.rb +++ b/config/initializers/mime_types.rb @@ -18,6 +18,4 @@ Mime::Type.unregister :json Mime::Type.register 'application/json', :json, [LfsRequest::CONTENT_TYPE, 'application/json'] -Mime::Type.register "application/activity+json", :apub, ["application/ld+json"] - Mime::Type.register 'image/x-icon', :ico diff --git a/config/routes.rb b/config/routes.rb index bdc6a3930d1e40..5a99ad44a82450 100644 --- a/config/routes.rb +++ b/config/routes.rb @@ -264,6 +264,7 @@ draw :git_http draw :api + draw :activity_pub draw :customers_dot draw :sidekiq draw :help diff --git a/config/routes/activity_pub.rb b/config/routes/activity_pub.rb new file mode 100644 index 00000000000000..f400d722e7612c --- /dev/null +++ b/config/routes/activity_pub.rb @@ -0,0 +1,31 @@ +# frozen_string_literal: true + +constraints(::Constraints::ActivityPubConstrainer.new) do + scope(module: 'activity_pub') do + constraints(::Constraints::ProjectUrlConstrainer.new) do + # Emulating route structure from routes/project.rb since we want to serve + # ActivityPub content with the proper "Accept" header to the same urls. See + # project routes file for rational behind this structure. + scope( + path: '*namespace_id', + as: :namespace, + namespace_id: Gitlab::PathRegex.full_namespace_route_regex + ) do + scope( + path: ':project_id', + constraints: { project_id: Gitlab::PathRegex.project_route_regex }, + module: :projects, + as: :project + ) do + scope '-' do + resources :releases, only: :index do + collection do + get 'outbox' + end + end + end + end + end + end + end +end diff --git a/config/routes/project.rb b/config/routes/project.rb index 79e23002bb341d..a4b513fc33ea45 100644 --- a/config/routes/project.rb +++ b/config/routes/project.rb @@ -231,10 +231,6 @@ resources :evidences, only: [:show] end end - - collection do - get :outbox, constraints: { format: :apub } - end end resources :starrers, only: [:index] diff --git a/lib/constraints/activity_pub_constrainer.rb b/lib/constraints/activity_pub_constrainer.rb new file mode 100644 index 00000000000000..383de2a84ea497 --- /dev/null +++ b/lib/constraints/activity_pub_constrainer.rb @@ -0,0 +1,15 @@ +# frozen_string_literal: true + +module Constraints + class ActivityPubConstrainer + def matches?(request) + mime_types.any? { |m| request.headers['Accept'].include?(m) } + end + + private + + def mime_types + ['application/activity+json', 'application/ld+json; profile="https://www.w3.org/ns/activitystreams"'] + end + end +end diff --git a/spec/controllers/activity_pub/projects/releases_controller_spec.rb b/spec/controllers/activity_pub/projects/releases_controller_spec.rb new file mode 100644 index 00000000000000..79f87b615d1f4f --- /dev/null +++ b/spec/controllers/activity_pub/projects/releases_controller_spec.rb @@ -0,0 +1,110 @@ +# frozen_string_literal: true + +require 'spec_helper' + +RSpec.describe ActivityPub::Projects::ReleasesController, feature_category: :groups_and_projects do + include AccessMatchersForController + + let_it_be(:project) { create(:project, :repository, :public) } + let_it_be(:private_project) { create(:project, :repository, :private) } + let_it_be(:developer) { create(:user) } + let_it_be(:release_1) { create(:release, project: project, released_at: Time.zone.parse('2018-10-18')) } + let_it_be(:release_2) { create(:release, project: project, released_at: Time.zone.parse('2019-10-19')) } + + before_all do + project.add_developer(developer) + end + + shared_examples 'common access controls' do + it 'renders a 200' do + get(action, params: params) + + expect(response).to have_gitlab_http_status(:ok) + end + + context 'when the project is private' do + let(:project) { private_project } + + context 'when user is not logged in' do + it 'renders a 404' do + get(action, params: params) + + expect(response).to have_gitlab_http_status(:not_found) + end + end + + context 'when user is a developer' do + before do + sign_in(developer) + end + + it 'still renders a 404' do + get(action, params: params) + + expect(response).to have_gitlab_http_status(:not_found) + end + end + end + end + + shared_examples_for 'ActivityPub response' do + it 'returns an application/activity+json content_type' do + expect(response.media_type).to eq 'application/activity+json' + end + + it 'is formated as an ActivityStream document' do + expect(json_response['@context']).to eq 'https://www.w3.org/ns/activitystreams' + end + end + + describe 'GET #index' do + before do + get(action, params: params) + end + + let(:action) { :index } + let(:params) { { namespace_id: project.namespace, project_id: project } } + + it_behaves_like 'common access controls' + it_behaves_like 'ActivityPub response' + + it "returns the project's releases actor profile data" do + expect(json_response['id']).to include project_releases_path(project) + end + end + + describe 'GET #outbox' do + before do + get(action, params: params) + end + + let(:action) { :outbox } + let(:params) { { namespace_id: project.namespace, project_id: project, page: page } } + + context 'with no page parameter' do + let(:page) { nil } + + it_behaves_like 'common access controls' + it_behaves_like 'ActivityPub response' + + it "returns the project's releases collection index" do + expect(json_response['id']).to include outbox_project_releases_path(project) + expect(json_response['totalItems']).to eq 2 + end + end + + context 'with a page parameter' do + let(:page) { 1 } + + it_behaves_like 'common access controls' + it_behaves_like 'ActivityPub response' + + it "returns the project's releases list" do + expect(json_response['id']).to include outbox_project_releases_path(project, page: 1) + + names = json_response['orderedItems'].map { |release| release['object']['name'] } + expect(names).to match_array([release_2.name, release_1.name]) + end + end + end +end diff --git a/spec/controllers/projects/releases_controller_spec.rb b/spec/controllers/projects/releases_controller_spec.rb index 2fb13075b4f2e0..35ac7ed0aa48e3 100644 --- a/spec/controllers/projects/releases_controller_spec.rb +++ b/spec/controllers/projects/releases_controller_spec.rb @@ -2,7 +2,7 @@ require 'spec_helper' -RSpec.describe Projects::ReleasesController, feature_category: :groups_and_projects do +RSpec.describe Projects::ReleasesController do include AccessMatchersForController let!(:project) { create(:project, :repository, :public) } @@ -77,16 +77,6 @@ end end - shared_examples_for 'ActivityPub response' do - it 'returns an application/activity+json content_type' do - expect(response.media_type).to eq 'application/activity+json' - end - - it 'is formated as an ActivityStream document' do - expect(json_response['@context']).to eq 'https://www.w3.org/ns/activitystreams' - end - end - describe 'GET #index' do context 'as html' do let(:format) { :html } @@ -137,30 +127,6 @@ end end end - - context 'as ActivityPub' do - before do - get_index - end - - let(:format) { :apub } - - it_behaves_like 'ActivityPub response' - - it "returns the project's releases actor profile data" do - expect(json_response['id']).to include project_releases_path(project) - end - - it_behaves_like 'common access controls' - - context 'when the project is private and the user is not logged in' do - let(:project) { private_project } - - it 'returns a redirect' do - expect(response).to have_gitlab_http_status(:redirect) - end - end - end end describe 'GET #new' do @@ -262,40 +228,6 @@ end end - describe 'GET #outbox' do - before do - get :outbox, params: { namespace_id: project.namespace, project_id: project, format: format, page: page } - end - - context 'as ActivityPub' do - let(:format) { :apub } - - context 'with no page parameter' do - let(:page) { nil } - - it_behaves_like 'ActivityPub response' - - it "returns the project's releases collection index" do - expect(json_response['id']).to include outbox_project_releases_path(project) - expect(json_response['totalItems']).to eq 2 - end - end - - context 'with a page parameter' do - let(:page) { 1 } - - it_behaves_like 'ActivityPub response' - - it "returns the project's releases list" do - expect(json_response['id']).to include outbox_project_releases_path(project, page: 1, format: :apub) - - names = json_response['orderedItems'].map { |release| release['object']['name'] } - expect(names).to match_array([release_2.name, release_1.name]) - end - end - end - end - describe 'GET #latest_permalink' do # Uses default order_by=released_at parameter. subject do -- GitLab From 11b3e9d4f84dcb07d1474fcf72ea94bb28f9d381 Mon Sep 17 00:00:00 2001 From: Olivier El Mekki Date: Wed, 30 Aug 2023 15:54:45 +0200 Subject: [PATCH 05/10] REFACTOR inherit from top-level ApplicationController Let's not figure out and duplicate all the stack in ::ApplicationController. --- app/controllers/activity_pub/application_controller.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/app/controllers/activity_pub/application_controller.rb b/app/controllers/activity_pub/application_controller.rb index 5697eadbcdc4be..452994ab52f208 100644 --- a/app/controllers/activity_pub/application_controller.rb +++ b/app/controllers/activity_pub/application_controller.rb @@ -1,9 +1,10 @@ # frozen_string_literal: true module ActivityPub - class ApplicationController < ActionController::Base + class ApplicationController < ::ApplicationController include RoutableActions + skip_before_action :authenticate_user! after_action :set_content_type # When we send activities out, we don't know if servers we federate with -- GitLab From 2aedd9fcf538487d10f695216fbd3a92ceba1016 Mon Sep 17 00:00:00 2001 From: Patrick Cyiza Date: Thu, 31 Aug 2023 12:23:39 +0000 Subject: [PATCH 06/10] Flag the documentation page as alpha --- doc/api/activity_pub/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/api/activity_pub/index.md b/doc/api/activity_pub/index.md index 534c27c971511c..71c74f061a2629 100644 --- a/doc/api/activity_pub/index.md +++ b/doc/api/activity_pub/index.md @@ -4,7 +4,7 @@ group: Source Code info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" --- -# ActivityPub +# ActivityPub **(EXPERIMENT)** The goal of those documents is to provide an implementation path for adding Fediverse capabilities to GitLab. -- GitLab From f9384d6479e76582d3bd145fb7bcd7ee423f413a Mon Sep 17 00:00:00 2001 From: Olivier El Mekki Date: Thu, 31 Aug 2023 14:58:03 +0200 Subject: [PATCH 07/10] REMOVE documentation --- doc/api/activity_pub/implementing-actor.md | 120 ------------ doc/api/activity_pub/index.md | 213 --------------------- 2 files changed, 333 deletions(-) delete mode 100644 doc/api/activity_pub/implementing-actor.md delete mode 100644 doc/api/activity_pub/index.md diff --git a/doc/api/activity_pub/implementing-actor.md b/doc/api/activity_pub/implementing-actor.md deleted file mode 100644 index 569b645f72c23b..00000000000000 --- a/doc/api/activity_pub/implementing-actor.md +++ /dev/null @@ -1,120 +0,0 @@ ---- -stage: Create -group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" ---- - -# Implementing an ActivityPub actor - -ActivityPub is based on three standard documents: - -- [ActivityPub](https://www.w3.org/TR/activitypub/) defines the HTTP - requests happening to implement federation. -- [ActivityStreams](https://www.w3.org/TR/activitystreams-core/) defines the - format of the JSON messages exchanged by the users of the protocol. -- [Activity Vocabulary](https://www.w3.org/TR/activitystreams-vocabulary/) - defines the various messages recognized by default. - -The first one is typically handled by controllers, while the two other are -related to what happen in serializers. - -To implement an ActivityPub actor, you must: - -- implement the profile page of the resource -- implement the outbox page -- handle incoming requests on the inbox - -All requests are made using the `application/ld+json; -profile="https://www.w3.org/ns/activitystreams"` mime-type. - -## Profile page - -Querying the profile page typically is useful for retrieving general info -about it (name, description, etc) and the urls for the inbox and the -outbox. - -To implement a profile page, you need to create an ActivityStreams -serializer in `app/serializers/activity_pub/`, making your serializer -inherit from `ActivityStreamsSerializer`. See below in the serializers -section about the mandatory fields. - -You call your serializer in your controller like this: - -```ruby -opts = { - inbox: nil, - outbox: outbox_project_releases_url(project) -} - -render json: ActivityPub::ReleasesActorSerializer.new.represent(project, opts) -``` - -`outbox` is the endpoint when to find the activities feed for this -actor, `inbox` is the one where to POST to subscribe to the feed (not -implemented yet, so you can just pass `nil`). - -## Outbox page - -The outbox is the list of activities for the resource. It's a feed for the -resource, and it allows ActivityPub clients to show public activity for -this actor without having yet subscribed to it. - -To implement an outbox page, you need to create an ActivityStreams -serializer in `app/serializers/activity_pub/`, making your serializer -inherit from `ActivityStreamsSerializer`. See below in the serializers -section about the mandatory fields. - -You call your serializer in your controller like this: - -```ruby -serializer = ActivityPub::ReleasesOutboxSerializer.new.with_pagination(request, response) -render json: serializer.represent(releases) -``` - -This will automatically convert the response to an `OrderedCollection` -ActivityPub type, with all the correct fields. - -## Inbox - -> Not yet implemented - -The inbox is where the ActivityPub compatible third-parties makes there -requests, to subscribe to the actor or send it messages. - -## ActivityStreams serializers - -The serializers implement half the core of ActivityPub support: they're all -about [ActivityStreams](https://www.w3.org/TR/activitystreams-core/), the -message format used by ActivityPub. - -To leverage the features doing most of the formatting for you, your -serializer should inherit from `ActivityPub::ActivityStreamsSerializer`. - -It's used as usual by calling the `#represent` method, although it will -require you provide `inbox` and `outbox` options as mentioned above if it -is an actor profile page. - -Each resource serialized (included other resources embedded in your -resource) must provide an `id` and a `type` field. - -`id` is a URL. It's meant to be a unique identifier for the resource, and -it must point to an existing page - ideally an actor, otherwise you can -just reference the closest actor and use an anchor, like this: - -```plaintext -https://gitlab.com/user/project/-/releases#release-1 -``` - -`type` should be taken from ActivityStreams core vocabulary -([here](https://www.w3.org/TR/activitystreams-vocabulary/#activity-types), -[here](https://www.w3.org/TR/activitystreams-vocabulary/#actor-types) and -[here](https://www.w3.org/TR/activitystreams-vocabulary/#object-types)). - -The properties you can use are all documented in [the ActivityStreams vocabulary document](https://www.w3.org/TR/activitystreams-vocabulary). -Given the type you have chosen for your resource, you will find a -`properties` list, telling you all available properties, direct or -inherited. - -It's worth noting that Mastodon adds one more property, `preferredName`, -and expects it to be set on any actor, or that actor won't be recognized by -Mastodon. diff --git a/doc/api/activity_pub/index.md b/doc/api/activity_pub/index.md deleted file mode 100644 index 71c74f061a2629..00000000000000 --- a/doc/api/activity_pub/index.md +++ /dev/null @@ -1,213 +0,0 @@ ---- -stage: Create -group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments" ---- - -# ActivityPub **(EXPERIMENT)** - -The goal of those documents is to provide an implementation path for adding -Fediverse capabilities to GitLab. - -This page will describe the conceptual and high level point of view, while -subpages will discuss implementation in more technical depth (as in, how to -implement this in the actual rails codebase of GitLab). - -- [What](#what) - - [The Fediverse](#the-fediverse) - - [ActivityPub](#activitypub) -- [Why](#why) -- [How](#how) - -## What - -Feel free to jump to [the Why section](#why) if you already know what -ActivityPub and the Fediverse are. - -Among the push for [decentralization of the web](https://en.wikipedia.org/wiki/Decentralized_web), -several projects tried different protocols with different ideals behind their reasoning -(some examples : [Secure Scuttlebutt](https://en.wikipedia.org/wiki/Secure_Scuttlebutt) or ssb for short, -[Dat](https://en.wikipedia.org/wiki/Dat_%28software%29), -[IPFS](https://en.wikipedia.org/wiki/InterPlanetary_File_System), -[Solid](https://en.wikipedia.org/wiki/Solid_%28web_decentralization_project%29)). -But one gained traction recently : what is known as -[ActivityPub](https://en.wikipedia.org/wiki/ActivityPub), better known for -the colloquial [Fediverse](https://en.wikipedia.org/wiki/Fediverse) built -on top of it, through applications like -[Mastodon](https://en.wikipedia.org/wiki/Mastodon_%28social_network%29) -(which could be described as some sort of decentralized Facebook) or -[Lemmy](https://en.wikipedia.org/wiki/Lemmy_%28software%29) (which could be -described as some sort of decentralized Reddit). - -We think that ActivityPub has several advantages that makes it attractive -to implementers and could explain its current success: - -- **It's built on top of HTTP**. You don't need to install new software or - to tinker with TCP/UDP to implement ActivityPub, if you have a webserver - or an application that provides an HTTP API (like a rails application), - you already have everything you need. -- **It's built on top of JSON**. All communications are basically JSON - objects, which webdevelopers are already used to. This makes adoption - really easy. -- **It's a w3c standard and already have multiple implementations**. Being - piloted by the w3c is a guarantee of stability and quality work. They - have profusely demonstrated in the past through their work on HTML, CSS - or other web standards that we can build on top of their work without - the fear of it becoming deprecated or irrelevant after a few years. - -### The Fediverse - -The core ideas behind Mastodon and Lemmy is called the Fediverse. Rather -than full decentralization, those applications rely on federation, in the -sense that there still are servers and clients (so it's not p2p like ssb, -Dat and IPFS), but there is a galaxy of servers chatting with each other -instead of having central servers controlled by a single entity (like -Facebook or Reddit). - -The user signs up to one of those servers (called **instances**), and they -can then interact with users either on this instance, or on other ones. -From the perspective of the user, they access a global network, and not -only their instance. They see the articles posted on other instances, they -can comment on them, upvote them, etc. What happens behind the scene is -that their instance know where the user they reply to is hosted, and it -contacts that other instance to let them know there is a message for them - -somewhat similar to what SMTP is doing. Similarly, when an user subscribes -to any sort of feed, their instance informs the instance where the feed is -hosted of this subscription, and then that target instance will post back -messages when new activities are created (allowing for a push model rather -than a constant poll model like RSS). Of course, what was just described is -the happy path, there is moderation, validation and fault tolerance -happening all the way. - -### ActivityPub - -Behind the Fediverse is the ActivityPub protocol. It's a simple HTTP API -attempting to be as general a social network implementation as possible, -while giving options to be extendable. - -The basic idea is that an `actor` will send and receive `activities`, which -are structured JSON messages with well defined properties, but extensible -to cover any need. An actor is defined by four endpoints, which are -contacted with the `application/ld+json; -profile="https://www.w3.org/ns/activitystreams"` Accept header: - -- `GET /inbox` : used by the actor to find new activities intended for them. -- `POST /inbox` : used by instances to push new activities intended for the - actor. -- `GET /outbox` : used by anyone to read the activities created by the - actor. -- `POST /outbox` : used by the actor to publish new activities. - -Among those, Mastodon and Lemmy currently only use `POST -/inbox` and `GET /outbox`, which is the minimum needed to implement -federation : instances push new activities for the actor on the inbox, and -reading the outbox allow to read the feed of an actor. - -Additionally, Mastodon and Lemmy implement a `GET /` endpoint (with the -mentioned Accept header) which responds with general information about the -actor, like name and URL of inbox and outbox. This is not required by the -standard, but it makes discovery easier. - -It's worth mentioning that while it's the main use case, an actor does not -necessarily map to a person. Anything can be an actor : a topic, a -subreddit, a group, an event, etc. It's easy to understand for GitLab : -anything that have activities (in the sense of what GitLab means by -"activity") can be an ActivityPub actor. So this includes projects, groups, -releases, etc. In those more abstract examples, an actor can be thought of -as an actionable feed. - -And that's it. Sounds too simple to be true? That's because it is. :) -ActivityPub by itself does not cover everything that is needed to implement -the Fediverse. Most notably, are left for the implementers to figure out: - -- finding a way to deal with spam, currently covered by authorizing or - blocking other instances (colloquially referred to as "defederating") -- discovering new instances -- performing network wide searches - -## Why - -Why would a social media protocol be useful for GitLab? - -There already has been several very popular discussions around this (see -[here](https://gitlab.com/gitlab-org/gitlab/-/issues/21582), -[here](https://gitlab.com/gitlab-org/gitlab/-/issues/14116) and the epic -[here](https://gitlab.com/groups/gitlab-org/-/epics/260)). The gist of it -is: what people really want is to have one global "GitLab network" to be -able to interact between various projects without having to register on -each of their hosts. - -The ideal workflow for this would be: - -- Alice registers to her favorite GitLab instance, like - `gitlab.example.org` -- She looks for a project on a given topic, and sees Bob's project popping - up, despite Bob being on `gitlab.com`. -- She clicks the "fork" button, and the `gitlab.com/Bob/project.git` is - forked to `gitlab.example.org/Alice/project.git` -- She makes her edits, and opens a merge request, which appears in Bob's - project on `gitlab.com`. -- Alice and Bob discuss the merge request, each one from their own GitLab - instance. -- Bob can send additional commits, which are picked up by Alice's instance -- When Bob accepts the merge request, his instance picks up the code from - Alice's instance. - -In this process, ActivityPub would help in: - -- letting Bob know a fork happened -- sending the MR to Bob -- allowing Alice and Bob to discuss the MR -- letting Alice know the code has been merged - -It will _not_ help in (please open an issue if I'm wrong): - -- implementing a network wide search -- implementing cross-instance forks (don't need it anyway, Git is kind of - good at that :) ) - -Those will need to get specific implementations. - -One may wonder : why use ActivityPub here rather than "just" implementing -cross-instance merge requests in a custom way? - -There are two reasons for that: - -- **Building on top of a standard makes it easy to reach beyond GitLab**. - While the workflow presented above only mentions GitLab, building on top - of a w3c standard means it will be easy for other forges to follow GitLab - there, and build a massive Fediverse of code sharing. -- **This is an opportunity to make GitLab more social**. To prepare the - architecture for the workflow above, smaller steps can be taken, allowing - people to subscribe to activity feeds from their Fediverse social - network. Basically, anything that has a RSS feed currently could become - an ActivityPub feed. This would mean that people on Mastodon could follow - their favorite developer, project or topic from GitLab and see the news - in their feed on Mastodon, hopefully raising engagement with GitLab. - -## How - -The idea of this implementation path is not to take the fastest route to -the feature with the most value added (cross-instance merge requests), but -to go on with the smallest useful step at each iteration, making sure each step -brings something actually useful immediately. - -1. implement ActivityPub for social following - After this, the Fediverse can follow activities on GitLab instances. - 1. ActivityPub to subscribe to project releases - 1. ActivityPub to subscribe to project creation in topics - 1. ActivityPub to subscribe to project activities - 1. ActivityPub to subscribe to group activities - 1. ActivityPub to subscribe to user activities -1. **implement cross-instance search**. After this, it's possible to - discover projects on other instances. -1. **implement cross-instance fork**. After this, it's possible to fork a - project from an other instance. -1. **implement ActivityPub for cross instance discussions** . After this, - it's possible to discuss on issues and MR from an other instance. - 1. in issues - 1. in merge requests -1. **implement ActivityPub to submit cross-instance merge request**. After - this, it's possible to submit merge requests to other instances. - -For now, you can see [how to implement an ActivityPub actor](implementing-actor.md). -- GitLab From fbf9b803c64a176210c287a9b5b78f4e2b917540 Mon Sep 17 00:00:00 2001 From: Olivier El Mekki Date: Thu, 31 Aug 2023 15:10:03 +0200 Subject: [PATCH 08/10] ADD make blocking private projects more explicit --- app/controllers/activity_pub/application_controller.rb | 7 ------- .../activity_pub/projects/application_controller.rb | 2 +- 2 files changed, 1 insertion(+), 8 deletions(-) diff --git a/app/controllers/activity_pub/application_controller.rb b/app/controllers/activity_pub/application_controller.rb index 452994ab52f208..aa369b8fa1a6a8 100644 --- a/app/controllers/activity_pub/application_controller.rb +++ b/app/controllers/activity_pub/application_controller.rb @@ -7,13 +7,6 @@ class ApplicationController < ::ApplicationController skip_before_action :authenticate_user! after_action :set_content_type - # When we send activities out, we don't know if servers we federate with - # will keep them private. So we don't publish any private data, even if - # cookies for authorized user are passed somehow. - def current_user - nil - end - def can?(object, action, subject = :global) Ability.allowed?(object, action, subject) end diff --git a/app/controllers/activity_pub/projects/application_controller.rb b/app/controllers/activity_pub/projects/application_controller.rb index 9d0a6bc5562319..9760c01693cc51 100644 --- a/app/controllers/activity_pub/projects/application_controller.rb +++ b/app/controllers/activity_pub/projects/application_controller.rb @@ -16,7 +16,7 @@ def project end def auth_proc - ->(project) { !project.pending_delete? } + ->(project) { project.public? && !project.pending_delete? } end end end -- GitLab From 54e7d4f5b9bd62a2d0051290a88e549fae0fdd0f Mon Sep 17 00:00:00 2001 From: Olivier El Mekki Date: Mon, 4 Sep 2023 16:15:18 +0200 Subject: [PATCH 09/10] ADD feature flags --- .../activity_pub/application_controller.rb | 5 ++++ .../projects/application_controller.rb | 5 ++++ .../development/activity_pub.yml | 8 +++++++ .../development/activity_pub_project.yml | 8 +++++++ .../projects/releases_controller_spec.rb | 24 +++++++++++++++++++ 5 files changed, 50 insertions(+) create mode 100644 config/feature_flags/development/activity_pub.yml create mode 100644 config/feature_flags/development/activity_pub_project.yml diff --git a/app/controllers/activity_pub/application_controller.rb b/app/controllers/activity_pub/application_controller.rb index aa369b8fa1a6a8..f9c2b14fe7701c 100644 --- a/app/controllers/activity_pub/application_controller.rb +++ b/app/controllers/activity_pub/application_controller.rb @@ -4,6 +4,7 @@ module ActivityPub class ApplicationController < ::ApplicationController include RoutableActions + before_action :ensure_feature_flag skip_before_action :authenticate_user! after_action :set_content_type @@ -18,5 +19,9 @@ def route_not_found def set_content_type self.content_type = "application/activity+json" end + + def ensure_feature_flag + not_found unless ::Feature.enabled?(:activity_pub) + end end end diff --git a/app/controllers/activity_pub/projects/application_controller.rb b/app/controllers/activity_pub/projects/application_controller.rb index 9760c01693cc51..e54a457743d4dc 100644 --- a/app/controllers/activity_pub/projects/application_controller.rb +++ b/app/controllers/activity_pub/projects/application_controller.rb @@ -4,6 +4,7 @@ module ActivityPub module Projects class ApplicationController < ::ActivityPub::ApplicationController before_action :project + before_action :ensure_project_feature_flag private @@ -18,6 +19,10 @@ def project def auth_proc ->(project) { project.public? && !project.pending_delete? } end + + def ensure_project_feature_flag + not_found unless ::Feature.enabled?(:activity_pub_project, project) + end end end end diff --git a/config/feature_flags/development/activity_pub.yml b/config/feature_flags/development/activity_pub.yml new file mode 100644 index 00000000000000..0a7cf9862a53d1 --- /dev/null +++ b/config/feature_flags/development/activity_pub.yml @@ -0,0 +1,8 @@ +--- +name: activity_pub +introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127023 +rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/424008 +milestone: '16.4' +type: development +group: group::source code +default_enabled: false \ No newline at end of file diff --git a/config/feature_flags/development/activity_pub_project.yml b/config/feature_flags/development/activity_pub_project.yml new file mode 100644 index 00000000000000..6630333ad5e034 --- /dev/null +++ b/config/feature_flags/development/activity_pub_project.yml @@ -0,0 +1,8 @@ +--- +name: activity_pub_project +introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/127023 +rollout_issue_url: https://gitlab.com/gitlab-org/gitlab/-/issues/424008 +milestone: '16.4' +type: development +group: group::source code +default_enabled: false \ No newline at end of file diff --git a/spec/controllers/activity_pub/projects/releases_controller_spec.rb b/spec/controllers/activity_pub/projects/releases_controller_spec.rb index 79f87b615d1f4f..8719756b2604b3 100644 --- a/spec/controllers/activity_pub/projects/releases_controller_spec.rb +++ b/spec/controllers/activity_pub/projects/releases_controller_spec.rb @@ -45,6 +45,30 @@ end end end + + context 'when activity_pub feature flag is disabled' do + before do + stub_feature_flags(activity_pub: false) + end + + it 'renders a 404' do + get(action, params: params) + + expect(response).to have_gitlab_http_status(:not_found) + end + end + + context 'when activity_pub_project feature flag is disabled' do + before do + stub_feature_flags(activity_pub_project: false) + end + + it 'renders a 404' do + get(action, params: params) + + expect(response).to have_gitlab_http_status(:not_found) + end + end end shared_examples_for 'ActivityPub response' do -- GitLab From d8c9cdc424abe8783803a9059e8f111ebb6013f9 Mon Sep 17 00:00:00 2001 From: Olivier El Mekki Date: Mon, 4 Sep 2023 17:21:50 +0200 Subject: [PATCH 10/10] ADD feature category in `ActivityPub::Projects::ReleasesController` --- app/controllers/activity_pub/projects/releases_controller.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/app/controllers/activity_pub/projects/releases_controller.rb b/app/controllers/activity_pub/projects/releases_controller.rb index c36f0029ba05c5..7c4c2a0322be69 100644 --- a/app/controllers/activity_pub/projects/releases_controller.rb +++ b/app/controllers/activity_pub/projects/releases_controller.rb @@ -3,6 +3,8 @@ module ActivityPub module Projects class ReleasesController < ApplicationController + feature_category :release_orchestration + def index opts = { inbox: nil, -- GitLab