Add alert for rate limited GroupHook and ProjectHook webhooks on project and group pages

Webhooks may be rate limited and as a consequence not fire.

We log the fact that a webhook has been rate limited in app/services/web_hook_service.rb, but it would be good to add alerting in the UI.

We could adopt the approach used to address this issue for adding alerts for disabled webhooks. This checks a redis cache which checks the web_hooks table to track whether a web hook is disabled. It uses the user_namespace_callouts DB table to track if an alert has been dismissed by a user.

We could do something similar for rate limited webhooks.

• add a column last_rate_limited column to the web_hooks table which is updated with a timestamp every time a web hook is rate limited.
• store a true/false value in redis for each group and project that indicates whether the group/project has any recently rate limited webhooks. Recent could mean within the last hour.
• display an alert if there is a recent rate limit. This won't display information about which webhook, just that a webhook on the group/project has been rate limited and a link to the webhooks troubleshoting page
• update the webhook API response to include the last_rate_limited timestamp.

The implementation would involve additions to the following files:

• app/assets/javascripts/persistent_user_callout.js

• app/assets/javascripts/persistent_user_callouts.js

• app/models/concerns/web_hooks/has_web_hooks.rb

• app/models/concerns/web_hooks/hook.rb

• app/helpers/web_hooks/web_hooks_helper.rb

• ee/app/helpers/ee/web_hooks/web_hooks_helper.rb

• app/helpers/users/callouts_helper.rb

• app/services/web_hook_service.rb

• app/services/web_hooks/log_execution_service.rb

• app/views/layouts/project.html.haml

• app/views/layouts/group.html.haml

• app/views/shared/web_hooks/_web_hook_rate_limited_alert.html.haml (a new file)

• ee/app/views/shared/web_hooks/_group_web_hook_rate_limited_alert.html.haml (a new file)

Plus a DB migration to add the last_rate_limited column to web_hooks.

Plus updating the API response endpoints.

• ee/spec/fixtures/api/schemas/public_api/v4/group_hook.json
• lib/api/entities/hook.rb

It should be split into multiple MRs.

We should be aware that at the time of creating this issue around 2.5K webhooks are rate limited each week on gitlab.com.