Dogfooding otel logs in GOB cluster

We need to research a way to dogfood logging in our GOB cluster, even though there is no stable implementation of OTEL Logging bridge ATM. See this paragraph in the design doc for more details/context.

Related issue: #2130 (closed)

TODO - native instrumentation:

• provisioning-api
• backup
• auth-proxy
• query-api
• trace-query-api
• scheduler
• otel collector
• errortracking
• smoke tests
• swagger ui

TODO - ingesting logs from pods:

• k8s node logs/node exporter
• prometheus
• redis, redis sentinel
• traefik

Outside of the scope:

• tenant operator - is going away
• gatekeeper - is going away
• clickhouse-operator - is going away
• jaeger-operator - is going away