profiler: add a prometheus backend

What

Add a prometheus backend for the profiler

Why

Continuous monitoring of profiler metrics using tools that are already use (i.e. prometheus and /metrics endpoint in octez-node)

How

Define a backend, and register it under the "prometheus" name.

Prometheus can't handle a too high cardinality in metrics, so this backend need to avoid registering section with unique identifiers such as block hash or (level, round). That is why in order to mark a profiling call you need to use {metadata=["prometheus", "metric_name"]} for prometheus to record the data. You can leave metric_name as empty string and it will use the name already provided in the call.

e.g.

[@profiler.record_f { metadata: ["prometheus", "block_validation"] } block_hash ]

Manually testing the MR

TEZOS_PPX_PROFILER=profiling make && PROFILING_BACKEND="prometheus" PROFILING=debug ./octez-node run --data-dir /home/ju/tezos/ghostnet

Visit the /metrics endpoint (likely http://127.0.0.1:8732/metrics)

#HELP ocaml_gc_allocated_bytes Total number of bytes allocated since the program was started.
#TYPE ocaml_gc_allocated_bytes counter
ocaml_gc_allocated_bytes 36103617856.000000
#HELP ocaml_gc_compactions Number of heap compactions since the program was started.
#TYPE ocaml_gc_compactions counter
ocaml_gc_compactions 0.000000
#HELP ocaml_gc_heap_words Total size of the major heap, in words.
#TYPE ocaml_gc_heap_words gauge
ocaml_gc_heap_words 33526272.000000
#HELP ocaml_gc_major_collections Number of major collection cycles completed since the program was started.
#TYPE ocaml_gc_major_collections counter
ocaml_gc_major_collections 80.000000
#HELP ocaml_gc_major_words Number of words allocated in the major heap since the program was started.
#TYPE ocaml_gc_major_words counter
ocaml_gc_major_words 642687291.000000
#HELP ocaml_gc_minor_collections Number of minor collection cycles completed since the program was started.
#TYPE ocaml_gc_minor_collections counter
ocaml_gc_minor_collections 17135.000000
#HELP ocaml_gc_top_heap_words Maximum size reached by the major heap, in words.
#TYPE ocaml_gc_top_heap_words counter
ocaml_gc_top_heap_words 33526272.000000
#HELP octez_distributed_db_message_block_header_messages Number of block_header messages
#TYPE octez_distributed_db_message_block_header_messages counter
octez_distributed_db_message_block_header_messages{action="received"} 5.000000
octez_distributed_db_message_block_header_messages{action="sent"} 0.000000
#HELP octez_distributed_db_message_checkpoint_messages Number of checkpoint messages
#TYPE octez_distributed_db_message_checkpoint_messages counter
octez_distributed_db_message_checkpoint_messages{action="received"} 0.000000
octez_distributed_db_message_checkpoint_messages{action="sent"} 0.000000
#HELP octez_distributed_db_message_current_branch_messages Number of current_branch messages
#TYPE octez_distributed_db_message_current_branch_messages counter
octez_distributed_db_message_current_branch_messages{action="received"} 53.000000
octez_distributed_db_message_current_branch_messages{action="sent"} 53.000000
#HELP octez_distributed_db_message_current_head_messages Number of current_head messages
#TYPE octez_distributed_db_message_current_head_messages counter
octez_distributed_db_message_current_head_messages{action="broadcasted"} 4436.000000
octez_distributed_db_message_current_head_messages{action="received"} 186583.000000
octez_distributed_db_message_current_head_messages{action="sent"} 6.000000
#HELP octez_distributed_db_message_deactivate_messages Number of deactivate messages
#TYPE octez_distributed_db_message_deactivate_messages counter
octez_distributed_db_message_deactivate_messages{action="received"} 0.000000
octez_distributed_db_message_deactivate_messages{action="sent"} 0.000000
#HELP octez_distributed_db_message_get_block_headers_messages Number of get_block_headers messages
#TYPE octez_distributed_db_message_get_block_headers_messages counter
octez_distributed_db_message_get_block_headers_messages{action="received"} 0.000000
octez_distributed_db_message_get_block_headers_messages{action="sent"} 5.000000
#HELP octez_distributed_db_message_get_checkpoint_messages Number of get_checkpoint messages
#TYPE octez_distributed_db_message_get_checkpoint_messages counter
octez_distributed_db_message_get_checkpoint_messages{action="received"} 0.000000
octez_distributed_db_message_get_checkpoint_messages{action="sent"} 0.000000
#HELP octez_distributed_db_message_get_current_branch_messages Number of get_current_branch messages
#TYPE octez_distributed_db_message_get_current_branch_messages counter
octez_distributed_db_message_get_current_branch_messages{action="received"} 53.000000
octez_distributed_db_message_get_current_branch_messages{action="sent"} 53.000000
#HELP octez_distributed_db_message_get_current_head_messages Number of get_current_head messages
#TYPE octez_distributed_db_message_get_current_head_messages counter
octez_distributed_db_message_get_current_head_messages{action="broadcasted"} 0.000000
octez_distributed_db_message_get_current_head_messages{action="received"} 6.000000
octez_distributed_db_message_get_current_head_messages{action="sent"} 5.000000
#HELP octez_distributed_db_message_get_operations_for_blocks_messages Number of get_operations_for_blocks messages
#TYPE octez_distributed_db_message_get_operations_for_blocks_messages counter
octez_distributed_db_message_get_operations_for_blocks_messages{action="received"} 28.000000
octez_distributed_db_message_get_operations_for_blocks_messages{action="sent"} 430.000000
#HELP octez_distributed_db_message_get_operations_messages Number of get_operations messages
#TYPE octez_distributed_db_message_get_operations_messages counter
octez_distributed_db_message_get_operations_messages{action="received"} 3544.000000
octez_distributed_db_message_get_operations_messages{action="sent"} 11921.000000
#HELP octez_distributed_db_message_get_predecessor_header_messages Number of get_predecessor_header messages
#TYPE octez_distributed_db_message_get_predecessor_header_messages counter
octez_distributed_db_message_get_predecessor_header_messages{action="received"} 0.000000
octez_distributed_db_message_get_predecessor_header_messages{action="sent"} 0.000000
#HELP octez_distributed_db_message_get_protocol_branch_messages Number of get_protocol_branch messages
#TYPE octez_distributed_db_message_get_protocol_branch_messages counter
octez_distributed_db_message_get_protocol_branch_messages{action="received"} 0.000000
octez_distributed_db_message_get_protocol_branch_messages{action="sent"} 0.000000
#HELP octez_distributed_db_message_get_protocols_messages Number of get_protocols messages
#TYPE octez_distributed_db_message_get_protocols_messages counter
octez_distributed_db_message_get_protocols_messages{action="received"} 0.000000
octez_distributed_db_message_get_protocols_messages{action="sent"} 0.000000
#HELP octez_distributed_db_message_operation_messages Number of operation messages
#TYPE octez_distributed_db_message_operation_messages counter
octez_distributed_db_message_operation_messages{action="received"} 15038.000000
octez_distributed_db_message_operation_messages{action="sent"} 3924.000000
#HELP octez_distributed_db_message_operations_for_block_messages Number of operations_for_block messages
#TYPE octez_distributed_db_message_operations_for_block_messages counter
octez_distributed_db_message_operations_for_block_messages{action="received"} 828.000000
octez_distributed_db_message_operations_for_block_messages{action="sent"} 83.000000
#HELP octez_distributed_db_message_predecessor_header_messages Number of predecessor_header messages
#TYPE octez_distributed_db_message_predecessor_header_messages counter
octez_distributed_db_message_predecessor_header_messages{action="received"} 0.000000
octez_distributed_db_message_predecessor_header_messages{action="sent"} 0.000000
#HELP octez_distributed_db_message_protocol_branch_messages Number of protocol_branch messages
#TYPE octez_distributed_db_message_protocol_branch_messages counter
octez_distributed_db_message_protocol_branch_messages{action="received"} 0.000000
octez_distributed_db_message_protocol_branch_messages{action="sent"} 0.000000
#HELP octez_distributed_db_message_protocol_messages Number of protocol messages
#TYPE octez_distributed_db_message_protocol_messages counter
octez_distributed_db_message_protocol_messages{action="received"} 0.000000
octez_distributed_db_message_protocol_messages{action="sent"} 0.000000
#HELP octez_distributed_db_requester_table_length Number of entries (to grab) from the network present
#TYPE octez_distributed_db_requester_table_length gauge
octez_distributed_db_requester_table_length{requester_kind="block_header", entry_type="entries"} 0.000000
octez_distributed_db_requester_table_length{requester_kind="block_header", entry_type="pending_requests"} 0.000000
octez_distributed_db_requester_table_length{requester_kind="operation", entry_type="entries"} 103.000000
octez_distributed_db_requester_table_length{requester_kind="operation", entry_type="pending_requests"} 1.000000
octez_distributed_db_requester_table_length{requester_kind="operations", entry_type="entries"} 0.000000
octez_distributed_db_requester_table_length{requester_kind="operations", entry_type="pending_requests"} 0.000000
octez_distributed_db_requester_table_length{requester_kind="protocol", entry_type="entries"} 0.000000
octez_distributed_db_requester_table_length{requester_kind="protocol", entry_type="pending_requests"} 0.000000
#HELP octez_external_rpc_process_calls External RPC endpoint call counts and sum of execution times.
#TYPE octez_external_rpc_process_calls summary
#HELP octez_mempool_pending_branch_delayed Mempool pending branch delayed operations count
#TYPE octez_mempool_pending_branch_delayed gauge
octez_mempool_pending_branch_delayed 0.000000
#HELP octez_mempool_pending_branch_refused Mempool pending branch refused operations count
#TYPE octez_mempool_pending_branch_refused gauge
octez_mempool_pending_branch_refused 0.000000
#HELP octez_mempool_pending_outdated Mempool pending outdated operations count
#TYPE octez_mempool_pending_outdated gauge
octez_mempool_pending_outdated 1000.000000
#HELP octez_mempool_pending_refused Mempool pending refused operations count
#TYPE octez_mempool_pending_refused gauge
octez_mempool_pending_refused 0.000000
#HELP octez_mempool_pending_unprocessed Mempool pending unprocessed operations count
#TYPE octez_mempool_pending_unprocessed gauge
octez_mempool_pending_unprocessed 0.000000
#HELP octez_mempool_pending_validated Mempool pending validated operations count
#TYPE octez_mempool_pending_validated gauge
octez_mempool_pending_validated 102.000000
#HELP octez_mempool_worker_completion_count Number of requests completed the block validator worker
#TYPE octez_mempool_worker_completion_count counter
octez_mempool_worker_completion_count 205576.000000
#HELP octez_mempool_worker_error_count Number of errors encountered by the block validator worker
#TYPE octez_mempool_worker_error_count counter
octez_mempool_worker_error_count 0.000000
#HELP octez_mempool_worker_request_count Number of requests received by the block validator worker
#TYPE octez_mempool_worker_request_count counter
octez_mempool_worker_request_count 205576.000000
#HELP octez_p2p_connections_active Number of active connections
#TYPE octez_p2p_connections_active gauge
octez_p2p_connections_active 38.000000
#HELP octez_p2p_connections_incoming Number of incoming connections
#TYPE octez_p2p_connections_incoming gauge
octez_p2p_connections_incoming 0.000000
#HELP octez_p2p_connections_outgoing Number of outgoing connections
#TYPE octez_p2p_connections_outgoing gauge
octez_p2p_connections_outgoing 38.000000
#HELP octez_p2p_connections_private Number of private connections
#TYPE octez_p2p_connections_private gauge
octez_p2p_connections_private 0.000000
#HELP octez_p2p_io_scheduler_current_inflow Current ingoing data rate
#TYPE octez_p2p_io_scheduler_current_inflow gauge
octez_p2p_io_scheduler_current_inflow 100081.000000
#HELP octez_p2p_io_scheduler_current_outflow Current outgoing data rate
#TYPE octez_p2p_io_scheduler_current_outflow gauge
octez_p2p_io_scheduler_current_outflow 92566.000000
#HELP octez_p2p_io_scheduler_total_recv Total amount of received data
#TYPE octez_p2p_io_scheduler_total_recv gauge
octez_p2p_io_scheduler_total_recv 94313121.000000
#HELP octez_p2p_io_scheduler_total_sent Total amount of sent data (in bytes)
#TYPE octez_p2p_io_scheduler_total_sent gauge
octez_p2p_io_scheduler_total_sent 90671559.000000
#HELP octez_p2p_messages_advertise_received Number of advertise received
#TYPE octez_p2p_messages_advertise_received counter
octez_p2p_messages_advertise_received 0.000000
#HELP octez_p2p_messages_advertise_sent Number of advertise sent
#TYPE octez_p2p_messages_advertise_sent counter
octez_p2p_messages_advertise_sent 55.000000
#HELP octez_p2p_messages_bootstrap_received Number of bootstrap received
#TYPE octez_p2p_messages_bootstrap_received counter
octez_p2p_messages_bootstrap_received 55.000000
#HELP octez_p2p_messages_bootstrap_sent Number of bootstrap sent
#TYPE octez_p2p_messages_bootstrap_sent counter
octez_p2p_messages_bootstrap_sent 0.000000
#HELP octez_p2p_messages_broadcast_message_sent Number of user message sent by broadcasting
#TYPE octez_p2p_messages_broadcast_message_sent counter
octez_p2p_messages_broadcast_message_sent 188904.000000
#HELP octez_p2p_messages_swap_ack_received Number of swap acks received
#TYPE octez_p2p_messages_swap_ack_received counter
octez_p2p_messages_swap_ack_received 3.000000
#HELP octez_p2p_messages_swap_ack_sent Number of swap acks sent
#TYPE octez_p2p_messages_swap_ack_sent counter
octez_p2p_messages_swap_ack_sent 7.000000
#HELP octez_p2p_messages_swap_requests_received Number of swap received
#TYPE octez_p2p_messages_swap_requests_received counter
octez_p2p_messages_swap_requests_received 11.000000
#HELP octez_p2p_messages_swap_requests_sent Number of swap sent
#TYPE octez_p2p_messages_swap_requests_sent counter
octez_p2p_messages_swap_requests_sent 9.000000
#HELP octez_p2p_messages_user_message_received Number of user message received
#TYPE octez_p2p_messages_user_message_received counter
octez_p2p_messages_user_message_received 206138.000000
#HELP octez_p2p_messages_user_message_received_error Number of user message received that resulted in error
#TYPE octez_p2p_messages_user_message_received_error counter
octez_p2p_messages_user_message_received_error 15.000000
#HELP octez_p2p_messages_user_message_sent Number of user message sent
#TYPE octez_p2p_messages_user_message_sent counter
octez_p2p_messages_user_message_sent 16480.000000
#HELP octez_p2p_peers_accepted Number of accepted connections
#TYPE octez_p2p_peers_accepted gauge
octez_p2p_peers_accepted 0.000000
#HELP octez_p2p_peers_disconnected Number of disconnected peers
#TYPE octez_p2p_peers_disconnected gauge
octez_p2p_peers_disconnected 127.000000
#HELP octez_p2p_peers_running Number of running peers
#TYPE octez_p2p_peers_running gauge
octez_p2p_peers_running 38.000000
#HELP octez_p2p_points_accepted Number of accepted points
#TYPE octez_p2p_points_accepted gauge
octez_p2p_points_accepted 0.000000
#HELP octez_p2p_points_disconnected Number of disconnected points
#TYPE octez_p2p_points_disconnected gauge
octez_p2p_points_disconnected 82.000000
#HELP octez_p2p_points_greylisted Number of greylisted points
#TYPE octez_p2p_points_greylisted gauge
octez_p2p_points_greylisted 35.000000
#HELP octez_p2p_points_running Number of running points
#TYPE octez_p2p_points_running gauge
octez_p2p_points_running 38.000000
#HELP octez_p2p_points_trusted Number of trusted points
#TYPE octez_p2p_points_trusted gauge
octez_p2p_points_trusted 0.000000
#HELP octez_p2p_swap_fail Number of failed swap
#TYPE octez_p2p_swap_fail counter
octez_p2p_swap_fail 9.000000
#HELP octez_p2p_swap_ignored Number of ignored swap
#TYPE octez_p2p_swap_ignored counter
octez_p2p_swap_ignored 4.000000
#HELP octez_p2p_swap_success Number of successful swap
#TYPE octez_p2p_swap_success counter
octez_p2p_swap_success 0.000000
#HELP octez_rpc_calls RPC endpoint call counts and sum of execution times.
#TYPE octez_rpc_calls summary
#HELP octez_store_alternate_heads_count Current number of alternated heads known
#TYPE octez_store_alternate_heads_count gauge
octez_store_alternate_heads_count 0.000000
#HELP octez_store_caboose_level Current caboose level
#TYPE octez_store_caboose_level gauge
octez_store_caboose_level 8572930.000000
#HELP octez_store_checkpoint_level Current checkpoint level
#TYPE octez_store_checkpoint_level gauge
octez_store_checkpoint_level 8605418.000000
#HELP octez_store_invalid_blocks Number of blocks known to be invalid stored on disk
#TYPE octez_store_invalid_blocks gauge
octez_store_invalid_blocks 0.000000
#HELP octez_store_last_merge_time Time, in seconds, for the completion of the last store merge
#TYPE octez_store_last_merge_time gauge
octez_store_last_merge_time 0.000000
#HELP octez_store_last_written_block_size Size, in bytes, of the last block written in store
#TYPE octez_store_last_written_block_size gauge
octez_store_last_written_block_size 32919.000000
#HELP octez_store_maintenance_target The level at which the storage maintenance is expected to be triggered. Set to -1 if no target is set
#TYPE octez_store_maintenance_target gauge
octez_store_maintenance_target -1.000000
#HELP octez_store_savepoint_level Current savepoint level
#TYPE octez_store_savepoint_level gauge
octez_store_savepoint_level 8572930.000000
#HELP octez_validator_block_already_commited_blocks_count Number of requests to validate a block already handled
#TYPE octez_validator_block_already_commited_blocks_count counter
octez_validator_block_already_commited_blocks_count 5593.000000
#HELP octez_validator_block_already_known_invalid_blocks_count Number of requests to validate a block already known as invalid
#TYPE octez_validator_block_already_known_invalid_blocks_count counter
octez_validator_block_already_known_invalid_blocks_count 0.000000
#HELP octez_validator_block_application_errors_after_validation_count Number of requests to validate an inapplicable but validated block
#TYPE octez_validator_block_application_errors_after_validation_count counter
octez_validator_block_application_errors_after_validation_count 0.000000
#HELP octez_validator_block_commit_block_failed_count Number of requests that failed to commit a block
#TYPE octez_validator_block_commit_block_failed_count counter
octez_validator_block_commit_block_failed_count 0.000000
#HELP octez_validator_block_last_finished_request_completion_timestamp Timestamp at which the latest request handled by the worker was completed
#TYPE octez_validator_block_last_finished_request_completion_timestamp gauge
octez_validator_block_last_finished_request_completion_timestamp 1729247388.425104
#HELP octez_validator_block_last_finished_request_push_timestamp Reception timestamp of the latest request handled by the worker
#TYPE octez_validator_block_last_finished_request_push_timestamp gauge
octez_validator_block_last_finished_request_push_timestamp 1729247388.374399
#HELP octez_validator_block_last_finished_request_treatment_timestamp Timestamp at which the worker started processing of the latest request it handled
#TYPE octez_validator_block_last_finished_request_treatment_timestamp gauge
octez_validator_block_last_finished_request_treatment_timestamp 1729247388.378198
#HELP octez_validator_block_operations_per_pass Number of operations per pass for the last validated block
#TYPE octez_validator_block_operations_per_pass gauge
octez_validator_block_operations_per_pass{pass_id="0"} 31.000000
octez_validator_block_operations_per_pass{pass_id="1"} 0.000000
octez_validator_block_operations_per_pass{pass_id="2"} 0.000000
octez_validator_block_operations_per_pass{pass_id="3"} 4.000000
#HELP octez_validator_block_preapplication_errors_count Number of refused application simulations of blocks
#TYPE octez_validator_block_preapplication_errors_count counter
octez_validator_block_preapplication_errors_count 0.000000
#HELP octez_validator_block_preapplied_blocks_count Number of successful application simulations of blocks
#TYPE octez_validator_block_preapplied_blocks_count counter
octez_validator_block_preapplied_blocks_count 0.000000
#HELP octez_validator_block_validated_blocks_count Number of requests to validate a valid block
#TYPE octez_validator_block_validated_blocks_count counter
octez_validator_block_validated_blocks_count 207.000000
#HELP octez_validator_block_validation_errors_count Number of requests to validate an invalid block
#TYPE octez_validator_block_validation_errors_count counter
octez_validator_block_validation_errors_count 0.000000
#HELP octez_validator_block_validation_failed_count Number of block validation requests where the validation of a block failed
#TYPE octez_validator_block_validation_failed_count counter
octez_validator_block_validation_failed_count 0.000000
#HELP octez_validator_block_worker_completion_count Number of requests completed the block validator worker
#TYPE octez_validator_block_worker_completion_count counter
octez_validator_block_worker_completion_count 5800.000000
#HELP octez_validator_block_worker_error_count Number of errors encountered by the block validator worker
#TYPE octez_validator_block_worker_error_count counter
octez_validator_block_worker_error_count 0.000000
#HELP octez_validator_block_worker_request_count Number of requests received by the block validator worker
#TYPE octez_validator_block_worker_request_count counter
octez_validator_block_worker_request_count 5800.000000
#HELP octez_validator_chain_branch_switch_count Number of times the chain_validator switched branch
#TYPE octez_validator_chain_branch_switch_count counter
octez_validator_chain_branch_switch_count{chain_id="NetXnHfVqm9ie"} 20.000000
#HELP octez_validator_chain_head_consumed_gas Gas consumed in the current node's head
#TYPE octez_validator_chain_head_consumed_gas gauge
octez_validator_chain_head_consumed_gas{chain_id="NetXnHfVqm9ie"} 20538.000000
#HELP octez_validator_chain_head_cycle Cycle of the current node's head
#TYPE octez_validator_chain_head_cycle gauge
octez_validator_chain_head_cycle{chain_id="NetXnHfVqm9ie"} 1225.000000
#HELP octez_validator_chain_head_increment_count Number of times the chain_validator incremented its head for a direct successor
#TYPE octez_validator_chain_head_increment_count counter
octez_validator_chain_head_increment_count{chain_id="NetXnHfVqm9ie"} 187.000000
#HELP octez_validator_chain_head_level Level of the current node's head
#TYPE octez_validator_chain_head_level gauge
octez_validator_chain_head_level{chain_id="NetXnHfVqm9ie"} 8605420.000000
#HELP octez_validator_chain_head_round Round of the current node's head
#TYPE octez_validator_chain_head_round gauge
octez_validator_chain_head_round{chain_id="NetXnHfVqm9ie"} 0.000000
#HELP octez_validator_chain_ignored_head_count Number of requests where the chain validator ignored a new valid block with a lower fitness than its current head
#TYPE octez_validator_chain_ignored_head_count counter
octez_validator_chain_ignored_head_count{chain_id="NetXnHfVqm9ie"} 0.000000
#HELP octez_validator_chain_is_bootstrapped Returns 1 if the node has bootstrapped, 0 otherwise.
#TYPE octez_validator_chain_is_bootstrapped gauge
octez_validator_chain_is_bootstrapped{chain_id="NetXnHfVqm9ie"} 1.000000
#HELP octez_validator_chain_last_finished_request_completion_timestamp Timestamp at which the latest request handled by the worker was completed
#TYPE octez_validator_chain_last_finished_request_completion_timestamp gauge
octez_validator_chain_last_finished_request_completion_timestamp{chain_id="NetXnHfVqm9ie"} 1729247388.432955
#HELP octez_validator_chain_last_finished_request_push_timestamp Reception timestamp of the latest request handled by the worker
#TYPE octez_validator_chain_last_finished_request_push_timestamp gauge
octez_validator_chain_last_finished_request_push_timestamp{chain_id="NetXnHfVqm9ie"} 1729247388.425102
#HELP octez_validator_chain_last_finished_request_treatment_timestamp Timestamp at which the worker started processing of the latest request it handled
#TYPE octez_validator_chain_last_finished_request_treatment_timestamp gauge
octez_validator_chain_last_finished_request_treatment_timestamp{chain_id="NetXnHfVqm9ie"} 1729247388.425142
#HELP octez_validator_chain_synchronisation_status Returns 0 if the node is unsynchronised, 1 if the node is synchronised, 2 if the node is stuck.
#TYPE octez_validator_chain_synchronisation_status gauge
octez_validator_chain_synchronisation_status{chain_id="NetXnHfVqm9ie"} 1.000000
#HELP octez_validator_chain_worker_completion_count Number of requests completed the block validator worker
#TYPE octez_validator_chain_worker_completion_count counter
octez_validator_chain_worker_completion_count{chain_id="NetXnHfVqm9ie"} 186858.000000
#HELP octez_validator_chain_worker_error_count Number of errors encountered by the block validator worker
#TYPE octez_validator_chain_worker_error_count counter
octez_validator_chain_worker_error_count{chain_id="NetXnHfVqm9ie"} 0.000000
#HELP octez_validator_chain_worker_request_count Number of requests received by the block validator worker
#TYPE octez_validator_chain_worker_request_count counter
octez_validator_chain_worker_request_count{chain_id="NetXnHfVqm9ie"} 186858.000000
#HELP octez_validator_peer_connections Number of time we connected to a peer.
#TYPE octez_validator_peer_connections counter
octez_validator_peer_connections 53.000000
#HELP octez_validator_peer_invalid_block Number of time we received an invalid block from a peer.
#TYPE octez_validator_peer_invalid_block counter
octez_validator_peer_invalid_block 0.000000
#HELP octez_validator_peer_invalid_locator Number of time we received an invalid locator from a peer.
#TYPE octez_validator_peer_invalid_locator counter
octez_validator_peer_invalid_locator 0.000000
#HELP octez_validator_peer_new_branch_completed Number of time we successfuly completed a new branch request from a peer.
#TYPE octez_validator_peer_new_branch_completed counter
octez_validator_peer_new_branch_completed 53.000000
#HELP octez_validator_peer_new_head_completed Number of time we successfuly completed a new head request from a peer.
#TYPE octez_validator_peer_new_head_completed counter
octez_validator_peer_new_head_completed 10462.000000
#HELP octez_validator_peer_on_no_request_count Number of time we did no hear new messages from a peer since the last timeout.
#TYPE octez_validator_peer_on_no_request_count counter
octez_validator_peer_on_no_request_count 5.000000
#HELP octez_validator_peer_operations_fetching_canceled_new_branch Number of time we canceled the fetching of operations on a new branch request for a peer.
#TYPE octez_validator_peer_operations_fetching_canceled_new_branch counter
octez_validator_peer_operations_fetching_canceled_new_branch 0.000000
#HELP octez_validator_peer_operations_fetching_canceled_new_known_valid_head Number of time we canceled the fetching of operations on a new head request for a peer.
#TYPE octez_validator_peer_operations_fetching_canceled_new_known_valid_head counter
octez_validator_peer_operations_fetching_canceled_new_known_valid_head 0.000000
#HELP octez_validator_peer_operations_fetching_canceled_new_unknown_head Number of time we canceled the fetching of operations on a new head request or an unknown head for a peer.
#TYPE octez_validator_peer_operations_fetching_canceled_new_unknown_head counter
octez_validator_peer_operations_fetching_canceled_new_unknown_head 0.000000
#HELP octez_validator_peer_system_error Number of time a request trigerred a system error from a peer.
#TYPE octez_validator_peer_system_error counter
octez_validator_peer_system_error 0.000000
#HELP octez_validator_peer_too_short_locator Number of time we received a too short locator from a peer.
#TYPE octez_validator_peer_too_short_locator counter
octez_validator_peer_too_short_locator 0.000000
#HELP octez_validator_peer_unavailable_protocol Number of time we received an unknown protocol from a peer.
#TYPE octez_validator_peer_unavailable_protocol counter
octez_validator_peer_unavailable_protocol 0.000000
#HELP octez_validator_peer_unknown_ancestor Number of time we received a locator with an unknown ancestor from a peer.
#TYPE octez_validator_peer_unknown_ancestor counter
octez_validator_peer_unknown_ancestor 0.000000
#HELP octez_validator_peer_unknown_error Number of time an unknown error happened for a peer.
#TYPE octez_validator_peer_unknown_error counter
octez_validator_peer_unknown_error 0.000000
#HELP octez_version Node version
#TYPE octez_version gauge
octez_version{version="Octez 21.0~rc3+dev", chain_name="TEZOS_ITHACANET_2022-01-25T15:00:00Z", distributed_db_version="2", p2p_version="1", commit_hash="cc0ef08cd4f66dcf4aa23322299089f794ba2bc5", commit_date="2024-10-18 09:38:34 +0200"} 0.000000
#HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
#TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 77.226559
#HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
#TYPE process_start_time_seconds counter
process_start_time_seconds 1729246357.246774
#HELP profiling_block_validator_counter Number of times a function is called
#TYPE profiling_block_validator_counter counter
profiling_block_validator_counter{id="record_protocol"} 115.000000
#HELP profiling_block_validator_summary Number of times a function is called and total time passed in this function
#TYPE profiling_block_validator_summary summary
profiling_block_validator_summary_sum{id="apply_operations"} 0.535462
profiling_block_validator_summary_count{id="apply_operations"} 206.000000
profiling_block_validator_summary_sum{id="begin_application"} 1.738615
profiling_block_validator_summary_count{id="begin_application"} 206.000000
profiling_block_validator_summary_sum{id="begin_validation"} 2.630060
profiling_block_validator_summary_count{id="begin_validation"} 206.000000
profiling_block_validator_summary_sum{id="commit"} 1.921067
profiling_block_validator_summary_count{id="commit"} 206.000000
profiling_block_validator_summary_sum{id="commit_block"} 0.147837
profiling_block_validator_summary_count{id="commit_block"} 206.000000
profiling_block_validator_summary_sum{id="compute_live_blocks"} 0.059729
profiling_block_validator_summary_count{id="compute_live_blocks"} 206.000000
profiling_block_validator_summary_sum{id="compute_metadata"} 0.049571
profiling_block_validator_summary_count{id="compute_metadata"} 206.000000
profiling_block_validator_summary_sum{id="finalize_application"} 0.357636
profiling_block_validator_summary_count{id="finalize_application"} 206.000000
profiling_block_validator_summary_sum{id="finalize_validation"} 0.004709
profiling_block_validator_summary_count{id="finalize_validation"} 206.000000
profiling_block_validator_summary_sum{id="get_predecessor_metadata"} 0.000519
profiling_block_validator_summary_count{id="get_predecessor_metadata"} 206.000000
profiling_block_validator_summary_sum{id="parse_operations"} 0.281885
profiling_block_validator_summary_count{id="parse_operations"} 412.000000
profiling_block_validator_summary_sum{id="post_validation"} 1.978932
profiling_block_validator_summary_count{id="post_validation"} 206.000000
profiling_block_validator_summary_sum{id="prepare_context"} 0.007520
profiling_block_validator_summary_count{id="prepare_context"} 206.000000
profiling_block_validator_summary_sum{id="read_predecessor"} 0.001719
profiling_block_validator_summary_count{id="read_predecessor"} 206.000000
profiling_block_validator_summary_sum{id="record_protocol"} 0.000099
profiling_block_validator_summary_count{id="record_protocol"} 91.000000
profiling_block_validator_summary_sum{id="validate_operations"} 0.204629
profiling_block_validator_summary_count{id="validate_operations"} 206.000000
#HELP profiling_chain_validator_counter Number of times a function is called
#TYPE profiling_chain_validator_counter counter
profiling_chain_validator_counter{id="chain_validator"} 0.000000
profiling_chain_validator_counter{id="notify branch received"} 53.000000
profiling_chain_validator_counter{id="notify head received"} 186011.000000
#HELP profiling_chain_validator_summary Number of times a function is called and total time passed in this function
#TYPE profiling_chain_validator_summary summary
profiling_chain_validator_summary_sum{id="validation request (set_head)"} 2.058834
profiling_chain_validator_summary_count{id="validation request (set_head)"} 206.000000
#HELP profiling_context_counter Number of times a function is called
#TYPE profiling_context_counter counter
#HELP profiling_context_summary Number of times a function is called and total time passed in this function
#TYPE profiling_context_summary summary
#HELP profiling_mempool_counter Number of times a function is called
#TYPE profiling_mempool_counter counter
profiling_mempool_counter{id="Freshly validated operation"} 14535.000000
profiling_mempool_counter{id="already fetching"} 128912.000000
profiling_mempool_counter{id="is already handled"} 0.000000
profiling_mempool_counter{id="is_classified"} 829808.000000
profiling_mempool_counter{id="is_live_operation"} 3877.000000
profiling_mempool_counter{id="not already handled"} 29096.000000
profiling_mempool_counter{id="outdated operation"} 7424.000000
profiling_mempool_counter{id="reclassified high priority operation"} 8594.000000
profiling_mempool_counter{id="reclassified low priority operation"} 135.000000
#HELP profiling_mempool_summary Number of times a function is called and total time passed in this function
#TYPE profiling_mempool_summary summary
profiling_mempool_summary_sum{id="advertise mempool"} 0.032198
profiling_mempool_summary_count{id="advertise mempool"} 14956.000000
profiling_mempool_summary_sum{id="classify consensus operation "} 5.394609
profiling_mempool_summary_count{id="classify consensus operation "} 29911.000000
profiling_mempool_summary_sum{id="classify manager operation"} 0.788225
profiling_mempool_summary_count{id="classify manager operation"} 777.000000
profiling_mempool_summary_sum{id="fetching thread"} 1.956498
profiling_mempool_summary_count{id="fetching thread"} 143462.000000
profiling_mempool_summary_sum{id="flush state"} 0.579268
profiling_mempool_summary_count{id="flush state"} 201.000000
profiling_mempool_summary_sum{id="flushed operations"} 0.054727
profiling_mempool_summary_count{id="flushed operations"} 16142.000000
profiling_mempool_summary_sum{id="handle_unprocessed"} 8.216780
profiling_mempool_summary_count{id="handle_unprocessed"} 204928.000000
profiling_mempool_summary_sum{id="may_fetch_operation"} 0.212152
profiling_mempool_summary_count{id="may_fetch_operation"} 185941.000000
profiling_mempool_summary_sum{id="on_advertise"} 0.487456
profiling_mempool_summary_count{id="on_advertise"} 4017.000000
profiling_mempool_summary_sum{id="on_arrived"} 0.719566
profiling_mempool_summary_count{id="on_arrived"} 14546.000000
profiling_mempool_summary_sum{id="on_flush"} 0.788322
profiling_mempool_summary_count{id="on_flush"} 201.000000
profiling_mempool_summary_sum{id="on_notify"} 7.552518
profiling_mempool_summary_count{id="on_notify"} 185941.000000
profiling_mempool_summary_sum{id="pending hashes"} 0.017300
profiling_mempool_summary_count{id="pending hashes"} 14957.000000
profiling_mempool_summary_sum{id="set mempool"} 0.102872
profiling_mempool_summary_count{id="set mempool"} 14957.000000
profiling_mempool_summary_sum{id="union validated hashes"} 0.037710
profiling_mempool_summary_count{id="union validated hashes"} 14957.000000
#HELP profiling_p2p_reader_counter Number of times a function is called
#TYPE profiling_p2p_reader_counter counter
#HELP profiling_p2p_reader_summary Number of times a function is called and total time passed in this function
#TYPE profiling_p2p_reader_summary summary
profiling_p2p_reader_summary_sum{id="Block_header"} 0.000110
profiling_p2p_reader_summary_count{id="Block_header"} 5.000000
profiling_p2p_reader_summary_sum{id="Current_branch"} 0.017128
profiling_p2p_reader_summary_count{id="Current_branch"} 53.000000
profiling_p2p_reader_summary_sum{id="Current_head"} 3.927509
profiling_p2p_reader_summary_count{id="Current_head"} 186011.000000
profiling_p2p_reader_summary_sum{id="Get_current_branch"} 0.171200
profiling_p2p_reader_summary_count{id="Get_current_branch"} 53.000000
profiling_p2p_reader_summary_sum{id="Get_current_head"} 0.000375
profiling_p2p_reader_summary_count{id="Get_current_head"} 6.000000
profiling_p2p_reader_summary_sum{id="Get_operations"} 0.148829
profiling_p2p_reader_summary_count{id="Get_operations"} 3540.000000
profiling_p2p_reader_summary_sum{id="Get_operations_for_blocks"} 0.014016
profiling_p2p_reader_summary_count{id="Get_operations_for_blocks"} 28.000000
profiling_p2p_reader_summary_sum{id="Operation"} 0.673537
profiling_p2p_reader_summary_count{id="Operation"} 14977.000000
profiling_p2p_reader_summary_sum{id="Operations_for_block"} 0.055137
profiling_p2p_reader_summary_count{id="Operations_for_block"} 828.000000
#HELP profiling_requester_counter Number of times a function is called
#TYPE profiling_requester_counter counter
profiling_requester_counter{id="cache hit"} 0.000000
profiling_requester_counter{id="cache miss"} 0.000000
profiling_requester_counter{id="disk miss"} 0.000000
profiling_requester_counter{id="fetch"} 0.000000
#HELP profiling_requester_summary Number of times a function is called and total time passed in this function
#TYPE profiling_requester_summary summary
profiling_requester_summary_sum{id="cache miss : request creation"} 2551.872363
profiling_requester_summary_count{id="cache miss : request creation"} 15379.000000
profiling_requester_summary_sum{id="disk hit"} 0.000007
profiling_requester_summary_count{id="disk hit"} 9.000000
profiling_requester_summary_sum{id="found"} 0.007758
profiling_requester_summary_count{id="found"} 9041.000000
profiling_requester_summary_sum{id="pending : add peer"} 15855.479091
profiling_requester_summary_count{id="pending : add peer"} 142352.000000
#HELP profiling_rpc_server_counter Number of times a function is called
#TYPE profiling_rpc_server_counter counter
#HELP profiling_rpc_server_summary Number of times a function is called and total time passed in this function
#TYPE profiling_rpc_server_summary summary
#HELP profiling_store_counter Number of times a function is called
#TYPE profiling_store_counter counter
profiling_store_counter{id="may_split_context"} 2.000000
profiling_store_counter{id="write_new_target"} 3.000000
#HELP profiling_store_summary Number of times a function is called and total time passed in this function
#TYPE profiling_store_summary summary
profiling_store_summary_sum{id="Compute live blocks with new head"} 0.057341
profiling_store_summary_count{id="Compute live blocks with new head"} 186.000000
profiling_store_summary_sum{id="compute live blocks"} 0.059064
profiling_store_summary_count{id="compute live blocks"} 207.000000
profiling_store_summary_sum{id="compute live blocks with alternative branch"} 0.454181
profiling_store_summary_count{id="compute live blocks with alternative branch"} 11.000000
profiling_store_summary_sum{id="compute live blocks with alternative head"} 0.009470
profiling_store_summary_count{id="compute live blocks with alternative head"} 20.000000
profiling_store_summary_sum{id="compute whole live blocks"} 0.049472
profiling_store_summary_count{id="compute whole live blocks"} 2.000000
profiling_store_summary_sum{id="get_pred_block"} 0.000724
profiling_store_summary_count{id="get_pred_block"} 206.000000
profiling_store_summary_sum{id="locked compute live blocks with cache"} 0.571231
profiling_store_summary_count{id="locked compute live blocks with cache"} 219.000000
profiling_store_summary_sum{id="may_split_context"} 0.001086
profiling_store_summary_count{id="may_split_context"} 204.000000
profiling_store_summary_sum{id="rollback livedata"} 0.003839
profiling_store_summary_count{id="rollback livedata"} 10.000000
profiling_store_summary_sum{id="set_head"} 0.612012
profiling_store_summary_count{id="set_head"} 206.000000
profiling_store_summary_sum{id="store_block"} 0.126678
profiling_store_summary_count{id="store_block"} 206.000000
profiling_store_summary_sum{id="updating live blocks"} 0.399231
profiling_store_summary_count{id="updating live blocks"} 206.000000
profiling_store_summary_sum{id="write_new_head"} 0.057209
profiling_store_summary_count{id="write_new_head"} 206.000000
profiling_store_summary_sum{id="write_new_target"} 0.000428
profiling_store_summary_count{id="write_new_target"} 203.000000
#HELP prometheus_logs_messages_total Total number of messages logged
#TYPE prometheus_logs_messages_total counter