diff --git a/src/bin_agnostic_baker/CHANGES.md b/src/bin_agnostic_baker/CHANGES.md new file mode 100644 index 0000000000000000000000000000000000000000..0c1d2d8153af3d8848df48b5364095c3e4da5719 --- /dev/null +++ b/src/bin_agnostic_baker/CHANGES.md @@ -0,0 +1,17 @@ +# Agnostic Baker Changelog + +- Add automatic patching mechanism [!16240](https://gitlab.com/tezos/tezos/-/merge_requests/16240) +- Improve existing documentation [!16491](https://gitlab.com/tezos/tezos/-/merge_requests/16491) +- Fix and improve tezt tests [!16492](https://gitlab.com/tezos/tezos/-/merge_requests/16492) +- Release agnostic baker as experimental [!16318](https://gitlab.com/tezos/tezos/-/merge_requests/16318) +- Agnostic baker uses generic watchdog [!15508](https://gitlab.com/tezos/tezos/-/merge_requests/15508) +- Change name into `octez-experimental-agnostic-baker` [!16434](https://gitlab.com/tezos/tezos/-/merge_requests/16434) +- Fix `--without-dal` issue in tezt tests [!16426](https://gitlab.com/tezos/tezos/-/merge_requests/16426) +- Add remote signer tezt tests [!16201](https://gitlab.com/tezos/tezos/-/merge_requests/16201) +- Refactoring and cleanup [!15325](https://gitlab.com/tezos/tezos/-/merge_requests/15324) +- Agnostic baker switches on new protocol [!15305](https://gitlab.com/tezos/tezos/-/merge_requests/15305) +- Logs improvement [!15072](https://gitlab.com/tezos/tezos/-/merge_requests/15072) +- Agnostic baker starts on known active protocol [!15071](https://gitlab.com/tezos/tezos/-/merge_requests/15071) +- Add monitoring of voting periods - [!15046](https://gitlab.com/tezos/tezos/-/merge_requests/15046) +- Prevent agnostic baker to be run without arguments - [!15202](https://gitlab.com/tezos/tezos/-/merge_requests/15202) +- Introduce dummy agnostic baker - [!15029](https://gitlab.com/tezos/tezos/-/merge_requests/15029) diff --git a/src/bin_agnostic_baker/README.md b/src/bin_agnostic_baker/README.md new file mode 100644 index 0000000000000000000000000000000000000000..54665d46c12390b1e7dc1f28034dfb45ed0f70ca --- /dev/null +++ b/src/bin_agnostic_baker/README.md @@ -0,0 +1,70 @@ +# Agnostic Baker (experimental) + +## Overview + +Agnostic Baker is a protocol-independent binary that dynamically selects the +appropriate baking binary based on the active protocol. It monitors the state of +the blockchain and automatically switches to the correct binary when a new +protocol is encountered (for example, during migrations, or at startup). + +It is designed to simplify the baking process for users, such that they will no +longer need to run two baker binaries at migration time. + +## Experimental purpose + +For now, the binary is continuously being developed and tested. This is the reason +why users are warned that the binary is experimental and that it should not be +used for real-life scenarios, for instance, baking on `mainnet`. + +## Usage + +To run the agnostic baker, the command line syntax is quite similar to the one +for the protocol-dependent baking binaries: + +```bash +./octez-experimental-agnostic-baker [OCTEZ-EXPERIMENTAL-AGNOSTIC-BAKER-COMMANDS] \ +-- [OCTEZ-BAKER-COMMANDS] +``` + +The `[OCTEZ-EXPERIMENTAL-AGNOSTIC-BAKER-COMMANDS]` list consists of arguments specific +to the agnostic baker binary and they include: + +- `--binaries-directory` : where to find the baker binaries to use +- `--endpoint` : endpoint to communicate with the connected node +- `--base-dir` : directory dedicated to the agnostic baker (that can contain logs +and configuration files) +-- `--help` : displays help information +-- `auto-patch-cli` : attempts to fix the CLI in case the baker binary arguments do not +obey the rules (specified in `baker_args_rules`) + +The `[OCTEZ-BAKER-COMMANDS]` list consists of all the arguments that can be used +for the specific protocol baking binary. To be more clear, if a user wants to use +the agnostic baker to replace a baking command which would be + +```bash +./octez-baker- [OCTEZ-BAKER-COMMANDS] +``` + +they can do this by using the same `[OCTEZ-BAKER-COMMANDS]` and let the agnostic +baker run the binary for ``, information obtained from the node. + +Notice that the two types of arguments are separated by a clear `--`. + +The automatic patching mechanism (specified by `--auto-patch-cli` flag) was added to solve +an existing issue: *potential incompatibility of command line arguments when swapping baking binaries*. +This happens because the `[OCTEZ-BAKER-COMMANDS]` is preserved when the baking binary is swapped, +therefore if there is an incompatibility between that list and the new protocol binary, the protocol +would be stopped. The patching approach relies on a list of hard-coded rules which, if not obeyed, +can be fixed (for instance if an argument is mandatory, it can be added with a default case, e.g. +`--liquidity-baking-toggle-vote` with default `pass`). The goal here is to increase the robustness +of the agnostic baker binary. + +## How it works + +1. **Initialization**: The daemon starts and connects to the specified Tezos node. +2. **Protocol Detection**: It fetches the currently active protocol. This is done via an RPC request on the `head` metadata against the Tezos node. +3. **Baker Selection**: Based on the active protocol, it selects the corresponding `octez-baker-` binary. +4. **Baker Execution**: The chosen binary is executed with the specified arguments +(`[OCTEZ-BAKER-COMMANDS]`). +5. **Chain Monitoring**: The daemon continuously monitors the chain for new blocks and protocol changes (based on the voting period). +6. **Protocol Updates**: If a new protocol is encountered, the daemon stops the current baker and starts a new one matching the new protocol. diff --git a/src/bin_agnostic_baker/agnostic_baker_events.ml b/src/bin_agnostic_baker/agnostic_baker_events.ml index cdb6585bf353e41550575bd349f0aae8fce65124..408afc2b28734fef961e45921dfb7c6a07678251 100644 --- a/src/bin_agnostic_baker/agnostic_baker_events.ml +++ b/src/bin_agnostic_baker/agnostic_baker_events.ml @@ -2,6 +2,7 @@ (* *) (* SPDX-License-Identifier: MIT *) (* Copyright (c) 2024 Nomadic Labs, *) +(* Copyright (c) 2025 TriliTech *) (* *) (*****************************************************************************) @@ -89,6 +90,30 @@ let period_status = ("period", string) ("remaining", int31) +let auto_patch_cli_activated = + declare_1 + ~section + ~alternative_color + ~level:Notice + ~name:"auto_patch_cli_activated" + ~msg: + "The automatic CLI patching for the agnostic baker has been activated. \ + This implies that the command line arguments will be checked against \ + the rules of the binary corresponding to the following protocol: \ + {proto_hash}" + ("proto_hash", Protocol_hash.encoding) + ~pp1:Protocol_hash.pp_short + +let solved_broken_rule = + declare_1 + ~section + ~alternative_color + ~level:Notice + ~name:"solved_broken_rule" + ~msg:"{solution_msg}" + ("solution_msg", string) + ~pp1:Format.pp_print_string + (* Warning *) let experimental_binary = declare_0 @@ -101,3 +126,13 @@ let experimental_binary = it is intended for testing purposes only. Please do not use it on \ `mainnet`." () + +let detected_broken_rule = + declare_1 + ~section + ~alternative_color + ~level:Warning + ~name:"detected_broken_rule" + ~msg:"{error_msg}" + ("error_msg", string) + ~pp1:Format.pp_print_string diff --git a/src/bin_agnostic_baker/baker_args_rules.ml b/src/bin_agnostic_baker/baker_args_rules.ml new file mode 100644 index 0000000000000000000000000000000000000000..b4cee7530f17c6101586ddfd8312e55c41358962 --- /dev/null +++ b/src/bin_agnostic_baker/baker_args_rules.ml @@ -0,0 +1,62 @@ +(*****************************************************************************) +(* *) +(* SPDX-License-Identifier: MIT *) +(* Copyright (c) 2025 TriliTech *) +(* *) +(*****************************************************************************) + +type t = + (* A mandatory rule which states that the baking argument [arg] must appear, + and if it doesn't, it will default to [default_value]. *) + | Mandatory of {arg : string; default_value : string} + (* A mandatory rule which states that one of the baking arguments from [args] + must appear, and if none does, the default is [default_arg]. *) + | One_of of {args : string list; default_arg : string} + +(* This list needs to be updated every time a new protocol is adopted + and every time a new baking argument rule is created in alpha/beta. *) +module Rules = struct + let mandatory_liquidity_rule = + Mandatory {arg = "--liquidity-baking-toggle-vote"; default_value = "pass"} + + let optional_dal_node_rule = + One_of + {args = ["--dal-node"; "--without-dal"]; default_arg = "--without-dal"} +end + +let warning_msg = function + | Mandatory {arg; _} -> Printf.sprintf "Missing mandatory argument: %s" arg + | One_of {args; _} -> + Printf.sprintf "One of [%s] must be provided" (String.concat "; " args) + +let solution_msg = function + | Mandatory {arg; default_value} -> + Printf.sprintf + "Adding mandatory argument `%s %s` to baking command" + arg + default_value + | One_of {args; default_arg} -> + Printf.sprintf + "Adding default argument `%s` to baking command. The other options are \ + `%s`." + default_arg + (String.concat "; " args) + +(** [is_broken_rule ~baker_args rule] checks if [rule] is violated within the + list of baking command arguments [~baker_args]. *) +let is_broken_rule ~baker_args rule = + match rule with + | Mandatory {arg; _} -> not (List.mem ~equal:String.equal arg baker_args) + | One_of {args; _} -> + not + (List.exists + (fun arg -> List.mem ~equal:String.equal arg baker_args) + args) + +let get_broken_rules ~baker_args rules = + List.filter (is_broken_rule ~baker_args) rules + +let solve_broken_rule ~baker_args rule = + match rule with + | Mandatory {arg; default_value} -> baker_args @ [arg; default_value] + | One_of {default_arg; _} -> baker_args @ [default_arg] diff --git a/src/bin_agnostic_baker/baker_args_rules.mli b/src/bin_agnostic_baker/baker_args_rules.mli new file mode 100644 index 0000000000000000000000000000000000000000..31ec7ea2e4eac5b40de76ed0fea7a36c99abc0d9 --- /dev/null +++ b/src/bin_agnostic_baker/baker_args_rules.mli @@ -0,0 +1,34 @@ +(*****************************************************************************) +(* *) +(* SPDX-License-Identifier: MIT *) +(* Copyright (c) 2025 TriliTech *) +(* *) +(*****************************************************************************) + +type t + +module Rules : sig + (** The [--liquidity-baking-toggle-vote] option must appear in the baking + command arguments, if it is not provided, its defaulted to ["pass"] *) + val mandatory_liquidity_rule : t + + (** Either one of the [--dal-node] or [--without-dal] options must appear + in the baking command arguments, otherwise the default is [--without-dal]. *) + val optional_dal_node_rule : t +end + +(** [warning_msg rule] returns a warning message regarding a broken [rule] *) +val warning_msg : t -> string + +(** [solution_msg rule] returns the message after solving the issue generated + by a broken [rule] *) +val solution_msg : t -> string + +(** [get_broken_rules ~baker_ags rules] returns all broken rules from + [rules] in the list of baker command arguments from [~baker_args]. *) +val get_broken_rules : baker_args:string list -> t list -> t list + +(** [solve_broken_rule ~baker_args broken_rule] takes a list of baker + command arguments [~baker_args] and returns a new list which solves + the issue encountered in the [broken_rule]. *) +val solve_broken_rule : baker_args:string list -> t -> string list diff --git a/src/bin_agnostic_baker/daemon.ml b/src/bin_agnostic_baker/daemon.ml index abfb1484ca47304cf4f5defa4f2ee5532aa478b6..c878af659f1515fc3653d36dec7034f7f8c0f53a 100644 --- a/src/bin_agnostic_baker/daemon.ml +++ b/src/bin_agnostic_baker/daemon.ml @@ -2,6 +2,7 @@ (* *) (* SPDX-License-Identifier: MIT *) (* Copyright (c) 2024 Nomadic Labs, *) +(* Copyright (c) 2025 TriliTech *) (* *) (*****************************************************************************) @@ -38,6 +39,7 @@ module type BAKER = sig Protocol_hash.t -> binaries_directory:string option -> baker_args:string trace -> + auto_patch_cli:bool -> (baker, tztrace) result Lwt.t end @@ -54,8 +56,45 @@ module MakeBaker (Name : Lwt_process_watchdog.NAME) : BAKER = struct let* () = Agnostic_baker_events.(emit stopping_baker) baker.protocol_hash in Watchdog.stop baker.process - let spawn_baker protocol_hash ~binaries_directory ~baker_args = + (** [check_rules ~baker_args ~rules] obtains the broken rules from [~rules] + in [~baker_args], warns the user of the broken rules, tells them how they + are going to get fixed, and then fixes them. *) + let check_rules ~baker_args ~rules = + let open Lwt_syntax in + let broken_rules = Baker_args_rules.get_broken_rules ~baker_args rules in + List.fold_left_s + (fun baker_args broken_rule -> + let* () = + Agnostic_baker_events.(emit detected_broken_rule) + (Baker_args_rules.warning_msg broken_rule) + in + let+ () = + Agnostic_baker_events.(emit solved_broken_rule) + (Baker_args_rules.solution_msg broken_rule) + in + Baker_args_rules.solve_broken_rule ~baker_args broken_rule) + baker_args + broken_rules + + (** [spawn_baker protocol_hash ~binaries_directory ~baker_args ~auto_patch_cli] + spawns a baker for the given [protocol_hash] using the [~binaries_directory] + as path for the baker binary and with [~baker_args] as command line arguments. + + If [~auto_patch_cli] argument is [true], then in case the baker command does + not "work" (i.e. does not obey the rules specified in `baker_args_rules`) regarding + its CLI arguments, then the command will pe patched automatically. *) + let spawn_baker protocol_hash ~binaries_directory ~baker_args ~auto_patch_cli + = let open Lwt_result_syntax in + let*! baker_args = + if auto_patch_cli then + let*! () = + Agnostic_baker_events.(emit auto_patch_cli_activated) protocol_hash + in + let rules = Parameters.protocol_baker_args_rules protocol_hash in + check_rules ~baker_args ~rules + else Lwt.return baker_args + in let args_as_string = Format.asprintf "%a" @@ -98,11 +137,15 @@ type 'a state = { binaries_directory : string option; node_endpoint : string; baker_args : string list; + auto_patch_cli : bool; mutable current_baker : (baker_instance * baker) option; } type 'a t = 'a state +(** [monitor_heads ~node_addr] creates a stream which returns the data + of the heads of the current network; this information is received + from the RPC calls at the endpoint given by [~node_addr]. *) let monitor_heads ~node_addr = let open Lwt_result_syntax in let uri = Format.sprintf "%s/monitor/heads/main" node_addr in @@ -127,6 +170,10 @@ let monitor_heads ~node_addr = ignore (loop () : unit Lwt.t) ; return stream +(** [hot_swap_baker ~state ~next_protocol_hash] performs a swap in the current + [~state] of the agnostic baker, exchanging the current baker with the one + corresponding to [~next_protocol_hash]. This is done by shutting down the + current baking binary and generating the new binary instead. *) let hot_swap_baker ~state ~next_protocol_hash = let open Lwt_result_syntax in let* (module CurrentBaker : BAKER), current_baker = @@ -162,12 +209,17 @@ let hot_swap_baker ~state ~next_protocol_hash = next_protocol_hash ~binaries_directory:state.binaries_directory ~baker_args:state.baker_args + ~auto_patch_cli:state.auto_patch_cli in return (Baker (module NewBaker), new_baker) in state.current_baker <- Some new_baker ; return_unit +(** [monitor_voting_periods ~state head_stream] creates a process which listens + to the [head_stream] stream (which returns the data of the heads of the network + chain) in order to know when to "hot swap" (fork) the current protocol baking + binary with the one associated with the next protocol. *) let monitor_voting_periods ~state head_stream = let open Lwt_result_syntax in let node_addr = state.node_endpoint in @@ -200,11 +252,11 @@ let monitor_voting_periods ~state head_stream = let* () = loop () in return_unit -(* Aims to start the baker associated to the current protocol. If - the protocol is considered as frozen (not active anymore), and - there is thus no actual baker binary anymore, the initial phase - consist in waiting until an active protocol is observed on - monitored heads. *) +(** [may_start_initial_baker state] aims to start the baker associated + to the current protocol. If the protocol is considered as [frozen] (not + [active] anymore), and there is thus no actual baker binary anymore, the + initial phase consists in waiting until an [active] protocol is observed on + monitored heads function. *) let may_start_initial_baker state = let open Lwt_result_syntax in let*! () = Agnostic_baker_events.(emit experimental_binary) () in @@ -240,6 +292,7 @@ let may_start_initial_baker state = protocol_hash ~binaries_directory:state.binaries_directory ~baker_args:state.baker_args + ~auto_patch_cli:state.auto_patch_cli in return (Baker (module NewBaker), new_baker) in @@ -270,8 +323,14 @@ let may_start_initial_baker state = in may_start ~head_stream:None () -let create ~binaries_directory ~node_endpoint ~baker_args = - {binaries_directory; node_endpoint; baker_args; current_baker = None} +let create ~binaries_directory ~node_endpoint ~baker_args ~auto_patch_cli = + { + binaries_directory; + node_endpoint; + baker_args; + auto_patch_cli; + current_baker = None; + } let run state = let open Lwt_result_syntax in diff --git a/src/bin_agnostic_baker/daemon.mli b/src/bin_agnostic_baker/daemon.mli index 3cdc92dd9ab7b3967633366319ca430b8d266bc1..d5ce5b5da4f2e8bdc04c0eb089c40569d8226d72 100644 --- a/src/bin_agnostic_baker/daemon.mli +++ b/src/bin_agnostic_baker/daemon.mli @@ -5,18 +5,31 @@ (* *) (*****************************************************************************) -(** Daemon handling the bakers life cycle. *) +(** Daemon handling the baker's life cycle. + +It is used to [create] and [run] a protocol-agnostic process which uses the existing +baking binaries in an adaptive way, depending on the current protocol obtained +from the chain. + +It relies on a [state] which contains the [endpoint] to contact the running node, +together with the current baker which is being run. + +To do so, it also spawns a "monitoring" process which follows the heads of the +chain, as reported by the node from the [state], more precisely which monitors +the voting period. By doing that, it decides when to swap to a different baking +binary. *) type 'a t -(** [create binaries_directory node_endpoint baker_args] returns a non - initialized daemon.*) +(** [create ~binaries_directory ~node_endpoint ~baker_args ~auto_patch_cli] + returns a non initialized daemon.*) val create : binaries_directory:string option -> node_endpoint:string -> baker_args:string trace -> + auto_patch_cli:bool -> 'a t (** [run t] Runs the daemon responsible for the spawn/stop of the - baker daemons. *) + baker daemons. *) val run : 'a t -> unit tzresult Lwt.t diff --git a/src/bin_agnostic_baker/main_agnostic_baker.ml b/src/bin_agnostic_baker/main_agnostic_baker.ml index ac627582e02caf07af369ba9e87aa54f4ae1d8db..2c78355393519057f5fa23d9d186d7be285341f1 100644 --- a/src/bin_agnostic_baker/main_agnostic_baker.ml +++ b/src/bin_agnostic_baker/main_agnostic_baker.ml @@ -7,7 +7,14 @@ let run () = let open Lwt_result_syntax in - let Run_args.{node_endpoint; base_dir; binaries_directory; baker_args} = + let Run_args. + { + node_endpoint; + base_dir; + binaries_directory; + baker_args; + auto_patch_cli; + } = Run_args.parse_args Sys.argv in let*! () = @@ -15,8 +22,10 @@ let run () = ~config:(Parameters.log_config ~base_dir) () in - let daemon = Daemon.create ~binaries_directory ~node_endpoint ~baker_args in - let* (_ : unit) = Daemon.run daemon in + let daemon = + Daemon.create ~binaries_directory ~node_endpoint ~baker_args ~auto_patch_cli + in + let* () = Daemon.run daemon in let*! () = Lwt_utils.never_ending () in return_unit diff --git a/src/bin_agnostic_baker/parameters.ml b/src/bin_agnostic_baker/parameters.ml index c985a297d8c693d69a194d2389a71faf67488ca7..795953dbe37843890211199d758c24e99c8a034c 100644 --- a/src/bin_agnostic_baker/parameters.ml +++ b/src/bin_agnostic_baker/parameters.ml @@ -2,6 +2,7 @@ (* *) (* SPDX-License-Identifier: MIT *) (* Copyright (c) 2024 Nomadic Labs, *) +(* Copyright (c) 2025 TriliTech *) (* *) (*****************************************************************************) @@ -64,7 +65,13 @@ let status_encoding = (fun () -> Frozen); ]) -(* From Manifest/Product_octez/Protocol*) +type protocol_info = { + short_hash : string; + status : status; + baker_args_rules : Baker_args_rules.t list; +} + +(* From Manifest/Product_octez/Protocol *) let protocol_info = function | ( "ProtoGenesisGenesisGenesisGenesisGenesisGenesk612im" | "Ps9mPmXaRzmzk35gbAYNCAw6UXdE2qoABTHbN2oEEc1qM7CwT9P" @@ -88,14 +95,41 @@ let protocol_info = function | "PtMumbai2TmsJHNGRkD8v8YDbtao7BLUC3wjASn1inAKLFCjaH1" | "PtNairobiyssHuh87hEhfVBGCVrK3WnS8Z2FT4ymB5tAa4r1nQf" | "ProxfordYmVfjWnRcgjWH36fW6PArwqykTFzotUxRs6gmTcZDuH" - | "PtParisBxoLz5gzMmn3d9WBQNoPSZakgnkMC2VNuQ3KXfUtUQeZ" ) as full_hash -> - (String.sub full_hash 0 8, Frozen) - | ( "PsParisCZo7KAh1Z1smVd9ZMZ1HHn5gkzbM94V3PLCpknFWhUAi" - | "PsQuebecnLByd3JwTiGadoG4nGWi3HYiLXUjkibeFV8dCFeVMUg" ) as full_hash -> - (String.sub full_hash 0 8, Active) - | "ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK" -> ("alpha", Active) - | _ -> (*We assume that unmatched protocols are beta ones*) ("beta", Active) + | "PtParisBxoLz5gzMmn3d9WBQNoPSZakgnkMC2VNuQ3KXfUtUQeZ" + | "PsParisCZo7KAh1Z1smVd9ZMZ1HHn5gkzbM94V3PLCpknFWhUAi" ) as full_hash -> + { + short_hash = String.sub full_hash 0 8; + status = Frozen; + baker_args_rules = []; + } + | "PsQuebecnLByd3JwTiGadoG4nGWi3HYiLXUjkibeFV8dCFeVMUg" as full_hash -> + { + short_hash = String.sub full_hash 0 8; + status = Active; + baker_args_rules = [Baker_args_rules.Rules.mandatory_liquidity_rule]; + } + | "ProtoALphaALphaALphaALphaALphaALphaALphaALphaDdp3zK" -> + { + short_hash = "alpha"; + status = Active; + baker_args_rules = + Baker_args_rules.Rules. + [mandatory_liquidity_rule; optional_dal_node_rule]; + } + | _ -> + (* We assume that unmatched protocols are next ones *) + { + short_hash = "next"; + status = Active; + baker_args_rules = + Baker_args_rules.Rules. + [mandatory_liquidity_rule; optional_dal_node_rule]; + } + +let protocol_short_hash h = + (protocol_info (Protocol_hash.to_b58check h)).short_hash -let protocol_short_hash h = fst (protocol_info (Protocol_hash.to_b58check h)) +let protocol_status h = (protocol_info (Protocol_hash.to_b58check h)).status -let protocol_status h = snd (protocol_info (Protocol_hash.to_b58check h)) +let protocol_baker_args_rules h = + (protocol_info (Protocol_hash.to_b58check h)).baker_args_rules diff --git a/src/bin_agnostic_baker/parameters.mli b/src/bin_agnostic_baker/parameters.mli index 1b57b0d250ec0ee35d70dba317c5df5faf4b4603..20d1342e17e8ef2b53d3f6a10f8a695c4b525a98 100644 --- a/src/bin_agnostic_baker/parameters.mli +++ b/src/bin_agnostic_baker/parameters.mli @@ -2,6 +2,7 @@ (* *) (* SPDX-License-Identifier: MIT *) (* Copyright (c) 2024 Nomadic Labs, *) +(* Copyright (c) 2025 TriliTech *) (* *) (*****************************************************************************) @@ -12,9 +13,9 @@ val default_node_endpoint : string val log_config : base_dir:string option -> Tezos_base.Internal_event_config.t (** Status of a protocol, based on Manifest/Product_octez/Protocol. A - protocol is considered as Active while it is running on a network, + protocol is considered as [Active] while it is running on a network, and thus, have dedicated binaries. Otherwise, the protocol is - Frozen as not running anymore and no associated binaries. + [Frozen] as not running anymore and no associated binaries. Warning, it is needed to update status for each new protocol added. *) @@ -27,3 +28,5 @@ val status_encoding : status t val protocol_short_hash : Protocol_hash.t -> string val protocol_status : Protocol_hash.t -> status + +val protocol_baker_args_rules : Protocol_hash.t -> Baker_args_rules.t trace diff --git a/src/bin_agnostic_baker/rpc_services.ml b/src/bin_agnostic_baker/rpc_services.ml index 67866da85720bc131b934912e207d0fb70a73729..725f47615932994d3c7b851ac9a7eb3e96c685f3 100644 --- a/src/bin_agnostic_baker/rpc_services.ml +++ b/src/bin_agnostic_baker/rpc_services.ml @@ -19,6 +19,9 @@ let request_uri ~node_addr ~uri = tzfail (Cannot_connect_to_node node_addr) | e -> raise e) +(** [call_and_wrap_rpc ~node_addr ~uri ~f] makes the RPC call given + by the [~uri] against [~node_addr], and in case of a well-formed + response, it applies [~f] to it. *) let call_and_wrap_rpc ~node_addr ~uri ~f = let open Lwt_result_syntax in let* resp, body = request_uri ~node_addr ~uri in diff --git a/src/bin_agnostic_baker/rpc_services.mli b/src/bin_agnostic_baker/rpc_services.mli index 5b962315d3c949cef73b9c831ce9363c836f6f66..bdf44fb647fb1216ffd51a49038d5922479db174 100644 --- a/src/bin_agnostic_baker/rpc_services.mli +++ b/src/bin_agnostic_baker/rpc_services.mli @@ -5,9 +5,9 @@ (* *) (*****************************************************************************) -(** request_uri ~node_addr ~uri] is a raw call that will return the - Cohttp response of a RPC call, given a [uri], against the - [node_addr]. *) +(** [request_uri ~node_addr ~uri] is a raw call that will return the + Cohttp response of an RPC call, given a [~uri], against the + [~node_addr]. *) val request_uri : node_addr:string -> uri:string -> @@ -18,12 +18,12 @@ val request_uri : block. *) val get_next_protocol_hash : node_addr:string -> Protocol_hash.t tzresult Lwt.t -(** [get_next_protocol_hash ~node_addr] returns the protocol hash of +(** [get_current_proposal ~node_addr] returns the protocol hash of the current voting period, if any. *) val get_current_proposal : node_addr:string -> Protocol_hash.t option tzresult Lwt.t -(** [get_next_protocol_hash ~node_addr] returns the current voting - period in addition to the number of remaining block until the end +(** [get_current_period ~node_addr] returns the current voting + period in addition to the number of remaining blocks until the end of the period. *) val get_current_period : node_addr:string -> (string * int) tzresult Lwt.t diff --git a/src/bin_agnostic_baker/run_args.ml b/src/bin_agnostic_baker/run_args.ml index 7b7654d93d299e2876170dd16311686ee6d542bb..f66b9373f335ebfd4456968d7db0a8d90e0c7bc8 100644 --- a/src/bin_agnostic_baker/run_args.ml +++ b/src/bin_agnostic_baker/run_args.ml @@ -15,6 +15,8 @@ let base_dir_arg = "--base-dir" let base_dir_short_arg = "--b" +let auto_patch_cli_arg = "--auto-patch-cli" + let help_arg = "--help" let print_help () = @@ -26,9 +28,11 @@ let print_help () = Format.printf "OCTEZ-EXPERIMENTAL-AGNOSTIC-BAKER-COMMANDS:\n\ \ %s: display help\n\ - \ %s: path to the octez-baker binaries@.@." + \ %s: path to the octez-baker binaries\n\ + \ %s: try to automatically patch the CLI errors@.@." help_arg - binaries_directory_arg ; + binaries_directory_arg + auto_patch_cli_arg ; Format.printf "OCTEZ-BAKER-COMMANDS:\n Run ./octez-baker- --help@." @@ -44,6 +48,9 @@ let version_cmd args = exit 0) else () +let auto_patch_cli_cmd args = + List.mem ~equal:String.equal auto_patch_cli_arg args + let split_args ?(on = "--") = let rec loop acc = function | [] -> (List.rev acc, []) @@ -79,6 +86,7 @@ type args = { base_dir : string option; binaries_directory : string option; baker_args : string list; + auto_patch_cli : bool; } let parse_args all_args = @@ -97,5 +105,6 @@ let parse_args all_args = (get_endpoint baker_args) in let binaries_directory = get_binaries_directory agnostic_baker_args in + let auto_patch_cli = auto_patch_cli_cmd agnostic_baker_args in let base_dir = get_base_dir baker_args in - {node_endpoint; base_dir; binaries_directory; baker_args} + {node_endpoint; base_dir; binaries_directory; baker_args; auto_patch_cli} diff --git a/src/bin_agnostic_baker/run_args.mli b/src/bin_agnostic_baker/run_args.mli index ff420d04bb7dc891fa2708666190ace456fcebb6..d137a880bf023e81738c9a0373c1ccde9f72f55d 100644 --- a/src/bin_agnostic_baker/run_args.mli +++ b/src/bin_agnostic_baker/run_args.mli @@ -11,9 +11,10 @@ type args = { base_dir : string option; binaries_directory : string option; baker_args : string list; + auto_patch_cli : bool; } -(** [parse_args args] is a raw utility that aims to parse the give +(** [parse_args args] is a raw utility that aims to parse the given arguments from the command line and to return, respectively, the - endpoint, base_dir, binaries_directory and baker_args. *) + [endpoint], [base_dir], [binaries_directory] and [baker_args]. *) val parse_args : string array -> args diff --git a/tezt/lib_tezos/agnostic_baker.ml b/tezt/lib_tezos/agnostic_baker.ml index 6543fe5102f4fb74559a3166cab243805f5b1f9f..f8241989c9faf58328a1296e38bff0bd45d06649 100644 --- a/tezt/lib_tezos/agnostic_baker.ml +++ b/tezt/lib_tezos/agnostic_baker.ml @@ -23,10 +23,10 @@ module Parameters = struct base_dir : string; node_data_dir : string; node_rpc_endpoint : Endpoint.t; + auto_patch_cli : bool; mutable pending_ready : unit option Lwt.u list; remote_mode : bool; liquidity_baking_toggle_vote : liquidity_baking_vote option; - use_dal_node : string option; } type session_state = {mutable ready : bool} @@ -53,8 +53,8 @@ let set_ready agnostic_baker = let create_from_uris ?runner ?(path = Uses.path Constant.octez_experimental_agnostic_baker) ?name ?color ?event_pipe ?(delegates = []) ?(remote_mode = false) - ?(liquidity_baking_toggle_vote = Some Pass) ?use_dal_node ~base_dir - ~node_data_dir ~node_rpc_endpoint () = + ?(liquidity_baking_toggle_vote = Some Pass) ~base_dir ~node_data_dir + ~node_rpc_endpoint ~auto_patch_cli () = let agnostic_baker = create ~path @@ -68,10 +68,10 @@ let create_from_uris ?runner base_dir; node_data_dir; node_rpc_endpoint; + auto_patch_cli; pending_ready = []; remote_mode; liquidity_baking_toggle_vote; - use_dal_node; } in agnostic_baker @@ -81,7 +81,7 @@ let handle_event node ({name; _} : event) = let create ?runner ?path ?name ?color ?event_pipe ?(delegates = []) ?(remote_mode = false) ?(liquidity_baking_toggle_vote = Some Pass) - ?use_dal_node node client = + ?(auto_patch_cli = false) node client = let agnostic_baker = create_from_uris ?runner @@ -89,13 +89,13 @@ let create ?runner ?path ?name ?color ?event_pipe ?(delegates = []) ?name ?color ?event_pipe - ?use_dal_node ~delegates ~remote_mode ~liquidity_baking_toggle_vote ~base_dir:(Client.base_dir client) ~node_data_dir:(Node.data_dir node) ~node_rpc_endpoint:(Node.as_rpc_endpoint node) + ~auto_patch_cli () in on_event agnostic_baker (handle_event agnostic_baker) ; @@ -113,6 +113,10 @@ let run ?event_level ?event_sections_levels (agnostic_baker : t) = let node_addr = Endpoint.as_string agnostic_baker.persistent_state.node_rpc_endpoint in + let auto_patch_cli_arg = + if agnostic_baker.persistent_state.auto_patch_cli then ["--auto-patch-cli"] + else [] + in let run_args = if agnostic_baker.persistent_state.remote_mode then ["remotely"] else ["with"; "local"; "node"; node_data_dir] @@ -123,14 +127,10 @@ let run ?event_level ?event_sections_levels (agnostic_baker : t) = liquidity_baking_vote_to_string agnostic_baker.persistent_state.liquidity_baking_toggle_vote in - let use_dal_node = - match agnostic_baker.persistent_state.use_dal_node with - | None -> ["--without-dal"] - | Some endpoint -> ["--dal-node"; endpoint] - in let arguments = - ["--"; "--endpoint"; node_addr; "--base-dir"; base_dir; "run"] - @ run_args @ delegates @ liquidity_baking_toggle_vote @ use_dal_node + auto_patch_cli_arg + @ ["--"; "--endpoint"; node_addr; "--base-dir"; base_dir; "run"] + @ run_args @ delegates @ liquidity_baking_toggle_vote in let on_terminate _ = @@ -167,7 +167,7 @@ let wait_for_ready agnostic_baker = let init ?runner ?(path = Uses.path Constant.octez_experimental_agnostic_baker) ?name ?color ?event_level ?event_pipe ?event_sections_levels - ?(delegates = []) ?remote_mode ?liquidity_baking_toggle_vote ?use_dal_node + ?(delegates = []) ?remote_mode ?liquidity_baking_toggle_vote ?auto_patch_cli node client = let* () = Node.wait_for_ready node in let agnostic_baker = @@ -179,7 +179,7 @@ let init ?runner ?(path = Uses.path Constant.octez_experimental_agnostic_baker) ?event_pipe ?remote_mode ?liquidity_baking_toggle_vote - ?use_dal_node + ?auto_patch_cli ~delegates node client diff --git a/tezt/lib_tezos/agnostic_baker.mli b/tezt/lib_tezos/agnostic_baker.mli index 183c2b134cc64e5e84c17dd369b0739b1236c039..a9bb00a4810f7bc378baf90909a9eb05bde594d7 100644 --- a/tezt/lib_tezos/agnostic_baker.mli +++ b/tezt/lib_tezos/agnostic_baker.mli @@ -95,10 +95,9 @@ val protocol_status : Protocol.t -> protocol_status then [--liquidity-baking-toggle-vote x] is passed. The default value is [Some Pass]. - [use_dal_node] is passed to the agnostic baker daemon through the - [--without-dal] or [--dal-node ] flags. If the flag is [None], the - former option is passed, otherwise if the flag is [Some ], then the - latter option is passed. The default value is [None]. + [auto_patch_cli] is a flag used for an automatic mechanism which tries to correct the + command line call of the baker, to obey the rules specific to the baker binary (this + can vary depending on the corresponding protocol). If [remote_mode] is specified, the agnostic baker will run in RPC-only mode. *) @@ -111,7 +110,7 @@ val create : ?delegates:string list -> ?remote_mode:bool -> ?liquidity_baking_toggle_vote:liquidity_baking_vote option -> - ?use_dal_node:string -> + ?auto_patch_cli:bool -> Node.t -> Client.t -> t @@ -139,14 +138,14 @@ val create_from_uris : ?delegates:string list -> ?remote_mode:bool -> ?liquidity_baking_toggle_vote:liquidity_baking_vote option -> - ?use_dal_node:string -> base_dir:string -> node_data_dir:string -> node_rpc_endpoint:Endpoint.t -> + auto_patch_cli:bool -> unit -> t -(** Initialize an agnositc baker. +(** Initialize an agnostic baker. This creates agnostic baker, waits for it to be ready, and then returns it. @@ -193,7 +192,7 @@ val init : ?delegates:string list -> ?remote_mode:bool -> ?liquidity_baking_toggle_vote:liquidity_baking_vote option -> - ?use_dal_node:string -> + ?auto_patch_cli:bool -> Node.t -> Client.t -> t Lwt.t diff --git a/tezt/tests/agnostic_baker_test.ml b/tezt/tests/agnostic_baker_test.ml index a9d2bcb9fb8fc94b917a12792917964714ef23d1..942177e015aecca146586457cf85d8519c4ff4a7 100644 --- a/tezt/tests/agnostic_baker_test.ml +++ b/tezt/tests/agnostic_baker_test.ml @@ -10,6 +10,15 @@ Component: Agnostic baker (octez-experimental-agnostic-baker) Invocation: dune exec tezt/tests/main.exe -- --file agnostic_baker_test.ml Subject: Ensure that the agnostic baker behaves as expected + + Tests + ----- + - each migration test starts from an [Active] protocol (which has a baker binary) + and migrates to the next protocol + - each migration test can either use remote signing or not + - each migration test can either be a resilience or normal test; the former testing + that killing the baker during migration results in a graceful restart + - there is also a start and stop test *) let team = Tag.layer1 @@ -20,6 +29,12 @@ let wait_for_active_protocol_waiting agnostic_baker = "waiting_for_active_protocol.v0" (fun _json -> Some ()) +let wait_for_auto_patch agnostic_baker = + Agnostic_baker.wait_for + agnostic_baker + "auto_patch_cli_activated.v0" + (fun _json -> Some ()) + let wait_for_baker_process agnostic_baker protocol_short_hash = Agnostic_baker.wait_for agnostic_baker @@ -33,8 +48,8 @@ let wait_for_baker_process agnostic_baker protocol_short_hash = test, making sure it is restarted as expected. *) let perform_protocol_migration ?(resilience_test = false) ?node_name ?client_name ?parameter_file ?(use_remote_signer = false) ~blocks_per_cycle - ~migration_level ~migrate_from ~migrate_to ~baked_blocks_after_migration () - = + ~migration_level ~migrate_from ~migrate_to ~baked_blocks_after_migration + ~auto_patch_cli () = assert (migration_level >= blocks_per_cycle) ; Log.info "Node starting" ; let* client, node = @@ -62,7 +77,7 @@ let perform_protocol_migration ?(resilience_test = false) ?node_name else unit in Log.info "Node %s initialized" (Node.name node) ; - let baker = Agnostic_baker.create node client in + let baker = Agnostic_baker.create node client ~auto_patch_cli in let wait_for_active_protocol_waiting = wait_for_active_protocol_waiting baker in @@ -169,24 +184,27 @@ let perform_protocol_migration ?(resilience_test = false) ?node_name let* () = Agnostic_baker.terminate baker in unit -let raw_migrate ~resilience_test ?(use_remote_signer = false) = +let raw_migrate ~migrate_from ~migrate_to ~resilience_test ~use_remote_signer + ~auto_patch_cli = let remote_signer_text, remote_signer = if use_remote_signer then (" using HTTP remote signer", [Constant.octez_signer]) else ("", []) in - let parameters = JSON.parse_file (Protocol.parameter_file Protocol.Alpha) in - Protocol.register_test + let parameters = JSON.parse_file (Protocol.parameter_file migrate_to) in + Test.register ~__FILE__ ~title: (Format.asprintf - "Protocol migration from protocol to alpha with agnostic baker%s %s" + "Protocol migration from protocol %s to protocol %s with agnostic \ + baker%s %s" + (Protocol.tag migrate_from) + (Protocol.tag migrate_to) remote_signer_text (if resilience_test then "and resilience test" else "")) - ~tags:["protocol"; "migration"; "agnostic"; "baker"; Tag.ci_disabled] - ~uses:(fun _protocol -> - [Constant.octez_experimental_agnostic_baker] @ remote_signer) - @@ fun protocol -> + ~tags:["protocol"; "migration"; "agnostic"; "baker"] + ~uses:([Constant.octez_experimental_agnostic_baker] @ remote_signer) + @@ fun () -> let blocks_per_cycle = JSON.(get "blocks_per_cycle" parameters |> as_int) in let consensus_rights_delay = JSON.(get "consensus_rights_delay" parameters |> as_int) @@ -198,41 +216,44 @@ let raw_migrate ~resilience_test ?(use_remote_signer = false) = ~use_remote_signer ~blocks_per_cycle ~migration_level - ~migrate_from:protocol - ~migrate_to:Protocol.Alpha + ~migrate_from + ~migrate_to ~baked_blocks_after_migration: (2 * consensus_rights_delay * blocks_per_cycle) + ~auto_patch_cli () in unit -let start_stop = - Protocol.register_test +let migrate = raw_migrate ~resilience_test:false ~auto_patch_cli:true + +let migrate_with_resilience_test = + raw_migrate ~resilience_test:true ~auto_patch_cli:true + +let register ~migrate_from ~migrate_to = + (* We want to migrate only from Active protocols *) + if Agnostic_baker.protocol_status migrate_from = Active then ( + migrate ~migrate_from ~migrate_to ~use_remote_signer:false ; + migrate ~migrate_from ~migrate_to ~use_remote_signer:true ; + migrate_with_resilience_test + ~migrate_from + ~migrate_to + ~use_remote_signer:false ; + migrate_with_resilience_test + ~migrate_from + ~migrate_to + ~use_remote_signer:true) + else () + +let register_protocol_independent () = + Test.register ~__FILE__ ~title:"Agnostic baker starts and stops" - ~tags:[team; "sandbox"; "agnostic"; "baker"; "init"; Tag.ci_disabled] - ~uses:(fun _protocol -> [Constant.octez_experimental_agnostic_baker]) - @@ fun _protocol -> + ~tags:[team; "sandbox"; "agnostic"; "baker"; "init"] + ~uses:[Constant.octez_experimental_agnostic_baker] + @@ fun () -> let* node, client = Client.init_with_node `Client () in let* baker = Agnostic_baker.init node client in let* () = Agnostic_baker.wait_for_ready baker in let* () = Agnostic_baker.terminate baker in unit - -let migrate = raw_migrate ~resilience_test:false - -let migrate_with_resilience_test = raw_migrate ~resilience_test:true - -let register ~protocols = - start_stop protocols ; - let filter_migratable_protocols protocol = - (* We want to migrate only from Active protocols *) - Agnostic_baker.protocol_status protocol = Active - in - let migratable_protocols = - List.filter filter_migratable_protocols protocols - in - migrate ~use_remote_signer:false migratable_protocols ; - migrate ~use_remote_signer:true migratable_protocols ; - migrate_with_resilience_test ~use_remote_signer:false migratable_protocols ; - migrate_with_resilience_test ~use_remote_signer:true migratable_protocols diff --git a/tezt/tests/main.ml b/tezt/tests/main.ml index f7d324e2ec3e970f5f388ba53b53d3a360bf5372..5635194bfd950c15469ea6ea80d5f7036e55250a 100644 --- a/tezt/tests/main.ml +++ b/tezt/tests/main.ml @@ -44,6 +44,7 @@ let alpha_can_stitch_from_its_predecessor = false (* Tests that are protocol-independent. They do not take a protocol as a parameter and thus need to be registered only once. *) let register_protocol_independent_tests () = + Agnostic_baker_test.register_protocol_independent () ; Binaries.register_protocol_independent () ; Bootstrap.register_protocol_independent () ; Cli_tezos.register_protocol_independent () ; @@ -68,6 +69,7 @@ let register_protocol_independent_tests () = (* Tests related to protocol migration. *) let register_protocol_migration_tests () = let migrate_from = Option.get @@ Protocol.previous_protocol migrate_to in + Agnostic_baker_test.register ~migrate_from ~migrate_to ; Mockup.register_constant_migration ~migrate_from ~migrate_to ; Protocol_migration.register ~migrate_from ~migrate_to ; Weeklynet.register () ; @@ -100,6 +102,7 @@ let register_old_protocol_migration_tests () = | _, Alpha -> () (* Already in register_protocol_migration_tests *) | None, _ -> () | Some migrate_from, migrate_to -> + Agnostic_baker_test.register ~migrate_from ~migrate_to ; Sc_rollup_migration.register ~migrate_from ~migrate_to) Protocol.all @@ -114,7 +117,6 @@ let register_old_protocol_migration_tests () = let register_protocol_tests_that_use_supports_correctly () = let protocols = Protocol.all in Adaptive_issuance.register ~protocols ; - Agnostic_baker_test.register ~protocols ; Bad_annot.register ~protocols ; Bad_indentation.register ~protocols ; Baker_test.register ~protocols ;