DE102022201702A1 - Computer-implemented method for protecting a machine learning model against deployment relocation, computer program, data carrier signal and automated transport system for transporting people and/or objects - Google Patents

Abstract

Computer-implemented method for protecting a machine learning model (M) against a deployment shift, comprising the steps: for frames on which the machine learning model (M) infers in a deployment phase, comprising frames from second domains (D2) with a respective domain shift (a) relative to the first domain (D1), in each case determining an epistemic uncertainty (c), forward paths of the machine learning model (M) being sequenced along a plurality of the individual images (V2); Using the epistemic uncertainty (c) as a measure of the performance (P) of the machine learning model (M) (V3); perceiving the deployment shift as a given decrease in performance (V4); during the deployment shift, feedback to a system (S) in which the machine learning model (M) is deployed in the deployment phase, that the machine learning model (M) infers (V5) outside of the first domain (D1).

Description

The invention relates to a computer-implemented method for protecting a machine learning model against a shift in use, a computer program, a data carrier signal and an automated transport system for transporting people and/or objects.

The following definitions, descriptions, and implementations retain their respective meaning and apply to the entire subject matter disclosed.

A machine learning model, for example an artificial neural network, for example a convolutional network, is usually trained with training data, with weights of connections of neurons being adjusted during this training process, for example using gradient descent methods, in order to minimize a loss function of the machine learning model. With this, predictions on the training data are learned from the training data. After the training, the machine learning model is validated with validation data, i.e. prediction errors of the machine learning model are minimized. For example, a data set can be divided into training data and validation data, where the ratio can be, for example, 70:30, 80:20 or 90:10. After the validation process, the machine learning model is tested for generalization with test data to avoid overfitting to the training data.

After the training process, the weights, possibly also other parameters, of the machine learning model are frozen for use of the machine learning model, ie for provision in a product or system. The deployment is based on the assumption that the data that the machine learning model processed during training, validation and/or testing and the data that the machine learning model processes during inference in the deployment are from the same domain.

However, in real applications, such as autonomous driving, domain shifts, also known as domain shifts or distributional shifts, i.e. changes in the shared distribution of input data, are to be expected at a certain point in time. Since the subject matter of the invention relates to the use, also referred to as deployment, of a machine learning model, the effect of domain shifts is referred to below as deployment shift. The deployment shift problem occurs when the product or system ignores that it left an initial domain in which the deployed machine learning model was trained, validated, and/or tested, resulting in a catastrophic drop in the performance of the machine learning model in the deployment , also called catastrophic deployment shift, catastrophic dropping in deployment performance or out of domain performance dropping under deployment. A catastrophic dropping in deployment performance occurs, for example, when a machine learning model used in a vehicle to classify objects, which was trained, validated and/or tested on daytime images, now enters the night domain and cannot recognize other vehicles there. The first domain is also called the source domain, for example the tag images belong to the source domain. The night domain is also called target domain in this example. If the domain shift between source and target domain is too large, the state of the system is critical to security.

Methods are known in the prior art for learning domain shifts in the training process. However, learning about domain shifts in the training process requires a great deal of effort, time and high costs in order to label new areas in the training data. The teachings of the DE 10 2019 211 672 A1 or DE 10 2020 200 998 A1 create.

The DE 10 2019 211 672 A1 discloses a training method for an artificial neural network, where the method can be referred to as self-supervised continuous domain adaptation and where training data from domains with relatively small domain shifts are processed continuously, for example to continuously learn the domain shift from day to night.

The DE 10 2020 200 998 A1 refers to a so-called catastrophic forgetting in the training process and discloses a so-called self-supervised continuous domain adaptation without catastrophic forgetting, whereby the catastrophic forgetting is circumvented by a common feature space for feature extraction and domain classification as an indication for a domain-specific, further classification.

The US 2020/0133281 A1 discloses a safety system for a vehicle wherein uncertainty data is determined indicative of uncertainty in one or more predictions of a driving model during operation of a vehicle.

The U.S. 9,406,017 B2 discloses a system for training an artificial neural network. A switch is connected to weights in at least some layers of the network. In each training epoch, the switch selectively turns off each of the weights according to a preconfigured probability. This process is known as dropout.

However, continuous validation is not always possible and this does not solve the problem of recognizing and notifying shifts in deployment. The object of the invention was how a machine learning model that has already been trained, validated and/or tested can be secured against a shift in use in an application.

The objects of claims 1 and 6 to 8 each solve this problem. Advantageous refinements of the invention result from the definitions, the dependent claims, the drawings and the description of preferred exemplary embodiments.

• • Bereitstellen des Maschinenlernmodells, das in einer ersten Domäne auf Klassifizierung, Lokalisierung und/oder Verfolgung von Objekten und/oder Bildsegmentierung trainiert, validiert und/oder getestet wurde;
• • für Einzelbilder, auf denen das Maschinenlernmodell in einer Einsatzphase inferiert, umfassend Einzelbilder aus zweiten Domänen mit einer jeweiligen Domäneverschiebung relativ zu der ersten Domäne, jeweils Bestimmen einer epistemischen Unsicherheit, wobei Vorwärtspfade des Maschinenlernmodells entlang mehreren der Einzelbildern sequenziert werden;
• • Verwerten der epistemischen Unsicherheit als Maß für die Leistungsfähigkeit des Maschinenlernmodells bezüglich der Klassifizierung, Lokalisierung und/oder Verfolgung von Objekten und/oder Bildsegmentierung auf den Einzelbildern, wobei eine hohe Unsicherheit einer geringen Leistungsfähigkeit und eine geringe Unsicherheit einer hohen Leistungsfähigkeit entsprechen;
• • Wahrnehmen der Einsatzverlagerung als eine gegebene Abnahme der Leistungsfähigkeit, wobei die Einsatzverlagerung einem Einsatz des Maschinenlernmodells außerhalb der ersten Domäne entspricht;
• • bei der Einsatzverlagerung Rückmelden an ein System, in dem das Maschinenlernmodell in der Einsatzphase eingesetzt wird, dass das Maschinenlernmodell außerhalb der ersten Domäne inferiert.

In one aspect, the invention provides a computer-implemented method for securing a machine learning model against deployment relocation. The procedure includes the steps:

• • Providing the machine learning model that has been trained, validated and/or tested in a first domain for classification, localization and/or tracking of objects and/or image segmentation;
• • for frames on which the machine learning model infers in a deployment phase, comprising frames from second domains with a respective domain shift relative to the first domain, each determining an epistemic uncertainty, wherein forward paths of the machine learning model are sequenced along several of the frames;
• • exploiting the epistemic uncertainty as a measure of the performance of the machine learning model with regard to the classification, localization and/or tracking of objects and/or image segmentation on the individual images, with high uncertainty corresponding to low performance and low uncertainty corresponding to high performance;
• • Perceiving the deployment shift as a given decrease in performance, the deployment shift corresponding to deployment of the machine learning model outside of the first domain;
• • In deployment shifting, feedback to a system in which the machine learning model is deployed in the deployment phase that the machine learning model is inferring outside of the first domain.

According to a further aspect, the invention provides a computer program. The computer program includes instructions that cause a computer to carry out the method according to the invention when the computer runs the computer program.

The instructions of the computer program according to the invention include machine instructions, source text or object code written in assembly language, an object-oriented programming language, for example C++, or in a procedural programming language, for example C. According to one aspect of the invention, the computer program is a hardware-independent application program which, for example, has a data carrier or the data carrier signal is provided for any hardware, according to one aspect using software over the air technology. The data carrier includes volatile memory, for example RAM, DRAM, SRAM, and non-volatile memory, for example ROM, flash EEPROM. The data carriers are, for example, flash memory cards, USB sticks. According to a further aspect, the computer program is a hardware-dependent program, for example firmware for the evaluation unit of the transport system.

According to a further aspect, the invention provides a data carrier signal which transmits the computer program according to the invention.

With regard to the data carrier signal and its provision using software over the air technology, the evaluation unit of the automated transport system or the transport system includes middleware via which the transmitted computer program registers as an application on an operating system of the evaluation unit or an operating system of the transport system. Middleware is an appl neutral program that mediates between applications and operating systems of hardware modules. Using the middleware, applications or also called applications can be used independently of a hardware module, for example the evaluation unit.

• • inferiert ein Maschinenlernmodell, das in einer ersten Domäne auf Klassifizierung, Lokalisierung und/oder Verfolgung von Objekten und/oder Bildsegmentierung trainiert, validiert und/oder getestet wurde, auf Daten des Umfelderfassungssensors;
• • sichert die Auswerteeinheit das Maschinenlernmodell gemäß des erfindungsgemäßen Verfahrens gegen eine Einsatzverlagerung ab.

According to a further aspect, the invention provides an automated transport system for transporting people and/or objects. The transport system comprises at least one environment detection sensor and an evaluation unit. During operation of the transport system

• • infers a machine learning model, which was trained, validated and/or tested in a first domain for classification, localization and/or tracking of objects and/or image segmentation, on data from the environment detection sensor;
• • the evaluation unit secures the machine learning model according to the method according to the invention against a shift in use.

Securing is a technical process in which, based on real or simulated data, e.g. data collected by sensors, e.g. vehicle environment sensors, e.g. camera, lidar, radar, microphone, odor sensors, proof of security, verification and/or or the validation of the system in which the machine learning model is used, for example an automated vehicle, in defined areas or areas of application, also operational design domain, is carried out through defined processing steps. For example, the "Verification Validation Methods" project, see https://www.vvm-projekt.de/projekt, deals with test procedures and systematics as well as methods to provide safety verification for automated vehicles.

Locating objects can include a distance determination. Tracking objects means following objects over time, also called tracking.

The individual images can be individual images of image sequences or individual images without a direct sequence character. Single images from image sequences means that the machine learning model continuously infers over time.

Epistemic uncertainty, also known as model uncertainty, is understood to refer to uncertainty stemming from lack of knowledge in representing model parameters, particularly in the data-constrained scenario. In contrast, the term aleatoric uncertainty, also referred to as input uncertainty, is understood to refer to uncertainty arising from noise associated with the observation. This type of uncertainty can be caused by noise from one or more sensors.

Die Unsicherheit c ist gegeben als $c=\frac{1}{T}{\displaystyle {\sum }_{i=0}^T{\left[f^i\left(x\right)-p\right]}^2.}$

Beispielsweise kann in Keras, einer in Python geschriebenen Open Source Deep-Learning-Bibliothek mit einer anwendungsspezifischen Programmierschnittstelle unter anderem für TensorFlow, Monte Carlo Dropout beispielsweise unter anderem mit den Befehlen

 Dropout (p) (input_tensor, training=True)
 mc_model = get_model(mc=True, act="relu")
 mc_predictions=[]
 for i in tqdm.tqdm(range(500)):
      y_p=mc_odel.predict(x_test, batch_size=1000)

      mc_predictions.append(y_p)
      accs=[]
      for y_p in mc_predictions:

      acc=accuracy_score(y_test.argmax(axis=1), y_p.argmax(axis=1))
      accs.append(acc)
 print ("MC accuracy:{:.1 %}".format(sum(accs)/len(accs)))
 implementiert werden.

In the prior art, it is known to model epistemic uncertainties in training processes of machine learning models, for example using Monte Carlo dropout, with several forward paths of the machine learning model per frame x each with different dropout masks f ⁱ (x)∈ based on Monte Carlo simulation {f ⁰ (x),f ¹ (x),...,f ^T (x)} are processed. The predictive posterior mean p, also called predictive posterior mean, is then given as $p=\frac{1}{T}{\displaystyle {\sum }_{i=0}^T{f}^i\left(x\right)}.$

The uncertainty c is given as $c=\frac{1}{T}{\displaystyle {\sum }_{i=0}^T{\left[f^i\left(x\right)-p\right]}^2.}$

For example, in Keras, an open source deep learning library written in Python with an application-specific programming interface for TensorFlow, among others, Monte Carlo Dropout can be used, for example, with the commands

 Dropout (p) (input_tensor, training=True)
 mc_model = get_model(mc=True, act="relu")
 mc_predictions=[]
 for i in tqdm.tqdm(range(500)):
      y_p=mc_odel.predict(x_test, batch_size=1000)

      mc_predictions.append(y_p)
      accs=[]
      for y_p in mc_predictions:

      acc=accuracy_score(y_test.argmax(axis=1), y_p.argmax(axis=1))
      accs.append(acc)
print ("MC accuracy:{:.1 %}".format(sum(accs)/len(accs)))
 to be implemented.

In contrast to this, the uncertainty over time is determined in the application by sequencing the forward paths along several individual images of the respective image sequences.

Various metrics are known in the prior art for determining the performance of a machine learning model, for example precision, recall, receiver operating curve or accuracy in Keras. The invention determines the performance of a machine learning model in a deployment of the machine learning model from epistemic uncertainties. This allows a shift in deployment, for example out of domain performance drops under deployment, to be monitored. The epistemic uncertainty can be determined for each frame from the second domains on which the machine learning model infers without reference data, i.e. without ground truth. Additional markings or labels of the data, as required when determining precision, recall, receiver operating curve or accuracy, are therefore no longer necessary. The invention thus provides an online function that encapsulates the machine learning model in the online case in the form of a safety envelope.

According to one aspect, the epistemic uncertainties are determined in live operation, i.e. online, for example during operational trips of the automated transport system. According to another aspect, the epistemic uncertainties are determined offline, especially when the machine learning model and data are accessible offline. This may be necessary if, for example, the system has little computing power.

If the machine learning model is used, for example, in automated transport systems for perception, ie environment perception, the invention enables so-called defensive driving system behavior in areas in which perception has previously unnoticed, reduced performance. The invention makes it possible to detect the hitherto unnoticed reduced performance when the deployment is shifted and, if detected accordingly, to influence the system through feedback and/or safety instructions. For example, an operator of the automated transport system or the automated transport system can receive the safety instruction to drive more slowly or to switch to a safe state, for example in the form of fail operational. The invention thus provides a so-called saftey envelope for perception based on the determination of epistemic uncertainties in a mission.

Further fields of application for machine learning models covered by the invention are, for example, classification of objects and/or image segmentation in imaging methods in the medical field, for example for diagnosing diseases, for example based on X-ray images. The system can be an X-ray machine, for example.

Automated transport systems are used to transport objects, goods, freight and/or people. Automated transport systems include internal transport systems, for example driverless transport vehicles, for example small parts load carriers. Automated transport systems also include area-bound transport systems, such as terminal tractors for use in ports or straddle carriers. Automated transport systems also include private transport systems, for example passenger cars or commercial vehicles. Automated transport systems also include transport systems for local and/or long-distance public transport, such as trains, buses or shuttles. Automated means that the transport system is designed to carry out a driving task automatically. Automated includes partially automated execution of the driving task with joint responsibility of a human driver up to autonomous, i.e. driverless, execution of the driving task. For example, the term automatically encompasses all levels of the SAE J3016 standard. System designates the components and functionalities at system level as well as the automated transport system, for example an automated vehicle, as such.

According to one aspect of the invention, the automated transport system is an autonomous shuttle that transports people and/or goods in an urban environment. Training data was collected, labeled and the deployed machine learning model was trained, validated and tested and finally frozen for deployment. During the mission, the shuttle and the machine learning model used run the risk of a drop in performance due to mission shifting, which is only noticed in the known prior art if the system exhibits a catastrophic malfunction. In contrast, the invention makes it possible to monitor deployment shifts with continuous inference of machine learning models.

The environment detection sensor of the automated transport system is, for example, a camera, radar, lidar, acoustic sensor, infrared sensor, ultrasonic sensor, odor sensor. According to one aspect, the automated transport system includes a plurality of the environment detection sensors mentioned above, individually and/or in a combination.

The evaluation unit of the automated transport system can be a computer or a computer module comprising, for example, application-specific integrated circuits, ASICs for short, integrated circuits with loadable logic circuits, FPGAs for short, central computers, CPUs for short, hardware accelerators, for example graphics processors, GPUs for short, tensor processing units , abbreviated TCUs. According to one aspect, the evaluation unit is a system on chip, or SoC for short. For example, the SoC is embedded in an on-board network of the automated transport system.

• • jeweils der letzten n Einzelbilder bestimmt, wobei n eine einstellbare Zahl ist, oder
• • für jedes k-te Einzelbild bestimmt, wobei k eine einstellbare Zahl ist.

According to a further aspect, the epistemic uncertainties

• • determines the last n frames, where n is an adjustable number, or
• • determined for every kth frame, where k is an adjustable number.

Determining the epistemic uncertainties of the last n individual images increases the robustness to outliers. The epistemic uncertainties of the last n frames are determined according to one aspect with a sliding window algorithm, also called sliding window. If the epistemic uncertainty is only determined for every kth frame, computing power can be saved. The value of k depends on the speed at which the insert is shifted. For example, k can be chosen to be relatively large for slow deployment shifts.

According to a further aspect, the epistemic uncertainties are determined using Monte Carlo dropouts. According to the invention, any method for determining the epistemic uncertainty can be used.

According to a further aspect, when the deployment is shifted, the system initiates a safety instruction to transfer the system to a state of heightened alertness. A state of increased attention includes a state with defensive perception, also called defensive perception, in which, for example, the machine learning model, for example an artificial neural network for environment recognition, with higher performance and higher resource requirements, in particular with regard to computing power and/or memory requirements, is used temporarily for environment recognition becomes. State of heightened alertness further includes reducing the speed of the transportation system. State of increased alertness includes in particular a safe state, for example in the form of a fail safe or fail operational state.

According to a further aspect, when the deployment is relocated, the system initiates a safety instruction to an operator of the system and, following the safety instruction, the operator transfers the system to a state of increased alertness, in particular to a safe state. The operator can be a human driver of the transport system or a fleet operator of a fleet of transport systems according to the invention, for example autonomous shuttles, people or cargo movers. Alternatively, the system is designed to reach a state of increased alertness, in particular a safe state, on the system side, and after the feedback on the shift in use, it reaches the state of increased alertness, in particular the safe state.

According to one aspect, the feedback and/or safety instructions take place depending on predefined uncertainty thresholds, for example green, orange, red, warning.

In a further embodiment of the transport system, the evaluation unit comprises an interface for reading in a computer-readable data carrier on which the computer program according to the invention is stored. The evaluation unit executes the computer program when the data carrier is read.

In a further embodiment of the transport system, the evaluation unit comprises a radio interface for receiving the data carrier signal according to the invention. The evaluation unit executes the computer program when the data carrier signal is received.

• 1 eine schematische Darstellung eines Ausführungsbeispiels des erfindungsgemäßen Verfahrens,
• 2 eine graphische Darstellung eines Ausführungsbeispiels der erfindungsgemäßen Wahrnehmung einer Einsatzverlagerung und
• 3 ein Ausführungsbeispiel eines erfindungsgemäßen Transportsystems.

The invention is illustrated in the following exemplary embodiments. Show it:

• 1 a schematic representation of an embodiment of the method according to the invention,
• 2 a graphic representation of an exemplary embodiment of the perception of a deployment shift according to the invention and
• 3 an embodiment of a transport system according to the invention.

In the figures, the same reference symbols denote the same or functionally similar reference parts. For the sake of clarity, only the relevant reference parts are highlighted in the individual figures.

In 1 a machine learning model M is provided in a method step V1, for example a convolution network, which was trained, validated and tested in a first domain D1, for example on daytime recordings, for the classification and localization of pedestrians in the urban traffic environment. Localization also includes a distance determination of a reference point of the system S in which the machine learning model M is used, for example a point on an outer skin of the in 3 shown automated transport system ATS, to a reference point of the object, here the pedestrian.

In a method step V2, the machine learning model M infers on individual images of image sequences in an application phase. The image sequences come from second domains D2 with a respective domain shift a relative to the first domain. One of the second domains D2 includes, for example, recordings from twilight situations between the beginning and/or end of a day changing to night. Epistemic uncertainties c are determined on these individual images, for example by means of a sliding window algorithm for the last 10 individual images. The uncertainties c are determined by sequencing forward paths of the machine learning model along several individual images in an image sequence, for example using Monte Carlo dropout.

In a method step V3, the previously determined epistemic uncertainties c are used as a measure of the performance P of the machine learning model M with regard to the classification and localization of pedestrians. A high level of uncertainty c means low performance P. A low level of uncertainty c means high performance P.

In a method step V4, the shift in use is detected as a given decrease in performance P, see also 2 .

In a method step V5, when the deployment is shifted to the system S in which the machine learning model M is deployed in the deployment phase, for example to the automated transport system ATS, feedback is given that the machine learning model M is inferring outside of the first domain D1. The feedback as well as the safety instructions to the system S or to an operator of the system S occur, for example, as a function of defined thresholds of the performance P being undershot, see also 2 .

For example, at 100% performance P, the feedback can be that the system S or the machine learning model M used infers reliably. The feedback can be displayed visually in green, for example. At 75% performance P, the feedback can be that the system S or the machine learning model M no longer infers safely and that a speed must be reduced. This feedback can be displayed visually in orange, for example. At 50% performance P, the feedback may be that the system S or the machine learning model M is being used critically with catastrophic mispredictions and that the system S is to be stopped. This feedback can be displayed visually, for example in red.

In 2 the epistemic uncertainties c are determined in a sliding window W using a sliding window algorithm.

3 shows an autonomous passenger shuttle of SAE J3016 level 4 as an example of an automated transport system ATS. The passenger shuttle includes environment detection sensors U, for example camera, radar, lidar, microphone, with which it perceives its environment. The invention provides a safety jacket for the passenger shuttle's sensing system.

Reference List

M

machine learning model

D1

first domain

D2

second domain

a

domain shift

c

epistemic uncertainty

P

efficiency

t

Time

W

sliding window

S

system

ATS

automated transport system

u

Environment detection sensor

V1-V5

process steps

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 102019211672 A1 [0006, 0007]
• DE 102020200998 A1 [0006, 0008]
• US20200133281A1 [0009]
• US 9406017 B2 [0010]

Claims (10)

Computer-implemented method for protecting a machine learning model (M) against a deployment shift, comprising the steps: • Providing the machine learning model (M), which was trained, validated and/or tested (V1) in a first domain (D1) for classification, localization and/or tracking of objects and/or image segmentation; • for frames on which the machine learning model (M) infers in a deployment phase, comprising frames from second domains (D2) with a respective domain shift (a) relative to the first domain (D1), each determining an epistemic uncertainty (c), where forward paths of the machine learning model (M) are sequenced (V2) along a plurality of the frames; • Utilizing the epistemic uncertainty (c) as a measure of the performance (P) of the machine learning model (M) with regard to the classification, localization and/or tracking of objects and/or image segmentation on the frames, with a high uncertainty (c) a low performance (P) and low uncertainty (c) correspond to high performance (P) (V3); • Perceiving the deployment shift as a given decrease in performance (P), the deployment shift corresponding to deployment of the machine learning model (M) outside the first domain (D1) (V4); • When shifting deployment, feedback to a system (S) in which the machine learning model (M) is deployed in the deployment phase that the machine learning model (M) infers (V5) outside of the first domain (D1). procedure after claim 1 , where the epistemic uncertainties (c) • are determined for each of the last n frames, where n is an adjustable number, or • are determined for every k-th frame, where k is an adjustable number. Method according to one of the preceding claims, wherein the epistemic uncertainties (c) are determined with Monte Carlo dropout. Method according to one of the preceding claims, wherein when the deployment is shifted, the system (S) initiates a safety instruction to transfer the system (S) to a state of heightened alertness. Method according to one of the preceding claims, in which the system (S) • initiates a safety instruction to an operator of the system (S) and the operator, following the safety instruction, transfers the system (S) to a state of heightened alertness, or • is designed to reach a state of heightened alertness on the system side, and reaches the state of heightened alertness after the response to the shift in use. Computer program comprising instructions that cause a computer to carry out the method according to one of the preceding steps when the computer runs the computer program. Disk signal that the computer program according to claim 6 transmits. Automated transport system (ATS) for transporting people and/or objects, the transport system (ATS) comprising at least one environment detection sensor (U) and an evaluation unit, wherein during operation of the transport system (ATS) • a machine learning model (M), which in a first domain (D1) has been trained, validated and/or tested for classification, localization and/or tracking of objects and/or image segmentation, infers data from the environment detection sensor (U); • the evaluation unit, the machine learning model (M) according to the method according to one of Claims 1 until 7 secured against a shift in use. Transport System (ATS) according to claim 8 , The evaluation unit comprises an interface for reading in a computer-readable data carrier on which the computer program claim 6 is stored, and the evaluation unit executes the computer program when the data carrier is read. Transport System (ATS) according to claim 8 , wherein the evaluation unit comprises a radio interface for receiving the data carrier signal claim 7 and the evaluation unit executes the computer program when the data carrier signal is received.

Images