DE102020108832A1 - System and method for monitoring a vehicle connection - Google Patents

Abstract

A method for monitoring a connection between vehicles, comprising extracting (210) an image of at least a part of the first vehicle (110) from a camera (101) mounted on the second vehicle (100), classifying (220) the image to identify at least two states of respective components of the link between the first and second vehicles, and recording (230) the identified states of the component of the link.

Description

Field of technology

The present application relates to a system and a method for monitoring a connection between vehicles. In particular, the system monitors components of a connection between a vehicle and a trailer.

background

Connections between vehicles, such as the connection between a car and a trailer, are common. Such connections usually include components that enable safe operation. Examples of such components include, for example, a coupler, a pull ball, a breakaway cable, a hand brake, a drawbar impeller, and a connector for electrical lighting. To ensure safe operation, a user must manually configure these components properly for each drive / towing scenario. For example, the drawbar wheel should be stowed away when the trailer is to be pulled straight to avoid damaging the drawbar wheel when pulling it. Incorrect configuration of one or more components could result in personal injury or damage to the vehicle, the trailer or the connection.

Known systems for monitoring vehicle connections are based on electronic signals and adaptations to the trailers and / or the connections.

the US2019386438A1 describes a vehicle connector for transmitting power and signals between a vehicle and a trailer that includes a power conductor and one or more video signal conductors.

the GB2429046A relates to an electrical system for monitoring the connection of a safety cable to a towing vehicle. The monitor can also detect the activation of the handbrake, detect the lights and / or monitor a hydraulic connection to the trailer.

It is an object of the present invention to monitor a vehicle link in a manner which overcomes at least some of the shortcomings of known methods.

Brief description

The present invention relates to a versatile camera-based system for monitoring components of a connection between vehicles. In some cases, the system can provide the user with a visual and / or audible warning when the conditions of components indicate a potential problem. Compared to known monitoring systems, the system described is reliable, expandable and can be operated without contact. The system described can also operate over a wide range of angles and will work with most existing trailers.

In some embodiments, the warning signal for the user can also immobilize a vehicle by sending a signal to the vehicle's system bus (BUS), for example a controller area network (CAN) bus.

Figure list

• die 1 ein Fahrzeug und einen durch eine Verbindung gekoppelten Anhänger zeigt, wobei das Fahrzeug mit einer auf den Anhänger ausgerichteten Kamera ausgestattet ist;
• die 2 Schritte in einem Verfahren zur Überwachung von Verbindungskomponenten zeigt;
• die 3 ein Ablaufdiagramm von beispielhaften Verfahren zur Überwachung von Verbindungskomponenten zeigt;
• die 4 einen Ausschnitt eines Bildes von einer Kamera zeigt, die an einem Fahrzeug montiert und zu einem durch eine Verbindung mit dem Fahrzeug gekoppelten Anhänger hin ausgerichtet ist;
• die 5 einen Bildausschnitt aus der 4 mit überlagerten interessierenden Bereichen für ein Deichsellaufrad und eine Handbremse zeigt; und
• die 6 eine Anzeige in einem Fahrzeug zeigt, die eine Verbindung zu einem Anhänger anzeigt.

Embodiments of the invention are described below by way of example with reference to the accompanying drawings, in which:

• the 1 Figure 11 shows a vehicle and a trailer coupled by a link, the vehicle being equipped with a camera aimed at the trailer;
• the 2 Fig. 11 shows steps in a method for monitoring link components;
• the 3 Figure 11 shows a flow diagram of exemplary methods for monitoring link components;
• the 4th shows a section of an image from a camera mounted on a vehicle and directed towards a trailer coupled by a link to the vehicle;
• the 5 an image section from the 4th shows with superimposed areas of interest for a drawbar impeller and a handbrake; and
• the 6th shows a display in a vehicle indicating a link to a trailer.

description

the 1 shows an example of a vehicle 100 with a trailer attached 110 . A trailer is any vehicle that can be pulled by another vehicle. (For example, a cart being pulled by another cart can be considered a trailer). The trailer 110 is through a connection 105 with the vehicle 100 coupled. It doesn't matter the connection 105 Part of the vehicle 100 , the trailer 110 or a separate item, provided it can couple vehicles together to enable towing. The connection 105 includes one or more components to ensure the safe operation of the trailer 110 to ensure. Examples of such components include couplers such as a pull ball, breakaway cables, hand brakes, drawbar wheels, and electrical connectors such as electrical connectors for electrical lighting.

The vehicle 100 includes a mounted camera 101 . The mounted camera 101 is aligned so that your field of vision 101a at least part of the trailer 110 and the connection 105 covers. The vehicle 100 does not need to be the source of motive power for pulling the trailer 110 be. For example, the vehicle 100 where the camera 101 is mounted, be a trailer itself, for example it can be a car-trailer-trailer arrangement.

The field of vision 101a is far, and this is usually achieved by having the camera 101 has a lens with a wide field of view, for example a fisheye lens. A fisheye lens is preferable because it is generally rotationally symmetrical. In other applications of the invention, the field of view can be less or more than 180 degrees. Although a fisheye lens is preferred, any other lens that provides a wide field of view can be used. In this context, a wide field of view is a lens with a field of view of over 100 degrees, preferably of over 150 degrees and particularly preferably of over 170 degrees.

The sensitivity of the camera used in the invention need not be limited to a specific wavelength range, but most often it will be used with cameras that are sensitive to visible light. The camera 101 will generally be in the form of a camera module comprising a housing for a lens and a sensor, the lens serving to focus light onto the sensor. The camera module can also have electronics to supply the sensor with power and to enable communication with the sensor. The camera module can also include electronics for processing the image. The processing can be low-level image signal processing, for example gain control, exposure control, white balance, noise suppression, etc., and / or it can be more powerful processing, for example for computer vision.

Thus, in embodiments, not only raw images or minimally processed images can be used, but it is also possible to use converted versions of a recorded image - so that, for example, some embodiments show images of histograms of oriented gradients (HOG), or generate integral images, or any other type of feature descriptor, from a captured image for use in downstream processing.

In some embodiments, a component of the connection to be monitored is a trailer handbrake. A handbrake is usually an independent braking system that is mounted on the trailer. A user should release the trailer handbrake before pulling the trailer to avoid inconvenience to the user or, in more severe cases, damage to the trailer braking system or the towing vehicle.

Another component of the link that is monitored in some embodiments is a drawbar impeller. A drawbar impeller is a small wheel, usually on the front of a trailer, that can be used to assist in maneuvering or maneuvering the trailer when it is not coupled to a vehicle. In general, a user should raise the drawbar impeller to a stowed position before pulling it. This in turn prevents inconvenience for the user or, in serious cases, damage to the drawbar impeller.

Different connections have different components. For example, small trailers (maximum total weight <750 kg) in Europe do not have to have a handbrake and therefore rarely have handbrakes. In some embodiments, an initial assessment of the components present on a connection under consideration is made. This initial assessment can be done automatically, e.g. using a trained convolution neural network (CNN), and / or it can be done manually. After this initial assessment, the condition of the detected component is assessed.

In some embodiments, to improve usability and efficiency, the system is configured to be selectively activated or deactivated when relevant events are detected. The relevant events include at least one of the following: changes in the component states, changes in the coupling state of the connection (coupled or not coupled) and changes in the driving state (regardless of whether the vehicle is moving). The coupling status of the connection and the driving status can be determined from the camera data, from electrical signals or from other sensors on the vehicle or on the trailer. A selective activation or deactivation can do so be set up so that it takes place a predetermined period of time after the relevant events have been recorded.

For example, the system can stop monitoring and deactivate it if: the connection is coupling a trailer to a vehicle, the handbrake is released and the vehicle is moving. For reasons of additional efficiency, the system can only be activated for a certain period of time, provided that no warning signals are issued. For example, the system can be activated for a few seconds after a connection has been paired and after the vehicle starts moving.

In one embodiment, the monitoring of the connection comprises that in the 2 steps shown. First, an image is recorded by a camera mounted on the vehicle and aimed at a trailer. A section of this picture shows a connection between a vehicle and a trailer. This section is extracted, 210 . This is followed by a classification step, 220 . This classification step consists of two parts. In the first part, the extracted image is analyzed to identify components of the connection, step 221 . The identified components are then analyzed to identify the states of the components, 222 . The identified conditions are then recorded, 230 .

Optionally, the recorded conditions are tested against expected predetermined conditions or in relation to one another, 240 . The expected, predetermined states can be called up from a database with the states expected in various driving scenarios. Depending on the identified conditions or the result of the check, an alarm can be output. 250 . Alarms can be manually disabled or muted when a user is satisfied that the components are configured correctly.

The flowchart in the 3 summarizes processes that some embodiments of the system follow. In summary, first a picture 310 taken by a camera mounted on the vehicle. Since a trailer is usually attached to the rear of a vehicle, a camera that allows a rear view of the vehicle is typically used. It doesn't matter whether it is a stand-alone rear-view camera, the rear-view camera of a surround-view system, or a rear-view mirror replacement camera (e.g. a camera that replaces the rear-view mirror). In principle, any camera that has the required view of the connection can be used.

The regions of interest assigned to the components of the connection within the image recorded by the camera can be identified in various ways. A particularly interesting area ("high-level region of interest", HL-ROI), 320 , extracted from the image, giving an image of the compound itself. The extracted image section does not have to be perfectly aligned with the connection, but just close enough to reduce irrelevant images and thus improve the efficiency of subsequent processing.

There are several ways to get HL ROI. One way is to capture the yaw angle of a link as well as the width and length of the link using prevailing vertical gradients, 320a . The image section to be extracted can also be defined using predefined values, 320b . These predefined values can be provided by a user or a trailer manufacturer, or they can be obtained from a database. Several trailer-specific HL-ROIs, from which a user can choose, can be stored in the database. In some embodiments, there may be a mixture of camera-based capture and user selection. For example, the system can automatically sense the yaw angle of a link and let the user enter the width and height of the link. Optionally, the user can adjust an HL ROI if necessary. In some embodiments, the HL-ROI can also be evaluated using a full-image-based CNN, 320c . Regardless of the method used to determine the HL-ROI, this can be triggered or caused by a relevant event, such as the coupling of a trailer.

the 4th shows a section of an image 400 from a camera and the HL-ROI 401 . The connection to the trailer is a drawbar 405 . The components of this drawbar include a drawbar impeller 402 and a handbrake 403 , both of which are in the HL-ROI. In this case, the yaw angle became the tiller 405 automatically captured by the camera and used to align the HL-ROI appropriately.

Now that the HL-ROI has been found, let's return to 3 go back and continue to step 330 . This step 330 includes capturing one or more regions of interest (ROIs) around the components of the connection. As with the HL-ROI, this can be done manually by the user and / or using specified values. In other embodiments, a machine learning approach is used - for example, a CNN or a Support Vector Machine (SVM) can be used. If a component is using a machine learning approach and it fails or has low confidence in the output, the system may use predetermined values and / or prompt the user to enter the ROI for the component.

the 5 shows a picture 500 the one from the picture 400 the 4th extracted HL-ROI which was processed to calculate the ROI of the drawbar impeller 502 and the ROI of the handbrake 503 to identify. A component's ROI can also be referred to as the component's bounding box.

Once the ROI of the components has been established, the system proceeds to determine the condition of the components. For most components, this is a binary judgment of whether they are engaged or disengaged. For example, the drawbar wheel is engaged when it is extended and disengaged when it is stowed, and the handbrake is engaged when it prevents movement of the trailer and disengaged when it allows movement.

The condition of the components can be determined in two ways. On the one hand, a binary offline classifier can be trained. The ROI of the handle is passed to the classifier, which returns the status "engaged" or "not engaged", 330a .

Alternatively or additionally, the template extraction and comparison process can be used to increase the confidence. 330b . For example, if the vehicle is driving with a trailer attached, it can be assumed that the handbrake is released (switched off) and the drawbar impeller is released (in a stowed position). Therefore, template images of the ROIs of the components can be extracted and saved while driving. The next time the driving operation of the vehicle is started, the current state of the ROIs of the components can be compared with the templates in order to confirm that the components are in a driving state. Known for comparing templates and images, which should not be further described here.

The condition of the components is then recorded, 340 . The recording can further comprise the sending of the recorded states or a signal derived from the recorded states for storage or display and / or for use in another system. For example, if the recorded conditions indicate that the trailer is not ready to be pulled (e.g. the handbrake is on), the system can signal this on the BUS, which can warn the driver and / or immobilize the vehicle.

In addition to camera-based measurements of component conditions, electrical signals can also be used to measure or confirm component conditions. The electrical signals can also determine the coupling of the connection or the states of other components that the camera cannot or cannot measure.

In some embodiments, the electrical signal is taken into account when the system assesses whether the identified component conditions warrant outputting a warning signal to the user. For example, when preparing to tow a trailer, the user should attach an electric light connector to the vehicle to activate the trailer lights. If the monitoring system detects that at least one component is in a pulling state and the connector for electrical lighting is not connected, it can provide this information to the user via a warning signal. Since the trailer lights are a central safety feature in order to warn other users of the user's intention to drive, the system can be configured to render the vehicle immobile by providing an output signal from the monitoring system to the BUS. As another example, the system may warn the driver if an electrical signal indicates that the electrical light connector is still attached, but at least one other component is not in a pulled condition. This reduces the risk of the vehicle driving away without the connector for electrical lighting being disconnected.

The warning signals that are given to the user can take many forms. These can be acoustic and / or visual warning signals. As already mentioned, they can also include a signal for influencing the travel of the vehicle.

Like in the 6th shown, the driver of the vehicle may in some cases have a screen in the vehicle that shows an image 600 showing at least part of the connection to the trailer. The warning signal can be displayed on this screen. The system may be configured to display the one or more components of the connection that caused the alert to be triggered. This display can be done as text and / or it can show the ROI of the components.

In some embodiments, the alert can be overridden manually by the user. In this way, they can continue to function if a user is convinced that the components are configured correctly.

The aforementioned machine learning algorithms including CNN can be trained in a known manner. Usually, the training takes place in a training phase using labeled data. When training the CNN to identify the HL-ROI, the CNN can be trained with images in which the HL-ROI has been annotated manually. Similarly, images of handbrake connections labeled as “engaged” or “disengaged” can be used to train the handbrake binary classifier. Since the creation of labeled data is labor intensive, it is possible to create additional training data by adapting existing labeled data. For example, the labeled data can be processed in such a way that the background is changed and / or the image is adapted in various ways. This significantly increases the available training data, making the trained machine learning algorithms more accurate. Manually entered data or user settings can also be recorded and used to retrain and improve machine learning algorithms.

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• US 2019386438 A1 [0004]
• GB 2429046 A [0005]

Claims (15)

A method for monitoring a connection between vehicles comprising: extracting (210) an image of at least a portion of the first vehicle (110) from a camera (101) mounted on the second vehicle (100); classifying (220) the image to identify at least two states of respective components of the connection between the first and second vehicles; and recording (230) the identified states of the component of the connection. Procedure according to Claim 1 wherein the connection is a drawbar connection (105) and at least one of the vehicles is a trailer (110). Procedure according to Claim 1 further comprising issuing an alarm in response to identifying at least one particular condition. Procedure according to Claim 1 further comprising: determining a mode of the second vehicle, the mode defining target states of the components of the connector; and outputting an alarm if, for a component of the connection, the identified state of the component deviates from the target state of the component. Procedure according to Claim 3 or 4th wherein the output of the alarm is dependent on whether the vehicle is configured to move and whether the link couples the first and second vehicles. Procedure according to Claim 5 wherein issuing an alarm comprises immobilizing the second vehicle (100). Procedure according to Claim 1 wherein at least two states of respective components include: a first state defining whether a wheel component of the connection is in use; and a second state defining whether a braking component of the connection is engaged. Procedure according to Claim 1 wherein the method further comprises identifying a condition of a component of the connection by measuring an electrical signal. Procedure according to Claim 7 wherein the component of the connection is a connector for electrical lighting. Procedure according to Claim 1 wherein extracting an image of at least a portion of the first vehicle from a camera mounted on the second vehicle comprises: identifying a portion of the image that includes the link; and identifying areas of the image detail, each area comprising at least one component of the connection. Procedure according to Claim 1 wherein identifying a portion of the image comprising the connection comprises passing the image through a trained convolutional neural network to identify the portion of the image comprising the connector; or wherein identifying areas of the image section comprises passing the image section through a trained neural convolution network in order to identify the areas. Procedure according to Claim 1 wherein classifying the image to identify at least two states of respective components of the connection comprises comparing images of the component with template images. Procedure according to Claim 12 wherein the system is selectively activated or deactivated a predetermined period of time after the change in at least one of the following: a signal indicating that the second vehicle is configured to move; a signal indicating that one of the first and second vehicles is moving; a signal indicating whether the link is coupling the second and first vehicles; and the states of the components of the connector. Module for a vehicle (100) which is set up to monitor a connection between a vehicle (100) and a coupled vehicle (110), the module operating according to one of the preceding claims. A computer program product comprising computer readable instructions stored on a computer readable medium and configured to, when executed in a module of a vehicle, perform the steps of any of the Claims 1 until 13th carry out.

Images