TWI857412B - Self propelled vehicle for following target and method thereof - Google Patents

Abstract

The present invention provides a self propelled vehicle for following an target and a method thereof. The method includes: using an image of an target to determine whether there is an obstacle in a track of the target; in response to determining that there exists the obstacle in the track, using the image to determine a following speed for following the target; and in response to determining that there does not exist the obstacle in the track, using a position information of the target to determine the following speed.

Description

Self-driving vehicle and method for following a target object

The present invention relates to a self-driving vehicle and a method for following a target object.

Today, warehouse and logistics companies still rely heavily on manual labor for transportation. Although automation has been partially introduced, the storage area is large and transportation back and forth is time-consuming and labor-intensive. Self-driving vehicles with following functions can follow the target (such as pickers) to pick up and transport goods, which can greatly reduce the workload of pickers.

The current method of following a self-driving car is to use a signal transmitter ( ie, Bluetooth) to send a coordinate signal so that the vehicle ( ie, self-driving car) can obtain the target position information when following. The signal can be a plane coordinate or a plane angle. When the self-driving car follows a target, if the target turns and the self-driving car still maintains a straight following path, the self-driving car will collide with the obstacle, causing damage to the self-driving car or losing the target.

The present invention provides a self-driving vehicle and method for following a target object, which can allow the self-driving vehicle to follow the picker more accurately, effectively avoiding the situation of losing or following the wrong person.

The self-driving vehicle for following a target object of the present invention includes an image capture device, a transceiver, a storage medium, and a processor. The image capture device obtains an image of the target object. The transceiver receives the position information of the target object. The storage medium stores multiple modules. The processor is coupled to the image capture device, the transceiver, and the storage medium, and accesses and executes multiple modules, wherein the multiple modules include an obstacle judgment module, an obstacle avoidance path planning module, and a target object positioning module. The obstacle judgment module uses an image to judge whether there is an obstacle in the track of the target object. The obstacle avoidance path planning module responds to the obstacle judgment module's determination that there are obstacles on the track and uses the image to determine the following speed of the target object. The target positioning module responds to the obstacle judgment module's determination that there are no obstacles on the track and uses the position information to determine the following speed.

The method of following a target object of the present invention includes: using an image of the target object to determine whether there is an obstacle in the track of the target object; using the image to determine the following speed of the target object in response to determining that there is an obstacle in the track; and using the position information of the target object to determine the following speed in response to determining that there is no obstacle in the track.

Based on the above, the self-driving vehicle and method for following a target object of the present invention can determine whether there are obstacles on the target's track, and use the image of the target or the position information of the target to determine the following speed of the target object. Based on this, the self-driving vehicle can follow the target object more stably. The present invention is to solve the problem that when the vehicle is following, the target turns or obstacles block the straight path, and the vehicle continues to receive the linear guidance signal of the transmitter, which will cause a collision.

The present invention uses image tracking technology to determine the target object's posture changes and geographical environment , dynamically switch guidance modes, and provide vehicle safety and obstacle avoidance path planning.

In order to make the above features and advantages of the present invention more clearly understood, the following is a detailed description of the embodiments with the accompanying drawings.

100: Self-driving car for following target objects

110: Image capture device

120: Transceiver

130: Storage media

131: Obstacle judgment module

132: Obstacle avoidance path planning module

133: Target positioning module

134: Target tracking and posture judgment module

135: Target recognition module

140: Processor

150: Motor controller

200: Target

210: Location information transmission module

S210, S220, S230: Steps

P0, P8, P2, P5: key points

300: Obstacles

A ₁ : The original route of the self-driving car

A ₂ : If the self-driving car follows the path of the target in a straight line

θ: Angle between route _A1 and route _A2

FIG1 is a schematic diagram of a self-driving vehicle for following a target object according to an embodiment of the present invention.

FIG2 is a flow chart of a method for following a target object according to an embodiment of the present invention.

Figure 3 is a schematic diagram of determining the posture of a target object according to an embodiment of the present invention.

FIG4 is a schematic diagram showing how to determine the following speed of a self-driving vehicle for a target object with a turning posture according to an embodiment of the present invention.

FIG5 is a flow chart showing how to determine the following speed using the image of the target object according to an embodiment of the present invention.

FIG6 is a flow chart showing how to determine the following speed using the location information of the target object according to an embodiment of the present invention.

FIG1 is a schematic diagram of a self-driving vehicle 100 for following a target object 200 according to an embodiment of the present invention. The self-driving vehicle 100 includes an image capture device 110, a transceiver 120, a storage medium 130, and a processor 140. The processor 140 is coupled to the image capture device 110, the transceiver 120, and the storage medium 130. In other embodiments, the self-driving vehicle 100 may further include a motor controller 150 coupled to the processor 140. In this embodiment, the self-driving vehicle 100 may follow the target object 200.

The image capture device 110 is, for example, a camera or a 3D image camera. In this embodiment, the image capture device 110 can generate an image of the target object 200.

The transceiver 120 transmits and receives signals wirelessly. In this embodiment, the transceiver 120 can receive location information of the target 200, and the location information can be generated by a location information transmission module 210. The location information transmission module 210 can be a Bluetooth, ultra-wideband or global positioning system. In one embodiment, the location information can be related to Bluetooth, ultra-wideband (UWB) 6.5Ghz or global positioning system (GPS). For example, the target 200 can wear a signal generator that supports Bluetooth signals, ultra-wideband signals or global positioning system signals. Then, the location information transmission module 210 of the target object 200 can transmit the location information related to Bluetooth, UWB or GPS to the transceiver 120.

The storage medium 130 is, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid state drive (SSD) or similar components or a combination of the above components, and is used to store multiple modules or various applications that can be executed by the processor 140. In this embodiment, the storage medium 130 can store the obstacle judgment module 131, the obstacle avoidance path planning module 132 and the target positioning module 133. In other embodiments, the storage medium 130 can also store the target tracking and posture determination module 134 and the target recognition module 135. The functions of these modules will be described later.

The processor 140 is, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose micro control unit (MCU), microprocessor, digital signal processor (DSP), programmable controller, application specific integrated circuit (ASIC), graphics processing unit (GPU), image signal processor (ISP), image processing unit (IPU), arithmetic logic unit (ALU), complex programmable logic device (CPLD), field programmable gate array (FPGA), or other similar components or combinations of the above components. The processor 140 can access and execute multiple modules and various applications stored in the storage medium 130.

FIG2 is a flow chart of a method for following a target object according to an embodiment of the present invention. Please refer to FIG1 and FIG2 simultaneously. The method of this embodiment is suitable for being executed by the self-driving vehicle 100 shown in FIG1. The following is a detailed description of the method for following a target object of the present invention in conjunction with the various components of the self-driving vehicle 100.

In step S210, the obstacle determination module 131 can use the image of the target object 200 to determine whether there is an obstacle in the trajectory of the target object 200.

In one embodiment, the obstacle determination module 131 may utilize a path trajectory detection model, such as an image signal receiver, and obtain the trajectory of the target object 200 by acquiring color image data through the image signal receiver, such as detecting the path trajectory by a neural network algorithm, and determining whether there is an obstacle in the trajectory. Specifically, the obstacle determination module 131 may input the image of the target object 200 into the path trajectory detection model pre-stored in the storage medium 130 to obtain the trajectory of the target object 200. Then, the obstacle determination module 131 may identify the obstacle from the image of the target object 200. Then, the obstacle judgment module 131 can judge whether there is an obstacle in the track. In one embodiment, the path track detection model may include a convolution layer, a pooling layer, a feature map, an upsampling layer, a deconvolution layer, and an activation function, but the present invention is not limited thereto. In this embodiment, the obstacle judgment module 131 can input the image of the target object 200 into the path track detection model pre-stored in the storage medium 130, and obtain the track of the target object 200 using a neural network algorithm.

Please continue to refer to Figure 2. In step S220, the obstacle avoidance path planning module 132 can use the image of the target object 200 to determine the following speed of the target object 200 in response to the obstacle judgment module 131 determining that there is an obstacle on the track (please refer to the following description). At this time, the mode in which the self-driving car 100 follows the target object 200 is also called the "curve following mode".

In one embodiment, the target tracking and posture determination module 134 can use an image tracking and posture recognition model, such as an image signal receiver, and the image tracking and posture recognition model is constructed using the Openpose method, and the key point coordinates of the target 200 determine the posture of the target 200. The following will continue to explain.

FIG3 is a schematic diagram of determining the posture of the target object 200 according to an embodiment of the present invention. Please refer to FIG1 and FIG3 at the same time. In this embodiment, the target object 200 is described by taking a person as an example. As shown in FIG3, the key point P0 can be a head skeleton point, the key point P8 can be a hip skeleton point, the key point P2 can be a left neck skeleton point, and the key point P5 can be a right neck skeleton point. The target object tracking and posture determination module 134 can use the displacement relationship between the key point coordinates of the key point P0 and the key point coordinates of the key point P8 to determine whether the target object 200 is in a squatting posture. If it is in a squatting posture, it continues to follow the straight line. In addition, the target tracking and posture judgment module 134 can also use the displacement relationship between the key point coordinates of the key point P2 and the key point coordinates of the key point P5 to judge whether the target 200 is in a turning posture. If a turning posture occurs, the turning following is used.

In other embodiments, the target tracking and posture determination module 134 can determine the posture of the target 200 according to the Openpose method. The Openpose method is the real-time recognition of two-dimensional multi-person skeleton points and the recognition of 15, 18 or 25 key points of the body/legs.

Furthermore, in this embodiment, when the target tracking and posture determination module 134 determines that the posture of the target 200 is a turning posture, the obstacle avoidance path planning module 132 can use the target recognition module 135 to recognize the posture image of the target 200 and determine to follow the turn.

FIG4 is a schematic diagram of determining the following speed of the self-driving vehicle 100 for a target object 200 in a turning posture according to an embodiment of the present invention. FIG5 is a flow chart of determining the following speed using an image of a target object according to an embodiment of the present invention. Please refer to FIG1, FIG4 and FIG5 simultaneously. As shown in FIG4, it is assumed that the target object 200 turns due to encountering an obstacle 300, and the route of the self-driving vehicle 100 in the cornering following mode is _A1 , and the route of the self-driving vehicle 100 if it follows the target object 200 in a straight line is _A2 . In addition, it is assumed that the angle between the route _A1 and the route _A2 is θ, and it is assumed that the straight line distance between the self-driving vehicle 100 and the target object 200 is d.

In this embodiment, the obstacle avoidance path planning module 132 may determine the following speed using the position of the obstacle 300 in the image of the target object 200. Specifically, in this embodiment, the following speed may include a linear speed and an angular speed. As shown in FIG5 , with respect to the position of the obstacle in the image, if the obstacle 300 is within the left safety zone (the left side of the preset safety zone), or if the obstacle 300 is within the right safety zone (the right side of the preset safety zone), the obstacle avoidance path planning module 132 may obtain the distance h using the following formula 1.

d × cosθ = h ...(Formula 1)

Then, if the distance h is less than or equal to a preset safety distance, the obstacle avoidance path planning module 132 may set the linear speed of the self-driving vehicle 100 to 0. Alternatively, if the distance h is greater than the safety distance and less than or equal to a preset following distance, the obstacle avoidance path planning module 132 may set a linear speed for the self-driving vehicle 100. For example, the obstacle avoidance path planning module 132 may set a plurality of different linear speeds for the self-driving vehicle 100 according to the distance between the target object 200 and the self-driving vehicle 100, so as to allow the self-driving vehicle 100 to keep up with the walking speed of the target object 200. Alternatively, if the distance h is greater than a preset following distance, the obstacle avoidance path planning module 132 may set the linear speed of the self-driving vehicle 100 to 0.

On the other hand, as shown in FIG5 , if the obstacle 300 is in the right dangerous area (the right side of a preset dangerous area), the obstacle avoidance path planning module 132 may set a left turn angular velocity for the self-driving vehicle 100. For example, the obstacle avoidance path planning module 132 may first multiply the linear velocity by 0.5 to obtain the left turn angular velocity, and set the moving direction of the self-driving vehicle 100 to the opposite direction of the obstacle 300, so that the self-driving vehicle 100 gradually moves away from the obstacle 300. After the distance between the self-driving vehicle 100 and the obstacle 300 is greater than the above safety distance, the self-driving vehicle 100 may continue to drive along the original path. In addition, if the obstacle 300 is in the left dangerous area (the left side of a preset dangerous area), the obstacle avoidance path planning module 132 can set a right turn angular velocity for the self-driving car 100. For example, the obstacle avoidance path planning module 132 can first multiply the above linear velocity by 0.5 to obtain the right turn angular velocity, and set the direction of travel of the self-driving car 100 to the opposite direction of the obstacle 300, so that the self-driving car 100 gradually moves away from the obstacle 300. After the distance between the self-driving car 100 and the obstacle 300 is greater than the above safety distance, the self-driving car 100 can continue to drive along the original path.

Finally, the obstacle avoidance path planning module 132 can combine the linear velocity and the angular velocity into the following velocity.

Please return to Figure 2. In step S230, the target positioning module 133 can determine the following speed using the position information of the target 200 in response to the obstacle judgment module 131 determining that there is no obstacle on the track. At this time, the mode in which the self-driving car 100 follows the target 200 is also called the "straight line following mode".

In one embodiment, when the target tracking and posture determination module 134 determines that the posture of the target 200 is a squatting posture, the target positioning module 133 can use the position information of the target 200 to determine the following speed. Specifically, as described in FIG. 3 and its embodiment, after the target tracking and posture determination module 134 uses the displacement relationship between the key point coordinates of the key point P0 and the key point coordinates of the key point P8 to determine that the posture of the target 200 is a squatting posture, the target positioning module 133 can use the position information of the target 200 to determine the following speed.

In one embodiment, when the self-driving car 100 starts to follow the target 200, the target recognition module 135 can record the characteristics of the target 200. When the mode of the self-driving car 100 following the target 200 is the "straight-line following mode" (the target 200 does not turn at this time), in order to confirm that the self-driving car 100 follows the correct target 200, the target recognition module 135 can use the pre-recorded characteristics to identify the target 200 from multiple candidates. Then, the transceiver 120 can receive its position information from this target 200.

FIG6 is a flow chart of determining the following speed using the position information of the target according to an embodiment of the present invention. Please refer to FIG1 and FIG6 at the same time. In one embodiment, the position information may include the distance between the transceiver 120 and the target 200, and may include the angle between the transceiver 120 and the target 200. Further, the following speed may include a linear speed and an angular speed. The target positioning module 133 may determine the linear speed using the distance and the following distance. Further, the target positioning module 133 may determine the angular speed using the angle.

As shown in FIG6 , if the distance between the transceiver 120 and the target 200 is less than or equal to a preset safety distance, the target positioning module 133 may set the linear speed of the self-driving vehicle 100 to 0. Alternatively, if the distance between the transceiver 120 and the target 200 is greater than the safety distance and less than or equal to a preset following distance, the target positioning module 133 may set the linear speed for the self-driving vehicle 100. Alternatively, if the distance between the transceiver 120 and the target 200 is greater than the following distance, the target positioning module 133 may set the linear speed of the self-driving vehicle 100 to 0.

On the other hand, as shown in FIG6 , regarding the acquisition of position information, the target positioning module 133 may determine the angular velocity according to the angle between the transceiver 120 and the target 200. If the target 200 is on the right side of the transceiver 120, the target positioning module 133 may set the right turn angular velocity for the self-driving vehicle 100. Alternatively, if the target 200 is on the left side of the transceiver 120, the target positioning module 133 may set the left turn angular velocity for the self-driving vehicle 100. Alternatively, if the target 200 is on the route of the self-driving vehicle 100, the target positioning module 133 may set the angular velocity of the self-driving vehicle 100 to 0.

Finally, the target positioning module 133 can combine the linear velocity and the angular velocity into the following velocity.

After determining the following speed, the processor 140 may transmit the following speed to the motor controller 150 (e.g., an encoder). For example, the processor 140 may first convert the linear speed and angular speed (in the following speed) into a format that complies with the motor controller 150 (e.g., converted into a CANbus format by a lower computer), and then transmit the following speed to the motor controller 150. Then, the motor controller 150 may drive the motor of the self-driving vehicle 100 based on the following speed.

In summary, the self-driving vehicle and method for following a target object of the present invention can determine whether there are obstacles in the track of the target object, and use the image of the target object or the position information of the target object to determine the following speed of the target object. Furthermore, the following speed of the target object can also be determined according to the posture of the target object. Based on this, the self-driving vehicle and method for following a target object of the present invention can dynamically switch between the "curve following mode" and the "straight following mode" to avoid the self-driving vehicle being damaged or losing the target object, so that the self-driving vehicle can follow the target object more stably.

Although the present invention has been disclosed as above by the embodiments, it is not intended to limit the present invention. Any person with ordinary knowledge in the relevant technical field can make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the attached patent application.

S210, S220, S230: Steps

Claims (18)

A self-propelled vehicle for following a target object comprises: an image capturing device for acquiring an image of the target object; a transceiver for receiving position information of the target object; a storage medium for storing a plurality of modules; and a processor for coupling the image capturing device, the transceiver and the storage medium, and accessing and executing the plurality of modules, wherein the plurality of modules comprises: an obstacle judging module for judging whether there is an obstacle in the trajectory of the target object by using the image; an obstacle avoidance path planning module; module, in response to the obstacle judgment module determining that the obstacle exists on the track and using the image to determine the following speed of the target object; a target positioning module, in response to the obstacle judgment module determining that the obstacle does not exist on the track and using the position information to determine the following speed; and a target tracking and posture judgment module, using the image tracking and posture recognition model and the key point coordinates in the target object to judge the posture of the target object. The self-driving car as described in claim 1, wherein when the target object tracking and posture judgment module determines that the posture of the target object is a turning posture, the obstacle avoidance path planning module uses the image to determine the following speed. A self-driving car as described in claim 1, wherein when the target tracking and posture determination module determines that the posture of the target is a crouching posture, the target positioning module uses the position information to determine the following speed. A self-driving car as described in claim 1, wherein the obstacle judgment module uses a path trajectory detection model to obtain the trajectory of the target object and judge whether the obstacle exists on the trajectory. A self-driving car as described in claim 1, wherein the location information is associated with Bluetooth, ultra-wideband or global positioning system. A self-driving car as described in claim 1, wherein the obstacle avoidance path planning module determines the following speed using the position of the obstacle in the image. A self-driving car as described in claim 1, wherein the position information includes the distance between the transceiver and the target object, and includes the angle between the transceiver and the target object, wherein the following speed includes the linear speed and the angular speed, wherein the target positioning module determines the linear speed using the distance and the following distance; and the target positioning module determines the angular speed using the angle. The self-driving car as described in claim 1 further includes a motor controller coupled to the processor, wherein the processor transmits the following speed to the motor controller. A self-driving car as described in claim 1, wherein the multiple modules further include a target object recognition module, wherein the target object recognition module recognizes the target object from multiple candidates. A method for following a target object, suitable for being performed by a self-driving vehicle, wherein the method comprises: using an image of the target object to determine whether there is an obstacle in the track of the target object; in response to determining that the obstacle exists in the track, using the image to determine the following speed of the target object; in response to determining that the obstacle does not exist in the track, using the position information of the target object to determine the following speed; and using an image tracking and posture recognition model and the coordinates of key points in the target object to determine the posture of the target object. The method described in claim 10 further includes: when the posture of the target object is a turning posture, using the image to determine the following speed. The method as described in claim 10 further includes: when the posture of the target object is a crouching posture, using the position information to determine the following speed. As described in claim 10, the step of using the image of the target object to determine whether the obstacle exists in the trajectory of the target object includes: using a path trajectory detection model to obtain the trajectory of the target object, and determining whether the obstacle exists in the trajectory. A method as claimed in claim 10, wherein the location information is associated with Bluetooth, ultra-wideband or global positioning system. As described in claim 10, the step of using the image to determine the following speed of the target object in response to determining that the obstacle exists in the trajectory includes: using the position of the obstacle in the image to determine the following speed. The method of claim 10, wherein the position information includes the distance between the self-driving vehicle and the target object, and includes the angle between the self-driving vehicle and the target object, wherein the following speed includes the linear speed and the angular speed, wherein the step of determining the following speed using the position information of the target object in response to determining that the obstacle does not exist on the track includes: determining the linear speed using the distance and the following distance; and determining the angular speed using the angle. The method as described in claim 10 further includes: transmitting the following speed to the motor controller. The method described in claim 10 further includes: identifying the target object from multiple candidates.

Images