CN110648299A - Image processing method, image processing apparatus, and computer-readable storage medium - Google Patents

Abstract

Embodiments of the present invention provide an image processing method, an apparatus, and a computer-readable storage medium, wherein the image processing method includes: acquiring a panoramic image and one or more partial images within the range of the panoramic image; acquiring panoramic semantics according to the panoramic image information, wherein the panoramic semantic information corresponds to a semantically divided area in the panoramic image; according to the panoramic semantic information and its corresponding semantically divided area, determine one or more of the one or more partial images in the one or more partial images The focus area is obtained, and detailed semantic information is obtained according to the determined focus area; image description information is obtained by using the panoramic semantic information and the detailed semantic information.

Description

Image processing method, image processing apparatus, and computer-readable storage medium

technical field

The present application relates to the field of image processing, and in particular, to an image processing method, an image processing apparatus, and a computer-readable storage medium.

Background technique

A multi-sensor imaging system consists of multiple and/or multiple sensors located in the same or different locations. After the image or video data is collected by the multi-sensor imaging system, multiple images or video information from the multi-sensors can be processed to output corresponding image processing results.

In the prior art, when an image acquired by a multi-sensor imaging system includes a panoramic image and one or more partial images within the range of the panoramic image, the obtained panoramic image and the partial image are generally fused, and after fusion is obtained panorama fusion image. However, the panoramic fusion image obtained by simply merging the above-mentioned panoramic image and the partial image cannot obtain all the application information expected by the user, such as relevant semantic information, description information, and the like.

SUMMARY OF THE INVENTION

In order to solve the above technical problems, according to one aspect of the present invention, an image processing method is provided, which includes: acquiring a panoramic image and one or more partial images within the range of the panoramic image; obtaining panoramic semantics according to the panoramic image information, wherein the panoramic semantic information corresponds to a semantically divided area in the panoramic image; according to the panoramic semantic information and its corresponding semantically divided area, determine one or more of the one or more partial images in the one or more partial images The focus area is obtained, and detailed semantic information is obtained according to the determined focus area; image description information is obtained by using the panoramic semantic information and the detailed semantic information.

According to another aspect of the present invention, there is provided an image processing apparatus, comprising: an acquisition unit for acquiring a panoramic image and one or more partial images within the range of the panoramic image; and a semantic dividing unit for acquiring according to the panoramic image Panoramic semantic information, wherein the panoramic semantic information corresponds to the semantic division area in the panoramic image; the focus area acquisition unit, according to the panoramic semantic information and the corresponding semantic division area, in the one or more local One or more focus areas are determined in the image, and detailed semantic information is obtained according to the determined focus areas; the description unit is used to obtain image description information by using the panoramic semantic information and the detailed semantic information.

According to another aspect of the present invention, there is provided an image processing apparatus, comprising: a processor; and a memory in which computer program instructions are stored, wherein, when the computer program instructions are executed by the processor, causing the processor to perform the following steps: acquiring a panoramic image and one or more partial images within the range of the panoramic image; acquiring panoramic semantic information according to the panoramic image, wherein the panoramic semantic information corresponds to the panoramic image Semantic division areas in the image; according to the panoramic semantic information and its corresponding semantic division areas, determine one or more focus areas in the one or more partial images, and obtain detailed semantic information according to the determined focus areas ; Obtain image description information by using the panoramic semantic information and the detailed semantic information.

According to another aspect of the present invention, there is provided a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the following steps: acquiring a panoramic image and storing the panoramic image one or more partial images within the range; obtain panoramic semantic information according to the panoramic image, wherein the panoramic semantic information corresponds to the semantic division area in the panoramic image; according to the panoramic semantic information and its corresponding semantics Divide regions, determine one or more focus regions in the one or more partial images, and obtain detailed semantic information according to the determined focus regions; and obtain image description information by using the panoramic semantic information and the detailed semantic information.

According to the above-mentioned image processing method, image processing device and computer-readable storage medium of the present invention, panoramic semantic information and detailed semantic information can be obtained respectively for a panoramic image and one or more partial images within the range of the panoramic image, and according to This gets the image description information. It can be seen that the image description information obtained according to the above method, device and computer-readable storage medium of the present invention can take into account the panoramic semantic information about the scene description of the panoramic image and the detailed semantic information about the detailed description of the focal area of the local image, which improves the image description. It can be effectively used in the fields of autonomous driving, robot interaction and so on.

Description of drawings

The above and other objects, features, and advantages of the present invention will become more apparent from the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings.

1 shows a flowchart of an image processing method according to an embodiment of the present invention;

Fig. 2(a) shows a panoramic image according to an embodiment of the present invention; Fig. 2(b) shows a partial infrared image according to an embodiment of the present invention, and Fig. 2(c) shows a The panoramic fusion image obtained after the panoramic image in 2(a) and the infrared partial image in Fig. 2(b) are fused;

Fig. 3(a) shows a panoramic fusion image according to an embodiment of the present invention; Fig. 3(b) shows a coordinate-transformed local fusion image obtained by performing coordinate transformation on the local fusion image in Fig. 3(a); Fig. 3(c) shows a schematic diagram of the positions of the clear area 1 and the blurred area 2 in the local fusion image after coordinate transformation in FIG. 3(b);

Fig. 4(a) shows a panoramic fusion image according to an embodiment of the present invention; Fig. 4(b) shows a coordinate-transformed local fusion image obtained by performing coordinate transformation on the local fusion image in Fig. 4(a); Fig. 4(c) shows that the local fused image after the coordinate transformation of Fig. 4(b) is resampled; Fig. 4(d) shows that the resampled local fused image in Fig. panoramic image;

FIG. 5 shows a schematic diagram of a panoramic image according to an embodiment of the present invention;

6 shows a schematic diagram of the positions of a partial image and a focus area in a panoramic image according to an embodiment of the present invention;

FIG. 7 shows a block diagram of an image processing apparatus according to an embodiment of the present invention;

FIG. 8 shows a block diagram of an image processing apparatus according to an embodiment of the present invention.

Detailed ways

An image processing method, an image processing apparatus, and a computer-readable storage medium according to embodiments of the present invention will be described below with reference to the accompanying drawings. In the drawings, the same reference numbers refer to the same elements throughout. It should be understood that the embodiments described herein are illustrative only and should not be construed as limiting the scope of the invention.

An image processing method according to an embodiment of the present invention will be described below with reference to FIG. 1 . The image processing method in the embodiment of the present invention can be applied to a still image, a video frame in a video that changes over time, and the like, which is not limited herein. FIG. 1 shows a flowchart of the image processing method 100 .

As shown in FIG. 1, in step S101, a panoramic image and one or more partial images within the range of the panoramic image are acquired.

In this step, the panoramic image and one or more partial images may be acquired using a multi-sensor system. The series of images acquired by the multi-sensor system may include a panoramic image acquired by a panoramic sensor in the multi-sensor imaging system, and a range of images within the panoramic image acquired by one or more local sensors in the multi-sensor imaging system One or more partial images. Wherein, the panoramic image can be obtained by shooting, for example, 360-degree scene image information by a panoramic sensor through a wide-angle method, and can be further mapped into a two-dimensional image through transformation of a latitude and longitude coordinate system. Correspondingly, within the scope of the scene captured by the panoramic image, one or more partial images may also be acquired through one or more partial sensors. The local sensor may be, for example, one or more of a high-definition sensor, an infrared sensor, a light field sensor, a point cloud sensor, a stereo vision sensor, and a laser sensor. Through the above-mentioned local sensors, one or more of corresponding high-definition local images, infrared local images, light field local images, point cloud local images, stereoscopic vision local images, and laser local images can be acquired.

After the panoramic image and one or more partial images within its scene range are obtained through the multi-sensor system, further, the one or more partial images can be fused according to the location of the obtained partial image. processing to obtain a local fusion image, wherein the local fusion image and the local image are in one-to-one correspondence. Finally, the panoramic image and the fused local fused image can be fused to obtain a panoramic fused image.

FIG. 2 shows a schematic diagram of a panoramic image, a partial image, and a panoramic fusion image according to an embodiment of the present invention. Specifically, Fig. 2(a) is a panoramic image obtained by using a panoramic sensor in an embodiment of the present invention; Fig. 2(b) is an infrared partial image obtained by using an infrared sensor, and Fig. 2(c) is a The panorama image obtained by fusing the panorama image with the infrared partial image in Fig. 2(b). As shown in Fig. 2(c), the infrared partial image in Fig. 2(b) can be fused, and the processed partial fusion image can be fused to the central area of the panoramic image in Fig. 2(a).

In this step, an independent panoramic image and one or more partial images can be obtained respectively; a panoramic fusion image such as shown in Figure 2(c) can also be obtained at the initial stage, and processed according to the panoramic fusion image to obtain the separation The panorama and partial images are processed in subsequent steps.

In one example, when the image acquired in the initial stage is a panoramic fused image, one or more local fused images may be first obtained from the panoramic fused image based on the location of the one or more local fused images in the panoramic fused image ; and then process the local fused image to obtain a local image and/or a panoramic image, respectively. In practical applications, optionally, the position of the local fused image in the panoramic fused image can be obtained through the position information of the local fused image included in the panoramic fused image. For example, the metadata of the panoramic fused image can be obtained. to obtain the position information, or the position information can be obtained through the relevant description in the picture file of the panoramic fusion image. After knowing the location of the local fusion image (or the local image) in the panoramic fusion image, the local fusion image can be separated from the panoramic fusion image. Here, the acquired local fusion image is generally a local fusion image obtained by distorting the local image in order to adapt to the latitude and longitude coordinate system of the panoramic image. Therefore, optionally, in order to obtain a partial image without distortion, in one example, coordinate transformation may be performed on the one or more partial fusion images, so as to remove the distortion to obtain the partial image. In another example, coordinate transformation may also be performed on the locally fused image to eliminate distortion; then one or more image-related features (for example, can be obtained from the coordinate transformed locally fused image) Start searching at the center to obtain image pixel-level features such as image resolution and/or focus information); finally, according to the obtained image features, remove the blurred area from the coordinate-transformed local fusion image to obtain the required image in this step. local image.

Figure 3 shows a schematic diagram of obtaining a partial image from a panoramic fusion image. FIG. 3( a ) shows a panoramic fusion image according to an embodiment of the present invention, and the dotted line in the image is the position of the local fusion image. FIG. 3(b) shows the coordinate transformation of the local fusion image in FIG. 3(a) to eliminate distortion, and the obtained local fusion image after the coordinate transformation is obtained. Further, feature extraction can be performed in the image of FIG. 3(b) to obtain features such as image resolution and/or focus information, and to remove blurred areas from the coordinate transformed local fusion image to obtain a local image. Wherein, the blurred area of the locally fused image may, for example, appear in the transition area of certain lines or colors in the image, or the edge area of the image. FIG. 3( c ) shows a schematic diagram of the positions of the clear area 1 and the blurred area 2 in the local fusion image after coordinate transformation in FIG. 3( b ). Among them, the area 1 in the central box is a clear area, and the area 2 in which the two boxes are nested is a blurred area. In one example, the blurred area 2 may be processed and removed using the extracted image features, so that the final obtained partial image (not shown) is sufficiently clear.

Optionally, when the image acquired in the initial stage is a panoramic fusion image, the panoramic image may also be acquired based on the panoramic fusion image and the local fusion image obtained according to its location. Specifically, firstly, coordinate transformation may be performed on the one or more local fused images to eliminate distortion; then, one or more image-related features may be obtained for the coordinate transformed local fused images, for example, from the coordinate transformed local fused images. Fusing the periphery of the image to obtain image pixel-level features such as image resolution and/or focus information; Therefore, the obtained image-related features can be used to resample the coordinate-transformed local fused image to Adjust the coordinate-transformed local fusion image to have the same features (image resolution, focus information, etc.) as the surrounding panoramic fusion image; finally, perform inverse coordinate transformation on the resampled local fusion image (that is, with the aforementioned coordinates The transformation process is reversed), that is to say, after the resampled local fusion image is distorted, it is projected back to the coordinate system of the panoramic fusion image for fusion, so as to fuse the area where the original local fusion image is located in the panoramic fusion image. , replaced with a processed local fusion image with the same image features as the panoramic fusion image to obtain a panoramic image. In one example of the present invention, resampling operates differently for different local sensors in a multi-sensor imaging system. For example, pixel resampling of high-definition data can be performed for high-definition sensors, infrared data acquisition and visible light processing can be performed for infrared sensors, resolution of light field data and focus information averaging and adjustment for light field sensors, and point cloud sensors. Projection and pixel supplementation of point cloud data can be performed, and depth information and resolution adjustment of stereo vision data can be performed for stereo vision sensors. The above resampling processing procedure and method are only examples, and in practical applications, any relevant resampling processing method may be adopted, which is not limited herein.

FIG. 4 shows a schematic diagram of obtaining a panoramic image from a panoramic fusion image. FIG. 4( a ) shows a panoramic fusion image according to an embodiment of the present invention, wherein the position of the local fusion image is framed by the dotted line. Fig. 4(b) shows the coordinate transformation of the locally fused image in Fig. 4(a) to eliminate distortion to obtain a locally fused image after coordinate transformation. Further, feature extraction can be performed around the image of FIG. 4(b) to obtain image features such as image resolution and/or focus information. FIG. 4( c ) shows a schematic diagram of resampling the local fused image after the coordinate transformation of FIG. 4( b ) using the extracted image features. Fig. 4(d) shows a schematic diagram of performing coordinate inverse transformation on the resampled local fused image in Fig. 4(c) to project it to the panoramic fused image in Fig. 4(a), and finally obtains images with consistent image features (such as image resolution, focus information).

In step S102, panoramic semantic information is obtained according to the panoramic image, wherein the panoramic semantic information corresponds to a semantically divided area in the panoramic image.

In this step, the panoramic image may be processed to obtain one or more panoramic semantic information of the panoramic image and its corresponding semantic division area. In an example, technologies such as image recognition may be used to obtain panoramic semantic information and the range where the corresponding semantically divided regions are located. Optionally, information related to the panoramic image, such as background information and/or scene description information of the image, may be further obtained according to the obtained panoramic semantic information.

FIG. 5 shows a schematic diagram of a panoramic image acquired according to an embodiment of the present invention. According to the example shown in FIG. 5 , the panoramic semantic information in the panoramic image can obtain information such as sky, ground, and people according to the image recognition content, which can respectively correspond to different regions obtained by image recognition, and the obtained Background information and/or scene description information may include, for example: outdoors, crowded people, face-to-face conversations, and the like.

In step S103, one or more focus regions may be determined in the one or more partial images according to the panoramic semantic information and its corresponding semantic division regions, and detailed semantic information may be obtained according to the determined focus regions. Optionally, one or more focus semantic information related to the one or more partial images may be selected from the panoramic semantic information, and according to the selected focus semantic information and its corresponding semantic division area, in the Corresponding regions of interest are obtained in one or more partial images through techniques such as neural network and image information processing as the one or more focus regions, and corresponding detailed semantic information is obtained according to the determined focus regions. Of course, the above specific operation methods are only examples. In practical applications, the determined focus area may not be completely in the partial image. For example, the focus area may only partially overlap with the partial image area, or the focus area may overlap with the partial image area. The partial image areas do not overlap at all, etc. Correspondingly, the selected focus semantic information for the focus area and the subsequently acquired detailed semantic information may be only partially related to the partial image, or not related to the partial image at all. At this time, in this step, one or more focus regions may be determined only according to the panoramic semantic information and its corresponding semantic division regions, and detailed semantic information may be acquired according to the determined focus regions.

FIG. 6 is a schematic diagram showing a panoramic image according to an embodiment of the present invention, and a schematic diagram of acquiring a partial image and a position of a focal area. According to the example shown in Figure 6, the dotted frame in the panoramic image is the area where the local fusion image obtained by the local sensor is fused, and the infrared image in the lower part of Figure 6 is in the local fusion image converted from the local fusion image. A zoomed-in schematic of the focal region identified in the image. According to the magnified infrared image of the focus area in FIG. 6 , corresponding detailed semantic information can be obtained, for example, the emotion of a person is happy.

In step S104, image description information is obtained by using the panoramic semantic information and the detailed semantic information. In this step, the panoramic semantic information and the detailed semantic information may be fused, and the image description information may be obtained by combining the weights. Optionally, the panoramic semantic information E used to describe the scene can be fused with the detailed semantic information S used to describe the details, and based on different model structures (such as weight averaging, Bayesian estimation, data fusion neural network) , reinforcement learning, etc.) to obtain the final image description information.

As mentioned above, the panoramic image and the partial image according to the embodiment of the present invention may both be static images, or may be a video frame in a video respectively. When the panoramic image and the partial image are video frames in the video, they can be one frame of panoramic image in the panoramic video collected at the same time i, and one or more frames of partial images in one or more partial videos. . When the panoramic image and the partial image are respectively a video frame collected at the same time in the video, the obtaining the image description information by using the panoramic semantic information and the detailed semantic information may include: combining the panoramic semantic information at different times The information and the detailed semantic information are respectively fused and processed according to time series to obtain the image description information that changes over time. That is to say, the image description information for the video may be information that changes gradually over time, rather than being fixed. At this time, the panoramic semantic information in the image description information can be represented as E _i , the detailed semantic information can be represented as S _i , and i is time. Correspondingly, the time series of panoramic semantic information changing with i can be E _i-2 , E _i-1 , E _i , E _i+1 . . . The time series of detailed semantic information changing with i can be S _i-2 , S _i-1 , S _i , S _i+1 . . . For example, an example of image description information changing over time may be: two people are talking outdoors, are happy at time i-1, start arguing at time i, and so on.

In an example, the image description information averaged by weights can be expressed as:

R=W _si S _i +W _ei E _i +W _s(i-1) S _i-1 +W _e(i-1) E _i-1 +…

Among them, R is the weighted average information of image semantics (ie, weighted image description information), _Wsi is the weight of detail semantic information, _Wei is the weight of panoramic semantic information, and i is time.

In another example, the image description information based on Bayesian estimation can be expressed as:

P _r =minL(P(P _si ,P _ei ))

where P _si is the Bayesian estimation of detail semantic information, P _ei is the Bayesian estimation of panoramic semantic information, i is time, P(P _si , P _ei ) is the joint distribution function, and P _r is the minimum likelihood of the joint distribution function P Natural estimation, that is, the fusion value of image description information.

In another example, the image description information based on reinforcement learning can be expressed as:

Turple(S,A,R,P)=turple((S _si ,S _ei ),A,(R _si ,R _ei ),P)

Wherein turtle(x) is the reinforcement learning four-element system; S, A, R are the input; P is the output; where S is the environmental information or state, which can be divided into detailed semantic information S _si and panoramic semantic information S _ei ; A is the state The behavior or action under ; R is the reward of each action in each state, which can be divided into the reward R _si brought by the detailed semantic information and the reward Re _ei brought by the panoramic semantic information; P is the image description information in the current state or its corresponding behavioral function.

According to the above-mentioned image processing method of the present invention, panoramic semantic information and detailed semantic information can be obtained respectively for a panoramic image and one or more partial images within the range of the panoramic image, and image description information can be obtained accordingly. It can be seen that the image description information obtained according to the above method of the present invention can take into account the panoramic semantic information about the scene description of the panoramic image and the detailed semantic information about the detailed description of the focal area of the local image, which improves the accuracy of the image description and can be effectively applied to Autonomous driving, robot interaction and other fields.

For example, in the field of robot interaction, in the prior art, the corresponding panoramic semantic information can only be obtained according to the panoramic image, so that only the description of the scene can be performed, and the corresponding detailed semantic analysis of the focal area cannot be targeted and made. reaction. According to the above method of the embodiment of the present invention, not only the panoramic semantic information for the scene description, but also the detailed semantic information for the focal area can be obtained, and the focal area can be transformed as needed, so that the scene and the scene can be taken into account. The description of the different focal points helps the presence of robots to communicate and respond more precisely and in a targeted manner.

Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described with reference to FIG. 7 . FIG. 7 shows a block diagram of an image processing apparatus 700 according to an embodiment of the present invention. The image processing apparatus in the embodiment of the present invention may be applied to a still image or a video frame in a video that changes over time, which is not limited herein. As shown in FIG. 7 , the image processing apparatus 700 includes an acquisition unit 710 , a semantic division unit 720 , a focus area acquisition unit 730 and a description unit 740 . In addition to these units, the apparatus 700 may also include other components, however, since these components are not related to the content of the embodiments of the present invention, their illustration and description are omitted here. In addition, since the specific details of the following operations performed by the image processing apparatus 700 according to the embodiment of the present invention are the same as those described above with reference to FIGS. 1 to 6 , repeated descriptions of the same details are omitted here to avoid repetition. .

The acquisition unit 710 of the image processing apparatus 700 in FIG. 7 is configured to acquire a panoramic image and one or more partial images within the range of the panoramic image.

The acquisition unit 710 may acquire the panoramic image and one or more partial images using a multi-sensor system. The series of images acquired by the multi-sensor system may include a panoramic image acquired by a panoramic sensor in the multi-sensor imaging system, and a range of images within the panoramic image acquired by one or more local sensors in the multi-sensor imaging system One or more partial images. Wherein, the panoramic image can be obtained by shooting, for example, 360-degree scene image information by a panoramic sensor through a wide-angle method, and can be further mapped into a two-dimensional image through transformation of a latitude and longitude coordinate system. Correspondingly, within the scope of the scene captured by the panoramic image, one or more partial images may also be acquired through one or more partial sensors. The local sensor may be, for example, one or more of a high-definition sensor, an infrared sensor, a light field sensor, a point cloud sensor, a stereo vision sensor, and a laser sensor. Through the above-mentioned local sensors, one or more of corresponding high-definition local images, infrared local images, light field local images, point cloud local images, stereoscopic vision local images, and laser local images can be acquired.

After the panoramic image and one or more partial images within its scene range are obtained through the multi-sensor system, further, the one or more partial images can be fused according to the location of the obtained partial image. processing to obtain a local fusion image, wherein the local fusion image and the local image are in one-to-one correspondence. Finally, the panoramic image and the fused local fused image can be fused to obtain a panoramic fused image.

FIG. 2 shows a schematic diagram of a panoramic image, a partial image, and a panoramic fusion image according to an embodiment of the present invention. Specifically, Fig. 2(a) is a panoramic image obtained by using a panoramic sensor in an embodiment of the present invention; Fig. 2(b) is an infrared partial image obtained by using an infrared sensor, and Fig. 2(c) is a The panorama image obtained by fusing the panorama image with the infrared partial image in Fig. 2(b). As shown in Fig. 2(c), the infrared partial image in Fig. 2(b) can be fused, and the processed partial fusion image can be fused to the central area of the panoramic image in Fig. 2(a).

During the specific operation, the acquiring unit 710 can acquire an independent panoramic image and one or more partial images respectively; it can also acquire, for example, a panoramic fusion image as shown in FIG. to process the separated panoramic and partial images for use in subsequent steps.

In one example, when the image acquired in the initial stage is a panoramic fused image, one or more local fused images may be first obtained from the panoramic fused image based on the location of the one or more local fused images in the panoramic fused image ; and then process the local fused image to obtain a local image and/or a panoramic image, respectively. In practical applications, optionally, the position of the local fused image in the panoramic fused image can be obtained through the position information of the local fused image included in the panoramic fused image. For example, the metadata of the panoramic fused image can be obtained. to obtain the position information, or the position information can be obtained through the relevant description in the picture file of the panoramic fusion image. After knowing the location of the local fusion image (or the local image) in the panoramic fusion image, the local fusion image can be separated from the panoramic fusion image. Here, the acquired local fusion image is generally a local fusion image obtained by distorting the local image in order to adapt to the latitude and longitude coordinate system of the panoramic image. Therefore, optionally, in order to obtain a partial image without distortion, in one example, coordinate transformation may be performed on the one or more partial fusion images, so as to remove the distortion to obtain the partial image. In another example, coordinate transformation may also be performed on the locally fused image to eliminate distortion; then one or more image-related features (for example, can be obtained from the coordinate transformed locally fused image) The center starts to search to obtain image pixel-level features such as image resolution and/or focus information); finally, according to the acquired image features, the blurred area is removed from the coordinate-transformed local fusion image to obtain the desired local image. .

Figure 3 shows a schematic diagram of obtaining a partial image from a panoramic fusion image. FIG. 3( a ) shows a panoramic fusion image according to an embodiment of the present invention, and the dotted line in the image is the position of the local fusion image. FIG. 3(b) shows the coordinate transformation of the local fusion image in FIG. 3(a) to eliminate distortion, and the obtained local fusion image after the coordinate transformation is obtained. Further, feature extraction can be performed in the image of FIG. 3(b) to obtain features such as image resolution and/or focus information, and to remove blurred areas from the coordinate transformed local fusion image to obtain a local image. Wherein, the blurred area of the locally fused image may, for example, appear in the transition area of certain lines or colors in the image, or the edge area of the image. FIG. 3( c ) shows a schematic diagram of the positions of the clear area 1 and the blurred area 2 in the local fusion image after coordinate transformation in FIG. 3( b ). Among them, the area 1 in the central box is a clear area, and the area 2 in which the two boxes are nested is a blurred area. In one example, the blurred area 2 may be processed and removed using the extracted image features, so that the final obtained partial image (not shown) is sufficiently clear.

Optionally, when the image acquired in the initial stage is a panoramic fusion image, the panoramic image may also be acquired based on the panoramic fusion image and the local fusion image obtained according to its location. Specifically, firstly, coordinate transformation may be performed on the one or more local fused images to eliminate distortion; then, one or more image-related features may be obtained for the coordinate transformed local fused images, for example, from the coordinate transformed local fused images. Fusing the periphery of the image to obtain image pixel-level features such as image resolution and/or focus information; Therefore, the obtained image-related features can be used to resample the coordinate-transformed local fused image to Adjust the coordinate-transformed local fusion image to have the same features (image resolution, focus information, etc.) as the surrounding panoramic fusion image; finally, perform inverse coordinate transformation on the resampled local fusion image (that is, with the aforementioned coordinates The transformation process is reversed), that is to say, after the resampled local fusion image is distorted, it is projected back to the coordinate system of the panoramic fusion image for fusion, so as to fuse the area where the original local fusion image is located in the panoramic fusion image. , replaced with a processed local fusion image with the same image features as the panoramic fusion image to obtain a panoramic image. In one example of the present invention, resampling operates differently for different local sensors in a multi-sensor imaging system. For example, pixel resampling of high-definition data can be performed for high-definition sensors, infrared data acquisition and visible light processing can be performed for infrared sensors, resolution of light field data and focus information averaging and adjustment for light field sensors, and point cloud sensors. Projection and pixel supplementation of point cloud data can be performed, and depth information and resolution adjustment of stereo vision data can be performed for stereo vision sensors. The above resampling processing procedure and method are only examples, and in practical applications, any relevant resampling processing method may be adopted, which is not limited herein.

FIG. 4 shows a schematic diagram of obtaining a panoramic image from a panoramic fusion image. FIG. 4( a ) shows a panoramic fusion image according to an embodiment of the present invention, wherein the position of the local fusion image is framed by the dotted line. Fig. 4(b) shows the coordinate transformation of the locally fused image in Fig. 4(a) to eliminate distortion to obtain a locally fused image after coordinate transformation. Further, feature extraction can be performed around the image of FIG. 4(b) to obtain image features such as image resolution and/or focus information. FIG. 4( c ) shows a schematic diagram of resampling the local fused image after the coordinate transformation of FIG. 4( b ) using the extracted image features. Fig. 4(d) shows a schematic diagram of performing coordinate inverse transformation on the resampled local fused image in Fig. 4(c) to project it to the panoramic fused image in Fig. 4(a), and finally obtains images with consistent image features (such as image resolution, focus information).

The semantic division unit 720 obtains panoramic semantic information according to the panoramic image, wherein the panoramic semantic information corresponds to the semantic division area in the panoramic image.

The semantic division unit 720 may process the panoramic image to obtain one or more panoramic semantic information of the panoramic image and its corresponding semantic division area. In an example, technologies such as image recognition may be used to obtain panoramic semantic information and the range where the corresponding semantically divided regions are located. Optionally, information related to the panoramic image, such as background information and/or scene description information of the image, may be further obtained according to the obtained panoramic semantic information.

FIG. 5 shows a schematic diagram of a panoramic image acquired according to an embodiment of the present invention. According to the example shown in FIG. 5 , the panoramic semantic information in the panoramic image can obtain information such as sky, ground, and people according to the image recognition content, which can respectively correspond to different regions obtained by image recognition, and the obtained Background information and/or scene description information may include, for example: outdoors, crowded people, face-to-face conversations, and the like.

The focal area acquiring unit 730 may determine one or more focal areas in the one or more partial images according to the panoramic semantic information and its corresponding semantic division areas, and acquire detailed semantic information according to the determined focal areas. Optionally, one or more focus semantic information related to the one or more partial images may be selected from the panoramic semantic information, and according to the selected focus semantic information and its corresponding semantic division area, in the Corresponding regions of interest are obtained in one or more partial images through techniques such as neural network and image information processing as the one or more focus regions, and corresponding detailed semantic information is obtained according to the determined focus regions. Of course, the above specific operation methods are only examples. In practical applications, the determined focus area may not be completely in the partial image. For example, the focus area may only partially overlap with the partial image area, or the focus area may overlap with the partial image area. The partial image areas do not overlap at all, etc. Correspondingly, the selected focus semantic information for the focus area and the subsequently acquired detailed semantic information may be only partially related to the partial image, or not related to the partial image at all. At this time, the focal area acquiring unit 730 may determine one or more focal areas only according to the panoramic semantic information and its corresponding semantic division areas, and acquire detailed semantic information according to the determined focal areas.

FIG. 6 shows a schematic diagram of a partial image and a focal area acquired by a panoramic image according to an embodiment of the present invention. According to the example shown in Figure 6, the dotted frame in the panoramic image is the area where the local fusion image obtained by the local sensor is fused, and the infrared image in the lower part of Figure 6 is in the local fusion image converted from the local fusion image. A zoomed-in schematic of the focal region identified in the image. According to the magnified infrared image of the focus area in FIG. 6 , corresponding detailed semantic information can be obtained, for example, the emotion of a person is happy.

The description unit 740 obtains image description information by using the panoramic semantic information and the detailed semantic information. The description unit 740 may fuse the panoramic semantic information and the detailed semantic information, and obtain the image description information by combining the weights. Optionally, the panoramic semantic information E used to describe the scene can be fused with the detailed semantic information S used to describe the details, and based on different model structures (such as weight averaging, Bayesian estimation, data fusion neural network) , reinforcement learning, etc.) to obtain the final image description information.

As mentioned above, the panoramic image and the partial image according to the embodiment of the present invention may both be static images, or may be a video frame in a video respectively. When both the panoramic image and the partial image are video frames in the video, they can be one frame of panoramic image in the panoramic video collected at the same moment i, and one frame or one frame of the collected one or more partial videos. Multi-frame partial images. When the panoramic image and the partial image are respectively a frame of video frames collected at the same time in the video, the obtaining the image description information by using the panoramic semantic information and the detailed semantic information may include: combining the panoramic semantic information at different times The information and the detailed semantic information are respectively fused and processed according to time series to obtain the image description information that changes over time. That is to say, the image description information for the video may be information that changes gradually over time, rather than being fixed. At this time, the panoramic semantic information in the image description information can be represented as E _i , the detailed semantic information can be represented as S _i , and i is time. Correspondingly, the time series of panoramic semantic information changing with i can be E _i-2 , E _i-1 , E _i , E _i+1 . . . The time series of detailed semantic information changing with i can be S _i-2 , S _i-1 , S _i , S _i+1 . . . In one example, the result of the time-varying image description information may be: two people talking outdoors, happy at time i-1, quarreling at time i, and so on.

In an example, the image description information averaged by weights can be expressed as:

R=W _si S _i +W _ei E _i +W _s(i-1) S _i-1 +W _e(i-1) E _i-1 +…

Among them, R is the weighted average information of image semantics (ie, weighted image description information), _Wsi is the weight of detail semantic information, _Wei is the weight of panoramic semantic information, and i is time.

In another example, the image description information based on Bayesian estimation can be expressed as:

P _r =minL(P(P _si ,P _ei ))

where P _si is the Bayesian estimation of detail semantic information, P _ei is the Bayesian estimation of panoramic semantic information, i is time, P(P _si , P _ei ) is the joint distribution function, and P _r is the minimum likelihood of the joint distribution function P Natural estimation, that is, the fusion value of image description information.

In another example, the image description information based on reinforcement learning can be expressed as:

Turple(S,A,R,P)=turple((S _si ,S _ei ),A,(R _si ,R _ei ),P)

Wherein turtle(x) is the reinforcement learning four-element system; S, A, R are the input; P is the output; where S is the environmental information or state, which can be divided into detailed semantic information S _si and panoramic semantic information S _ei ; A is the state The behavior or action under ; R is the reward of each action in each state, which can be divided into the reward R _si brought by the detailed semantic information and the reward Re _ei brought by the panoramic semantic information; P is the image description information in the current state or its corresponding behavioral function.

According to the above-mentioned image processing apparatus of the present invention, panoramic semantic information and detailed semantic information can be obtained respectively for a panoramic image and one or more partial images within the range of the panoramic image, and image description information can be obtained accordingly. It can be seen that the image description information obtained according to the above method of the present invention can take into account the panoramic semantic information about the scene description of the panoramic image and the detailed semantic information about the detailed description of the focal area of the local image, which improves the accuracy of the image description and can be effectively applied to Autonomous driving, robot interaction and other fields.

For example, in the field of robot interaction, in the prior art, the corresponding panoramic semantic information can only be obtained according to the panoramic image, so that only the description of the scene can be performed, and the corresponding detailed semantic analysis of the focal area cannot be targeted and made. reaction. According to the above method of the embodiment of the present invention, not only the panoramic semantic information for the scene description, but also the detailed semantic information for the focal area can be obtained, and the focal area can be transformed as needed, so that the scene and the scene can be taken into account. The description of the different focal points helps the presence of robots to communicate and respond more precisely and in a targeted manner.

Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described with reference to FIG. 8 . FIG. 8 shows a block diagram of an image processing apparatus 800 according to an embodiment of the present invention. As shown in FIG. 8, the apparatus 800 may be a computer or a server.

As shown in FIG. 8 , the image processing apparatus 800 includes one or more processors 810 and a memory 820. Of course, in addition to this, the image processing apparatus 800 may also include a multi-sensor imaging system and an output device (not shown), etc., These components may be interconnected by bus systems and/or other forms of connection mechanisms. It should be noted that the components and structures of the image processing apparatus 800 shown in FIG. 8 are only exemplary and non-limiting, and the image processing apparatus 800 may also have other components and structures as required.

Processor 810 may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may utilize computer program instructions stored in memory 820 to perform desired functions, which may include: Obtain a panoramic image and one or more partial images within the range of the panoramic image; obtain panoramic semantic information according to the panoramic image, wherein the panoramic semantic information corresponds to a semantically divided area in the panoramic image; the panoramic semantic information and its corresponding semantic division area, determine one or more focal areas in the one or more partial images, and obtain detailed semantic information according to the determined focal area; use the panoramic semantic information and all the The detailed semantic information is described to obtain the image description information.

Memory 820 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 810 may execute the program instructions to implement the functions of the image processing apparatus of the embodiments of the present invention described above and/or other desired functions, and/or the image processing method according to the embodiment of the present invention may be performed. Various application programs and various data may also be stored in the computer-readable storage medium.

In the following, a computer-readable storage medium according to an embodiment of the present invention is described, on which computer program instructions are stored, wherein, when the computer program instructions are executed by a processor, the following steps are implemented: acquiring a panoramic image and within the range of the panoramic image one or more partial images; obtain panoramic semantic information according to the panoramic image, wherein the panoramic semantic information corresponds to a semantically divided area in the panoramic image; according to the panoramic semantic information and its corresponding semantically divided area , determine one or more focus areas in the one or more partial images, and acquire detailed semantic information according to the determined focus areas; obtain image description information by using the panoramic semantic information and the detailed semantic information.

Of course, the above-mentioned specific embodiments are only examples rather than limitations, and those skilled in the art can combine and combine some steps and devices from the above-mentioned separately described embodiments according to the concept of the present invention to achieve the effect of the present invention. Combinations and combinations of embodiments are also included in the present invention, and such combinations and combinations are not individually described herein.

Note that the advantages, advantages, effects, etc. mentioned in the present invention are only examples and not limitations, and these advantages, advantages, effects, etc. should not be considered as necessarily possessed by each embodiment of the present invention. In addition, the specific details of the above-mentioned invention are only for the purpose of example and easy understanding, but not for limitation, and the above-mentioned details do not limit the present invention to be realized by the above-mentioned specific details.

The block diagrams of the devices, apparatus, apparatuses, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that the connections, arrangements, and configurations must be in the manner shown in the block diagrams. As those skilled in the art will appreciate, these means, apparatuses, apparatuses, systems may be connected, arranged, configured in any manner. Words such as "including", "including", "having" and the like are open-ended words meaning "including but not limited to" and are used interchangeably therewith. As used herein, the words "or" and "and" refer to and are used interchangeably with the word "and/or" unless the context clearly dictates otherwise. As used herein, the word "such as" refers to and is used interchangeably with the phrase "such as but not limited to".

The flowcharts of steps in the present invention and the above method descriptions are merely illustrative examples and are not intended to require or imply that the steps of the various embodiments must be performed in the order presented. As those skilled in the art will recognize, the sequence of steps in the above embodiments may be performed in any order. Words such as "thereafter," "then," "next," etc. are not intended to limit the order of the steps; these words are merely used to guide the reader through the description of the methods. In addition, any reference to an element in the singular, eg, using the articles "a," "an," or "the," should not be construed as limiting that element to the singular.

In addition, the steps and devices in the various embodiments herein are not limited to be implemented in a certain embodiment. In fact, some steps and devices in the various embodiments herein can be combined according to the concept of the present invention to achieve New embodiments are contemplated and included within the scope of the present invention.

The various operations of the methods described above may be performed by any suitable means capable of performing the corresponding function. The means may include various hardware and/or software components and/or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors.

A general purpose processor, digital signal processor (DSP), ASIC, field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or The various illustrated logic blocks, modules and circuits described are implemented or carried out in transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

The steps of a method or algorithm described in connection with the present invention may be directly embedded in hardware, in a software module executed by a processor, or in a combination of the two. A software module may exist in any form of tangible storage medium. Some examples of storage media that may be used include random access memory (RAM), read only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, and the like. A storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral with the processor. A software module may be a single instruction or many instructions, and may be distributed over several different code segments, among different programs, and across multiple storage media.

The methods of the invention herein include one or more acts for implementing the described methods. The methods and/or actions may be interchanged with each other without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and/or use of specific actions may be modified without departing from the scope of the claims.

The functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions on a tangible computer-readable medium. A storage medium can be any available tangible medium that can be accessed by a computer. By way of example and not limitation, such computer readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices or may be used to carry or store desired in the form of instructions or data structures program code and any other tangible medium that can be accessed by a computer. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disc, and blu-ray disc, where a disc typically reproduces data magnetically, while a disc The data is reproduced optically with a laser.

Accordingly, a computer program product may perform the operations set forth herein. For example, such a computer program product may be a tangible computer-readable medium having instructions physically stored (and/or encoded) thereon, the instructions executable by one or more processors to perform the operations described herein. The computer program product may include packaging materials.

Software or instructions may also be transmitted over a transmission medium. For example, software may be transmitted from a website, server, or other remote source using a transmission medium such as coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology such as infrared, radio, or microwave.

Furthermore, modules and/or other suitable means for carrying out the methods and techniques described herein may be downloaded and/or otherwise obtained by user terminals and/or base stations as appropriate. For example, such a device may be coupled to a server to facilitate the transfer of means for carrying out the methods described herein. Alternatively, the various methods described herein may be provided via storage means (eg, RAM, ROM, physical storage media such as CDs or floppy disks, etc.) so that user terminals and/or base stations may Various methods are obtained when the device provides storage components. Furthermore, any other suitable techniques for providing the methods and techniques described herein to a device may be utilized.

Other examples and implementations are within the scope and spirit of the invention and appended claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hardwiring, or any combination of these. Features implementing functions may also be physically located at various locations, including being distributed so that portions of functions are implemented at different physical locations. Also, as used herein, including in the claims, the use of "or" in the recitation of items beginning with "at least one" indicates separate recitations, such as "at least one of A, B, or C." An enumeration means A or B or C, or AB or AC or BC, or ABC (ie A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

Various changes, substitutions and alterations to the techniques described herein can be made without departing from the techniques taught by the appended claims. Furthermore, the scope of the claims of the present invention is not limited to the specific aspects of the process, machine, manufacture, composition of events, means, methods and acts described above. A currently existing or later-to-be-developed process, machine, manufacture, composition, means, method, or act of performing substantially the same function or achieving substantially the same results as the corresponding aspects described herein may be utilized. Accordingly, the appended claims include within their scope such processes, machines, manufacture, compositions of events, means, methods, or acts.

The above description of the inventive aspects is provided to enable any person skilled in the art to make or use the invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the invention. Thus, the present invention is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features of the invention herein.

The foregoing description has been presented for the purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the invention to the forms of invention herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims (13)

1. An image processing method comprising:

acquiring a panoramic image and one or more local images within the panoramic image;

acquiring panoramic semantic information according to the panoramic image, wherein the panoramic semantic information corresponds to semantic division areas in the panoramic image;

determining one or more focus areas in the one or more local images according to the panoramic semantic information and the corresponding semantic division areas thereof, and acquiring detailed semantic information according to the determined focus areas;

and obtaining image description information by using the panoramic semantic information and the detail semantic information.

2. The method of claim 1, wherein the acquiring the panoramic image and the one or more local images within the panoramic image comprises:

acquiring one or more local fusion images from a panoramic fusion image based on the positions of the one or more local fusion images in the panoramic fusion image, wherein the panoramic fusion image is obtained by fusing the panoramic image and the one or more local images, and the local images are fused and correspond to the local fusion images in the panoramic fusion image one to one;

and processing the local fusion image to obtain the panoramic image and/or the local image.

3. The method of claim 2, wherein the processing the locally fused image to obtain the panoramic image and/or the local image comprises:

and carrying out coordinate transformation on the one or more local fusion images to obtain the local images.

4. The method of claim 2, wherein the processing the locally fused image to obtain the panoramic image and/or the local image comprises:

performing coordinate transformation on the one or more locally fused images;

resampling the local fusion image after coordinate transformation;

and carrying out coordinate inverse transformation on the local fusion image after resampling so as to obtain the panoramic image.

5. The method of claim 1, wherein the obtaining panoramic semantic information from the panoramic image further comprises:

and acquiring background information and/or scene description information of the panoramic image according to the panoramic semantic information of the panoramic image and the corresponding semantic division area.

6. The method of claim 1, wherein the determining one or more focal regions in the one or more local images according to the panoramic semantic information and its corresponding semantic zoning regions comprises:

and selecting one or more pieces of focus semantic information related to the local images from the panoramic semantic information, and determining one or more focus areas in the one or more local images according to the selected focus semantic information and the corresponding semantic division areas.

7. The method of claim 1, wherein the deriving image description information using the panorama semantic information and the detail semantic information comprises:

and fusing the panoramic semantic information and the detail semantic information to obtain the image description information.

8. The method of claim 1, wherein,

the panoramic image and the local image are respectively a frame of video frame collected at the same time in the video.

9. The method of claim 8, wherein when the panoramic image and the local image are respectively a frame of video frames captured at the same time in a video, the obtaining image description information by using the panoramic semantic information and the detail semantic information comprises:

and respectively fusing the panoramic semantic information and the detail semantic information at different moments, and processing according to a time sequence to obtain the image description information which changes along with time.

10. The method of any one of claims 1-9,

the panoramic image is acquired by a panoramic sensor in a multi-sensor imaging system;

the one or more local images are acquired by one or more local sensors in a multi-sensor imaging system.

11. An image processing apparatus comprising:

an acquisition unit that acquires a panoramic image and one or more partial images within the panoramic image;

the semantic dividing unit acquires panoramic semantic information according to the panoramic image, wherein the panoramic semantic information corresponds to semantic dividing areas in the panoramic image;

a focus area obtaining unit, which determines one or more focus areas in the one or more local images according to the panoramic semantic information and the corresponding semantic division areas thereof, and obtains detail semantic information according to the determined focus areas;

and the description unit is used for obtaining image description information by utilizing the panoramic semantic information and the detail semantic information.

12. An image processing apparatus comprising:

a processor;

and a memory having computer program instructions stored therein,

wherein the computer program instructions, when executed by the processor, cause the processor to perform the steps of:

acquiring a panoramic image and one or more local images within the panoramic image;

acquiring panoramic semantic information according to the panoramic image, wherein the panoramic semantic information corresponds to semantic division areas in the panoramic image;

determining one or more focus areas in the one or more local images according to the panoramic semantic information and the corresponding semantic division areas thereof, and acquiring detailed semantic information according to the determined focus areas;

and obtaining image description information by using the panoramic semantic information and the detail semantic information.

13. A computer readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the steps of:

acquiring a panoramic image and one or more local images within the panoramic image;

acquiring panoramic semantic information according to the panoramic image, wherein the panoramic semantic information corresponds to semantic division areas in the panoramic image;

determining one or more focus areas in the one or more local images according to the panoramic semantic information and the corresponding semantic division areas thereof, and acquiring detailed semantic information according to the determined focus areas;

and obtaining image description information by using the panoramic semantic information and the detail semantic information.

Images