CN114897915A - Image segmentation method, device, electronic device and storage medium - Google Patents

Abstract

The present disclosure provides an image segmentation method, an apparatus, an electronic device, and a storage medium, the image segmentation method including: receiving an interactive operation aiming at an image; clipping a corresponding area image from the image according to the interactive operation range; acquiring a result mask of the interactive operation according to a first image mask obtained by the previous interactive operation of the interactive operation, the area image and the mask of the interactive operation; and updating the first image mask to a second image mask by using the result mask of the interactive operation, and obtaining a segmented image based on the second image mask.

Description

Image segmentation method, device, electronic device and storage medium

technical field

The present disclosure relates to the technical field of image processing, and in particular, to an interactive image segmentation method, apparatus, electronic device and storage medium.

Background technique

Image segmentation technology is a very important computer vision task, which has many applications in image retrieval, image editing and film and television production. As a sub-task in the field of image segmentation, Interactive Segmentation aims to achieve the distinction between objects of interest and background with the least user input and reasoning time, and achieve the best segmentation accuracy. Due to the variety of user input methods (such as clicking, scribbling, bounding boxes, etc.), the interactive segmentation method provides users with great flexibility and can effectively adjust the current segmentation results according to user guidance.

The interactive method based on the interactive segmentation technology of the related art mainly includes two interactive operations, click and scribble. In each interactive process, segmentation is performed based on the whole image. The disadvantage of this method is that the full-image-based method reduces the effective resolution of the interactive model network, so it has great limitations for the segmentation of details.

SUMMARY OF THE INVENTION

According to a first aspect of the present disclosure, there is provided an image segmentation method, characterized by comprising: receiving an interactive operation on an image; cropping a corresponding area image from the image according to the scope of the interactive operation; The first image mask, the area image, and the mask of the interactive operation obtained from the previous interactive operation of the operation are obtained, and the result mask of the interactive operation is obtained; An image mask is updated to a second image mask, and a segmented image is obtained based on the second image mask.

According to the first aspect of the present disclosure, the interactive operation includes a smear and/or click operation on an image.

According to the first aspect of the present disclosure, clipping a corresponding area image from the image according to the scope of the interactive operation includes: determining an interactive area corresponding to a trajectory of the interactive operation; and clipping from the image corresponding to the interactive area of the area image.

According to the first aspect of the present disclosure, the interaction area is a circular area, and the clipping the area image corresponding to the interaction area from the image includes: clipping and enclosing the circular area from the image the area image corresponding to the rectangular area of

According to the first aspect of the present disclosure, the interactive area is a rectangular area, and the clipping the area image corresponding to the interactive area from the image includes: clipping the image corresponding to the rectangular area from the image area image.

According to the first aspect of the present disclosure, cropping the area image corresponding to the rectangular area from the image includes: expanding the rectangular area at a predetermined ratio, and cropping out the area corresponding to the expanded rectangular area from the image area image.

According to the first aspect of the present disclosure, the predetermined ratio is inversely related to the size of the rectangular area, and/or the predetermined ratio is inversely related to the degree to which the image is enlarged.

According to the first aspect of the present disclosure, expanding the rectangular area at a predetermined ratio includes enlarging the length of the rectangular area at a first ratio and enlarging the width of the rectangular area at a second ratio.

According to the first aspect of the present disclosure, when the ratio of the size of the rectangular area to the size of the image is smaller than a first threshold value, the predetermined ratio is determined as the first value; when the ratio of the size of the rectangular area to the size of the image is smaller than the first threshold When the ratio is greater than the first threshold, the predetermined ratio is determined as a second value, wherein the first value is greater than the second value, and the first value and the second value are negatively correlated with the magnification of the image.

According to the first aspect of the present disclosure, when the ratio of the size of the area image to the size of the image is smaller than a second threshold value, the area image is expanded into an area image of a first preset size, wherein the second threshold value is smaller than the second threshold value A threshold and inversely related to the factor by which the image is magnified.

According to the first aspect of the present disclosure, the method further includes: adjusting the size of the area image to a second preset size, and obtaining a result mask of the interaction operation based on the adjusted area image.

According to a second aspect of the present disclosure, there is provided an image segmentation apparatus, comprising: an interaction unit configured to receive an interaction operation with respect to an image; a cropping unit configured to crop out a corresponding image from the image according to the range of the interaction operation The inference unit is configured to obtain the result mask of the interactive operation according to the first image mask obtained from the previous interactive operation of the interactive operation, the regional image and the mask of the interactive operation a mask updating unit configured to use the result mask of the interactive operation to update the first image mask to a second image mask, and obtain a segmented image based on the second image mask.

According to the second aspect of the present disclosure, the interactive operation includes a smear and/or click operation in the image.

According to the second aspect of the present disclosure, the cropping unit is configured to: determine the interaction area corresponding to the trajectory of the interaction operation; and crop the area image corresponding to the interaction area from the image.

According to the second aspect of the present disclosure, the interaction area is a circular area, and the cropping unit is configured to: crop the area image corresponding to the rectangular area surrounding the circular area from the image; The partial image corresponding to the circular area is processed into a rectangular image to obtain the area image.

According to the second aspect of the present disclosure, the interaction area is a rectangular area, and the cropping unit is configured to: crop an area image corresponding to the rectangular area from the image.

According to the second aspect of the present disclosure, the cropping unit is configured to expand the rectangular area at a predetermined ratio, and crop the area image corresponding to the expanded rectangular area from an image.

According to the second aspect of the present disclosure, the predetermined ratio is inversely related to the size of the rectangular area, and/or the predetermined ratio is inversely related to the degree to which the image is enlarged.

According to the second aspect of the present disclosure, the trimming unit is configured to enlarge the length of the rectangular area at a first ratio and enlarge the width of the rectangular area at a second ratio.

According to the second aspect of the present disclosure, the cropping unit is configured to: determine the predetermined ratio as a first value when the ratio of the size of the rectangular area to the size of the image is smaller than a first threshold; When the ratio of the size to the size of the image is greater than the first threshold, the predetermined ratio is determined as a second value, wherein the first value is greater than the second value, and the first value and the second value are negatively correlated with the magnification of the image .

According to the second aspect of the present disclosure, the cropping unit is configured to: when the ratio of the size of the area image to the size of the image is smaller than a second threshold, expand the area image into an area image having a first preset size, Wherein, the second threshold is smaller than the first threshold and negatively correlated with the magnification of the image.

According to the second aspect of the present disclosure, the inference unit is configured to: adjust the size of the area image to a second preset size, and based on the adjusted area image, obtain a result mask of the interactive operation according to the present disclosure In a third aspect, an electronic device is provided, comprising: at least one processor; at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause all The at least one processor performs the image segmentation method as described above.

According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium that, when executed by instructions in the computer-readable storage medium, enables the at least one processor to execute the above-described Image segmentation method.

According to a fifth aspect of the present disclosure, there is provided a computer program product whose instructions are executed by at least one processor to perform the image segmentation method as described above.

The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects: the image segmentation method according to the exemplary embodiments of the present disclosure can adaptively locally crop the image according to the scope of the user operation in each interaction, and input the image to the interactive The interactive operations of the segmentation network are reduced. Therefore, the image segmentation method according to the exemplary embodiment of the present disclosure is simple, and even if some wrong operations occur in the user interaction operation, resulting in an unsatisfactory segmentation result of a certain interaction operation, subsequent operations will not be affected. The prediction process has a strong fault tolerance rate, which is very convenient for users to segment and extract objects of interest. At the same time, the method of cropping the region image according to the user interaction operation according to the embodiment of the present disclosure can obtain a finer local region, thus helping the user to extract a high-precision object-of-interest mask, thereby obtaining a more accurate segmented image.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure and do not unduly limit the present disclosure.

FIG. 1 is a flowchart illustrating an image segmentation method according to an exemplary embodiment of the present disclosure.

FIG. 2 is a schematic diagram illustrating an interactive operation in an image segmentation method according to an exemplary embodiment of the present disclosure.

FIG. 3 is a schematic diagram illustrating a local cropping process in an image segmentation method according to an exemplary embodiment of the present disclosure.

FIG. 4 is a block diagram illustrating an image segmentation apparatus according to an exemplary embodiment of the present disclosure.

FIG. 5 is a schematic diagram illustrating an electronic device for image segmentation according to an exemplary embodiment of the present disclosure.

FIG. 6 is a schematic diagram illustrating an electronic device for image segmentation according to another exemplary embodiment of the present disclosure.

Detailed ways

In order to make those skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

It should be noted that the terms "first", "second" and the like in the description and claims of the present disclosure and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used may be interchanged under appropriate circumstances such that the embodiments of the disclosure described herein can be practiced in sequences other than those illustrated or described herein. The implementations described in the following examples are not intended to represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as recited in the appended claims.

It should be noted here that "at least one of several items" in the present disclosure all means including "any one of the several items", "a combination of any of the several items", The three categories of "the whole of the several items" are juxtaposed. For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "execute at least one of step 1 and step 2", which means the following three parallel situations: (1) execute step 1; (2) execute step 2; (3) execute step 1 and step 2.

FIG. 1 is a flowchart of an image segmentation method according to an exemplary embodiment of the present disclosure. It should be understood that the image segmentation scheme according to the exemplary embodiments of the present disclosure can be implemented in any device with image processing capabilities. For example, the method can be performed in a terminal device with image processing capability. Here, the terminal device may be a mobile phone, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, a netbook, a personal digital assistant (PDA), an augmented reality (AR)/virtual reality , VR) equipment. An executable application can be run on the terminal device, and image segmentation processing can be implemented in the application. The image segmentation method can also be performed on, for example, a server. The exemplary embodiments of the present disclosure do not limit it.

As shown in FIG. 1, in step S110, an interactive operation for an image is received.

Here, the image may be an image in a predetermined format (eg, an RGB picture) displayed to the user through the screen of the electronic device. There are target areas to be segmented and background areas other than the target that the user is interested in in the image, and the user can segment the image of the target area from the image through multiple interactive operations. According to an exemplary embodiment of the present disclosure, for each user interaction operation, corresponding processing as described with reference to FIG. 1 may be performed.

According to an exemplary embodiment of the present disclosure, the interactive operation may include a smear operation and/or a click operation. For example, a user can perform a scribbling operation by moving an input device (eg, a stylus, a finger, etc.) in a certain trajectory in the image displayed on the touch screen, and the electronic device can receive the signal by processing the signal received by the sensor. User interaction. In general, smearing operations performed in the target area of interest may be referred to as positive smearing, while smearing operations performed in the background area may be referred to as negative smearing. In an interaction task, in order to obtain a satisfactory result for the user, it is usually necessary to go back and forth for many times of positive and negative interaction.

FIG. 2 shows a schematic diagram of an interaction operation according to an exemplary embodiment of the present disclosure. As shown on the left in FIG. 2 , the user wants to segment the target area of interest of a tennis player from the image, so the user can draw a trajectory as a line on the player's body part to perform positive smearing. In the interactive heatmap on the right shown in Figure 2, you can set the pixel value of the smeared position to 255 (i.e., white as shown in Figure 2), and set the pixel value of the rest of the position to 0 (i.e., except for The areas outside the lines are all black), so that you can get a mask of the trajectory that the user painted. The width of the line can be adjusted according to the application form of the specific product.

In step S120, a corresponding region image is cut out from the image according to the scope of the interactive operation.

According to an exemplary embodiment of the present disclosure, cropping a corresponding area image from the image according to the scope of the interaction operation may include: determining an interaction area corresponding to a trajectory of the interaction operation; cropping the interaction area from the image corresponding to the region image.

According to an exemplary embodiment of the present disclosure, the interaction area may be a circular area. For example, the area of the smallest enclosing circle surrounding the interaction trajectory may be determined as the interaction area. In one example, the maximum length of the coordinates connecting the trajectories may be determined, and the minimum enclosing circle may be obtained with the maximum length as the diameter. The cropping the area image corresponding to the interaction area from the image may include cropping the area image corresponding to the rectangular area surrounding the circular area from the image. In an example, the circle area may be filled with 0 or average pixels to obtain the smallest rectangular area surrounding the circular area, and then an area image corresponding to the smallest rectangular area is clipped from the image. Alternatively, a part of the image corresponding to the circular area may be cropped from the image, and the part of the image may be processed into a rectangular image to obtain the area image. For example, pixels of 0 or average value can be added around the area image of the circular area to obtain an image of the rectangular area.

According to an exemplary embodiment of the present disclosure, the interaction area may be a rectangular area, and the clipping the area image corresponding to the interaction area from the image includes: clipping out the image corresponding to the rectangular area the area image.

It should be understood that the shape of the interaction area is not limited here, and an area of any shape covering the trajectory of the user interaction operation may be used, and the interaction area may be processed into a rectangular image in various ways for subsequent segmentation processing.

According to an exemplary embodiment of the present disclosure, the scope of the interactive operation may be the smallest rectangular area covering the trajectory of the user interactive operation, that is, the smallest rectangular area is the interactive area corresponding to the trajectory of the interactive operation. For example, as shown in FIG. 3 , the rectangular area indicated by 310 is the smallest rectangular area covering the user's smearing track. The minimum rectangular area can be determined by the maximum and minimum values of the coordinates of the track on the screen coordinate axis. In addition, according to an exemplary embodiment of the present disclosure, the region image acquired through the user's interactive operation can be expanded to obtain better contextual information to accurately determine the target region of interest to the user. According to an exemplary embodiment of the present disclosure, the rectangular area obtained according to the range of the user interaction operation may be expanded at a predetermined ratio in step S120. As shown in FIG. 3 , the rectangular area 310 covered by the user interaction can be adaptively expanded according to a predetermined expansion ratio, thereby obtaining a more accurate segmented area 320 . The expansion ratio here refers to the ratio of the length of the expanded part to the original length.

According to an exemplary embodiment of the present disclosure, the predetermined ratio for performing the enlargement is negatively correlated with the size of the rectangular area covered by the user interaction operation, and/or the predetermined ratio is negatively correlated with the degree to which the image is enlarged. That is to say, the expansion ratio according to the type of embodiment of the present disclosure is related to two factors, namely, the size ratio of the original image occupied by the rectangular area covered by the user's interactive operation and the multiple of the user's enlargement of the original image during the interaction process. The larger the size of the original image occupied by the rectangular area covered by the interactive operation, the relatively smaller the expansion ratio can be, and the greater the multiple that the user enlarges the original image in the interactive operation, the relatively smaller the expansion ratio can be. In this way, it can be ensured that a relatively suitable clipping area can be obtained for different user interaction operations, and the clipping area can contain enough information for subsequent target segmentation operations, and will not contain too much information to affect the processing speed.

According to an exemplary embodiment of the present disclosure, expanding the rectangular area at a predetermined ratio includes enlarging a length of the rectangular area at a first ratio, and enlarging a width of the rectangular area at a second ratio. The first ratio and the second ratio may be different or the same. That is to say, the length and width of the rectangular area can be expanded by the same ratio, or the length and width of the rectangular area can be expanded by different ratios, or the first ratio and the second ratio can have a preset relationship, here No restrictions.

In addition, according to an exemplary embodiment of the present disclosure, the cropped area image may be determined according to the smallest rectangular area in a segmented manner. When the ratio of the size of the rectangular area to the size of the image is smaller than the first threshold, the predetermined ratio is determined as the first value; when the ratio of the size of the rectangular area to the size of the image is greater than the first threshold, the predetermined ratio is determined as the first value; The predetermined ratio is determined as a second value, wherein the first value is greater than the second value, and the first value and the second value are negatively correlated with the magnification of the image by the user. For example, when the ratio of the size of the rectangular area to the size of the image is smaller than the first threshold, the predetermined ratio ER is determined as ER=/; when the ratio of the size of the rectangular area to the size of the image is greater than the first threshold , the predetermined ratio ER is determined as ER=β/z, where α>β, z represents the magnification of the image by the user, and α and β are preset according to the business scenario, for example, α=2.0, β= 1.5. Here, the first threshold may be a value between 0-1, for example, may be 0.3. Therefore, the piecewise function can be set as Equation 1 as follows:

Among them, Area(R ^min ) represents the size of the smallest rectangular area, and Area(I ^RGB ) represents the size of the original image.

According to an exemplary embodiment of the present disclosure, when the ratio of the size of the cropped area image to the size of the image is smaller than a second threshold, the cropped area image is expanded into an area with a preset size, wherein the second threshold is smaller than the first threshold and negatively correlated with the factor by which the user magnifies the image. Here, the area image of the preset size is, for example, an area image whose area is a predetermined ratio (eg, 0.2) of the original image. The ratio here can be adjusted according to the actual situation. That is to say, after clipping through the expansion of the minimum rectangular area as described above, there may still be situations in which the clipped area image is small, so that the target of interest to the user cannot be quickly segmented. Therefore, a clipping area Area ( The lower limit of R ^expand ), when Area(R ^expand )/Area(I ^RGB ) is less than a certain hyperparameter threshold τ (ie, the second threshold), the R ^expand area can be further expanded, and τ is defined as follows:

τ=γ/z

Among them, when z is larger, τ will be smaller in response. According to an exemplary embodiment of the present disclosure, the default value of γ may be 0.34. Through the above segmented expansion and the limitation of the lower limit value, the local area image can be more refined and cropped.

It should be understood that the above manners of expanding the scope of the user's interactive operations are merely illustrative, and those skilled in the art can make various changes according to the needs of actual business scenarios.

In other embodiments of the present disclosure, the interaction area may also be other geometric shapes than circles and rectangles, such as ellipses, or target contours targeted by the interaction operation. There are various implementations of cropping region images:

In one example, the area image corresponding to the rectangular area surrounding the interaction area is cropped from the image. For example, 0 or average pixels can be filled around the interactive area to obtain the smallest rectangular area surrounding the interactive area, and then an area image corresponding to the smallest rectangular area can be cropped from the image. It should be understood that the minimum rectangular area is only an example, and the obtained rectangular area surrounding the interaction area may not be the minimum rectangular area, and those skilled in the art can make various changes according to actual business scenarios.

In another example, a part of the image corresponding to the interaction area is cropped from the image, and the part of the image is processed into a rectangular image to obtain the area image. For example, zero or average pixels can be added around the area image of the interaction area to obtain an image of a rectangular area.

It can be seen that the obtained region image is a rectangular image.

In step S130, a result mask of the interactive operation is obtained according to the first image mask, the region image and the mask of the interactive operation obtained by the previous interactive operation of the interactive operation. The resulting mask output here is the segmentation mask corresponding to the cropped region image.

As mentioned above, in each interactive operation of the user, the image mask obtained after the last interaction, the region image cropped in step S120 and the mask of this interactive operation will be used, and these will be input into the In the trained interactive image segmentation inference model, the result mask of this interaction can be output. The inference model here can be a Convolutional Neural Network (CNN) model for interactive image segmentation trained by mask-guided restorative interactions, which can be based on input pictures, user guidance, and masks from the last interaction As a result, a predicted mask is generated for the input image after passing through the neural network. It should be understood that the present application may also use other trained inference models in the form of forward feedback to perform inference to obtain corresponding masks.

According to an exemplary embodiment of the present disclosure, the size of the cropped region through the interactive operation is adjusted to a predetermined size and then input to the interactive image segmentation inference model. This is because, typically, models require fixed-size inputs. Therefore, the clipped area can be adjusted to a fixed size for the convenience of the model.

In step S140, the first image mask is updated to a second image mask using the result mask of the interaction operation, and a segmented image is obtained based on the second image mask. After the result mask corresponding to the clipping area is obtained, the result mask is updated to the corresponding area of the full-image mask after the last interactive operation on the image, so as to obtain the full-image mask after this interaction. That is to say, the position of the result mask of the interactive operation can be recorded, and the obtained result mask can be updated to the full image mask obtained by the previous interactive operation according to the position of the result mask, so that a new full image mask can be obtained. code. After the new full-image mask is obtained, the image of the target region of interest to the user can be obtained through the full-image mask as the final image segmentation result. It should be understood that if there is no previous interactive operation before the current interactive operation (ie, the current interactive operation is the first interactive operation), the first image mask is the original image mask, and all the pixels of the original image mask The value can be 0.

For example, if the user's interactive input is positive smearing (ie, smearing on the object of interest), the output result mask can be updated with a pixel value of 255 into the corresponding area of the full-image mask. If the user's input is a negative smear (ie, a smear on the background area), the output result mask can be updated with a pixel value of 0 into the corresponding area of the full-image mask.

The image segmentation method according to the exemplary embodiment of the present disclosure can adaptively crop the image locally according to the scope of the user's operation in each interaction, and then feed it into the interactive segmentation model to generate the segmentation mask of the user's interest target. In addition, in the related art, the results of all interactive operations before the current interactive operation need to be input into the interactive segmentation model for processing, so if an error occurs in one interactive operation, it will interfere with the final image segmentation result, and the error tolerance rate is relatively high. Small. However, according to the image segmentation method according to the exemplary embodiment of the present disclosure, the interactive operations input to the interactive segmentation network are reduced. Therefore, the image segmentation method according to the exemplary embodiment of the present disclosure is simple, even if some wrong operations occur in the user interactive operation. As a result, the segmentation result of an interactive operation is not ideal, and it will not affect the subsequent prediction process. At the same time, the method of cropping the region image according to the user interaction operation according to the embodiment of the present disclosure can obtain a finer local region, thus helping the user to extract a high-precision object-of-interest mask, thereby obtaining a more accurate segmented image.

FIG. 4 is a block diagram illustrating an image segmentation apparatus according to an exemplary embodiment of the present disclosure. It should be understood that the image segmentation apparatus can be implemented in an electronic device with image processing in a software/hardware or a combination of software and hardware.

As shown in FIG. 4 , the image segmentation apparatus 400 according to an exemplary embodiment of the present disclosure may include an interaction unit 410 , a cropping unit 420 , an inference unit 430 and a mask updating unit 440 .

The interaction unit 410 is configured to receive an interaction operation with respect to an image. The cropping unit 420 is configured to crop a corresponding area image from the image according to the range of the interactive operation.

The reasoning unit 430 is configured to obtain a result mask of the interaction operation according to the first image mask obtained by the previous interaction operation of the interaction operation, the region image and the mask of the interaction operation.

The mask update unit 440 is configured to update the first image mask to the second image mask using the result mask of the interaction operation, and obtain a segmented image based on the second image mask.

According to an exemplary embodiment of the present disclosure, the interactive operation includes a smear and/or click operation in the image.

According to an exemplary embodiment of the present disclosure, the cropping unit is configured to: determine an interaction area corresponding to the trajectory of the interaction operation; and crop the area image corresponding to the interaction area from an image.

According to an exemplary embodiment of the present disclosure, the interaction area is a circular area, and the cropping unit is configured to: crop out the area image corresponding to the rectangular area surrounding the circular area from the image; or crop out the image a partial image corresponding to the circular area, and processing the partial image into a rectangular image to obtain the area image.

According to an exemplary embodiment of the present disclosure, the interaction area is a rectangular area, and the cropping unit is configured to: crop an area image corresponding to the rectangular area from the image.

According to an exemplary embodiment of the present disclosure, the cropping unit 420 may be configured to expand the rectangular area at a predetermined ratio, and crop the area image corresponding to the expanded rectangular area from an image.

According to an exemplary embodiment of the present disclosure, the predetermined ratio is inversely related to the size of the rectangular area, and/or the predetermined ratio is inversely related to the degree to which the image is enlarged in the interactive operation.

According to an exemplary embodiment of the present disclosure, the trimming unit 420 may be configured to enlarge the length of the rectangular area at a first ratio and enlarge the width of the rectangular area at a second ratio. The first ratio and the second ratio may be independent of each other.

According to an exemplary embodiment of the present disclosure, the cropping unit 420 may be configured to: when the ratio of the size of the rectangular area to the size of the image is smaller than a first threshold, determine the predetermined ratio as a first value; When the ratio of the size of the rectangular area to the size of the image is greater than the first threshold, the predetermined ratio is determined as a second value, wherein the first value is greater than the second value, and the first value and the second value and the image are enlarged by the user The multiples of are negatively correlated.

According to an exemplary embodiment of the present disclosure, the cropping unit 420 may be configured to: when the ratio of the size of the area image to the size of the image is smaller than a second threshold, expand the area image to have a first preset size , where the second threshold is smaller than the first threshold and negatively correlated with the magnification of the image.

According to an exemplary embodiment of the present disclosure, the reasoning unit 430 may be configured to: adjust the size of the area image to a second preset size, and obtain a result mask of the interaction operation based on the adjusted area image .

The image segmentation method according to an exemplary embodiment of the present disclosure has been described above with reference to FIG. 1 , and each step in the method corresponds to each unit of the image segmentation apparatus shown in FIG. 4 , so the description will not be repeated.

FIG. 5 is a structural block diagram illustrating an electronic device 500 for video processing for image segmentation according to an exemplary embodiment of the present disclosure. The electronic device 500 may be, for example, a smart phone, a tablet computer, an MP4 (Moving PictureExperts Group Audio Layer IV) player, a notebook computer or a desktop computer. Electronic device 500 may also be called user equipment, portable terminal, laptop terminal, desktop terminal, and the like by other names.

Generally, the electronic device 500 includes: a processor 501 and a memory 502 .

The processor 501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 501 may be implemented by at least one hardware form among DSP (Digital Signal Processing, digital signal processing), FPGA (FieldProgrammable Gate Array, field programmable gate array), and PLA (Programmable Logic Array, programmable logic array). The processor 501 may also include a main processor and a coprocessor. The main processor is a processor used to process data in a wake-up state, also called a CPU (Central Processing Unit, central processing unit); A low-power processor for processing data in a standby state. In some embodiments, the processor 501 may be integrated with a GPU (Graphics Processing Unit, image processor), and the GPU is used for rendering and drawing the content that needs to be displayed on the display screen. In an exemplary embodiment of the present disclosure, the processor 501 may further include an AI (Artificial Intelligence, artificial intelligence) processor for processing computing operations related to machine learning.

Memory 502 may include one or more computer-readable storage media, which may be non-transitory. Memory 502 may also include high-speed random access memory, as well as non-volatile memory, such as one or more disk storage devices, flash storage devices. In some embodiments, a non-transitory computer-readable storage medium in memory 502 is used to store at least one instruction for execution by processor 501 to implement image segmentation of exemplary embodiments of the present disclosure method.

In some embodiments, the electronic device 500 may also optionally include: a peripheral device interface 503 and at least one peripheral device. The processor 501, the memory 502 and the peripheral device interface 503 may be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 503 through a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 504 , a touch display screen 505 , a camera 506 , an audio circuit 507 , a positioning component 508 and a power supply 509 .

The peripheral device interface 503 may be used to connect at least one peripheral device related to I/O (Input/Output) to the processor 501 and the memory 502 . In some embodiments, processor 501, memory 502, and peripherals interface 503 are integrated on the same chip or circuit board; in some other embodiments, any one of processor 501, memory 502, and peripherals interface 503 or The two can be implemented on a separate chip or circuit board, which is not limited in this embodiment.

The radio frequency circuit 504 is used for receiving and transmitting RF (Radio Frequency, radio frequency) signals, also called electromagnetic signals. The radio frequency circuit 504 communicates with the communication network and other communication devices via electromagnetic signals. The radio frequency circuit 504 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 504 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 504 may communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: metropolitan area network, mobile communication networks of various generations (2G, 3G, 4G and 5G), wireless local area network and/or WiFi (Wireless Fidelity, wireless fidelity) network. In some embodiments, the radio frequency circuit 504 may further include a circuit related to NFC (Near Field Communication, short-range wireless communication), which is not limited in the present disclosure.

The display screen 505 is used to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 505 is a touch display screen, the display screen 505 also has the ability to acquire touch signals on or above the surface of the display screen 505 . The touch signal may be input to the processor 501 as a control signal for processing. At this time, the display screen 505 may also be used to provide virtual buttons and/or virtual keyboards, also referred to as soft buttons and/or soft keyboards. In some embodiments, there may be one display screen 505, which is arranged on the front panel of the electronic device 500; in other embodiments, there may be at least two display screens 505, which are respectively arranged on different surfaces of the terminal 500 or in a folded design. ; In some embodiments, the display screen 505 may be a flexible display screen, disposed on a curved surface or a folding surface of the terminal 500 . Even, the display screen 505 can also be set as a non-rectangular irregular figure, that is, a special-shaped screen. The display screen 505 can be made of materials such as LCD (Liquid Crystal Display, liquid crystal display), OLED (Organic Light-Emitting Diode, organic light emitting diode).

The camera assembly 506 is used to capture images or video. Optionally, the camera assembly 506 includes a front camera and a rear camera. Usually, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth-of-field camera to realize the background blur function, the main camera It is integrated with the wide-angle camera to achieve panoramic shooting and VR (Virtual Reality, virtual reality) shooting functions or other integrated shooting functions. In some embodiments, the camera assembly 506 may also include a flash. The flash can be a single color temperature flash or a dual color temperature flash. Dual color temperature flash refers to the combination of warm light flash and cold light flash, which can be used for light compensation under different color temperatures.

Audio circuitry 507 may include a microphone and speakers. The microphone is used to collect the sound waves of the user and the environment, convert the sound waves into electrical signals and input them to the processor 501 for processing, or input them to the radio frequency circuit 504 to realize voice communication. For the purpose of stereo collection or noise reduction, there may be multiple microphones, which are respectively disposed in different parts of the terminal 500 . The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signal from the processor 501 or the radio frequency circuit 504 into sound waves. The loudspeaker can be a traditional thin-film loudspeaker or a piezoelectric ceramic loudspeaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for distance measurement and other purposes. In some embodiments, the audio circuit 507 may also include a headphone jack.

The positioning component 508 is used to locate the current geographic location of the electronic device 500 to implement navigation or LBS (Location Based Service, location-based service). The positioning component 508 may be a positioning component based on the GPS (Global Positioning System, global positioning system) of the United States, the Beidou system of China, the Grenas system of Russia, or the Galileo system of the European Union.

Power supply 509 is used to power various components in electronic device 500 . The power source 509 may be alternating current, direct current, disposable batteries or rechargeable batteries. When the power source 509 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

In some embodiments, the electronic device 500 also includes one or more sensors 510 . The one or more sensors 510 include, but are not limited to, an acceleration sensor 511 , a gyro sensor 512 , a pressure sensor 513 , a fingerprint sensor 514 , an optical sensor 515 and a proximity sensor 516 .

The acceleration sensor 311 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 500 . For example, the acceleration sensor 511 can be used to detect the components of the gravitational acceleration on the three coordinate axes. The processor 501 can control the touch display screen 505 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 511 . The acceleration sensor 511 can also be used for game or user movement data collection.

The gyroscope sensor 512 can detect the body direction and rotation angle of the terminal 500 , and the gyroscope sensor 512 can cooperate with the acceleration sensor 511 to collect 3D actions of the user on the terminal 500 . The processor 501 can implement the following functions according to the data collected by the gyro sensor 512 : motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

The pressure sensor 513 may be disposed on the side frame of the terminal 500 and/or the lower layer of the touch display screen 505 . When the pressure sensor 513 is disposed on the side frame of the terminal 500, the user's holding signal of the terminal 500 can be detected, and the processor 501 can perform left and right hand identification or quick operation according to the holding signal collected by the pressure sensor 513. When the pressure sensor 513 is disposed on the lower layer of the touch display screen 505 , the processor 501 controls the operability controls on the UI according to the user's pressure operation on the touch display screen 505 . The operability controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

The fingerprint sensor 514 is used to collect the user's fingerprint, and the processor 501 identifies the user's identity according to the fingerprint collected by the fingerprint sensor 514, or the fingerprint sensor 514 identifies the user's identity according to the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 501 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 514 may be provided on the front, back, or side of the electronic device 500 . When the electronic device 500 is provided with physical buttons or a manufacturer's logo, the fingerprint sensor 514 may be integrated with the physical buttons or the manufacturer's logo.

Optical sensor 515 is used to collect ambient light intensity. In one embodiment, the processor 501 may control the display brightness of the touch display screen 505 according to the ambient light intensity collected by the optical sensor 515 . Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 505 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 505 is decreased. In another embodiment, the processor 501 may also dynamically adjust the shooting parameters of the camera assembly 506 according to the ambient light intensity collected by the optical sensor 515 .

Proximity sensor 516 , also referred to as a distance sensor, is typically provided on the front panel of electronic device 500 . The proximity sensor 516 is used to collect the distance between the user and the front of the electronic device 500 . In one embodiment, when the proximity sensor 516 detects that the distance between the user and the front of the terminal 500 gradually decreases, the processor 501 controls the touch display screen 505 to switch from the bright screen state to the off screen state; when the proximity sensor 516 detects When the distance between the user and the front of the electronic device 500 gradually increases, the processor 501 controls the touch display screen 505 to switch from the off-screen state to the bright-screen state.

Those skilled in the art can understand that the structure shown in FIG. 3 does not constitute a limitation on the electronic device 500, and may include more or less components than the one shown, or combine some components, or adopt different component arrangements.

FIG. 6 is a structural block diagram of another electronic device 600 . For example, the electronic device 600 may be provided as a server. Referring to FIG. 6 , an electronic device 600 includes one or more processing processors 610 and a memory 620 . The memory 620 may include one or more programs for performing the above image segmentation method. The electronic device 600 may also include a power supply assembly 630 configured to perform power management of the electronic device 600, a wired or wireless network interface 640 configured to connect the electronic device 600 to a network, and an input output (I/O) interface 650 . Electronic device 600 may operate based on an operating system stored in memory 620, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

According to an embodiment of the present disclosure, there may also be provided a computer-readable storage medium storing instructions, wherein the instructions, when executed by at least one processor, cause the at least one processor to perform the image segmentation method according to the present disclosure. Examples of the computer-readable storage medium herein include: Read Only Memory (ROM), Random Access Programmable Read Only Memory (PROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Random Access Memory (RAM) , dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM , DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or Optical Disc Storage, Hard Disk Drive (HDD), Solid State Hard disk (SSD), card memory (such as a multimedia card, Secure Digital (SD) card, or Extreme Digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other apparatus configured to store and provide the computer program and any associated data, data files and data structures in a non-transitory manner with the computer program and any associated data, data files and data structures The computer program is given to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer readable storage medium can be executed in an environment deployed in a computer device such as a client, a host, a proxy device, a server, etc. Furthermore, in one example, the computer program and any associated data, data files and data structures are distributed over networked computer systems so that the computer programs and any associated data, data files and data structures are stored, accessed and executed in a distributed fashion by one or more processors or computers.

According to an embodiment of the present disclosure, a computer program product can also be provided, wherein instructions in the computer program product can be executed by a processor of a computer device to complete the above-mentioned image segmentation method.

The image segmentation method according to the exemplary embodiment of the present disclosure can adaptively locally crop the image according to the range of user operations in each interaction, reduce the number of interaction operations input to the interaction segmentation network, and have a strong error tolerance rate. A finer local area can be obtained and a more accurate segmented image can be obtained.

Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the schemes disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or techniques in the technical field not disclosed by the present disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (10)

1. an image segmentation method, is characterized in that, comprises: receive interactive operations on images; Cut out the corresponding area image from the image according to the scope of the interactive operation; Obtain the result mask of the interactive operation according to the first image mask, the area image and the mask of the interactive operation obtained by the previous interactive operation of the interactive operation; The first image mask is updated to a second image mask using the result mask of the interaction operation, and a segmented image is obtained based on the second image mask. 2. The method according to claim 1, wherein, according to the scope of the interactive operation, cutting out a corresponding region image from the image, comprising: determining the interaction area corresponding to the trajectory of the interaction operation; The region image corresponding to the interaction region is cropped from the image. 3. The method according to claim 2, wherein the interactive area is a rectangular area, and the clipping the area image corresponding to the interactive area from the image comprises: The area image corresponding to the rectangular area is cropped from the image. 4. The method of claim 3, wherein clipping the area image corresponding to the rectangular area from the image comprises: The rectangular area is expanded at a predetermined ratio, and the area image corresponding to the expanded rectangular area is cropped from the image. 5. The method of claim 4, wherein the predetermined ratio is inversely related to the size of the rectangular area, and/or The predetermined ratio is inversely related to the degree to which the image is enlarged. 6. The method of claim 5, wherein expanding the rectangular area by a predetermined ratio comprises: enlarging a length of the rectangular area by a first ratio and a width of the rectangular area by a second ratio. 7. The method of claim 5, wherein, When the ratio of the size of the rectangular area to the size of the image is smaller than a first threshold, determining the predetermined ratio as a first value; When the ratio of the size of the rectangular area to the size of the image is greater than a first threshold value, determining the predetermined ratio as a second value, Wherein, the first value is greater than the second value, and the first value and the second value are negatively correlated with the magnification of the image. 8. An image segmentation device, comprising: an interaction unit configured to receive an interaction operation for the image; a cropping unit, configured to crop a corresponding area image from the image according to the scope of the interactive operation; an inference unit, configured to obtain the result mask of the interactive operation according to the first image mask, the area image and the mask of the interactive operation obtained from the previous interactive operation of the interactive operation; A mask updating unit configured to update the first image mask to a second image mask using the result mask of the interactive operation, and obtain a segmented image based on the second image mask. 9. An electronic device, characterized in that, comprising: at least one processor; at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, cause the at least one processor to perform the method of any one of claims 1-7. 10. A computer-readable storage medium, the instructions in the computer-readable storage medium, when executed by at least one processor, enable the at least one processor to perform the execution of any one of claims 1 to 7 method described.

Images