CN111258408A - An object boundary determination method and device for human-computer interaction - Google Patents

Abstract

The invention provides an object boundary determining method and device for man-machine interaction, wherein the method comprises the following steps: shooting scene images in real time by using a wide-angle camera, and sending each frame of shot scene images to a computing board; the computing board judges whether the current scene can determine a boundary or not based on each acquired scene image, and if so, determines the boundary range of the object. The main advantages of the invention are: after the scene image is judged, the position of the blank area is obtained by removing the object in the scene image, so that the delimiting accuracy is ensured, the invention supports the free switching of the delimitation of the user operation interface and the delimitation of the display content, can further extract the content of the user operation interface, is accompanied by the reminding of characters and frames, and is added with the real-time tracking effect of projection, so that the user operation interface projected on a moving object is realized by having good visual effect no matter the display condition of the projected area or the display condition of the panel and realizing the real-time update of the delimitation, and the user experience is greatly improved.

Description

An object boundary determination method and device for human-computer interaction

technical field

The present invention relates to the technical field of human-computer interaction, in particular to an object boundary determination method and device for human-computer interaction.

Background technique

With the development of information technology, many real-world technologies continue to develop. In particular, augmented reality technology has been widely used in entertainment, engineering, etc., allowing people in the real world to solve some virtual things at close range. The technologies involved include multimedia, Three-dimensional modeling, real-time tracking, intelligent interaction, sensing and other technical means.

Human-computer interaction is the study of the interaction between systems and users. A system can be a variety of machines, as well as computerized systems and software. Human-computer interface usually refers to the part that is visible to the user. The user communicates with the system through the human-computer interface and performs operations.

In the prior art, when the user interface is projected through the projector, the position, size, etc. of the user interface are generally adjusted manually, which is time-consuming and laborious, and the prior art is difficult to project the user interface on a non-blank interface, such as in textbooks. The user operation interface is projected, and in the prior art, it is impossible to automatically identify the displayed content in the operation interface to perform preliminary delimitation, and in the existing delimitation mode, it is impossible to adaptively determine the delimitation method according to the projected content. , resulting in low human-computer interaction efficiency, and the existing technology cannot adaptively adjust the distance between the projector and the projection interface according to the size of the delimiting object, resulting in blurred projection and affecting user experience.

SUMMARY OF THE INVENTION

The present invention proposes the following technical solutions in view of the above-mentioned defects in the prior art.

An object boundary determination method for human-computer interaction, the method comprising:

In the step of acquiring scene information, a wide-angle camera is used to capture a scene image in real time, and each frame of the captured scene image is sent to the computing board;

In the determining step, the computing board determines whether the current scene can determine the boundary based on the acquired scene images of each frame, and if so, determines the boundary range of the object.

Further, the object is a user operation interface or displayed content.

Further, when the object is a user operation interface, the determining step includes:

The computing board receives each frame of scene image transmitted by the wide-angle camera in real time, and uses the mobilenet-ssd detection network to process the object distribution of the current scene to determine the shape of each object and the corresponding category of the object;

The calculation board calculates the position of each object in the space of the scene image based on the scene image and the determined object shape, and generates an object data set after merging the position of each object and the corresponding category;

The computing board reads the position information from the object data set, subtracts all the object distribution information from the scene image based on the position information to obtain blank area information, and then the computing board will determine whether the boundary can be performed according to user settings. Determine, if yes, calculate the position information of the delimitable area to determine the boundary range, and store the boundary range;

The computing board transmits the stored boundary range to the projection unit, and at the same time transmits a signal of successful delimitation, when the projection unit receives the signal of successful delimitation, obtains the stored setting information of the current user from the computing board ;

The computing board determines whether to perform projection in the blank area, if so, determines the projection area according to the boundary range and the setting information, and projects the user's operation interface in the projection area; if not, the user selects the projection area. The object to be projected, then the computing board reads the position information of the object in the object data set, and projects the user's operation interface based on the position information of the object.

Further, when the object is a user operation interface, the method further includes:

In the first update step, the computing board processes the scene image transmitted by the wide-angle camera at the first time interval and compares the scene image with the previously determined boundary. If the comparison result is inconsistent, the boundary is determined again.

Further, when the object is a user operation interface, the updating step includes:

The computing board obtains a frame of scene image from the wide-angle camera every second, and uses the mobilenet-ssd detection network to obtain the distribution state of all objects in the frame of scene image;

The computing board obtains the setting information of the current projection unit, and compares it with the distribution state of the object. If the error of the comparison result is greater than the first threshold, the boundary determination cannot be performed, and the projection unit is updated to the non-delimitable warning state. , and adjust the state of the computing board to a state that determines whether it can be delimited in real time; if the error of the comparison result is less than the first threshold, a new boundary range will be determined, and compared with the previously stored boundary range, if the comparison If the result is less than the second threshold, no update is performed; otherwise, the new boundary range is stored and transmitted to the projection unit, and the projection unit adjusts the projection area accordingly according to the new boundary range.

Further, when the object is displayed content, the content is the content displayed on the user operation interface, and the determining step includes:

The wide-angle camera captures a scene image in real time and transmits it to the computing board at a second time interval, and the computing board transmits the scene image to a cloud server; the cloud server uses a deep learning learning network to predict the Chinese characters of the content position, and at the same time, the picture containing the text is cropped to obtain the first sub-picture and stored;

The cloud server uses the ctc algorithm to identify the text content of the first sub-picture, and generates a content data set from the text and the corresponding position after the recognition;

the server transmits the content data set to the computing pad, the computing pad transmits information of the location in the content data set to the projection unit;

The projection unit projects the content data set obtained by the computing board, and the user selects the content that needs to be delimited.

Further, the content data set obtained by the projection unit projected by the computing board and the content that needs to be delimited selected by the user includes:

The projection unit monitors the computing board in real time, and displays the content in a light color in the projection area after receiving the recognized text and position sent by the computing board;

The user selects according to the displayed identified content, and after selection, the boundary at the position of the corresponding content will be made obvious, indicating that the current content has been selected.

Further, the obvious is adding an outer frame to display.

Further, when the object is displayed content, the method further includes:

In the second update step, the computing board performs in-depth identification of the content selected by the user, obtains the specific information of the content, and updates the content to the projection unit for projection display.

Further, the updating step includes:

When the user selects an identified area, the computing board obtains the user's selected area and records the location of the selected area;

The computing board cuts the region selected by the user into a second sub-picture based on the position of the selected region, and uses an intelligent recognition API to analyze the text or picture information in the second sub-picture;

The computing board combines the specific information and position information of the analyzed text or picture to obtain detailed information of the selected area, and extracts the effective part of the detailed information for normalization to obtain standardized data and transmits it to the projection unit. ;

After the projection unit receives the normative data from the computing board, the corresponding display is updated in the user operation area in the projection area.

The present invention also provides an object boundary determination device for human-computer interaction, the device comprising: a projection unit, a wide-angle camera and a computing board;

The wide-angle camera captures the scene image in real time, and sends each frame of the captured scene image to the computing board; after the computing board receives the each frame of the scene image, it determines whether the current scene can be Determine the bounds, and if so, determine the bounds of the object.

Further, the object is a user operation interface or displayed content.

Further, when the object is a user operation interface, after receiving the each frame of the scene image, the computing board determines whether the current scene can determine the boundary based on each frame of the acquired scene image, and if so, determines the The bounds of the object include:

The computing board receives each frame of scene image transmitted by the wide-angle camera in real time, and uses the mobilenet-ssd detection network to process the object distribution of the current scene to determine the shape of each object and the corresponding category of the object;

The calculation board calculates the position of each object in the space of the scene image based on the scene image and the determined object shape, and generates an object data set after merging the position of each object and the corresponding category;

The computing board reads the position information from the object data set, subtracts all the object distribution information from the scene image based on the position information to obtain blank area information, and then the computing board will determine whether the boundary can be performed according to user settings. Determine, if yes, calculate the position information of the delimitable area to determine the boundary range, and store the boundary range;

The computing board transmits the stored boundary range to the projection unit, and at the same time transmits a signal of successful delimitation, when the projection unit receives the signal of successful delimitation, obtains the stored setting information of the current user from the computing board ;

The computing board determines whether to perform projection in the blank area, if so, determines the projection area according to the boundary range and the setting information, and projects the user's operation interface in the projection area; if not, the user selects the projection area. The object to be projected, then the computing board reads the position information of the object in the object data set, and projects the user's operation interface based on the position information of the object.

Further, when the object is a user operation interface, the computing board processes the scene image transmitted by the wide-angle camera at the first time interval and compares it with the previously determined scene image of the boundary. If the comparison results are inconsistent, then Redo the boundary determination.

Further, when the object is a user operation interface, the computing board processes the scene image transmitted by the wide-angle camera at the first time interval and compares it with the previously determined scene image of the boundary. If the comparison results are inconsistent, then Redefining boundaries includes:

The computing board obtains a frame of scene image from the wide-angle camera every second, and uses the mobilenet-ssd detection network to obtain the distribution state of all objects in the frame of scene image;

The computing board obtains the setting information of the current projection unit, and compares it with the distribution state of the object. If the error of the comparison result is greater than the first threshold, the boundary determination cannot be performed, and the projection unit is updated to the non-delimitable warning state. , and adjust the state of the computing board to a state that determines whether it can be delimited in real time; if the error of the comparison result is less than the first threshold, a new boundary range will be determined, and compared with the previously stored boundary range, if the comparison If the result is less than the second threshold, no update is performed; otherwise, the new boundary range is stored and transmitted to the projection unit, and the projection unit adjusts the projection area accordingly according to the new boundary range.

Further, when the object is the displayed content, the content is the content displayed on the user operation interface, and the computing board determines the current image based on each frame of the scene image obtained after receiving the each frame of the scene image. Whether the scene can determine the boundary, if so, determine the boundary range of the object including:

The wide-angle camera captures a scene image in real time and transmits it to the computing board at a second time interval, and the computing board transmits the scene image to a cloud server; the cloud server uses a deep learning learning network to predict the Chinese characters of the content position, and at the same time, the picture containing the text is cropped to obtain the first sub-picture and stored;

The cloud server uses the ctc algorithm to identify the text content of the first sub-picture, and generates a content data set from the text and the corresponding position after the recognition;

the server transmits the content data set to the computing pad, the computing pad transmits information of the location in the content data set to the projection unit;

The projection unit projects the content data set obtained by the computing board, and the user selects the content that needs to be delimited.

Further, the content data set obtained by the projection unit projected by the computing board and the content that needs to be delimited selected by the user includes:

The projection unit monitors the computing board in real time, and displays the content in a light color in the projection area after receiving the recognized text and position sent by the computing board;

The user selects according to the displayed identified content, and after selection, the boundary at the position of the corresponding content will be made obvious, indicating that the current content has been selected.

Further, the obvious is adding an outer frame to display.

Further, when the object is displayed content, the computing board performs in-depth identification of the content selected by the user, obtains the specific information of the content, and updates it to the projection unit for projection display.

Further, the computing board performs in-depth identification of the content selected by the user, and after obtaining the specific information of the content, updating it to the projection unit and performing projection display includes:

When the user selects an identified area, the computing board obtains the user's selected area and records the location of the selected area;

The computing board cuts the region selected by the user into a second sub-picture based on the position of the selected region, and uses an intelligent recognition API to analyze the text or picture information in the second sub-picture;

The computing board combines the specific information and position information of the analyzed text or picture to obtain detailed information of the selected area, and extracts the effective part of the detailed information for normalization to obtain standardized data and transmits it to the projection unit. ;

After the projection unit receives the normative data from the computing board, the corresponding display is updated in the user operation area in the projection area.

The technical effect of the present invention is: an object boundary determination method for human-computer interaction according to the present invention, the method includes: the step of acquiring scene information, using a wide-angle camera to capture scene images in real time, and recording each frame of the captured scene images Send to the computing board; in the determining step, the computing board determines whether the current scene can determine the boundary based on each frame of scene images obtained, and if so, determines the boundary range of the object. The main advantages of the present invention are: after judging the scene image, the present invention obtains the position of the blank area by removing the objects in the scene image, so as to make the delimitation accuracy, and then make the projected user interface very clear; and the present invention supports The delimitation of the user interface and the delimitation of the displayed content can be freely switched. Therefore, it is helpful to add other operations during the projection and delimitation. For example, the content can be further extracted, such as extracting specific text and retrieving pictures. In addition, the information can also be displayed directly by means of projection. For content delimitation, the projection automatically gives the recognized area, and comes with reminders of text and borders, and coupled with the real-time tracking effect of the projection, the delimitation , both the projected area display and the panel display have good visual effects, and the delimitation is updated in real time, thereby realizing the projection user interface on the moving object, and can realize the projector and interface based on the size of the object. The automatic adjustment of the distance greatly improves the user experience.

Description of drawings

Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments taken with reference to the following drawings.

FIG. 1 is a flowchart of an object boundary determination method for human-computer interaction according to one of the embodiments of the present invention.

FIG. 2 is a schematic diagram of an object boundary determination device for human-computer interaction according to one of the embodiments of the present invention.

Detailed ways

The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the related invention, but not to limit the invention. In addition, it should be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.

It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

FIG. 1 shows a method for determining an object boundary for human-computer interaction according to the present invention, and the method includes:

The scene information acquisition step S101 is to use a wide-angle camera to capture a scene image in real time, and send each frame of the captured scene image to the computing board;

In determination step S102, the computing board determines whether the current scene can determine the boundary based on each frame of the acquired scene image, and if so, determines the boundary range of the object.

The method of the present invention can be applied to a smart desk lamp. The smart desk lamp has a projection unit, that is, a projector, a wide-angle camera, a depth camera, an infrared camera, etc., and has a computing board inside. In order to complete the processing of data, etc., of course, it must also have a power supply, a power supply controller, and the like. The projection unit may be a projector, and through the method of the present invention, the boundary of the projection unit to project an operation interface on the desktop can be determined. When the user operates on the operation interface, the boundary of the displayed content can also be determined. As for which object to delimit, the computing board judges according to the content projected by the current projection unit, and determines whether to delimit the user operation interface according to the judgment result. Or delimit the display content, and then perform the corresponding delimitation operation. For example, during initialization, the projected user operation interface is delimited. After the user operation interface is provided, when the user operates on the operation interface, the The content displayed on the operation interface is delimited, that is, the present invention supports the free switching between the delimitation of the user operation interface and the delimitation of the display content.

In one embodiment, when the computing board judges according to the content projected by the current projection unit, and determines according to the judgment result that the user operation interface is delimited, that is, when the object is the user operation interface, the determining step S102 includes:

The computing board receives each frame of scene image transmitted by the wide-angle camera in real time, and uses the mobilenet-ssd detection network to process the object distribution in the current scene to determine the shape of each object and the corresponding category of each object;

The calculation board calculates the position of each object in the space of the scene image based on the scene image and the determined object shape, and generates an object data set after merging the position of each object and the corresponding category;

The computing board reads the position information from the object data set, subtracts all the object distribution information from the scene image based on the position information to obtain blank area information, and then the computing board will determine whether the boundary can be performed according to user settings. Determine, if yes, calculate the position information of the delimitable area to determine the boundary range, and store the boundary range;

The computing board transmits the stored boundary range to the projection unit, and at the same time transmits a signal of successful delimitation, when the projection unit receives the signal of successful delimitation, obtains the stored setting information of the current user from the computing board ;

The computing board determines whether to perform projection in the blank area, if so, determines the projection area according to the boundary range and the setting information, and projects the user's operation interface in the projection area; if not, the user selects the projection area. The object to be projected, then the computing board reads the position information of the object in the object data set, and projects the user's operation interface based on the position information of the object.

There are two main ways of delimitation, one is the projection of the blank area (projection on the blank area), if the size of the area meets the projection range set by the user, it means that it can be delimited, which is a more traditional way; The first method is based on the projection of the recognized object, that is, projected on a specific book or paper. If there is an object that can be projected, such as a book, paper, etc., it means that the boundary can be delimited. The present invention removes the scene image through the specific operation determined above. The position of the blank area is obtained by the method of middle object, and then the delimitation is carried out, so that the delimitation accuracy is improved, and the projected user interface is very clear, which is an important invention point of the present invention.

In one embodiment, when the object is a user operation interface, the method further includes:

In the first update step S103, the computing board processes the scene image transmitted by the wide-angle camera at the first time interval and compares it with the scene image of the previously determined boundary. If the comparison result is inconsistent, the boundary is determined again. In this application, All 'if the comparison results are inconsistent' means that the distribution of objects in the scene after the two detections before and after the detection is quite different, which indicates that the position of the object has been greatly changed. When the object is a user operation interface, the first update step S103 includes:

The computing board obtains a frame of scene image from the wide-angle camera every second, and uses the mobilenet-ssd detection network to obtain the distribution state of all objects in the frame of scene image;

The computing board obtains the setting information of the current projection unit, and compares it with the distribution state of the object. If the error of the comparison result is greater than the first threshold, the boundary determination cannot be performed, and the projection unit is updated to the non-delimitable warning state. , and adjust the state of the computing board to a state that determines whether it can be delimited in real time; if the error of the comparison result is less than the first threshold, a new boundary range will be determined, and compared with the previously stored boundary range, if the comparison If the result is less than the second threshold, no update is performed; otherwise, the new boundary range is stored and transmitted to the projection unit, and the projection unit adjusts the projection area accordingly according to the new boundary range.

Through the above updating operation, the present invention enables the delimitation to be updated in real time, thereby realizing the projection user operation interface on the moving object, that is, the projection can track the movement of the object, which is convenient for the user's operation, that is, the tracking effect realized by the real-time refresh makes the The ability to delimit is further improved, so that the user can ensure the automatic tracking of the projection area when the device is not moving in a large range, which greatly improves the user's experience, which is another important invention point of the present invention.

In one embodiment, when the computing board makes a judgment according to the content projected by the current projection unit, and determines according to the judgment result that the displayed content is to be delimited, that is, when the object is the displayed content, the content is displayed on the user The content on the operation interface, the determining step S102 includes:

The wide-angle camera captures a scene image in real time and transmits it to the computing board at a second time interval, and the computing board transmits the scene image to a cloud server; the cloud server uses a deep learning learning network to predict the Chinese characters of the content position, and at the same time, the picture containing the text is cropped to obtain the first sub-picture and stored;

The cloud server uses the ctc algorithm to identify the text content of the first sub-picture, and generates a content data set from the text and the corresponding position after the recognition;

the server transmits the content data set to the computing pad, the computing pad transmits information of the location in the content data set to the projection unit;

The projection unit projects the content data set obtained by the computing board, and the user selects the content that needs to be delimited.

In one embodiment, the projection unit projects the content data set obtained by the computing board and the content that needs to be delimited selected by the user includes:

The projection unit monitors the computing board in real time, and displays the content in a light color in the projection area after receiving the recognized text and position sent by the computing board;

The user selects according to the displayed identified content, and after selection, the boundary at the position of the corresponding content will be made obvious, indicating that the current content has been selected. The manifesting is adding a frame to the display.

Through the delimitation of the above content, the displayed content can be further extracted, such as extracting specific text, retrieving the deep information of the picture, and the information can also be displayed directly by means of projection. For content delimitation, projection automatically gives recognition It also has a reminder of text and borders, plus the real-time tracking effect of projection, and allows users to obtain (assign) again, so that users have a new feeling when obtaining information, which belongs to another aspect of the present invention. an important invention.

In one embodiment, when the object is displayed content, the method further includes:

In the second update step S104, the computing board performs an in-depth identification of the content selected by the user, obtains the specific information of the content, and updates the content to the projection unit for projection display. In one embodiment, the second update step S104 includes:

When the user selects an identified area, the computing board obtains the user's selected area and records the location of the selected area;

The computing board cuts the region selected by the user into a second sub-picture based on the position of the selected region, and uses an intelligent recognition API to analyze the text or picture information in the second sub-picture;

The computing board combines the specific information and position information of the analyzed text or picture to obtain detailed information of the selected area, and extracts the effective part of the detailed information for normalization to obtain standardized data and transmits it to the projection unit. ; Normalization refers to extracting the effective part of the information, such as the interpretation of idioms.

After the projection unit receives the normative data from the computing board, the corresponding display is updated in the user operation area in the projection area.

The delimitation of the displayed content is a method implemented to better record the marked positions related to the text and pictures, and at the same time, the delimitation also helps to collect information for the next application. Through the above update operation, the present invention makes the displayed content The real-time update facilitates the user's operation of the displayed content and greatly improves the user's experience, which is another important invention point of the present invention.

In addition, in one embodiment, in order to ensure excellent projection effects on objects of different sizes and distances, the present invention adopts a method of adjusting the focal length of the projector based on the center point distance. The specific operations are as follows: the calculation board has been based on The user's selection determines the object to be delimited (for example, a book), and calculates the position of the boundary to be projected in the scene; based on the delimited boundary, the calculation board calculates the delimited area based on the four boundaries of the object The intersection of the diagonal lines, that is, the position of the center point, is stored again; then the computing board enables the depth camera to shoot the scene, and temporarily stores it after obtaining the complete RGB-D information of the scene; the computing board is based on the obtained RGB-D information. D information, after extracting the depth information from it, combine it with the position of the center point of the bounding area, and then obtain the distance between the center point and the camera, and then obtain the distance between the projector and the center of the bounding area after fine-tuning based on the positions of the camera and the projector; Then, the computing board calls the initialization method of the projector, and uses the distance as the original focal length. After the projector's own keystone correction processing during the initialization process, a clear projection display of the corresponding position can be realized based on the distance. Through this operation, when delimiting, the center of the projection unit and the delimiting area are automatically adjusted based on the size of the delimiting object, so that the projected interface is clearer, which is another important invention point of the present invention.

FIG. 2 shows an object boundary determination device for human-computer interaction according to the present invention, the device includes: a projection unit, a wide-angle camera, a computing board, and the like;

The wide-angle camera captures the scene image in real time, and sends each frame of the captured scene image to the computing board; after the computing board receives the each frame of the scene image, it determines whether the current scene can be Determine the bounds, and if so, determine the bounds of the object.

The device of the present invention can be on a smart desk lamp, the smart desk lamp has a projection unit, that is, a projector, a wide-angle camera, a depth camera, an infrared camera, etc., and has a computing board inside, and the computing board at least has a processor and a memory for To complete the processing of data, etc., of course, it must also have a power supply, a power supply controller, and the like. The projection unit may be a projector, and through the method of the present invention, the boundary of the projection unit to project an operation interface on the desktop can be determined. When the user operates on the operation interface, the boundary of the displayed content can also be determined. As for which object to delimit, the computing board judges according to the content projected by the current projection unit, and determines whether to delimit the user operation interface according to the judgment result. Or delimit the display content, and then perform the corresponding delimitation operation. For example, during initialization, the projected user operation interface is delimited. After the user operation interface is provided, when the user operates on the operation interface, the The display content on the operation interface is delimited, that is, the present invention supports the free switching between the delimitation of the user operation interface and the delimitation of the display content, and the smart desk lamp can interact with the server. The server is shown in FIG. 2, but the server Not part of the smart desk lamp, the server can be a cloud server, etc.

In one embodiment, when the computing board judges according to the content projected by the current projection unit, and determines according to the judgment result that the user operation interface is delimited, that is, when the object is the user operation interface, the computing board receives the After describing each frame of scene image, it is determined whether the current scene can determine the boundary based on each frame of scene image obtained, and if so, then determining the boundary range of the object includes:

The computing board receives each frame of scene image transmitted by the wide-angle camera in real time, and uses the mobilenet-ssd detection network to process the object distribution in the current scene to determine the shape of each object and the corresponding category of each object;

The calculation board calculates the position of each object in the space of the scene image based on the scene image and the determined object shape, and generates an object data set after merging the position of each object and the corresponding category;

The computing board reads the position information from the object data set, subtracts all the object distribution information from the scene image based on the position information to obtain blank area information, and then the computing board will determine whether the boundary can be performed according to user settings. Determine, if yes, calculate the position information of the delimitable area to determine the boundary range, and store the boundary range;

The computing board transmits the stored boundary range to the projection unit, and at the same time transmits a signal of successful delimitation, when the projection unit receives the signal of successful delimitation, obtains the stored setting information of the current user from the computing board ;

The computing board determines whether to perform projection in the blank area, if so, determines the projection area according to the boundary range and the setting information, and projects the user's operation interface in the projection area; if not, the user selects the projection area. The object to be projected, then the computing board reads the position information of the object in the object data set, and projects the user's operation interface based on the position information of the object.

There are two main ways of delimitation, one is the projection of the blank area (projection on the blank area), if the size of the area meets the projection range set by the user, it means that it can be delimited, which is a more traditional way; The first method is based on the projection of the recognized object, that is, projected on a specific book or paper. If there is an object that can be projected, such as a book, paper, etc., it means that the boundary can be delimited. The present invention removes the scene image through the specific operation determined above. The position of the blank area is obtained by the method of middle object, and then the delimitation is carried out, so that the delimitation accuracy is improved, and the projected user interface is very clear, which is an important invention point of the present invention.

In one embodiment, when the object is a user interface, the computing board processes the scene image transmitted by the wide-angle camera at a first time interval and compares it with the scene image with the determined boundary before, if the comparison result is inconsistent , the boundary is determined again.

In one embodiment, when the object is a user interface, the computing board processes the scene image transmitted by the wide-angle camera at a first time interval and compares it with the scene image with the determined boundary before, if the comparison result is inconsistent , the boundary determination again includes:

The computing board obtains a frame of scene image from the wide-angle camera every second, and uses the mobilenet-ssd detection network to obtain the distribution state of all objects in the frame of scene image;

The computing board obtains the setting information of the current projection unit, and compares it with the distribution state of the object. If the error of the comparison result is greater than the first threshold, the boundary determination cannot be performed, and the projection unit is updated to the non-delimitable warning state. , and adjust the state of the computing board to a state that determines whether it can be delimited in real time; if the error of the comparison result is less than the first threshold, a new boundary range will be determined, and compared with the previously stored boundary range, if the comparison If the result is less than the second threshold, no update is performed; otherwise, the new boundary range is stored and transmitted to the projection unit, and the projection unit adjusts the projection area accordingly according to the new boundary range.

Through the above updating operation, the present invention enables the delimitation to be updated in real time, thereby realizing the projection user operation interface on the moving object, that is, the projection can track the movement of the object, which is convenient for the user's operation, that is, the tracking effect realized by the real-time refresh makes the The ability to delimit is further improved, so that the user can ensure the automatic tracking of the projection area when the device is not moving in a large range, which greatly improves the user's experience, which is another important invention point of the present invention.

In one embodiment, when the computing board makes a judgment according to the content projected by the current projection unit, and determines according to the judgment result that the displayed content is to be delimited, that is, when the object is the displayed content, the content is displayed on the user The content on the operation interface, after receiving the each frame of scene image, the computing board judges whether the current scene can determine the boundary based on each frame of the scene image obtained, and if so, then determine the boundary range of the object includes:

The wide-angle camera captures a scene image in real time and transmits it to the computing board at a second time interval, and the computing board transmits the scene image to a cloud server; the cloud server uses a deep learning learning network to predict the Chinese characters of the content position, and at the same time, the picture containing the text is cropped to obtain the first sub-picture and stored;

The cloud server uses the ctc algorithm to identify the text content of the first sub-picture, and generates a content data set from the text and the corresponding position after the recognition;

the server transmits the content data set to the computing pad, the computing pad transmits information of the location in the content data set to the projection unit;

The projection unit projects the content data set obtained by the computing board, and the user selects the content that needs to be delimited. Preferably, the projection unit projects the content data set obtained by the computing board and the content that needs to be delimited selected by the user includes:

The projection unit monitors the computing board in real time, and displays the content in a light color in the projection area after receiving the recognized text and position sent by the computing board;

The user selects according to the displayed identified content, and after selection, the boundary at the position of the corresponding content will be made obvious, indicating that the current content has been selected. The manifesting is adding a frame to the display.

Through the delimitation of the above content, the displayed content can be further extracted, such as extracting specific text, retrieving the deep information of the picture, and the information can also be displayed directly by means of projection. For content delimitation, projection automatically gives recognition It also has a reminder of text and borders, plus the real-time tracking effect of projection, and allows users to obtain (assign) again, so that users have a new feeling when obtaining information, which belongs to another aspect of the present invention. an important invention.

In one embodiment, when the object is displayed content, the computing board performs in-depth identification of the content selected by the user, obtains specific information of the content, and updates it to the projection unit for projection display. The computing board performs in-depth identification of the content selected by the user, and after obtaining the specific information of the content, updating it to the projection unit and performing projection display includes:

When the user selects an identified area, the computing board obtains the user's selected area and records the location of the selected area;

The computing board cuts the region selected by the user into a second sub-picture based on the position of the selected region, and uses an intelligent recognition API to analyze the text or picture information in the second sub-picture;

The computing board combines the specific information and position information of the analyzed text or picture to obtain detailed information of the selected area, and extracts the effective part of the detailed information for normalization to obtain standardized data and transmits it to the projection unit. ; Normalization refers to extracting the effective part of the information, such as the interpretation of idioms.

After the projection unit receives the normative data from the computing board, the corresponding display is updated in the user operation area in the projection area.

The delimitation of the displayed content is a method implemented to better record the marked positions related to the text and pictures, and at the same time, the delimitation also helps to collect information for the next application. Through the above update operation, the present invention makes the displayed content The real-time update facilitates the user's operation of the displayed content and greatly improves the user's experience, which is another important invention point of the present invention.

In addition, in one embodiment, in order to ensure excellent projection effects on objects of different sizes and distances, the present invention adopts a method of adjusting the focal length of the projector based on the center point distance. The specific operations are as follows: the calculation board has been based on The user's selection determines the object to be delimited (for example, a book), and calculates the position of the boundary to be projected in the scene; based on the delimited boundary, the calculation board calculates the delimited area based on the four boundaries of the object The intersection of the diagonal lines, that is, the position of the center point, is stored again; then the computing board enables the depth camera to shoot the scene, and temporarily stores it after obtaining the complete RGB-D information of the scene; the computing board is based on the obtained RGB-D information. D information, after extracting the depth information from it, combine it with the position of the center point of the bounding area, and then obtain the distance between the center point and the camera, and then obtain the distance between the projector and the center of the bounding area after fine-tuning based on the positions of the camera and the projector; Then, the computing board calls the initialization method of the projector, and uses the distance as the original focal length. After the projector's own keystone correction processing during the initialization process, a clear projection display of the corresponding position can be realized based on the distance. Through this operation, when delimiting, the center of the projection unit and the delimiting area are automatically adjusted based on the size of the delimiting object, so that the projected interface is clearer, which is another important invention point of the present invention.

For the convenience of description, when describing the above device, the functions are divided into various units and described respectively. Of course, when implementing the present application, the functions of each unit may be implemented in one or more software and/or hardware.

From the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products can be stored in storage media, such as ROM/RAM, magnetic disks , CD, etc., including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute the methods described in various embodiments or some parts of the embodiments of the present application.

Finally, it should be noted that the above embodiments are only to illustrate rather than limit the technical solutions of the present invention. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: the present invention can still be modified or Equivalent replacements, and any modifications or partial replacements that do not depart from the spirit and scope of the present invention, shall all be included in the scope of the claims of the present invention.

Claims (10)

1. An object boundary determination method for human-computer interaction, the method comprising:

a scene information acquisition step, namely shooting a scene image in real time by using a wide-angle camera and sending each frame of shot scene image to a computing board;

and a determining step, wherein the computing board judges whether the boundary of the current scene can be determined or not based on each acquired scene image, and if so, the boundary range of the object is determined.

2. The method of claim 1, wherein the object is a user interface or displayed content.

3. The method according to claim 2, wherein when the object is a user operation interface, the determining step comprises:

the computing board receives each frame of scene image transmitted by the wide-angle camera in real time, and processes the object distribution of the current scene by using a mobilent-ssd detection network to determine the shape of each object and the corresponding category of each object;

the computing board calculates the position of each object in the space of the scene image based on the scene image and the determined object shape, and generates an object data set after combining the position of each object and the corresponding category;

the computing board reads position information from the object data set, all object distribution information is subtracted from the scene image based on the position information to obtain blank area information, then the computing board determines whether the boundary can be determined according to user setting, if so, the position information of the delimitable area is computed to determine the boundary range, and the boundary range is stored;

the computing board transmits the stored boundary range to the projection unit, transmits a successful delimitation signal at the same time, and acquires the stored setting information of the current user from the computing board after the projection unit receives the successful delimitation signal;

the computing board determines whether to project in the blank area, if so, determines a projection area according to the boundary range and the setting information, and projects an operation interface of a user in the projection area; if not, selecting an object to be projected by the user, reading the position information of the object in the object data set by the computing board, and projecting an operation interface of the user based on the position information of the object.

4. The method of claim 3, wherein when the object is a user operation interface,

the method further comprises the following steps:

and a first updating step, wherein the computing board processes the scene image transmitted by the wide-angle camera at a first time interval, then compares the processed scene image with the previous scene image with the determined boundary, and if the comparison result is inconsistent, the boundary is determined again.

5. The method according to claim 4, wherein when the object is a user operation interface, the updating step comprises:

the computing board acquires a frame of scene image from the wide-angle camera again every second, and acquires the distribution states of all objects in the frame of scene image by using the mobilene-ssd detection network;

the computing board acquires the setting information of the current projection unit, compares the setting information with the distribution state of the object, if the error of the comparison result is larger than a first threshold value, the boundary can not be determined, the projection unit is updated to a warning state which can not be delimited, and the state of the computing board is adjusted to a state which can judge whether the delimitation can be realized in real time; if the error of the comparison result is smaller than the first threshold value, determining a new boundary range, comparing the new boundary range with the previously stored boundary range, if the comparison result is smaller than a second threshold value, not updating, otherwise, storing the new boundary range and transmitting the new boundary range to the projection unit, and the projection unit correspondingly adjusts the projection area according to the new boundary range.

6. The method according to claim 3, wherein when the object is displayed content, the content is content displayed on a user operation interface, and the determining step includes:

the wide-angle camera shoots a scene image in real time and transmits the scene image to the computing board at a second time interval, and the computing board transmits the scene image to a cloud server; the cloud server predicts the positions of the characters in the content by using a deep learning network, and cuts the picture containing the characters to obtain a first sub-picture and stores the first sub-picture;

the cloud server identifies the text content of the first sub-picture by using a ctc algorithm, and generates a content data set by the text and the corresponding position after identification;

the server transmits the content data set to the computing board, and the computing board transmits information of a position in the content data set to the projection unit;

and the projection unit projects the content data set obtained by the computing board, and the user selects the content needing to be delimited.

7. An object boundary determination device for human-computer interaction, the device comprising: the system comprises a projection unit, a wide-angle camera and a computing board;

the wide-angle camera shoots scene images in real time and sends each frame of shot scene images to the computing board; and after receiving each frame of scene image, the computing board judges whether the boundary of the current scene can be determined based on each acquired frame of scene image, and if so, determines the boundary range of the object.

8. The apparatus according to claim 7, wherein the object is a user operation interface or displayed content, and when the object is a user operation interface, the computing board, after receiving the each frame of scene image, determines whether a boundary of the current scene can be determined based on the acquired each frame of scene image, and if so, determining a boundary range of the object includes:

the computing board receives each frame of scene image transmitted by the wide-angle camera in real time, and processes the object distribution of the current scene by using a mobilent-ssd detection network to determine the shape of each object and the corresponding category of the object;

the computing board calculates the position of each object in the space of the scene image based on the scene image and the determined object shape, and generates an object data set after combining the position of each object and the corresponding category;

the computing board reads position information from the object data set, all object distribution information is subtracted from the scene image based on the position information to obtain blank area information, then the computing board determines whether the boundary can be determined according to user setting, if so, the position information of the delimitable area is computed to determine the boundary range, and the boundary range is stored;

the computing board transmits the stored boundary range to the projection unit, transmits a successful delimitation signal at the same time, and acquires the stored setting information of the current user from the computing board after the projection unit receives the successful delimitation signal;

the computing board determines whether to project in the blank area, if so, determines a projection area according to the boundary range and the setting information, and projects an operation interface of a user in the projection area; if not, selecting an object to be projected by the user, reading the position information of the object in the object data set by the computing board, and projecting an operation interface of the user based on the position information of the object.

9. The device of claim 8, wherein when the object is a user-operated interface, the computing board processes the scene image transmitted by the wide-angle camera at a first time interval and then compares the processed scene image with the previously determined boundary scene image, and if the comparison result is inconsistent, the boundary determination is performed again.

10. The apparatus of claim 9, wherein when the object is a user-operated interface, the computing board processes the scene image transmitted by the wide-angle camera at a first time interval and compares the processed scene image with a previously determined boundary scene image, and if the comparison result is inconsistent, re-performing the boundary determination comprises:

the computing board acquires a frame of scene image from the wide-angle camera again every second, and acquires the distribution states of all objects in the frame of scene image by using the mobilene-ssd detection network;

the computing board acquires the setting information of the current projection unit, compares the setting information with the distribution state of the object, if the error of the comparison result is larger than a first threshold value, the boundary can not be determined, the projection unit is updated to a warning state which can not be delimited, and the state of the computing board is adjusted to a state which can judge whether the delimitation can be realized in real time; if the error of the comparison result is smaller than the first threshold value, determining a new boundary range, comparing the new boundary range with the previously stored boundary range, if the comparison result is smaller than a second threshold value, not updating, otherwise, storing the new boundary range and transmitting the new boundary range to the projection unit, and the projection unit correspondingly adjusts the projection area according to the new boundary range.

Images