TWI885711B - Electronic apparatus and windows arrangement method thereof - Google Patents

Abstract

An electronic apparatus and a windows arrangement method thereof are disclosed. The method is adapted to the electronic apparatus and includes the following steps. A window of an application is displayed via a display. Window Information of the window is obtained. The window is classified into one of window layout categories based on the window Information of the window and application information of the application. A window size of the window is adjusted according to a size criterion of one of the window layout categories.

Description

Electronic device and window layout method thereof

The present invention relates to a management technology of an electronic device, and in particular to an electronic device and a window layout method thereof.

In recent years, windowed operating systems have become quite common. By displaying application windows on the display screen, users can operate the application to complete the corresponding tasks. On the other hand, with the evolution of technology, large-size display screens are increasingly widely used. Although large-size display screens are indeed very convenient in many usage scenarios (such as audio-visual entertainment, multitasking, or computer visual drawing, etc.), there are times when displaying large-size windows on a large-size display screen does not meet user needs. For example, the window screen of some applications or the web page content displayed in the window is not complicated. If it is displayed through a large-size window, it will cause the application content or controls to be too scattered. This will further lead to problems such as the mouse cursor moving too long, thus affecting work efficiency or causing shoulder and neck pain for the user. Whenever a user opens a new window, the user can only manually adjust the size and position of the window, which is quite inconvenient.

The present invention provides an electronic device and a window layout method thereof, which can improve the above-mentioned problem.

The embodiment of the present invention provides a window layout method applicable to an electronic device including a display. The method comprises the following steps. Displaying a window of an application program through a display. Obtaining window information of the window. Classifying the window into one of a plurality of window layout categories according to the window information of the window and application information of the application. Adjusting the window size of the window according to a window size criterion of one of the plurality of window layout categories.

The embodiment of the present invention further provides an electronic device, which includes a display, a storage device and a processor. The processor is coupled to the display and the storage device and is configured to perform the following operations. Display the window of the application through the display. Obtain window information of the window. Classify the window into one of a plurality of window layout categories based on the window information of the window and the application information of the application. Adjust the window size of the window according to the window size criteria of one of the plurality of window layout categories.

Based on the above, in the embodiment of the present invention, the window can be classified into one of a plurality of window layout categories according to the application information and the window information, so as to automatically adjust the window size according to the classification result. In this way, the user can easily enjoy the benefits of a large-size display screen without having to manually adjust the window size, and the operation convenience and work efficiency are improved.

FIG1 is a block diagram of an electronic device according to an embodiment of the present invention. Referring to FIG1 , the electronic device 100 may be a laptop computer, a desktop computer, a smart phone, a tablet computer, an industrial computer, or a server. The electronic device 100 may include a display 110, a line of sight detection device 120, a storage device 130, and a processor 140.

The display 110 is used to display images. For example, the display 110 may include various types of displays such as a liquid crystal display (LCD), a thin film transistor LCD (TFT LCD), a field sequential LCD (FS LCD), a light emitting diode (LED), an active-matrix organic light-emitting diode (AMOLED) display, or an electrowetting display.

The sight detection device 120 is used to detect the sight information of the user of the electronic device 100. For example, the sight detection device 120 may include a camera that can shoot an image toward the user. The sight information of the user can be estimated based on the user image. Alternatively, the sight detection device 120 may include an eye tracking device that can track the eye position and eye movement through an infrared sensor or a dedicated camera. The sight information of the user can be generated based on the eye position and eye movement.

The storage device 130 is used to store data such as images and program codes, and may include a volatile memory module and a non-volatile memory module. The storage device 130 may be any type of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk or other similar devices, integrated circuits and combinations thereof.

The processor 140 is coupled to the display 110, the line of sight detection device 120, and the storage device 130. The processor 140 can be used to control the operation of the electronic device 100 in whole or in part. For example, the processor 140 can include any programmable general-purpose or special-purpose microprocessor, digital signal processor (DSP), programmable controller, application specific integrated circuits (ASIC), programmable logic device (PLD) or other similar devices or combinations of these devices. In some embodiments, the processor 140 can execute instructions, programs or software modules in the storage device 130 to implement various operations performed by the processor 140 herein.

It should be noted that, in one embodiment, the electronic device 100 may also include a power supply device and various input/output devices. The power supply device may include a battery and/or a power management circuit. The output/output device may include a network connection interface, a touch panel, a mouse, a keyboard, a speaker and/or a microphone, etc., and the present invention is not limited thereto.

In some embodiments, the processor 140 may instruct the display 110 to display at least one window. The present invention does not limit the number of windows displayed by the display 110. It should be particularly noted that, compared to the traditional method where the user needs to manually adjust the window size of the window through an input device, in the embodiment of the present invention, the processor 140 can automatically and adaptively adjust the window size according to the window information and application information of the window, such as enlarging or reducing the window. In this way, the user experience can be effectively improved and the convenience of operation can be enhanced.

For example, FIG. 2 is a schematic diagram of automatically adjusting the window size according to an embodiment of the present invention. Referring to FIG. 2 , after starting an application, the display 110 may display a window 210 of the application. For example, the window 210 may present a user interface of the application. The processor 140 may adjust the window size of the window 210 in real time according to the window information and application information of the window 210. In some embodiments, the window size of the window 210 may change in response to changes in the window information of the window 210, and the above-mentioned window information may include a window title, a window screen, a layout of components of the window, web page content or other information related to the window 210.

For example, in response to the change of the window image of the window 210, the processor 140 may enlarge the window size of the window 210 from a first preset size (e.g., window width W1) to a second preset size (e.g., window width W2). Then, in response to the change of the window image of the window 210, the processor 140 may reduce the window size of the window 210 from the second preset size (e.g., window width W2) to a third preset size (e.g., window width W3). The window size of the window 210 may be automatically adjusted based on the window image without user intervention.

FIG3 is a flow chart of a window layout method according to an embodiment of the present invention. Please refer to FIG1 and FIG3 simultaneously. The following is a detailed description of the window layout method of this embodiment in conjunction with the components and devices of the electronic device 100 in FIG1.

In step S310, the processor 140 displays the application window through the display 110. In step S320, the processor 140 obtains the window information of the window. Specifically, the processor 140 may first obtain the basic information of the current active window (Active Window). For example, through the Windows application programming interface (API), the processor 140 may obtain the window title and application name of the current active window. The above-mentioned Windows application programming interface is, for example, GetForegroundWindow(), GetWindowTextLength(), GetWindowText(), GetWindowThreadProcessId(), or GetModuleBaseName(), etc., and the present invention is not limited thereto.

In step S330, the processor 140 classifies the window into one of a plurality of window layout categories according to the window information of the window and the application information of the application. Further, the application information of the application may include the application name, the application type or other information related to the application. In addition, the window information may also include the window screen, the component layout, the webpage content, the window content, or the derived window information generated by analyzing the above information, etc.

In some embodiments, the step of classifying the window into one of a plurality of window layout categories according to the window information of the window and the application information of the application includes at least one of a first operation and a second operation. The first operation includes: in response to the application being not a browser program, dividing the window into a plurality of sub-window blocks; capturing a plurality of component types respectively corresponding to the plurality of sub-window blocks through an application program interface component; and determining whether the window is classified into the first window layout category or the second window layout category according to the plurality of component types of the plurality of sub-window blocks. The second operation includes: in response to the application being a browser program, capturing at least one image block and at least one text block in the window; and determining whether the window is classified into a first window layout category or a second window layout category according to the quantity of the at least one image block, the block range of the at least one image block, and the block range of the at least one text block.

In different embodiments, the processor 140 may classify a window into one of a plurality of window layout categories according to different classification criteria. The following embodiments will be described in more detail. It should be particularly noted that different window layout categories correspond to different window size criteria. In other words, by classifying a window into one of a plurality of window layout categories, the processor 140 may obtain the window size criteria applicable to the window.

Therefore, in step S340, the processor 140 adjusts the window size of the window according to the window size criteria of one of the plurality of window layout categories. The window size criteria of each window layout category may include a default size or a default scaling ratio, etc. For example, when the window is classified into a window layout category corresponding to a certain window size criteria, the processor 140 may enlarge the window to the maximum according to the window size criteria. Alternatively, when the window is classified into a window layout category corresponding to a certain window size criteria, the processor 140 may adjust the window size to be the same as the target default size. The above target default size may be determined according to the screen size of the display 110. For example, the target width in the target default size may be the screen width multiplied by M1%, where M1 is a preset value between 1 and 100. The target height in the target default size may be the screen height multiplied by M2%, where M2 is a preset value between 1 and 100.

In some embodiments, the step of adjusting the window size of the window according to the window size criteria of one of the plurality of window layout categories includes at least one of a first operation, a second operation, and a third operation. The first operation includes: detecting a user's line of sight; determining whether the line of sight is projected on the window; and in response to the line of sight being projected on the window, adjusting the window size of the window according to the window size criteria. The second operation includes: in response to one of the plurality of window layout categories being the first window layout category, adjusting the window size of the window to a first default size. The third operation includes: in response to one of the plurality of window layout categories being the second window layout category, adjusting the window size of the window to a second default size, and moving the window to a default display position.

Further, in some embodiments, the processor 140 may determine whether to adjust the window size of the window according to the user's line of sight information. Please refer to FIG. 4, which is a flow chart of adjusting the window size based on line of sight according to an embodiment of the present invention. Step S340 shown in FIG. 3 may be implemented as steps S341 to S343.

In step S341, the processor 140 detects a user's line of sight through the line of sight detection device 120. In step S342, the processor 140 determines whether the line of sight is projected on the window. Specifically, the processor 140 can determine whether the user's line of sight on the display 110 is located within the window range of the window.

If the judgment in step S342 is yes, in step S343, in response to the line of sight being projected on the window, the processor 140 adjusts the window size of the window according to the window size criteria. That is, when the user focuses on a certain window, the processor 140 will adjust the window size of the window in real time. On the contrary, if the judgment in step S342 is no, the processor 140 may disable the adjustment of the window size of the window according to the window size criteria. That is, when the user does not focus on a certain window, the processor 140 will not adjust the window size of the window. It is worth mentioning that when no line of sight is detected, the processor 140 may disable the adjustment of the window size of the window according to the window size criteria. That is, when no user is looking at the display 110, the processor 140 will not adjust the window size of the window.

In some embodiments, the processor 140 can capture the user's facial image through the line of sight detection device 120 and calculate the head posture angle based on the facial image. Afterwards, the processor 140 can determine the line of sight projected on the display 110 according to the head posture angle.

In more detail, the processor 140 may perform a face recognition operation on the image captured by the line of sight detection device 120 to obtain a face block on the head image. The processor 140 may detect a plurality of facial landmarks in the face block for marking the facial contour, the shape of the facial features, and the position of the facial features. Afterwards, the processor 140 estimates the user's head posture angle based on the facial landmarks. For example, by adopting a perspective-n-point (PnP) algorithm, the processor 140 may estimate the user's head posture angle in the world coordinate system based on the coordinate positions of these facial landmarks. The head posture angle may include a head pitch angle relative to the first coordinate axis and a head yaw angle relative to the second coordinate axis. Based on the head yaw angle boundary associated with the display 110 and the current head posture angle, the processor 140 may determine the point of view projected on the display 110. The head yaw angle boundary includes a maximum yaw angle corresponding to the right screen boundary, a minimum yaw angle corresponding to the left screen boundary, a maximum pitch angle corresponding to the upper screen boundary, and a minimum pitch angle corresponding to the lower screen boundary.

For example, FIG5A and FIG5B are schematic diagrams of detecting sight lines according to an embodiment of the present invention. Referring to FIG5A and FIG5B , the processor 140 may perform interpolation calculations based on the maximum deflection angle θymax, the minimum deflection angle θymin, the X-axis coordinate in the screen coordinate system corresponding to the maximum deflection angle θymax (i.e., Sw), the X-axis coordinate in the screen coordinate system corresponding to the minimum deflection angle θymin (i.e., 0), and the user's current head deflection angle θy to obtain the X-coordinate value of the sight line landing point F2 (i.e., Sx). Similarly, the processor 140 can perform interpolation calculations based on the maximum tilt angle θpmax, the minimum tilt angle θpmin, the Y-axis coordinate in the screen coordinate system corresponding to the maximum tilt angle θpmax (i.e., 0), the X-axis coordinate in the screen coordinate system corresponding to the minimum tilt angle θpmin (i.e., Sh), and the user's current head tilt angle θp to obtain the Y-coordinate value of the gaze position F2 (i.e., Sy). In this way, the processor 140 can predict that the user's gaze position is (Sx, Sy) in the screen coordinate system. Here, the screen coordinate system can be a coordinate system defined in pixels and based on the screen resolution. In one embodiment, the processor 140 can calculate the coordinates of the sight point according to the following formula (1) and formula (2). Formula (1) Formula (2)

When the processor 140 determines that the coordinates of the view point under the screen coordinate system are within the window range of the window, the processor 140 can determine that the view is projected on the window. Conversely, when the processor 140 determines that the coordinates of the view point under the screen coordinate system are not within the window range of the window, the processor 140 can determine that the view is not projected on the window.

FIG6 is a flow chart of a window layout method according to an embodiment of the present invention. Please refer to FIG1 and FIG6 simultaneously. The following is a detailed description of the window layout method of this embodiment in conjunction with the components and devices of the electronic device 100 in FIG1.

In step S610, the processor 140 displays the application window through the display 110. In step S620, the processor 140 obtains the window information of the window. The implementation details of step S610 to step S620 can be referred to the above description and will not be elaborated here.

It should be noted that in step S630, the processor 140 determines whether the window meets the common condition according to the window's active window time, the number of interactive events, and the eye-retention time. In other words, the processor 140 can determine whether a window is a common window.

In some embodiments, the active window time is the total time that a window is an active window in a unit time period. The number of interactive events is the number of user interactive events received by a window in a unit time period. User interactive events include mouse events, keyboard events, touch events or a combination thereof. The gaze dwell time is the total time that the user's gaze stays on a window in a unit time period.

In some embodiments, the processor 140 may calculate the window usage score of the window according to the window's active window time, the number of interactive events, and the eye retention time. For example, the processor 140 may calculate the window usage score of the window according to the following formula (3). ΔOx = (Wa*Ax + We*Ex + Ws*Mx) / (Wa + We + Ws)   Formula (3) Wherein, ΔOx represents the window usage score; Ax represents the active window time; Ex represents the number of interactive events; Mx represents the eye retention time; Wa, We, and Ws are weight factors, respectively.

In some embodiments, the processor 140 may sort the window and at least one other window according to the window common score of the window. The processor 140 may determine whether the window meets the common condition according to the sorting order of the window. Specifically, the processor 140 may calculate the window common score for each of the multiple windows displayed by the display 110, and filter out the common windows according to the window common scores of all the windows. For example, when the sorting order of a certain window is the first n, the processor 140 may determine that the window meets the common condition. Otherwise, when the sorting order of a certain window is not the first n, the processor 140 may determine that the window does not meet the common condition. Alternatively, in some embodiments, the processor 140 may determine whether the window common score is greater than a threshold value to determine whether the window meets the common condition. When the window common score of a certain window is greater than the threshold value, the processor 140 may determine that the window meets the common condition. Otherwise, when the window common score of a certain window is not greater than the threshold value, the processor 140 may determine that the window does not meet the common condition.

If the determination in step S630 is yes, in step S640, the processor 140 classifies the window into one of a plurality of window layout categories according to the window information and the application information, reflecting that the window meets the common conditions. In other words, the processor 140 can automatically adjust the window size for the window with a high frequency of interaction with the user, and does not automatically adjust the window size for the window with a low frequency of interaction with the user.

In step S650, the processor 140 adjusts the window size of the window according to the window size criteria of one of the plurality of window layout categories. In this embodiment, step S650 can be implemented as steps S651 and S652. In step S651, in response to one of the plurality of window layout categories being a first window layout category, the processor 140 adjusts the window size of the window to a first default size. In step S652, in response to one of the plurality of window layout categories being a second window layout category, the processor 140 adjusts the window size of the window to a second default size and moves the window to a default display position.

In some embodiments, the first window layout category is a large window size category, and the second window layout category is a small window size category. The first default size may be a maximized size, and the second default size may be a non-maximized size. The default display position is, for example, the center position of the screen of the display 110. That is, in response to the window being classified into different window layout categories, the processor 140 will adjust the window size to different default sizes. In addition, the processor 140 may move the window position of the window according to the window layout category to which the window belongs. Taking FIG. 2 as an example, when the window 210 is classified into the first window layout category, the window 210 will be maximized to have a window width W2 (i.e., the maximized size). When the window 210 is classified as the second window layout category, the processor 140 may adjust the window 210 to have a window width W3 (ie, a non-maximized size), and display the window 210 at the center of the screen of the display 110 .

FIG7 is a flow chart of classifying windows into one of a plurality of window layout categories according to an embodiment of the present invention. Please refer to FIG1 and FIG7 simultaneously. The following is a detailed description of the steps of this embodiment in conjunction with the components and devices of the electronic device 100 in FIG1.

In step S702, the processor 140 determines whether the window information of the window and the application information of the application match the database data recorded in a database. If the determination in step S702 is yes, in step S704, the processor 140 determines whether the window is classified into the first window layout category or the second window layout category according to the database data, in response to the window information and the application information matching the database data.

In different embodiments, the database may be a local database or a cloud database. The database data includes a plurality of information combinations of different window information and different application information, and the window layout categories corresponding to the above information combinations. For example, the database data may be a classification setting list shown in Table 1. Window title Application Name Window layout class Title A Program Name A First window layout class Title B Program Name B Second window layout class Title C Program Name B Second window layout class Table 1 Based on the example of Table 1, when the window title of the window matches "Title A" and the application name of the window matches "Program Name A", the processor 140 determines to classify the window into the first window layout category according to the database data. When the window title of the window matches "Title B" and the application name of the window matches "Program Name B", the processor 140 determines to classify the window into the first window layout category according to the database data. When the window title of the window matches "Title D" and the application name of the window matches "Program Name D", the processor 140 determines that the window information and the application information do not match the database data.

In some embodiments, the application is a browser application, and the application information includes a web link address (URL). That is, when the application of the window is a browser application, the processor 140 can determine whether the web link address matches the database data to classify the window into one of a plurality of window layout categories. That is, the processor 140 can determine the window size adjustment method according to the web link address.

If the determination in step S702 is no, it means that the processor 140 cannot classify the window into one of the plurality of window layout categories according to the database data. Therefore, in step S706, the processor 140 determines whether the application is a browser program. In this embodiment, when the application is a browser program, the processor 140 can classify the window according to the amount of images and text in the window. When the application is not a browser program, the processor 140 can analyze the panel layout of the window to classify the window.

If the determination in step S706 is yes, it means that the application of the window is a browser program, and the window screen of the window is a web page. Therefore, in step S708, in response to the application being a browser program, the processor 140 captures at least one image block and at least one text block in the window. For example, the processor 140 can capture the text block and the image block in the web page through the JavaScript API supported by the browser program.

For example, FIG8 is a schematic diagram of multiple image blocks and multiple text blocks in a window according to an embodiment of the present invention. Referring to FIG8 , the processor 140 can extract multiple image blocks (e.g., image blocks IB1, IB2) and multiple text blocks (e.g., text blocks TB1, TB2, TB3) from the window 810. The processor 140 can obtain the positions and block ranges of these image blocks. The processor 140 can obtain the positions and block ranges of these text blocks.

Therefore, in step S710, the processor 140 determines whether the window is classified into the first window layout category or the second window layout category according to the number of at least one image block, the block range of at least one image block, and the block range of at least one text block. In some embodiments, the processor 140 may determine whether the number of image blocks is greater than a quantity threshold (e.g., 10). When the number of image blocks is greater than the quantity threshold, the processor 140 may determine that the window is classified into the first window layout category. In addition, the processor 140 may determine whether the ratio of the total block range of the image blocks to the overall range of the webpage is greater than a ratio threshold (e.g., 50%). When the ratio of the total block range of the image blocks to the overall range of the webpage is greater than the ratio threshold, the processor 140 may determine that the window is classified as the first window layout category. In addition, the processor 140 may determine whether the first ratio of the total block range of the image blocks to the overall range of the webpage is greater than the second ratio of the total block range of the text blocks to the overall range of the webpage. When the first ratio is greater than the second ratio, the processor 140 may determine that the window is classified as the first window layout category. When the number of image blocks, the block range of the image blocks, and the block range of the text blocks do not meet any of the above conditions, the processor 140 may determine that the window is classified as the second window layout category.

If step S706 is judged as no, it means that the application of the window is not a browser program. Therefore, in step S712, reflecting that the application is not a browser program, the processor 140 divides the window into a plurality of sub-window blocks. The present invention does not limit the number of sub-window blocks, which can be designed according to actual applications. For example, FIG. 9 is a schematic diagram of a plurality of sub-window blocks drawn according to an embodiment of the present invention. Referring to FIG. 9, based on the preset division ratios A% and B%, the processor 140 can divide the window 910 into 9 sub-window blocks Z1 to Z9.

In step S714, the processor 140 captures multiple component types corresponding to the multiple sub-window blocks through the application program interface component. For example, the processor 140 can use the API provided by the operating system to obtain the multiple component types corresponding to the multiple sub-window blocks. The above API can be a Windows UI Automation API, such as AutomationControlType Enum(), but is not limited thereto.

In some embodiments, the processor 140 may input a reference position in each sub-window block to an API provided by the operating system to obtain a component type (ControlType) and a component range of a component in each sub-window block. The component type may be, for example, "Edit", "Document", "Pane", "List", "ListItem", "Tree", "TreeItem", "TreeView", "Table", "Group", "ToolBar", or "MenuBar", etc. Taking FIG. 9 as an example, the processor 140 may obtain a component type and a component range of a component in each sub-window block Z1 to Z9.

In step S716, the processor 140 detects whether the multiple component types meet the specific conditions. Specifically, the processor 140 determines whether these component types are sufficient to analyze the panel layout of the window. In some embodiments, when these component types are all the same or are all unknown, the processor 140 may determine that the multiple component types do not meet the specific conditions. For example, when the component types of multiple sub-window blocks are all the same and the sum of the ranges of the corresponding component ranges is greater than the threshold value, the processor 140 will not be able to know the panel layout of the window based on the component types of the multiple sub-window blocks. The above situation is common in game applications.

In some embodiments, according to the component types of the plurality of sub-window blocks, the processor 140 may determine the panel types of the plurality of operation panels composed of at least one sub-window block. Taking FIG. 9 as an example, the adjacent sub-window blocks Z1 to Z3 may form an upper operation panel. The adjacent sub-window blocks Z7 to Z9 may form a lower operation panel. The adjacent sub-window blocks Z1, Z4, and Z7 may form a left operation panel. The adjacent sub-window blocks Z3, Z6, and Z9 may form a right operation panel. The sub-window block Z5 may form a central operation panel.

Taking FIG. 9 as an example, according to the component types of sub-window blocks Z1 to Z3, the processor 140 can determine the panel type of the upper operation panel. According to the component types of sub-window blocks Z7 to Z9, the processor 140 can determine the panel type of the lower operation panel. According to the component types of sub-window blocks Z1, Z4, and Z7, the processor 140 can determine the panel type of the left operation panel. According to the component types of sub-window blocks Z3, Z6, and Z9, the processor 140 can determine the panel type of the right operation panel. According to the component type of sub-window block Z5, the processor 140 can determine the panel type of the middle operation panel. For example, the processor 140 can determine the panel type of each operation panel according to the following Table 2. Panel Type Component Type Text Editor Panel Edit, Document, ... Main control panel Pane (only for central operation panel) List Panel List, ListItem, Tree, TreeItem, TreeView, Table, ... Menu Panel Group, Pane, ToolBar, MenuBar, Tab, TabItem... Table 2

Taking the upper operation panel of FIG. 9 as an example, assuming that the component types of the sub-window blocks Z1 to Z3 of the upper operation panel are all "Group", the processor 140 can determine the panel type of the upper operation panel as "Menu Panel" based on Table 2. In some embodiments, when the component types of multiple sub-window blocks of an operation panel are different, the processor 140 can determine the panel type of the operation panel according to the component type of one of the multiple sub-window blocks of the operation panel. Taking the upper operation panel of FIG. 9 as an example, assuming that the component type of the sub-window block Z2 of the upper operation panel is "Group", the processor 140 can determine the panel type of the upper operation panel as "Menu Panel" based on Table 2. In some embodiments, when the component types of a plurality of sub-window blocks of an operation panel are mostly the same, the processor 140 may determine the panel type of the operation panel according to the same component types of the majority of sub-window blocks. Taking the upper operation panel of FIG. 9 as an example, assuming that the component types of the sub-window blocks Z1 and Z2 of the upper operation panel are "Group" but the component type of the sub-window block Z3 is "Document", the processor 140 may determine that the panel type of the upper operation panel is "Menu Panel".

In some embodiments, the processor 140 may search a lookup table according to the panel types of these operation panels to determine the classification result of the window. When the panel types of these operation panels do not conform to the panel layout recorded in the above-mentioned lookup table, the processor 140 may determine that these component types do not conform to the specific conditions. For example, Table 3 is an example of the above-mentioned lookup table. If the processor 140 determines that the panel types of multiple operation panels of a certain window do not conform to the panel layout recorded in Table 3, the processor 140 may determine that these component types do not conform to the specific conditions. Panel Layout Window layout class Central operating panel Upper operation panel Lower operation panel Left operation panel Right operation panel Multi-control panel First window layout class Main Control Panel/Text Editor Panel Menu Panel List Panel List Panel List Panel Multimedia player First window layout class Main Dashboard/List Panel Unlimited Menu Group Related List Panel Unlimited Video Conferencing First window layout class Menu Panel Unable to detect Unable to detect Unable to detect Unable to detect Office document software First window layout class Main Control Panel/Text Editor Panel Menu Group Related Unlimited Unlimited Unlimited Computer folder Second window layout class List Panel Menu Panel Unlimited List Panel Unlimited Communication software Second window layout class Unlimited Unlimited Text Editor Panel List Panel Unlimited Widget Second window layout class No panel No panel No panel No panel No panel Table 3

If the judgment in step S716 is yes, in step S718, the processor 140 determines whether the window is classified into the first window layout category or the second window layout category according to the multiple component types of the multiple sub-window blocks. As shown in Table 3 above, according to the panel types of the multiple operation panels, the processor 140 can determine whether the window is classified into the first window layout category or the second window layout category by looking up a table. For example, when the processor 140 determines that the panel types of the multiple operation panels meet the panel layout "Office document software" in Table 3, the processor 140 can classify the window into the first window layout category.

If the determination in step S716 is no, in step S720, the processor 140 inputs the window image of the window into a machine learning model to obtain a model classification result. In step S722, the processor 140 determines whether the window is classified into the first window layout category or the second window layout category according to the model classification result.

FIG10 is a schematic diagram of determining a window layout category using a machine learning model according to an embodiment of the present invention. Referring to FIG10 , the processor 140 may capture a window screen WMG1 of a window and input the window screen WMG1 into a trained machine learning model M1. The machine learning model M1 is a trained classification model, such as a convolutional neural network model. The processor 140 may obtain a model classification result R1 based on the output of the machine learning model M1. For example, the machine learning model M1 may classify the window screen WMG1 into one of a plurality of preset model categories, such as one of a plurality of game categories or one of a plurality of web page modes. The processor 140 can use the model classification result R1 to look up the lookup table LT1 to determine the window layout category WC1 of the window. The window layout category WC1 can be the first window layout category or the second window layout category. Therefore, the processor 140 can adjust the windows of multiple game applications of different game categories to different window sizes.

In some embodiments, the processor 140 may capture multiple target window images of the window after the application has been running for a preset time. The above-mentioned preset time can be set according to the actual application, and the present invention is not limited to this. By waiting for the preset time before capturing the target window image, it is possible to avoid the situation where the animation when the application is opened misclassifies the window. The processor 140 can capture multiple target window images of the window in a sampling cycle, for example, capturing a target window image every p seconds. The processor 140 inputs the multiple target window images into the machine learning model respectively, and obtains multiple classification confidence values of multiple preset model categories for each target window image. The processor 140 determines the model classification result according to the statistical results of multiple classification confidence values of multiple default model categories of each target window image.

For example, FIG11 is a schematic diagram of determining a model classification result based on multiple target window images according to an embodiment of the present invention. Referring to FIG11 , the target window image WMG2 and the target window image WMG3 are respectively input into the machine learning model M1 for classification processing. The machine learning model M1 can output the image classification result of the target window image WMG2 and the image classification result of the target window image WMG3. The image classification result of the target window image WMG2 includes multiple classification confidence values C1 for classifying the target window image WMG2 into multiple preset model categories. The image classification result of the target window image WMG3 includes multiple classification confidence values C2 for classifying the target window image WMG3 into multiple preset model categories. These classification confidence values can be probability values. That is, for each default model category, the machine learning model M1 will generate a corresponding classification confidence value based on the input image. Afterwards, the processor 140 can perform weighted statistical processing on the multiple classification confidence values of the multiple target window images to obtain multiple fused classification confidence values C3 for the multiple default model categories. For example, as shown in FIG11 , the processor 140 can perform weighted operations on the "classification confidence value 1-1" and the "classification confidence value 2-1" to generate the "fused classification confidence value 1" for "category 1". By comparing multiple fused classification confidence values, the processor 140 can determine that the model classification result R1 is the default model category corresponding to the highest fused classification confidence value.

For example, the processor 140 may calculate multiple fusion classification confidence values of multiple default model categories according to the following formula (4). Formula (4) Wherein, Cs is the fusion classification confidence value of a certain default model category; Cns is the classification confidence value of the default model category of the nth target window screen; N is the number of target window screens; the target window screens after the Rth target window screen can be given a standard weight of "1"; the target window screens before the Rth target window screen can be given a weight of "n/R". In other words, a lower weight "n/R" is given to the 1st target window screen to the R-1th target window screen, and a standard weight of "1" is given to the target window screens from the Rth target window screen to the Nth target window screen. For example, the processor 140 may capture 10 target window images (N=10), and the processor 140 may set the weights of the 1st to 5th target window images to 1/6, 2/6, ..., 5/6 respectively, and the weights of the 6th to 10th target window images are all 1 (R=6).

In summary, in the embodiment of the present invention, the window can be classified into one of multiple window layout categories according to the application information and the window information, and the window size can be automatically adjusted according to the classification result. In this way, the user can easily enjoy the benefits of a large-size display screen without having to manually adjust the window size, and the operation convenience and work efficiency are improved. In addition, the embodiment of the present invention can apply different classification rules to different applications, so that the window size adjustment result can better meet the user's needs.

Although the present invention has been disclosed as above by the embodiments, they are not intended to limit the present invention. Any person with ordinary knowledge in the relevant technical field can make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be defined by the scope of the attached patent application.

100: electronic device 110: display 120: line of sight detection device 130: storage device 140: processor 210, 810, 910: window W1, W2, W3: window width θymax: maximum deflection angle θymin: minimum deflection angle θpmax: maximum tilt angle θpmin: minimum tilt angle θy: head deflection angle θp: head tilt angle F2: line of sight point TB1, TB2, TB3: text block IB1, IB2: image block Z1~Z9: sub-window block WMG1: window screen WMG2, WMG3: target window screen M1: Machine learning model R1: Model classification result LT1: Lookup table WC1: Window layout category C1, C2: Classification confidence value C3: Fusion classification confidence value S310~S340, S341~S343, S610~S652, S702~S722: Steps

FIG. 1 is a block diagram of an electronic device according to an embodiment of the present invention. FIG. 2 is a schematic diagram of automatically adjusting the window size according to an embodiment of the present invention. FIG. 3 is a flow chart of a window layout method according to an embodiment of the present invention. FIG. 4 is a flow chart of adjusting the window size based on the line of sight according to an embodiment of the present invention. FIG. 5A and FIG. 5B are schematic diagrams of detecting the line of sight according to an embodiment of the present invention. FIG. 6 is a flow chart of a window layout method according to an embodiment of the present invention. FIG. 7 is a flow chart of classifying a window into one of a plurality of window layout categories according to an embodiment of the present invention. FIG8 is a schematic diagram of multiple image blocks and multiple text blocks in a window drawn according to an embodiment of the present invention. FIG9 is a schematic diagram of multiple sub-window blocks drawn according to an embodiment of the present invention. FIG10 is a schematic diagram of determining a window layout category using a machine learning model according to an embodiment of the present invention. FIG11 is a schematic diagram of determining a model classification result based on multiple target window images according to an embodiment of the present invention.

S310~S340: Steps

Claims (10)

A window layout method, applicable to an electronic device including a display, comprises: displaying a window of an application program through the display; obtaining window information of the window; classifying the window into one of a plurality of window layout categories according to the window information of the window and application information of the application program; and adjusting the window size of the window according to a window size criterion of one of the plurality of window layout categories, wherein the window size of the window is adjusted according to the window information of the window. The step of classifying the window into one of the plurality of window layout categories based on the application information of the application includes: capturing at least one image block and at least one text block in the window in response to the application being a browser program; and determining whether the window is classified into the first window layout category or the second window layout category according to the number of the at least one image block, the block range of the at least one image block, and the block range of the at least one text block. The window layout method as described in claim 1, wherein the step of classifying the window into one of the plurality of window layout categories according to the window information of the window and the application information of the application comprises: judging whether the window meets common conditions according to the active window time, the number of interactive events and the eye retention time of the window; and in response to the window meeting the common conditions, classifying the window into one of the plurality of window layout categories according to the window information and the application information. The window layout method as described in claim 2, wherein the step of judging whether the window meets the common condition according to the active window time, the number of interactive events and the sight dwelling time of the window comprises: calculating a window common score of the window according to the active window time, the number of interactive events and the sight dwelling time of the window; sorting the window and at least one other window according to the window common score of the window; and judging whether the window meets the common condition according to the sorting order of the window. The window layout method of claim 1, wherein the step of adjusting the window size of the window according to the window size criteria of one of the plurality of window layout categories comprises at least one of a first operation, a second operation, and a third operation, wherein the first operation comprises: detecting a user's line of sight; determining whether the line of sight is projected on the window; and responding to the line of sight being projected on the window, adjusting the window size according to the window size criteria. The second operation includes: in response to one of the plurality of window layout categories being a first window layout category, adjusting the window size of the window to a first default size; and the third operation includes: in response to one of the plurality of window layout categories being a second window layout category, adjusting the window size of the window to a second default size, and moving the window to a default display position. The window layout method as described in claim 1, wherein the step of classifying the window into one of the plurality of window layout categories according to the window information of the window and the application information of the application includes: determining whether the window information of the window and the application information of the application match database data recorded in a database; and in response to the window information and the application information matching the database data, determining whether the window is classified into the first window layout category or the second window layout category according to the database data. A window layout method as described in claim 5, wherein the application is a browser application, and the application information includes a web page link address. The window layout method as described in claim 1, wherein the step of classifying the window into one of the plurality of window layout categories according to the window information of the window and the application information of the application includes: dividing the window into a plurality of sub-window blocks in response to the fact that the application is not a browser program; capturing a plurality of component types respectively corresponding to the plurality of sub-window blocks through an application program interface component; and determining whether the window is classified into the first window layout category or the second window layout category according to the plurality of component types of the plurality of sub-window blocks. The window layout method as described in claim 1, wherein the step of classifying the window into one of the plurality of window layout categories according to the window information of the window and the application information of the application comprises: inputting the window screen of the window into a machine learning model to obtain a model classification result; and determining, according to the model classification result, whether the window is classified into the first window layout category or the second window layout category. The window layout method as described in claim 8, wherein the window screen includes multiple target window screens, and the application information and the window screen of the window are input into the machine learning model, and the step of obtaining the model classification result includes: after the application is executed for a preset time, capturing multiple target window screens of the window; inputting the multiple target window screens into the machine learning model respectively, and obtaining multiple classification confidence values of multiple preset model categories of each of the multiple target window screens; and determining the model classification result according to the statistical results of the multiple classification confidence values of the multiple preset model categories of each of the multiple target window screens. An electronic device comprises: a display; a storage device; and a processor coupled to the display and the storage device and configured to: display a window of an application through the display; obtain window information of the window; classify the window into one of a plurality of window layout categories according to the window information of the window and application information of the application; and adjust the window size of the window according to a window size criterion of one of the plurality of window layout categories, wherein The operation of classifying the window into one of the plurality of window layout categories according to the window information of the window and the application information of the application includes: capturing at least one image block and at least one text block in the window in response to the application being a browser program; and determining whether the window is classified into the first window layout category or the second window layout category according to the number of the at least one image block, the block range of the at least one image block, and the block range of the at least one text block.

Images