WO2000025267A1

WO2000025267A1 - A method for magnifying a computer generated image and a graphic user interface incorporating said method

Info

Publication number: WO2000025267A1
Application number: PCT/SG1999/000104
Authority: WO
Inventors: David Poston; Timothy Poston; Shehan Vidarshana Warusavithana; James Kolenchery Rappel
Original assignee: Johns Hopkins Singapore Pte Ltd
Priority date: 1998-10-27
Filing date: 1999-10-22
Publication date: 2000-05-04
Also published as: JP2002528981A; AU6494299A; SG74082A1; EP1131787A1

Abstract

A method and graphic user interface incorporating the method for magnifying a portion of a computer generated image (14) displayed on a screen (10). The method involves generating an image of a panel (12) that is movable over or within the image (14). The panel (12) is sectioned into a central magnifying region (16) and a peripheral compression region (18). An area (20) of the image (14) underlying or within the footprint (22) of panel (12) is mapped into the central and peripheral regions (16, 18) in the following manner. A portion (24) of the area of the image (20) is mapped with a degree of magnification into the central region (16). A margin or portion (26) of the area (20) of image (14) that lies outside the portion (24) but within the footprint (22) is mapped into the peripheral region (18) in a compressed and/or otherwise translated manner. The mapping of the area (20) of image (14) into panel (12) is performed so that the magnified image appearing in the central region (16) is continuous with the compressed image appearing in the peripheral region (18) which in turn is continuous with the remainder of the image (14) outside the footprint (22).

Description

Title

A METHOD FOR MAGNIFYING A COMPUTER GENERATED IMAGE

AND A GRAPHIC USER INTERFACE INCORPORATING SAID METHOD

Field of the Invention

This invention relates to a method of magnifying a portion of a computer generated image on a screen of an electronic visual display unit (VDU) and to a graphic user interface incorporating said method, which allows magnification of a portion of an image without obscuring its contextual relationship with the remaining portion of the image.

Background of the Invention

Throughout this specification the acronym GUI is used to denote "graphic user interface". "Zoom" is used to mean an increase or decrease in the magnification of an image. "Click" means pointing at a specific location on a VDU screen using an input device and simultaneously^" pressing a button on the device to indicate or execute a command. "Drag" means a movement of a screen cursor which pulls a part of an image to which the cursor is attached across the screen. "Scrolling" is where an image larger than the screen is moved across the screen vertically or horizontally in order that it may be viewed sequentially. A "thumbnail" is a miniature version of a larger complete image. "Footprint" is used to denote an area directly beneath or bound by an external periphery of a panel overlaid on or incorporated in an image.

The size of a computer screen or VDU restricts the amount of information and detail that can be displayed at any one time. Many computer graphics packages include a zoom function that can be used to gain a closer look at a part of an image but in doing so reduces the amount of the total image displayed on the screen at one time. Scrolling allows only part of the image to be inspected at any one time. Where an image comprises several pages, the entire image cannot be examined at once and there is a possible risk of confusion.

With complex images in which the interrelationship of different areas of the image is important, the zoom and scroll facilities currently available are unsatisfactory because the magnified or displayed section of the image is not shown within a significant context of the complete imag&- To overcome this problem, some software programs use a thumbnail image in association with a zoom function, where an indication of the area of the magnified image is outlined on the thumbnail in order to show how it relates to the whole image.

Summary of the Invention

The present invention was developed with a view to providing a method and associated graphic user interface that allows magnification of a part of an image while maintaining continuity with the remainder of the image. In this way, magnification is achieved without obscuring any substantive part of the remainder of the image and allowing the magnified portion to be viewed in the context of the whole image.

According to one aspect of the present invention there is provided a method of magnifying a portion of a computer generated image displayed on a screen of an electronic visual display unit, the method involving: generating an image of a panel that is moveable over or within an image displayed on the screen of a visual display unit; sectioning the panel into a central magnifying region and a peripheral compression region; and, mapping:

(i) with a degree of magnification, a portion of an area of the image within the footprint of the panel into the central region of the panel to display a magnified image thereof in the central portion; and, (ii) with a degree of compression, a marginal portion of the image bound between said portion of the area of the image and the remainder of the area of the image within the footprint into the peripheral region to display a compressed image thereof in the peripheral region, said mapping arranged so that the magnified image in the central region is continuous with the compressed image in the peripheral region which in turn is continuous with the image outside the footprint; whereby, in use, a user can move the panel to a desired location on the screen to magnify a-portion- of the image while maintaining the continuity of the image inside and outside the panel.

Advantageously, the continuity means not only that smooth curves in the original image appear unbroken in the transformed image (they are C°), but that their directions change continuously along them (they are C¹).

Preferably the method includes the provision of a control panel at at least one location within the panel which can be pointed to via a user input interface of the computer such as a mouse or keyboard to indicate and execute predetermined commands. Advantageously, these commands include a command to enlarge or reduce the footprint of the panel; a command to allow the panel to be dragged or otherwise moved across the screen; a command to increase or decrease the degree of magnification within the central region of the panel; and a command to change the shape of the panel and/or the shape of the central region and/or peripheral region of the panel.

According to another aspect of the present invention there is provided a graphic user interface for manipulating a computer generated image on a screen of an electronic visual display unit, the graphic user interface including: means for generating a panel visible on the screen of a visual display unit and moveable over or within an image displayed on the screen, the panel having a central magnifying region and a peripheral compression region; and, means for mapping:

(i) with a selected degree of magnification, a portion of an area of the image within the footprint of the panel into the central region of the panel to produce a magnified image thereof in the central region; and, (ii) with a degree of compression, a marginal portion of the image bound between said portion of the area of the image and the remainder of the area of the image within the footprint into the peripheral region to display a compressed image thereof in the peripheral region; wherein the means for mapping maps the image within the footprint of the panel into the panel in a manner so that the magnified image in the central region is continuous with the compressed image in the peripheral region which in turn is continuous with the image outside the footprint whereby, in use, the graphic user interface enables a user to move the panel to a desired location on the screen to magnify a portion of the image while maintaining continuity of the image inside and outside of said panel.

Preferably the graphic user interface includes a control panel visible on the panel and displaying various executable- demands that can be pointed to via a separate user input device connected to the computer such as a mouse or key board for executing said commands.

Preferably the graphic user interface includes a means for enlarging or reducing the footprint of the panel.

Preferably the graphic user interface includes means for dragging or otherwise moving the panel across the screen.

Preferably the graphic user interface includes means for increasing or decreasing the degree of magnification within the central region of the panel.

Preferably the graphic user interface includes means for changing the shape of the panel.

Preferably the graphic user interface includes means for changing the shape of the central region and/or the peripheral region of the panel. Brief Description of the Drawings

Embodiments of the present invention will now be described by way of example only with reference to the accompanying drawings in which:

Figure 1 illustrates one form of a magnifying panel provided by an embodiment of the graphic user interface;

Figure 2 is a representation of the magnifying panel shown in Figure 1 without an underlying image;

Figure 3 is a schematic representation of the method for magnifying an image;

Figure 4 illustrates a simple blockwise mapping scheme incorporated in one embodiment of the method and graphic user interface;

Figure 5 graphically illustrates a second embodiment of a mapping scheme for producing a magnified image;

Figure 6 illustrates the level sets of the square norm useable in defining a shape of a magnifying panel arising from the method and graphic user interface;

Figure 7 illustrates the level sets of the rectangular norm useable in defining a shape of a magnifying panel arising from the method and graphic user interface;

Figure 8 is a representation of a linear interpolation over the range where the image is to be compressed between the constant-multiplier Vfe and the constant 1, and the graph of the resulting radius-to-radius map; Figure 9 is a representation of a non-linear interpolation between constant multipliers, giving a piecewise linear radius-to-radius map;

Figure 10 is a representation of a once continuously differentiable interpolation between constant multipliers giving a differentiable radius-to-radius map;

Figure 11 is a representation of the level sets of the norm ^[(au ^+ v)²"], for n = 2, 3, 4 and 6 useable in defining panel shape;

Figure 12 is the level set of a norm with triangular symmetry, useable in defining panel shape;

Figure 13 represents- level sets of a norm with hexagonal symmetry, useable in defining panel, shape; and,

Figure 14 is a representation of level sets of a non-convex function with hexagonal symmetry, useable in defining panel shape.

Detailed Description of the Invention

Referring to the accompanying drawings and in particular Figures 1-3, embodiments of the present invention provide a method of magnifying a portion of a computer generated image on a screen 10 of a visual display unit (not shown) and a GUI incorporating the same. The method involves generating a image of a panel 12 that is moveable over or within an image 14 displayed on the screen 10. The panel 12 is sectioned into a central magnifying region 16 and a peripheral compression region 18. An area 20 of the image 14 underlying or within the footprint 22 of the panel 12 is mapped into the central and peripheral regions 16 and 18 in the following manner. A portion 24 of the area of the image 20 is mapped with a degree of magnification into the central region 16. A marginal portion 26 of the area 20 of the image 14 that lies outside the portion 24 but within the footprint 22 is mapped into the peripheral region 18 in a compressed and/or otherwise translated manner. The mapping of the area 20 of the image 14 into the panel 12 is performed so that the magnified image appearing in the central region 16 is continuous with the compressed image appearing in the peripheral region 18 and that the compressed image in the peripheral region 18 is in turn continuous with the remainder of the image 14 outside of the footprint 22. This is shown most clearly in Figure 1 where it can be seen that various lines on the image 14 are continuous across the screen 10 and panel 12. Therefore the continuity of the image 14 as a whole remains unaltered and the visual context of the magnified image in central portion 16 is maintained. The panel 12 can be moved to a desired location on the screen 10 by use of a conventional user input interface such as a mouse or key pad to magnify selected portions or areas of the image 14.

In effect, the panel 12 is a visual metaphor of a flat magnifying glass or lens having a central region of constant thickness but with bevelled edges. In such a metaphor the central portion of the magnifying glass is equated with a central region 16 of the panel and the bevelled edges equated with the peripheral region 18. In the same way as the metaphoric magnifying glass can be moved over an image on the piece of paper the panel 12 can be moved across the screen 10 by conventional means. The compression of the marginal portion 26 of the image as it appears in the peripheral region 18 compensates for the increase in space taken up by the magnified portion 24 appearing in the central region 16. Therefore, the panel 12 does not obscure any portion of the image 14 appearing on the screen 10.

When the portion 24 of the area 20 of image 14 to be magnified is in a corner or at the side of the screen 10 the method and GUI operate to cause the entire image 14 to move in an opposite direction sufficiently to allow the marginal region 26 to also be shown in the peripheral region 18. Thus, the action of pushing the panel 12 against the side or corner causes a shift in the whole image 14 in an opposite direction. Once the panel 12 is moved away from the side or corner the image 14 automatically shifts back to its centralised position. While this results in a part of the image 14 most distant the panel 12 to be temporarily out of sight this is not considered to be of any substantial significance as it is reasonable to assume that that area is not of immediate interest while an opposite extremity of the image 14 is being magnified.

The GUI is provided with various user functions that can be activated by clicking on various commands icons that appear in one or more control panels 28 on the panel 12. In Figures 1 and 2, the four control panels 28 are shown, one at each corner of the panel 12. A movement icon 30 is provided in each of the control panels 28 other than that for the top right hand corner. The movement icon 30 can be clicked on by use of a user interface such as a mouse, space ball or key pad to drag or otherwise move the panel 12 about the screen 10. A magnification icon 32 is provided in the control panel 28 at the upper right hand corner. Clicking on this icon allows the user to increase or decrease the magnification of the portion 24 of the image. By implication this also inversely changes the degree of compression of the marginal portion 26 of the image appearing in the peripheral region 18.

Examples of algorithms and strategies for mapping the image in the footprint 22 to the panel 12 will now be described.

Referring to Figures 4 and 5, for each pixel point (u,v) on the screen, there is a point (x_>y) of the original image /, whose colour must be copied to (u,v). (Also, if a mouse is clicked at (u,v), f(u,v) must be evaluated to find which sensitivity region of an object defined in J the mouse click is intended to hit.) For best results / has higher resolution than the unmagnified screen view, so that magnification will reveal more detail, rather than show the lack of it as an enlarged blur. Since (x,y) will rarely be an exact pixel point, standard anti-aliasing techniques can be useful. For the purposes of this description, however, the correspondence of the screen and panel 12 "is" the mapping/.

For convenience, assume below that the units in the screen and in / correspond, so that the map J for which colouring (u,v) with the colour of (x,y) = J(u,v) produces the standard undistorted image can be written as (x,y) = (u,v). Adjusting to pixel units in the description of / is a final step, suppressed here. Assume also for clarity (u,v) is mapped to f(u,v) = jiu-u_&v-Vo) + U^V_Q), keeping the same size and shape.)

Most mappings are much easier to evaluate forwards than to solve, so it is more efficient to work with (x,y) =f(u,v) where / (rather than the inverse off) has a simple expression. Since / must be evaluated for every pixel within the panel footprint, it is preferable to keep it quickly computable.

For all the mapping functions below, the constants W and H are the width and height of the area covered by the panel 12, while w and h are the width and height of the distortion-free area inside it, where the image is magnified by λ. That is, for the example of twofold multiplication (λ = 2), the screen coordinate ( ,v) must there receive the colour at the point (x,y) = (ul2,vl2).

The simplest scheme is to break the panel 12 into nine rectangles these being eight peripheral rectangles 34 that constitute the peripheral region 18 of panel 13 and a central rectangle 36, constituting the central region 16 (Figure 4). Mapping the rectangles one at a time by the obvious scaling reduces to piecewise linear formulae on the two coordinates separately:

■ / (u, υ) = (^χ,y) ( " ) where

if V < -H/2

- λ-ξ(-Η----Λ2)" _ - 2-λ--(!-tf±---/-ι-l) if -H/2 < ≤ -h/2 y =

with ( λ - w) I λ (W - w),Ww (1 - λ) / 2 λ (W - w),(H - λh) I λ (H - h) and Hh (1 - λ) / 2 λ (H - h) as constants fixed when the panel 12 is defined. Note that we must have 0 < to < W (4)

0 < h < H (5)

0 < ω/λ < W (6) < h/\ < H (7)

for sensible results. With λ < 1 (that is, with actual magnification), the last two of these conditions imply the first two.

As Figure 4 illustrates, the blockwise approach creates 24 distinct boundaries, where the mapping is continuous but (except for vertical and horizontal lines) all directions change. These are somewhat distracting to the eye, as the angles they create may be seen as "features" of the image competing visually with the corners and boundaries in the image itself. Instead of rectangular blocks, the area of the image within footprint 22 and panel 12 can be broken into a central rectangle and four surrounding trapezia, with pure scaling on the central region and bilinear interpolation in each surrounding trapezial piece of the bevel (Figure 5). In the left-most trapezium, for example, the mapping / is defined by

, , ( W ( W\ Wλ - 2w ( W\ - l \

( W ( W\ Wλ- 2w .- ^■ , „-, w- X \

This gives only twelve lines or boundaries in the panel 12 in which there is a change in direction of the image. Such lines can be entirely avoided, by observing that this example is a member of a wider class defined as follows.

Define a radial map as a mapping of the form x

Λ (u, υ) (8) y where A (u,v) is a scalar function of u and v. Any such map takes rays (radial lines) to rays, in the same direction. We have a radial panel 12 with magnification λ if the value of Λ is exactly λ for (M, V) near to (0,0) and exactly 1 for large values. It is a homeomorphic radial panel 12 if it is continuous and has a continuous inverse.

Along a ray A can be expressed as a function of radius. In the mappings described here, this function is assumed to be given by a formula independent of which ray is involved, though "radius" need not be the usual Euclidean distance measure Vft t-v²]. We take A of the form A (u,v) = l(p(u,v)), where p is any function that increases outward along rays, and / maps the values of this to the multiplier Λ we will apply to points. With Euclidean distance as p, we get for an a rotationally symmetric panel 12; useful for some purposes, but note the best for a rectangularly organised image. We illustrate here some other choices for p.

One of these is the square norm p («_f >) = max(H , |t |) C** ) whose level sets are shown in Figure 6. These correspond to the 'contours' of a geographical map; curves of equal height in that case, curves of equal p here. The region where p is less than or equal to a certain value P is the region within the curve where p is exactly equal to P. Evidently, if the multiplier / is 1 / λ up to where this p reaches the value P, we have a square region where display points correspond to original image points 1 / λ as far from the centre, so that in this region the image is magnified by λ. For a rectangular region, we need only add coefficients. If

P ,_b (u_>v) = max (a \u\ , b \υ\) ( _{i 0})

then its level sets when α = 2 and b = 3 are 2/3 as wide as they are tall. Whatever / does, the effects of / (pα,b(u,v)) will be organised in rectangular regions, just as those of / ( [M² + V²]) will be organised in disks and annuli.

Before going on to other radial functions, consider candidates for /. The simplest uses piecewise linearity, as in (2). In the inner region, magnification by λ requires division by it: linearly interpolate between this and (outside the panel) multiplication

The curve with level parts in Figure 8 graphs / with λ = 3, r - 3 and R = 4, a constant multiplier V3 up to p = 3, a constant multiplier 1 beyond p = 4, and a straight (linear) interpolation between these. The ascending graph of /p shows how multiplication by ¹Λ stretches the range from 0 to 3 in screen space only over the range from 0 to 1 in the original image, and thus displays the latter range enlarged by 3. The map in the squeezed region is non-linear: to be piecewise linear along rays, set

giving piecewise linearity after multiplication (figure 9). With the Euclidean p, this gives directional discontinuity only on the two circles bounding the panel and the magnified image: the level sets of p = r and p = R for this case. With the square norm, it reproduces trapezial mapping.

These formulae allow us fewer boundaries, but at these boundaries we will have changes in direction. It is smoother to use an interpolating cubic, requiring that it be level where it meets the constant regions. As with linear interpolation, polynomial interpolation for the multiplier can lead to many-to-one mappings, so it is better to divide by p. Setting l/λ if 0 < p ≤ r i = l_n+l, t>+hf>'+l*P³ j < p < R HI) if R < p

^- = 2l₃p + l₂ - ^ or 0 dp f? cm)

we can require k + I_IΓ + .₂Γ² + .₃Γ³ 1 λ

2.₃r + -2 - k CI6) lo + hR + hR + hR³ = 1 tπ)

R

Our coefficients are given by

( -2o)

Our definition of / thus becomes

1/λ if p < τ l(p) = , l afΛ- l)r-^l Λ^a . «r' l+r«^a4-lt^a ---X-riAfl-<rλltⁱ - -³ +.-R + R² -λ-J -λ-R-ΛR³ . , r+R-rA-RA -a l if _r < ₀ <

I K-r)3, + A(Λ-r.3 ^{2 '} «->³ " + A<J-—)3 * | " ^r - P - ^Λ

1 if i- < /> with the results shown in Figure 10.

We have now made / once-differentiable (C¹). In combination with p = [«² + v²] the resulting map is C¹ also. By the use of application of standard interpolation formulae, this differentiability can be extended to C* by any person skilled in the art. However, the square norm and its rectangular relatives are not differentiable along the lines where the choice made by the max function changes. We can get an analytic (C) form for p throughout the non-constant multiplier region, replacing the rectangular boundary by an analytic curve near it, if we use

for some n > 1. Figure 11 shows the contours of this, when n - 2, n = 3, π = 4 and n = 6, for the values a = 2 and b = 3 used to rectangularise the square norm. These are easily used in managing the size and shape of the panel, since along the axes p reduces simply to a lu I and b \v I. Without loss of generality (since the shape is controlled only by the ratio of a and b), we can fix b = 1. So to arrange a panel 12 whose inner region is h high and whose overall height is H, we use On with / as in (11,12,13) and r = h/2, R - H/2. Setting a = HIW, we then get p = R when u = ± W/2, correctly setting the edges of the region where / ≠ 1. This automatically forces a width w = Wh/H, with the inner region a scaled copy of the whole panel shape, and a bevel in proportion. This is best in many cases, but generalisations to cover separate control of all widths and heights are straightforward to a person skilled in the art. A great variety of panel shapes are possible in this overall scheme. The above examples are all round or rectangular, but the class of radial maps here defined includes triangularly symmetric shapes (Figure 12) with

hexagonally symmetric shapes (Figure 13) with p = g (u + v + [ - 3uvⁱ)^{2 z<}*) and so on. They need not even be convex: with α4 = 0.1 the hexagonal form (24) gives the shapes shown in Figure 14. These variations illustrate the power of the general scheme.

Claims

The Claims Defining the Invention are as Follows:

1. A method of magnifying a portion of a computer generated image displayed on a screen of an electronic visual display unit, the method involving: generating an image of a panel that is moveable over or within an image displayed on the screen of a visual display unit; sectioning the panel- into a^{^}central magnifying region and a peripheral compression region; and, mapping:

(i) with a degree of magnification, a portion of an area of the image within the footprint of the panel into the central region of the panel to display a magnified image thereof in the central portion; and, (ϋ) with a degree of compression, a marginal portion of the image bound between said portion of the area of the image and the remainder of the area of the image within the footprint into the peripheral region to display a compressed image thereof in the peripheral region, said mapping arranged so that the magnified image in the central region is continuous with the compressed image in the peripheral region which in turn is continuous with the image outside the footprint; whereby, in use, a user can move the panel to a desired location on the screen to magnify a portion of the image while maintaining the continuity of the image inside and outside the panel.

2. The method according to claim 1 further including the provision of a control panel at at least one location within the panel which can be pointed to via a user input device connected to the computer and execute predetermined commands.

3. The method according to claim 2 wherein one command is to enlarge or reduce the footprint of the panel.

4. The method according to claim 3 wherein a further command is to allow the panel to be dragged or otherwise moved across the screen.

5. The method according to claim 4 wherein a further command is to increase or decrease the degree of magnification within the central region of the panel.

6. A method according to claim 5 wherein-, a further corn-tnand is to change the shape of the panel and/or the shape of the central region and/or peripheral region of the panel.

7. A graphic user interface for manipulating a computer generated image on a screen of an electronic visual display unit, the graphic user interface including: means for generating a panel visible on the screen of a visual display unit and moveable over or within an image displayed on the screen, the panel having a central magnifying region and a peripheral compression region; and, means for mapping: (i) with a selected degree of magnification, a portion of an area of the image within the footprint of the panel into the central region of the panel to produce a magnified image thereof in the central region; and,

(ii) with a degree of compression, a marginal portion of the image bound between said portion of the area of the image and the remainder of the area of the image within the footprint into the peripheral region to display a compressed image thereof in the peripheral region; wherein the means for mapping maps the image within the footprint of the panel into the panel in a manner so that the magnified image in the central region is continuous with the compressed image in the peripheral region which in turn is continuous with the image outside the footprint whereby, in use, the graphic user interface enables a user to move the panel to a desired location on the screen to magnify a portion of the image while maintaining continuity of the image inside and outside of said panel.

8. A graphic user interface according to claim 7 further including a control panel visible on the panel and displaying various executable demands that can be pointed to via a separate user input device connected to the computer for executing said commands.

9. A graphic user interface according to claim 8 further including means for enlarging or reducing the footprint of the panel.

10. A graphic user interface according to claim 9 further including means for dragging or otherwise moving the panel across the screen.

11. A graphic user interface according to claim 10 means for increasing or decreasing the degree of magnification within the central region of the panel.

12. A graphic user interface according to claim 11 further including means for changing the shape of the panel.

13. A graphic user interface according to claim 12 further including means or changing the shape of the central region and/or the peripheral region of the panel.

14. A graphic user interface according to claim 7 further including means for shifting the whole of the computer generated image when the panel is moved to an edge of the screen, the shifting means shifting the image in a direction and by a displacement so that the marginal portion of the image is displayed in the peripheral region.