HK1073199A - Method and device for image display - Google Patents

Description

Image display method and apparatus

Technical Field

The invention relates to a method for image display according to the preamble of claim 1 and to an apparatus for image display according to the preamble of claim 10.

Known techniques

In typical open surgery, the surgeon views the surgical site directly with his eyes and can use this typical stereoscopic vision to perform various actions with great accuracy. However, it is becoming increasingly common to use closed surgical techniques, where the procedure is performed in vivo through only a small external incision. This makes it difficult for the surgeon to obtain a good view of the surgical site. To solve this problem, a technique has been employed in which an endoscope is inserted into the surgical site and an image viewable by the surgeon on a display screen is obtained by means of this. However, such imaging has the disadvantage that it does not provide information at any depth and furthermore the image is transferred from the surgical site, as a result of which the surgeon finds it difficult to operate with the required accuracy.

Efforts have been made to achieve stereoscopic imaging of images with a certain separation and displayed on a monitor with a certain offset, viewing the monitor with special 3D glasses, and thus experiencing a perception of depth of the image. However, such solutions prove difficult to use and fatigue the surgeon, especially during operations that require a long time.

In order to give the surgeon a more comfortable working position and to allow his head to be directed at the surgical site at all times, attempts have been made to use head mounted image display devices. However, such devices have been found to be rather heavy and thus inconvenient to use, and have the additional drawback of obscuring the surroundings to a considerable extent, with consequent difficulty for the user to point himself around the operation. In addition to causing neck pain, this also has problems with the back, eye fatigue, etc., with the result that the user often feels a boat-like feel, with the result that problems are caused in precision surgical work.

Against this background, there is a clear need for new and better solutions in this area.

Objects of the invention

The aim of the invention is, for example, to enable the surgeon to obtain a better visual perception of the hidden surgical site than was previously possible, while at the same time enabling the surgeon to perceive the surrounding situation, in particular his own hands and instruments used for the operation. Another object is that the solution must be quiet for the user with good optical properties and must be simple to use. Another object is to achieve a head mounted device which does not feel heavy and which is easy to adapt to different individuals.

These objects are achieved according to the invention with a method having the features indicated in claim 1 and also by means of a device having the features indicated in claim 10.

The use of a head mounted device that projects an image onto the front of each eye enables the image to be at a desired distance in front of the eye, creating a perception of depth of the image. Also by having these image display positions each occupy a relatively limited portion of the user's total field of view, a free view can be achieved above and below the device and sideways, which helps to allow the user to largely maintain a good orientation in the space in which the device is used, which is particularly important during extended periods of operation.

Furthermore, according to the invention, the device obtained is easy to adapt to different individuals by changing one or more elements, without the need to change all other parts of the device. For example, each user can use his own device specifically adapted to his own vision, which can easily be supplemented by additional devices for certain working situations.

Since the heavy part of the device is worn high at the waist corners of the user, the head-mounted part of the device can be made low in weight, which gives the user increased freedom of movement and is comfortable, especially for prolonged use. In this regard, however, as a result of further development of the elements, it is desirable that the device be lightweight so that the entire unit can be mounted to the head.

Electronic image processing enables the displayed image to be controlled without changing the optics by selecting a specific image processing program for image processing, and as a result the image is particularly adapted to the specific work situation. In this way, the focusing can be varied to a certain extent, for example by electronic means, which is the same for light intensity and contrast, for example. It is also possible in a simple manner to store the desired image types by electronic means and to retrieve these images from the memory for use. In a corresponding manner, for example, appropriate settings for different working environments and lighting conditions can be stored and retrieved quickly from the memory.

But also other information, such as X-ray images, images produced by magnetic cameras, magnetic recorders etc. can be displayed temporarily to the user when required by the electronic means and used to return to the actual working environment.

The solution according to the invention also enables, by adding one or more devices or one or more monitors, to display the working scenario that the surgeon experiences during the operation, for example a situation of value for training purposes.

Other features and advantages will be apparent from the following description and from the patent claims.

The invention will be described in more detail hereinafter with reference to an embodiment shown in the drawings.

Description of the drawings

In the drawings:

figure 1 shows a block diagram of an image display device according to the invention,

figure 2 shows a perspective view of a head mounted display according to the invention,

figure 3 shows a side view of the partially exploded display of figure 2,

figure 4 shows a perspective view of the display module seen diagonally from the back,

figure 5 shows a vertical section through one cell in a display module,

figure 6 shows a user of a device according to the invention,

figure 7 shows a front view of an embodiment of a display according to the invention,

FIG. 8 shows a side view of the display of FIG. 7, an

Fig. 9 shows a view from above of the display in fig. 7 and 8.

Description of the embodiments

Fig. 1 schematically shows a block diagram of an image display device 1 constructed in accordance with the present invention with which a user 2 can view an object 3 that is normally hidden from the user. The user 2 may for example be a surgeon who needs to observe an internal organ in the human body, which organ is relevant for performing a surgery by means of closed surgery, that is to say a surgery in the patient, without the surgeon having to directly see the operation site. In this case, the object 3, or the surgical site, can be observed using an endoscope 4 inserted into or inside the body, which transmits an optical signal to an image processing device 5 connected to an image observation device 6 worn by the user 2, through which the user 2 can see the object 3 with both eyes.

The optical signal transmitted from the endoscope 4 is input to the camera 7, which converts the optical signal into an electric signal representing an image of the object 3. The signals are input from the camera 7 to a computer 8, where appropriate signal processing takes place, and where the signal processing can be controlled in various ways according to the needs of the different occasions. The different settings and imaging results can be visually monitored using a monitor 30 connected to the computer 8. The computer 8 together with the camera 7 can advantageously be designed as a processing image basis for displaying stereoscopic images to the user. The monitor 30 is not equipped or used to view such images, but is used to display two-dimensional images. The camera 7 and the computer 8 can of course also display two-dimensional images if this is desired.

By means of a suitable interface, signals are transmitted from the computer 8 to the image-viewing device 6, where they first reach the driver unit 9 and then, after having been so processed, they reach the display 10, which comprises a display unit 11 intended to be attached to a support 12, which is worn by the user 2 in a manner corresponding to glasses. The display unit 11 is again divided into two different display modules 13 and 14, one for each eye of the user. For example by using the endoscope 4 to stereoscopically image the object 3, the computer 8 and the driver unit 9 are able to display different images in the two display modules 13, 14 after appropriate image processing in the control unit 7, so that the user 2 experiences the perception of depth of the image of the object 3 displayed by the two display modules 13, 14 as a result of the stereoscopic vision. Thus, it is desirable that the user experience that the virtual object displayed in front of his eyes is at substantially the same distance from the actual object and that the focal length of the virtual image is a predetermined distance from the user, which is within a usual working distance for the user, i.e. the surgeon. In addition, a normal two-dimensional picture can be selected.

The design of the display 10 is shown in more detail in fig. 2 and 3. The holder 12 is suitably designed as a pair of goggles for protecting the glasses of the user, which goggles are advantageously adapted optically to the wearer. For example, the optical adaptation may be ground into the spectacle lens itself, or may include a special attachment fitted inside on the spectacle. Such that each user has a cradle 12 that is compatible with his own individual. In fig. 2, the display 10 is shown ready for use, with the display unit 11 mounted on the stand 12 such that the display modules 13, 14 are each positioned in front of one eye of the user, and in fig. 3, the display unit 11 is shown removed from the stand 12. For example for close-up work or fine work, the user's vision can be adapted in a simple manner to suit the desired work situation by mounting the display unit 11 adapted to different visual needs to the support 12. According to fig. 6 the driver unit 9 is adapted to be worn at the waist corner of the user 2, and in order to form an electrical connection between the display unit 11 and the driver unit 9, the stand 12 has one or more wires 15 which can be connected to the display unit 11 via a connector 16 partly mounted on the stand 12 and partly on the display unit 11. This connector 16 can be designed in a manner as desired to provide individual connections of the display modules 13, 14 to the driver unit 9.

As also shown in fig. 2 and 3, the design of the display unit 11 is such that the display modules 13 and 14 only partially block the view of the user 2, which allows the user to freely see the sides of the display unit 11 and above and below it. Substantially enabling the user to see his hands and the instruments used and providing a good spatial orientation and feel in the area where the work is taking place, especially during long periods of work. Being able to see up and down eliminates the risk of the user feeling unstable and seasick. Further, according to the present invention, the display 10 as a combination of the display unit 11 and the stand 12 is low in weight, preferably less than 100g, which means that there is no trouble of feeling strain of the neck and back. Furthermore, the power consumption of the device is low, which means that no troublesome temperature problems are caused.

The construction of the display unit 11 is shown in more detail in fig. 4 and 5. Fig. 4 shows that the two display modules 13 and 14 are supported on a binding holder 17, which is designed to be attached to the stand 12, which holder 17 suitably allows the two display modules 13 and 14 to be moved so that their separation distance matches the pupil distance d of the user to provide optimal vision. The two display modules 13 and 14 are also shown to be substantially rectangular in shape, for example, with a width greater than a height, as viewed by a user. The size is such that the user can see the diagonal within an angle a of at most about 50, suitably about 30. The visible circular image should be able to form an angle beta of at least about 15 deg., suitably about 25 deg.. The nature of the two images displayed is furthermore such that they give the user the impression that the observed size 3 lies at a predetermined working distance, for example approximately 0.2-1.2m from the eye. For sitting work a suitable distance is about 0.4m, and for standing work a suitable distance is 0.6-0.8m, which is the length of the arm. For microscopes, on the other hand, a suitable distance may be less than up to 0.2 m.

Fig. 5 shows a vertical section through a display module 13, the corresponding structure being applicable to another display module 14. The user sees the image displayed in the specially designed prism 18, and the perception is that the displayed object is located above the prism in the direction in which its gaze is directed. The image is produced electronically over the prism 18 by means of a substantially horizontally positioned image screen 19, suitably of the LCD type, backlit by means of a light source 20 and capable of being viewed from the prism 18. In order to have a good view during the day it is therefore important to be able to set the appropriate contrast and light intensity. In order to clearly distinguish the image field of view from the surroundings, a frame, for example a black frame, surrounds the image field of view. The electronic circuitry 21 for controlling the picture screen 19 is located above the light source 20. In more detail, the above-mentioned components are suitably protected by a cover of a not shown lightweight material. In this way, a compact, lightweight structure is obtained for the display module 13, wherein the prism 18 may be of glass or other suitable material. Wires 15 connect the display module to the driver 9 for transmission of electrical signals and power.

In order to enable a stable vision of the user with relaxed eye muscles, the display module 13 is suitably positioned according to fig. 5 such that the user's gaze is directed slightly downwards at an angle delta relative to the horizontal. The magnitude of this angle delta should suitably be in the range of about 2-12 deg., and preferably about 5-9 deg.. In order to enable a good individual fit for the user during the day, it is also expedient to be able to adapt other adjustments to other physical characteristics of the user in addition to the above-mentioned setting of the pupil distance. Thus, for example, adaptation to different users can be achieved by forming variously adjustable holders 17 between the display unit 11 and the stand 12. This makes it possible, for example, to pivot the display unit 11 about a horizontal axis with its centre at the pupil, by the entire display unit 11 being mounted such that it can be rotated relative to the support 12 such that it can be moved in the vertical direction along a suitably curved trajectory at the holder 17. By this means the direction of observation can be varied in the vertical direction, for example from about 10 ° above the horizontal to about 10 ° below the horizontal. Furthermore, by forming the holder 17 such that it allows the distance from the display unit 11 to the user's eyes to vary, in cooperation with adapting the user's pupil distance, a good individual adaptation between the user and the display unit can be achieved.

In order to achieve a good individual fit of the prism 18, the shape of this surface facing the eye should have a shape corresponding to the shape of the eye of the observer. Since the eye typically has a larger radius of curvature when viewed in the horizontal plane than when viewed in the vertical plane, the prism should have a curvature corresponding to the eye-facing surface, with the curved surface with the eye and prism viewed in both the horizontal and vertical planes having substantially the same center of curvature. This means that users with different eye sizes should adjust the display unit individually to achieve optimal vision. By making the prism 18 adjustable relative to the eye, the wearer can be given an optimal position for the prism. In this respect, it is desirable that the prism or the display unit is adapted well to different eyes, and should be able to be rotated slightly sideways, so that the correct position can be set for the geometry of the eye, in particular the position of the macula in the eye.

Electronic image processing enables the image that the user will see to be manipulated in several different ways by means of suitable image processing programs adapted to the connected hardware. For example, the color, contrast, and light intensity of the image are simply changed as needed by the operational control of the driver unit 9. By this means, appropriate changes can be made as work progresses, as needed. It is also possible to use several previously determined parameter sets and to choose between these combinations as desired. Another possibility is, for example, for a better detailed view and a better overall view, to manipulate the image by means of suitable software such that it shows the user that it is moving closer or farther away. Electronic image processing also enables it to be easily adapted to different types of peripheral devices, such as monitors and display devices, from each other.

According to fig. 1, the image viewing apparatus 6 shown can also be provided with sound means 21 enabling the user to perform acoustic communication, voice control, etc. For this purpose, for example, a conductive microphone may be coupled to the arm of the support 12, shaped like a spectacle frame and thus able to come into contact with the skull of the user, while the speaker consists of an earphone fitted to the ear of the user. In fig. 6 the user 2 is shown connected to the communication processing means 5 by a wire 22 for transmitting signals and power, but it is of course within the scope of the invention to place the necessary power supply in the form of a battery in the driver unit 9 and to use wireless signal transmission to the image processing means 5 and other means. For example without any hindrance to the connection of more than one image viewing means 6 to the image processing means 5, which can also be provided with more than one monitor if desired.

By providing the device with a motion sensor for detecting head movements of the user, the image can be moved in relation to the eyes of the user when the head is moving.

Each of the two images displayed is adapted in color and has, for example, about 800 horizontal pixels and about 600 vertical pixels for each of the three primary colors RGB (red, green and blue), so that each image can contain three times the number of pixels, i.e. about 1.4 million pixels. However, cameras and monitors currently available on the market do not have such high resolution, and therefore the quality of images that can be used is still limited by these devices. As mentioned, the properties of the image can be changed by the user by means of a control 23 for proper operation of the driver unit 9, or for example by means of voice control.

The image signals generated by the endoscope 4 and the camera 7 can be transmitted to the two display modules 13 and 14 in various ways for stereoscopic vision. One possibility is to use separate conductors, one carrying the right-hand image and the other carrying the left-hand image. Another possibility is to use a shared conductor almost directly opposite the display elements and to alternate the transmission of right-hand and left-hand images through this conductor and to divide the signal into two conductors only close to the display elements, one for each display element. This latter reduces the need for wires as much as possible.

Of course, the device according to the invention can be used for other purposes than surgery. One possible field of application of this kind is, for example, the inspection or working of difficult-to-access areas in mechanical devices.

In order to ensure that the device functions well, it is convenient that the computer 8 monitors the device by automatically performing regular functional control, at least when the device is put into use and is regular during use.

Further embodiments of the display 10 according to the invention are shown in more detail in fig. 7-9. As shown, the brace 12 is designed here with the frame 24 and goggles 25 adjacent to the user's face and eyes. The visor 25 can be fitted on the frame 24 so that it can be removed, for example for cleaning. As in the previously described embodiment, the display unit 11 is mounted on the stand 12 such that it can be removed and electrically connected to the stand 12 by means of a connector (not shown), which in turn is designed to be connected to the driver unit 9 by means of a wire 15. The conductor 15 is suitably divided according to fig. 9 into two branches 15a and 15b at the support 12, each of which runs through one arm of the support 12, through to the respective display unit 13, 14.

In order to make the display 10 more lightweight to wear, the two arms of the frame 24 are connected at the back by a strap 26, the strap 26 having a pad 27 thereon designed to contact the back of the user's head, the length of the strap 26 and the position of the pad 27 being suitably adjustable to fit the user. A suitable position of the pad 27 when in use is shown in figure 6. To relieve strain on the back of the user's head and its back, a tension receptor 28, for example in the form of a clip or the like, may be provided to grip onto the user's clothing to reduce the load from the wires 15 on the back of the user's head.

To relieve pressure on the user's nose to make the display 10 more comfortable to wear, a strap 28 may be attached to each side of the front of the frame 24, as shown in fig. 9, extending over the top of the user's head and supported by two straps 29 spaced apart. By adjusting the length of the straps 28, pressure on the user's nose can be relieved to the desired extent and, since the straps 29 are spaced apart, pressure on sensitive central areas of the skull is avoided.

According to the invention the viewer's sufficient perception of the depth of the image is obtained by the focal plane being positioned substantially in the center of the image. With this arrangement, it is naturally perceived that the subject is in front of or behind the focal plane.

It has been proposed above that the display unit 11 can be adjusted to the user's pupillary distance d by making the two display modules 13 and 14 movable in relation to each other. Another advantageous possibility is to manufacture the display unit 11 in a small number of standard sizes for the pupil distance d, for example in three different standard sizes. The result is a simplification of the display unit 11, while at the same time most users will be able to show one of these standard sizes. In connection with this, in order to be able to fine-tune the interpupillary distance individually, it can advantageously be configured such that the image in each of the two display modules 13 and 14 can be moved slightly in the horizontal and vertical directions by electronic means. The appropriate value for this shift may be, for example, 8 pixels in each lateral direction and 6 pixels in each up and down direction starting from the normal position.

In order to be able to fully release the user's field of view, the display unit 11 can be made so that it can be flipped up, so that it can be flipped up away from the user's field of view if desired.

As described above, the focal plane of the image is located at an appropriate distance from the user who is working. For certain jobs, this distance can be increased to about 2 meters.

For a user wearing glasses, as described above, the optical adjustment of the display 10 can be performed by means of its holder 12 fitted with optically adjustable goggles. However, the user's own glasses may also be used as the installer for the display unit 11.

Claims (22)

1. An image display method in which an object (3) is imaged and after image processing the image is displayed on a head mounted apparatus (10) in front of the eyes of a user of the apparatus, characterised in that the image is displayed in front of each eye of the user within a restricted portion of the field of view of each eye, so that the user can perceive its environment at least beneath and to the side of this image.

2. A method according to claim 1, characterized in that two different images are displayed to the user, one in front of each eye, so that the user perceives a stereoscopic image with a sense of depth at a predetermined distance in front of his eyes in a part of his field of view.

3. A method according to claim 2, wherein each image is displayed such that the perceived image is perceived to be within the normal arm length and working distance of the user, the distance being approximately 0.2-1.2 meters from the eyes of the user.

4. A method according to claims 1-3, characterized in that the display of the image is performed such that, when looking, the user can point his gaze slightly downwards, suitably up to about 2-12 °, relative to the horizontal plane.

5. A method according to claims 1-4, characterized in that each image is displayed in a circular field of view with a viewing angle (β) of at least about 15 ° for each eye.

6. A method according to any one of claims 1 to 5, characterized in that each image is displayed in colour.

7. A method according to any of claims 1-6, characterized in that each image is displayed by electronic means.

8. A method according to any of claims 1-7, characterized in that each image is displayed in a position individually adapted to the eyes of the user.

9. A method according to any of claims 1-8, characterized in that each image is displayed in a dark frame in order to clearly distinguish it from the surroundings.

10. Apparatus for image display, with image processing means (5) connected to imaging means (4), and image viewing means (6) connected to the image processing means, which image viewing means incorporate a display (10) for wearing on the head of a user in order to display said image of an imaged object in front of the eyes of the user, characterized in that the display (10) is formed by two display modules (13, 14) which are designed to be positioned in front of each eye, in a limited part of the field of view of each eye, each display module being arranged to display its image in said part of the field of view of the user and being constructed to leave a free field of view towards the environment at least downwards and to the side of the user.

11. A device according to claim 10, characterized in that the display module (13, 14) is arranged to display two different images, one in front of each eye, so that the user experiences a stereoscopic image with a sense of depth in a part of his field of view at a predetermined distance in front of his eyes.

12. A device according to claim 10 or 11, wherein each display module has a maximum viewing angle of about 50 ° for each eye, preferably about 30 °.

13. A device according to any one of claims 10-12, characterized in that the two display modules (13, 14) are integrated together into a display unit (11) designed to be mounted to a support (12) shaped to fit the head of a user, the display unit and the support together forming the display (10).

14. Device according to claim 13, characterized in that the holder (12) is configured as a pair of safety glasses for the user, which safety glasses are advantageously capable of optically adapting to the user, and in that the display module (13, 14) is configured to be located outside these safety glasses.

15. A device according to any one of claims 10-14, characterized in that the image viewing device (6) comprises, in addition to the display (10), a driver unit (9) connected thereto, which is designed to be worn by the user separately from the display (10).

16. An arrangement according to any one of claims 10-15, characterized in that both display modules (13, 14) are adjustable to individual position settings to adapt at least the user's interpupillary distance.

17. A device according to any one of claims 10-16, characterized in that each display module (13, 14) comprises a prism (18) designed for viewing, and a picture screen (19) arranged on the upper side of the prism, which screen is arranged to be illuminated from behind by means of a light source (20) so that the picture screen can be viewed through the prism.

18. A device as claimed in claim 17, characterized in that each display module (13, 14) is provided with an electronic circuit (21) for controlling the picture screen (19), which electronic circuit is preferably located above the light source (20).

19. A device according to claim 17, characterized in that the side of the prism (18) facing the viewer is concave and preferably has a curvature corresponding to the shape of the user's eye.

20. A device according to any one of claims 10-19, characterized in that the display module (13, 14) is positioned relative to the eyes of the observer in such a way that the observer can direct his gaze direction slightly downwards relative to the horizontal plane.

21. Device according to any of claims 10-20, characterized in that the pair of display modules (13, 14) is designed for a predetermined type of task and can be changed for different tasks and visual needs.

22. A device according to any of claims 10-20, characterized in that the display modules (13, 14) are arranged to display the image in a dark frame in order to clearly define the image and the surroundings.