JP2001005994A - Device and method for image processing - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize an image processor and an image processing method capable of registering, reproducing and displaying a real scenic image with a simple configuration. SOLUTION: Image data from an image pickup part 12 are registered in a main camera 10 corresponding to a block in map information stored in a map memory 30, when a three-dimensional map corresponding to positions is displayed on a monitor display device 22, an image representing image data corresponding to the block is pasted and shown, information guiding a route to a destination is shown on the basis of road information of the map information, display images like real scenes are displayed up to the destination along a travel route.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and an image processing method capable of generating a stereoscopic image.

[0002]

2. Description of the Related Art Conventionally, there has been a system for displaying a digital map image of a desired point by a computer device. For example, a car navigation system that displays map information such as roads and city blocks on the original screen by computer graphics (CG) processing and guides the route to the destination requires a bird's-eye (bird's-eye) view of the road from the sky, There is one that displays a route to a destination.

[0003] For example, Japanese Patent Laying-Open No. 6-348815 discloses a method of creating a landscape image using a three-dimensional model.
JP-A-8-101924 describes a method of synthesizing a CG moving image with a real image, and JP-A-10-42282 discloses a video display in which map information is superimposed and displayed on aerial image data. The system was listed.

[0004]

However, for example, JP-A-6-348815 mentioned above discloses a method of generating a three-dimensional computer graphic image by adjusting the height of a three-dimensional model to a second photograph. In order to realize this, a large-scale system was required. Also, when taking photos, you only need to consider the viewpoint during playback,
Conversely, many processing procedures were required to generate a stereoscopic image.

[0005] Portable devices, such as cameras and information terminals, are required to be smaller and consume less power. Such compact devices can be used to change the current position, destination, and landscape while moving. It is difficult to display the information and to easily grasp the current location, and to register such peripheral information in the device.

For example, the present location is grasped by a portable device,
There is a need for a device that determines a shortest course to a plurality of major destinations in the map by comparing the map with the surrounding situation map stored in the memory, and generates a simulation image that displays the landscape from the current location to the destination. Was. However, buildings and other structures are represented one by one in the same way as the actual landscape.
When a three-dimensional model is generated by computer graphics processing, the amount of processing becomes extremely large, especially due to the increase in the number of polygons for generating a three-dimensional solid and the complexity of various types of image processing. A circuit configuration is required, and power consumption increases. For this reason, for example, the battery driving time is shortened, and the method as in the conventional example cannot be simply applied to devices such as portable cameras and information terminals.

An object of the present invention is to provide an image processing apparatus and an image processing method capable of solving the drawbacks of the prior art and registering and reproducing and displaying an actual landscape image with a simple configuration. Aim.

[0008]

SUMMARY OF THE INVENTION In order to solve the above-mentioned problems, the present invention relates to an image processing apparatus for displaying a three-dimensional state of a structure so that the three-dimensional state of the structure can be displayed. Block creation means for creating block data indicated by simplified blocks, storage means for storing first image data representing a surface image of the block corresponding to the block, and first data stored in the storage means Based on the image data and the block data, an image corresponding to the first image data corresponding to the block is pasted as a plane image of the block, and second image data representing a plurality of structures three-dimensionally is formed. It is characterized by including image processing means for creating, and image output means for outputting the second image data processed by the image processing means to a display means.

In order to solve the above-mentioned problems, the present invention provides an image processing method for displaying a three-dimensional state of a structure so that the three-dimensional state of the structure can be displayed. A block creation step of creating block data indicated by a block, and a first step representing a surface image of the block.
And a first image corresponding to the block as a surface image of the block based on the first image data and the block data stored in the storing step, wherein the first image data and the block data are stored in the storing step. An image processing step of pasting an image corresponding to the data to create second image data representing a plurality of structures three-dimensionally, and outputting the second image data processed in the image processing step to a display unit And an image output step.

[0010]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of a digital camera to which the present invention is applied will be described in detail with reference to the accompanying drawings. Referring to FIG. 1, there is shown a block diagram of a digital camera 10 capable of creating and displaying a three-dimensional stereoscopic map image. This camera 10 has a function of pasting a photographed image to a three-dimensional map to create a three-dimensional map close to the actual scenery in addition to a usual photographing record and image reproducing function, and a destination according to the created three-dimensional map. An image capturing / recording and image display device having a function of clearly displaying directions to the vehicle and a current position and its surrounding information.

For example, when a store providing an image data print service is selected as one of a plurality of registered destinations, the camera 10 moves from the current position to another location via another transit point or the like. The route to the point is shown, and the landscape around the moving route is sequentially displayed according to the position to guide the user to the destination. Upon arriving at the store, the user obtains a photographic image by requesting a service for printing image data and other information recorded on the memory card 26 by, for example, photographing with the camera 10 and recording on the optical disk. Further, at such a store, a memory card in which other map information and photographic images, and further, sightseeing information and the like are recorded is obtained, and the information is supplied to the camera 10, and a map or photographic image corresponding to these data is obtained. To perform image reproduction and route guidance based on a new map. In this way, the camera 10 has a photography function,
It is a multifunctional mobile terminal that includes a route guidance function and a landscape creation display function using a digital map, and an input / output and display function of various information.

Hereinafter, a detailed configuration of the camera 10 will be described. The camera 10 has an imaging unit 12 that photoelectrically converts an optical image corresponding to an object scene, and the imaging unit 12 is generated by a light receiving unit (not shown) arranged in a plurality of horizontal and vertical scanning directions. An image sensor that reads out the electric charge as an electric signal and outputs an RGB color image signal, an analog signal processing circuit that amplifies the image signal output from the image sensor and adjusts the signal level and the gradation of the image, and the like. An analog-to-digital conversion circuit for converting a signal into digital-valued image data, and a digital signal processing circuit for interpolating and generating pixels as a still image based on the converted image data are included. The digital signal processing circuit has an electronic zoom function for enlarging and reducing an image size. This zoom function is used, for example, when adjusting a photographed image to a desired size while viewing the image displayed on the monitor display device 22, when performing pixel interpolation processing or pixel thinning to a desired ratio in accordance with the size of the display image. This is a function to perform processing. By this zoom processing, the size of an image to be shot or an image after shooting can be adjusted to a desired size. Further, a lens disposed on the front surface of the image sensor may have a zoom lens configuration.
The output of the digital signal processing circuit constitutes the output of the imaging unit 12 and is connected to the bus 14. Image data output from the imaging unit 12 is stored in a memory (RAM) 18 connected via a bus 14.
Is stored under the control of the control unit (CPU) 16.

A display control unit 20 outputs image data and graphic data input from the bus 14 and various character information and icons generated by the control unit 16 in a format corresponding to the output destination. Interface. A monitor display device 22 is connected to the output of the display control unit 20, and the display control unit 20 outputs image data in RGB line order and generates a drive signal for driving the monitor display device 22. As the monitor display device 22, a thin display panel such as a color liquid crystal display is advantageously applied. For example, as shown in FIG.
On the display screen 300, a moving image or a still image enabling the angle of view to be adjusted when photographing an actual scene is displayed in the first display frame 302 according to the photographed image data.
In the display screen 300, a second display frame 304 on which a planar map or a three-dimensional map is displayed is set. These display frames can be moved within the screen 300 in response to an operation on the operation unit 32, and portions where the display positions overlap each other are combined and displayed as images. By utilizing this, for example, when photographing a building group of each block, it is possible to make adjustments while checking the angle of view, size, and the like. In addition, a three-dimensional map corresponding to the current position and a camera 10
When the moving image captured by the camera is displayed in each frame, the consistency between the map display and the current position can be confirmed by looking at the screen display of the camera 10 at hand. Further, the size of these frames can be arbitrarily enlarged or reduced to the size of the display screen 300, for example, and any of the display frames may be displayed with priority.

The 3D processing unit 24 includes a second display frame 304
Is generated based on the map information. The 3D processing unit 24 generates a block 306 indicating the outline of these structures corresponding to a plurality of structures existing in each block on the three-dimensional map. Although not shown in FIG. 3 for simplicity, the 3D processing unit 24 maps the processed image of the photographic image on the surface of the block 306 as a texture. In the present embodiment, in a three-dimensional map displayed on the second display frame 304, a block in which a plurality of structures such as buildings are built is represented by a three-dimensional three-dimensional block formed by combining polygons, and the actual plurality of structures are displayed. Image information obtained by photographing and recording an object is mapped as a surface image of the block. Note that a plurality of blocks may be set for one block in accordance with the situation, not limited to the actual block.
These block-corresponding polygon data and image data representing the surface image of the building are supplied by the memory cards 27 and 26 of the external recording medium detachably attached to the camera 10 and stored in the map memory 30. . In addition, the camera 10 has a townscape scene information creation function of photographing such a townscape, creating image data corresponding to the above-described blocks, and storing the image data in the memory card 26.

The 3D processing unit 24 has a geometric engine, a raster engine, a texture memory and the like for three-dimensionally displaying various information, and creates a three-dimensional map image representing blocks and roads set for each block. . The 3D processing unit 24 performs a process of creating a three-dimensional map image and a process of pasting image data as appearance information of each block to a corresponding block. The 3D processing unit 24 has a function of transforming an image represented by image data by, for example, affine transformation in accordance with a viewpoint serving as a reference when displaying a stereoscopic map, and converts a texture image to be pasted on a plurality of building surfaces. Correct the shape according to the viewing angle.

The recording control unit 28 is a controller that controls writing and reading of information to and from the memory cards 26 and 27. In the present embodiment, the recording control unit 28 includes two connection units that can simultaneously load two memory cards 26 and 27. Have. These memory cards 26 and 27 are, for example, NAND flash EEPROMs.
An information recording medium having a semiconductor memory device such as a smart media using an OM or a PC card having a memory backup function. Describing the logical format of the storage area, the memory card 26 for storing captured image data is formed, for example, as shown in FIG. The image recording memory card 26 includes a header area 200 and an image data area 202. The header area 200 has attribute information 20 indicating that it is a memory card for storing image data.
4, directory information 206 of each file, and a memory allocation table (MA
T) 208 and an image management area 210 in which information relating to image data recorded in the image data area 202 is recorded. The image management area 210 is an area for managing each of the image files 0001 to nnnn in the image data area 202 for each frame of each image. In addition to the image management information, the position information indicating the place where the image was shot, and the structure information such as the block number and the block name indicating the correspondence between the shooting direction and the block in the three-dimensional map are included.

The memory card 27 for storing map information differs from the format shown in FIG. 2 in that the memory card 27 has a storage area for storing map information instead of the image data area 202. Attribute information indicating that the recording medium is stored and map management information are recorded in the header area 200. The memory card 27 includes map data capable of generating a three-dimensional image for each block that three-dimensionally shows the block shape as one or a plurality of groups, roads and passages that can be a traversable road and can be a moving route, and sidewalks. And road information indicating information about the network model of nodes and links, and additional information such as a street name, an intersection name, a town name, and the name of each structure is linked and stored in the road information. You. The additional information includes, besides the name of the building, a line name, a station name, and the like related to each transportation such as a train and a bus.

The map data includes position information (longitude / latitude) of structures such as roads and buildings and a planar map representing the shape of a block, and stereoscopic information representing a plurality of structures in each block as blocks of a three-dimensional stereoscopic image. And include each file. Each block is given a block number along with its name. The road information is classified into roads that can be walked by pedestrians, roads that cannot be walked, and roads that can pass vehicles. When performing route guidance indicating the route from the starting point to the destination, the user moves A route to the destination is calculated using a walkable road selected according to the form.
When traveling by vehicle, the control unit 16 performs a route calculation to the destination using a road that can pass through the vehicle.

Further, information on a possible departure place and destination is stored in the memory card 27. For example, items related to destinations, such as print service stores and favorite restaurants, include store information such as the store name, address, telephone number, URL, service content, business hours, holidays, and location information of the store photo. A destination is searched using these item contents as keys, and a desired destination is selected. These pieces of information are read from the memory card 27 and stored in the map memory 30 provided in the camera 10. Also,
The map information can be updated at any time by rewriting the storage contents of the memory card 27, and the camera 10 can refer to the latest information.

In this embodiment, a memory card having a semiconductor memory element is used as an information recording medium for storing image data and map information. However, the shape is not limited to a card type, and other storage media may be used. A medium, for example, a magnetic disk or a magneto-optical disk for magnetically and optically storing information, or an information recording medium such as an optical card memory may be adopted. Further, in the present embodiment, the map information and the image information are respectively recorded on the two memory cards. However, the present invention is not limited to this. For example, the image information and the map information are stored on one memory card according to the storage capacity. May be stored. Further, map information may be supplied to the map memory 30 of the camera 10 through an information storage medium such as a CD-ROM or a DVD.

Returning to FIG. 1, the recording control unit 28 checks the attribute information of the memory card inserted in any one of the slots, and reads out information according to the attribute of the memory card. Upon detecting the memory card 27 on which the map information is recorded, the recording control unit 28 transfers the data required for the map display to the map memory 30 and stores it in response to the control of the control unit 16. Further, the recording control unit 28 allows the memory card 27 and the map memory to update the point information of the destination created by the control unit 16.
Record at 30.

Upon detecting the memory card 26 on which the photographed image is recorded, the recording control unit 28 stores the image data to be mapped on the surface of each block on the displayed three-dimensional map on the memory card.
It is read from 26 and transferred to the map memory 30. Further, the recording control unit 28 writes image data of a building or the like photographed corresponding to the block on the three-dimensional map into the memory card 26 together with its block number and name, and registers the image data corresponding to the block. Further, the recording control unit 28 has a function of compression-encoding the image data supplied from the bus 14, and records, for example, still image data compression-encoded by the JPEG method on the memory card 26. The recording control unit 28 is a memory card
The data read from 26 is expanded and decoded, and the decoded image data is output to the bus 14 and stored in the RAM 18.

As described above, in this embodiment, the surface image of the block corresponding to the block formed by polygons is
Is used, and the image data is compression-encoded and recorded on the memory card 26 or the like. The recording control unit 28 reads the image data of the blocks necessary for display from the memory card 26, sequentially stores the read image data in the map memory 30, and deletes the image data unnecessary for display from the map memory 30. As a result, each memory storage capacity is reduced, and the processing load is reduced.

An input interface (I / F) unit 34 is an operation unit
32, a GPS 36, and a direction sensor 38, and supplies information output from these to the control unit 16 via the bus 14.
The operation unit 32 is provided with a release switch for photographing the scene, a menu display key for selecting various functions, and a cross key (control key), and I / F for operating information according to the operation. Control unit via unit 34 and bus 14
Notify 16. The operation unit 32 further includes a mode selection dial for selecting one of a shooting mode, a movement mode for performing route guidance, a playback mode, and a registration mode for registering a photographic image corresponding to a block. The control unit 16 is notified. The operation unit 32 may be provided with buttons for inputting various character inputs.
For example, various kinds of information obtained in a town are stored in correspondence with a displayed map, a related object scene is photographed and its image information is stored, and the information is stored in a memory card.
26 and 27 can be newly registered as future destinations.

The position measurement processing unit (GPS) 36 is a processing unit that detects the current position of the camera 10 and outputs position information indicating the current position. The position positioning processing unit 36 in the present embodiment includes:
Global positioning system including GPS antenna and GPS receiver is applied, and multiple GPS orbiting the earth
Receives and demodulates the spread spectrum modulated L1 charged wave transmitted from the satellite, and calculates the position coordinates of the receiving point based on the C / A code and the navigation message transmitted from a plurality of satellites. Advantageously, a differential GPS that receives correction data indicating error information and corrects the calculated position coordinates
Relative positioning such as a method is applied. The position positioning processing unit 36
Depending on the receiving location, e.g. 139 ° 43'xx east longitude, north latitude
Data showing longitude and latitude information of 35 degrees 39 minutes xx seconds "E 139.43.x
x "and" N35.39.xx "and time data indicating the reception time.If the current position is already known or the radio wave from GPS satellites cannot be received effectively, the operation A character such as an address is input to the unit 32, a location is specified on a displayed map, and the information is converted into position information.In this case, the control unit 16 converts the input information into a map memory. The information is converted into longitude and latitude information on the map data stored in 30. The control unit 16 has a function of storing the area of the section where the GPS information could not be obtained in the RAM 18 and displaying the area on the map. This gives the operator information to infer the current position.

The azimuth sensor 38 is a detection device disposed inside the camera 10 for detecting the attitude of the camera 10. The azimuth sensor 38 in the present embodiment recognizes the direction of the camera 10 relative to magnetic north detected by terrestrial magnetism, and generates azimuth information of 8 to 16 directions indicating the front direction of the imaging lens. Further, the direction sensor 38 includes an angle sensor that detects the inclination of the camera 10. After detecting the initial azimuth, the azimuth information may be updated by, for example, detecting a change in the direction using an acceleration sensor or the like. Alternatively, a small magnetic compass may be provided in the camera 10, and information corresponding to the pointing direction may be manually input to the camera 10 from the operation unit 32. In addition, it is preferable to detect the moving route of the camera 10, consider the current traveling direction recognized from the position information of each passing point as the azimuth of the camera 10, and use this azimuth information when guiding the route.

The map memory 30 is a built-in memory for storing map information read from the memory card 27. The map memory 30 stores map data of a map file and a three-dimensional information file, road information, and additional information. In this embodiment, polygon data forming a plurality of structures existing in a predetermined block in a town in one block is read from the memory card 27 and registered in the map memory 30, and the polygon data is used for each block. A block representing a schematic shape of a plurality of existing structures is created. When not all the storage information of the memory card 27 is stored in the map memory 30 at once, it is preferable to sequentially store information on a portion related to display and information on the periphery thereof. When the map memory 30 is configured to have a large storage capacity, for example, a non-volatile flash memory is used as the map memory 30,
Any map information stored in the memory card 27 may be transferred to the map memory 30 in advance. The information stored in the map memory 30 is read out in response to the control of the control unit 16, and is supplied to the 3D processing unit 24 and the display control unit 20.

The map memory 30 has a memory card 26
It has a storage area for storing image data and structure information read out from the corresponding block, and a storage area for storing point information.

In a main storage unit (RAM) 18 having a random access memory to which information can be read and written, a control program for defining a processing procedure in the control unit 16 is stored in a ROM (R) (not shown).
eReadOnly Memory) and stored in a predetermined storage area. The main storage unit 18 further includes a storage area that temporarily stores various information and is used as a work area, in which image data and various information supplied from the input I / F 34 are temporarily stored. . The memory for storing the control program is constituted by, for example, an EEPROM, and the control program recorded on an external recording medium such as a memory card is stored in the EEPROM.
By writing to the OM, the control program itself can be
You can register for 10 or update the program.

The control unit 16 includes a photographing mode for recording image data obtained by photographing on the memory card 26, a reproducing mode for reading out image data stored on the memory card 26, and reproducing and outputting the image data. The operation unit 32 is provided with either a registration mode in which is registered in correspondence with a block formed on a three-dimensional map, or a movement mode in which route guidance and movement simulation are performed from a departure point (current location) to a destination point. Each part is set according to the set state of the mode setting dial to be set, and each part is controlled by a processing control procedure according to a control program corresponding to each setting mode.

More specifically, when the release button is depressed in the photographing mode, the control unit 16 generates a control signal for causing the image pickup unit 12 to pick up an image of the object scene, and outputs a control signal 1 from the image pickup unit 12.
The image signal for the frame is stored in the image storage area of the RAM 18. The control unit 16 reads out the image data, transfers the image data to the recording control unit 28, and performs compression encoding. Further, the control unit 16
In this shooting mode, the object scene is continuously imaged, and the image data of the continuous frame is transferred to the display control unit 20.
A finder function for displaying a moving image on the monitor display device 22 is provided. The photographer can perform photographing adjustments such as the angle of view and framing of the object scene and the focus adjustment while checking the displayed image.

In the reproduction mode, the control section 16 outputs to the recording control section 28 an instruction signal for designating and reading desired image data recorded on the memory card 26. The image data read by the recording control unit 28 is temporarily stored in the RAM 18. The control unit 16 repeatedly reads out the image data for one frame stored in the RAM 18 and transfers it to the display control unit 20.

The control unit 16 in the registration mode associates the image data obtained by photographing the structure of each block with the blocks of the three-dimensional map, and registers the building surface image data in the memory card 27 and the map memory 30. Specifically, the control unit 16
At the position where the surface of the building is to be photographed, the position information and the azimuth information of the camera are acquired, and a three-dimensional map image corresponding to the information is created and displayed on the second display frame 304. At this time, the control unit 16 displays blocks corresponding to the blocks on the map and allows the operator to select a desired block. The control unit 16 recognizes the block number of the selected block, causes the operator to shoot an image corresponding to the selected block, and temporarily stores the image data in the RAM 18. Control unit 16
Causes the image data stored in the RAM 18 to be stored in the memory card 27 and the map memory 30 together with the corresponding block number, position information and direction information. At the time of shooting, the control unit 16 supplies an image represented by the image data to the display control unit 20 and continuously converts the image of the object scene into a first display frame 302.
To allow the operator to adjust the shooting range and the angle of view.

The operator can adjust the angle of view of the camera 10,
By adjusting the image size of the captured image, of the plurality of structures corresponding to the selected block, the structure located almost at the center is set as the center of the scene, the front side and side surface of the group of structures A plurality of structures corresponding to the block are photographed from a plurality of surfaces such as sides. The control unit 16 converts the image data output from the imaging unit 12 by such a shooting process into RA data.
Store it temporarily in M18. The control unit 16 corrects the image represented by the image data obtained by the photographing into a normal shape that compensates for the characteristics of the imaging lens. Specifically, the control unit 16
The image data output from the imaging unit 12 is supplied to the 3D processing unit 24 and the image correction is instructed to the 3D processing unit 24. 3D
Upon receiving this instruction, the processing unit 24 performs distortion conversion on the image data of the structure image deformed by the lens characteristics so as to conform to the square or rectangular shape of the corresponding block. This distortion correction process is performed not only in the vertical direction but also in the horizontal direction. Even if the shooting position does not correspond to the center position of the corresponding block, shooting was performed from a shooting position orthogonal to the center of the block surface. The image data similar to the above is obtained.

The control unit 16 may recognize the ratio between the size of the block displayed on the three-dimensional map and the size of the image data to be pasted on the block. That is, when image data of an original size close to the number of pixels of the image sensor is used as a pasted image, a ratio corresponding to the display image displayed on the monitor and the size of the original image data is recognized. When the image data is pasted to the block, the image size can be changed in accordance with the ratio. In this case, when displaying the block at the position corresponding to the close distance and the surface image thereof, a fine image can be obtained. In this case, a large storage area for storing image data is required. For this reason, in the present embodiment, when displaying an image on the compact monitor display device 22 of about several inches, the image size of the front surface image to be attached to the block is changed in accordance with the block, and the size-changed image data is displayed. On the block surface. In this case, since the image data corresponding to the block size is used, the above-described ratio information does not need to be used. Thus, in the present embodiment, the storage capacity of each memory for storing image data is saved, and the processing load is reduced.

The controller 16 associates the azimuth information supplied from the position measurement processor 36 with the block number and name of the photographed block in the image data thus normalized with respect to the block. In the memory card 27 and the map memory 30. In addition, this image data may be stored in the memory card 26 in advance together with the azimuth information and the block number, and a process of corresponding to the corresponding block, the azimuth information and the block number may be performed later. That is, without performing the imaging process in the registration mode, the image data corresponding to each block may be obtained from an external storage medium such as the memory card 26, and the image data corresponding to the block may be registered.

The control unit 16 stores the current position, image data and azimuth information corresponding to each block number into the 3D processing unit 24, respectively.
Transfer to The 3D processing unit 24 is controlled by the control unit 16
For each block currently displayed, determine the viewpoint when displaying the three-dimensional map based on the position information and the azimuth information, add the deformation process and light / dark process according to the viewpoint to the image data, and process the processed image data. Is mapped to the block surface of the corresponding block number. In this case, the control unit
The detection unit 16 detects an operation state of the operation unit 32 and performs a correction process for finely adjusting the state of the pasting position in accordance with the operation state. By such control and processing in the registration mode, for example, as shown in FIG.
306 and the correspondence between the image data 600 and 602 are held,
It is displayed on the monitor in the second display frame 304. These operations are performed for each desired block to create a landscape on a three-dimensional map. When mapping the image data to each block in the movement mode, the control unit 16 specifies each image data according to the correspondence.

The control unit 16 converts the image data to be pasted into each block into the 3D processing unit 24 together with the change of the viewpoint on the three-dimensional map.
, And the display image of the three-dimensional map is updated. Each time the current position changes, the control unit 16 causes the 3D processing unit 24 and the imaging unit 12 to perform a deformation process and an image size correction process according to the viewpoint of the stereoscopic map. As described above, the control unit 16 generates the viewpoint information according to the current position, sets the viewpoint on the three-dimensional map, and displays the map image following the change in the current position. Control unit
The unit 16 can generate arbitrary position information instead of the current position, for example, in response to an operation state of the operation unit 32, and can create a three-dimensional map from a viewpoint corresponding to the operation. In this case, the control unit 16 can perform a movement simulation for displaying a landscape or the like on the movement route by sequentially updating the three-dimensional map display according to the result of the route calculation to the destination.

In the registration mode, information on the destination can be set. The control unit 16 selects a starting point, a waypoint, and a destination point from a plurality of points, and stores detailed information at each point in the memory card 27 or the map memory 30. Register according to the input information from.

In the movement mode, the control unit 16 generates a three-dimensional map display image at the current position and an image representing the surroundings of the current position, and displays these images. Further, the control unit 16 selects a destination point according to an operation on the operation unit 32,
Calculate the route to travel from your current location to your destination,
The route guidance information representing the route is generated.

More specifically, the control unit 16 calculates an appropriate route for reaching the destination from the selected destination point and the current position, displays the route on a plane map, and sequentially changes the route. Instruction information indicating a route guidance instruction is generated according to the current position. The control unit 16 supplies the instruction information to the display control unit 20, and the display control unit 20 according to the instruction information,
For example, an image representing a straight-ahead instruction, a left turn, a right turn, or the like in the vicinity of an intersection is generated, and is displayed on the display screen, for example, in the second display frame 304. The control unit 16 determines whether or not it is near an intersection or the like according to the moving speed of the camera 10 itself, and displays the instruction image at a timing farther from the intersection when moving at high speed. Good. Note that this instruction information may be audibly output as voice.

The control unit 16 searches for a route from the current location to the destination by route calculation, and performs route guidance according to the current location that changes as the camera 10 moves. Control unit 16
Calculates the shortest route based on the road information, for example, by Dijkstra's algorithm. The control unit 16 displays the obtained shortest route on a map corresponding to the road information, and generates route guidance instruction information for instructing a moving direction according to a current position such as an intersection. As shown in FIG. 4, when performing a route guidance of turning right at an intersection 404 in front while heading from a starting point 400 to a destination point 402, the control unit 16 calls, for example, “next right turn” for instructing a right turn. Instruction information such as icons representing characters and arrows
500 is generated at the display point 406 and the instruction information is displayed on the second display frame 304. FIG. 5 shows a display example. In this figure, illustration of the surface image of the block is omitted for simplification of the figure. At this time, the control unit 16
Calculates the travel speed from the distance from the current position to the destination and the change in the current position, predicts the expected arrival time based on the travel speed and the distance to the destination, and displays the prediction result together with the instruction information Good to do. In the case where there are a plurality of shortest routes, for example, route guidance information such as adopting a route with as few turns as possible or passing through a street that is easy to pass using a boulevard where a lot of point information exists is used. It is good to generate. In these cases, the route may be longer than the shortest route to the destination. When the current position deviates greatly from the route indicated by the route information, the control unit 16 detects this, performs a route search from the latest current location to the destination again, and provides guidance information and instruction based on the updated latest route. The information is regenerated, and the fact that the route has been recreated deviating from the initial route is displayed on the monitor.

The control unit 16 reads out the map data stored in the map memory 30 and the image data to be pasted on the displayed block according to the viewpoint to be displayed, and reads out the 3D processing unit 24.
Control of generating a stereoscopic map image to be supplied to the computer.

When the control unit 16 performs a process of pasting one frame image of a plurality of buildings into one block, the control unit 16 captures a plurality of frames according to, for example, a block extending horizontally. It is preferable to have a function of connecting the images of the frames to form a panoramic image by combining the images, thereby creating a single-frame image in a wider range. In this case, one frame of the wide image created by the combining process can be pasted on a horizontally formed block according to the block. As a result,
A large block is divided into a plurality of blocks, and the blocks can be represented by a small number of blocks without generating a plurality of blocks corresponding to each of the divided blocks, or by reducing the number of blocks. The load is reduced.

The control unit 16 in the movement mode creates a three-dimensional map image corresponding to the current position in the same manner as the image mapping processing in the registration mode, and generates image data mapped to blocks on the map in the 3D processing unit 24. Let it. The control unit 16 generates viewpoint information for displaying the three-dimensional map based on the current position and the azimuth information. Specifically, the control unit 16 determines, based on the azimuth information and the position information detected by the azimuth sensor 38 and the position measurement processing unit 36, the azimuth of the camera 10 and the current position where the camera 10 is present. And the recognition result is supplied to the 3D processing unit 24. 3D
The processing unit 24 creates a three-dimensional map image corresponding to the current direction and the current position according to the information, and supplies the created three-dimensional map data to the display control unit 20. The 3D processing unit 24 creates three-dimensional map data according to the change of the current position. The display control unit 20 performs data processing such that a three-dimensional map image corresponding to the three-dimensional map data is displayed on the second display frame 304 of the monitor display device 22. The display control unit 20 includes a frame memory for sequentially storing display data transferred from the bus 14, reads information stored in the frame memory, and outputs the information to the monitor display device 22.

The communication control unit 50 in the present embodiment has a function of receiving correction data used in the position positioning processing unit 36, and transmits various kinds of information to stores and the like at the destination.
Conversely, it has a function of receiving map information and the like. For example, the communication control unit 50 receives a request from the control unit 16, accesses a network that supplies correction data, and receives correction data (error information) for correcting a positioning result. In addition,
The communication control unit 50 may be configured to receive the error information transmitted by an FM multiplex wave or the like. Communication control unit 50
Supplies the error information included in the received signal to the position positioning processing unit 36. The communication control unit 50 is connected to a wireless communication line for connecting to a network such as the Internet, for example, to search for information on each destination according to a URL, or to search for information on a homepage or the like established at a store or the like. It is possible to access the contents and display them on a monitor or register them as point information.

The communication control unit 50 has a function of transmitting information indicating the recognized current position to a remote terminal device. In this case, the camera 10 is stored in a non-volatile memory (not shown).
For example, the controller 16 registers unique information indicating which GPS device is used, and when the camera 10 is accessed from a remote terminal device, the controller 16 The information, the current position information, and the information indicating the latest travel route can be transmitted to the terminal device. With such a configuration, it is possible to determine where the camera 10 is located from a terminal device distant from the camera 10, and to find out, for example, when the camera 10 is lost.

The operation of the camera 10 according to this embodiment having the above configuration will be described with reference to FIGS. First, if the registration mode is set when no map information is stored in the map memory 30, map information such as map data, road information and additional information and point information stored in the memory card 27 are read out. Then, the data is transferred to the map memory 30 via the bus 14 and stored.

Thus, in the registration mode in which the map information is stored in the map memory 30, the image data picked up and processed by the image pickup unit 12 is transferred via the bus 14 and the RAM 18 as shown in FIG. The data is sequentially transferred to the display control unit 20. In response to the drive signal supplied from the display controller 20, an image represented by the image data is sequentially displayed on the first display frame 302 of the display screen of the monitor display device 22 (step 900).

Next, the routine proceeds to step 902, where the position information measured by the position positioning processing unit 36 is transmitted to the control unit via the bus 14.
When the position information is input to the map data 16, the position information is recognized as the current position. In the subsequent step 904, the map data in the map information corresponding to the position information is read from the map memory 30 and is read via the bus 14 via the 3D processing unit Transferred to 24. When the azimuth information detected by the azimuth sensor 38 is recognized by the control unit 16 (step 906), the 3D processing unit 24 generates a map representing a three-dimensional map image based on a viewpoint corresponding to the current position. Image data is generated. The generated 3D map image data is 3
When transferred from the D processing unit 24 to the display control unit 20, a first display frame 302 is set on the display screen of the monitor display device 22, and an image represented by image data obtained by imaging is displayed on the first display frame 302. And a second display frame
304 is set and a three-dimensional map image is displayed thereon (step 910).

The processes of steps 900 to 910 are repeated, and the position and orientation of the camera 10 are adjusted and zoom processing is performed by the operator, and the structure of the block adapted to the block of the map image selected as the processing target Are adjusted so as to be appropriately displayed in the first display frame 302 corresponding to the blocks displayed in the second display frame 304.

When the fine-tuning of the image to be displayed corresponding to the block is completed in this way, the release button is pressed, and the control unit 16 recognizes the detection information indicating the ON state of the release switch, the imaging unit 12 An instruction to process a still image is output. When the imaging instruction is input to the imaging unit 12, the electric charge that has been photoelectrically converted by the imaging element is read out as an electric signal, and various signal processing is performed on the imaging signal, and the processed image data is transmitted to the bus. Stored in RAM 18 via 14. This image data is subjected to distortion characteristic removal processing and image size normalization processing in the 3D processing unit 24, and the normalized image data is stored in the memory card 27 and the map memory 30 (step 914). . Along with the image storage, the block number and name of the corresponding block, and the position information and the azimuth information are stored corresponding to the image (step 916).

As described above, when the image data to be mapped to each block is registered in the map memory 30, when displaying a three-dimensional map image, an image to which a block number corresponding to each displayed block is assigned. Data can be pasted and displayed. This situation will be described by taking the route guidance in the movement mode as an example.

When the movement mode is set as shown in FIG. 10, a destination selection menu image for selecting a destination is displayed on the monitor display device 22 in step 1000.
A location corresponding to the registered location information or a location indicated by the map data is selected as the destination. Next, when the current position is confirmed in step 1002, the shortest path calculation processing in step 1004 is performed. In the control unit 16, based on the position information of the selected destination point and the current position, a route calculation according to the user's movement mode is performed using the road information stored in the map memory 30, and the route between these points is calculated. For example, an appropriate route such as the shortest route is calculated.

When the route is calculated, the process proceeds to step 1006, in which route guidance information (navigation information) for displaying a direction instruction, reference information, and the like on the calculated route.
Is generated and stored in the RAM 18. The display point on the calculated route is set in this information, and when the current position approaches the display point in a later process, the instruction information is displayed and output on the monitor display device 22.

At step 1008, the control unit 16 acquires the position information and the azimuth information, and sets the viewpoint on the three-dimensional map. In step 1010, it is determined whether or not the viewpoint information has been changed after being set. If it is determined that there has been a change, the process proceeds to step 1012, where map data and road data corresponding to the current position information are read from the map memory 30, and in step 1014, the three-dimensional map data is Created by the processing unit 24.

In step 1016, the surface image data corresponding to each block created on the stereo map data is
It is read from the map memory 30 based on each block number and supplied to the 3D processing unit 24. In step 1018, these block-corresponding image data are pasted on the corresponding blocks, respectively, and a mapped stereoscopic map image is created according to the viewpoint information. In step 1020, it is determined whether or not the current position is a display point for displaying the route guidance instruction information according to the current moving speed. If the current position is not a display point, the process directly proceeds to step 1024. The image indicating the instruction information is read from the map memory 30 and synthesized with the stereoscopic map image, and the process proceeds to step 1024.
Proceed to. In step 1024, the created three-dimensional map image data is transferred from the 3D processing unit 24 to the display control unit 20, and the three-dimensional map image is displayed on the second display frame 704 of the display screen of the monitor display device.

Thereafter, returning to step 1008, the current position and azimuth information are obtained. If it is determined in step 1010 that there is no change, the flow advances to step 1020 to determine whether or not the display point exists. You. If it is confirmed in step 1010 that the change has been made, steps 1010 to 10
After the processing at 18, the process proceeds to step 1020. If it is determined in step 1020 that the point is a display point, an image indicating the route guidance instruction information is combined and displayed on the three-dimensional map.

In this way, the three-dimensional map image displayed according to the current position and the traveling direction is updated and displayed on the monitor. For example, instruction information indicating the traveling direction such as going straight, turning right or turning left at an intersection is displayed. Monitor display at point. In step 1008, if it is detected that the acquired position information is significantly different from the route calculated in the route calculation, the process returns to step 1004 to recalculate the route from the current location to the destination, and obtain the same. A route guidance process is performed by a new route, and a monitor display device indicates the fact.
Displayed on the 22 display screen. This is when you switch to a different travel mode from the current travel mode on the way, and use a means of transportation such as a bus or railroad that travels on a route different from the calculated route, and get off at an appropriate bus stop or station. It is effective in such as.

The above description is an example in which the image display is updated according to the change in the current position. However, the movement simulation can be performed by forcibly changing the position information indicating the current position. For example, by recognizing the calculated position information corresponding to the movement route as the current position and updating the position information at predetermined time intervals along the movement route, even if the actual current position has not changed. The state of the change in the scenery to the destination can be displayed on the monitor display device 22. By selecting the position information to be changed arbitrarily according to, for example, an operation, the landscape at the selected position can be displayed on the monitor display device 22. By utilizing this, it is possible to display a three-dimensional map indicating a desired block, shift to the registration mode, retake a surface image of the structure group to be pasted on the block, and update the image data to be pasted as a texture.

As described above, this camera 10 can be used, for example, to find a favorite shop or the like and return to the favorite place again. Objects and the like can be photographed and recorded according to the digital map, and the recorded image can be used to create a three-dimensional stereo map showing the surrounding situation. Then, at the next visit, the recorded information is synthesized and displayed on the structure on the three-dimensional map, and further information for route guidance is displayed. The operator walks and moves with a vehicle while checking the display. Also in this case, it is possible to register and update the newly photographed image data as information representing the surface of the structure on the three-dimensional map. By visually recognizing the three-dimensional map created in this way and the actual landscape, it is possible to intuitively confirm that the landscape from the current position matches the landscape shown in the three-dimensional map, You can reach the shop with confidence.

If the directions to the destination are complicated, or similar buildings are lined up in the block, the display of the conventional map and the current position shows the actual landscape at the current location and the conventional map. Because the connection with the map was sparse, the gap between the actual scenery obtained at my current position and the map display was large. And, without a doubt, he could not feel that he was in this place, and as a result, for example, a person who came to an unknown place could not fully eliminate his anxiety.

However, in the camera 10 of this embodiment, not only the present position display on the map but also the display image on the three-dimensional three-dimensional map matches the actual scene visually.
Since the state of the surface of the structure can be recorded and recorded and can be displayed on a 3D frame in a composite manner, and furthermore, a three-dimensional structure on a three-dimensional map is represented by an actual image taken during the travel to the destination, as shown in FIGS. 7 and 8. Landscape at each position 700 to 712 until reaching the destination 800 or
While checking 810 one by one, a building or the like that will be seen next with the movement can be checked on the three-dimensional map displayed on the second display frame 304. In these figures, the surface images pasted on each block are not shown for simplicity, but in actuality the color and shape of structures such as buildings, as well as the brightness and texture, etc. Ten
It is possible to easily determine whether or not the object matches the structure in the three-dimensional map displayed in. As a result,
Not only during the second and subsequent visits to the destination, but also during the first visit, by creating necessary information for the camera 10, the actual scenery and the displayed stereoscopic image are displayed in agreement. So you can head to your destination with peace of mind. In addition, even if you do not actually go to the place again, you can memorize the travel route from one point to another, simulate the scenery on that route and reproduce it, and just walk around that place You can get the scenery that you are.

In the present embodiment, such a camera 10 expresses a texture to be mapped to a block displayed on a three-dimensional map, for example, by a plurality of image groups representing windows, entrances and the like constituting one building. Instead of forming a three-dimensional block in each predetermined block or a certain area obtained by dividing the block, and using the image data corresponding to the block as a texture, the landscape display can be performed regardless of a large-scale configuration. This makes it possible to create and display a simple three-dimensional stereoscopic map image in a compact and portable configuration.

[0065]

As described above, according to the present invention, based on the first image data and the block data representing the plane image of the block in which a plurality of structures are simplified three-dimensionally, the block plane image is obtained. Is created by pasting an image corresponding to the first image data corresponding to the first image data, so that a three-dimensional image can be expressed with a simple configuration. Can be registered or reproduced and displayed on a portable simple device.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a camera to which the present invention has been applied.

FIG. 2 is a diagram showing a storage format of a memory card.

FIG. 3 is a diagram showing a screen display example.

FIG. 4 is a diagram showing a state of route guidance.

FIG. 5 is a diagram illustrating an example of route guidance instruction information.

FIG. 6 is a diagram showing a state of pasting a front surface image.

FIG. 7 is a diagram illustrating an example of each display point on a movement route.

FIG. 8 is a diagram showing an example of a screen display on a moving route.

FIG. 9 is a flowchart showing processing in a registration mode.

FIG. 10 is a flowchart illustrating a process in a movement mode.

[Explanation of symbols]

 10 Camera 12 Imaging unit 16 Control unit (CPU) 18 Memory (RAM) 24 3D processing unit 28 Recording control unit 30 Map memory

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) G06F 15/66 450 F term (Reference) 2C032 HB11 HC01 HC23 HD07 5B050 BA09 BA17 EA13 EA19 5B057 AA16 BA02 CD11 CE08 9A001 HH29 HH32 JJ11 JJ72

Claims (18)

[Claims] 1. An image processing apparatus for processing a three-dimensional state of a structure so that the three-dimensional state of the structure can be displayed, the apparatus comprising: block creation means for creating block data indicating a plurality of adjacent structures as three-dimensionally simplified blocks A storage unit that stores first image data representing a surface image of the block corresponding to the block; and a block based on the first image data and the block data stored in the storage unit. An image processing means for pasting an image corresponding to the first image data corresponding to the block as a surface image of the second image data to create second image data three-dimensionally representing the plurality of structures; and the image processing means And an image output means for outputting the second image data processed by the image processing means to a display means. 2. The image processing apparatus according to claim 1, wherein the block creating unit creates block data that forms the block by using polygons, and the image processing unit creates the first image data corresponding to the block. An image processing apparatus, wherein is attached to a polygon surface of the block. 3. The image processing apparatus according to claim 1, wherein the apparatus is connected to an output of an image pickup means for picking up an image of the object scene, and the storage means is picked up and processed corresponding to the block. An image processing apparatus storing the first image data. 4. The image processing apparatus according to claim 3, wherein the apparatus includes the imaging unit, and registers image data obtained by imaging in the storage unit in correspondence with the block. apparatus. 5. The image processing apparatus according to claim 1, wherein the block creating unit creates block data indicating a block represented by the map information as a block, and the storage unit captures the block corresponding to the block. An image processing apparatus for storing image data that has been processed in a manner as described above. 6. The image processing apparatus according to claim 5, wherein said block creating means creates block data indicating each of said blocks by a plurality of blocks. 7. The image processing device according to claim 5, wherein the device includes a storage unit that stores the map information, wherein the image processing unit stores the map information and the second image data. An image processing apparatus, wherein the image output means outputs an image represented by the stereoscopic map image data so as to be displayable. 8. The image processing apparatus according to claim 7, wherein the apparatus includes a route calculation unit that calculates a route to a destination based on the map information, wherein the image processing unit performs An image processing device for creating a three-dimensional map image according to the image processing device. 9. The image processing apparatus according to claim 8, wherein the apparatus includes position input means for inputting position information of the apparatus, wherein the image processing means generates a three-dimensional map image corresponding to the position information. An image processing apparatus characterized by being created. 10. An image processing method for displaying a three-dimensional state of a structure so as to be able to be displayed, the method comprising: a block creation step of creating block data indicating a plurality of adjacent structures as three-dimensionally simplified blocks. And a storage step of storing first image data representing a plane image of the block corresponding to the block; based on the first image data and the block data stored in the storage step An image processing step of pasting an image corresponding to the first image data corresponding to the block as a plane image of the block to create second image data three-dimensionally representing the plurality of structures; An image output step of outputting the second image data processed in the image processing step to a display means. 11. The image processing method according to claim 10, wherein the block processing means generates block data for forming the block by using polygons, and wherein the image processing step includes a first image data corresponding to the block. Is attached to the polygon surface of the block. 12. The image processing method according to claim 10, wherein the method is connected to an output of an imaging unit that captures an object scene, and the storing step is performed by capturing an image corresponding to the block. An image processing method characterized by storing the first image data. 13. The image processing method according to claim 12, wherein said method registers image data obtained by imaging by said imaging means in correspondence with said blocks. 14. The image processing method according to claim 10, wherein the block creating step creates block data indicating a block represented by the map information as a block, and the storing step captures the block corresponding to the block. An image processing method characterized by storing image data that has been processed. 15. The image processing method according to claim 14, wherein the block creating step creates block data indicating each of the blocks by a plurality of blocks. 16. The image processing method according to claim 14, further comprising a storing step of storing the map information, wherein the image processing step includes the step of storing the map information and the second image. An image processing method, wherein three-dimensional map data corresponding to the three-dimensional map data is created, and the image output step outputs an image represented by the three-dimensional map image data so that the image can be displayed. 17. The image processing method according to claim 16, wherein the method includes a route calculating step of calculating a route to a destination based on the map information. An image processing method comprising creating a three-dimensional map image according to the image processing method. 18. The image processing method according to claim 17, wherein the method includes a position input step of inputting position information, wherein the image processing step generates a stereoscopic map image according to the position information. An image processing method characterized by the following.