US20100104141A1 - System for and method of processing laser scan samples an digital photographic images relating to building facades - Google Patents
System for and method of processing laser scan samples an digital photographic images relating to building facades Download PDFInfo
- Publication number
- US20100104141A1 US20100104141A1 US12/311,461 US31146106A US2010104141A1 US 20100104141 A1 US20100104141 A1 US 20100104141A1 US 31146106 A US31146106 A US 31146106A US 2010104141 A1 US2010104141 A1 US 2010104141A1
- Authority
- US
- United States
- Prior art keywords
- laser scan
- scan samples
- samples
- laser
- processor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C11/00—Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
- G01C11/02—Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
- G01C11/025—Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures by scanning the object
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/40—Analysis of texture
- G06T7/41—Analysis of texture based on statistical description of texture
- G06T7/42—Analysis of texture based on statistical description of texture using transform domain methods
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20048—Transform domain processing
- G06T2207/20056—Discrete and fast Fourier transform, [DFT, FFT]
Definitions
- the present invention relates to a system for and method of processing laser scan samples comprising laser scan samples relating to building façades.
- one field of application is the detection of building façade texture elements from mobile mapping system data that includes images from cameras and output data from laser scanners.
- MMS mobile mapping system
- a MMS may take the form of a car that is driven along roads of interest while the processor controls the camera(s) to take pictures of building façades.
- the locations of texture elements, like windows and doors, in the façade pictures are identified.
- the texture elements as present in these façade pictures are then substituted by standard texture elements as stored in a library that is accessible by the processor. Since these standard texture elements are used to construct the façades, the memory space required to store façades with their texture elements is much less than is necessary for storing all the original pictures with their original textures. Further reference is made to unpublished PCT/EP2005/055317.
- the present invention provides a system as defined in claim 1 .
- the system according to the invention provides the user with more automatic detection of texture elements in façades than was possible in the prior art.
- an enormous amount of labour in the sense of manpower as well as money can be saved.
- a realistic building façade can be constructed from these texture elements, for example in a 3D model of the buildings along a street.
- the invention relates to a method as defined in the independent method claim.
- the invention relates to a computer program product comprising instructions and data arranged to instruct a processor to perform such a method.
- the invention relates to a data carrier comprising such a computer program product.
- FIG. 1 shows a MMS system with a camera and a laser scanner
- FIG. 2 shows a diagram of location and orientation parameters
- FIG. 3 shows a diagram of a computer arrangement with which the invention can be performed
- FIG. 4 shows a flow diagram of a method according to the invention
- FIG. 5 shows a histogram based on laser scan samples
- FIG. 6 shows several examples of laser scans of building façades
- FIGS. 7 a , 7 b , and 7 c show the façade of the last example of FIG. 6 , a variance over this façade and a vertical histogram over the façade of the building, respectively;
- FIG. 8 shows the output of a FFT performed on the data shown in FIG. 7 c
- FIGS. 9 a and 9 b show middle floors of a scanned façade and an averaged image of FIG. 9 a , respectively;
- FIGS. 10 a - 10 d show how masks can be derived from the images of FIGS. 9 a and 9 b;
- FIGS. 11 a and 11 b show how façade wall data is derived with the aid a mask
- FIG. 12 shows texture elements of façades as derived with the aid of a mask shown in FIG. 10 d;
- FIG. 13 shows a picture of a façade with upper and lower floors
- FIG. 14 shows a façade pattern with only average texture elements
- FIGS. 15 a , 15 b , and 15 c show an original picture of a façade, a façade picture as resulting from the invention and some stored texture elements, respectively.
- FIG. 1 shows a MMS system that takes the form of a car 1 .
- the car 1 can be driven by a driver along roads of interest.
- the car 1 is provided with a plurality of wheels 2 . Moreover, the car 1 is provided with a high accuracy position determination device. As shown in FIG. 1 , the position determination device comprises the following components:
- the system as shown in FIG. 1 is a so-called “mobile mapping system” which collect geographic data, for instance by taking pictures with one or more camera(s) 9 ( i ) mounted on the car 1 .
- the camera(s) are connected to the microprocessor ⁇ P.
- the laser scanner(s) 3 ( j ) take laser samples while the car 1 is driven along building blocks of interest. They are also connected to the ⁇ P and send these laser samples to the ⁇ P.
- the pictures and laser samples include information as to building block façades.
- the laser scanner(s) 3 ( j ) are arranged to produce an output with minimal 50 Hz and 1 deg resolution in order to produce a dense enough output for the method.
- a laser scanner such as MODEL LMS291-S05 produced by SICK is capable of producing such output.
- FIG. 2 shows which position signals can be obtained from the three measurement units GPS, DMI and IMU shown in FIG. 1 .
- FIG. 2 shows that the microprocessor ⁇ P is arranged to calculate 6 different parameters, i.e., 3 distance parameters x, y, z relative to an origin in a predetermined coordinate system and 3 angle parameters ⁇ x , ⁇ y , and ⁇ z , respectively, which denote a rotation about the x-axis, y-axis and z-axis respectively.
- the z-direction coincides with the direction of the gravity vector.
- the microprocessor in the car 1 and memory 9 may be implemented as a computer arrangement.
- An example of such a computer arrangement is shown in FIG. 3 .
- FIG. 3 an overview is given of a computer arrangement 10 comprising a processor 11 for carrying out arithmetic operations.
- the processor 11 would be the microprocessor 7 .
- the processor 11 is connected to a plurality of memory components, including a hard disk 12 , Read Only Memory (ROM) 13 , Electrically Erasable Programmable Read Only Memory (EEPROM) 14 , and Random Access Memory (RAM) 15 . Not all of these memory types need necessarily be provided. Moreover, these memory components need not be located physically close to the processor 11 but may be located remote from the processor 11 .
- the processor 11 is also connected to means for inputting instructions, data etc. by a user, like a keyboard 16 , and a mouse 17 .
- a user like a keyboard 16 , and a mouse 17 .
- Other input means such as a touch screen, a track ball and/or a voice converter, known to persons skilled in the art may be provided too.
- a reading unit 19 connected to the processor 11 is provided.
- the reading unit 19 is arranged to read data from and possibly write data on a data carrier like a floppy disk 20 or a CDROM 21 .
- Other data carriers may be tapes, DVD, CD-R. DVD-R, memory sticks etc. as is known to persons skilled in the art.
- the processor 11 is also connected to a printer 23 for printing output data on paper, as well as to a display 18 , for instance, a monitor or LCD (Liquid Crystal Display) screen, or any other type of display known to persons skilled in the art.
- a printer 23 for printing output data on paper
- a display 18 for instance, a monitor or LCD (Liquid Crystal Display) screen, or any other type of display known to persons skilled in the art.
- LCD Liquid Crystal Display
- the processor 11 may be connected to a loudspeaker 29 .
- the processor 11 may be connected to a communication network 27 , for instance, the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), the Internet etc. by means of I/O means 25 .
- the processor 11 may be arranged to communicate with other communication arrangements through the network 27 .
- the data carrier 20 , 21 may comprise a computer program product in the form of data and instructions arranged to provide the processor with the capacity to perform a method in accordance with the invention.
- computer program product may, alternatively, be downloaded via the telecommunication network 27 .
- the processor 11 may be implemented as stand alone system, or as a plurality of parallel operating processors each arranged to carry out subtasks of a larger computer program, or as one or more main processors with several sub-processors. Parts of the functionality of the invention may even be carried out by remote processors communicating with processor 11 through the network 27 .
- the computer arrangement does not need to have all components shown in FIG. 3 .
- the computer arrangement does not need to have a loudspeaker and printer then.
- the computer arrangement needs at least processor 11 , some memory to store a suitable program and some kind of interface to receive instructions and data from an operator and to show output data to the operator.
- the pictures and scans as taken by the camera(s) 9 ( i ) and the laser scanner(s) 3 ( j ) a similar arrangement as the one shown in FIG. 3 will be used, be it that that one will not be located in the car 1 but may conveniently be located in a building for off-line post-processing.
- the pictures and scans as taken by camera(s) 9 ( i ) and scanner(s) 3 ( j ) are stored in one of the memories 12 - 15 . That can be done via storing them first on a DVD, memory stick or the like, or transmitting them, possibly wirelessly, from the memory 9 .
- façade textures are decomposed by using both the pictures taken by the camera(s) 9 ( i ) and the laser scans taken by the laser scanner(s) 3 ( j ).
- the method uses a unique combination of techniques from both the field of image processing and laser scanning technology.
- FIG. 4 shows a flow chart of a method in accordance with the invention.
- FIG. 4 shows the following actions:
- the laser scanner(s) 3 ( j ) are, in an embodiment 2D laser scanner(s).
- a 2D laser scanner 3 ( j ) provides a triplet of data comprising time of measurement, angle of measurement, and distance to nearest solid object that is visible at this angle from the laser scanner 3 ( j ).
- a good method for finding façade points in the laser scan is to use a histogram analysis.
- FIG. 5 shows such a histogram.
- the histogram comprises a number of laser scan samples as taken by the laser scanner(s) 3 ( j ) at a certain distance as seen in a direction perpendicular to a trajectory traveled by the MMS system and summed along a certain distance traveled by the car 1 .
- the laser scanner(s) scan in an angular direction over, for instance, 180° in a surface substantially perpendicular to the earth surface (in practice there is a small deviation from the perpendicular plane due to the car 1 moving along the façade).
- the laser scanner(s) may take 180 samples each deviating by 1° from its adjacent samples.
- the MMS system will follow a route along a line that is directed along a certain road (only when changing lanes for some reason the traveled path will show deviations from this). So, in most cases, are the laser scans in a direction perpendicular to the road direction.
- FIG. 5 there are shown two histograms:
- the peak on histogram 63 indicates the presence of a flat solid surface parallel to the car heading.
- the approximate distance between the car 1 and the façade 65 can be determined by any available method. For instance, the method as explained in a co-pending patent application (attorneys file P6011543) can be used for that purpose. Alternatively, GPS (or other) data indicating the trajectory traveled by the car 1 and data showing locations of footprints of buildings can be compared and, thus, render such approximate distance data between the car 1 and the façade 65 . By analysing the histogram data within a certain area about this approximate distance, the local maximal peak within this area is identified as being the base of a façade 65 .
- FIG. 6 shows eight examples of such images, associated with eight different building façades. The images of FIG. 6 are presented with different grey scales where a certain grey intensity corresponds to a certain projected horizontal distance to the car 1 as measured to the laser scanner trajectory on the road.
- Examples of detected façade textures in FIG. 6 are:
- the next action in the process is the computation of the number of floors in a building as well as a height of a single floor.
- a variance filtering is applied with a window size of 3 ⁇ 3.
- the variance filtering is applied for a plurality of horizontal lines in the entire area of the image. each horizontal line is at a different height above ground. For every horizontal line an average value of such a variance over the complete line is calculated.
- Such average data is stored in a table in memory.
- the table comprises distance variance data averaged for each horizontal line (height) where “distance” refers to a perpendicular distance from the car 1 .
- a Fast Fourier Transform (FFT) is applied to this average variance, height dependent data in the table to find frequency characteristics of the height component of the image. Such transformation allows one to find a repeatable pattern in the average changes over the height of the building.
- FFT Fast Fourier Transform
- the next lowest frequency peak in the FFT output data will be caused by patterns of windows and other architectural elements outside the plane of the façade itself and therefore will represent the basic floor pattern of the building
- the highest peak in the FFT output data will correspond to the frequency of appearance of windows and other floor associated architectural elements in the façade. From the FFT output data, a size (height) of a floor can be computed, as will be explained now in greater detail with reference to FIGS. 7 a , 7 b , 7 c , and 8 .
- FIGS. 7 a , 7 b , and 7 c show the façade of the last example of FIG. 6 , the results of a variance filter applied over this image and a variance vs height profile over the façade of the building, respectively.
- FIG. 7 c shows a total variance sum for each horizontal line in the picture of FIG. 7 b .
- Brightness of the pixel in FIG. 7 b corresponds to variance value of depth calculated over of 3 ⁇ 3 area surrounding that pixel.
- FIG. 7 c one sees a measure of the average distortion in grey scale intensity at a height z above the earth.
- An FFT is applied to this variance data shown in FIG. 7 c , treating the vertical building height as the time axis, and variance as the value of the signal.
- the result of the FFT is shown in FIG. 8 .
- FIG. 8 the output of the Fourier Transform, shows a building's vertical harmonical frequency amplitude spectrum i.
- the vertical axis shows an amplitude whereas the horizontal axis shows the spatial frequency.
- the actual height of the floor is proportional to the inverse of the spatial frequency as determined by parameters of the FFT.
- Action 44 has rendered the floor size (height).
- action 46 the floors are averaged.
- a “middle floor” is defined as a floor with a minimal and maximal height that occurs more than once in the façade of the building.
- “middle floors” are vertically repeatable in the façade.
- the lower floor of most buildings is not a “middle floor”: its height is in most cases not within a certain minimal and maximal height as is the case for higher floors (in most buildings the lower floor is higher than other floors).
- the upper floor is, in most cases not a “middle floor” since its height can not be established very well with the method above because there is no floor above the upper floor with windows that can be identified and used.
- FIGS. 9 a and 9 b further clarify this.
- FIG. 9 a shows the façade from FIG. 7 a after removing of pixels representing the lowest and highest floor (this is done using a size of floor derived from the height of the façade and the number of floors).
- FIG. 9 b shows the same façade after an averaging process.
- the resolution of the façade data as would be captured by the laser scanner 3 ( j ) may be 3 times higher, for instance may increase from 10 cm to 3 cm resolution in given example).
- one pixel on the laser output corresponds to 20 pixels on the image as produced by the cameras. That is why the process of increasing resolution is very important in case when the laser scanner has a lower resolution then the camera (which is typical on currently available equipment). In that manner a higher resolution mask can be obtained, which later on will be applied to the image.
- FIG. 10 a shows the same laser scan as FIG. 9 b but replicated to reproduce the three floors.
- FIG. 10 b is a histogram showing the number of laser scanner samples of horizontal projected distances from the laser scanner to the car. It clearly shows one peak which is the most common distance value (also known as the mode value). Those samples which have the mode value are further treated as being associated with the walls of the building and the other samples are treated as being associated with texture elements. Such other samples are mainly associated with windows.
- the image is separated into two parts or masks.
- the first part contains all samples associated with the mode value and is used as a mask indicating the wall of the façade.
- the second part contains all samples associated with non-mode values and is used as a mask for deriving library elements from the pictures taken by the camera(s) 9 ( i ). Note that both masks contain samples where each sample contains location information.
- such a camera or cameras
- such a laser scanner or laser scanners
- the laser scanner may be part of a LIDAR (Light Detection And Ranging or Laser Imaging Detection And Ranging) system.
- LIDAR Light Detection And Ranging or Laser Imaging Detection And Ranging
- the present invention relates to the method and to the computer arrangement with such a processor, but also to a computer program product comprising suitable instructions and data for the described method, as well as a data carrier, like a CD, DVD, etc. comprising such a computer program.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Probability & Statistics with Applications (AREA)
- Image Processing (AREA)
Abstract
Description
- The present invention relates to a system for and method of processing laser scan samples comprising laser scan samples relating to building façades. one field of application is the detection of building façade texture elements from mobile mapping system data that includes images from cameras and output data from laser scanners.
- Systems for detection of texture elements from data obtained by a mobile mapping system (MMS) is known for some years. Such a MMS is provided with one or more cameras that take pictures of the environment as controlled by a processor on board the MMS. A MMS may take the form of a car that is driven along roads of interest while the processor controls the camera(s) to take pictures of building façades. In an off-line process, nowadays, the locations of texture elements, like windows and doors, in the façade pictures are identified. The texture elements as present in these façade pictures are then substituted by standard texture elements as stored in a library that is accessible by the processor. Since these standard texture elements are used to construct the façades, the memory space required to store façades with their texture elements is much less than is necessary for storing all the original pictures with their original textures. Further reference is made to unpublished PCT/EP2005/055317.
- To build a textured 3D façade model, a good method for identification of such texture elements in the raw picture of the façade is required. In the prior art, there exist methods to analyze and decompose texture from pictures. However, there is a rather high chance that incorrect objects are identified as being texture elements, such as trees, cars, people, and other obstacles in front of a façade. Also like all highly advanced image based technologies, the methods require heavy computational power. Moreover, the prior art process requires a lot of human interaction to obtain good quality for the textural representation of the façade. The speed of manual extraction may be only 1 km/h, on average, which, in total, would bring a cost of thousands of man hours for an average sized city.
- It is an object of the invention to provide a system and method to detect a texture element in a façade as well as its location that requires much less man power than in the prior art.
- To that effect, the present invention provides a system as defined in
claim 1. - The system according to the invention provides the user with more automatic detection of texture elements in façades than was possible in the prior art. Thus, an enormous amount of labour in the sense of manpower as well as money can be saved. And a realistic building façade can be constructed from these texture elements, for example in a 3D model of the buildings along a street.
- In an embodiment, the invention relates to a method as defined in the independent method claim.
- In a further embodiment, the invention relates to a computer program product comprising instructions and data arranged to instruct a processor to perform such a method.
- Moreover, the invention relates to a data carrier comprising such a computer program product.
- The invention will be explained in detail with reference to some drawings that are intended to illustrate the invention but not to limit its scope which is defined by the annexed claims and its equivalent embodiments.
- In the drawings:
-
FIG. 1 shows a MMS system with a camera and a laser scanner; -
FIG. 2 shows a diagram of location and orientation parameters; -
FIG. 3 shows a diagram of a computer arrangement with which the invention can be performed; -
FIG. 4 shows a flow diagram of a method according to the invention; -
FIG. 5 shows a histogram based on laser scan samples; -
FIG. 6 shows several examples of laser scans of building façades; -
FIGS. 7 a, 7 b, and 7 c, respectively, show the façade of the last example ofFIG. 6 , a variance over this façade and a vertical histogram over the façade of the building, respectively; -
FIG. 8 shows the output of a FFT performed on the data shown inFIG. 7 c; -
FIGS. 9 a and 9 b, respectively, show middle floors of a scanned façade and an averaged image ofFIG. 9 a, respectively; -
FIGS. 10 a-10 d show how masks can be derived from the images ofFIGS. 9 a and 9 b; -
FIGS. 11 a and 11 b show how façade wall data is derived with the aid a mask; -
FIG. 12 shows texture elements of façades as derived with the aid of a mask shown inFIG. 10 d; -
FIG. 13 shows a picture of a façade with upper and lower floors; -
FIG. 14 shows a façade pattern with only average texture elements; -
FIGS. 15 a, 15 b, and 15 c, respectively, show an original picture of a façade, a façade picture as resulting from the invention and some stored texture elements, respectively. -
FIG. 1 shows a MMS system that takes the form of acar 1. Thecar 1 is provided with one or more cameras 9(i), i=1, 2, 3, . . . I, and one or more laser scanners 3(j), j=1, 2, 3, . . . J. Thecar 1 can be driven by a driver along roads of interest. - The
car 1 is provided with a plurality ofwheels 2. Moreover, thecar 1 is provided with a high accuracy position determination device. As shown inFIG. 1 , the position determination device comprises the following components: -
- a GPS (global positioning system) unit connected to an
antenna 8 and arranged to communicate with a plurality of satellites SLi (i=1, 2, 3, . . . ) and to calculate a position signal from signals received from the satellites SLi. The GPS unit is connected to a microprocessor μP. Based on the signals received from the GPS unit, the microprocessor μP may determine suitable display signals to be displayed on a monitor 4 in thecar 1, informing the driver where the car is located and possibly in what direction it is travelling. - a DMI (Distance Measurement Instrument). This instrument is an odometer that measures a distance traveled by the
car 1 by sensing the number of rotations of one or more of thewheels 2. The DMI is also connected to the microprocessor μP to allow the microprocessor μP to take the distance as measured by the DMI into account while calculating the display signal from the output signal from the GPS unit. - an IMU (Inertial Measurement Unit). Such an IMU can be implemented as 3 gyro units arranged to measure rotational accelerations and translational accelerations along 3 orthogonal directions. The IMU is also connected to the microprocessor μP to allow the microprocessor μP to take the measurements by the DMI into account while calculating the display signal from the output signal from the GPS unit.
- a GPS (global positioning system) unit connected to an
- The system as shown in
FIG. 1 is a so-called “mobile mapping system” which collect geographic data, for instance by taking pictures with one or more camera(s) 9(i) mounted on thecar 1. The camera(s) are connected to the microprocessor μP. - Moreover, the laser scanner(s) 3(j) take laser samples while the
car 1 is driven along building blocks of interest. They are also connected to the μP and send these laser samples to the μP. - It is a general desire to provide as accurate as possible location and orientation measurement from the 3 measurement units: GPS, IMU and DMI. These location and orientation data are measured while the camera(s) 9(i) take pictures and the laser scanner(s) 3(j) take laser samples. Both the pictures and the laser samples are stored for later use in a suitable memory of the μP in association with corresponding location and orientation data of the
car 1 at the time these pictures and laser samples were taken. - The pictures and laser samples include information as to building block façades. In an embodiment, the laser scanner(s) 3(j) are arranged to produce an output with minimal 50 Hz and 1 deg resolution in order to produce a dense enough output for the method. A laser scanner such as MODEL LMS291-S05 produced by SICK is capable of producing such output.
-
FIG. 2 shows which position signals can be obtained from the three measurement units GPS, DMI and IMU shown inFIG. 1 .FIG. 2 shows that the microprocessor μP is arranged to calculate 6 different parameters, i.e., 3 distance parameters x, y, z relative to an origin in a predetermined coordinate system and 3 angle parameters ωx, ωy, and ωz, respectively, which denote a rotation about the x-axis, y-axis and z-axis respectively. The z-direction coincides with the direction of the gravity vector. - The microprocessor in the
car 1 andmemory 9 may be implemented as a computer arrangement. An example of such a computer arrangement is shown inFIG. 3 . - In
FIG. 3 , an overview is given of acomputer arrangement 10 comprising aprocessor 11 for carrying out arithmetic operations. In the embodiment shown inFIG. 2 , theprocessor 11 would be the microprocessor 7. - The
processor 11 is connected to a plurality of memory components, including ahard disk 12, Read Only Memory (ROM) 13, Electrically Erasable Programmable Read Only Memory (EEPROM) 14, and Random Access Memory (RAM) 15. Not all of these memory types need necessarily be provided. Moreover, these memory components need not be located physically close to theprocessor 11 but may be located remote from theprocessor 11. - The
processor 11 is also connected to means for inputting instructions, data etc. by a user, like akeyboard 16, and amouse 17. Other input means, such as a touch screen, a track ball and/or a voice converter, known to persons skilled in the art may be provided too. - A
reading unit 19 connected to theprocessor 11 is provided. Thereading unit 19 is arranged to read data from and possibly write data on a data carrier like afloppy disk 20 or aCDROM 21. Other data carriers may be tapes, DVD, CD-R. DVD-R, memory sticks etc. as is known to persons skilled in the art. - The
processor 11 is also connected to aprinter 23 for printing output data on paper, as well as to adisplay 18, for instance, a monitor or LCD (Liquid Crystal Display) screen, or any other type of display known to persons skilled in the art. - The
processor 11 may be connected to aloudspeaker 29. - The
processor 11 may be connected to acommunication network 27, for instance, the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), the Internet etc. by means of I/O means 25. Theprocessor 11 may be arranged to communicate with other communication arrangements through thenetwork 27. - The
20, 21 may comprise a computer program product in the form of data and instructions arranged to provide the processor with the capacity to perform a method in accordance with the invention. However, such computer program product may, alternatively, be downloaded via thedata carrier telecommunication network 27. - The
processor 11 may be implemented as stand alone system, or as a plurality of parallel operating processors each arranged to carry out subtasks of a larger computer program, or as one or more main processors with several sub-processors. Parts of the functionality of the invention may even be carried out by remote processors communicating withprocessor 11 through thenetwork 27. - It is observed that when applied in the
car 1 the computer arrangement does not need to have all components shown inFIG. 3 . For instance, the computer arrangement does not need to have a loudspeaker and printer then. As for the implementation in thecar 1, the computer arrangement needs at leastprocessor 11, some memory to store a suitable program and some kind of interface to receive instructions and data from an operator and to show output data to the operator. - For post-processing the pictures and scans as taken by the camera(s) 9(i) and the laser scanner(s) 3(j) a similar arrangement as the one shown in
FIG. 3 will be used, be it that that one will not be located in thecar 1 but may conveniently be located in a building for off-line post-processing. The pictures and scans as taken by camera(s) 9(i) and scanner(s) 3(j) are stored in one of the memories 12-15. That can be done via storing them first on a DVD, memory stick or the like, or transmitting them, possibly wirelessly, from thememory 9. - In the present invention, façade textures are decomposed by using both the pictures taken by the camera(s) 9(i) and the laser scans taken by the laser scanner(s) 3(j). The method uses a unique combination of techniques from both the field of image processing and laser scanning technology.
-
FIG. 4 shows a flow chart of a method in accordance with the invention. -
FIG. 4 shows the following actions: -
- A. action 42: extraction of façade points in the laser scan (which is the output of the laser scanner)
- B. action 44: floor height determination based on Fourier analysis of extracted points in the laser scan
- C. action 46: floor height averaging
- D. action 48: extraction of median-based library elements from laser scan
- E. action 50: creating a multi floor mask
- F. action 52: removing details from a façade picture
- G. action 54: creating façade pattern tile
- H. action 56: finding duplicate details on the façade picture
- I. action 58: dealing with highest and lowest floor elements
- J. action 60: storing the details as resulting from
54, 56 and 58actions - K. action 62: minimizing size of library.
- These actions will now be explained in detail below.
- A. Action 42: Extraction of Façade Points in Laser Scan
- The laser scanner(s) 3(j) are, in an embodiment 2D laser scanner(s). A 2D laser scanner 3(j) provides a triplet of data comprising time of measurement, angle of measurement, and distance to nearest solid object that is visible at this angle from the laser scanner 3(j). A good method for finding façade points in the laser scan is to use a histogram analysis.
FIG. 5 shows such a histogram. The histogram comprises a number of laser scan samples as taken by the laser scanner(s) 3(j) at a certain distance as seen in a direction perpendicular to a trajectory traveled by the MMS system and summed along a certain distance traveled by thecar 1. The laser scanner(s) scan in an angular direction over, for instance, 180° in a surface substantially perpendicular to the earth surface (in practice there is a small deviation from the perpendicular plane due to thecar 1 moving along the façade). E.g., the laser scanner(s) may take 180 samples each deviating by 1° from its adjacent samples. Most of the time, the MMS system will follow a route along a line that is directed along a certain road (only when changing lanes for some reason the traveled path will show deviations from this). So, in most cases, are the laser scans in a direction perpendicular to the road direction. - In
FIG. 5 , there are shown two histograms: -
- 1.
distance histogram 61—thishistogram 61 shows the number of laser scan samples as a function of distance to thecar 1 as summed over a certain travel distance, including samples close to thecar 1. There is a peak shown close to thecar 1 indicating a laser “echo” close to thecar 1. This peak relates to many echo's being present close to thecar 1 because of the angular sweep made by the laser scanning. Moreover, there is a second peak present at a greater distance which relates to an object identified at that greater distance from thecar 1. - 2.
distance histogram 63 showing only the second peak at a certain distance from thecar 1 indicating only one object. This histogram is achieved by eliminating the higher density of laser scan samples in the direct neighbourhood of thecar 1 due to the angular distribution of the laser scanning. The effect of this elimination is that one will better see objects at a certain distance away from thecar 1, i.e. the façade of abuilding 65.
- 1.
- The peak on
histogram 63 indicates the presence of a flat solid surface parallel to the car heading. The approximate distance between thecar 1 and thefaçade 65 can be determined by any available method. For instance, the method as explained in a co-pending patent application (attorneys file P6011543) can be used for that purpose. Alternatively, GPS (or other) data indicating the trajectory traveled by thecar 1 and data showing locations of footprints of buildings can be compared and, thus, render such approximate distance data between thecar 1 and thefaçade 65. By analysing the histogram data within a certain area about this approximate distance, the local maximal peak within this area is identified as being the base of afaçade 65. All laser scan samples that are within a perpendicular distance of, for instance, 0.5 m from this local maximal peak are considered as architectural detail of thefaçade 65 and marked as “façade points”. All other points are considered as “ghosts” and are marked so. It is observed that the distance of 0.5 m is only given as an example. Other distances may be used, if required. - From points marked as “façade” a depth map perpendicular to the vehicle direction is created and stored as an image. This depth-map contains all laser scan samples within, for instance, 0.5 m from the local maximal peak.
FIG. 6 shows eight examples of such images, associated with eight different building façades. The images ofFIG. 6 are presented with different grey scales where a certain grey intensity corresponds to a certain projected horizontal distance to thecar 1 as measured to the laser scanner trajectory on the road. - Examples of detected façade textures in
FIG. 6 are: -
- 1. Brighter grey color indicates objects that are closer than the façade plane itself, such as balconies.
- 2. Grey color indicates objects being more distant than the façade plane itself, usually doors or less transparent windows.
- 3. White color indicates places where depth is not known, because there was no laser sample at the analyzed distance.
- Below, the last façade of
FIG. 6 is analyzed in detail. - B. Action 44: Floor Height Size Determination Based on Fourier Analysis of Extracted Points of Laser Scan
- The next action in the process is the computation of the number of floors in a building as well as a height of a single floor.
- On the image generated in step 42 a variance filtering is applied with a window size of 3×3. The variance filtering is applied for a plurality of horizontal lines in the entire area of the image. each horizontal line is at a different height above ground. For every horizontal line an average value of such a variance over the complete line is calculated. Such average data is stored in a table in memory. Thus, the table comprises distance variance data averaged for each horizontal line (height) where “distance” refers to a perpendicular distance from the
car 1. A Fast Fourier Transform (FFT) is applied to this average variance, height dependent data in the table to find frequency characteristics of the height component of the image. Such transformation allows one to find a repeatable pattern in the average changes over the height of the building. There is a very low, zero frequency representing the constant background of the building's distance to the car. The next lowest frequency peak in the FFT output data will be caused by patterns of windows and other architectural elements outside the plane of the façade itself and therefore will represent the basic floor pattern of the building The highest peak in the FFT output data will correspond to the frequency of appearance of windows and other floor associated architectural elements in the façade. From the FFT output data, a size (height) of a floor can be computed, as will be explained now in greater detail with reference toFIGS. 7 a, 7 b, 7 c, and 8. -
FIGS. 7 a, 7 b, and 7 c, respectively, show the façade of the last example ofFIG. 6 , the results of a variance filter applied over this image and a variance vs height profile over the façade of the building, respectively. I.e.,FIG. 7 c shows a total variance sum for each horizontal line in the picture ofFIG. 7 b. Brightness of the pixel inFIG. 7 b corresponds to variance value of depth calculated over of 3×3 area surrounding that pixel. Stated in other terms, inFIG. 7 c one sees a measure of the average distortion in grey scale intensity at a height z above the earth. An FFT is applied to this variance data shown inFIG. 7 c, treating the vertical building height as the time axis, and variance as the value of the signal. The result of the FFT is shown inFIG. 8 . - Thus,
FIG. 8 , the output of the Fourier Transform, shows a building's vertical harmonical frequency amplitude spectrum i. The vertical axis shows an amplitude whereas the horizontal axis shows the spatial frequency. For this example, there is a peak at spatial frequency 6. That peak represents the dominate spatial frequency of the building which is assumed to be the floor spacing (and hence window pacing etc.). The actual height of the floor is proportional to the inverse of the spatial frequency as determined by parameters of the FFT. - C. Action 46: Floor Size Averaging
-
Action 44 has rendered the floor size (height). Inaction 46, the floors are averaged. In order to ease the overall process and make it more accurate, only “middle floors” are taken into account. A “middle floor” is defined as a floor with a minimal and maximal height that occurs more than once in the façade of the building. One can also say, “middle floors” are vertically repeatable in the façade. For instance, the lower floor of most buildings is not a “middle floor”: its height is in most cases not within a certain minimal and maximal height as is the case for higher floors (in most buildings the lower floor is higher than other floors). Similarly, the upper floor is, in most cases not a “middle floor” since its height can not be established very well with the method above because there is no floor above the upper floor with windows that can be identified and used. -
FIGS. 9 a and 9 b further clarify this.FIG. 9 a shows the façade fromFIG. 7 a after removing of pixels representing the lowest and highest floor (this is done using a size of floor derived from the height of the façade and the number of floors).FIG. 9 b shows the same façade after an averaging process. - By averaging all middle floors, as the resolution of the façade data as would be captured by the laser scanner 3(j) may be 3 times higher, for instance may increase from 10 cm to 3 cm resolution in given example). In it's original resolution one pixel on the laser output corresponds to 20 pixels on the image as produced by the cameras. That is why the process of increasing resolution is very important in case when the laser scanner has a lower resolution then the camera (which is typical on currently available equipment). In that manner a higher resolution mask can be obtained, which later on will be applied to the image.
- In a test of a large number of buildings in a European city, it was observed that over ninety percent of these buildings had at least two such similar middle floors So, the property of high similarity between two or more floors in a building can be used to virtually increase the laser resolution. This is especially important for higher buildings since laser resolution lowers with building height due to the angular nature of laser measurements. Moreover, averaging allows reconstructing places where trees were in front of part of the façade. This is because trees and other obstacles are filtered out in
action 42, and as such these points are not taken into the average.Action 46 delivers a “floor pattern”. - D. Action 48: Extraction of Median Based Library Elements from Laser Scan
-
FIG. 10 a shows the same laser scan asFIG. 9 b but replicated to reproduce the three floors.FIG. 10 b is a histogram showing the number of laser scanner samples of horizontal projected distances from the laser scanner to the car. It clearly shows one peak which is the most common distance value (also known as the mode value). Those samples which have the mode value are further treated as being associated with the walls of the building and the other samples are treated as being associated with texture elements. Such other samples are mainly associated with windows. - E. Action 50: Creating a Multi Floor Mask
- Then, the image is separated into two parts or masks. The first part contains all samples associated with the mode value and is used as a mask indicating the wall of the façade. The second part contains all samples associated with non-mode values and is used as a mask for deriving library elements from the pictures taken by the camera(s) 9(i). Note that both masks contain samples where each sample contains location information.
- to a person skilled in the art, such a camera (or cameras) and such a laser scanner (or laser scanners) may alternatively be located on an airborne vehicle. The laser scanner may be part of a LIDAR (Light Detection And Ranging or Laser Imaging Detection And Ranging) system.
- It is observed that the method as explained above can be performed by
processor 11 as instructed by a suitable software program stored in one of the memories 12-15 or stored elsewhere. The present invention relates to the method and to the computer arrangement with such a processor, but also to a computer program product comprising suitable instructions and data for the described method, as well as a data carrier, like a CD, DVD, etc. comprising such a computer program.
Claims (20)
Applications Claiming Priority (4)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/NL2006/050259 WO2008044913A1 (en) | 2006-10-13 | 2006-10-13 | System for and method of processing laser scan samples and digital photographic images relating to building façade |
| NLPCT/NL2006/050259 | 2006-10-13 | ||
| WOPCT/NL2006/050259 | 2006-10-13 | ||
| PCT/NL2006/050265 WO2008044914A1 (en) | 2006-10-13 | 2006-10-20 | System for and method of processing laser scan samples and digital photographic images relating to building facades |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| US20100104141A1 true US20100104141A1 (en) | 2010-04-29 |
| US8396255B2 US8396255B2 (en) | 2013-03-12 |
Family
ID=38255518
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| US12/311,461 Active 2028-10-30 US8396255B2 (en) | 2006-10-13 | 2006-10-20 | System for and method of processing laser scan samples and digital photographic images relating to building facades |
Country Status (5)
| Country | Link |
|---|---|
| US (1) | US8396255B2 (en) |
| EP (1) | EP2074378B1 (en) |
| JP (1) | JP2010506328A (en) |
| CN (1) | CN101529202A (en) |
| WO (2) | WO2008044913A1 (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080310733A1 (en) * | 2007-06-15 | 2008-12-18 | Microsoft Corporation | Facade rendering system |
| EP2463813A3 (en) * | 2010-12-09 | 2012-06-20 | Bilfinger Berger Industrial Services GmbH | Device, system and method for inspecting scaffolds |
| US20130141433A1 (en) * | 2011-12-02 | 2013-06-06 | Per Astrand | Methods, Systems and Computer Program Products for Creating Three Dimensional Meshes from Two Dimensional Images |
| US8970850B2 (en) | 2012-12-14 | 2015-03-03 | University Of New Brunswick | Method and apparatus for surface profilometry |
| US20150242542A1 (en) * | 2012-08-13 | 2015-08-27 | Auto-Measure Pty Limited | Building modelling system |
| EP3385662A1 (en) * | 2017-04-05 | 2018-10-10 | Testo SE & Co. KGaA | Method and device for determining a geometric parameter of an object |
Families Citing this family (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2010507127A (en) | 2006-10-20 | 2010-03-04 | テレ アトラス ベスローテン フエンノートシャップ | Computer apparatus and method for matching position data of different sources |
| US8970579B2 (en) | 2007-07-27 | 2015-03-03 | Procedural Ag | Computer system and method for generating a 3D geometric model |
| US8639234B2 (en) * | 2008-03-31 | 2014-01-28 | General Motors Llc | System and method for processing vehicle communications |
| CA2725564A1 (en) | 2008-12-19 | 2010-06-24 | Tele Atlas B.V. | Dynamically mapping images on objects in a navigation system |
| US8699755B2 (en) | 2009-02-20 | 2014-04-15 | Navteq B.V. | Determining travel path features based on retroreflectivity |
| WO2011023243A1 (en) | 2009-08-25 | 2011-03-03 | Tele Atlas B.V. | Digital map editing process using active contour manipulation |
| CN101908228B (en) * | 2010-07-16 | 2012-04-25 | 浙江大学 | Digital building measuring and drawing method for acquiring building vertical face map |
| WO2012089264A1 (en) * | 2010-12-30 | 2012-07-05 | Tele Atlas Polska Sp.Z.O.O | Method and apparatus for determining the position of a building facade |
| EP2495688A1 (en) * | 2011-03-02 | 2012-09-05 | Harman Becker Automotive Systems GmbH | Floor Number Determination in Buildings |
| US10380316B2 (en) | 2015-02-09 | 2019-08-13 | Haag Engineering Co. | System and method for visualization of a mechanical integrity program |
| CN105701471B (en) * | 2016-01-15 | 2019-04-12 | 昆明理工大学 | A kind of modification method of laser scanning Wave anomaly data |
| CN112614051A (en) * | 2020-12-08 | 2021-04-06 | 上海裕芮信息技术有限公司 | Building facade image splicing method, system, equipment and storage medium |
| CN116399266A (en) * | 2023-04-06 | 2023-07-07 | 洛阳理工学院 | A detection method for building exterior wall ceramic tile veneer |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060152522A1 (en) * | 2002-07-10 | 2006-07-13 | Marek Strassenburg-Kleciak | System for texturizing electronic representations of objects |
| US20080111815A1 (en) * | 2003-12-08 | 2008-05-15 | Gmj Citymodels Ltd | Modeling System |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4099776B2 (en) * | 2004-03-05 | 2008-06-11 | 日本電気株式会社 | 3D model creation device, 3D model creation method, and 3D model creation program |
| WO2007045272A1 (en) | 2005-10-17 | 2007-04-26 | Tele Atlas North America, Inc. | Method for generating an enhanced map |
-
2006
- 2006-10-13 WO PCT/NL2006/050259 patent/WO2008044913A1/en not_active Ceased
- 2006-10-20 CN CNA200680056051XA patent/CN101529202A/en active Pending
- 2006-10-20 EP EP06812724.0A patent/EP2074378B1/en active Active
- 2006-10-20 WO PCT/NL2006/050265 patent/WO2008044914A1/en not_active Ceased
- 2006-10-20 JP JP2009532310A patent/JP2010506328A/en active Pending
- 2006-10-20 US US12/311,461 patent/US8396255B2/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060152522A1 (en) * | 2002-07-10 | 2006-07-13 | Marek Strassenburg-Kleciak | System for texturizing electronic representations of objects |
| US20060188143A1 (en) * | 2002-07-10 | 2006-08-24 | Marek Strassenburg-Kleciak | Scanning system for three-dimensional objects |
| US20080111815A1 (en) * | 2003-12-08 | 2008-05-15 | Gmj Citymodels Ltd | Modeling System |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080310733A1 (en) * | 2007-06-15 | 2008-12-18 | Microsoft Corporation | Facade rendering system |
| US8224098B2 (en) * | 2007-06-15 | 2012-07-17 | Microsoft Corporation | Façade rendering system |
| EP2463813A3 (en) * | 2010-12-09 | 2012-06-20 | Bilfinger Berger Industrial Services GmbH | Device, system and method for inspecting scaffolds |
| US20130141433A1 (en) * | 2011-12-02 | 2013-06-06 | Per Astrand | Methods, Systems and Computer Program Products for Creating Three Dimensional Meshes from Two Dimensional Images |
| US20150242542A1 (en) * | 2012-08-13 | 2015-08-27 | Auto-Measure Pty Limited | Building modelling system |
| US10769864B2 (en) | 2012-08-13 | 2020-09-08 | Auto-Measure Pty Limited | Method and system for modelling a building structure |
| US8970850B2 (en) | 2012-12-14 | 2015-03-03 | University Of New Brunswick | Method and apparatus for surface profilometry |
| EP3385662A1 (en) * | 2017-04-05 | 2018-10-10 | Testo SE & Co. KGaA | Method and device for determining a geometric parameter of an object |
Also Published As
| Publication number | Publication date |
|---|---|
| EP2074378B1 (en) | 2017-12-13 |
| EP2074378A1 (en) | 2009-07-01 |
| WO2008044914A9 (en) | 2015-11-26 |
| JP2010506328A (en) | 2010-02-25 |
| CN101529202A (en) | 2009-09-09 |
| WO2008044914A1 (en) | 2008-04-17 |
| WO2008044913A1 (en) | 2008-04-17 |
| US8396255B2 (en) | 2013-03-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8396255B2 (en) | System for and method of processing laser scan samples and digital photographic images relating to building facades | |
| US9679362B2 (en) | System and method for generating textured map object images | |
| US20100118116A1 (en) | Method of and apparatus for producing a multi-viewpoint panorama | |
| JP2022101552A (en) | Device and method for analyzing objects | |
| US8325979B2 (en) | Method and apparatus for detecting objects from terrestrial based mobile mapping data | |
| US8571354B2 (en) | Method of and arrangement for blurring an image | |
| US8884962B2 (en) | Computer arrangement for and method of matching location data of different sources | |
| AU2008241689A1 (en) | Method of and apparatus for producing road information | |
| JPWO2018146279A5 (en) | ||
| CN101218602A (en) | Image processing device and environmental information observation device | |
| Kang et al. | The change detection of building models using epochs of terrestrial point clouds | |
| Campi et al. | Indoor and outdoor mobile mapping systems for architectural surveys | |
| HK1134132A (en) | System for and method of processing laser scan samples and digital photographic images relating to building facades | |
| Gunay et al. | Semi-automatic true orthophoto production by using LIDAR data | |
| Wac et al. | Assessment of the technical condition of a historic building using photogrammetry and terrestrial laser scanning: the case of the Juliusz Osterwa Theatre in Lublin | |
| Freeman et al. | Quantifying and visualising the uncertainty in 3D building model walls using terrestrial lidar data | |
| HK1136899A (en) | Method of and apparatus for producing a multi-viewpoint panorama | |
| HK1134129A (en) | Computer arrangement for and method of matching location data of different sources | |
| HK1148345A (en) | Method of and arrangement for mapping range sensor data on image sensor data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| AS | Assignment |
Owner name: TELE ATLAS N.V.,NETHERLANDS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:KMIECIK, MARCIN MICHAL;GLISZCZYNSKI, RAFAL JAN;REEL/FRAME:023597/0519 Effective date: 20091013 Owner name: TELE ATLAS N.V., NETHERLANDS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:KMIECIK, MARCIN MICHAL;GLISZCZYNSKI, RAFAL JAN;REEL/FRAME:023597/0519 Effective date: 20091013 |
|
| FEPP | Fee payment procedure |
Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
| AS | Assignment |
Owner name: TOMTOM GLOBAL CONTENT B.V., NETHERLANDS Free format text: CHANGE OF NAME;ASSIGNOR:TELE ATLAS B.V.;REEL/FRAME:029627/0615 Effective date: 20110125 Owner name: TELE ATLAS B.V., NETHERLANDS Free format text: CHANGE OF NAME;ASSIGNOR:TELE ATLAS N.V.;REEL/FRAME:029627/0578 Effective date: 20080730 |
|
| STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
| FPAY | Fee payment |
Year of fee payment: 4 |
|
| CC | Certificate of correction | ||
| MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |
|
| MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 12TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1553); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 12 |