JP2013161445A - Computer device, information processing system, and program - Google Patents

Abstract

The present invention provides a computer device capable of displaying a photograph taken with a camera and a sketch drawn with an electronic pen in an appropriate arrangement in accordance with a direction.
An image data input means for inputting image data photographed in all directions of 360 degrees by a camera, and a handwritten stroke written by an electronic pen on a sketch paper on which a coded pattern is formed. Receiving means for receiving information related to and a sketch corresponding to the handwritten stroke entered on the sketch paper based on the filling information received by the receiving means arranged in accordance with the direction in which the sketch is entered and displayed Display control means for displaying a 360-degree panoramic photograph corresponding to the image data input to the image data input means in accordance with the direction.
[Selection] Figure 1

Description

  The present invention relates to the technical field of displaying handwritten strokes entered with an electronic pen or the like.

  2. Description of the Related Art Conventionally, electronic pens that digitize written information have been developed, and “Anot pen” developed by Swedish company Anoto is known as a representative one. For example, Patent Document 1 describes that the anotopen reads a predetermined dot pattern printed on a sheet, generates entry information, and transmits the entry information to the terminal device. Patent Document 2 describes a rotation correction processing function that corrects the arrangement of dots on an image caused by the angle of an electronic pen facing the dot pattern. Patent Document 3 describes a technique for drawing the answer contents of each answerer entered with an electronic pen in a collaborative learning system using the electronic pen.

  In addition, Patent Document 4 describes a technique related to the present invention. Patent Document 4 describes a camera incorporating an electronic compass.

Japanese Patent No. 3874498 Special table 2003-529853 gazette JP 2009-276479 A JP 2009-88683 A

  By the way, it would be convenient if a landscape photograph taken with a camera and a landscape sketch drawn with an electronic pen can be arranged and displayed in accordance with the direction of the photograph or sketch. Therefore, the present invention provides a computer device, an information processing system, and a program capable of displaying a photograph taken with a camera and a sketch drawn with an electronic pen in an appropriate arrangement in accordance with a direction. Is the main purpose.

  In one aspect of the present invention, a computer apparatus fills in an image data input means for inputting image data photographed in all directions of 360 degrees by a camera, and sketch paper on which a coded pattern is formed, using an electronic pen. Receiving means for receiving written information relating to the handwritten stroke, and based on the written information received by the receiving means, a sketch corresponding to the handwritten stroke written on the sketch paper is entered in the sketch Display control means for displaying and arranging 360-degree panoramic photos corresponding to the image data inputted to the image data input means in accordance with the direction.

  In the above computer apparatus, the image data input means is inputted with the image data photographed in all directions of 360 degrees by the camera, and the receiving means is entered with the electronic pen on the sketch paper on which the coded pattern is formed. Receives information about handwritten strokes. Then, the display control means arranges and displays the sketch corresponding to the handwritten stroke entered on the sketch paper in accordance with the direction in which the sketch is entered, and the 360-degree panoramic photograph corresponding to the image data in the direction. Display together.

  According to the above computer apparatus, both the 360-degree panoramic photograph and the sketch written with the electronic pen can be appropriately displayed in accordance with the direction. Such a display is convenient when used in a weather class or the like. For example, it can be used effectively when learning the position of the sun according to the time, how to evaluate the weather according to the shape of the clouds and the amount of clouds. Moreover, according to said display, a sketch can be evaluated appropriately by contrasting a sketch with a 360-degree panoramic photograph.

  In this specification, “photographing in all directions of 360 degrees” does not mean that the image of “360 degrees” is strictly photographed, but an image of an angle slightly smaller than 360 degrees is photographed. And taking a azimuth of an angle slightly larger than 360 degrees is also included.

  In addition, the “direction” represents a direction in a horizontal plane at a certain point in relation to a reference direction, in other words, an orientation. “Direction” includes four directions of east, west, south, and north with azimuth intervals of 90 degrees, and eight directions of east, west, south, north, northeast, southeast, northwest, and southwest with azimuth intervals of 45 degrees. Is used. In addition, 16 orientations with an azimuth interval of 22.5 degrees and 32 orientations with an azimuth interval of 11.25 degrees may be used.

  In one aspect of the above computer apparatus, the image data input means is inputted with a direction detected by a sensor built in the camera together with the image data, and the display control means is supplied to the image data input means. Based on the input direction, the 360-degree panoramic photograph is displayed in accordance with the direction in which the sketch is written. Thereby, a 360 degree panoramic photograph can be appropriately displayed according to the direction.

  In another aspect of the above computer apparatus, the image data input means is inputted with the date and time of photographing by the camera together with the image data, and the display control means is the photographing that is inputted to the image data input means. The date and time are displayed together with the 360-degree panoramic photo. Thereby, it is possible to appropriately display the shooting date and time together with the 360-degree panoramic photograph.

  Preferably, in the above computer apparatus, the sketch sheet is provided with a sketch entry field for entering the sketch, and a direction selection field for selecting a direction in which the sketch is entered, and the sketch entry field, Different encoding patterns are formed in the direction selection field, and the display control means is configured to select the direction based on the entry information corresponding to the entry with the electronic pen in the sketch entry field and the direction selection field. The sketches entered in the sketch entry column are displayed in an arrangement corresponding to the direction selected in the selection column. Accordingly, the sketch can be appropriately displayed in association with the direction in which the sketch is written with the electronic pen.

  Preferably, in the above computer apparatus, the sketch sheet is provided with a sketch entry field for entering the sketch and a direction entry field for entering the direction in which the sketch is entered, and the sketch entry is provided. Different coding patterns are formed in the field and the direction entry field, and the display control means is based on the entry information corresponding to the entry with the electronic pen in the sketch entry field and the direction entry field, The sketches entered in the sketch entry column are displayed in an arrangement corresponding to the direction entered in the direction entry column. This also makes it possible to display the sketch appropriately in association with the direction in which the sketch is written with the electronic pen.

  Also preferably, the sketch paper is provided with a date and time entry column for entering the date and time when the sketch was entered, and a coding pattern is formed in the date and time entry column, and the display control means includes the date and time Based on the entry information corresponding to the entry with the electronic pen in the entry field, the date and time entered in the date and time entry field are arranged and displayed in the direction together with the sketch. Thereby, the date and time when the sketch was entered can be appropriately displayed together with the sketch.

  In another aspect of the above-described computer apparatus, the display control unit displays all or a part of the plurality of sketches and the 360-degree panoramic photograph in the horizontal direction or the vertical direction in association with the plurality of directions. When a part of the plurality of sketches and the 360-degree panoramic photograph is displayed, the part to be displayed in the plurality of sketches and the 360-degree panoramic photograph can be scrolled according to a user operation.

  In a preferred example, the camera is a fish-eye lens camera capable of photographing all directions of the 360 degrees.

  In another preferred example, the camera is a video camera that generates image data of a moving image obtained by photographing the 360 degrees of all directions, and the computer device converts the image data of the moving image into the 360-degree panoramic photograph. And a means for outputting the 360-degree panoramic photograph to the display control means.

  In another aspect of the present invention, the computer device, a camera that generates image data obtained by photographing 360 degrees in all directions, a sketch sheet on which a coding pattern is formed and a handwritten stroke is written by an electronic pen, An information processing system comprising: an electronic pen that reads the coded pattern formed on a sketch sheet and generates entry information related to the coded pattern. The image data input means of the computer device is generated by the camera. The image data is input, and the receiving means of the computer apparatus receives the entry information related to the coded pattern formed on the sketch paper, which is read by the electronic pen.

  In still another aspect of the present invention, the program causes the computer apparatus to function. By executing the program, the above-described computer device can be realized.

  According to the present invention, both a 360-degree panoramic photograph and a sketch written with an electronic pen can be appropriately displayed in accordance with the direction.

It is a system configuration figure of the information processing system concerning this embodiment. The sketch paper concerning this embodiment is shown. It is explanatory drawing which shows the relationship between the arrangement | positioning of the dot in a dot pattern, and the value converted. (A) schematically shows a dot pattern, and (b) is a diagram showing an example of information corresponding thereto. It is the schematic which shows the structure of an electronic pen. It is a functional block diagram of a fisheye lens camera. It is a functional block diagram of a computer apparatus. An example of a situation where a 360-degree landscape is photographed with a fish-eye lens camera is shown. An example of how each entrant sketches a landscape in a different direction. The example of the sketch entered in the sketch paper concerning this embodiment is shown. The pull-down menu of the tool menu used when displaying a sketch and a 360-degree panoramic photograph is shown. The example of a display screen of a sketch and a 360-degree panoramic photograph according to the present embodiment is shown. The processing flow concerning this embodiment is shown. The sketch paper concerning the modification 1 is shown. The example of a sketch written in the sketch paper concerning the modification 1 is shown. The video camera which concerns on the modification 2 is shown.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[Configuration of information processing system]
FIG. 1 is a system configuration diagram of an information processing system 10 according to the present embodiment. As shown in FIG. 1, the information processing system 10 fills in sketch paper 5A, 5B,... (5) distributed to each of a plurality of writers and sketches on the sketch paper 5, and sends the entered information to the outside. Electronic pens 1A, 1B,... (1) for transmitting by a wireless communication method such as Bluetooth (registered trademark), a fisheye lens camera 3 that captures 360 degrees of all directions and generates image data of the photographed photographs, A computer apparatus 2 is provided that receives entry information transmitted from each electronic pen 1, receives image data of a photograph taken by the fisheye lens camera 3, and displays a sketch or a photograph on a display (display means).

  Each electronic pen 1 can use an anotopen, and the computer device 2 can use a laptop personal computer or the like. The electronic pen 1 is used by a writer such as a student, and the computer device 2 is used by an operator such as a teacher. The sketch paper 5 is printed with a dot pattern (coded pattern) that can be read by the electronic pen 1. The projector 4 is a device that receives display information from the computer device 2 and projects and displays the screen of the computer device 2 on the screen 6.

  The fish-eye lens camera 3 is a digital camera having a fish-eye lens that can photograph 360 degrees in all directions, and generates image data of the photographed photograph. In this specification, “omnidirectional” means “360 ° omnidirectional”, and hereinafter, the term “360 °” will be omitted as appropriate.

  Here, a usage form of the information processing system 10 shown in FIG. 1 will be briefly described. Each entrant draws a sketch of a landscape on the sketch paper 5 with the electronic pen 1 outdoors. For example, each entrant draws sketches for different directions (eg, east, west, south, and north). On the other hand, the operator uses the fisheye lens camera 3 to photograph a landscape in all directions outdoors. Specifically, the operator uses the fisheye lens camera 3 to shoot an omnidirectional landscape including the landscape drawn by the writer. The computer device 2 receives the entry information from each entry person's electronic pen 1 and receives image data from the fisheye lens camera 3 so that the sketch drawn by the electronic pen 1 is in the direction in which the sketch is drawn. While arranging and displaying them together, a photograph (360-degree panoramic photograph) taken by the fisheye lens camera 3 is displayed in accordance with the direction. In one example, such an information processing system 10 is used for observation of the sun and clouds in a weather class.

[Sketch paper]
The sketch paper 5 according to the present embodiment will be described with reference to FIG. As shown in FIG. 2, on the sketch paper 5, a sketch entry field 5a for entering a sketch, a direction selection field 5b for selecting a direction in which the sketch is entered, and a date and time when the sketch is entered are entered. The date entry column 5c is provided. A plurality of directions (north, northeast, east, southeast, south, southwest, west, northwest) are printed in the direction selection column 5b, and a direction in which a sketch is entered can be selected from the plurality of directions. ing. Specifically, the direction selection column 5b can select a direction in which a sketch is written by enclosing one of a plurality of directions with a circle. The date and time entry column 5c can be filled with the date and time (hereinafter referred to as “sketch entry date and time”) when the sketch is entered.

  In the sketch entry field 5a, the direction selection field 5b, and the date / time entry field 5c, unique dot patterns (coded patterns) having different coordinate ranges are printed. The dot pattern is printed with ink containing carbon that absorbs infrared rays. On the other hand, on the sketch paper 5, characters (characters such as sketches, characters indicating directions, characters such as dates), frame lines, figures, and the like are printed with ink that does not absorb in the infrared region.

  A plurality of sketch papers 5 are prepared for distribution to each entry person or group. However, a unique dot pattern may be printed on the plurality of sketch papers 5 or transmitted by the electronic pen 1. Since the entry person can be identified by the pen ID to be used, the same dot pattern may be printed on the same area of each sketch paper 5.

[Dot pattern]
Next, an Anoto dot pattern (coded pattern) printed on the sketch paper 5 will be described with reference to FIGS. 3 and 4. FIG. 3 is a diagram for explaining the relationship between the dots of the dot pattern printed on the sketch paper 5 and the values to which the dots are converted. As shown in FIG. 3, each dot of the dot pattern is associated with a predetermined value depending on its position. In other words, each dot is associated with a value of 0 to 3 depending on which direction the top, bottom, left, or right is shifted from the reference position of the grid (intersection of the vertical line and horizontal line). The value of each dot can be further converted into a first bit value for the X coordinate and a second bit value for the Y coordinate. The position coordinates on the sketch paper 5 are determined by the combination of the information thus associated.

  FIG. 4A shows an arrangement of a certain dot pattern. As shown in FIG. 4A, 6 × 6 dots are arranged within a range of about 2 mm in length and width so that a 6 × 6 dot from any part on sketch paper 5 has a unique pattern. Has been. The dot pattern formed by these 36 dots holds position coordinates (for example, where the dot pattern is on the sketch paper 5). FIG. 4B is a diagram in which each dot shown in FIG. 4A is converted into a corresponding value based on the regularity shown in FIG. 3 according to the shift direction from the reference position of the lattice. This conversion is performed by the electronic pen 1 that captures an image of a dot pattern.

[Electronic pen]
Next, the electronic pen 1 will be described with reference to FIG. FIG. 5 is a schematic view showing the structure of the electronic pen 1. As shown in FIG. 5, the electronic pen 1 includes an ink cartridge 104, an LED 105, a CMOS camera 106, a pressure sensor 107, a processor 108 including a CPU, a memory 109 such as a ROM and a RAM, A communication unit 111 including a real-time clock 110, an antenna, and the like, and a battery 112 are provided. The tip of the ink cartridge 104 is a pen tip portion 103, and each entry person makes a stroke (handwritten stroke) on the sketch paper 5 by bringing the pen tip portion 103 of the electronic pen 1 into contact with the sketch paper 5. Alternatively, the pen tip portion 103 of the electronic pen 1 is brought into contact with the sketch paper 5 and tapped (tap on the sketch paper 5 by the pen tip portion 103). The ink of the ink cartridge 104 does not include a material that absorbs infrared rays (for example, carbon). Here, contact of the pen tip portion 103 of the electronic pen 1 with the sketch paper 5 is referred to as pen-down, and removal of the pen tip portion 103 from the contacted (contacted) state is referred to as pen-up. A trajectory written between the pen-down and pen-up of the electronic pen 1 is one stroke, and characters, figures, etc. are composed of one or a plurality of strokes. Since the ink of the ink cartridge 104 does not include a material that absorbs infrared rays, the reading of the dot pattern by the processor 103 is not hindered.

  The battery 112 is for supplying electric power to each component in the electronic pen 1. For example, the battery 112 is configured to turn on / off the electronic pen 1 itself by attaching and detaching a cap (not shown) of the electronic pen 1. May be. The real time clock 110 transmits time information indicating the current time (time stamp) and supplies the time information to the processor 108. The pressure sensor 107 detects a pressure applied through the ink cartridge 104 from the pen tip portion 103 when the writer enters a handwritten stroke on the sketch paper 5 or taps the sketch paper 5 with the electronic pen 1, that is, a writing pressure. Is transmitted to the processor 108.

  The processor 108 switches on / off the switches of the LED 105 and the CMOS camera 106 based on the writing pressure data supplied from the pressure sensor 107. That is, when the writer enters or taps a handwritten stroke on the sketch paper 5 with the electronic pen 1, the pen pressure is applied to the pen tip 103, and when the pressure sensor 107 detects a writing pressure higher than a predetermined value. The processor 108 determines that the writer has started filling in and activates the LED 105 and the CMOS camera 106. Then, the communication unit 111 associates the pen-down information PD detected by the pressure sensor 107 with a pen ID described later, and transmits it to the computer apparatus 2 as entry information. Further, when the writer finishes entering one stroke and moves the electronic pen 1 away from the sketch paper 5, the pressure sensor 107 detects pen-up because no writing pressure exceeding a predetermined value is detected. Then, the communication unit 111 associates the pen-up information detected by the pressure sensor 107 with the pen ID, and transmits it to the computer apparatus 2 as entry information.

  The LED 105 and the CMOS camera 106 are attached near the pen tip portion 103 of the electronic pen 1, and an opening 102 is formed in a portion of the housing 101 that faces the LED 105 and the CMOS camera 106. The LED 105 illuminates infrared rays toward the vicinity of the pen tip portion 103 on the sketch paper 5. The region is slightly shifted from the position where the pen tip 103 contacts the sketch paper 5. The CMOS camera 106 is provided with an infrared transmission filter that transmits infrared rays and blocks non-infrared rays, captures a dot pattern in the area illuminated by the LED 105, and supplies image data of the dot pattern to the processor 108. Here, since carbon absorbs infrared rays, the infrared rays irradiated by the LED 105 are absorbed by the carbon contained in the dots. Therefore, the dot portion has a small amount of infrared reflection, and the portion other than the dot has a large amount of infrared reflection. By photographing with the CMOS camera 106, a threshold value is provided based on the difference in the amount of reflected infrared light, so that a dot region containing carbon can be distinguished from other regions. Therefore, even when characters, frame lines, etc. are printed on the sketch paper 5, the processor 108 can recognize the dot pattern because the printed ink does not absorb in the infrared region. Note that the shooting area by the CMOS camera 106 is a range including a size of about 2 mm × about 2 mm as shown in FIG. 4A, and the shooting by the CMOS camera 106 is performed at regular intervals of about 50 to 100 times per second. Is called. Further, the CMOS camera 106 has a sufficient depth of field in order to photograph dots clearly.

  The processor 108 uses the dot pattern of the image data supplied by the CMOS camera 106 to perform X and Y coordinates (hereinafter, simply “ "Position coordinates" or "coordinate information") is continuously calculated. That is, the processor 108 converts the image data of the dot pattern as shown in FIG. 4 (a) supplied by the CMOS camera 106 into the data array shown in FIG. 4 (b). The data is converted into a Y coordinate bit value, and X and Y coordinate data are calculated from the data array by a predetermined calculation method. Note that the processor 108 has a rotation correction processing function for correcting the arrangement of dots on the image due to the angle of the electronic pen 1 facing the dot pattern, and this function is exhibited during the coordinate calculation. Then, the processor 108 associates the current time (time stamp: entered time information) transmitted from the real-time clock 110, writing pressure data, and X and Y coordinate data. Hereinafter, these associated data are collectively referred to as “coordinate attribute information”. Since the 6 × 6 dot pattern on the sketch paper 5 does not overlap in the sketch paper 5, when the writer enters characters or the like with the electronic pen 1, the written position is any position on the sketch paper 5. Can be specified by coordinate calculation by the processor 108.

  The memory 109 stores property information such as pen IDs “pen01” to “pen05”, pen manufacturer numbers, and pen software versions for identifying the electronic pens 1A to 1E. Then, the communication unit 111 associates the pen ID, time information (time stamp), writing pressure data, and X and Y coordinate data, and transmits them to the computer apparatus 2 as entry information. Transmission to the computer apparatus 2 by the communication unit 111 is performed immediately and sequentially by wireless transmission such as Bluetooth (registered trademark). Here, one or a plurality of coordinate attribute information generated between the pen-down and pen-up of the electronic pen 1 and transmitted to the computer device 2 is stored as stroke information by the computer device 2. In other words, one stroke is made up of one or a plurality of X, Y coordinates (coordinate points), and the computer apparatus 2 can use one or a plurality of pieces constituting one stroke based on the pen-down information and the pen-up information. Recognize coordinate attribute information.

  In the situation where the electronic pen 1 cannot communicate with the computer apparatus 2 (for example, the situation where radio waves do not reach), the entry information as described above is stored in the memory 108. Then, the electronic pen 1 transmits the entry information stored in the memory 108 in this way to the computer apparatus 2 collectively when it can communicate with the computer apparatus 2. In one example, when the writer is sketching outdoors, the electronic pen 1 stores the entry information in the memory 108 so that the writer can return to the indoor classroom and communicate with the computer apparatus 2. When the status is reached, the entry information is sent together.

[Fisheye lens camera]
Next, the fisheye lens camera 3 will be described. FIG. 6 shows a functional block diagram of the fisheye lens camera 3. The fisheye lens camera 3 includes a photographing unit 31, a direction detecting unit 32, a clock unit 33, a control unit 34, and a recording medium 35.

  The imaging means 31 receives light corresponding to a 360-degree omnidirectional landscape via a fisheye lens (not shown), and generates image data by converting the received light into an electrical signal. That is, the photographing unit 31 generates image data corresponding to a 360-degree panoramic photograph. The photographing means 31 is an image sensor (optical sensor) and is composed of, for example, a CCD or a CMOS. The direction detecting means 32 is constituted by a triaxial type geomagnetic sensor (electronic compass) or the like, and detects a direction (direction) in a horizontal plane at a position where the fisheye lens camera 3 is installed. The clock means 33 transmits time information indicating the current time (time stamp).

  The control unit 34 receives data from the imaging unit 31, the direction detection unit 32, and the clock unit 33, and stores in the recording medium 35 data with the direction and the imaging date / time added to the image data obtained by imaging. Specifically, the control means 34 detects the direction detection means 32 in order to associate the position on the photograph corresponding to the image data with the direction (that is, to associate which position on the photograph is in which direction). The direction is attached to the image data generated by the photographing means 31. Specifically, the control unit 34 attaches a numerical value (an angle or the like) indicating the direction detected by the direction detection unit 32 to the image data so that the position of the corresponding image can be understood. Alternatively, the control unit 34 converts the numerical value indicating the direction detected by the direction detection unit 32 into data indicating one of the east, south, west, and north directions, or the east, southeast, south, southwest, Data converted to data indicating any one of the west, northwest, north, and northeast directions is attached to the image data so that the position of the corresponding image can be understood. On the other hand, the control unit 34 generates a shooting date and time (data indicating the date and time when shooting was performed) based on the data transmitted from the clock unit 33, and attaches the shooting date and time to the image data.

  The recording medium 35 stores image data (image data corresponding to a 360-degree panoramic photo) with a direction and a shooting date and time by the control unit 34. The recording medium 35 is configured to be removable from the fisheye lens camera 3. For example, the recording medium 35 is a memory card.

[Computer device]
Next, the computer apparatus 2 will be described. The computer apparatus 2 is configured as a hardware such as a processor such as a CPU, a memory such as a ROM or a RAM, a display, a personal computer including a mouse, a keyboard, or the like. FIG. 7 is a functional block diagram of the computer apparatus 2. Functionally, the computer apparatus 2 includes an input unit 21, a reception unit 22, a processing unit 24, a storage unit 25, a display unit 26, a transmission unit 27, and an interface 28.

  The input means 21 includes a mouse and a keyboard for inputting various commands and data. When the display unit 26 is a touch panel system, the touch panel provided on the display screen of the display unit 26 also functions as the input unit 21. The receiving means 22 is constituted by an antenna receiving circuit or the like, receives entry information from the electronic pen 1 and transmits it to the processing means 24. The display means 26 is constituted by a display or the like, and displays the contents instructed by the processing means 24. The transmission means 27 is a means for transmitting the same content as the content displayed on the display means 26 from the projector 4 to the screen 6 and transmitting the display signal to the display means 26 to the projector 4 in synchronization.

  The interface 28 is attached with the recording medium 35 of the fisheye lens camera 3 described above, and reads data stored in the recording medium 35. Specifically, the interface 28 reads out the image data with the direction and the shooting date / time from the recording medium 35, and outputs the image data to the processing unit 24. For example, the interface 28 is a memory card reader. The interface 28 is an example of the “image data input unit” in the present invention.

  The storage means 25 is configured by a memory such as a hard disk, ROM, or RAM. The storage unit 25 stores coordinate definition information related to the coordinate range of the dot pattern printed on the sketch paper 5. Specifically, the storage means 25 stores coordinate ranges of different dot patterns respectively assigned to the sketch entry field 5a, the direction selection field 5b, and the date / time entry field 5c of the sketch paper 5. That is, the storage unit 25 stores coordinate definition information in which the sketch entry field 5a, the direction selection field 5b, the date and time entry field 5c are associated with the coordinate range of the dot pattern assigned to each of them. The coordinate range of the dot pattern is defined by the rectangular corner position coordinates (Xn, Yn), height (H), and width (W).

  Further, the storage means 25 stores the entry information received from the electronic pen 1 for each pen ID according to the processing instruction of the processing means 24, and further secures a storage area according to the instruction of the processing means 24, and from the pen down to the pen. The coordinate attribute information included in the entry information until the up is stored as stroke information for each pen ID. In addition to the above, the storage unit 25 stores various information generated by the execution of the program according to instructions from the processing unit 24.

  The processing means 24 is constituted by a processor such as a CPU and controls the entire computer apparatus 2. Specifically, the processing unit 24 includes a display control unit 241. The display control means 241 displays both the sketch written with the electronic pen 1 and the photograph (360-degree panoramic photograph) taken by the fisheye lens camera 3 in accordance with the direction. Specifically, the display control unit 241 performs a process for displaying a sketch and a process for displaying a 360-degree panoramic photograph in parallel, and displays the sketch and the 360-degree panoramic photograph in accordance with the direction. .

  First, processing for displaying a sketch will be described. The display control unit 241 receives the handwriting stroke corresponding to the sketch written on the sketch paper 5 (hereinafter, based on the entry information corresponding to the writing on the sketch paper 5 by the electronic pen 1 received by the receiving unit 22. The handwritten stroke corresponding to the sketch is also simply called “sketch”.) Are arranged and displayed in accordance with the direction in which the sketch is written. In other words, the display control means 241 displays the sketches (landscapes) entered by a plurality of writers in association with the direction in which those sketches are entered (the direction of the landscape drawn as a sketch).

  More specifically, the display control means 241 first recognizes the direction selected by each writer based on the entry information corresponding to the entry in the direction selection field 5b of each sketch paper 5 by each electronic pen 1. Then, the arrangement for displaying the sketches of each writer is determined. In this case, the display control unit 241 arranges the directions defined on the display screen (for example, “east → south → west → north” or “east → south → south → southwest → west → northwest → north → northeast”). The layout for displaying the sketches of each writer is determined based on the direction selected by each writer so that the sketches are displayed side by side. Then, the display control means 241 performs handwriting strokes corresponding to the sketches entered by each writer based on the entry information corresponding to the entry in the sketch entry field 5a of each sketch paper 5 by each electronic pen 1. Display them side by side according to the determined sequence. The display control means 241 also calculates the date and time (sketch entry date and time) entered by each writer based on the entry information corresponding to the entry in the date and time entry field 5 c of each sketch paper 5 by each electronic pen 1. In addition, it is displayed in association with the direction.

  Next, processing for displaying a 360-degree panoramic photograph will be described. Based on the image data read out by the interface 28 and stored in the recording medium 35 of the fisheye lens camera 3, the display control means 241 adds a 360 degree panoramic photograph corresponding to the image data to the image data. Display according to the direction. In this case, the display control unit 241 displays the 360-degree panoramic photograph in accordance with the direction in which the sketch is entered.

  More specifically, the display control unit 241 first recognizes the direction attached to the image data input from the interface 28. Then, the display control unit 241 displays the 360-degree panoramic photograph corresponding to the image data in an arrangement according to the recognized direction. In this case, the display control unit 241 arranges the directions defined on the display screen (for example, “east → south → west → north” or “east → south → south → southwest → west → northwest → north → northeast”). 360 degree panoramic photos are displayed in accordance with the above order. That is, the display control unit 241 displays the 360-degree panoramic photograph so that the direction on the 360-degree panoramic photograph matches the direction defined on the display screen. The display control unit 241 displays the shooting date and time attached to the image data together with the 360-degree panoramic photograph.

  Note that the display control unit 241 has a case where the photograph corresponding to the input image data has a circular scene (for example, a photograph by an equidistant projection method in which the distance from the center is proportional to the angle). A 360-degree panoramic photograph is generated by performing known processing such as distortion correction and planar development on such image data. However, if such processing has already been performed by the control unit 34 of the fisheye lens camera 3, the display control unit 241 does not perform the processing, but displays a 360 degree panoramic photograph corresponding to the input image data. Use as is.

  As described above, the display control unit 241 performs the process for displaying the sketch and the process for displaying the 360-degree panoramic photograph in parallel, so that both the sketch and the 360-degree panoramic photograph are directed. Display according to.

[Shooting example]
FIG. 8 shows an example of a situation where a 360-degree landscape is photographed by the fish-eye lens camera 3. The operator installs the fish-eye lens camera 3 at a position where an omnidirectional landscape including the landscape sketched by the writer is photographed, and shoots with the fish-eye lens camera 3.

[Sketch example]
Next, a sketch example using the sketch paper 5 according to the present embodiment will be described with reference to FIGS.

  FIG. 9 shows an example of how each entry person sketches (draws) a landscape in a different direction. FIG. 9 shows an example in which four writers are sketching landscapes in four directions, east, south, west, and north, respectively. In one example, each writer confirms the direction that he / she is facing using a compass, and the direction indicated by the teacher (either east, west, south or north). And sketch.

  FIG. 10 shows a sketch example entered on the sketch paper 5 by the electronic pen 1. As shown in FIG. 10, a sketch of a landscape is drawn in the sketch entry field 5a, a direction in which the sketch is drawn is selected in the direction selection field 5b, and a sketch is drawn in the date and time entry field 5c. The date and time are entered. Here, a case where the writer selects “south” as the direction in which the sketch is drawn by enclosing the “south” in the direction selection field 5b with a circle with the electronic pen 1 is illustrated. For example, each writer confirms the direction in which he / she is facing using a magnetic compass or the like, and selects the direction from the direction selection field 5b using the electronic pen 1.

[Display screen example]
Next, an example of a screen displayed on the display unit 26 of the computer apparatus 2 will be described with reference to FIGS. 11 and 12.

  FIG. 11 shows a screen on which a pull-down menu of a tool menu 90 used for displaying a sketch and a 360-degree panoramic photograph is shown. Note that the display of sketches is executed by an application such as “education tool”.

  As shown in FIG. 11, as a pull-down menu of the tool menu 90, a direction array menu 90a for displaying a sketch and a 360-degree panoramic photo in accordance with the direction, and a pen management menu 90b for displaying the connection status of the electronic pen 1 are displayed. And a paper management menu 90c for setting an image of the sketch paper 5 as a background for displaying stroke information, and a menu 90d for displaying the version and property information of the application. When these menus are selected, each function is executed by the processing means 24.

  FIG. 12 shows a display screen example of sketches and 360-degree panoramic photos according to this embodiment. This screen is displayed when the direction menu 90a of the tool menu 90 is selected. Further, this screen is displayed on the display means 26 by the display control means 241. In FIG. 12, for convenience of explanation, the portion of the photograph is represented by hatching.

  As shown in FIG. 12, the display control means 241 includes sketches 61 a, 61 b, 61 c (61) written by the respective writers with the electronic pen 1 and a 360-degree panoramic photograph 66 taken by the fisheye lens camera 3. Display in the same direction. Specifically, the display control unit 241 displays the sketches 61 side by side in the left-right direction according to the direction in which the sketch 61 is written, and according to the direction detected by the direction detection unit 32 of the fisheye lens camera 3. A 360-degree panoramic photograph 66 is displayed in the left-right direction. In this case, the display control means 241 displays the sketch 61 and the 360-degree panoramic photograph 66 in the left-right direction in accordance with the order of the directions such as “East → South → West → North” defined on the display screen. As a result, the sketch 61 and the portion of the 360-degree panoramic photograph 66 corresponding to the scenery in the same direction are lined up and down. That is, on the sketch 61, a 360-degree panoramic photograph 66 where the landscape in the direction in which the sketch 61 is drawn is photographed is located.

  Further, as shown in FIG. 12, the display control unit 241 displays the sketch entry date and time 62 a, 62 b, 62 c (62) entered by each entry person with the electronic pen 1 in association with the direction along with the sketch 61. In this case, the display control means 241 displays the characters (numbers) obtained by performing the character recognition process on the handwritten stroke entered in the date / time entry field 5 c with the electronic pen 1 as the sketch entry date / time 62. Further, the display control unit 241 displays the shooting date and time 67 attached to the image data corresponding to the 360 degree panoramic photograph 66 together with the 360 degree panoramic photograph 66. In addition, the display control means 241 displays information input to the computer apparatus 2 for recording by the operator (teacher) as indicated by reference numeral 63. For example, the operator inputs information related to the landscape (information such as the ratio of clouds, the type of clouds, and weather).

  Further, the display control means 241 activates the scroll bar 64 to display a part of the sketch 61 and the sketch entry date and time 62 entered by each writer, and a part of the 360-degree panoramic photograph 66. In the example shown in FIG. 12, the sketch 61 b is displayed in its entirety in the “south” direction in the example shown in FIG. 12. Is displayed, and the sketch is not displayed for the direction of “north” (the sketch entry date and time are not displayed). The operator can appropriately switch the parts to be displayed in the plurality of sketches 61 and 360-degree panoramic photos 66 by scrolling the screen by operating the scroll bar 64 with a mouse or a touch panel. (Even when scrolling, the information indicated by reference numeral 63 and the shooting date and time 67 do not scroll).

  In addition, although the example which displays four sketches 61 about four directions of east, south, west, and the north was shown above, eight directions of east, southeast, south, southwest, west, northwest, north, and northeast were shown. When the sketch 61 is entered, the display control means 241 displays the eight sketches 61 side by side in association with the direction in which those sketches are entered. For example, the display control means 241 displays the eight sketches 61 side by side in the order of “east → southeast → south → southwest → west → northwest → north → northeast”. However, if there is a direction in which the sketch 61 is not filled in the east, south, west and north directions, or the sketch 61 is in the east, southeast, south, southwest, west, northwest, north and northeast directions. When there is a direction not filled in, the sketch 61 for that direction is not displayed (for example, the direction is displayed in a blank).

  Further, in the above, an example is shown in which the character obtained by performing the character recognition process on the handwritten stroke entered in the date and time entry field 5 c with the electronic pen 1 is displayed as the sketch entry date and time 62. The handwritten stroke entered in the date / time entry field 5c in 1 may be displayed as the sketch entry date / time 62 as it is. Furthermore, in the above description, the sketch 61 and the 360-degree panoramic photograph 66 are displayed in the left-right direction. However, the sketch 61 and the 360-degree panoramic photograph 66 may be displayed in the vertical direction. In this case, the corresponding sketch 61 and the 360-degree panoramic photograph 66 are arranged side by side. In this case, a scroll bar for a vertical stroke is displayed instead of a scroll bar 64 for a horizontal stroke. Further, in the above description, the scroll bar 64 for scrolling the screen is displayed. However, when the display unit 26 is configured as a touch panel, the scroll bar 64 may not be displayed. In this case, it is only necessary to scroll the screen when an operation of tracing the screen with a finger is performed, and the display control means 241 can perform “east” → “south” → “west” even when scrolling in one direction. “→” North ”→“ East ”→... May be repeated to display the same photograph or sketch.

  Moreover, although the example which displays a part of several sketch 61 and 360 degree panoramic photos 66 on the screen was shown above, you may display the whole several sketch 61 and 360 degree panoramic photos 66 on a screen. In this case, an image obtained by reducing the image of the sketch 61 and the 360-degree panoramic photograph 66 when a part is displayed may be displayed. In this case, the scroll bar 64 may not be displayed.

[Processing flow]
Next, a processing flow according to the present embodiment will be described with reference to FIG. FIG. 13 shows a processing flow for displaying both a 360-degree panoramic photograph and a sketch. This processing flow is repeatedly executed by the computer apparatus 2 at a predetermined cycle.

  In the processing flow, processing for displaying a 360-degree panoramic photograph (steps S11 to S13) and processing for displaying a sketch (steps S14 to S17) are executed in parallel. First, processing (steps S11 to S13) for displaying a 360-degree panoramic photograph will be described. In step S <b> 11, the interface 28 reads image data from the recording medium 35, and the processing unit 24 stores the image data in the storage unit 25.

  Next, the display control means 241 of the processing means 24 recognizes the direction (correspondence between the image and the direction) attached to the image data (step S12). When a numerical value (such as an angle) indicating the direction detected by the direction detection unit 32 of the fisheye lens camera 3 is input, the display control unit 241 recognizes such a numerical value. On the other hand, the display control unit 241 converts the numerical value indicating the direction detected by the direction detection unit 32 into data indicating any one of east, south, west, and north, or east, southeast, south, When data converted into data indicating any one of the southwest, west, northwest, north and northeast is input, such direction is recognized.

  Thereafter, the display control unit 241 determines an arrangement for displaying a 360-degree panoramic photograph based on the correspondence relationship (including numerical values such as an angle) with the recognized direction image (step S13). Specifically, the display control unit 241 arranges the directions defined on the display screen (for example, “east → south → west → north” or “east → south → south → southwest → west → northwest → north → The 360-degree panoramic photograph is displayed so that the 360-degree panoramic photograph is displayed in the order of “northeast”, that is, the direction on the 360-degree panoramic photograph matches the direction defined on the display screen. Decide the arrangement for display.

  Next, the process (steps S14 to S17) for displaying a sketch will be described. In step S <b> 14, the receiving unit 22 receives entry information transmitted from the electronic pen 1, and the processing unit 24 stores the entry information in the storage unit 25. Next, the display control means 241 recognizes the direction selected by each writer based on the entry information corresponding to the entry in the direction selection field 5b of each sketch paper 5 by each electronic pen 1 (step S15). . Specifically, the display control unit 241 refers to the coordinate definition information about the coordinate range of the dot pattern assigned to the direction selection field 5b stored in the storage unit 25, so that each handwriting person can create a handwritten stroke. The position on the entered direction selection column 5b is specified, and the direction selected by each writer is recognized based on the position.

  Thereafter, the display control means 241 determines the arrangement for displaying the sketches of each writer based on the recognized direction (step S16). Specifically, the display control unit 241 arranges the directions defined on the display screen (for example, “east → south → west → north” or “east → south → south → southwest → west → northwest → north → Based on the direction selected by each writer, the arrangement for displaying the sketches of each writer is determined so that the sketches are displayed side by side in the order of “northeast”.

  Next, the display control means 241 draws handwritten strokes corresponding to the sketches entered by each writer based on the entry information corresponding to the entry in the sketch entry field 5 a of each sketch paper 5 by each electronic pen 1. (Step S17). In this case, the display control means 241 refers to the coordinate definition information about the coordinate range of the dot pattern assigned to the sketch entry field 5a, stored in the storage means 25, so that the handwritten stroke entered by each writer. Draw.

  Finally, the display control unit 241 displays the 360-degree panoramic photograph corresponding to the image data in accordance with the arrangement determined in step S13, and arranges the strokes drawn as described above according to the arrangement determined in step S17. It is displayed (step S18). That is, the display control unit 241 displays both the 360-degree panoramic photograph and the sketch in the direction.

[Effects of this embodiment]
According to the present embodiment described above, both the 360-degree panoramic photograph taken by the fisheye lens camera 3 and the sketch written by the electronic pen 1 can be appropriately displayed according to the direction. Such a display according to the present embodiment is convenient when used in a weather class or the like. For example, it can be used effectively when learning the position of the sun according to the time, how to evaluate the weather according to the shape of the clouds and the amount of clouds. Moreover, according to the display according to the present embodiment, the sketch can be appropriately evaluated by comparing the sketch with the 360-degree panoramic photograph.

[Modification]
Below, the modification of above-described embodiment is demonstrated. Note that the following modifications can be applied to the embodiment in any combination. In the following, the configuration different from the embodiment will be mainly described, and the description of the same configuration as the embodiment will be omitted as appropriate. That is, components and processes that are not specifically described are the same as those in the embodiment.

(Modification 1)
Modification 1 relates to another example of the sketch paper 5 described above. The sketch paper 5 according to the embodiment is provided with the direction selection field 5b for selecting the direction. However, in the sketch paper according to the first modification, the direction is directly selected by the electronic pen 1 instead of the direction selection field 5b. There is a direction entry field for entry.

  FIG. 14 shows a sketch paper 51 according to the first modification. As shown in FIG. 14, on the sketch paper 51, a sketch entry field 51a for entering a sketch, a direction entry field 51b for entering the direction in which the sketch is entered, and the date and time when the sketch is entered are entered. The date entry column 51c is provided. In the direction entry field 51b, a unique dot pattern (coded pattern) having a different coordinate range from the sketch entry field 51a and the date / time entry field 51c is printed, and the direction in which the sketch is entered can be entered. The sketch entry field 51a and the date entry field 51c are the same as the sketch entry field 5a and the date entry field 5c described above.

  FIG. 15 shows a sketch example entered on the sketch paper 51 by the electronic pen 1. As shown in FIG. 15, a sketch of a landscape is drawn in the sketch entry field 51a, and the direction in which the sketch was drawn (in this example, “south”) is entered in the direction entry field 51b. In 51c, the date and time when the sketch was drawn (the date and time when the sketch was entered) is entered. Here, in the first modification, the display control means 241 refers to the coordinate range of the dot pattern assigned to the direction entry field 51b, and uses the electronic pen 1 to write characters for the handwritten stroke entered in the direction entry field 51b. By performing the recognition process, the direction entered by each writer is recognized. Then, the display control means 241 displays handwritten strokes corresponding to the sketches entered by each writer in an array according to the direction thus recognized.

  Even when the sketch paper 51 as described above is used, the sketch written on the sketch paper 51 with the electronic pen 1 can be appropriately arranged and displayed according to the direction in which the sketch is written.

(Modification 2)
In the second modification, a video camera 3x as shown in FIG. 16 is used instead of the fisheye lens camera 3 described above. In the above-described embodiment, a 360-degree panoramic photograph is obtained by photographing a still image with the fish-eye lens camera 3 capable of photographing all directions of 360 degrees. On the other hand, in the modification 2, the video camera 3x is used to shoot a moving image for 360 degrees in all directions. For example, the video camera 3x is rotated 360 degrees at a constant angular velocity around one point, thereby shooting a moving image in all directions of 360 degrees. The computer apparatus 2 acquires the image data of the moving image shot by the video camera 3x in this way (for example, reads out the image data of the moving image recorded on a recording medium such as a memory card), and 360 converts the image data of the moving image to 360. Convert to a panoramic photo. Various known methods can be used for this conversion.

  In addition, the video camera 3x incorporates a triaxial type geomagnetic sensor (electronic compass) and the like, detects the direction in which the video camera 3x is facing during moving image shooting, and uses the detected direction as moving image data. Attached. The display control means 241 of the computer apparatus 2 displays a 360-degree panoramic photograph in accordance with the direction in the same manner as in the above-described embodiment, based on the direction attached to the moving image data.

  Also according to the second modification described above, both the 360-degree panoramic photograph obtained by photographing with the video camera 3x and the sketch filled in with the electronic pen 1 can be appropriately displayed in accordance with the direction.

(Modification 3)
The present invention is not limited to application to a general PC (personal computer), and can also be applied to a tablet PC such as “iPad (registered trademark)”.

  In the above description, the Anoto method is used for the electronic pen, the dot pattern (coded pattern), and the entry information, but the use of the Anoto method is not limited.

DESCRIPTION OF SYMBOLS 1 ... Electronic pen 2 ... Computer apparatus 3 ... Fish eye lens camera 3x ... Video camera 5 ... Sketch paper 10 ... Information processing system 21 ... Input means 22 ... Receiving means 24 ... Processing means 25 ... Storage means 26 ... Display means 28 ... Interface

Claims (11)

Image data input means for inputting image data taken in all directions of 360 degrees by the camera;
Receiving means for receiving entry information relating to a handwritten stroke written by an electronic pen on a sketch paper on which a coded pattern is formed;
Based on the entry information received by the receiving means, the sketch corresponding to the handwritten stroke entered on the sketch paper is arranged and displayed according to the direction in which the sketch is entered, and the image data is displayed. And a display control means for displaying a 360-degree panoramic photograph corresponding to the image data input to the input means in accordance with the direction. In the image data input means, a direction detected by a sensor built in the camera is input together with the image data,
2. The display control unit according to claim 1, wherein the 360-degree panoramic photograph is displayed in accordance with the direction in which the sketch is written, based on the direction input to the image data input unit. Computer equipment. In the image data input means, the shooting date and time by the camera is input together with the image data,
The computer apparatus according to claim 1, wherein the display control unit displays the shooting date and time input to the image data input unit together with the 360-degree panoramic photograph. The sketch sheet is provided with a sketch entry field for entering the sketch and a direction selection field for selecting a direction in which the sketch is entered, and the sketch entry field and the direction selection field are encoded differently. A pattern is formed,
The display control means has an arrangement according to the direction selected in the direction selection field based on the entry information corresponding to the entry with the electronic pen in the sketch entry field and the direction selection field, and the sketch. The computer apparatus according to any one of claims 1 to 3, wherein the sketch entered in the entry column is displayed. The sketch sheet is provided with a sketch entry field for entering the sketch and a direction entry field for entering the direction in which the sketch is entered, and the coding is different depending on the sketch entry field and the direction entry field. A pattern is formed,
The display control means, based on the entry information corresponding to the entry with the electronic pen in the sketch entry field and the direction entry field, in an arrangement according to the direction entered in the direction entry field, The computer apparatus according to any one of claims 1 to 3, wherein a sketch written in a column is displayed. The sketch paper is provided with a date and time entry column for entering the date and time when the sketch was entered, and a coded pattern is formed in the date and time entry column,
The display control means displays the date and time entered in the date and time entry column according to the direction along with the sketch based on the entry information corresponding to the entry with the electronic pen in the date and time entry column. The computer apparatus according to claim 1, wherein The display control means includes
In correspondence with a plurality of the directions, the whole or a part of the plurality of sketches and the 360-degree panoramic photograph is displayed in the horizontal direction or the vertical direction,
When a part of the plurality of sketches and the 360-degree panoramic photograph is displayed, the part to be displayed in the plurality of sketches and the 360-degree panoramic photograph is scrolled according to a user operation. Item 7. The computer apparatus according to any one of Items 1 to 6.   The computer apparatus according to any one of claims 1 to 7, wherein the camera is a fisheye lens camera capable of photographing all directions of the 360 degrees. The camera is a video camera that generates image data of a moving image obtained by photographing the 360 degrees of all directions,
8. The apparatus according to claim 1, further comprising: a unit that converts the image data of the moving image into the 360-degree panoramic photograph, and outputs the 360-degree panoramic photograph to the display control unit. Computer device. A computer device according to any one of claims 1 to 9,
A camera that generates image data obtained by photographing 360 degrees in all directions;
Sketch paper on which a coded pattern is formed and handwritten strokes are filled in with an electronic pen;
An electronic pen that reads the coded pattern formed on the sketch paper and generates entry information related to the coded pattern,
The image data input means of the computer device receives the image data generated by the camera,
The information processing system according to claim 1, wherein the receiving unit of the computer apparatus receives entry information related to the coded pattern formed on the sketch paper, which is read by the electronic pen.   A program for causing a computer apparatus according to any one of claims 1 to 9 to function.