HK1164518B - Using accelerometer information for determining orientation of pictures and video images - Google Patents

Description

Using accelerometer information to determine orientation of pictures and video images

Technical Field

The present invention is directed to determining the orientation of pictures and video images, and more particularly to using accelerometer information to determine the orientation of pictures and video images.

Background

Mobile devices incorporating digital cameras and position sensing technology are becoming increasingly common. Since such mobile devices can rotate in any direction when capturing a picture or video, there is a lack of uniformity in the orientation of the picture or video image when it is stored on the device. For example, a user may take a picture using a mobile device that is rotated slightly to the right or left. Sometimes such rotation is intentional (which represents an ideal picture composition), but sometimes not. Regardless of whether the rotation is intentional, the user may want to adjust the picture at a later time. Rotation of a picture or image is often a manual process (such as using a photo editing application) and can be difficult and inaccurate.

The difference in direction between pictures or video images is problematic when dealing with common image processing tasks. For example, when comparing a picture to one or more other pictures, differences in direction, such as rotation, can increase the difficulty of finding a matching picture or finding matching features within a picture. Similarly, when matching features between two or more pictures during stitching of the pictures to a mosaic or panoramic view, the difference in orientation can make it more difficult for the task to be performed successfully and accurately.

Therefore, there is a sufficient opportunity for improvement of the technology related to picture and image capturing and processing using directional information.

Disclosure of Invention

Various techniques can be applied regarding picture and image capture and processing using directional information.

For example, a method for capturing and processing video data is provided. The method includes, for each of a plurality of video images, obtaining orientation information of a computing device that captured the video image, the orientation information indicating an orientation of the computing device at a time the video image was captured, and adjusting the captured video image based on the orientation information. The adjusted captured video image can be displayed (e.g., in real-time during video capture, or later). A three-axis accelerometer can be used to capture directional information. In some implementations, the direction information includes three values: acceleration in the x-axis, acceleration in the y-axis, and acceleration in the z-axis of the computing device performing the capturing.

As another example, a mobile device for capturing and processing video data is provided. The mobile device includes a three-axis accelerometer, a memory, a digital camera, and a processing unit. The mobile device is configured to obtain, for each of a plurality of video images, direction information indicative of a direction of the mobile device when the video image was captured, and store the captured video and associated direction information. The associated orientation information can be stored in the same file as the captured video image or in a different file, and can be stored as an acceleration value, a rotation angle value, or a value expressed in another way. In some cases, the orientation information is streamed from the mobile device to the network server along with the video image data (with or without adjustments based on the orientation information).

Alternatively, a two-axis accelerometer (e.g., rather than a three-axis accelerometer) can be used to capture directional information (e.g., acceleration values in two axes), which can be used to determine and compensate for rotation in one plane.

As another example, a method for matching image data is provided. The method includes obtaining image data of a first image captured by a digital camera and associated orientation information, and obtaining image data of a second image captured by the digital camera and associated orientation information. The orientation information can, for example, represent three-axis accelerometer information obtained during image capture. Next, at least one of the first and second captured images is adjusted using the orientation information to compensate for differences in orientation of the digital camera along the at least one plane of rotation. Image matching is performed by matching points of interest between the first and second images as adjusted.

The foregoing and other features and advantages of the invention will become further apparent from the following detailed description, read in conjunction with the accompanying drawings.

Brief Description of Drawings

FIG. 1 is a flow chart showing an exemplary method for capturing video data and processing the video data for display using orientation information.

Fig. 2 is a flow chart showing an exemplary method for capturing video data and storing the video data using orientation information.

FIG. 3 is a diagram showing an example mobile device including an accelerometer, where the mobile device is generally aligned with a plane defined by an X-axis and a Y-axis, and perpendicular to a Z-axis.

FIG. 4 is a block diagram illustrating an example mobile device including an accelerometer.

Fig. 5 is a diagram illustrating an example image matching operation using direction information.

FIG. 6 is a flow chart showing an exemplary method for matching image data using orientation information.

FIG. 7 is a block diagram illustrating details of an example mobile device.

FIG. 8 is a diagram illustrating an example implementation environment.

Detailed description of exemplary embodiments

The following description is directed to techniques and schemes for image processing using orientation information. The various techniques and solutions may be combined or used separately. Different embodiments may implement one or more of the described techniques and solutions.

I. Example Direction information

In the techniques and schemes described herein, orientation information is used for various image processing tasks. As used herein, directional information refers to information that can describe the direction of a device, such as a mobile device. For example, the orientation information includes information that can describe the rotation of the device in at least one plane.

In some instancesCurrently, orientation information is obtained from a three-axis accelerometer (sometimes referred to as a g-sensor). The three-axis accelerometer provides three values, each representing acceleration along a different axis. In particular, a three-axis accelerometer provides values representing acceleration along the x-axis, values representing acceleration along the y-axis, and values representing acceleration along the z-axis. Conventionally, the value returned by the accelerometer has a value of m/s²(m/s)²) Unit (of which about 9.80665m/s²Is equal to a gravitational force, or 1 g). As used herein, the terms "ax," "ay," and "az" refer to acceleration values obtained from a three-axis accelerometer with respect to x-axis, y-axis, and z-axis directions.

In some implementations, the direction information includes three acceleration values (ax, ay, and az) obtained from a three-axis accelerometer. In other implementations, the directional information includes values calculated from acceleration values (e.g., left/right roll angles and/or forward/backward tilt angles). In yet other implementations, only two-axis acceleration values are obtained (e.g., only ax and ay or only az and ay), which can be used to determine rotation in one plane (e.g., only left/right roll or only forward/backward tilt).

FIG. 3 is a diagram illustrating an example mobile device 300 including an accelerometer. The accelerometer provides orientation information describing the orientation of the mobile device 300. With a three-axis accelerometer, directional information of the x, y, and z-axes can be obtained. The position of the accelerometer within the mobile device 300 will determine the orientation of the various axes. In the example of the mobile device 300, the accelerometer has been oriented such that the y-axis of the accelerometer points downward (toward the earth) when the mobile device 300 is held both vertically and horizontally. The x-axis extends left and right from the mobile device 300 and the z-axis extends forward and backward from the mobile device 300. The accelerometer can be oriented differently within the device (e.g., the z-axis points downward when the device is held vertical and horizontal), which would require corresponding modifications to the axes for the various operations described herein.

When the mobile device 300 is oriented vertically and horizontally (as depicted in the figures), the three-axis accelerometer will provide a y-axis of approximately 9.80665m/s²Value of (a), x-axis 0.0m/s²Value of (representing a left-right roll of 0 degrees), and z-axis 0.0m/s²(indicating a backward-forward tilt of 0 degrees). If the mobile device 300 is rotated slightly to the right, the y-axis acceleration value will decrease and the x-axis acceleration value will increase, indicating a rotation in the (y, x) plane (if the device 300 is not rotated forward or backward, the z-axis value will remain unchanged). Similarly, if the mobile device 300 is rotated slightly forward, the y-axis acceleration value will decrease and the z-axis acceleration value will increase, indicating a rotation in the (y, z) plane (if the device 300 is not rotated left or right, the x-axis value will remain unchanged). The combination of left/right and forward/backward rotation will result in corresponding changes in x, y and z-axis acceleration values. However, regardless of how the device rotates, the size of the vector (ax, ay, az) is theoretically 9.80665m/s²(when the device is stationary or has a constant speed).

The mobile device 300 may be any type of mobile computing device. For example, mobile device 300 may be a mobile phone, a Personal Digital Assistant (PDA), a smart phone, a tablet computer, a laptop computer, a digital camera, a digital video recorder, and so forth. Although fig. 3 depicts a mobile device 300, the techniques and aspects described herein are not limited to mobile devices. In general, any type of computing device (e.g., digital camera and/or video camera) including accelerometers and imaging devices can implement the techniques and aspects described herein. For example, a desktop computer with an accelerometer and an integrated camera (or an attached camera with an accelerometer) can obtain orientation information during the capture of a picture or video image.

Example Association Direction information

In the techniques and solutions described herein, orientation information can be associated with still pictures and video images. When a picture or video image is captured, orientation information for the device performing the capture (e.g., a mobile device or another type of computing device) can be obtained and associated with the captured picture or video image. For example, the orientation information is saved in the same file as the picture or video (e.g., in a header of the picture or video) or in a separate file associated with the picture or video file.

For example, a smart phone with a camera and an accelerometer is used to take still pictures. When a picture is taken, directional information from the accelerometer is obtained and associated with the captured picture. The orientation information describes the orientation of the smartphone when the picture was taken. In a particular implementation, the orientation information for the still picture (e.g., ax, ay, and az values obtained from a tri-axial accelerometer) is saved in the IPTC header of the picture file according to an Information Interchange Model (IIM) provided by international news teleconferencing (IPTC). Alternatively, the direction information is stored in a separate metadata file.

In another example, a smartphone with a camera (which supports video capture) and an accelerometer is used to record video. During recording of the video, directional information from the accelerometer is obtained and associated with each individual video image that is captured (e.g., each individual video frame or field). The orientation information describes the orientation of the smartphone during the capture of each individual video image. The orientation information can be stored in a file with the video images or in a separate association file (e.g., including information associating sets of individual captured orientation information values (such as ax, ay, and az values) with their corresponding video images). In addition to, or instead of, saving the video images and the directional information, the video images and directional information can be streamed from the smartphone to one or more destinations (e.g., to a network server for storage or to another device for display).

In a particular implementation, the video images and the directional information are streamed using real-time transport protocol (RTP). In particular, the original (unadjusted) video image and the direction information are stored in the payload field of the RTP protocol and transmitted to one or more destinations (e.g., a network server or other device). Alternatively, the direction information can be inserted in the header field. At the destination, the image data and the direction information are extracted. The directional information can be used to adjust the video image at the destination. Alternatively, the video image can be adjusted using the orientation information prior to streaming the video image (the orientation information can still be sent with the adjusted video image).

Example calculation of rotation angles

Using a three-axis accelerometer, the rotation angle can be calculated for both left/right roll and forward/backward tilt of the device. Using the directions depicted in fig. 3, the rotation angle in the (y, x) plane (left/right roll of the device) is defined as α. The rotation angle in the (y, z) plane (forward/backward tilting of the device) is defined as β.

In a particular implementation, the rotation angle is determined by computing a 3D rotational transformation between a vector (0, | y |, 0) representing the gravitational acceleration and a vector (ax, ay, az) generated by the accelerometer values. The value of | y | is theoretically 9.80665. However, due to the non-ideality of the accelerometer, for calibration purposes, the value of | y | is calculated as sqrt (ax)²+ay²+az²). The 3D conversion is calculated using the following equation:

using the above 3D conversion, the rotation angles α and β can be directly calculated.

Example rotation Compensation

Once the rotation angle (the values of α and β above) is calculated, the captured picture or video image can be adjusted to compensate for the device rotation. Compensation can be performed for only one plane (e.g., only for alpha representing roll) or for both planes (taking roll and pitch into account).

Compensation can be performed to return any given picture or image to the reference direction. The reference direction indicates the direction of the device and may indirectly indicate the direction of the reference picture or video image. The reference direction may be any desired direction and depends on the needs of a particular situation. For example, the reference direction may be an α value of 0 degrees and a β value of 0 degrees, representing the vertical/horizontal direction of the device capturing the picture or image or considered ideal for image capture. As another example, the reference direction may be an alpha value of 90 degrees.

For example, if a picture or video image is captured in which the orientation information indicates a roll of 20 degrees to the right (alpha value of 20 degrees), the picture or video image can be adjusted to change the orientation by 20 degrees to the left to return the picture or image to the reference orientation (the reference manner is an alpha value of 0 degrees).

As another example, the reference direction may be the direction of the computing device when the initial image within the video sequence was captured. So, if an initial video image within the sequence is captured with the directional information indicating a 5 degree roll to the left and a subsequent video image is captured with the directional information indicating an 8 degree roll to the left, then the subsequent video image is adjusted to change direction by 3 degrees to the right to return the image to the reference direction.

The conversion applied to a picture or video image to compensate for differences in device orientation depends on the implementation. In some implementations, the device performs rotation, translation, scaling, and/or cropping operations using affine transformation when compensating for differences in device orientation. For example, the parameters of the conversion are set in dependence on the rotation angles α and/or β.

V. example method of capturing video images

Fig. 1 illustrates an exemplary method 100 for capturing video data and processing the video data using directional information. At 110, orientation information for the captured video image is obtained (i.e., orientation information at least approximately simultaneously with capturing the video image is obtained). For example, the directional information includes ax, ay, and az values from a three-axis accelerometer.

At 120, the captured video image is adjusted using the orientation information. For example, the 3D conversion described above can be used to determine the angle of rotation of the captured video image in one or two planes (e.g., roll only or both roll and tilt). The captured video image can be adjusted according to the rotation angle(s). For example, if the captured video image is captured at 10 degrees roll to the right (using the orientation depicted in fig. 3 to rotate in the (y, x) plane), then the captured video image can be compensated for by rotating 10 degrees to the left (to return the captured image to the reference orientation of 0 degrees left/right roll). In addition to or instead of roll adjustment, the captured video image can also be adjusted for forward/backward tilting. For example, the captured video image is warped by an affine transformation (for image translation, scaling, rotation and/or cropping) or a perspective transformation.

In a specific implementation, compensation for roll and pitch is performed as follows, using the α and β values obtained from the calculations in "example calculation of angle of rotation" above. First, the angle in the (y, x) plane (roll) is used to adjust the picture or image by using a 2x2 rotation matrix, which has the general form:

second, once the roll has been adjusted, compensation for tilt is performed using the value of β. Specifically, the x-value of the picture or image remains the same while the y-value is multiplied by cos (β).

At 130, the adjusted captured image is displayed. For example, the adjusted captured image can be output for display by the same device that captured the image (e.g., a smartphone), or the adjusted captured image can be output from the capture device for display by another device (e.g., transmitted from the smartphone to the other device or to a server).

The example method 100 can be performed for each of a plurality of video images. For example, the example method 100 is performed on individual video images captured during video recording. The example method 100 can be performed in real-time during video recording. For example, orientation information for a video image is obtained 110, the image is then adjusted 120 and the video is displayed 130 as the next image being captured (e.g., allowing the user to view the adjusted video image in real-time during video capture).

Fig. 2 illustrates an exemplary method 200 for capturing video data and storing the video data using orientation information. At 210, orientation information for the captured video image is obtained (i.e., orientation information at least approximately simultaneous with the captured video image is obtained). For example, the directional information includes ax, ay, and az values from a three-axis accelerometer.

At 220, the captured video image is stored with its associated orientation information. For example, the orientation information is saved in the same file as the captured video information (e.g., in the header of an image file or structure), or the orientation information is saved in a separate file associated with the video image file.

The example method 200 can be performed for each of a plurality of video images. For example, the example method 200 is performed on individual video images captured during video recording. The example method 200 can be performed in real-time during video recording. For example, orientation information 210 for a video image is obtained and then stored 220 with the captured video image as subsequent video being captured.

The captured video images can be adjusted (e.g., in real-time) at the time of ongoing capture of other images or adjusted at a later time (e.g., using the saved orientation information and saved video images). Adjusting the video image using the orientation information can include compensating for a rotation of the video image relative to a reference orientation. For example, the reference direction can be a vertical/horizontal direction as depicted in fig. 3 (where ay ═ 9.80665, ax ═ 0.0, and az ═ 0.0). Compensation for rotation (e.g., roll in the (y, x) plane depicted for fig. 3 and/or tilt in the (y, z) plane depicted for fig. 3) can be performed in one or two planes. For example, rotation compensation can be performed for each plane by calculating an angle of rotation within the plane, determining a difference in angle of rotation between the calculated angle of rotation and a reference angle of rotation, and adjusting the captured video image to compensate for the rotation (e.g., to return the captured video image to the reference angle of rotation).

VI. an example mobile device

In the techniques and solutions described herein, a mobile device can be used to capture pictures and/or video images, as well as to capture associated directional information. The mobile device can also perform other operations based on the captured picture/image and associated orientation information, such as adjusting the picture or image to compensate for rotation, saving the original and/or adjusted picture or image, saving orientation information, streaming the original and/or adjusted picture or image, and streaming orientation information.

The mobile device may be any type of mobile computing device. For example, the mobile device may be a mobile phone, a Personal Digital Assistant (PDA), a smart phone, a tablet computer, a laptop computer, a digital camera, a digital video recorder, and so forth. Further, computing devices that are not generally considered "mobile," such as desktop computers, can also implement the techniques and aspects described herein (e.g., desktop computers with built-in cameras and accelerometers or with cameras including accelerometers).

FIG. 4 is a block diagram showing an example mobile device 410 capable of capturing pictures and/or images and associated orientation information. The mobile device 410 includes a display 420, storage 430, a processing unit 440, an accelerometer 450, and a digital camera 460.

The processing unit 440 can be configured to read the directional information from the accelerometer 450. For example, if accelerometer 450 is a three-axis accelerometer, processing unit 440 reads the ax, ay, and az values from accelerometer 450 when an image is captured using the camera (i.e., at approximately the same time, such that the obtained orientation information accurately represents the time at which the image was captured as compared to the time of capture of other images). The processing unit 440 can store the orientation information and captured pictures or images in the storage 430.

The processing unit 440 can use the orientation information to adjust the captured picture or image and cause the adjusted captured picture or image to be displayed on the display 420. The processing unit 440 can also transmit or stream raw or adjusted pictures or images (e.g., using a wireless data network component (not shown), such as a WiFi transceiver or a mobile data network transceiver).

Example image matching

The techniques and schemes described herein can be used to match pictures and video images. For example, the orientation information can be used to facilitate the task of matching features or points of interest within pictures and video images.

When orientation information is not available for a picture or image, image matching can be difficult and computationally expensive. For example, if it is not known that two pictures have the same direction (e.g., the same roll, as a rotation within the (y, x) plane depicted in fig. 3), then a special type of descriptor that can be performed regardless of such rotation is needed when comparing points of interest. However, these special types of descriptors are more computationally expensive than simple descriptors and in some cases tend to be numerically inaccurate. An example of one type of rotation invariant point of interest descriptor is the Speeded Up Robust Features (SURF) descriptor.

Using the techniques and solutions described herein, pictures and images can be adjusted to compensate for device rotation before performing imaging and matching tasks. For example, the orientation information (e.g., ax, ay, and az values) is used to adjust two or more pictures or images to compensate for device rotation (e.g., within the (y, x) and/or (y, z) planes as depicted in fig. 3). Once the pictures/images are adjusted, features or points of interest within a first picture or image can be selected and compared to potentially matching features or points of interest within one or more other pictures or images. Since the picture or image has been adjusted to compensate for differences in device orientation (in one or more planes of rotation), point of interest matching can be performed using a relatively simple pixel comparison operation that is less prone to numerical errors and computationally less expensive than operations involving the rotation invariant descriptors discussed above (assuming that the images have the same scale, otherwise compensation for differences in scale/scale needs to be performed).

Image matching using orientation information can help improve efficiency and accuracy in a number of aspects. For example, the orientation information is used to adjust pictures or images in a picture or image library or database to compensate for device rotation to facilitate searching for features or points of interest. As another example, pictures or images can be adjusted to compensate for device rotation to facilitate stitching or connecting two or more pictures together (e.g., to create a panoramic picture or image).

Fig. 5 is an example illustration 500 showing how image matching and stitching may be accomplished using orientation information. The matching example depicted at 510 shows two pictures (which may also be a single video image) to be matched. The two pictures depicted at 510 (left and middle pictures) are captured using the same device orientation relative to left-right rotation (roll). To match the two pictures, a point of interest is selected within the first picture (e.g., a 9x9 pixel grid depicted at 512). The selected interest point 512 in the first picture is used to find a matching interest point in the second picture (e.g., a 9x9 pixel grid depicted at 514). Since the two pictures have the same orientation, a relatively simple pixel matching operation can be used when matching the points of interest 512 and 514. In this example 510, the two images are then stitched (or connected) together based on the matching points of interest 512 and 514 (and possibly other matching points of interest) to create a single stitched picture (the right picture).

The matching example depicted at 520 also displays two pictures (which may also be a single video image) to be matched. The first (left) picture depicted at 520 is captured by a camera that is rotated (rolled) by approximately 45 degrees. The first picture depicted at 520 is not adjusted to compensate for device rotation. The second (middle) picture depicted at 520 is captured by a horizontal (no roll) camera. As a result, in order to match a point of interest within the first picture depicted at 520 with a point of interest within the second picture depicted at 520, a rotation invariant point of interest descriptor (e.g., a 9x9 grid of pixels as shown at 522 and 524) needs to be used. This is a less accurate and more computationally expensive operation than the pixel matching of example 510 (indicating matching points of interest or creating a stitched image is difficult or impossible, as with the question mark shown in 520).

The matching example depicted at 530 shows two pictures (which may also be a single video image) to be matched. The first (left) picture depicted at 530 is captured by a camera that is rotated (rolled) by approximately 45 degrees. However, unlike the corresponding first picture depicted at 520, the first picture in 530 has been adjusted based on the orientation information to compensate for the device rotation. Since the first and second (intermediate) pictures at 530 actually have the same device orientation, the points of interest can be more accurately and efficiently matched (as depicted in the point of interest pixel grids 532 and 534). For example, the points of interest can be matched using a simple pixel matching operation instead of the less accurate and more computationally expensive rotation invariant descriptors required for matching in 520. By matching points of interest within the first and second pictures within 530, the first and second pictures can be stitched together to create a third (right) image within 530.

Example method for image matching

FIG. 6 shows an exemplary method 600 for matching images using orientation information. The matched image data can include still picture data and video image data. The computing device that captured the picture or image can perform the matching, or the computing device can perform the method using the orientation information and image data received from another computing device or camera.

At 610, image data and associated orientation information (e.g., ax, ay, and az values) for the first captured image are obtained. At 620, image data and associated orientation information for the second captured image is obtained. In a particular implementation, the orientation information of the first and second images includes three-axis accelerometer information obtained during image capture.

At 630, at least one of the first and second captured images is adjusted using the orientation information to compensate for differences in camera orientation within the at least one plane of rotation. For example, both images are adjusted to compensate for differences in camera direction within the (y, x) plane (as depicted in fig. 3) and/or both images are adjusted to compensate for differences in camera direction within the (y, z) plane (as depicted in fig. 3). In order to adjust the two images to have virtually the same capture direction, one or both images can be adjusted in at least one plane of rotation relative to the reference camera direction.

At 640, at least one point of interest is matched between the first and second captured images as adjusted. For example, when the pictures actually have the same camera orientation, a simple pixel matching operation can be used when matching the points of interest.

Various results can be provided (e.g., output) by the example method 600. For example, an indication can be output (e.g., displayed) as to whether the point of interest is matched between the first and second images. The results can also be used to perform other tasks, such as image stitching, in which case the stitched (or panoramic) image can be output (e.g., displayed or saved).

IX. example Mobile device details

Fig. 7 depicts a detailed example of a mobile device 700 capable of implementing the techniques and aspects described herein. The mobile device 700 includes various optional hardware and software components, shown generally at 702. Any component 702 in the mobile device may communicate with any other component, but not all connections are shown for ease of illustration. The mobile device may be any of a variety of computing devices (e.g., cellular phone, smart phone, handheld computer, laptop computer, notebook computer, tablet device, netbook, Personal Digital Assistant (PDA), camera, camcorder, etc.) and may allow wireless two-way communication with one or more mobile communication networks 704, such as Wi-Fi, cellular or satellite networks.

The illustrated mobile computing device 700 may include a controller or processor 710 (e.g., signal processor, microprocessor, ASIC, or other control and processing logic circuitry) for performing tasks such as signal coding, data processing, input/output processing, power control, and/or other functions. An operating system 712 can control allocation and usage of the components 702 and support for one or more application programs 714. The application programs may include public mobile computing applications (e.g., email applications, calendars, contact managers, web browsers, messaging applications), or any other computing application.

The illustrated mobile computing device 700 may include memory 720. Memory 720 can include non-removable memory 722 and/or removable memory 724. Removable memory 722 may include RAM, ROM, flash memory, a hard disk, or other well known memory storage technologies. The removable memory 724 may include flash memory or a Subscriber Identity Module (SIM) card, which are well known in GSM communication systems, or other well-known memory storage technologies, such as "smart cards. Memory 720 may be used to store data and/or code for running operating system 712 and application programs 714. Example data may include web pages, text, images, sound files, video data, or other data sets sent to and/or received from one or more network servers or other devices via one or more wired or wireless networks. The memory 720 can be employed to store a subscriber identifier, such as an International Mobile Subscriber Identity (IMSI), and an equipment identifier, such as an International Mobile Equipment Identifier (IMEI). Such identifiers may be transmitted to a network server to identify users and equipment.

The mobile device 700 can support one or more input devices 730, such as a touch screen 732, a microphone 734, a camera 736 (e.g., capable of capturing still and/or video images), a physical keyboard 738, and a trackball 740, and one or more output devices 750, such as a speaker 752 and a display 754. Other possible output devices (not shown) may include piezoelectric or other haptic output devices. Some devices may serve more than one input/output function. For example, the touchscreen 732 and the display 754 may be combined in a single input/output device.

The wireless modem 760 may be coupled to an antenna (not shown) and may support bi-directional communication between the processor 710 and external devices, as is well understood in the art. The modem 760 is shown generically and may include a cellular modem for communicating with the mobile communications network 704 and/or other radio-based modems (e.g., bluetooth 764 or Wi-Fi 762). The wireless modem 760 is typically configured to communicate with one or more cellular networks, such as a GSM network for data and voice communications within a single cellular network, between cellular networks, or between a mobile device and the Public Switched Telephone Network (PSTN).

The mobile device may further include at least one input/output port 780, a power supply 782, a satellite navigation system receiver 784, such as a Global Positioning System (GPS) receiver, an accelerometer 786, a transceiver 788 (for wirelessly transmitting analog or digital signals), and/or a physical connector 737, which may be a USB port, an IEEE 1394 (firewire) port, and/or an RS-232 port. The illustrated components 702 are not required or all-inclusive, as any components can be deleted and other components can be added.

The mobile device 700 can implement the techniques described herein. For example, processor 710 can obtain orientation information (e.g., ax, ay, and az values) from accelerometer 786 during capture of a picture or video image by camera 736. The orientation information can be used to adjust (e.g., by the processor 710) the captured picture or video image to compensate for rotation of the mobile device 700. The picture or video image (in raw or adjusted form) can be saved in memory (722 or 724). Further, the orientation information can be saved to memory (722 or 724) (e.g., in the same file as the picture or video image, or in a separate file). The picture or video image may be transmitted (e.g., streamed) over a wireless network (e.g., via wireless modem 760 or transceiver 788) in raw or adjusted form, with or without associated direction information.

X. example implementation Environment

FIG. 8 illustrates one general example of a suitable implementation environment 800 in which the described embodiments, techniques, and technologies may be implemented.

In the example environment 800, various types of services (e.g., computing services) are provided by 810. For example, the cloud 810 may comprise a collection of computing devices, which may be centrally located or distributed, that provide cloud-based services to various types of users and devices via a network such as the internet. The implementation environment 800 may be used to implement computing tasks in a different manner. For example, some tasks (e.g., processing user input and presenting a user interface) may be performed on a local computing device (e.g., connected device 830 and 832), while other tasks (e.g., storing data to be used in subsequent processing) may be performed in cloud 810.

In the example environment 800, the cloud 810 provides services to connected devices 830 and 832 having various screen capabilities. Connected device 830 represents a device having a computer screen (e.g., a mid-sized screen). For example, the connected device 830 may be a personal computer, such as a desktop computer, a laptop computer, a notebook, a netbook, and so forth. Connected device 831 represents a device having a mobile device screen (e.g., a small screen). For example, the connected device 831 can be a mobile phone, a smart phone, a personal digital assistant, a tablet computer, and the like. Connected device 832 represents a device with a large screen. For example, connected device 832 may be a television screen (e.g., a smart television) or another device connected to a television (e.g., a set-top box or game console), and so on. One or more of connected devices 830-832 may include touch screen capabilities. Devices without screen capabilities may also be used in the example environment 800. For example, the cloud 810 can provide services to one or more computers (e.g., server computers) that do not have displays.

Services may be provided by the cloud 810 through a service provider 820, or through other providers of online services (not depicted). For example, the cloud service may be customized to the screen size, display capabilities, and/or touch screen capabilities of a particular connected device (e.g., connected device 830 and 832).

In the example environment 800, the cloud 820 uses, at least in part, a service provider 830 to provide the techniques and solutions described herein to various connected devices 830 and 832. For example, service provider 820 may provide a centralized solution to various cloud-based services. Service provider 820 can manage service subscriptions for users and/or devices (e.g., connected devices 830 and 832 and/or their respective users).

For example, connected devices such as device 831 may include cameras and accelerometers capable of obtaining digital pictures or video images and associated orientation information. Connected device 831 can communicate the picture or image and associated orientation information to other connected devices (e.g., 830) and 832 or to service provider 820 (e.g., for display or storage). Service provider 820 can perform various image processing tasks by using the received picture/image data and associated orientation information (e.g., using the orientation information to adjust the image to compensate for device rotation, match points of interest within the adjusted image, perform stitching operations on the adjusted image, and distribute the image to other connected devices, or using the orientation information to adjust the image to compensate for device rotation, then distribute the image to other connected devices).

Example substitutions and variations

Although some of the operations of the disclosed methods are described in a particular, sequential order for convenient presentation, it should be understood that this method of description encompasses rearrangement, unless a particular ordering is required by specific language set forth below. For example, in some cases, operations described sequentially may be rearranged or performed concurrently. Moreover, for the sake of brevity, the attached figures may not show the various ways in which the disclosed methods can be used in conjunction with other methods.

Any of the disclosed methods can be implemented as stored on one or more computer-readable media (e.g., non-transitory computer-readable media, such as one or more optical media discs, volatile memory components (such as DRAM or SRAM), or non-volatile memory components (such as hard drives))

And computer-executable instructions executing on a computer (e.g., any commercially available computer, including smart phones or other mobile devices that include computing hardware). Any of the computer-executable instructions for implementing the disclosed techniques, as well as any data created and used during implementation of the disclosed embodiments, can be stored on one or more computer-readable media (e.g., non-transitory computer-readable media). The computer-executable instructions may be, for example, a dedicated software application or software application that is accessed or downloaded via a web browser or other software application, such as a remote computing application. Such software can be executed, for example, on a single local computer (e.g., any suitable commercially available computer) or in a network environment (e.g., via the internet, a wide-area or local-area network, a client-server network (such as a cloud computing network), or other such network) using one or more network computers.

For clarity, only certain selected aspects of software-based implementations are described. Other details known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any particular computer language, or program. For example, the disclosed technology may be implemented by software written in C + +, Java, Perl, JavaScript, AdobeFlash, or any other suitable programming language. Also, the disclosed techniques are not limited to any particular computer or type of hardware. Certain details of suitable computers and hardware are well known and need not be set forth in detail in this disclosure.

Further, any of the software-based embodiments (including, for example, computer-executable instructions for causing a computer to perform any of the disclosed methods) may be uploaded, downloaded, or accessed remotely via suitable communication means. Such suitable communication means include, for example, the internet, the world wide web, an intranet, software applications, electrical cables (including fiber optic cables), magnetic communication means, electromagnetic communication means (including RF, microwave, and infrared communications), electronic communication means, or any such communication means.

The disclosed methods, apparatus, and systems should in no way be construed as limiting. Rather, the invention is directed to all novel and non-obvious features and aspects of the various disclosed embodiments, alone or in various combinations and subcombinations with one another. The disclosed methods, apparatus, and systems are not limited to any specific aspect or feature or combination thereof, nor do the disclosed embodiments require that any one or more specific advantages be present or problems be solved. Accordingly, all that comes within the spirit and scope of these claims is claimed as the invention.

Claims (13)

1. A method, implemented at least in part by a computing device, for capturing and processing video data, the method comprising:

for each of a plurality of video images:

obtaining, by the computing device, orientation information (110) of the computing device at a time the video image was captured, wherein the computing device comprises a three-axis accelerometer, and wherein the orientation information is obtained from the three-axis accelerometer;

adjusting, by the computing device, the captured video image based on the orientation information (120), wherein adjusting the captured video image based on the orientation information comprises:

obtaining a reference direction, the reference direction being a direction of the computing device at a time an initial video image of the video data was captured; and

adjusting the captured video image to return the captured video image to the reference direction; and

outputting, by the computing device, for display of the adjusted captured video image (130);

wherein the obtaining, the adjusting, and the displaying are performed in real-time during recording of the plurality of video images by the computing device for each of the plurality of video images such that the adjusted captured video image is a video of a next image being captured.

2. The method of claim 1, wherein the direction information comprises:

a first value representing acceleration along an x-axis of the computing device;

a second value representing acceleration along a y-axis of the computing device; and

a third value representing acceleration along a z-axis of the computing device.

3. The method of claim 1, wherein adjusting the captured video image based on the orientation information comprises compensating for a difference between the obtained orientation information of the captured video image and a reference orientation.

4. The method of claim 1, wherein adjusting the captured video image based on the orientation information comprises:

calculating a rotation angle in a first plane using the direction information of the captured video image;

determining a difference in rotation angle between the calculated rotation angle in the first plane and a reference rotation angle in the first plane; and

the captured video image is adjusted to compensate for the difference in rotation angle.

5. The method of claim 1, wherein adjusting the captured video image based on the orientation information comprises:

calculating a rotation angle in a first plane using the direction information of the captured video image;

calculating a rotation angle in a second plane using the orientation information of the captured video image, wherein the first plane is different from the second plane;

determining a difference in rotation angle between the calculated rotation angle in the first plane and a reference rotation angle in the first plane;

determining a difference in rotation angle between the calculated rotation angle in the second plane and the reference rotation angle in the second plane; and

the captured video image is adjusted to compensate for the difference in rotation angle in the first plane and the difference in rotation angle in the second plane.

6. The method of claim 1, further comprising:

for each of the plurality of video images:

storing, by the computing device, the adjusted captured video image.

7. The method of claim 6, further comprising:

for each of the plurality of video images:

storing, by the computing device, direction information associated with the stored adjusted captured video image, wherein the direction information is stored as a plurality of values representing acceleration along an axis of the computing device or as a plurality of angles of rotation of the computing device.

8. A mobile device (410) for capturing and processing video data, the mobile device comprising:

an accelerometer (450);

storing (430);

a digital camera (460); and

a processing unit (440) configured to perform operations comprising:

for each of a plurality of video images:

obtaining, by the processing unit, orientation information (210) of the mobile device from the accelerometer, the orientation information indicating an orientation of the mobile device when capturing video images;

adjusting the captured video image in real-time based on the orientation information, wherein adjusting the captured video image based on the orientation information comprises:

obtaining a reference direction, the reference direction being a direction of the mobile device at a time an initial video image of the video data was captured; and

adjusting the captured video image to return the captured video image to the reference direction; and;

storing, by the processing unit, the adjusted captured video image and the orientation information in storage (220) as subsequent video being captured, wherein the orientation information is associated with its respective captured video image.

9. The mobile device of claim 8, wherein the plurality of captured video images are stored in a first data file, and wherein the orientation information is stored in a second data file different from the first data file.

10. The mobile device of claim 8, wherein the accelerometer is a three-axis accelerometer, wherein a plurality of captured video images and the orientation information are stored in the same data file, and wherein the orientation information is stored as a plurality of values representing acceleration along an axis of a computing device or as a plurality of angles of rotation of the computing device.

11. The mobile device of claim 8, further comprising:

for each of the plurality of video images:

streaming the obtained orientation information and the captured video image from the mobile device to a network server.

12. The mobile device of claim 8, further comprising:

for each of the plurality of video images:

adjusting, by the mobile device, the captured video image based on the orientation information; and

streaming the adjusted captured video image from the mobile device to a network server.

13. The mobile device of claim 8, wherein the accelerometer is a three-axis accelerometer, and wherein the orientation information comprises:

a first value representing acceleration along an x-axis of the mobile device;

a second value representing acceleration along a y-axis of the mobile device; and

a third value representing acceleration along a z-axis of the mobile device.