WO2016202111A1 - Audio output method and apparatus based on photographing - Google Patents

Abstract

Embodiments of the present disclosure provide an audio output method and apparatus based on photographing. The method comprises: determining, when a camera collects data of a preview image, one or more target objects in the data of the preview image; calculating sound-field angles of the one or more target objects relative to a loudspeaker; calculating deviation angles of the one or more target objects relative to the loudspeaker; and driving, upon reception of a photographing instruction, the loudspeaker to output audio according to the sound-field angles and the deviation angles.

Description

Photo-based audio output method and device

Cross-reference to related applications

The present application claims priority to Chinese Patent Application No. 20151034529, filed on Jun. 19, 2015, which is hereby incorporated by reference.

Technical field

The present disclosure relates to the field of audio processing technologies, and in particular, to a camera-based audio output method and a camera-based audio output device.

Background technique

With the development of technology, various electronic devices, especially mobile devices such as mobile phones and tablet computers, are increasingly used in people's work, study, and daily communication.

Most of the electronic devices such as mobile phones and tablet computers are equipped with cameras, making photography an important application of electronic devices.

At present, in the scene of photographing, a prompt sound is usually sent through the speaker to prompt the user and the time when the photographed person takes a photo.

However, due to factors such as environmental noise, there may be cases where the person being photographed cannot hear the prompt sound. If the person being photographed does not hear the prompt, and the picture is blurred when the camera is moved a little, especially when shooting a child, it is difficult to take pictures of the child because it does not attract the attention of the child.

If the photo is blurred, it is necessary to re-photograph, waste the resources of the electronic device, and the photographing efficiency is low and the cost is high.

Moreover, in scenes with a large number of people, the sound of the photo may be transmitted to other people (non-photographed), affecting others.

Summary of the invention

In view of the above problems, embodiments of the present disclosure have been made in order to provide a photo-based audio output method and a corresponding photo-based audio output device that overcome the above problems or at least partially solve the above problems.

In order to solve the above problem, an embodiment of the present disclosure discloses an audio output method based on photographing, including:

Determining one or more target objects in the preview image data when the camera captures the preview image data;

Calculating an acoustic field angle relative to the speaker for the one or more target objects;

Calculating a deflection angle relative to the speaker for the one or more target objects;

When receiving the photographing instruction, the driving speaker outputs the audio in accordance with the sound field angle and the deflection angle.

The embodiment of the present disclosure further discloses a camera-based audio output device, including:

a target object determining module, configured to determine one or more target objects in the preview image data when the camera captures the preview image data;

a sound field angle calculation module, configured to calculate an acoustic field angle relative to the speaker for the one or more target objects;

a deflection angle calculation module, configured to calculate a deflection angle relative to the speaker for the one or more target objects;

And an audio directional output module, configured to drive the speaker to output audio according to the sound field angle and the deflection angle when receiving the photographing indication.

The embodiments of the present disclosure include the following advantages: the embodiment of the present disclosure calculates the sound field angle and the deflection angle by one or more target objects in the preview image data, and outputs audio according to the sound field angle and the deflection angle when photographing, and prompts the photographing. The sound is output to the target object in the actual target, so that the photographer can hear the prompt sound under the environment and other factors, thereby improving the success rate of the photograph, avoiding the chance of re-photographing, and reducing the electronic device. Waste of resources, improve the efficiency of photographing, and reduce the cost of photographing; in addition, areas other than directional output audio generally do not hear the sound of photographing, reducing the impact on others.

DRAWINGS

1 is a flow chart showing the steps of an embodiment of a camera-based audio output method according to the present disclosure;

2A and 2B are diagrams showing an example of a scene of a sound field angle and a deflection angle of the present disclosure;

3A and 3B are diagrams showing an example of calculation of an acoustic field angle of the present disclosure;

4A to 4C are diagrams showing an example of calculation of a deflection angle of the present disclosure;

FIG. 5 is a structural block diagram of an embodiment of a photo-based audio output device of the present disclosure.

detailed description

The above-described objects, features and advantages of the present disclosure will become more apparent from the aspects of the appended claims.

Referring to FIG. 1 , a flow chart of steps of an embodiment of a camera-based audio output method according to the present disclosure is shown. Specifically, the method may include the following steps:

Step 101: When the camera collects preview image data, determine one or more target objects in the preview image data;

It should be noted that the embodiments of the present disclosure may be applied to various electronic devices, such as a mobile phone, a tablet computer, a personal digital assistant, a wearable device (such as glasses, a watch, etc.), etc., and the embodiment of the present disclosure does not limit this. .

The operating system of the electronic device may include Android (Android), IOS, Windows Phone, Windows, etc., and generally supports the operation of the camera and the speaker.

The camera is a hardware on the electronic device, which can be used for photographing and photographing, and can be front-mounted (in the same direction as the screen of the electronic device) or rear-mounted (in the opposite direction to the screen of the electronic device), and the implementation of the present disclosure This example does not limit this.

In practical applications, the image generated by the scene through the lens of the camera (Lens) is projected onto the surface of the image sensing processor (Sensor), and then converted into an electrical signal, which is converted by A/D (Analog to Digital Conversion). For digital image signals, digital image signals are compressed by a digital signal processing chip (DSP) or an encoding library and converted into a specific image file format, which is transmitted to a mobile device's Central Processing Unit (CPU) through a data bus. Processing can be displayed on the display of the electronic device.

It should be noted that the preview is relative to the photographing, and the preview image data is image data that is provided to the user for adjustment and selection before being stored in the photograph, and is saved in the cache.

The camera captures a series of preview image data, that is, preview image data of multiple frames. In the embodiment of the present disclosure, the sound field angle and the deflection angle can be continuously calculated while the camera continuously collects the preview image data until the user takes a photo. .

Certainly, a selection control may also be provided, by which the user can select whether to output audio at the time of photographing, and the embodiment of the present disclosure can determine the state of the selection control before determining one or more target objects in the preview image data. If the state of the control is selected to select the directional output audio, the step of determining one or more target objects in the preview image data may be continued. If the state of the selected control is undirected output audio, the step of determining one or more target objects in the preview image data may not be performed to reduce resource consumption of the electronic device.

In an optional embodiment of the present disclosure, step 101 may include the following sub-steps:

Sub-step S11, detecting a face in the preview image data, and determining one or more target objects.

In general, when the camera captures preview image data, the camera can perform auto focus through face detection. The so-called face detection can refer to calibrating the position and size of all faces from one frame of preview image data.

Then, in the embodiment of the present disclosure, the object that is successfully detected may be identified as the target object, and the photographed person actually mapped by the target object may be a person.

Further, taking the Android (Android) system as an example, two professional APIs (Application Program Interface), android.media.FaceDetector and android.media.FaceDetector are provided in the Android system. Face, which implements face detection on a bitmap.

In another optional embodiment of the present disclosure, step 101 may include the following sub-steps:

Sub-step S11, when receiving the focus operation instruction, determining that one or more human faces in the preview image data corresponding to the focus operation instruction are one or more target objects;

In the embodiment of the present disclosure, the user can perform manual focusing, and the focus operation instruction is triggered by clicking the preview image data, selecting the focus frame, and the like, and the camera can perform the focusing operation on the object selected by the user according to the focus operation instruction.

In the embodiment of the present disclosure, the object corresponding to the focus operation indication may be identified as the target object, and the photographed person actually mapped by the target object may be a person, an animal, or a still life.

or,

Sub-step S12, when the cancellation operation instruction is received, the determined one or more target objects are canceled.

In the case where the person who recognizes the camera autofocus is not the desired subject, the user can trigger the cancel operation instruction by clicking the target object or the like, and cancel the target object.

In a specific implementation, the target object determined in the preview image data may be one, or may be multiple (ie, two or more), which is not limited by the embodiment of the present disclosure.

Step 102: Calculate an acoustic field angle with respect to the speaker for the one or more target objects;

The speaker can be a hardware that outputs audio, such as a miniature piezoelectric film ultrasonic sensor.

Assuming that the speaker can hear the audio within a certain range when the speaker is directed to output audio, and the other person outside the range cannot generally hear the audio, the angle of the range with respect to the speaker may be referred to as the sound field angle.

That is, as shown in FIGS. 2A and 2B, the sound field angle R can be an angular range in which the speaker 201 can hear the audio in the case of outputting audio.

In an optional embodiment of the present disclosure, step 102 may include the following sub-steps:

Sub-step S21, measuring a target distance between the speaker and the one or more target objects;

It should be noted that the target distance refers to the linear distance between the speaker and the target object as a whole, and does not necessarily mean the linear distance between the speaker and a certain target object.

In an optional example of an embodiment of the present disclosure, the sub-step S21 may further include the following sub-steps:

Sub-step S211, when the target object is one, acquiring a candidate distance between the camera and the target object as a target distance;

or,

Sub-step S212, when the target object is multiple, respectively acquiring a plurality of candidate distances between the camera and the plurality of target objects;

Sub-step S213, calculating the target distance by using the plurality of candidate distances.

Since the camera and the speaker are both disposed in the same electronic device, the difference between the two is generally small, so the candidate distance between the camera and the target object (the linear distance between the two), and, the speaker and the target object The difference between the candidate distances (the linear distance between the two) is small and is generally within the acceptable difference.

And, the candidate distance between the camera and the target object is calculated by previewing the image data, and therefore, in this example, in order to avoid adding additional hardware, the candidate distance between the camera and the target object can be replaced with the speaker and the target object. Candidate distance between.

When the target object is a single, the candidate distance between the camera and the target object can be directly set as the target distance between the speaker and the target object.

When there are multiple target objects, multiple candidates between the camera and multiple target objects can be used. The distance between the calculated speaker and the target object is calculated, such as calculating a plurality of candidate distance averages, selecting a maximum value of the plurality of candidate distance species, selecting a minimum of the plurality of candidate distances, and the like, which is not Limit it.

Further, the candidate distance between the camera and the target object can be calculated by one or more of the following methods:

1, stereo vision.

Imitating human stereo perception analysis method, the binocular or multi-view camera observes the same target object at different viewpoints, acquires two-dimensional images of target objects at different viewing angles, and calculates the positional deviation of image pixels, ie, parallax, by triangulation principle. Get the 3D information of the target object.

2. Motion ranging method.

The two-dimensional image of the continuous target object is acquired by the barrage camera at different times or different spatial positions, and the distance and other parameters of the target object are calculated by the time or space change of the target object in the two-dimensional image sequence.

3. Monocular ranging.

Image-based ranging methods in monocular ranging include: Depth from Focus (DFF) and Depth from Defocus (DFD).

The focus ranging method captures a series of image data by adjusting the optical writing parameters, finds the clearest image data among the image data, and calculates the distance based on the imaging parameters of the image data using the imaging principle of geometric optics.

The defocusing distance method is based on the principle that the defocusing degree of the object is larger and the image is more blurred. The two or three frames of image data captured under different optical parameters are used to determine the diffusion parameter of the scattered focus diffusion function, according to the defocus diffusion parameter. The depth calculation is performed on the relationship with the distance of the target object.

Of course, the above calculation manner is only an example. When the embodiment of the present disclosure is implemented, other calculation manners may be set according to actual conditions, and the embodiment of the present disclosure does not limit this. In addition, in addition to the foregoing calculation manners, other calculation methods may be used by those skilled in the art according to actual needs, and the embodiments of the present disclosure do not limit this.

In addition, in addition to multiplexing the candidate distance between the camera and the target object, the candidate distance between the speaker and the target object can be directly measured by the active ranging method, that is, using a beam such as a laser or a light having a certain texture structure to find a target object, By analyzing the texture deformation or measurement of the reflected light of the target object The propagation time of the light speed is used to determine the distance of the object, and the embodiment of the present disclosure does not limit this.

Sub-step S22, acquiring an audition range distance that matches the one or more target objects;

It is assumed that when the audio is directionally output, the audio can be heard within a certain range, and the audio is generally not heard outside the range, and the distance of the range is called the audition range distance.

That is, the range of the audition range can be the distance from which the range of the directional output audio can be heard.

With the embodiment of the present disclosure, the matching audition range distance can be set in advance according to the target object, for example, the audition range of one target object is 35 cm, the audition range of two target objects is 45 cm, and the like.

Of course, it is also possible to calculate a suitable audition range distance according to the focal length, the difference of the target object in the preview image data, and the like, and the embodiment of the present disclosure does not limit this.

Sub-step S23, calculating a sound field angle according to the target distance and the distance of the audition range.

In a specific implementation, the target distance and the listening range distance may be used, and the sound field angle is calculated according to a trigonometric function relationship.

In one embodiment, the isosceles triangle is constructed with the target distance being high and the audition range distance as the base, and the sound field angle is calculated according to the following trigonometric relationship:

tanR/2=(K/2)/L

Where R is the sound field angle, K is the distance of the audition range, and L is the target distance.

Of course, in addition to the tangent tan function, other trigonometric function relationships may be used to calculate the acoustic field angle, which is not limited by the embodiments of the present disclosure.

For example, as shown in FIG. 3A, when the target object is one, the candidate distance between the camera and the target object is measured as L ₀ , that is, the target distance between the speaker 301 and the target object is L ₀ , and the distance of the audition range of one target object is For K ₀ , the isosceles triangle is constructed with the target distance L ₀ as the height and the audition range distance K ₀ as the base, and the sound field angle R _{0 is} calculated according to the following trigonometric relationship:

tanR ₀ /2=(K ₀ /2)/L ₀

For another example, as shown in FIG. 3B, when the target object is three, the candidate distance distribution of the camera and the target object is measured as L ₂ , L ₃ , L ₄ , that is, the target distance L _{1 of the} speaker 301 and the target object is L ₁ ( L ₂ +L ₃ +L ₄ )/3, the distance of the audition range of the three target objects is K ₁ , and the isosceles triangle is constructed with the target distance L ₁ as the height and the audition range distance K ₁ as the base. The function relationship calculates the sound field angle R ₁ :

tanR ₁ /2=(K ₁ /2)/L ₁

Step 103: Calculate a deflection angle with respect to the speaker for the one or more target objects;

As shown in FIGS. 2A and 2B, the deflection angle S may be an angle at which the subject actually mapped by the target object deviates from the forward direction of the speaker 201.

In an optional embodiment of the present disclosure, step 103 may include the following sub-steps:

Sub-step S31, projecting the target object in the preview image into a preset coordinate system, the coordinate system is constructed based on the position of the speaker;

Since the preview image data collected by the camera is generally proportional to the actual scene, the value of the target object deviating from the speaker in the preview image data is generally the same as the value of the bias angle of the subject with respect to the speaker mapped by the target object in practice. it's the same.

With the embodiment of the present disclosure, the coordinate system can be constructed in advance based on the position of the speaker.

It should be noted that the position of the speaker and the sound hole (the hole on the outer casing of the electronic device, the sound emitted by the speaker can propagate through the sound hole) is usually opposite, and the position of the speaker on the electronic device (ie, electronically) The back side of the device acts as a projection surface, and the position at which the speaker is projected onto the projection surface is often coincident with the sound hole. Therefore, the coordinate system can also be directly constructed based on the sound hole.

Among them, the projection is a method in which a projection line is projected through an object (such as a speaker) to a selected projection surface, and a graphic is obtained on the surface.

In the embodiment of the present disclosure, a coordinate system such as a Cartesian coordinate system may be constructed with the speaker or the sound hole as an origin.

The plane of the coordinate system is used as a projection plane, and the target object in the preview image is projected into the coordinate system to calculate the deflection angle.

Sub-step S32, in the coordinate system, calculating focus coordinates of the one or more target objects;

The focus coordinate, which can be the coordinates of the focus when the target object is focused.

In an optional example of an embodiment of the present disclosure, sub-step S32 may further include the following sub-steps:

Sub-step S321, when the target object is one, searching for the first coordinate of the upper left corner of the target object and the second coordinate of the lower right corner;

Sub-step S322, calculating an average value of the first coordinate and the second coordinate, sitting as a focus Standard

In this example, the target object is an area, and the midpoint of the area can be used as the focus coordinate.

or,

Sub-step S323, when the target object is multiple, find the third coordinate of the upper left corner of the leftmost target object, the fourth coordinate of the lower right corner, and the fifth coordinate of the upper left corner of the rightmost target object, The sixth coordinate of the lower right corner;

Sub-step S324, respectively calculating an average value of the third coordinate and the fourth coordinate, and an average value of the fifth coordinate and the sixth coordinate as focus coordinates.

In this example, the target object is an area, and the midpoint of the area can be used as the focus coordinate.

If there are multiple target objects, the focus coordinates of the leftmost target object and the focus coordinates of the rightmost target object can be calculated, for a total of two focus coordinates.

Sub-step S33, the deflection angle is calculated using the focus coordinates.

In a specific implementation, the focus angle can be used to calculate the deflection angle according to a trigonometric function relationship.

In an optional example of the embodiment of the present disclosure, when the target object is one, a right-angled triangle may be constructed with the value of the X-axis coordinate of the focus coordinate and the value of the Y-axis coordinate being a right-angled edge, and calculated according to the following trigonometric relationship. Deflection angle:

tanS=X ₀ /Y ₀

Where S is the deflection angle, X ₀ is the value of the X-axis coordinate of the focus coordinate, and Y ₀ is the value of the Y-axis coordinate of the focus coordinate.

Of course, in addition to the tangent tan function, other trigonometric function relationships may be used to calculate the deflection angle, which is not limited by the embodiments of the present disclosure.

In another optional example of an embodiment of the present disclosure, sub-step S33 may further include the following sub-steps:

Sub-step S331, when the target object is multiple, calculate a first candidate deflection angle by using focus coordinates of the leftmost target object;

Sub-step S332, calculating a second candidate deflection angle by using focus coordinates of the rightmost target object;

Sub-step S333, if the leftmost target object and the rightmost target object are located in the two speakers Side, the first feature angle is set to a deflection angle;

Wherein the first feature angle is a half of a difference between the first candidate deflection angle and the second candidate deflection angle;

Sub-step S334, if the leftmost target object and the rightmost target object are located on the same side of the speaker, the second feature angle is set to a deflection angle;

The second feature angle is half of a sum of the first candidate deflection angle and the second candidate deflection angle.

In this example, for each focus coordinate, a right-angled triangle may be constructed with a value of an X coordinate of the focus coordinate and a value of the Y coordinate, and a first candidate deflection angle and a second candidate deflection angle may be calculated according to a trigonometric function relationship. If the leftmost target object and the rightmost target object are located on either side of the forward direction of the speaker, then the deflection angle:

S=(S ₁ -S ₂ )/2

If the leftmost target object and the rightmost target object are located on the same side of the speaker in the forward direction (such as the left or right side), the deflection angle:

S=(S ₁ +S ₂ )/2

Where S is the deflection angle, S ₁ is the first candidate deflection angle, and S ₂ is the second candidate deflection angle.

It should be noted that the above calculation

For example, as shown in FIG. 4A, the XY coordinate system is constructed with the projection O of the speaker as a dot. When the target object is one, the upper left corner of the target object is point A (X ₁ , Y ₁ ), and the lower right corner is a point. B(X ₂ , Y ₂ ), then the focus coordinate U((X ₁ +X ₂ )/2, (Y ₁ +Y ₂ )/2), then the X-axis coordinate of the focus coordinate U (X ₁ +X ₂ The /2 and Y-axis coordinates (Y ₁ +Y ₂ )/2 are right-angled edges, and a right-angled triangle is constructed. The deflection angle S _{3 is} calculated according to the following trigonometric relationship:

tanS ₃ = (X ₁ +X ₂ )/2/(Y ₁ +Y ₂ )/2;

For another example, as shown in FIG. 4B, the XY coordinate system is constructed with the projection O of the speaker as a dot. When the target object is three, the upper left corner of the leftmost target object is the point C (X ₃ , Y ₃ ). , the lower right corner is the point D (X ₄ , Y ₄ ), then the focus coordinate V ((X ₃ +X ₄ )/2, (Y ₃ +Y ₄ )/2), the upper left corner of the rightmost target object is Point E (X ₅ , Y ₅ ), the lower right corner is point F (X ₆ , Y ₇ ), then the focus coordinate W ((X ₅ +X ₆ )/2, (Y ₅ +Y ₆ )/2);

Then, the X-axis coordinate (X ₃ +X ₄ )/2 and the Y-axis coordinate (Y ₃ +Y ₄ )/2 of the focus coordinate V are respectively a right-angled edge, and the X-axis coordinate of the focal coordinate W (X ₅ +X) ₆ )/2 and the Y-axis coordinate (Y ₅ +Y ₆ )/2 are right-angled sides, and a right-angled triangle is constructed, and the first candidate deflection angle S ₅ and the second candidate deflection angle S _{6 are} calculated according to the following trigonometric relationship:

tanS ₅ = (X ₃ +X ₄ )/2/(Y ₃ +Y ₄ )/2;

tanS ₆ = (X ₅ +X ₆ )/2/(Y ₅ +Y ₆ )/2;

Assuming S ₅ is 30° and S ₆ is 50°, the deflection angle S ₄ =(S ₆ -S ₅ )/2=10°, indicating that the three target objects are entirely shifted by 10° toward the right side of the speaker.

For another example, as shown in FIG. 4C, the XY coordinate system is constructed with the projection O of the speaker as a dot. When the target object is three, the upper left corner of the leftmost target object is the point G (X ₇ , Y ₇ ). , the lower right corner is point H (X ₈ , Y ₈ ), then the focus coordinate is M ((X ₇ +X ₈ )/2, (Y ₇ +Y ₈ )/2), the upper left corner of the rightmost target object For point I(X ₉ , Y ₉ ), the lower right corner is point J (X ₁₀ , Y ₁₀ ), then the focus coordinate is N((X ₉ +X ₁₀ )/2, (Y ₉ +Y ₁₀ )/2) ;

Then, the X-axis coordinate (X ₇ +X ₈ )/2 and the Y-axis coordinate (Y ₇ +Y ₈ )/2 of the focus coordinate V are respectively a right-angled edge, and the X-axis coordinate of the focal coordinate W (X ₉ +X) ₁₀ )/2 and the Y-axis coordinate (Y ₉ +Y ₁₀ )/2 are right-angled sides, and a right-angled triangle is constructed, and the first candidate deflection angle S ₈ and the second candidate deflection angle S _{9 are} calculated according to the following trigonometric relationship:

tanS ₈ = (X ₇ +X ₈ )/2/(Y ₇ +Y ₈ )/2;

tanS ₉ = (X ₉ +X ₁₀ )/2/(Y ₉ +Y ₁₀ )/2;

Assuming S ₈ is 50° and S ₉ is 30°, the deflection angle S ₇ =(S ₈ +S ₉ )/2=40°, indicating that the three target objects are entirely offset from the right side of the speaker by 40°.

Step 104: When receiving the photographing instruction, the driving speaker outputs the audio according to the sound field angle and the deflection angle.

In a specific implementation, the user can trigger the photographing instruction by clicking the photographing control, clicking on the preview image data, and the like, and the camera performs the photographing process, and at the same time, driving the speaker to output the audio according to the sound field angle and the deflecting angle, that is, the actual target to the target object. A photo is sounded in the area to remind the person being photographed to take a picture.

Directional output audio can be produced by utilizing the nonlinear propagation effects of ultrasound in air to produce highly directional audible sound (ie, audio orientation).

According to the theory of nonlinear acoustics, two plane waves propagate nonlinearly in an inhomogeneous medium. When two columns of electrical signals of frequency f ₁ and f ₂ are input to the ultrasonic transducer (one of the components of the loudspeaker), the ultrasonic transducer passes The mechanical vibration emits two ultrasonic waves of frequencies f ₁ and f ₂ into the air. When these two columns of ultrasonic waves propagate in the air, a nonlinear interaction will occur, resulting in the inclusion of the fundamental frequencies f ₁ , f ₂ , the sum frequency f ₁ +f ₂ , the difference frequencies f ₁ -f ₂ and the various orders Complex sound waves including harmonics. Since the acoustic attenuation coefficient α is proportional to the square of the frequency, the higher frequency ultrasonic signals f ₁ , f ₂ , f ₁ +f ₂ and the harmonics will be quickly absorbed by the air, leaving the difference in the audio frequency range. The frequency signals f ₁ -f ₂ continue to propagate in the air.

Whether the sound wave has directivity is closely related to the ratio of the wavelength of the sound wave to the size of the sound source. When the wavelength of the sound wave is much larger than the size of the sound source, the sound wave has no directivity; when the wavelength of the sound wave is close to far smaller than the size of the sound source, the sound wave will gradually exhibit more and more directivity. Therefore, when the greetings select the ultrasonic frequencies f ₁ and f ₂ , the beat signals f ₁ -f ₂ can be made to be in the audible range, thereby generating the acoustic waves by the ultrasonic waves.

Further, in the parametric acoustic array theory, an ultrasonic transducer (one of the components of the loudspeaker) emits a strongly modulated ultrasonic wave into the air medium, the ultrasonic wave being in the direction of its main axis of propagation (such as the direction of the sound field and the direction in which the deflection angle is directed) During the process of traveling, the audio signals are continuously modulated by nonlinear interaction, and these continuously demodulated audio waves are accumulated and superimposed, whereby an end-fire virtual array is realized in this way. . This virtual sound source array is a so-called parametric acoustic array. The parametric acoustic array makes the energy of the acoustic wave continuously strengthened in the direction of the sound wave. Since the ultrasonic wave has strong directivity, the superposition enhancement effect will be weak except for the direction of the propagation principal axis (such as the direction of the sound field and the direction of the deflection angle), which ultimately causes the acoustic wave to be in the direction of the main propagation axis (such as the sound field angle and the deflection angle). The direction of pointing) has a strong directivity.

It should be noted that, for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should understand that the embodiments of the present disclosure are not limited by the described action sequence, because In accordance with embodiments of the present disclosure, certain steps may be performed in other sequences or concurrently. In the following, those skilled in the art should also understand that the embodiments described in the specification are optional embodiments, and the actions involved are not necessarily required by the embodiments of the present disclosure.

Referring to FIG. 5, a structural block diagram of an embodiment of a camera-based audio output device according to the present disclosure is shown, which may specifically include the following modules:

The target object determining module 501 is configured to determine one or more target objects in the preview image data when the camera captures the preview image data;

a sound field angle calculation module 502, configured to calculate, relative to the speaker, the one or more target objects Sound field angle of the device;

a deflection angle calculation module 503, configured to calculate a deflection angle relative to the speaker for the one or more target objects;

The audio directional output module 504 is configured to drive the speaker to output audio according to the sound field angle and the deflection angle when receiving the photographing indication.

In an optional embodiment of the present disclosure, the target object determining module 501 may include the following sub-modules:

The first determining submodule is configured to detect a face in the preview image data and determine the target object as one or more targets.

In an optional embodiment of the present disclosure, the target object determining module 501 may include the following sub-modules:

a second determining submodule, configured to: when receiving the focus operation indication, determine that one or more human faces in the preview image data corresponding to the focus operation instruction are one or more target objects;

or,

The cancel submodule is used to cancel the determined one or more target objects upon receiving the cancel operation indication.

In an optional embodiment of the present disclosure, the sound field angle calculation module 502 may include the following sub-modules:

a target distance measurement submodule for measuring a target distance between the speaker and the one or more target objects;

The audition range distance obtaining sub-module is configured to obtain an audition range distance matching the one or more target objects, where the audition range distance is a distance in which the range of the audio can be heard;

And a first calculating submodule configured to calculate a sound field angle according to the target distance and the distance of the audition range.

In an optional example of the embodiments of the present disclosure, the target distance measurement submodule may further include the following submodules:

a first obtaining submodule, configured to acquire a candidate distance between the camera and the target object as the target distance when the target object is one;

Or,

a second obtaining submodule, configured to acquire a plurality of candidate distances between the camera and the plurality of target objects respectively when the target object is multiple;

And a second calculation submodule configured to calculate the target distance by using the plurality of candidate distances.

In an optional embodiment of the present disclosure, the deflection angle calculation module 503 may include the following sub-modules:

a projection submodule, configured to project a target object in the preview image data into a preset coordinate system, where the coordinate system is constructed based on a position of the speaker;

a focus coordinate calculation submodule, configured to calculate focus coordinates of the one or more target objects in the coordinate system;

a third calculation sub-module for calculating a deflection angle using the focus coordinates.

In an optional example of the embodiments of the present disclosure, the focus coordinate calculation sub-module may further include the following sub-modules:

a first search submodule, configured to search for a first coordinate of the upper left corner of the target object and a second coordinate of a lower right corner when the target object is one;

a fourth calculation submodule, configured to calculate an average value of the first coordinate and the second coordinate as a focus coordinate;

or,

a second search submodule, configured to: when the target object is multiple, find the third coordinate of the upper left corner of the leftmost target object, the fourth coordinate of the lower right corner, and the upper left corner of the rightmost target object The fifth coordinate, the sixth coordinate of the lower right corner;

And a fifth calculating submodule, configured to respectively calculate an average value of the third coordinate and the fourth coordinate, and an average value of the fifth coordinate and the sixth coordinate as a focus coordinate.

In an optional example of the embodiment of the present disclosure, the third calculating submodule may further include the following submodule:

a first candidate deflection angle calculation submodule, configured to calculate a first candidate deflection angle by using focus coordinates of the leftmost target object when the target object is multiple;

a second candidate deflection angle calculation submodule, configured to calculate a second candidate deflection angle by using focus coordinates of the rightmost target object;

The first setting sub-module is used to target the leftmost target object and the rightmost target object. When the two sides of the sounder are set, the first feature angle is set to a deflection angle;

a second setting sub-module, configured to set a second feature angle as a deflection angle when the leftmost target object and the rightmost target object are located on the same side of the speaker;

The first feature angle is a half of a difference between the first candidate deflection angle and the second candidate deflection angle; the second feature angle is the first candidate deflection angle and the second candidate bias Half of the sum of the horns.

For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.

The various embodiments in the present specification are described in a progressive manner, and each embodiment focuses on differences from other embodiments, and the same similar parts between the various embodiments can be referred to each other.

Those skilled in the art will appreciate that embodiments of the disclosed embodiments can be provided as a method, apparatus, or computer program product. Thus, embodiments of the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or a combination of software and hardware. Moreover, embodiments of the present disclosure may take the form of a computer program product embodied on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.

Embodiments of the present disclosure are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each flow and/or block of the flowchart illustrations and/or FIG. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing terminal device to produce a machine such that instructions are executed by a processor of a computer or other programmable data processing terminal device Means are provided for implementing the functions specified in one or more of the flow or in one or more blocks of the flow chart.

The computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device. The instruction device implements the functions specified in one or more blocks of the flowchart or in a flow or block of the flowchart.

These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device such that a series of operational steps are performed on the computer or other programmable terminal device to produce computer-implemented processing, such that the computer or other programmable terminal device The instructions executed above provide steps for implementing the functions specified in one or more blocks of the flowchart or in a block or blocks of the flowchart.

Although alternative embodiments of the disclosed embodiments have been described, those skilled in the art can make additional changes and modifications to the embodiments once they are aware of the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all alternatives and modifications of the embodiments.

Finally, it should also be noted that in this context, relational terms such as first and second are used merely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these entities. There is any such actual relationship or order between operations. Furthermore, the terms "comprises" or "comprising" or "comprising" or any other variations are intended to encompass a non-exclusive inclusion, such that a process, method, article, or terminal device that includes a plurality of elements includes not only those elements but also Other elements that are included, or include elements inherent to such a process, method, article, or terminal device. An element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that comprises the element, without further limitation.

The above is a detailed description of a photo-based audio output method and a photo-based audio output device provided by the present disclosure. The principle and implementation of the present disclosure are described in the following. The descriptions are only used to help understand the method of the present disclosure and its core ideas; at the same time, for those of ordinary skill in the art, according to the idea of the present disclosure, there will be changes in the specific embodiments and application scopes. The description is not to be construed as limiting the disclosure.

Claims (16)

A camera-based audio output method, comprising: Determining one or more target objects in the preview image data when the camera captures the preview image data; Calculating an acoustic field angle relative to the speaker for the one or more target objects; Calculating a deflection angle relative to the speaker for the one or more target objects; When receiving the photographing instruction, the driving speaker outputs the audio in accordance with the sound field angle and the deflection angle. The method of claim 1 wherein said step of determining one or more target objects in said pre-image data comprises: A face in the preview image data is detected and determined as one or more target objects. The method of claim 1 wherein said step of determining one or more target objects in said pre-image data comprises: Determining, when the focus operation instruction is received, that the one or more human faces in the preview image data corresponding to the focus operation instruction are one or more target objects; or, When the cancel operation indication is received, the determined one or more target objects are canceled. The method of claim 1 or 2 or 3, wherein the step of calculating an acoustic field angle relative to the speaker for the one or more target objects comprises: Measuring a target distance between the speaker and the one or more target objects; Obtaining an audition range distance that matches the one or more target objects, the audition range distance being a distance of a range in which the audio can be heard; The sound field angle is calculated based on the target distance and the distance of the audition range. The method of claim 4 wherein said step of measuring a target distance between said speaker and said one or more target objects comprises: Obtaining a candidate distance between the camera and the target object as the target distance when the target object is one; Or, When the target object is multiple, a plurality of candidate distances between the camera and the plurality of target objects are respectively acquired; and the target distance is calculated by using the plurality of candidate distances. The method of claim 1 or 2 or 3 or 5, wherein the step of calculating a deflection angle with respect to the speaker for the one or more target objects comprises: Projecting the target object in the preview image data into a preset coordinate system, the coordinate system being constructed based on the position of the speaker; In the coordinate system, a focus coordinate of the one or more target objects is calculated; and a deflection angle is calculated using the focus coordinates. The method of claim 6 wherein said step of calculating focus coordinates of said one or more target objects comprises: When the target object is one, searching for a first coordinate of the upper left corner of the target object and a second coordinate of a lower right corner; Calculating an average value of the first coordinate and the second coordinate as a focus coordinate; or, When the target object is multiple, the third coordinate of the upper left corner of the target object on the leftmost side, the fourth coordinate of the lower right corner, and the fifth coordinate and the lower right corner of the upper left corner of the target object of the rightmost side are searched. Six coordinates An average value of the third coordinate and the fourth coordinate, and an average value of the fifth coordinate and the sixth coordinate are respectively calculated as focus coordinates. The method according to claim 6 or 7, wherein said step of calculating a deflection angle using said focus coordinates comprises: When the target object is multiple, the first candidate deflection angle is calculated by using the focus coordinates of the leftmost target object; Calculating a second candidate deflection angle by using focus coordinates of the rightmost target object; If the leftmost target object and the rightmost target object are located on both sides of the speaker, the first feature angle is set to a deflection angle; If the leftmost target object and the rightmost target object are located on the same side of the speaker, the second feature angle is set to a deflection angle; Wherein the first feature angle is the first candidate deflection angle and the second candidate deflection angle One half of the difference; the second feature angle is half of the sum of the first candidate deflection angle and the second candidate deflection angle. A camera-based audio output device comprising: a target object determining module, configured to determine one or more target objects in the preview image data when the camera captures the preview image data; a sound field angle calculation module, configured to calculate an acoustic field angle relative to the speaker for the one or more target objects; a deflection angle calculation module, configured to calculate a deflection angle relative to the speaker for the one or more target objects; And an audio directional output module, configured to drive the speaker to output audio according to the sound field angle and the deflection angle when receiving the photographing indication. The apparatus of claim 9, wherein the target object determination module comprises: The first determining submodule is configured to detect a face in the preview image data and determine the target object as one or more targets. The apparatus of claim 9, wherein the target object determination module comprises: a second determining submodule, configured to: when receiving the focus operation indication, determine that one or more human faces in the preview image data corresponding to the focus operation instruction are one or more target objects; or, The cancel submodule is used to cancel the determined one or more target objects upon receiving the cancel operation indication. The apparatus according to claim 9 or 10 or 11, wherein the sound field angle calculation module comprises: a target distance measurement submodule for measuring a target distance between the speaker and the one or more target objects; The audition range distance obtaining sub-module is configured to obtain an audition range distance matching the one or more target objects, where the audition range distance is a distance in which the range of the audio can be heard; And a first calculating submodule configured to calculate a sound field angle according to the target distance and the distance of the audition range. The apparatus of claim 12, wherein the target distance measurement sub-module comprises: a first obtaining submodule, configured to acquire a candidate distance between the camera and the target object as the target distance when the target object is one; or, a second obtaining submodule, configured to acquire a plurality of candidate distances between the camera and the plurality of target objects respectively when the target object is multiple; And a second calculation submodule configured to calculate the target distance by using the plurality of candidate distances. The apparatus according to claim 9 or 10 or 11 or 13, wherein the deflection angle calculation module comprises: a projection submodule, configured to project a target object in the preview image data into a preset coordinate system, where the coordinate system is constructed based on a position of the speaker; a focus coordinate calculation submodule, configured to calculate focus coordinates of the one or more target objects in the coordinate system; a third calculation sub-module for calculating a deflection angle using the focus coordinates. The apparatus of claim 14, wherein the focus coordinate calculation sub-module comprises: a first search submodule, configured to search for a first coordinate of the upper left corner of the target object and a second coordinate of a lower right corner when the target object is one; a fourth calculation submodule, configured to calculate an average value of the first coordinate and the second coordinate as a focus coordinate; or, a second search submodule, configured to: when the target object is multiple, find the third coordinate of the upper left corner of the leftmost target object, the fourth coordinate of the lower right corner, and the upper left corner of the rightmost target object The fifth coordinate, the sixth coordinate of the lower right corner; And a fifth calculating submodule, configured to respectively calculate an average value of the third coordinate and the fourth coordinate, and an average value of the fifth coordinate and the sixth coordinate as a focus coordinate. The apparatus of claim 14 or 15, wherein the third computing sub-module comprises: a first candidate deflection angle calculation submodule, configured to calculate a first candidate deflection angle by using focus coordinates of the leftmost target object when the target object is multiple; a second candidate deflection angle calculation submodule, configured to calculate a second candidate deflection angle by using focus coordinates of the rightmost target object; a first setting sub-module, configured to set a first feature angle as a deflection angle when the leftmost target object and the rightmost target object are located on both sides of the speaker; a second setting sub-module, configured to set a second feature angle as a deflection angle when the leftmost target object and the rightmost target object are located on the same side of the speaker; The first feature angle is a half of a difference between the first candidate deflection angle and the second candidate deflection angle; the second feature angle is the first candidate deflection angle and the second candidate bias Half of the sum of the horns.

Images