WO2025135388A1 - Electronic device for performing image stabilization and operation method therefor - Google Patents

Abstract

An electronic device according to various embodiments comprises a sensor, a camera module, one or more processors, and a memory for storing one or more instructions, wherein the camera module may comprise a lens assembly, an image sensor, and a driving unit for transferring the lens assembly or the image sensor. The one or more instructions, when executed by the one or more processors individually or collectively, may instruct the electronic device to: control the driving unit to move the position of the lens assembly or the image sensor according to movement information on the basis of a first operation mode; control the driving unit such that the lens assembly or the image sensor is positioned at a reference position on the basis of a second operation mode; perform an image stabilization process; and when operating on the basis of the first operation mode, enable the driving unit to control the position of the lens assembly or the image sensor on the basis of a first gain value, and when operating on the basis of the second operation mode, enable the driving unit to control the position of the lens assembly or the image sensor on the basis of a second gain value.

Description

Electronic device performing image stabilization and method of operation thereof

The present disclosure relates to an electronic device performing image stabilization and a method of operating the same.

Recently, as the functions of mobile devices have diversified, the demand for improved photo and video shooting functions using mobile devices has also increased. Accordingly, technology to compensate for the shaking of electronic devices when shooting video is being developed.

The electronic device may include a camera module capable of performing an OIS (optical image stabilization) function. The OIS function is a function to compensate for shaking by moving an OIS driving unit (e.g., a lens assembly or an image sensor) included in the camera module within the OIS driving range, and the camera module may move the lens assembly or the image sensor in a direction that offsets the movement of the electronic device for OIS driving.

An electronic device can perform video digital image stabilization (VDIS) and/or electronic image stabilization (EIS) on multiple image frames. VDIS is a method of reducing shaking through digital processing in a mobile device, and a processor can compensate for multiple image frames through VDIS.

The above information may be provided as related art for the purpose of assisting in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art in connection with the present disclosure.

An electronic device according to one embodiment may include a sensor for detecting movement of the electronic device, a camera module, one or more processors, and a memory for storing one or more instructions. The camera module may include a lens assembly, an image sensor, and a driving unit for moving the lens assembly or the image sensor within a driving range. The one or more instructions may cause the electronic device, when individually or collectively executed by one or more processors, to control the driving unit to move a position of the lens assembly or the image sensor based on movement information acquired from the sensor, based on a first operation mode among a plurality of operation modes associated with the operation of the driving unit. The one or more instructions may cause the electronic device, when individually or collectively executed by one or more processors, to control the driving unit to position the lens assembly or the image sensor at a reference position within the driving range, based on a second operation mode different from the first operation mode among the plurality of operation modes. The one or more instructions may cause the electronic device, when individually or collectively executed by one or more processors, to perform image processing on image frames acquired through the camera module while controlling the driving unit to perform image stabilization processing to reduce image shake. One or more instructions, when executed by one or more processors individually or collectively, may cause the electronic device to: when operating based on the first operating mode, cause the driver to control the position of the lens assembly or the image sensor based on a first gain value, and when operating based on the second operating mode, cause the driver to control the position of the lens assembly or the image sensor based on a second gain value different from the first gain value.

In one embodiment, a method of operating an electronic device including a camera module may include an operation of controlling a driving unit to move a position of a lens assembly or an image sensor based on movement information acquired from a sensor detecting a movement of the electronic device, based on a first operation mode among a plurality of operation modes associated with an operation of a driving unit that moves a lens assembly or an image sensor within a driving range. The method of operating an electronic device may include an operation of controlling a driving unit so that the lens assembly or the image sensor is positioned at a reference position within the driving range, based on a second operation mode different from the first operation mode among the plurality of operation modes. The method of operating an electronic device may include an operation of performing image stabilization processing to reduce shaking of an image by performing image processing on image frames acquired through the camera module while controlling the driving unit. The method of operating an electronic device may include an operation of controlling a position of the lens assembly or the image sensor based on a first gain value when operating based on the first operation mode, and an operation of controlling a position of the lens assembly or the image sensor based on a second gain value different from the first gain value when operating based on the second operation mode.

A computer-readable, non-transitory recording medium can cause an electronic device including a camera module, when executed, to perform a method including: controlling a driving unit to move a position of a lens assembly or an image sensor according to movement information acquired from a sensor detecting a movement of the electronic device, based on a first operation mode among a plurality of operation modes associated with an operation of a driving unit that moves a lens assembly or an image sensor within a driving range. The computer-readable, non-transitory recording medium can cause an electronic device including a camera module, when executed, to perform a method including: controlling a driving unit to position the lens assembly or the image sensor at a reference position within the driving range, based on a second operation mode different from the first operation mode among a plurality of operation modes. The computer-readable, non-transitory recording medium can cause an electronic device including a camera module, when executed, to perform a method including: performing image processing on an image frame acquired through the camera module while controlling the driving unit to perform image stabilization processing to reduce shaking of the image. A computer-readable non-transitory recording medium can cause an electronic device including a camera module to perform a method including: when the electronic device operates based on a first operation mode, the driving unit controls a position of a lens assembly or an image sensor based on a first gain value, and when the electronic device operates based on a second operation mode, the driving unit controls the position of the lens assembly or the image sensor based on a second gain value different from the first gain value.

FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

FIG. 2 is a block diagram illustrating a camera module according to various embodiments.

Figure 3 is a block diagram showing the configuration of an electronic device according to one embodiment.

Figure 4 is an exploded perspective view of a camera module according to one embodiment.

FIG. 5 is a perspective view of a camera module according to one embodiment.

FIG. 6 is a block diagram illustrating a structure in which an electronic device performs image stabilization processing according to one embodiment.

FIG. 7 is a drawing for explaining an operation of controlling a driving unit based on a first operation mode according to one embodiment.

FIG. 8 is a drawing for explaining an operation of controlling a driving unit based on a second operation mode according to one embodiment.

FIG. 9 is a diagram for explaining an operation of controlling the position of a lens assembly or an image sensor based on a gain value according to one embodiment.

FIG. 10 is a flowchart illustrating a process in which an electronic device performs a first operation mode or a second operation mode based on movement information according to an embodiment of the present invention.

FIG. 11 is a flowchart illustrating a process in which an electronic device changes a first operation mode to a second operation mode and performs the second operation mode according to one embodiment.

FIG. 12 is a diagram for explaining an operation for determining whether the frequency of motion information satisfies a specified condition according to one embodiment.

FIG. 13 is a diagram for explaining an operation for determining whether the frequency of motion information satisfies a specified condition according to one embodiment.

FIG. 14 is a flowchart illustrating a process of a second operation mode according to one embodiment.

FIG. 15 is a diagram for explaining an operation of controlling the position of a lens assembly or an image sensor based on a first gain value and a second gain value different from each other according to one embodiment.

FIG. 16 is a flowchart illustrating a process for fixing a target value to a specified value according to one embodiment.

FIG. 17 is a diagram for explaining the effect of improving the performance of image stabilization processing by fixing a target value to a specified value according to one embodiment.

FIG. 18 is a flowchart illustrating a process for performing an operation mode based on user input according to one embodiment.

FIG. 19 is a flowchart illustrating a process for performing an operation mode based on detection of an external device according to one embodiment.

Hereinafter, embodiments will be described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present invention. However, the disclosed embodiments may be implemented in various different forms and are not limited to the embodiments described herein.

When the movement of the electronic device shakes above a certain frequency, there is a problem of difficulty in controlling the driving unit that moves the lens or image sensor within the driving range and a problem of reduced performance of the VDIS (and/or EIS).

In one embodiment, an electronic device and an operating method thereof may be provided that allow a position of a lens or a position of an image sensor to be stably maintained even when the movement of the electronic device shakes more than a specific frequency. For example, a camera module that controls the position of a lens or the position of an image sensor so that the position of a lens or the position of an image sensor can be stably maintained even when the electronic device shakes more than a specific frequency, an electronic device including the camera module, and an operating method thereof may be provided. Accordingly, even when a video or a preview video is shot in a situation where the electronic device shakes significantly, the performance of image stabilization can be improved, and a user can obtain a video or a preview video with improved quality. Even in a situation where the electronic device shakes or is shaken by various impacts while shooting a preview video or video through the electronic device, the user can obtain a preview video or video with shake-compensated preview video.

The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the art pertaining to the present disclosure from the description of the present disclosure.

FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may include one or more other components. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphic processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a given function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

The auxiliary processor (123) may control at least a part of functions or states associated with at least one of the components of the electronic device (101) (e.g., the display module (160), the sensor module (176), or the communication module (190)), for example, while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, in the electronic device (101) itself on which the artificial intelligence model is executed, or may be performed through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers. The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may additionally or alternatively include a software structure.

The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or nonvolatile memory (134).

The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

The audio output module (155) can output an audio signal to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive an incoming call. According to one embodiment, the receiver can be implemented separately from the speaker or as a part thereof.

The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) can include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) can include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can obtain sound through an input module (150), or output sound through an audio output module (155), or an external electronic device (e.g., an electronic device (102)) (e.g., a speaker or a headphone) directly or wirelessly connected to the electronic device (101).

The sensor module (176) can detect an operating state (e.g., power or temperature) of the electronic device (101) or an external environmental state (e.g., user state) and generate an electric signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., the electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that a user can perceive through a tactile or kinesthetic sense. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module (180) can capture still images and moving images. According to one embodiment, the camera module (180) can include one or more lenses, image sensors, image signal processors, or flashes.

The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery (189) can power at least one component of the electronic device (101). In one embodiment, the battery (189) can include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

The communication module (190) may support establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., the electronic device (102), the electronic device (104), or the server (108)), and performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., the application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, a corresponding communication module may communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) may use subscriber information (e.g., international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196) to identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199).

The wireless communication module (192) can support a 5G network and next-generation communication technology after a 4G network, for example, NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), terminal power minimization and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) may support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) may support various requirements specified in an electronic device (101), an external electronic device (e.g., electronic device (104)), or a network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL) each, or 1 ms or less for round trip) for URLLC realization.

The antenna module (197) can transmit or receive signals or power to or from the outside (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and the external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC positioned on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) positioned on or adjacent a second side (e.g., a top side or a side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

At least some of the above components can be connected to each other and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

According to one embodiment, a command or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of executing the function or service by itself or in addition, request one or more external electronic devices to perform at least a part of the function or service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result, as is or additionally processed, as at least a part of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an IoT (Internet of Things) device. The server (108) may be an intelligent server using machine learning and/or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199). The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

Electronic devices according to various embodiments disclosed in this document may be devices of various forms. The electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliance devices. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The various embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

The term "module" used in various embodiments of this document may include a unit implemented in hardware, software or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrally configured component or a minimum unit of the component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the called at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

According to one embodiment, the method according to the various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play StoreTM) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a part of the computer program product may be at least temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or an intermediary server.

According to various embodiments, each component (e.g., a module or a program) of the above-described components may include a single or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the above-described components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, the multiple components (e.g., a module or a program) may be integrated into one component. In such a case, the integrated component may perform one or more functions of each of the multiple components identically or similarly to those performed by the corresponding component of the multiple components before the integration. According to various embodiments, the operations performed by the module, program or other components may be executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. The lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

The flash (220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can convert light emitted or reflected from a subject and transmitted through the lens assembly (210) into an electrical signal, thereby obtaining an image corresponding to the subject. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as, for example, an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) may be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (240) can detect such movement of the camera module (180) or the electronic device (101) by using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). In one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer. The memory (250) can temporarily store at least a portion of an image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to a shutter, or when a plurality of images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed by, for example, an image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130), or as a separate memory that operates independently therefrom.

The image signal processor (260) can perform one or more image processing operations on an image acquired through an image sensor (230) or an image stored in a memory (250). The one or more of the image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) may perform control (e.g., exposure time control, read-out timing control, etc.) for at least one of the components included in the camera module (180) (e.g., the image sensor (230)). The image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or may be provided to an external component of the camera module (180) (e.g., the memory (130), the display module (160), the electronic device (102), the electronic device (104), or the server (108)). According to one embodiment, the image signal processor (260) may be at least a part of the processor (120). The image signal processor (260) may be configured as a separate processor that operates independently of the processor (120). When the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) as is or after undergoing additional image processing by the processor (120).

According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

Figure 3 is a block diagram showing the configuration of an electronic device (300) according to one embodiment.

The electronic device (300) of FIG. 3 may be referenced by the electronic device (101) of FIG. 1. With respect to FIG. 3, the configuration described in FIGS. 1 and 2 may be briefly described or the description may be omitted.

In FIG. 3, the electronic device (300) is illustrated as including a sensor (310), a camera module (320), a processor (330), and a memory (340), but the configuration of the electronic device (300) is not limited thereto. For example, the electronic device (300) may omit at least one of the above-described configurations, or may further include at least one other configuration.

According to one embodiment, the electronic device (300) may include a sensor (310). The sensor (310) of FIG. 3 may be included in the sensor module (176) of FIG. 1. According to one embodiment, the sensor (310) may detect movement of the electronic device (300). For example, the electronic device (300) may obtain movement information of the electronic device (300) through the sensor (310). In one example, the sensor (310) may include at least one of a gyro sensor or an acceleration sensor. For example, the electronic device (300) may obtain angular velocity information of the electronic device (300) through the gyro sensor. For example, the electronic device (300) may obtain acceleration information of the electronic device (300) through the acceleration sensor. However, the present invention is not limited thereto. For example, the electronic device (300) may include various sensors capable of detecting movement of the electronic device (300).

According to one embodiment, the sensor (310) may be formed separately from the camera module (320) or may be included in the camera module (320). In one example, the sensor (310) may be included in a printed circuit board (not shown) on which the processor (330) is mounted.

According to one embodiment, the electronic device (300) may include a camera module (320). According to one embodiment, the camera module (320) may include a lens assembly (321), an image sensor (322), and a driving unit (333). However, the configuration of the camera module (320) is not limited thereto. For example, the camera module (320) may omit at least one of the above-described configurations, or may further include at least one other configuration.

According to one embodiment, the driving unit (333) can move the lens assembly (321) or the image sensor (322) within the driving range. For example, the driving unit (333) can move the lens assembly (321) or the image sensor (322) within the OIS (optical image stabilization) driving range. For example, the driving unit (333) can move the lens assembly (321) or the image sensor (322) within the AF (auto focus) driving range. For example, the driving unit (333) can move the lens assembly (321) or the image sensor (322) within the OIS driving range and the AF driving range.

According to one embodiment, the electronic device (300) may perform an OIS function for image stabilization. The OIS may include a lens shift method in which a lens assembly (321) moves or tilts in a direction substantially perpendicular to an optical axis, and/or a sensor shift method in which an image sensor (322) moves or tilts in a direction substantially perpendicular to an optical axis. In one example, the OIS may include a method in which a reflective member (e.g., a prism or a mirror) is tilted, or a reflective member shift method. The reflective member may be disposed on the lens, and/or disposed between the lens assembly and the image sensor. However, the present invention is not limited thereto. For example, the OIS may include a module tilt method in which a camera module is tilted.

According to one embodiment, in a lens shift type OIS, the driving unit (333) may include a lens assembly (321). For example, the camera module (320) may include a lens assembly (321) that is movably arranged, and an OIS carrier that accommodates the lens assembly (321) and moves or tilts in a direction substantially perpendicular to the optical axis together with the lens assembly (321).

According to one embodiment, in a sensor shift type OIS, the driving unit (333) may include an image sensor (322). For example, the camera module (320) may include an image sensor (322) arranged to be movable.

The present disclosure may be applied to at least one of an electronic device using a lens shift type OIS, an electronic device using a sensor shift type OIS, an electronic device using a module tilt type OIS, or an electronic device using an OIS using a tilting reflective member type OIS.

The present disclosure can also be applied when the electronic device (300) includes two or more camera modules (320). According to one embodiment, the electronic device (300) may include two or more camera modules (320). For example, the electronic device (300) may include a first camera module including a telephoto lens, and a second camera module including a wide-angle lens. For example, the electronic device (300) may also include a first camera module including a telephoto lens, a second camera module including a wide-angle lens, and a third camera module including an ultra-wide lens.

According to one embodiment, at least some of the camera modules (320) included in the electronic device (300) may support the OIS function. According to one embodiment, at least some of the camera modules (320) included in the electronic device (300) may not support the OIS function. For example, the first camera module and the second camera module may support the OIS function, and the third camera module may not support the OIS function. According to one embodiment, when the electronic device (300) includes two or more camera modules (320), the electronic device (300) may include two or more OIS processing units that control the respective OIS driving units. For example, the electronic device (300) may include a first camera module and a second camera module that support the OIS function, and the first OIS processing unit may control the OIS driving unit included in the first camera module, and the second OIS processing unit may control the OIS driving unit included in the second camera module.

According to one embodiment, when the electronic device (300) includes two or more camera modules (320), the electronic device (300) may include one OIS processing unit that controls each of the OIS driving units. For example, the electronic device (300) may include a first camera module and a second camera module that support an OIS function, and may control both the OIS driving unit included in the first camera module and the OIS driving unit included in the second camera module through a single OIS processing unit. Although FIGS. 3 to 17 of the present disclosure describe one camera module included in the electronic device, this is only one example and does not limit the technical idea of the present disclosure.

According to one embodiment, the electronic device (300) may include a processor (330) and a memory (340). In one example, the electronic device (300) may include one or more processors (330) and a memory (340) storing one or more instructions. In the present disclosure, it may be understood that the electronic device (300) is controlled by the one or more processors (330) executing instructions stored in the memory (340). For example, it may be understood that the operation of at least one of the sensors (310) or the camera module (320) is controlled by the one or more processors (330) executing instructions stored in the memory (340). For example, one or more commands stored in the memory (340) may be executed by one or more processors (330) so that the electronic device (300) may control the driving unit (333) to move the position of the lens assembly (321) or the image sensor (322) according to the movement information obtained from the sensor (310) based on the first operation mode. For example, one or more commands stored in the memory (340) may be executed by one or more processors (330) so that the electronic device (300) may control the driving unit (333) so that the lens assembly (321) or the image sensor (322) is positioned at a reference position within the operation range based on the second operation mode.

Fig. 4 is an exploded perspective view of a camera module (400) according to one embodiment. Fig. 5 is a perspective view of a camera module (400) according to one embodiment.

The camera module (400) of FIGS. 4 and 5 may be referenced by the camera module (180) of FIGS. 1 and 2 and/or the camera module (320) of FIG. 3.

Referring to FIGS. 4 and 5, a camera module (400) according to an embodiment may include a lens assembly (410), an optical image stabilization (OIS) driving unit for driving the optical image stabilization (OIS) of the camera module (400), an AF driving unit for driving the auto focus (AF) of the camera module (400), a housing (450), and an image sensor (460). However, the configuration of the camera module (400) is not limited thereto. For example, the camera module (400) may omit at least one component among the above-described components, or may further include at least one other component. For example, the camera module (400) may further include at least one of a base on which the housing (450) is placed, a cover covering at least a portion of the housing (450), or a printed circuit board connected to the image sensor (460).

In one embodiment, the lens assembly (410) can include at least one lens arranged to form an image of a subject onto the image sensor (460). The camera module (400) can receive light through the lens assembly (410). For example, the light can enter the interior of the camera module (400) through at least a portion of the at least one lens. In one example, the light can enter the interior of the camera module (400) in a direction (40) substantially parallel to an optical axis (e.g., in the -z-axis direction).

According to one embodiment, at least a portion of the lens assembly (410) may be exposed to the exterior of the camera module (400). For example, at least a portion of the lens assembly (410) may protrude from an upper surface of the housing (450) (e.g., a surface facing the +z direction).

According to one embodiment, at least a portion of the lens assembly (410) may be disposed inside the housing (450). In one example, the lens assembly (210) may be disposed to be movable relative to the housing (450). For example, the lens assembly (210) may be movable in at least one of a first direction (e.g., in the x-axis direction or the -x-axis direction) and/or a second direction (e.g., in the y-axis direction or the -y-axis direction) relative to the housing (450) by an OIS driving unit (e.g., the first carrier (420) and/or the second carrier (430)) for driving the OIS of the camera module (400). For example, the lens assembly (210) may be movable in a third direction (e.g., in the z-axis direction or the -z-axis direction) relative to the housing (450) by an AF driving unit (e.g., the third carrier (440)) for driving the AF of the camera module (400).

According to one embodiment, the camera module (400) may include an OIS driving unit for optical image stabilization function. The OIS driving unit may move the lens assembly (410) in a first direction (e.g., in the x-axis direction or the -x-axis direction) and/or a second direction (e.g., in the y-axis direction or the -y-axis direction). The first direction may be a different direction than the second direction. According to one embodiment, the OIS driving unit may include an OIS actuator configured to move the lens assembly (410) for optical image stabilization. For example, the OIS driving unit may move in a direction substantially perpendicular to a third direction (e.g., in the z-axis direction or the -z-axis direction) in which a distance between the lens assembly (410) and the image sensor (460) is changed for optical image stabilization.

According to one embodiment, the OIS drive unit may include a first carrier (420) that transports the lens assembly (410) in a first direction (e.g., in the x-axis direction or the -x-axis direction) and/or a second direction (e.g., in the y-axis direction or the -y-axis direction). In one example, the lens assembly (410) may be coupled to the first carrier (420). For example, the lens assembly (410) may be fixed to the first carrier (420). For example, the lens assembly (410) may be at least partially surrounded by the first carrier (420).

According to one embodiment, a first magnet (421) may be disposed on a first side (e.g., a side facing the -y direction) of a first carrier (420). The first magnet (421) may be disposed to face a first coil (451). The first coil (451) may be disposed on an inner side of a housing (450) facing the first side of the first carrier (420) on which the first magnet (421) is disposed. The first carrier (420) may be moved relative to the housing (450) by a force in a second direction (e.g., in the y-axis direction or the -y-axis direction) applied to the first magnet (421) by a magnetic field generated by a current flowing in the first coil (451). As the first carrier (420) moves, the lens assembly (410) may be moved in the second direction.

According to one embodiment, a second magnet (422) may be disposed on a second side (e.g., a side facing the -x direction) of the first carrier (420). The second magnet (422) may be disposed to face the second coil (452). The second coil (452) may be disposed on the inner side of the housing (450) facing the second side of the first carrier (420) on which the second magnet (422) is disposed. The first carrier (420) may be moved relative to the housing (450) by a force in a first direction (e.g., in the x-axis direction or the -x-axis direction) applied to the second magnet (422) by a magnetic field generated by a current flowing in the second coil (452). As the first carrier (420) moves, the lens assembly (410) may be moved in the first direction.

According to one embodiment, the first side of the first carrier (420) can be substantially perpendicular to the second side of the first carrier (420). The first magnet (421) can be arranged substantially perpendicular to the second magnet (422). The first coil (451) can be arranged substantially perpendicular to the second coil (452).

According to one embodiment, the third magnet (445) may be positioned substantially perpendicular to the first magnet (421). The second magnet (422) may be positioned opposite the third magnet (445). The second magnet (422) may be positioned substantially parallel to the third magnet (445). However, the present invention is not limited thereto.

According to one embodiment, the OIS actuator may include a first carrier (420) coupled with the lens assembly (410), and a second carrier (430) that transports the first carrier (420) in a second direction (e.g., in the y-axis direction or the -y-axis direction). For example, the first carrier (420) may be disposed on the second carrier (430), such that as the second carrier (430) moves in the second direction, the first carrier (420) may be transported in the second direction. For example, the first carrier (420) and/or the second carrier (430) may be moved in the first direction and/or the second direction by the sphere.

According to one embodiment, when the OIS driving unit further includes a second carrier (430), a first sphere (432) may be placed between the first carrier (420) and the second carrier (430). The first carrier (420) may move on the second carrier (430) by the first sphere (432). For example, the first carrier (420) may move in a first direction (for example, in the x-axis direction or the -x-axis direction) with respect to the housing (450) on the second carrier (430) by the first sphere (432). However, the present invention is not limited thereto. For example, the second carrier (430) and the first sphere (432) may be omitted in the OIS driving unit.

According to one embodiment, when the OIS driving unit further includes a second carrier (430), at least one of the first carrier (420) or the second carrier (430) may include a first guide portion (431) that guides the first sphere (432) to move in the first direction. The first sphere (432) may be disposed on the first guide portion (431). As the first sphere (432) is guided along the first guide portion (431), the first carrier (420) may move in the first direction. For example, as the first sphere (432) moves, rolls, rotates, or tilts along the first guide portion (431), the first carrier (420) may move in the first direction. When the first carrier (420) moves in the first direction, the lens assembly (410) accommodated in the first carrier (420) can move together in the first direction.

According to one embodiment, a second sphere (442) may be disposed between the first carrier (420) and the third carrier (440). In one example, when the OIS driving unit further includes a second carrier (430), the second sphere (442) may be disposed between the second carrier (430) and the third carrier (440). The first carrier (420) may move on the third carrier (440) by the second sphere (442). For example, the first carrier (420) may move in a first direction and/or a second direction with respect to the housing (450) by the second sphere (442).

According to one embodiment, at least one of the first carrier (420) or the third carrier (440) may include a second guide portion (441) that guides the second sphere (442) to move in the first direction and/or the second direction. In one example, when the OIS driving unit further includes the second carrier (430), at least one of the second carrier (430) or the third carrier (440) may include a second guide portion (441) that guides the second sphere (442) to move in the second direction. The second sphere (442) may be disposed on the second guide portion (441). The second sphere (442) may be disposed on the second guide portion (441). By guiding the second sphere (442) along the second guide portion (441), the first carrier (420) may move in the first direction and/or the second direction. For example, when the second sphere (442) moves, rolls, rotates, or tilts along the second guide portion (441), the first carrier (420) can move in the first direction and/or the second direction. When the first carrier (420) moves in the first direction and/or the second direction, the lens assembly (410) accommodated in the first carrier (420) can move together in the first direction and/or the second direction.

According to one embodiment, at least one of the first guide portion (431) or the second guide portion (441) may have a guide groove or rail shape, but is not limited thereto.

According to one embodiment, the OIS actuator may include at least one of a plurality of first spheres (432), a plurality of first guide portions (431), a plurality of second spheres (442), or a plurality of second guide portions (441).

According to one embodiment, the camera module (400) may include an AF driving unit for an auto-focus function. The AF driving unit may move the lens assembly (410) in a third direction (e.g., in the z-axis direction or the -z-axis direction). According to one embodiment, the AF driving unit may include an AF actuator configured to move the lens assembly (410) for the auto-focus function. For example, the AF driving unit may move the lens assembly (410) in a third direction in which a distance between the image sensor (460) and the lens assembly (410) is changed for the auto-focus function.

According to one embodiment, the AF driving unit may include a third carrier (440) and a third sphere (444). However, the components of the AF driving unit are not limited thereto. For example, the AF driving unit may omit at least one of the components described above, or may further include at least one other component. For example, the AF driving unit may further include a third magnet (445) and a third coil (453). In one example, the AF driving unit may also include a plurality of third spheres (444).

In one embodiment, the third carrier (440) can transport at least a portion of the OIS drive unit that accommodates the lens assembly (410) in the third direction. For example, the third carrier (440) can transport at least one of the first carrier (420) that accommodates the lens assembly (410) or the second carrier (430) in the third direction. In one example, the third carrier (440) can surround at least a portion of a side surface of the second carrier (430).

According to one embodiment, the third sphere (444) can be positioned between the third carrier (440) and the housing (450). The third sphere (444) can be positioned to move in a third direction with respect to the housing (450). For example, a plurality of third spheres (444) can be positioned side by side along the third direction.

According to one embodiment, a third magnet (445) may be disposed on one side (e.g., a side facing the +x direction) of a third carrier (440). A third coil (453) may be disposed to face the third magnet (445). For example, the third coil (453) may be disposed on an inner side of a housing (450) facing the one side of the third carrier on which the third magnet (445) is disposed. The third carrier (440) may be moved relative to the housing (450) by a third direction force acting on the third magnet (445) due to a magnetic field generated by a current flowing in the third coil (453). As the third carrier (440) moves, the lens assembly (410) may be moved in the third direction. For example, as the third carrier (440) moves, a distance from an image sensor (460) to the lens assembly (410) may be changed.

According to one embodiment, the third carrier (440) may include a third guide portion (443) that guides the third sphere (444) to move in a third direction. The third guide portion (443) may be formed to allow the third carrier (440) to move in the third direction. For example, the third guide portion (443) may be formed to have the third direction as a longitudinal direction. For example, the third guide portion (443) may have a guide groove or rail shape, but is not limited thereto. In one example, the third carrier (440) may include a plurality of third guide portions (443). For example, the third carrier (440) may include a plurality of third guide portions (443) arranged with a third magnet (445) therebetween. Although not shown in FIGS. 4 and 5, the housing (450) may include a guide portion formed at a position corresponding to the third guide portion (443) of the third carrier (440).

According to one embodiment, the third sphere (444) may be placed on the third guide portion (443). The third guide portion (443) may guide the third sphere (444) to move in a third direction. For example, the third sphere (444) may move, roll, rotate, or tilt on the third guide portion (443). However, the present invention is not limited thereto.

According to one embodiment, the third sphere (444) is guided along the third guide portion (443), so that the third carrier (440) can be transported in the third direction. When the third carrier (440) is transported in the third direction, at least some of the OIS driving units arranged on the third carrier (440) can be transported in the third direction. The lens assembly (410) can be transported in the third direction by the movement of at least some of the OIS driving units.

According to one embodiment, the housing (450) may form an outer appearance of the camera module (400). For example, the housing (450) may provide a skeleton for supporting other components. The shape of the housing (450) may be modified depending on the shape of the camera module (400) or components included in the camera module (400). For example, the housing (450) may have a box shape. However, the present invention is not limited thereto.

According to one embodiment, the housing (450) may provide a space for accommodating other components. Components included in the camera module (400) may be arranged inside the housing (450). For example, at least a portion of the lens assembly (410), an OIS driving unit for driving the OIS of the camera module (400), or at least one of an AF driving unit for driving the AF of the camera module (400) may be arranged in the space. However, the present invention is not limited thereto.

According to one embodiment, the image sensor (460) can detect light incident in the z-axis direction through the lens assembly (410). The electronic device can control the OIS driving unit (e.g., the first carrier (420) and/or the second carrier (430)) to move the position of the image detected through the image sensor (460).

FIGS. 4 and 5 are only examples for explaining the configuration of the camera module (400), and the configuration of the camera module (400) can be applied in various ways. For example, the OIS driving unit of the camera module (400) may be configured to move the position of the image sensor (460) rather than the lens assembly (410). Alternatively, the camera module (400) may include a curved camera in which the path (40) of light incident on the image sensor (460) is curved one or more times.

The method described in FIGS. 4 and 5 is described on the premise of a VCM (voice coil motor), but this is only an example and the technical idea of the present disclosure described in FIGS. 4 and 5 can also be applied to camera modules using motors of other types. Among the VCM type camera modules, depending on the form of electromagnetic force (e.g., Lorenz, solenoid), the shape of the magnet and the coil (e.g., the first coil (451) and the second coil (452)) and/or the driving method of the OIS driving unit may differ from the contents described in FIGS. 4 and 5.

FIG. 6 is a block diagram illustrating a structure in which an electronic device performs image stabilization processing according to one embodiment.

The content of Fig. 6 may be referenced by the content of Fig. 1, Fig. 2, Fig. 3, Fig. 4 and/or Fig. 5 described above. The same terminology is used for the same or substantially same configuration as described above.

Referring to FIG. 6, an electronic device (500) according to an embodiment may include a sensor (510), an OIS MCU (520), a processor (530), and a position control unit (540). However, the configuration of the electronic device (500) is not limited thereto. For example, the electronic device (500) may omit at least one of the above-described components, or may further include at least one other component. For example, the electronic device (500) may further include an AF control unit (531), a lens assembly (610), one or more magnets (611), one or more coils (612), and/or a position sensor (613) included in the processor (530).

According to one embodiment, the sensor (510) may be a separate configuration from the camera module (e.g., the camera module (320) of FIG. 3) and the processor (530). However, the present invention is not limited thereto. For example, the sensor (510) may be included in the camera module or may be included in the processor (530). For example, the sensor (510) may be included in a printed circuit board on which the processor (530) is mounted. In one example, the sensor (510) and the OIS MCU (520) may be a single configuration.

According to one embodiment, the OIS MCU (micro controller unit) (520) may be a separate configuration from the camera module (e.g., the camera module (320) of FIG. 3) and the processor (530). For example, the OIS MCU (520) may be a separate integrated circuit (IC). For example, the OIS MCU (520) may be included in a processor different from the processor (530). However, the present invention is not limited thereto. For example, the OIS MCU (520) may be included in the camera module or the processor (530). For example, the OIS MCU (520) may be included in a printed circuit board on which the processor (530) is mounted.

According to one embodiment, the position control unit (540) may be a separate configuration from the OIS MCU (520) and the processor (530). However, the present invention is not limited thereto. For example, the position control unit (540) may be included in the OIS MCU (520) or may be included in the processor (530). In one example, the position control unit (540) may be included in a processor different from the processor (530).

According to one embodiment, in the present disclosure, it may be understood that the operation of at least one of the sensor (510) or the camera module (for example, the camera module (320) of FIG. 3) is controlled by at least one of the processor (530) or the OIS MCU (520) executing instructions stored in a memory (for example, the memory (130) of FIG. 1). For example, it may be understood that the operation of the position control unit (540) is controlled by at least one of the processor (530) or the OIS MCU (520) executing instructions stored in a memory.

According to one embodiment, the sensor (510) can detect the movement of the electronic device (500). For example, the sensor (510) can detect at least one of the angular velocity or acceleration of the electronic device (500). The electronic device (500) can obtain movement information from the sensor (510). For example, the electronic device (500) can obtain at least one of the angular velocity information or the acceleration information from the sensor (510). However, the present invention is not limited thereto. For example, the electronic device (500) can obtain various information related to the movement of the electronic device (500) from the sensor (510).

According to one embodiment, the OIS MCU (520) may determine a target value related to OIS driving based on movement information of the electronic device (500) obtained from the sensor (510). For example, the OIS MCU (520) may determine at least one of an angular value or an angular velocity value based on the movement information. For example, the OIS MCU (520) may calculate at least one of an angular value or an angular velocity value related to OIS driving.

According to one embodiment, the OIS MCU (520) may include a first filter. In one example, the first filter may include a phase compensation filter (PCF). The first filter may compensate for a delay of angular velocity information acquired from the sensor (510). According to one embodiment, the OIS MCU (520) may include a second filter. In one example, the second filter may remove a direct current (DC) component of angular velocity information acquired from the sensor (510). For example, the second filter may remove a DC component of angular velocity information compensated by the first filter. According to one embodiment, the OIS MCU (520) may include a first operation unit. The first operation unit may change angular velocity information acquired from the sensor (510) into angle information. For example, the first operation unit may integrate angular velocity information acquired from the sensor (510) and change it into angle information. For example, the first operation unit can integrate the angular velocity value from which the DC component is removed by the second filter and change it into an angle value. According to one embodiment, the OIS MCU (520) can include a second operation unit. The second operation unit can adjust a suppression ratio (SR) of the angular velocity information acquired from the sensor (510). For example, the second operation unit can adjust the suppression ratio of the angular value acquired from the first operation unit. According to one embodiment, the OIS MCU (520) can include a variable attenuator. The variable attenuator can secure a margin for moving the lens assembly or the image sensor to a reference position.

According to one embodiment, the OIS MCU (520) can determine a target value related to OIS operation through the above-described configurations. However, the configuration of the OIS MCU (520) is not limited to the above-described configurations. For example, the OIS MCU (520) may omit at least one of the above-described configurations, or may further include at least one other configuration.

According to one embodiment, the processor (530) may include a VDIS driving unit, an EIS driving unit, and an AF control unit (531). However, the configuration of the processor (530) is not limited thereto. For example, the processor (530) may omit at least one of the above-described configurations, or may further include at least one other configuration. In one example, the processor (530) may include a sensor (510), an OIS MCU (520), and/or a position control unit (540).

According to one embodiment, the electronic device (500) may include a position sensor (613) that obtains position information of the lens assembly (610) or position information of the image sensor. For example, the camera module may include the position sensor (613). In one example, the camera module may include a plurality of position sensors (613). In one example, the position sensors (613) may include Hall sensors. However, the present invention is not limited thereto. For example, the camera module may include at least one of a tunnel magnetoresistance (TMR) sensor, an anisotropic magneto-resistance (AMR) sensor, or a giant magneto-resistance (GMR) sensor. The camera module can detect the movement (rotation) of a magnetic body based on a resistance value that changes based on the relative angle of a plurality of magnetic bodies by using a tunnel magneto-resistance sensor, or can detect the movement (rotation) of a magnetic body by using at least one of an anisotropic magneto-resistance (AMR) sensor or a giant magneto-resistance (GMR) sensor.

According to one embodiment, the position control unit (540) can obtain a target value related to the determined OIS driving from the OIS MCU (520). For example, the position control unit (540) can obtain at least one of the determined angle value or angular velocity value.

According to one embodiment, the position control unit (540) can obtain a sensing value related to the position information of the lens assembly (610) (or, image sensor) from the position sensor (613). For example, the position control unit (540) can obtain a sensing value related to at least one of the velocity information or the acceleration information of the lens assembly (610) (or, image sensor) from the position sensor (613).

According to one embodiment, the position control unit (540) may determine an output value for controlling the driving unit based on the determined target value and the sensing value. For example, the position control unit (540) may determine an output value related to the intensity of the current applied to the coil (612) based on the determined target value and the sensing value. The electronic device (500) may control the driving unit to move the position of the lens assembly (610) (e.g., the lens assembly (321) of FIG. 3) or the image sensor (e.g., the image sensor (322) of FIG. 3) based on the determined output value.

According to one embodiment, the target value determined by the OIS MCU (520) may be transmitted to the processor (530). For example, at least one of the determined angle value or angular velocity value may be transmitted to the processor (530). For example, the target value determined by the OIS MCU (520) may be transmitted to the VDIS driving unit and/or the EIS driving unit. The processor (530) may perform image processing on image frames acquired through the camera module based on the determined target value to perform image stabilization processing to reduce image shaking. For example, the processor (530) may perform image processing on image frames acquired through the camera module based on at least one of the determined angle value or angular velocity value to perform image stabilization processing to reduce image shaking. For example, the processor (530) may perform at least one of VDIS (video digital image stabilization) or EIS (electronic image stabilization) on the image frames. For example, the processor (530) may perform at least one of video digital image stabilization (VDIS) or electronic image stabilization (EIS) on image frames to compensate for shaking of the electronic device (500).

According to one embodiment, the processor (530) may obtain a sensing value related to the position information of the lens assembly (610) (or the image sensor) from the position sensor (613). For example, the processor (530) may obtain a sensing value related to the position information of the lens assembly (610) (or the image sensor) directly from the position sensor (613). For example, the processor (530) may obtain a sensing value related to at least one of the velocity information or the acceleration information of the lens assembly (610) (or the image sensor) from the position sensor (613). For example, the sensing value obtained from the position sensor (613) may be transmitted to the VDIS driving unit and/or the EIS driving unit. The processor (530) may perform image processing on image frames obtained through the camera module based on the sensing value to perform image stabilization processing to reduce shaking of the image. For example, the processor (530) may perform image stabilization processing to reduce image shake by performing image processing on image frames acquired through the camera module based on at least one of an angular value or an angular velocity value. For example, the processor (530) may perform at least one of VDIS (video digital image stabilization) or EIS (electronic image stabilization) on the image frames. For example, the processor (530) may perform at least one of VDIS (video digital image stabilization) or EIS (electronic image stabilization) on the image frames to compensate for shake of the electronic device (500).

According to one embodiment, the electronic device (500) may determine the camera operation mode as one of a plurality of operation modes associated with the operation of the driving unit. For example, the electronic device (500) may support a plurality of operation modes. For example, the electronic device (500) may support a first operation mode and a second operation mode. However, the present invention is not limited thereto. For example, the electronic device (500) may further support a third operation mode.

According to one embodiment, the second operation mode may include an operation mode that is performed when the movement of the electronic device shakes more than a certain frequency than the first operation mode. The third operation mode may include an operation mode that is performed when the movement of the electronic device shakes more than a certain frequency than the first operation mode and the second operation mode.

According to one embodiment, the electronic device (500) may determine the camera operation mode based on the movement information acquired from the sensor (510). For example, the electronic device (500) may perform at least one of the first operation mode, the second operation mode, or the third operation mode based on whether the frequency of the movement information satisfies a specified condition. For example, the electronic device (500) may perform the first operation mode based on the detection of a frequency component lower than a specified frequency value from the movement information. For example, the electronic device (500) may perform the second operation mode based on the detection of a frequency component higher than a specified frequency value from the movement information. However, the present invention is not limited thereto. For example, the electronic device (500) may perform at least one of the first operation mode, the second operation mode, or the third operation mode based on a user input. For example, the electronic device (500) may also perform at least one of the first operation mode, the second operation mode, or the third operation mode based on whether an external device is detected. In one example, at least two of the first operation mode, the second operation mode, or the third operation mode may be different operation modes. However, the present invention is not limited thereto. In one example, the first operation mode, the second operation mode, and the third operation mode may be performed independently. For example, the first operation mode, the second operation mode, and the third operation mode may be performed selectively.

According to one embodiment, when the electronic device (500) operates based on the first operation mode, the driving unit may control the position of the lens assembly (610) or the image sensor based on the first gain value. For example, when the electronic device (500) operates based on the first operation mode, the position control unit (540) may determine an output value for controlling the driving unit based on the first gain value.

According to one embodiment, when the electronic device (500) operates based on the second operation mode, the driving unit may control the position of the lens assembly (610) or the image sensor based on the second gain value. For example, when the electronic device (500) operates based on the second operation mode, the position control unit (540) may determine an output value for controlling the driving unit based on the second gain value. The second gain value may be a value greater than the first gain value.

According to one embodiment, when the electronic device (500) operates based on the third operation mode, the driving unit may control the position of the lens assembly (610) or the image sensor based on the third gain value. For example, when the electronic device (500) operates based on the third operation mode, the position control unit (540) may determine an output value for controlling the driving unit based on the third gain value. The third gain value may be a value greater than the second gain value. The third gain value may be a value greater than the first gain value.

According to one embodiment, the electronic device (500) may set a position corresponding to a target value determined based on movement information acquired from the sensor (510) based on the first operation mode as a target position to which the lens assembly (610) or the image sensor is moved. The electronic device (500) may control a driving unit to move the position of the lens assembly (610) or the image sensor to the target position according to the movement information based on the first operation mode.

According to one embodiment, the electronic device (500) may set a reference position within the driving range as a target position to which the lens assembly (610) or the image sensor is moved based on the second operation mode. The electronic device (500) may control the driving unit to position the lens assembly (610) or the image sensor at the reference position within the driving range based on the second operation mode. For example, the electronic device (500) may control the driving unit to move the lens assembly (610) or the image sensor to the reference position within the driving range based on the second operation mode.

According to one embodiment, the electronic device (500) may set a reference position within the driving range as a target position to which the lens assembly (610) or the image sensor is moved based on the third operation mode. The electronic device (500) may control the driving unit so that the lens assembly (610) or the image sensor is positioned at the reference position within the driving range based on the third operation mode. For example, the electronic device (500) may control the driving unit so that the lens assembly (610) or the image sensor is moved to the reference position within the driving range based on the third operation mode. In the present disclosure, the content regarding the third operation mode may be referenced by the content regarding the second operation mode.

FIG. 7 is a drawing for explaining an operation of controlling a driving unit based on a first operation mode according to one embodiment.

The camera module (700) of Fig. 7 can be referenced by the camera module described above.

Referring to FIG. 7, a camera module (700) according to one embodiment may include a lens assembly (710) and a housing (750). However, the present invention is not limited thereto. In the contents of FIG. 7, the lens assembly (710) may be replaced with an image sensor.

According to one embodiment, the lens assembly (710) (or, image sensor) may be arranged to be movable in a first axis (720) direction and a second axis (730) direction substantially perpendicular to an optical axis of the lens assembly (710) (e.g., optical axis (40) of FIG. 5) within the housing (750). The first axis (720) may be substantially perpendicular to the second axis (730). However, the present invention is not limited thereto. For example, the lens assembly (710) (or, image sensor) may be arranged to be movable in a third axis direction substantially parallel to the optical axis of the lens assembly (710) within the housing (750).

According to one embodiment, the electronic device may control the driving unit (for example, the driving unit (333) of FIG. 3) based on the first operation mode to move the position of the lens assembly (710) (or the image sensor). For example, the electronic device may control the OIS driving unit based on the first operation mode to move the position of the lens assembly (710) on the first and second axis planes. For example, the electronic device may move the position of the lens assembly (710) on the x-y axis plane. However, the present invention is not limited thereto. For example, the electronic device may control the AF driving unit based on the first operation mode to move the position of the lens assembly (710) (or the image sensor) on the third axis that is substantially perpendicular to the first and second axis planes. The lens assembly (710) (or the image sensor) may move within the driving range. Referring to FIG. 7, the driving range may include the interior of the camera module (700) and/or the interior of the housing (750). However, it is not limited to this.

According to one embodiment, the electronic device may control the driving unit according to movement information acquired from a sensor (e.g., the sensor (310) of FIG. 3) based on the first operation mode. For example, the electronic device may control the OIS driving unit (e.g., the OIS driving unit of FIG. 4) to control the position of the lens assembly (710) in the first axis (720) direction. The electronic device may control the OIS driving unit to control the position of the lens assembly (710) in the second axis (730) direction. The electronic device may move the lens assembly (710) (or the image sensor) in the target movement direction on the x-y axis plane through the position movement in the first axis (720) direction and the position movement in the second axis (730) direction.

According to one embodiment, by controlling the OIS driving unit, when no offset is applied to the lens assembly (710), the shape of the subject can appear at a first position in an image acquired through the camera. When the lens assembly (710) moves in a target movement direction by controlling the OIS driving unit, the shape of the subject can appear at a second position in the image that has moved in the opposite direction.

In one embodiment, when operating based on the first operation mode, the electronic device may control the driving unit to control the position of the lens assembly (710) (or, image sensor) based on the first gain value. For example, the electronic device may control the driving unit to fix the position of the lens assembly (710) (or, image sensor) based on the first gain value.

FIG. 8 is a drawing for explaining an operation of controlling a driving unit based on a second operation mode according to one embodiment. In the present disclosure, an operation of controlling a driving unit based on a third operation mode may be referred to as an operation of controlling a driving unit based on the second operation mode.

The camera module (700) of Fig. 8 can be referenced by the camera module described above.

According to one embodiment, the electronic device may control the driving unit (for example, the driving unit (333) of FIG. 3) based on the second operation mode to move the lens assembly (710) (or the image sensor) to a reference position (740) within the driving range. For example, the electronic device may control the OIS driving unit based on the second operation mode to move the lens assembly (710) to the reference position (740). For example, the electronic device may move the lens assembly (710) to a reference position (740) on an x-y axis plane. However, the present invention is not limited thereto. For example, the electronic device may control the AF driving unit based on the second operation mode to move the lens assembly (710) (or the image sensor) to a reference position (not shown) on a third axis that is substantially perpendicular to the first and second axis planes.

According to one embodiment, the electronic device can control the OIS driving unit (e.g., the OIS driving unit of FIG. 4) to move the lens assembly (710) to a reference position (740) in the direction of the first axis (720). The electronic device can control the OIS driving unit to move the lens assembly (710) to a reference position (740) in the direction of the second axis (730). The electronic device can move the lens assembly (710) (or the image sensor) in a direction toward the reference position (740) on the x-y axis plane through the position movement in the direction of the first axis (720) and the position movement in the direction of the second axis (730).

According to one embodiment, the reference position (740) may include at least one of the center of the driving range on the first axis (720) or the center of the driving range on the second axis (730), but is not limited thereto. For example, the reference position (740) may include at least one of the center of the camera module (700) or the center of the housing (750). In one example, the reference position with respect to the AF driving unit may include at least one of the infinity focus position or the 1.5 to 3 M focus position on the third axis, but is not limited thereto.

According to one embodiment, when operating based on the second operation mode, the electronic device may control the driving unit to control the position of the lens assembly (710) (or the image sensor) based on the second gain value. For example, the electronic device may control the driving unit so that the position of the lens assembly (710) (or the image sensor) is maintained (e.g., fixed) based on the second gain value. For example, the electronic device may control the OIS driving unit based on the second gain value so that the position of the lens assembly (710) (or the image sensor) is maintained at the reference position (740) with respect to the first direction (e.g., the first axis (720) direction or the x-axis direction) and the second direction (e.g., the second axis (730) direction or the y-axis direction). For example, the electronic device may control the AF driving unit based on the second gain value so that the position of the lens assembly (710) (or the image sensor) is maintained at the reference position with respect to the third direction (e.g., the third-axis direction or the z-axis direction).

In one embodiment, the second gain value can be a different value than the first gain value. For example, the second gain value can be a value greater than the first gain value.

According to one embodiment, the electronic device controls the driving unit based on a second gain value that is greater than the first gain value, so that the position of the lens assembly (710) (or the image sensor) can be more strongly maintained in the second operation mode than in the first operation mode. For example, the position of the lens assembly (710) (or the image sensor) can be more strongly maintained at the reference position (740). In one example, being 'strongly maintained' can include that the displacement that occurs when the same disturbance is applied can be less.

FIG. 9 is a diagram for explaining an operation of controlling the position of a lens assembly or an image sensor based on a gain value according to one embodiment.

The contents of Fig. 9 may be referenced by the contents of other drawings.

According to one embodiment, the position control unit (840) can obtain a target value from the OIS MCU (820). For example, the position control unit (840) can obtain at least one of an angular velocity value or an acceleration value determined by the OIS MCU (820) from the OIS MCU (820). The target value can be determined based on movement information obtained from a sensor. For example, the target value can be determined based on at least one of angular velocity information obtained from a gyro sensor or acceleration information obtained from an acceleration sensor.

According to one embodiment, the electronic device may include a position sensor (913) that obtains position information of the lens assembly (910) or position information of the image sensor. For example, the camera module may include the position sensor (913). In one example, the electronic device may include a plurality of position sensors (913). In one example, the position sensors (913) may include a Hall sensor. However, the present invention is not limited thereto. For example, the camera module may include at least one of a tunnel magnetoresistance (TMR) sensor, an anisotropic magneto-resistance (AMR) sensor, or a giant magneto-resistance (GMR) sensor. The camera module may use the tunnel magnetoresistance sensor to detect movement (rotation) of a plurality of magnetic bodies based on a resistance value that changes based on a relative angle of the magnetic bodies, or may use at least one of the anisotropic magneto-resistance (AMR) sensor or the giant magneto-resistance (GMR) sensor to detect movement (rotation) of the magnetic body.

According to one embodiment, the driving unit may include one or more magnets (911) arranged in the driving unit and one or more coils (912) for controlling the position of the lens assembly (910) or the position of the image sensor. The magnets (911) may be arranged to face the coils (912). The magnets (911) may be referenced by at least one of the first magnets (421), the second magnets (422), or the third magnets (445) of FIGS. 4 and 5 described above. The coils (912) may be referenced by at least one of the first coils (451), the second coils (452), or the third coils (453) of FIGS. 4 and 5 described above.

According to one embodiment, the position control unit (840) can obtain sensing values related to position information of the lens assembly (910) or the image sensor from the position sensor (913). The position sensor (913) can detect the position of the lens assembly (910) or the image sensor. The position sensor (913) can detect the position of the driving unit that moves the lens assembly (910) or the image sensor.

According to one embodiment, the position control unit (840) may determine an output value for controlling the intensity of a current applied to one or more coils (912) based on a target value obtained from the OIS MCU (820) and a sensing value obtained from the position sensor (913). For example, the position control unit (840) may determine an output value based on a difference value between the target value obtained from the OIS MCU (820) and the sensing value obtained from the position sensor (913). For example, the position control unit (840) may determine an output value by adjusting a difference value between the target value obtained from the OIS MCU (820) and the sensing value obtained from the position sensor (913). For example, the position control unit (840) can determine an output value by multiplying a difference value between a target value acquired from the OIS MCU (820) and a sensing value acquired from the position sensor (913) by a specific value (first control (841)), integrating the difference value (second control (842)), and/or differentiating the difference value (third control (843)).

According to one embodiment, the position control unit (840) can determine the output value based on the gain value. For example, the position control unit (840) can determine the output value by adjusting the gain value. For example, the position control unit (840) can determine the output value by adjusting the gain value related to the difference value and/or the coefficient related to the difference value. The operation of adjusting the gain value may include the operation of adjusting the gain value related to the difference value and/or the coefficient related to the difference value. For example, the operation of adjusting the gain value may include the operation of adjusting the coefficient related to the first control (841), the second control (842), and/or the third control (843). For example, the operation of adjusting the gain value may include the operation of adjusting at least one of a coefficient multiplied by a value obtained by integrating the difference value or a coefficient multiplied by a value obtained by differentiating the difference value.

In one embodiment, when operating based on the first operation mode, the electronic device may control the driving unit to control the position of the lens assembly (910) or the image sensor based on the first gain value. When operating based on the second operation mode, the electronic device may control the driving unit to control the position of the lens assembly (910) or the image sensor based on the second gain value. In one example, when operating based on the first operation mode, the position control unit (840) may determine the output value based on the first gain value. When operating based on the second operation mode, the position control unit (840) may determine the output value based on the second gain value. The second gain value may be a value greater than the first gain value. For example, when operating based on the second operation mode, the position control unit (840) may determine the output value based on the second gain value greater than the first gain value.

According to one embodiment, based on the determined output value, the electronic device can control the current flowing in the coil (912). For example, when operating based on the first operation mode, the electronic device can control the intensity of the current flowing in the coil (912) based on the first output value determined based on the first gain value. For example, when operating based on the second operation mode, the electronic device can control the intensity of the current flowing in the coil (912) based on the second output value determined based on the second gain value. According to one embodiment, since the second gain value is greater than the first gain value, the position of the lens assembly (910) (or, the image sensor) can be maintained more strongly in the second operation mode than in the first operation mode. For example, the position of the lens assembly (910) (or, the image sensor) can be maintained more strongly at a reference position (e.g., the reference position (740) of FIG. 8).

FIG. 10 is a flowchart illustrating a process (1000) in which an electronic device performs a first operation mode or a second operation mode based on movement information according to one embodiment.

In the present disclosure, the operation of the electronic device may be understood as being performed by one or more processors (e.g., the processor (120) of FIG. 1) executing instructions stored in a memory (e.g., the memory (130) of FIG. 1) to perform calculations or control components of the electronic device.

According to one embodiment, in operation 1001, the electronic device can verify the motion information. For example, the electronic device can verify the motion information acquired from the sensor. For example, the electronic device can verify the frequency of the motion information. For example, the electronic device can verify whether a frequency component higher than a specified frequency value is detected from the motion information. For example, the electronic device can verify whether a frequency component lower than a specified frequency value is detected from the motion information. However, the present invention is not limited thereto. For example, the electronic device can verify the time for which the frequency of the motion information is maintained. For example, the electronic device can verify the time for which the detection of a frequency component higher than a specified frequency value is maintained from the motion information. For example, the electronic device can verify the time for which the detection of a frequency component lower than a specified frequency value is maintained from the motion information.

According to one embodiment, based on the movement information confirmed in operation 1001, in operation 1002, the electronic device can determine the type of the camera operation mode. For example, the electronic device can determine which operation mode the currently performed camera operation mode (hereinafter referred to as “current operation mode”) is among a plurality of operation modes. For example, the electronic device can determine whether the current operation mode is a second operation mode. For example, the electronic device can determine whether the current operation mode is a first operation mode. However, the present invention is not limited thereto. For example, the electronic device can determine whether the current operation mode is a third operation mode.

According to one embodiment, in response to determining that the camera operation mode (e.g., the current operation mode) is the first operation mode in operation 1002, in operation 1003, the electronic device may determine whether the motion information satisfies a specified condition. In one example, the electronic device may determine whether a frequency of the motion information satisfies a specified condition. For example, the electronic device may determine whether a frequency component higher than a specified frequency value is detected from the motion information acquired from the sensor. For example, the electronic device may determine whether a frequency component lower than a specified frequency value is detected from the motion information acquired from the sensor. For example, the electronic device may detect a frequency component higher than a specified frequency value based on at least one of a periodicity of the motion information or a differential value of the motion information. For example, the electronic device may detect a frequency component lower than a specified frequency value based on at least one of a periodicity of the motion information or a differential value of the motion information. In one example, the specified frequency value may include a frequency value higher than a band of interest of the OIS operation. For example, the specified frequency value may be greater than or equal to 10 Hz. However, it is not limited thereto. In one example, the specified frequency value may include the frequency value of the motion information acquired during drone filming. In one example, the specified frequency value may include the frequency value of the motion information acquired during filming while the electronic device is mounted on a motorcycle, cart, or bicycle. In one example, the electronic device may determine whether the time for which the frequency of the motion information is maintained satisfies the specified condition. For example, the electronic device may determine whether the time for which the detection of a frequency component higher than the specified frequency value from the motion information is maintained is longer than the specified time. For example, the electronic device may determine whether the time for which the detection of a frequency component higher than the specified frequency value from the motion information is maintained is shorter than the specified time. For example, the electronic device may determine whether the time for which the detection of a frequency component lower than the specified frequency value from the motion information is maintained is longer than the specified time. For example, the electronic device may determine whether the time for which the detection of a frequency component lower than the specified frequency value from the motion information is maintained is shorter than the specified time.

According to one embodiment, if it is determined that the motion information satisfies a specified condition in operation 1003, the electronic device may change the first operation mode to the second operation mode in operation 1004. For example, the electronic device may change the operation mode from the first operation mode, which is the current operation mode, to the second operation mode. For example, if it is determined that the frequency of the motion information satisfies the specified condition, the electronic device may change to the second operation mode. For example, if it is determined that a frequency component higher than a specified frequency value is detected from the motion information, the electronic device may change the camera operation mode so that the current operation mode becomes the second operation mode from the first operation mode. For example, if it is determined that a frequency value higher than 10 Hz is detected from the motion information, the electronic device may change to the second operation mode. However, the present invention is not limited thereto. For example, if it is determined that the time for which the frequency of the motion information is maintained satisfies the specified condition, the electronic device may change to the second operation mode. For example, if it is determined that the detection of a frequency component higher than a specified frequency value from the motion information continues for a specified period of time or longer, the electronic device can change the camera operation mode so that the current operation mode is from the first operation mode to the second operation mode.

According to one embodiment, based on the change to the second operation mode in operation 1004, in operation 1005, the electronic device can perform the second operation mode. For example, based on the second operation mode, the electronic device can control the driving unit so that the lens assembly or the image sensor is positioned at a reference position within the driving range. The electronic device can control the position of the lens assembly or the image sensor based on the second gain value. The electronic device can fix the target value output from the position sensor to a designated value. While controlling the driving unit, the electronic device can perform image processing on image frames acquired through the camera module to perform image stabilization processing to reduce shaking of the image.

According to one embodiment, in operation 1007, the electronic device can determine whether the operation of capturing an image through the camera has ended. If the operation of capturing an image has ended, the electronic device can end the process for capturing an image. If the operation of capturing an image has not ended, the electronic device can check movement information in operation 1001.

According to one embodiment, if it is determined that the motion information does not satisfy the specified condition in operation 1003, then in operation 1006, the electronic device may perform the first operation mode. For example, the electronic device may maintain the first operation mode, which is the current operation mode. For example, if it is determined that the frequency of the motion information does not satisfy the specified condition, the electronic device may perform the first operation mode. For example, if it is determined that a frequency component higher than the specified frequency value is not detected from the motion information, the electronic device may perform the first operation mode, which is the current operation mode. For example, if it is determined that a frequency component lower than the specified frequency value is detected from the motion information, the electronic device may perform the first operation mode, which is the current operation mode. For example, if it is determined that a frequency value lower than 10 Hz is detected from the motion information, the electronic device may perform the first operation mode. However, the present invention is not limited thereto. For example, if it is determined that the time during which the frequency of the motion information is maintained does not satisfy the specified condition, the electronic device may perform the first operation mode. For example, if it is determined that the time for which detection of a frequency component lower than a specified frequency value from the motion information is maintained is longer than a specified time, the electronic device can perform the first operation mode.

According to one embodiment, in operation 1006, the electronic device may control the driving unit to move the position of the lens assembly or the position of the image sensor based on the motion information acquired from the sensor based on the first operation mode. The electronic device may control the position of the lens assembly or the image sensor based on the first gain value. For example, the electronic device may control the driving unit to move the lens assembly or the image sensor in a target motion direction based on the acquired motion information. The lens assembly or the image sensor may be moved to the target position. While controlling the driving unit, the electronic device may perform image processing on image frames acquired through the camera module to perform image stabilization processing to reduce shaking of the image.

According to one embodiment, in operation 1007, the electronic device can determine whether the operation of capturing an image through the camera has ended. If the operation of capturing an image has ended, the electronic device can end the process for capturing an image. If the operation of capturing an image has not ended, the electronic device can check movement information in operation 1001.

In one embodiment, in response to determining in operation 1002 that the camera operation mode (e.g., the current operation mode) is not the first operation mode, in operation 1008, the electronic device may determine whether the motion information satisfies the specified condition. For example, in response to determining that the camera operation mode is the second operation mode, the electronic device may determine whether the motion information satisfies the specified condition. In one example, the electronic device may determine whether the frequency of the motion information satisfies the specified condition. For example, the electronic device may determine whether a frequency component greater than or equal to a specified frequency value is detected from the motion information acquired from the sensor. For example, the electronic device may determine whether a frequency component less than or equal to a specified frequency value is detected from the motion information acquired from the sensor. For example, the electronic device may detect a frequency component greater than or equal to a specified frequency value based on at least one of a periodicity of the motion information or a differential value of the motion information. For example, the electronic device may detect a frequency component less than or equal to a specified frequency value based on at least one of a periodicity of the motion information or a differential value of the motion information. In one example, the specified frequency value may include a frequency value higher than the band of interest of the OIS operation. For example, the specified frequency value may be greater than or equal to 10 Hz. However, the present invention is not limited thereto. In one example, the specified frequency value may include a frequency value of motion information acquired during drone filming. In one example, the specified frequency value may include a frequency value of motion information acquired during filming while the electronic device is mounted on a motorcycle, a cart, or a bicycle. In one example, the electronic device may determine whether the time for which the frequency of the motion information is maintained satisfies a specified condition. For example, the electronic device may determine whether the time for which the detection of a frequency component higher than or equal to the specified frequency value from the motion information is maintained is longer than or equal to the specified time. For example, the electronic device may determine whether the time for which the detection of a frequency component higher than or equal to the specified frequency value from the motion information is maintained is shorter than or equal to the specified time. For example, the electronic device may determine whether the time for which the detection of a frequency component lower than or equal to the specified frequency value from the motion information is maintained is longer than or equal to the specified time.

According to one embodiment, if it is determined that the motion information satisfies the specified condition in operation 1008, then in operation 1009, the electronic device may perform the second operation mode. For example, the electronic device may maintain the second operation mode, which is the current operation mode. For example, if it is determined that the frequency of the motion information satisfies the specified condition, the electronic device may perform the second operation mode. For example, if it is determined that a frequency component higher than the specified frequency value is detected from the motion information, the electronic device may perform the second operation mode, which is the current operation mode. For example, if it is determined that a frequency component higher than the specified frequency value is detected from the motion information, the electronic device may perform the second operation mode, which is the current operation mode. For example, if it is determined that a frequency value higher than 10 Hz is detected from the motion information, the electronic device may perform the second operation mode. However, the present invention is not limited thereto. For example, if it is determined that the time during which the frequency of the motion information is maintained satisfies the specified condition, the electronic device may perform the second operation mode. For example, if it is determined that the detection of a frequency component higher than a specified frequency value from the motion information continues for a specified period of time, the electronic device may perform the second operation mode.

According to one embodiment, in operation 1009, the electronic device may control the driving unit so that the lens assembly or the image sensor is positioned at a reference position within the driving range based on the second operation mode. The electronic device may control the position of the lens assembly or the image sensor based on the second gain value. The electronic device may fix the target value output from the position sensor to a designated value. While controlling the driving unit, the electronic device may perform image processing on image frames acquired through the camera module to perform image stabilization processing to reduce shaking of the image.

According to one embodiment, in operation 1007, the electronic device can determine whether the operation of capturing an image through the camera has ended. If the operation of capturing an image has ended, the electronic device can end the process for capturing an image. If the operation of capturing an image has not ended, the electronic device can check movement information in operation 1001.

According to one embodiment, if it is determined that the motion information does not satisfy the specified condition in operation 1008, then in operation 1010, the electronic device may change the second operation mode to the first operation mode. For example, the electronic device may change the operation mode from the second operation mode, which is the current operation mode, to the first operation mode. For example, if it is determined that the frequency of the motion information does not satisfy the specified condition, the electronic device may change to the first operation mode. For example, if it is determined that a frequency component lower than the specified frequency value is detected from the motion information, the electronic device may change the camera operation mode such that the current operation mode becomes the first operation mode from the second operation mode. For example, if it is determined that a frequency value lower than 10 Hz is detected from the motion information, the electronic device may change to the first operation mode. However, the present invention is not limited thereto. For example, if it is determined that the time for which the frequency of the motion information is maintained satisfies the specified condition, the electronic device may change to the first operation mode. For example, if it is determined that the detection of a frequency component lower than a specified frequency value from the motion information is maintained for a specified period of time or longer, the electronic device can change the camera operation mode so that the current operation mode is from the second operation mode to the first operation mode.

According to one embodiment, based on the change to the first operation mode in operation 1010, in operation 1011, the electronic device can perform the first operation mode. For example, the electronic device can control the driving unit to move the position of the lens assembly or the position of the image sensor according to the motion information acquired from the sensor based on the first operation mode. The electronic device can control the position of the lens assembly or the image sensor based on the first gain value. For example, the electronic device can control the driving unit to move the lens assembly or the image sensor in a target movement direction based on the acquired motion information. The lens assembly or the image sensor can be moved to the target position. While controlling the driving unit, the electronic device can perform image processing on image frames acquired through the camera module to perform image stabilization processing to reduce shaking of the image.

According to one embodiment, in operation 1007, the electronic device can determine whether the operation of capturing an image through the camera has ended. If the operation of capturing an image has ended, the electronic device can end the process for capturing an image. If the operation of capturing an image has not ended, the electronic device can check movement information in operation 1001.

The operation of the electronic device based on the second operation mode is specifically described in Fig. 14.

FIG. 11 is a flowchart illustrating a process (1100) in which an electronic device changes a first operation mode to a second operation mode and performs the second operation mode according to one embodiment.

FIG. 11 is a flowchart illustrating a process when it is determined that the frequency of movement information acquired from a sensor satisfies a specified condition, as an example of the embodiment of FIG. 10, when the current operation mode is the first operation mode.

According to one embodiment, in operation 1110, the electronic device may control the driving unit based on the first operation mode. For example, if the frequency of the motion information acquired from the sensor does not satisfy the specified condition, the electronic device may control the driving unit based on the first operation mode. For example, the electronic device may control the driving unit to move the position of the lens assembly or the image sensor according to the motion information acquired from the sensor.

In one embodiment, in operation 1110, when operating based on the first operation mode, the electronic device can control the driving unit to control the position of the lens assembly or the image sensor based on the first gain value.

According to one embodiment, in operation 1120, the electronic device may determine whether to change the first operation mode to the second operation mode based on the frequency of the motion information. For example, the electronic device may determine whether to change the first operation mode to the second operation mode based on whether the frequency of the motion information satisfies a specified condition. For example, the electronic device may determine whether to change the first operation mode to the second operation mode based on whether a frequency component higher than a specified frequency value is detected from the motion information. For example, the electronic device may change the first operation mode to the second operation mode if the frequency of the motion information satisfies the specified condition. For example, the electronic device may change the first operation mode to the second operation mode if a frequency component higher than a specified frequency value is detected from the motion information. For example, the electronic device may maintain the first operation mode if the frequency of the motion information does not satisfy the specified condition. For example, the electronic device may maintain the first operation mode if a frequency component higher than a specified frequency value is not detected from the motion information.

According to one embodiment, in response to changing the first operation mode to the second operation mode in operation 1120, in operation 1130, the electronic device may position the lens assembly or the image sensor at the reference position based on the second operation mode. For example, the electronic device may control the driving unit to position the lens assembly or the image sensor at the reference position based on the second operation mode. For example, the electronic device may control the OIS driving unit to position the lens assembly or the image sensor at at least one of the center of the driving range along the first axis or the center of the driving range along the second axis. For example, the electronic device may control the AF driving unit to position the lens assembly or the image sensor at the reference position along the third axis.

In one embodiment, in operation 1130, when operating based on the second operation mode, the electronic device may control the driving unit to control the position of the lens assembly or the image sensor based on the second gain value. In one example, the second gain value may be a different value from the first gain value. For example, the second gain value may be greater than the first gain value. However, the present invention is not limited thereto.

According to one embodiment, in operation 1140, the electronic device may perform image processing on image frames acquired through the camera module while controlling the driving unit based on the second operation mode. The electronic device may perform image stabilization processing to reduce shaking of the image by performing image processing on the image frames. For example, the electronic device may perform at least one of video digital image stabilization (VDIS) or electronic image stabilization (EIS) on the image frames. For example, the electronic device may perform at least one of video digital image stabilization (VDIS) or electronic image stabilization (EIS) on the image frames to correct shaking of the electronic device (500). The electronic device (500) may output a video or preview image on which image stabilization is performed through VDIS (and/or EIS) through a display.

According to one embodiment, in response to not changing the first operation mode to the second operation mode in operation 1120, the electronic device may perform the first operation mode. For example, the electronic device may maintain the first operation mode, which is the current operation mode. The electronic device may perform image processing on image frames acquired through the camera module while controlling the driving unit based on the first operation mode. The electronic device may perform image stabilization processing to reduce shaking of the image by performing image processing on the image frames. For example, the electronic device may perform at least one of video digital image stabilization (VDIS) or electronic image stabilization (EIS) on the image frames. For example, the electronic device may perform at least one of video digital image stabilization (VDIS) or electronic image stabilization (EIS) on the image frames to compensate for shaking of the electronic device (500). The electronic device (500) may output a video or preview image on which image stabilization is performed through VDIS (and/or EIS) through a display.

FIG. 12 is a diagram for explaining an operation for determining whether a frequency of motion information satisfies a specified condition according to one embodiment. FIG. 13 is a diagram for explaining an operation for determining whether a frequency of motion information satisfies a specified condition according to one embodiment.

The following description of FIGS. 12 and 13 may be examples of an operation for determining whether a frequency of motion information satisfies a specified condition. For example, the electronic device may detect a frequency component greater than or equal to a specified frequency value based on at least one of a periodicity of the motion information (see FIG. 12) or a differential value of the motion information (see FIG. 13). However, the present invention is not limited thereto. For example, the electronic device may detect a frequency component greater than or equal to a specified frequency value based on a value obtained by Fourier transforming the motion information.

The contents of Fig. 12 may be referenced by the contents of other drawings. The contents of Fig. 13 may be referenced by the contents of other drawings.

Operation 1010 of Fig. 10 can be referenced by the contents of Fig. 12.

Referring to FIG. 12, according to one embodiment, the electronic device may determine whether a frequency of the motion information satisfies a specified condition based on the motion information acquired from the sensor. For example, the electronic device may detect a frequency component higher than a specified frequency value from the motion information. In one example, the specified frequency value may include a frequency value higher than a band of interest of the OIS operation. For example, the specified frequency value may be greater than or equal to 10 Hz. However, the present invention is not limited thereto. In one example, the specified frequency value may include a frequency value of motion information acquired during drone filming. In one example, the specified frequency value may also include a frequency value of motion information acquired during filming while the electronic device is mounted on a cart or a bicycle.

According to one embodiment, the electronic device can detect a frequency component greater than or equal to a specified frequency value based on the periodicity of the motion information. For example, referring to FIG. 12, the electronic device can detect a frequency component greater than or equal to a specified frequency value based on a period in which an angular velocity signal (1210) acquired from a sensor has a value of 0. For example, if the interval (1211) of the period of the angular velocity signal (1210) satisfies a specified condition, the electronic device can determine that the frequency of the motion information has a frequency component greater than or equal to the specified frequency value. For example, if the interval (1211) of the period of the angular velocity signal (1210) becomes smaller than the specified interval, the electronic device can determine that the frequency of the motion information has a frequency component greater than or equal to the specified frequency value. In one example, if the interval (1211) of the period of the angular velocity signal (1210) remains smaller than the specified interval, the electronic device can determine that the frequency of the motion information has a frequency component greater than or equal to the specified frequency value.

According to one embodiment, the amplitude of the angular velocity signal (1210) may be affected by the determination of a specified condition. For example, if the amplitude is not large but is a high frequency greater than a specified frequency interval (1211), it may be a general noise signal and does not affect the lens shake, and thus does not affect the clarity of the image. However, if it is a high frequency signal with a large amplitude, it may affect the movement of the lens. Therefore, the electronic device may determine a given condition by considering the amplitude of the angular velocity signal (1210).

Operation 1010 of Fig. 10 can be referenced by the contents of Fig. 13.

Referring to FIG. 13, according to one embodiment, the electronic device may determine whether a frequency of the motion information satisfies a specified condition based on the motion information acquired from the sensor. For example, the electronic device may detect a frequency component higher than a specified frequency value from the motion information. In one example, the specified frequency value may include a frequency value higher than a band of interest of the OIS operation. For example, the specified frequency value may be greater than or equal to 10 Hz. However, the present invention is not limited thereto. In one example, the specified frequency value may include a frequency value of motion information acquired during drone filming. In one example, the specified frequency value may also include a frequency value of motion information acquired during filming while the electronic device is mounted on a cart or a bicycle.

According to one embodiment, the electronic device may detect a frequency component higher than a specified frequency value based on a differential value of the motion information. For example, referring to FIG. 13, a first graph (1310) may be a graph representing an angular velocity value of the motion information over time, and a second graph (1320) may be a graph representing a differential value of the angular velocity of the motion information over time. A first line (1311) of the first graph (1310) and a second line (1321) of the second graph (1320) may represent whether a frequency of the motion information has a frequency component higher than a specified frequency value.

When referring to the first line (1311) and the second line (1312), it may be easier to use the second graph (1320) than the first graph (1310) to determine whether the frequency of the motion information has a frequency component higher than a specified frequency value.

FIG. 14 is a flowchart illustrating a process (1070) of a second operation mode according to one embodiment.

The process (1070) of FIG. 14 may be referenced by the operation 1070 of FIG. 10. The contents of FIG. 14 may be referenced by the contents of other drawings.

In one embodiment, in operation 1071, the electronic device may position the lens assembly or the image sensor at a reference position. For example, the electronic device may control a driving unit so that the lens assembly or the image sensor is positioned at the reference position. For example, the electronic device may control an OIS driving unit so that the lens assembly or the image sensor is positioned at the reference position. For example, the electronic device may control an AF driving unit so that the lens assembly or the image sensor is positioned at the reference position.

According to one embodiment, in operation 1073, the electronic device may control the position of the lens assembly or the image sensor based on the second gain value. For example, the electronic device may maintain the position of the lens assembly or the image sensor based on the second gain value. For example, the electronic device may maintain the position of the lens assembly or the image sensor at the reference position based on the second gain value. For example, the electronic device may control the OIS driving unit based on the second gain value so that the lens assembly or the image sensor is maintained at the reference position with respect to the first direction (e.g., the x-axis direction) and the second direction (e.g., the y-axis direction). For example, the electronic device may control the AF driving unit based on the second gain value so that the lens assembly or the image sensor is maintained at the reference position with respect to the third direction (e.g., the z-axis direction). Therefore, in a mode (e.g., the second operation mode) in which image stabilization is performed using VDIS (and/or EIS), the position of the lens assembly or the image sensor may be stably maintained even if the electronic device shakes more than a specific frequency.

In one embodiment, the second gain value may be different from the first gain value when operating based on the first operating mode. For example, the second gain value may be greater than the first gain value. In one embodiment, the first gain value and the second gain value may include gain values for controlling current flowing in one or more coils (e.g., the first coil (451), the second coil (452), the third coil (453) of FIGS. 4 and 5 and/or the coil (612) of FIG. 6) based on position information acquired by the driving unit from the position sensor.

According to one embodiment, in operation 1075, the electronic device may perform image stabilization processing on image frames acquired through the camera module while controlling the driving unit based on the second operation mode. For example, the electronic device may perform image stabilization processing on image frames acquired through the camera module while maintaining the position of the lens assembly or the position of the image sensor based on the second operation mode. For example, the electronic device may perform image stabilization processing using VDIS (and/or EIS) on image frames acquired through the camera module while maintaining the position of the lens assembly or the position of the image sensor based on the second operation mode. Accordingly, even when the electronic device captures a video and/or a preview video in a situation where it shakes significantly, the position of the lens assembly or the position of the image sensor is maintained, so that the performance of the VDIS (and/or EIS) may be improved and a video and/or a preview video with improved quality may be provided.

FIG. 15 is a diagram for explaining an operation of controlling the position of a lens assembly or an image sensor based on a first gain value and a second gain value different from each other according to one embodiment.

The contents of Fig. 15 may be referenced by the contents of other drawings.

The first graph (1510) of FIG. 15 may be a Bode plot illustrating a first gain value (1511) and a first phase (1512) in a first operation mode according to an embodiment, and the second graph (1520) may be a Bode plot illustrating a second gain value (1521) and a second phase (1522) in a second operation mode according to an embodiment. Referring to FIG. 15, the second gain value (1521) of the second operation mode may be greater than the first gain value (1511) of the first operation mode.

FIG. 16 is a flowchart illustrating a process (1600) for fixing a target value to a specified value according to one embodiment.

According to one embodiment, operation 1610 may be a subsequent operation to operation 1073 of FIG. 10, but is not limited thereto. According to one embodiment, operation 1075 of FIG. 10 may be a subsequent operation to operation 1620, but is not limited thereto.

In one embodiment, at operation 1610, the electronic device can fix the target value to a specified value. For example, the electronic device can set the target value to a value of 0. In one example, the target value can include a value output from a position sensor (e.g., position sensor 612 of FIG. 6). In one example, the target value can include an output value transmitted from the position sensor to the VDIS (and/or EIS) drive. In one example, setting the target value to a value of 0 can include maintaining the lens assembly or the image sensor positioned at a reference position.

According to one embodiment, at operation 1620, the electronic device may control the intensity of a current applied to one or more coils (e.g., the first coil (451), the second coil (452), the third coil (453) of FIGS. 4 and 5, and/or the coil (612) of FIG. 6) based on a sensing value of a position sensor (e.g., the position sensor (612) of FIG. 6). For example, the electronic device may control the intensity of a current applied to one or more coils based on a sensing value based on a position of a lens assembly or a position of an image sensor.

According to one embodiment, in operation 1075, the electronic device may perform image stabilization processing to reduce shaking of the image by performing image processing on the image frames based on at least one of a target value, a sensing value, or motion information. For example, the electronic device may perform image stabilization processing to reduce shaking of the image by performing image processing on the image frames based on at least one of a target value of 0, a sensing value based on a position of the lens assembly or a position of the image sensor, or motion information.

According to one embodiment, by fixing the target value to a specified value (e.g., a value of 0), the movement of the driving unit can be reduced or prevented from acting as noise due to the frequency of the movement information having a frequency component higher than the specified frequency value.

However, the operation of preventing the movement of the driving unit from acting as noise due to the frequency of the movement information having a frequency component higher than the specified frequency value is not limited thereto. For example, the electronic device may not fix the target value to the specified value and may not use the target value. For example, the electronic device may perform image stabilization processing to reduce shaking of the image by performing image processing on image frames based on the movement information acquired from the sensor without using the target value.

FIG. 17 is a diagram for explaining the effect of improving the performance of image stabilization processing by fixing a target value to a specified value according to one embodiment.

The contents of Fig. 17 may be referenced by the contents of other drawings. For example, the target values of Fig. 17 may be referenced by the target values of Fig. 16.

The first graph (1710) of FIG. 17 may be a graph representing a first angular velocity motion vector (1711) and a first image motion vector (1712) when the target value is not fixed to a specified value (e.g., a value of 0). The second graph (1720) of FIG. 17 may be a graph representing a second angular velocity motion vector (1711) and a second image motion vector (1712) when the target value is fixed to a specified value (e.g., a value of 0). In one example, the angular velocity motion vector may include a motion vector based on motion information. The image motion vector may include an amount of pixels moved compared to a previous image. The electronic device may compare the image motion vector and the angular velocity motion vector. For example, the electronic device may determine whether the angular velocity motion vector and the image motion vector are similar and/or the degree of similarity. The electronic device can determine whether the video digital image stabilization (VDIS) (and/or electronic image stabilization (EIS)) has been performed stably based on the comparison result. For example, the electronic device can determine that the video digital image stabilization (VDIS) (and/or electronic image stabilization (EIS)) has been performed stably based on the comparison result that the angular velocity motion vector and the image motion vector are determined to be similar.

According to one embodiment, when the target value is fixed to the specified value (e.g., 0), the angular velocity motion vector and the image motion vector may be more similar than when the target value is not fixed to the specified value (e.g., 0). For example, when referring to the first graph (1710) and the second graph (1720), when comparing the degree of similarity between the first angular velocity motion vector (1711) and the first image motion vector (1712), and the degree of similarity between the second angular velocity motion vector (1721) and the second image motion vector (1722), the degree of similarity between the second angular velocity motion vector (1721) and the second image motion vector (1722) may be greater. When the target value is fixed to a specified value (e.g., 0), video digital image stabilization (VDIS) (and/or electronic image stabilization (EIS)) can be performed more stably compared to when the target value is not fixed to a specified value (e.g., 0).

FIG. 18 is a flowchart illustrating a process (1800) for performing an operation mode based on user input according to one embodiment.

The contents of Fig. 18 may be referenced by the contents of other drawings.

According to one embodiment, a process (1800) for performing an operation mode based on a user input may include an operation (1810) for obtaining a user input and an operation (1820) for performing an operation mode based on the user input. However, the operations of the process (1800) are not limited thereto. For example, the process (1800) may omit at least one of the operations described above, or may further include at least one other operation.

In one embodiment, at operation 1810, the electronic device may obtain a user input. For example, the electronic device may obtain a user input to change an operation mode of a camera module (e.g., a current operation mode). In one example, the electronic device may obtain the user input through a display (e.g., the display module (160) of FIG. 1). For example, the electronic device may obtain the user input when a user touches a visual object displayed on the display (e.g., a button and/or icon related to changing the operation mode). However, the present invention is not limited thereto. According to various embodiments, the user input may be obtained through various methods. For example, the electronic device may obtain the user input when a user touches a physical button.

In one embodiment, the user input may include a user input to change an operating mode of the camera module (e.g., a current operating mode). For example, the user input may include a user input to change to a first operating mode. For example, the user input may include a user input to change to a second operating mode. For example, the user input may include a user input to change to a third operating mode.

According to an embodiment, in operation 1820, the electronic device may perform an operation mode based on a user input. For example, the electronic device may perform one of a plurality of operation modes based on the user input. For example, the electronic device may perform an operation mode corresponding to the user input. According to an embodiment, the electronic device may change to one of the plurality of operation modes based on the user input. For example, the electronic device may change the current operation mode to another operation mode among the plurality of operation modes based on the user input. For example, the electronic device may change to the first operation mode based on a user input to change to the first operation mode. For example, the electronic device may change the current operation mode from the second operation mode or the third operation mode to the first operation mode. The electronic device may perform the first operation mode in response to the change to the first operation mode. For example, the electronic device may change to the second operation mode based on a user input to change to the second operation mode. For example, the electronic device may change the current operation mode from the first operation mode or the third operation mode to the second operation mode. The electronic device may perform the second operating mode in response to changing to the second operating mode. For example, the electronic device may change to the third operating mode based on a user input causing the electronic device to change to the third operating mode. For example, the electronic device may change the current operating mode from the first operating mode or the second operating mode to the third operating mode. The electronic device may perform the third operating mode in response to changing to the third operating mode.

FIG. 19 is a flowchart illustrating a process (1900) for performing an operation mode based on detection of an external device according to one embodiment.

The contents of Fig. 19 may be referenced by the contents of other drawings.

In one embodiment, at operation 1910, the electronic device may detect an external device. For example, the electronic device may detect that the electronic device is connected to the external device. For example, the electronic device may detect that the electronic device is physically connected to the external device. For example, the electronic device may detect that the electronic device is in physical contact with the external device. For example, the electronic device may detect that the electronic device is placed in the external device. For example, the electronic device may detect that the electronic device is accommodated in the external device. For example, the electronic device may detect that the electronic device is seated in the external device. In one example, the external device may include the electronic device and/or an accessory. For example, the external device may include a drone, a motorcycle, a bicycle, a cart, and/or a mount. For example, the electronic device may detect that the electronic device is mounted on the drone. For example, the electronic device may detect that the electronic device is placed on the mount. For example, the electronic device may detect that the electronic device is placed on the mount of the motorcycle.

In one embodiment, in response to the electronic device not detecting the external device in operation 1910, in operation 1920, the electronic device can perform a first operational mode. For example, the electronic device can maintain the first operational mode.

In one embodiment, in response to the electronic device detecting the external device in operation 1910, in operation 1930, the electronic device may perform a designated operation mode from among a plurality of operation modes. For example, the electronic device may perform the designated operation mode based on the external device. For example, the electronic device may perform the designated operation mode based on the type of the external device.

In one embodiment, the electronic device may, in response to detecting an external device, control the driving unit to position the lens assembly or the image sensor at a reference position within the driving range based on a specified operating mode. For example, the electronic device may control the driving unit based on a second operating mode. For example, the electronic device may control the driving unit based on a third operating mode. For example, the electronic device may perform the second operating mode based on the electronic device being placed on a motorcycle (e.g., on a mount of the motorcycle). For example, the electronic device may change to the second operating mode or maintain the second operating mode based on the electronic device being placed on the motorcycle (e.g., on a mount of the motorcycle). For example, the electronic device may perform the third operating mode based on the electronic device being mounted on a drone. For example, the electronic device may change to the third operating mode or maintain the third operating mode based on the electronic device being mounted on a drone.

As described above, an electronic device according to an embodiment (e.g., the electronic device (101) of FIG. 1) may include a sensor for detecting a movement of the electronic device, a camera module, one or more processors, and a memory for storing one or more instructions. The camera module may include a lens assembly, an image sensor, and a driving unit for moving the lens assembly or the image sensor within a driving range. The one or more instructions may cause the electronic device, when individually or collectively executed by the one or more processors, to: control the driving unit to move a position of the lens assembly or the image sensor according to movement information acquired from the sensor, based on a first operation mode among a plurality of operation modes associated with an operation of the driving unit. The one or more instructions may cause the electronic device, when individually or collectively executed by the one or more processors, to: control the driving unit to position the lens assembly or the image sensor at a reference position within the driving range, based on a second operation mode different from the first operation mode among the plurality of operation modes. The one or more instructions may cause the one or more processors, when individually or collectively executed, to cause the electronic device to: perform image stabilization processing to reduce image shake by performing image processing on image frames acquired through the camera module while controlling the driving unit. The one or more instructions may cause the electronic device, when individually or collectively executed by the one or more processors, to: control the position of the lens assembly or the image sensor based on a first gain value when the driving unit operates based on the first operation mode, and control the position of the lens assembly or the image sensor based on a second gain value different from the first gain value when the driving unit operates based on the second operation mode.

In one embodiment, the one or more instructions may cause the one or more processors, when individually or collectively executed, to determine whether the electronic device will change to one of the plurality of operation modes based on a frequency of the motion information. The one or more instructions may cause the one or more processors, when individually or collectively executed, to cause the electronic device, based on a third operation mode different from the first operation mode and the second operation mode among the plurality of operation modes, to control the driving unit such that the lens assembly or the image sensor is positioned at a reference position within the driving range. The one or more instructions may cause the one or more processors, when individually or collectively executed, to control the position of the lens assembly or the image sensor based on a third gain value different from the first gain value and the second gain value when the electronic device operates based on the third operation mode.

In one embodiment, the one or more instructions, when executed by the one or more processors individually or collectively, may cause the electronic device to change to at least one of the second operating mode or the third operating mode based on detection of a frequency component greater than or equal to a specified frequency value from the motion information.

In one embodiment, the one or more instructions, when executed individually or collectively by the one or more processors, may cause the electronic device to detect a frequency component greater than or equal to the specified frequency value based on at least one of a periodicity of the motion information or a derivative of the motion information.

In one embodiment, the third gain value may be greater than the second gain value.

In one embodiment, the one or more instructions, when executed by the one or more processors individually or collectively, may cause the electronic device to obtain a user input that causes the electronic device to change into one of the plurality of operating modes. The one or more instructions, when executed by the one or more processors individually or collectively, may cause the electronic device to change into one of the plurality of operating modes based on the user input.

In one embodiment, the one or more instructions may cause the one or more processors, when individually or collectively executed, to determine whether the electronic device has detected an external device. The one or more instructions may cause the one or more processors, when individually or collectively executed, to control the driving unit such that the lens assembly or the image sensor is positioned at a reference position within the driving range, based on a designated operating mode among the plurality of operating modes, in response to the electronic device detecting the external device.

According to one embodiment, the electronic device may include a housing that accommodates the lens assembly and the driving unit. The driving unit may be arranged to be movable in a first axial direction substantially perpendicular to an optical axis of the lens assembly and a second axial direction substantially perpendicular to the first axis within the housing. The reference position may include at least one of a center of the driving range along the first axis or a center of the driving range along the second axis.

According to one embodiment, the driving unit may include at least one of an OIS driving unit that moves the lens assembly or the image sensor in at least one of a first direction or a second direction different from the first direction, or an AF driving unit that moves the lens assembly or the image sensor in a third direction substantially perpendicular to the first direction and the second direction according to focus-related information for an image captured through the image sensor.

According to one embodiment, the one or more instructions may cause the one or more processors, when individually or collectively executed, to cause the electronic device to operate based on the second operating mode: control the OIS driving unit based on the second gain value such that the position of the lens assembly or the position of the image sensor is maintained at the reference position with respect to the first direction and the second direction, and control the AF driving unit based on the second gain value such that the position of the lens assembly or the position of the image sensor is maintained at the reference position with respect to the third direction.

According to one embodiment, the camera module may include a position sensor for obtaining position information of the lens assembly or position information of the image sensor. The driving unit may include one or more coils for controlling the position of the lens assembly or the position of the image sensor. The first gain value and the second gain value may include gain values for the driving unit to control current flowing in the one or more coils based on the position information.

According to one embodiment, the camera module may include a position sensor for obtaining position information of the lens assembly or position information of the image sensor. The driving unit may include one or more coils for controlling the position of the lens assembly or the position of the image sensor. The one or more commands, when individually or collectively executed by the one or more processors, may cause the electronic device to fix a target value output from the position sensor to a designated value. The one or more commands, when individually or collectively executed by the one or more processors, may cause the electronic device to control the intensity of a current applied to the one or more coils based on a sensing value obtained from the position sensor.

According to one embodiment, the sensor for detecting movement of the electronic device may include at least one of a gyro sensor or an acceleration sensor.

In one embodiment, the second gain value may be greater than the first gain value.

As described above, the operating method of an electronic device including a camera module according to an embodiment (for example, the electronic device (101) of FIG. 1) may include an operation of controlling the driving unit to move a position of the lens assembly or the image sensor based on movement information acquired from a sensor detecting movement of the electronic device, based on a first operation mode among a plurality of operation modes associated with an operation of the driving unit that moves the lens assembly or the image sensor within a driving range. The operating method of the electronic device may include an operation of controlling the driving unit based on a second operation mode different from the first operation mode among the plurality of operation modes, so that the lens assembly or the image sensor is positioned at a reference position within the driving range. The operating method of the electronic device may include an operation of performing image processing on image frames acquired through the camera module while controlling the driving unit to perform image stabilization processing to reduce shaking of the image. The operating method of the electronic device may include an operation in which, when operating based on the first operation mode, the driving unit controls the position of the lens assembly or the image sensor based on a first gain value, and when operating based on the second operation mode, the driving unit controls the position of the lens assembly or the image sensor based on a second gain value different from the first gain value.

According to one embodiment, the operating method of the electronic device may include an operation of determining whether to change to one of the plurality of operating modes based on a frequency of the movement information. The operating method of the electronic device may include an operation of controlling the driving unit so that the lens assembly or the image sensor is positioned at a reference position within the driving range based on a third operating mode different from the first operating mode and the second operating mode among the plurality of operating modes. The operating method of the electronic device may include an operation of controlling the position of the lens assembly or the image sensor based on a third gain value different from the first gain value and the second gain value when the electronic device operates based on the third operating mode.

According to one embodiment, the operating method of the electronic device may include an operation of changing to at least one of the second operating mode or the third operating mode based on detection of a frequency component equal to or greater than a specified frequency value from the motion information or detection of a frequency component equal to or greater than a predetermined value. The operating method of the electronic device may include an operation of detecting a frequency component equal to or greater than the specified frequency value based on at least one of a periodicity of the motion information or a differential value of the motion information.

According to one embodiment, the driving unit may include at least one of an OIS driving unit that moves the lens assembly or the image sensor in at least one of a first direction or a second direction different from the first direction, or an AF driving unit that moves the lens assembly or the image sensor in a third direction substantially perpendicular to the first direction and the second direction according to focus-related information for an image captured through the image sensor.

According to one embodiment, when operating based on the second operation mode, the operation of controlling the position of the lens assembly or the image sensor based on the second gain value different from the first gain value by the driving unit may include an operation of controlling the OIS driving unit based on the second gain value so that the position of the lens assembly or the position of the image sensor is maintained at the reference position with respect to the first direction and the second direction, and an operation of controlling the AF driving unit based on the second gain value so that the position of the lens assembly or the position of the image sensor is maintained at the reference position with respect to the third direction.

According to one embodiment, the operating method of the electronic device may include an operation of fixing a target value output from the position sensor to a specified value. The operating method of the electronic device may include an operation of controlling an intensity of a current applied to one or more coils for controlling a position of the lens assembly or a position of the image sensor based on a sensing value obtained from the position sensor.

According to one embodiment, a computer-readable, non-transitory recording medium may cause an electronic device including a camera module to perform a method including an operation of controlling the driving unit to move a position of the lens assembly or the image sensor based on movement information acquired from a sensor detecting movement of the electronic device, based on a first operation mode among a plurality of operation modes associated with an operation of a driving unit that moves a lens assembly or an image sensor within a driving range when the electronic device is executed. The computer-readable, non-transitory recording medium may cause an electronic device including a camera module to perform a method including an operation of controlling the driving unit to position the lens assembly or the image sensor at a reference position within the driving range, based on a second operation mode different from the first operation mode among the plurality of operation modes when the electronic device including the camera module is executed. The computer-readable, non-transitory recording medium may cause an electronic device including a camera module to perform a method including an operation of performing image processing on an image frame acquired through the camera module while controlling the driving unit when the electronic device including the camera module is executed to perform image stabilization processing to reduce shaking of the image. The above computer-readable non-transitory recording medium can cause an electronic device including a camera module to perform a method including an operation in which, when the electronic device operates based on the first operation mode, the driving unit controls the position of the lens assembly or the image sensor based on a first gain value, and when the electronic device operates based on the second operation mode, the driving unit controls the position of the lens assembly or the image sensor based on a second gain value different from the first gain value.

According to one embodiment, the recording medium may cause the electronic device, when executed, to execute an operation of determining whether to change to one of the plurality of operation modes based on a frequency of the movement information. The recording medium may cause the electronic device, when executed, to execute an operation of controlling the driving unit so that the lens assembly or the image sensor is positioned at a reference position within the driving range based on a third operation mode different from the first operation mode and the second operation mode among the plurality of operation modes. The recording medium may cause the driving unit to execute an operation of controlling the position of the lens assembly or the image sensor based on a third gain value different from the first gain value and the second gain value when the electronic device operates based on the third operation mode when executed.

According to one embodiment, the recording medium may cause the electronic device to perform an operation of detecting a frequency component greater than or equal to the specified frequency value based on at least one of a periodicity of the motion information or a differential value of the motion information when the electronic device is executed.

According to one embodiment, the recording medium may cause the electronic device, when executed, to execute an operation of fixing a target value output from a position sensor that acquires position information of the lens assembly or position information of the image sensor to a designated value. The recording medium may cause the electronic device, when executed, to execute an operation of controlling the intensity of a current applied to one or more coils for controlling the position of the lens assembly or the position of the image sensor based on the sensing value acquired from the position sensor.

The methods according to the embodiments described in the claims or specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

In the case of software implementation, a computer-readable storage medium storing one or more programs (software modules) may be provided. The one or more programs stored in the computer-readable storage medium are configured for execution by one or more processors in an electronic device. The one or more programs include instructions that cause the electronic device to execute methods according to embodiments described in the claims or specification of the present disclosure.

In the present disclosure, a function or operation performed by an electronic device may be performed by one or more processors executing one or more instructions stored in a memory. The function or operation of the electronic device mentioned in the present disclosure may be performed by one processor executing one or more instructions, or may be performed by a combination of a plurality of processors executing one or more instructions. The processor mentioned in the present disclosure may be understood to include a circuit for performing a calculation or controlling other components of the electronic device. For example, the one or more processors may include a central processing unit (CPU), a microprocessor unit (MPU), an application processor (AP), a communication processor (CP), a neural processing unit (NPU), a system on chip (SoC), or an integrated circuit (IC) configured to execute one or more instructions. The one or more processors may be configured to perform the operation of the electronic device described above.

In the present disclosure, a program (software module, software) may be stored in a non-volatile memory including a random access memory, a flash memory, a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a magnetic disc storage device, a compact disc-ROM (CD-ROM), digital versatile discs (DVDs) or other forms of optical storage devices, a magnetic cassette. Or, it may be stored in a memory formed by a combination of some or all of these. The memory may be formed by a single storage medium, or may be formed by a combination of a plurality of storage media. The one or more commands may be stored in a single storage medium, or may be distributed and stored in a plurality of storage media.

Additionally, the program may be stored in an attachable storage device that is accessible via a communication network, such as the Internet, an Intranet, a local area network (LAN), a wide LAN (WLAN), or a storage area network (SAN), or a combination thereof. The storage device may be connected to a device performing an embodiment of the present disclosure via an external port. Additionally, a separate storage device on the communication network may be connected to the device performing an embodiment of the present disclosure.

In the specific embodiments of the present disclosure described above, the components included in the disclosure are expressed in the singular or plural form according to the specific embodiments presented. However, the singular or plural expressions are selected to suit the presented situation for the convenience of explanation, and the present disclosure is not limited to the singular or plural components, and even if a component is expressed in the plural form, it may be composed of the singular form, or even if a component is expressed in the singular form, it may be composed of the plural form.

Additionally, in the present disclosure, terms such as “part”, “module”, etc. may refer to a hardware component such as a processor or a circuit, and/or a software component executed by a hardware component such as a processor.

A “component”, a “module” may be implemented by a program stored in an addressable storage medium and executed by a processor. For example, a “component”, a “module” may be implemented by components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.

The specific implementations described in this disclosure are only examples and are not intended to limit the scope of the disclosure in any way. For the sake of brevity, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted.

Additionally, in the present disclosure, “comprising at least one of a, b, or c” may mean “comprising only a, or comprising only b, or comprising only c, or comprising a combination of two or more of them (including a and b, or comprising b and c, or comprising a and c, or comprising all of a, b, and c).

Meanwhile, although the detailed description of the present disclosure has described specific embodiments, it is obvious that various modifications are possible without departing from the scope of the present disclosure. Therefore, the scope of the present disclosure should not be limited to the described embodiments, but should be determined not only by the scope of the claims described below, but also by equivalents of the scope of the claims.

Claims (15)

In electronic devices, A sensor for detecting movement of the electronic device; A camera module comprising a lens assembly, an image sensor, and a driving unit for moving the lens assembly or the image sensor within a driving range; one or more processors; and Contains a memory that stores one or more instructions, The one or more instructions, when individually or collectively executed by the one or more processors, cause the electronic device to: Controlling the driving unit to move the position of the lens assembly or the image sensor based on the movement information acquired from the sensor based on a first operation mode among a plurality of operation modes associated with the operation of the driving unit, Based on a second operation mode different from the first operation mode among the plurality of operation modes, the driving unit is controlled so that the lens assembly or the image sensor is positioned at a reference position within the driving range, While controlling the above driving unit, image stabilization processing is performed to reduce image shaking by performing image processing on image frames acquired through the camera module. When operating based on the first operation mode, the driving unit controls the position of the lens assembly or the image sensor based on the first gain value, and when operating based on the second operation mode, the driving unit controls the position of the lens assembly or the image sensor based on a second gain value different from the first gain value. Electronic devices. In claim 1, The one or more instructions, when executed individually or collectively by the one or more processors, cause the electronic device to: Based on the frequency of the above movement information, it is determined whether to change to one of the above multiple operation modes, Based on a third operation mode different from the first operation mode and the second operation mode among the plurality of operation modes, the driving unit is controlled so that the lens assembly or the image sensor is positioned at a reference position within the driving range, When operating based on the third operation mode, the driving unit controls the position of the lens assembly or the image sensor based on a third gain value different from the first gain value and the second gain value. Electronic devices. In claim 2, The one or more instructions, when executed individually or collectively by the one or more processors, cause the electronic device to: To change to at least one of the second operation mode or the third operation mode based on the detection of a frequency component higher than a specified frequency value from the movement information or the detection of a frequency component higher than a certain value. Electronic devices. In claim 3, The one or more instructions, when executed individually or collectively by the one or more processors, cause the electronic device to: To detect a frequency component higher than the specified frequency value based on at least one of the periodicity of the motion information or the differential value of the motion information. Electronic devices. In claim 2, The third gain value is greater than the second gain value, Electronic devices. In claim 1, The one or more instructions, when executed individually or collectively by the one or more processors, cause the electronic device to: Obtaining a user input to change one of the above multiple operation modes, Based on the above user input, changing to one of the above multiple operation modes, Electronic devices. In claim 1, The one or more instructions, when executed individually or collectively by the one or more processors, cause the electronic device to: Determine whether an external device has been detected, In response to detecting the external device, the driving unit is controlled to position the lens assembly or the image sensor at a reference position within the driving range based on a designated operating mode among the plurality of operating modes. Electronic devices. In claim 1, Further comprising a housing accommodating the lens assembly and the driving unit, The driving unit is arranged to be movable in a first axis direction substantially perpendicular to the optical axis of the lens assembly and a second axis direction substantially perpendicular to the first axis within the housing, An electronic device, wherein the reference position includes at least one of the center of the driving range on the first axis or the center of the driving range on the second axis. In claim 1, An electronic device, wherein the driving unit includes at least one of an OIS driving unit that moves the lens assembly or the image sensor in at least one of a first direction or a second direction different from the first direction, or an AF driving unit that moves the lens assembly or the image sensor in a third direction substantially perpendicular to the first direction and the second direction according to focus-related information for an image captured through the image sensor. In claim 9, The one or more instructions, when executed individually or collectively by the one or more processors, cause the electronic device to: When operating based on the above second operation mode: Controlling the OIS driving unit based on the second gain value so that the position of the lens assembly or the position of the image sensor is maintained at the reference position with respect to the first direction and the second direction, Controlling the AF driving unit based on the second gain value so that the position of the lens assembly or the position of the image sensor is maintained at the reference position with respect to the third direction. Electronic devices. In claim 1, The above camera module further includes a position sensor that obtains position information of the lens assembly or position information of the image sensor, The above driving unit includes one or more coils, The one or more instructions, when executed individually or collectively by the one or more processors, cause the electronic device to: The target value output from the above position sensor is fixed to a specified value, Controlling the intensity of the current applied to one or more coils based on the sensing value obtained from the position sensor. Electronic devices. In claim 1, The above sensor comprises at least one of a gyro sensor or an acceleration sensor. Electronic devices. In claim 1, The second gain value is greater than the first gain value, Electronic devices. A method of operating an electronic device including a camera module, An operation of controlling the driving unit to move the position of the lens assembly or the image sensor based on movement information acquired from a sensor detecting movement of the electronic device, based on a first operation mode among a plurality of operation modes associated with the operation of the driving unit that moves the lens assembly or the image sensor within the driving range; An operation of controlling the driving unit so that the lens assembly or the image sensor is positioned at a reference position within the driving range based on a second operation mode different from the first operation mode among the plurality of operation modes; An operation of performing image stabilization processing to reduce image shaking by performing image processing on image frames acquired through the camera module while controlling the driving unit; and A method comprising: an operation in which, when operating based on the first operation mode, the driving unit controls the position of the lens assembly or the image sensor based on a first gain value, and when operating based on the second operation mode, the driving unit controls the position of the lens assembly or the image sensor based on a second gain value different from the first gain value; In a computer-readable non-transitory recording medium, an electronic device including a camera module, when executed: An operation of controlling the driving unit to move the position of the lens assembly or the image sensor based on movement information acquired from a sensor detecting movement of the electronic device, based on a first operation mode among a plurality of operation modes associated with the operation of the driving unit that moves the lens assembly or the image sensor within the driving range; An operation of controlling the driving unit so that the lens assembly or the image sensor is positioned at a reference position within the driving range based on a second operation mode different from the first operation mode among the plurality of operation modes; An operation of performing image stabilization processing to reduce image shaking by performing image processing on image frames acquired through the camera module while controlling the driving unit; and A computer program for causing a method to be performed, comprising: an operation in which, when operating based on the first operation mode, the driving unit controls the position of the lens assembly or the image sensor based on a first gain value; and when operating based on the second operation mode, the driving unit controls the position of the lens assembly or the image sensor based on a second gain value different from the first gain value; A recording medium having recorded thereon a computer program for causing the method to be performed.

Images