TWI877843B - Integration system of artificial intelligence iot gateway and near-end audio and video and its integration method - Google Patents

Abstract

The invention provides an integrated system of artificial intelligence Internet of Things gateway and near-end audio and video, including: a wireless network monitoring device, a plurality of sensors and an artificial intelligence Internet of Things gateway device. The wireless network monitoring device covers and monitors a plurality of near-end audio and video in a status area, and the wireless network monitoring device further includes a plurality of audio and video capture units arranged in the status area, and the audio and video capture unit can capture Obtain audio and video information of the near-end video. The sensor is adjacent to the audio and video capture unit and can sense an environmental information. The artificial intelligence IoT gateway device is connected to the wireless network monitoring device and the sensor via telecommunications, and the artificial intelligence IoT gateway device simultaneously receives the audio-visual information and the environmental information, and when the comparison determines that it meets a crisis mode, An input/output device will be executed.

Description

Artificial intelligence Internet of Things gateway and near-end audio and video integration system and integration method

The present invention relates to an artificial intelligence Internet of Things gateway, and more particularly to an integrated system and method of an artificial intelligence Internet of Things gateway and near-end audio and video that can execute imaging and sensing in an edge computing environment.

Industry 4.0 means the "fourth industrial revolution" and can also be called smart manufacturing. Industry 1.0 used steam as a power source and machines replaced labor; Industry 2.0 entered the era of electrified production; Industry 3.0 used computers to assist human production and entered the information age; Industry 4.0 focuses on "smart manufacturing" of smart production.

The purpose of Industry 4.0/smart manufacturing is to enable factories to use advanced manufacturing models to closely integrate all aspects of the "smart machinery" production process with applications such as the Internet of Things, cloud computing, big data, and AI artificial intelligence, so that machinery and equipment, and between machinery and equipment and people can communicate with each other, and assist the manufacturing industry in digital transformation through transparent management.

In addition, many digital twin IoT applications of Industry 4.0 include field-based sensing and control. However, non-imaging sensors alone are not enough. For example, traditional programmable logic controllers (PLCs) can also integrate sensing and control at the local end, or can be used with a centralized system (SCADA) that monitors and controls all devices for integrated control. However, there is no integrated interpretation mechanism combined with images. Therefore, there are still many problems that need to be solved.

The purpose of the present invention is to provide an integrated system and method for integrating artificial intelligence IoT gateway and near-end audio and video. By combining an artificial intelligence IoT gateway device with edge computing and image protocol, the artificial intelligence IoT gateway device can process near-end image recognition, and use edge computing image recognition to perform image interpretation, perform real-time field control, and report the overall remote situation, thereby achieving the effect of solving real-time and integration problems.

The purpose of the present invention is to provide an integrated system of an artificial intelligence IoT gateway and near-end audio and video and an integration method thereof, by which the artificial intelligence IoT gateway device can obtain near-end images for interpretation, integrate information for control, and interact with the remote end.

To achieve the above-mentioned purpose, the present invention provides an integrated system of artificial intelligence Internet of Things gateway and near-end audio and video, comprising: a wireless network monitoring device, a plurality of sensors and an artificial intelligence Internet of Things gateway device. The wireless network monitoring device covers and monitors a plurality of near-end audio and video in a situation area. The wireless network monitoring device further includes a plurality of audio and video capture units arranged in the situation area, and the audio and video capture unit can capture audio and video information of the near-end audio and video. The sensor is adjacent to the audio and video capture unit and can sense environmental information. The artificial intelligence Internet of Things gateway device is telecommunication-connected to the wireless network monitoring device and the sensor. The artificial intelligence Internet of Things gateway device simultaneously receives the video and audio information and the environmental information, and when comparing and judging that they meet a crisis mode, an input/output device is executed.

In a preferred embodiment of the present invention, the artificial intelligence IoT gateway device receives the video information and the environmental information using a video access protocol. The video access protocol is a real-time streaming protocol (Real Time Streaming Protocol, RTSP) or an open network video interface forum (ONVIF).

In a preferred embodiment of the present invention, the integrated system further includes an instant messaging host and a terminal device, the instant messaging host is located on a network cloud, and is connected to the artificial intelligence IoT gateway device by a persistent connection, the terminal device is directly or indirectly connected to the instant messaging host, and the artificial intelligence IoT gateway device can receive a video data transmission request of the terminal device through the instant messaging host. The persistent connection is a message queuing telemetry transport (MQTT). The integrated system further includes an end-to-end connection control host, the artificial intelligence IoT gateway device is connected to the end-to-end connection control host by a network interface, and the artificial intelligence IoT gateway device uses the network interface to establish an end-to-end connection with the terminal device through the end-to-end connection control host. The artificial intelligence IoT gateway device can use a streaming protocol to use the network interface. The streaming protocol is a real-time streaming protocol (Real Time Streaming Protocol, RTSP). The network interface is an Ethernet, a wireless network (Wifi) or a long-term evolution technology (Long Term Evolution, LTE).

In a preferred embodiment of the present invention, the artificial intelligence IoT gateway device further includes an edge computing unit, which uses an edge computing to interpret the audio and video information.

In a preferred embodiment of the present invention, the sensor is at least one of a temperature sensor and a smoke sensor.

To achieve the above-mentioned purpose, the present invention provides a method for integrating an artificial intelligence IoT gateway with near-end audio and video, comprising the following steps:

Step (a): providing a wireless network monitoring device and an artificial intelligence IoT gateway device, wherein the wireless network monitoring device further comprises a plurality of video and audio capture units arranged in a situation area, the artificial intelligence IoT gateway device is telecommunication-connected to the wireless network monitoring device, and the artificial intelligence IoT gateway device further comprises an edge computing unit.

Step (b): Capturing video information with the video capture unit.

Step (c): The edge operation unit uses an edge operation to read the video information.

Step (d): Whether the video information is in a crisis mode.

Step (e): If yes, execute an input/output device; if no, repeat step (b).

In a preferred embodiment of the present invention, the step (e) further includes the following steps:

Step (f): providing a plurality of sensors, wherein the sensors are adjacent to the video capture unit and are electronically connected to the wireless network monitoring device.

Step (g): The sensor senses environmental information.

Step (h): Whether the environmental information is in the crisis mode.

Step (i): If yes, execute the input/output device; if no, repeat step (g).

The following disclosure provides different embodiments or examples to implement different features of the subject matter provided. The specific examples of components and arrangements described below are for the purpose of simplifying the disclosure and are not intended to be limiting; the size and shape of the components are not limited by the disclosed range or values, but may depend on the process conditions or required characteristics of the components. For example, cross-sectional views are used to describe the technical features of the present invention, and these cross-sectional views are idealized schematic views of embodiments. Therefore, it is foreseeable that the shapes of the diagrams will be different due to manufacturing processes and/or tolerances, and should not be limited thereto.

Furthermore, spatially relative terms, such as “below,” “under,” “lower than,” “above,” and “higher than,” are intended to easily describe the relationship between elements or features depicted in the diagrams; in addition, spatially relative terms include different directions of the elements when they are in use or operation in addition to the directions depicted in the diagrams.

Please refer to FIG. 1, which is a block diagram of a preferred embodiment of the integrated system of artificial intelligence IoT gateway and near-end audio and video of the present invention. The integrated system of artificial intelligence IoT gateway and near-end audio and video of the present invention comprises: a wireless network monitoring device 1, a plurality of sensors 2, and an artificial intelligence IoT gateway device 3. The wireless network monitoring device 1 covers and monitors a plurality of near-end audio and video 81 in a situation area 8. The wireless network monitoring device 1 further comprises a plurality of audio and video capture units 11 arranged in the situation area 8. The audio and video capture unit 11 can capture audio and video information of the near-end audio and video 81. The sensor 2 is adjacent to the video capture unit 11. The sensor 2 is at least one of a temperature sensor and a smoke sensor, and can sense environmental information, which can be the state of temperature and smoke concentration.

The artificial intelligence IoT gateway device 3 is connected to the wireless network monitoring device 1 and the sensor 2. The artificial intelligence IoT gateway device 3 receives the video and audio information and the environmental information at the same time. When it is compared and judged to meet a crisis mode, an input/output device 4 is executed. The input/output device 4 can be a sprayer and located in the condition area 8, or it can be a drone and located outside the condition area 8. In a preferred embodiment of the present invention, the artificial intelligence IoT gateway device 3 receives the video and audio information and the environmental information using a video and audio access protocol 32. The video access protocol 32 is a real-time streaming protocol (RTSP) or an open network video interface forum (ONVIF). The artificial intelligence IoT gateway device 3 further includes an edge computing unit 31, and the edge computing unit 31 uses an edge computing to read the video information.

The integrated system of the present invention further includes an instant messaging host 5 and a terminal device 6. The instant messaging host 5 is located on a network cloud 50 and is connected to the artificial intelligence IoT gateway device 3 via a fixed connection 51 (persistant connection). The fixed connection 51 is a message queuing telemetry transport (MQTT). The terminal device 6 is directly or indirectly connected to the instant messaging host 5. The artificial intelligence IoT gateway device 3 can receive an audio and video data transmission request 61 from the terminal device 6 through the instant messaging host 5. In addition, the data transmission and control requirements of the audio and video data transmission request 61 also include Bluetooth, Zigbee protocols and network connections.

The integrated system of the present invention further includes an end-to-end connection control host 7, the artificial intelligence IoT gateway device 3 is connected to the end-to-end connection control host 7 by a network interface 33, and the artificial intelligence IoT gateway device 3 uses the network interface 33 to establish an end-to-end connection with the terminal device 6 through the end-to-end connection control host 7. The artificial intelligence IoT gateway device 3 can use a streaming protocol 34 to use the network interface 33. The streaming protocol 34 is a real-time streaming protocol (Real Time Streaming Protocol, RTSP). The network interface is an Ethernet, a wireless network (Wifi) or a long-term evolution technology (Long Term Evolution, LTE), and can be connected with a connection with minimal delay.

The data transmission and control requirements of the terminal device 6 also include the local I/O data of the artificial intelligence IoT gateway device 3. The artificial intelligence IoT gateway device 3 uses its I/O communication interface and the local I/O control to obtain information. In other words, the artificial intelligence IoT gateway device 3 can be detected and controlled through the local I/O or the sensor of the sensing network combined with image recognition, and the local I/O or the sensing of the sensing network can also be independently controlled from the remote terminal device.

Please refer to FIG. 2, which is a flow chart of a preferred embodiment of the present invention, which is a flow chart of a preferred embodiment of the method for integrating an artificial intelligence IoT gateway with near-end audio and video. The present invention provides a method for integrating an artificial intelligence IoT gateway with near-end audio and video, which includes the following steps:

Step 91: Provide a wireless network monitoring device and an artificial intelligence Internet of Things gateway device, wherein the wireless network monitoring device further includes a plurality of video and audio capture units arranged in a situation area, the artificial intelligence Internet of Things gateway device is telecommunication-connected to the wireless network monitoring device, and the artificial intelligence Internet of Things gateway device further includes an edge computing unit.

Step 92: Capture video information with the video capture unit.

Step 93: The edge operation unit uses an edge operation to read the video information.

Step 94: whether the video and audio information is a crisis mode; if so, execute step 95: execute an input and output device; if not, repeat the step 92.

In a preferred embodiment of the present invention, step 95 further includes the following steps:

Step 96: Provide a plurality of sensors, wherein the sensors are adjacent to the video capture unit and are electronically connected to the wireless network monitoring device.

Step 97: The sensor senses environmental information.

Step 98: Whether the environmental information is the crisis mode; if so, execute step 95: the input/output device; if not, repeat the step 97.

The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention is described in detail with reference to the preferred embodiments, ordinary technicians in this field should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

1: Wireless network monitoring device 11: Video capture unit 2: Sensor 3: Artificial intelligence IoT gateway device 31: Edge computing unit 32: Video access protocol 33: Network interface 34: Streaming protocol 4: Input and output device 5: Instant messaging host 50: Network cloud 51: Fixed connection 6: Terminal device 61: Video data transmission requirements 7: End-to-end connection control host 8: Status area 81: Near-end video 91~98: Steps

In order to make the above and other purposes, features, advantages and embodiments of the present invention more understandable, the attached diagrams are described as follows: Figure 1 is a system block diagram of a preferred embodiment of the integration system of the artificial intelligence Internet of Things gateway and near-end audio and video of the present invention. Figure 2 is a flow block diagram of a preferred embodiment of the integration method of the artificial intelligence Internet of Things gateway and near-end audio and video of the present invention.

According to conventional operation methods, various features and components in the figure are not drawn according to the actual scale, and the drawing method is to present the specific features and components related to the present invention in the best way. In addition, between different figures, the same or similar element symbols are used to refer to similar elements and components.

1: Wireless network monitoring device

11: Video and audio capture unit

2: Sensor

3: Artificial Intelligence Internet of Things Gateway Device

31: Edge Operation Unit

32: The video access protocol

33: Network interface

34: Streaming Protocol

4: Input and output devices

5: Instant messaging host

50: Internet cloud

51: Fixed connection

6: Terminal device

61: Audio and video data transmission requirements

7: End-to-end connection control host

8: Condition area

81: Near-end video

Claims (9)

An integrated system of artificial intelligence Internet of Things gateway and near-end audio and video, comprising: A wireless network monitoring device, covering and monitoring a plurality of near-end audio and video in a situation area, the wireless network monitoring device further comprising a plurality of audio and video capture units arranged in the situation area, the audio and video capture units can capture audio and video information of the near-end audio and video; A plurality of sensors, the sensors are adjacent to the audio and video capture units, and can sense environmental information; An artificial intelligence IoT gateway device is telecommunication-connected to the wireless network monitoring device and the sensor. The artificial intelligence IoT gateway device simultaneously receives the video and audio information and the environmental information. When it is compared and determined to be in accordance with a crisis mode, an input/output device is executed; Wherein, the artificial intelligence IoT gateway device receives the video and audio information and the environmental information using a video and audio access protocol; Wherein, the video and audio access protocol is a real-time streaming protocol (Real Time Streaming Protocol, RTSP) or a network video monitoring protocol (Open Network Video Interface Forum, ONVIF). An integrated system of an artificial intelligence IoT gateway and near-end audio and video as described in claim 1, wherein the integrated system further includes an instant messaging host and a terminal device, the instant messaging host is located on a network cloud and is telecommunication-connected to the artificial intelligence IoT gateway device via a persistent connection, the terminal device is directly or indirectly connected to the instant messaging host, and the artificial intelligence IoT gateway device can receive an audio and video data transmission request from the terminal device through the instant messaging host. The integrated system of artificial intelligence Internet of Things gateway and local audio and video as described in claim 2, wherein the fixed connection is a message queuing telemetry transport (MQTT). An integrated system of artificial intelligence Internet of Things gateway and near-end audio and video as described in claim 2, wherein the integrated system further includes an end-to-end connection control host, the artificial intelligence Internet of Things gateway device is telecommunication-connected to the end-to-end connection control host via a network interface, and the artificial intelligence Internet of Things gateway device uses the network interface to establish an end-to-end connection with the terminal device through the end-to-end connection control host. As described in claim 4, the integrated system of artificial intelligence Internet of Things gateway and near-end audio and video, wherein the network interface is an Ethernet, a wireless network (Wifi) or a long-term evolution technology (Long Term Evolution, LTE), and can be connected with minimal delay. As described in claim 1, the integrated system of artificial intelligence Internet of Things gateway and proximal audio and video, wherein the artificial intelligence Internet of Things gateway device further includes an edge computing unit, and the edge computing unit uses an edge computing to interpret the audio and video information. An integrated system of artificial intelligence Internet of Things gateway and near-end audio and video as described in claim 1, wherein the sensor is at least one of a temperature sensor and a smoke sensor. A method for integrating an artificial intelligence Internet of Things gateway and near-end audio and video includes the following steps: (a) providing a wireless network monitoring device and an artificial intelligence Internet of Things gateway device, wherein the wireless network monitoring device further includes a plurality of audio and video capture units arranged in a situation area, the artificial intelligence Internet of Things gateway device is telecommunication-connected to the wireless network monitoring device, and the artificial intelligence Internet of Things gateway device further includes an edge computing unit; (b) capturing audio and video information with the audio and video capture unit; (c) the edge computing unit interpreting the audio and video information with an edge computing; (d) determining whether the audio and video information is a crisis mode; (e) If yes, execute an input/output device; if no, repeat the step (b); wherein the artificial intelligence IoT gateway device receives the video information and the environmental information by a video access protocol; wherein the video access protocol is a real-time streaming protocol (Real Time Streaming Protocol, RTSP) or a network video monitoring protocol (Open Network Video Interface Forum, ONVIF). The method for integrating an artificial intelligence IoT gateway with near-end audio and video as described in claim 8, wherein the step (e) further includes the following steps: (f) providing a plurality of sensors, the sensors being adjacent to the audio and video capture unit and being electronically connected to the wireless network monitoring device; (g) the sensors sensing the environmental information; (h) whether the environmental information is the distress mode; (i) if so, executing the input/output device; if not, repeating the step (g).

Images