KR102894893B1 - Method for building a manufacturing language database for obtaining automated manufacturing injection molding data - Google Patents

Abstract

A method for constructing a manufacturing language database for securing automated manufacturing injection molding data according to one embodiment of the present invention comprises the steps of: a) a step in which a code of a web crawler is written from a terminal, and the web crawler is periodically executed by the code to visit web pages on the Internet at specific times and periodically collect documents related to a manufacturing injection molding process to be used for training a large language model; b) a step in which the documents related to the manufacturing injection molding process are stored in a manufacturing language database; c) a loading step in which the terminal loads documents related to the manufacturing injection molding process of the manufacturing language database into a memory; d) a step in which the terminal parses data related to the manufacturing injection molding process from documents related to the manufacturing injection molding process loaded into the memory; e) a step in which the terminal preprocesses documents related to the manufacturing injection molding process that have been parsed; f) a step in which the terminal converts data related to the manufacturing injection molding process from the preprocessed documents related to the manufacturing injection molding process so that the large language model can learn from the documents related to the manufacturing injection molding process; and g) a step of updating the manufacturing language database by repeating steps a) to f).

Description

Method for Building a Manufacturing Language Database for Obtaining Automated Manufacturing Injection Molding Data

The present invention relates to a method for constructing a manufacturing language database for securing automated manufacturing injection molding data, and more particularly, to a method for automatically constructing a manufacturing language database for storing documents related to a manufacturing injection molding process for training a large language model specialized for the manufacturing injection molding process.

A Large Language Model (LLM) is an artificial intelligence (AI)-based, massive language model that learns from a massive amount of data and performs natural language processing (NLP) tasks. It is typically trained using datasets consisting of large parameters (weights) and billions of words, and can perform various language-related tasks.

In order to train these large-scale language models into large-scale language models specialized for the manufacturing injection molding process, it is essential to collect and preprocess various data related to the manufacturing injection molding process.

In addition, data related to the manufacturing injection molding process includes not only structured data but also unstructured data. However, existing methods for collecting and preprocessing unstructured data often rely on manual work, which is time-consuming and costly, and has the problem of low efficiency.

In particular, when collecting and preprocessing unstructured data manually, there are concerns about the decrease in accuracy and reliability of data analysis due to incompleteness, redundancy, and structural problems of unstructured data.

Republic of Korea Patent No. 10-2674954 (registered on June 10, 2024)

Accordingly, the present invention has been devised to solve the above problems, and the purpose of the present invention is to provide a manufacturing language database construction method for automatically securing manufacturing injection molding data, which can automatically construct a manufacturing language database storing documents related to a manufacturing injection molding process that have gone through the steps of collecting, loading/parsing, preprocessing, and converting sequentially to train a large language model specialized for the manufacturing injection molding process.

However, the technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

In order to achieve the above object, according to one embodiment of the present invention, a method for constructing a manufacturing language database for securing automated manufacturing injection molding data comprises the steps of: a) a step in which a code of a web crawler is written from a terminal, and the web crawler is periodically executed by the code to visit web pages on the Internet at specific times and periodically collect documents related to a manufacturing injection molding process to be used for training a large language model; b) a step in which the documents related to the manufacturing injection molding process are stored in a manufacturing language database; c) a loading step in which the terminal loads documents related to the manufacturing injection molding process of the manufacturing language database into a memory; d) a step in which the terminal parses data related to the manufacturing injection molding process from documents related to the manufacturing injection molding process loaded into the memory; e) a step in which the terminal preprocesses documents related to the manufacturing injection molding process that have been parsed; f) a step in which the terminal converts data related to the manufacturing injection molding process from the preprocessed documents related to the manufacturing injection molding process so that the large language model can learn from the documents related to the manufacturing injection molding process; and g) a step of updating the manufacturing language database by repeating steps a) to f).

In addition, the step a) may include a-1) a step in which the web crawler searches Wikipedia at specific times and performs web crawling to collect documents related to the manufacturing injection molding process from web pages on the Internet; a-2) a step in which the web crawler collects domestic and foreign papers related to the manufacturing injection molding process from among domestic and foreign papers registered in an academic database at specific times; a-3) a step in which the web crawler collects domestic and foreign specialized books related to the manufacturing injection molding process from among domestic and foreign specialized books registered on web pages on the Internet at specific times; a-4) a step in which the web crawler collects manufacturing-specialized newspaper articles related to the manufacturing injection molding process from among manufacturing-specialized newspaper articles registered on web pages on the Internet; and a-5) a step in which the large language model mutually translates the languages of the documents, domestic and foreign papers, domestic and foreign specialized books, and manufacturing-specialized newspaper articles related to the manufacturing injection molding process collected in steps a-1) to a-4) into English and Korean.

And the step d) above may include: d-1) a step in which the terminal secures or converts a document related to the manufacturing injection molding process loaded in the memory into a PDF file, and then parses a text block related to the manufacturing injection molding process from the PDF file; d-2) a step in which the terminal parses a table and formula related to the manufacturing injection molding process from the PDF file; d-3) a step in which the terminal extracts and tags a reference image and a graph image related to the manufacturing injection molding process from the PDF file; and d-4) a step in which the terminal extracts structured data from the parsed text block and table, and the extracted and tagged reference image and graph image, converts the structured data into the form of a relational database, and then stores the same in the manufacturing language database.

Additionally, the above-mentioned structured data may include temperature, pressure, time, speed, and equipment specifications, which are structured data of the manufacturing injection molding process.

And the step d-1) may include: d-1-1) a step in which the terminal extracts a region of interest including data related to the manufacturing injection molding process from each page of the PDF file when a document related to the manufacturing injection molding process loaded in the memory is secured or converted into a PDF file, and removes unnecessary regions excluding the region of interest; d-1-2) a step in which the terminal parses the region of interest by distinguishing the left/right of a text block composed of text related to the manufacturing injection molding process among the regions of interest; d-1-3) a step in which the terminal is equipped with a Fitz library, and a step in which the terminal extracts reference images and graph images related to the manufacturing injection molding process from each page of the PDF file using the Fitz library and then performs tagging; and d-1-4) a step in which the terminal tags metadata of the PDF file.

Additionally, the area of interest may include text blocks, tables, formulas, reference images, and graph images that constitute text related to the manufacturing injection molding process.

And the metadata of the PDF file may include the title, author, organization, page number, revision content and date, relationship between documents, and subject (topic) of the PDF file.

In addition, in the step d-2), the terminal can detect the border of the table and the location of the cell from each page of the PDF file based on OCR (Optical Character Recognition) and image analysis technology, and extract the table from each page.

And the step d-2) above is a procedure for detecting merged cells from the table extracted by the terminal, remembering the row merge property and column merge property of each cell, and arranging the cells in the parsed table according to the merge property of each cell when parsing the table so that the row/column positions are accurately maintained, a procedure for extracting the row information and column information to which each cell belongs together with the data of each cell when extracting the table through the row merge and column merge property of each cell, and parsing even empty cells without data so that the row/column structure in the parsed table is not disrupted, and a procedure for refining outliers or missing values of each cell from the extracted table can be automated.

In addition, in the step d-3), the terminal can extract and tag reference images and graph images from each page of the PDF file based on OCR (Optical Character Recognition) and image processing technology.

And the step d-3) above can be automated, including a procedure in which the terminal detects a pattern of a graph in the PDF file to identify the graph, extracts only the identified graph portion to obtain the graph image, a procedure in which metadata of the reference image and the graph image are extracted and tagged, and a procedure in which tags for the subject of the reference image and the graph image are generated based on deep learning-based image recognition technology, and a procedure in which the reference image and the graph image, and tags for the subject of the reference image and the graph image are stored in the manufacturing language database.

In addition, the step e) may include: e-1) a step in which the terminal performs quality filtering of documents related to the manufacturing injection molding process for which parsing has been completed; e-2) a step in which the terminal removes duplicate data from data related to the manufacturing injection molding process in the manufacturing language database; e-3) a step in which the terminal performs a privacy scrubbing process of personal identification information included in the data related to the manufacturing injection molding process in the manufacturing language database; and e-4) a step in which the terminal performs a toxic and biased text filtering process of data related to the manufacturing injection molding process stored in the manufacturing language database.

And in the above step e-1), the terminal can perform quality filtering of documents related to the manufacturing injection molding process for which parsing has been completed based on a heuristic method or a classifier method.

The present invention automatically collects documents related to a manufacturing injection molding process, including structured data as well as unstructured data that were previously dependent on manual work, thereby minimizing repetitive and time-consuming tasks and thereby increasing the efficiency of data processing, thereby reducing labor and operating costs.

In addition, the present invention reduces the possibility of input errors and duplication of data related to the manufacturing injection molding process due to manual work, thereby increasing the accuracy of data related to the manufacturing injection molding process, ensures reliability by applying a consistent processing method for data related to the manufacturing injection molding process, and periodically updates the manufacturing language database, thereby increasing the reliability of a large language model and supporting rapid decision-making.

In addition, the present invention can construct an accurate and rich manufacturing language database through mutual translation between English and Korean of data related to a manufacturing injection molding process collected from web pages on the Internet.

In addition, the present invention can facilitate the learning of a large language model by preventing the loss of row/column information in tabular information through PDF file and table parsing of data related to a manufacturing injection molding process.

Furthermore, the present invention can enhance data privacy, ensure compliance with data protection laws, increase data usability, reduce the risk of data misuse and unauthorized access, and alleviate concerns about potential leakage of personal information through a privacy scrubbing process of data related to a manufacturing injection molding process.

Additionally, the present invention enables improved user experience, legal and regulatory compliance, reduced bias, and maintained fairness through a process of filtering out toxic and biased text from data related to a manufacturing injection molding process.

And the present invention can be utilized for fact-checking as a manufacturing language database as a verification methodology for illusion/false information of injection molding ontology or manufacturing injection molding-related generative AI.

However, the effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

FIG. 1 is a flowchart illustrating a process of a method for constructing a manufacturing language database for securing automated manufacturing injection molding data according to one embodiment of the present invention.
Figure 2 is a flowchart illustrating a detailed process of a data collection step according to one embodiment of the present invention.
FIG. 3 is a flowchart illustrating a detailed process of parsing during the data loading/parsing step according to one embodiment of the present invention.
Figure 4 is a drawing for explaining the detailed process of the PDF file parsing step.
Figure 5 is a flowchart illustrating a detailed process of a data preprocessing step according to one embodiment of the present invention.
Figure 6 is a flowchart illustrating the detailed process of the heuristic-based quality filtering step.
Figure 7 is a flowchart illustrating the detailed process of the classifier-based quality filtering step.
Figure 8 is a flowchart illustrating a detailed process of a data conversion step according to one embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings so that those skilled in the art can easily practice the present invention. However, the description of the present invention is merely an embodiment for structural and functional explanation, and therefore the scope of the present invention should not be construed as being limited by the embodiments described in the text. That is, since the embodiments can be modified in various ways and can have various forms, the scope of the present invention should be understood to include equivalents that can realize the technical idea. In addition, the purposes or effects presented in the present invention do not mean that a specific embodiment must include all of them or only such effects, and therefore the scope of the present invention should not be construed as being limited thereby.

The meanings of terms described in the present invention should be understood as follows.

Terms such as "first" and "second" are intended to distinguish one component from another, and the scope of the rights should not be limited by these terms. For example, a first component could be referred to as a second component, and similarly, a second component could also be referred to as a first component. When a component is referred to as being "connected" to another component, it should be understood that it may be directly connected to that other component, but there may also be other components in between. Conversely, when a component is referred to as being "directly connected" to another component, it should be understood that there are no other components in between. Meanwhile, other expressions describing the relationship between components, such as "between" and "immediately between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

Singular expressions should be understood to include plural expressions unless the context clearly indicates otherwise, and terms such as "comprises" or "has" should be understood to specify the presence of stated features, numbers, steps, operations, components, parts, or combinations thereof, but not to exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted to be consistent with their meaning within the context of the relevant technology, and should not be interpreted as having ideal or overly formal meanings unless explicitly defined herein.

How to build a manufacturing language database

Hereinafter, a preferred embodiment of a manufacturing language database construction method (S100) for securing automated manufacturing injection molding data according to the present invention will be described in detail with reference to the attached drawings.

FIG. 1 is a flowchart illustrating a process of a method for constructing a manufacturing language database for securing automated manufacturing injection molding data according to one embodiment of the present invention.

Referring to FIG. 1, the manufacturing language database construction method (S100) of the present invention is a method for constructing a manufacturing language database (not shown) that stores documents related to a manufacturing injection molding process for training a large language model (LLM) specialized for a manufacturing injection molding process, and in one embodiment, the method proceeds in the following order: a data collection step (S110), a data loading/parsing step (S120), a data preprocessing step (S130), a data conversion step (S140), and a manufacturing language database update step (S150), and the manufacturing language database can be constructed and updated based on repeating the processes of the above steps (S110 to S150).

That is, the manufacturing language database of the present invention may be a database that stores documents related to a manufacturing injection molding process for training a large language model (LLM) specialized in the manufacturing injection molding process.

Meanwhile, each step (S110 to S150) of the manufacturing language database construction method (S100) can be performed by a user (or administrator) who uses a large language model, but below, it will be explained assuming that it is performed using a terminal equipped with a large language model specialized for the manufacturing injection molding process so that it can answer questions about the manufacturing injection molding process by learning based on documents related to the manufacturing injection molding process stored in the manufacturing language database.

In the above data collection step (S110), the user sets the operation schedule of the web crawler by writing a code of the web crawler using a programming language such as Python through the operation of the terminal in order to collect documents related to the manufacturing injection molding process for training a large language model, and the web crawler is periodically executed by the code to collect the latest data at specific times.

At this time, a web crawler is an automated program or bot that explores websites and collects information. It can visit various web pages on the Internet, follow links, analyze content, and store or index necessary data.

In addition, the latest data periodically collected by the web crawler from the user's terminal refers to documents related to the manufacturing injection molding process required to train the large language model to be specialized in the manufacturing injection molding process, and the documents related to the manufacturing injection molding process may include structured data or unstructured data related to the manufacturing injection molding process.

Here, structured data refers to structured data such as numerical data such as temperature, pressure, time, speed, and equipment specifications of the manufacturing injection molding process, and unstructured data refers to unstructured data such as tables, formulas, reference images, and graph images.

In the above data collection step (S110), the user can operate the terminal to automatically build an ETL (Extract, Transform, Load) pipeline on the terminal to process extraction, transformation, and loading of the latest data collected by the web crawler, and automate the data collection process for each manufacturing subject area to store the latest data in a central data warehouse and a manufacturing language database (preferably, a manufacturing language database).

At this time, the ETL pipeline can be a system that processes the latest data by automating the process of the Extract step for collecting documents related to the manufacturing injection molding process, which is the latest data, from the data source, the Transform step for parsing, refining, and converting the collected latest data into the desired format, and the Load step for storing the converted latest data in a central data warehouse or manufacturing language database.

Additionally, the central data warehouse serves as a central repository for storing the latest data loaded through the ETL pipeline in a structured format. Data related to the manufacturing injection molding process can be stored in the central data warehouse by manufacturing subject area. However, the following description assumes that the latest data is stored only in the manufacturing language database.

In the above data collection step (S110), the user can use a large language model to create a table of contents related to each manufacturing subject area and automatically create questions and answers for each detailed table of contents by operating the terminal.

At this time, the large language model generates a table of contents related to each manufacturing subject area because the latest data collected through web crawlers is often very large and unstructured, so this latest data is structured by manufacturing subject area.

Additionally, the large-scale language model automatically generates questions and answers by detailed table of contents to reduce the time required for users to directly analyze and organize the massive amount of up-to-date data collected through web crawlers, thereby minimizing human resources consumption and operational costs incurred in managing up-to-date data.

Furthermore, the large-scale language model undergoes continuous model training based on documents related to the manufacturing injection molding process, which are the latest data collected through web crawlers. As time goes by, it will be able to accurately generate questions and answers about detailed table of contents of manufacturing subject areas related to the manufacturing injection molding process.

In the above data collection step (S110), the detailed process of collecting documents related to the manufacturing injection molding process, which are the latest data, at specific time intervals by a web crawler scheduled to be run periodically by the user through the operation of the terminal is as shown in Fig. 2.

Figure 2 is a flowchart illustrating a detailed process of a data collection step according to one embodiment of the present invention.

Referring to FIG. 2, the data collection step (S110) of the present invention can collect the latest data in the following order: a web scraping-based data collection step (S111), a domestic and foreign paper collection step (S112), a domestic and foreign specialized book collection step (S113), and a manufacturing specialized newspaper article collection step (S114) when a web crawler searches a website at a specific time through scheduling. The steps (S111 to S114) can be performed sequentially or simultaneously.

In the above web scraping-based data collection step (S111), the web crawler can collect data related to the manufacturing injection molding process by searching Wikipedia at scheduled specific times through manipulation of the user's terminal, and additionally, can perform web crawling to collect documents related to the manufacturing injection molding process from various web pages on the Internet other than Wikipedia.

In the above domestic and international paper collection step (S112), the web crawler can search academic databases such as ArXiv and DBPia at scheduled specific times by manipulating the user's terminal to collect domestic and international papers related to the manufacturing injection molding process among the domestic and international papers registered in the academic database.

In the above domestic and international specialized book collection step (S113), the web crawler can collect domestic and international specialized books related to the manufacturing injection molding process among domestic and international specialized books registered on various websites on the Internet at scheduled specific times by manipulating the user's terminal.

In the above manufacturing specialized newspaper article collection step (S114), the web crawler can collect manufacturing specialized newspaper articles related to the manufacturing injection molding process among manufacturing specialized newspaper articles registered on various websites on the Internet at scheduled specific times by manipulating the user's terminal.

After these collection steps (S111 to S114), the data collection step (S110) may proceed with a translation step (S115) in which the large language model mutually translates (or bidirectionally translates) the languages of the latest data collected through the collection steps (S111 to S114).

In the above translation step (S115), the large language model can automatically translate into Korean if the language of the document related to the manufacturing injection molding process collected in the collection step (S111 to S114) is English, and conversely, if the language of the document related to the manufacturing injection molding process is Korean, it can automatically translate into English.

At this time, the large language model was described as translating documents related to the manufacturing injection molding process based on mutual translation between English and Korean in the translation step (S115), but it is not limited to this, and mutual translation between other languages (e.g., Japanese, Chinese, etc.) and Korean can also be performed in addition to English.

In this way, as the large language model performs mutual translation of documents related to the manufacturing injection molding process, the central data warehouse and manufacturing language database of the present invention can be constructed as a multilingual database that stores documents related to the manufacturing injection molding process based on various languages.

Additionally, the large-scale language model can provide answers to questions related to the manufacturing injection molding process in various languages, not just in a specific language, as the manufacturing language database is built as a multilingual database that stores documents related to the manufacturing injection molding process based on various languages.

Referring again to FIG. 1, the data loading/parsing step (S120) may be performed in the order of the loading step and the parsing step.

In the loading step among the above data loading/parsing steps (S120), the terminal can be operated by the user to load a document related to the manufacturing injection molding process stored in the manufacturing language database into the terminal's memory (not shown).

In the parsing step among the above data loading/parsing steps (S120), the terminal can parse a document related to the manufacturing injection molding process of the manufacturing language database loaded into the memory by being manipulated by the user, and the detailed process of data parsing related to the manufacturing injection molding process is as illustrated in FIG. 3.

FIG. 3 is a flowchart illustrating a detailed process of parsing during the data loading/parsing step according to one embodiment of the present invention.

Referring to FIG. 3, the parsing step among the data loading/parsing step (S120) of the present invention may proceed in the following order: PDF file parsing step (S121), table parsing step (S122), formula parsing step (S123), image extraction and tagging step (S124), and structured data RDB storage/management step (S125).

In the above PDF file parsing step (S121), the terminal can be operated by the user to obtain or convert a document related to the manufacturing injection molding process of the manufacturing language database loaded into the memory into a PDF (Portable Document Format) file.

At this time, data related to the manufacturing injection molding process included in the PDF file may be unstructured data.

In this way, the detailed process of converting a document related to the manufacturing injection molding process into a PDF file and then parsing the PDF file will be explained based on the PDF file shown in Fig. 4.

Figure 4 is a drawing for explaining the detailed process of the PDF file parsing step.

Referring to FIG. 4, in the PDF file parsing step (S121), when a document related to a manufacturing injection molding process is secured or converted into a PDF file, the terminal can extract (crop) data related to the manufacturing injection molding process from each page of the PDF file into an area of interest of the manufacturing injection molding process, and remove unnecessary areas excluding this ('①' in FIG. 4).

At this time, the area of interest extracted by the terminal from the PDF file includes text blocks, tables, formulas, reference images, and graph images that constitute text related to the manufacturing injection molding process.

Additionally, the unnecessary area that is removed from the PDF file may be the upper, lower, left, and right outer borders included in the PDF file.

In the above PDF file parsing step (S121), the terminal can parse the area of interest by distinguishing the left/right of the text block so that the text block composed of text related to the manufacturing injection molding process among the areas of interest of the manufacturing injection molding process extracted from each page of the PDF file is arranged to the left/right of the same page due to the format structure characteristics of the PDF file, so as to maintain the form of the text block ('②' in FIG. 4).

In the above PDF file parsing step (S121), the terminal is equipped with a Fitz library, and by using the Fitz library, reference images and graph images related to the manufacturing injection molding process can be extracted from each page of the PDF file, and then tagging can be performed ('③' in Fig. 4).

At this time, tagging reference images and graph images is to systematically classify and store reference images and graph images, which are unstructured data, in a manufacturing language database, and then use them for image retrieval related to the query when applying RAG (Retrieval Augmented Generation).

In the above PDF file parsing step (S121), the terminal can tag the metadata of the PDF file ('④' in Fig. 4).

At this time, the metadata of the PDF file may include the title, author, organization, page number, revision content and date, relationship between documents, subject (topic), etc. of the PDF file.

In addition, the terminal tags the metadata of PDF files to systematically classify and store documents related to the manufacturing injection molding process acquired or converted into PDF files in a manufacturing language database, and to utilize them for retrieving documents related to queries in RAG (Retrieval Augmented Generation).

In the above table parsing step (S122), the terminal can be operated by the user to parse the table, which is data related to the manufacturing injection molding process, in a manner that prevents loss of row/column information.

In the above table parsing step (S122), when a document related to a manufacturing injection molding process is acquired or converted into a PDF file by the PDF file parsing step (S121), the terminal can extract a table from each page of the PDF file by detecting the border of the table and the location of the cell from each page of the PDF file based on OCR (Optical Character Recognition) and image analysis technology.

However, in the above table parsing step (S122), the terminal may also parse the table from a document related to the manufacturing injection molding process collected from a web page on the Internet through a web crawler.

At this time, when a document related to a manufacturing injection molding process is collected from a web page on the Internet through a web crawler, the terminal can analyze HTML tags (<table>, <tr>, <td>, <th>) and CSS styles from the document related to the manufacturing injection molding process, and extract a table from the data related to the manufacturing injection molding process.

The terminal of the present invention can automate the following procedures in the table parsing step (S122) to parse the table.

In the above table parsing step (S122), the terminal detects merged cells from a table extracted from a document related to a manufacturing injection molding process, remembers the row merge attribute and column merge attribute of each cell, and arranges them in accordance with the merge attribute of each cell in the parsed table when parsing the table so that the row/column positions are accurately maintained.

At this time, when the terminal parses a table from a document related to a manufacturing injection molding process collected from a web page on the Internet through a web crawler, if a table is extracted from the document related to the manufacturing injection molding process through the row merge and column merge attributes of the <td> or <th> tag in the HTML tag, cells from the table can be merged.

In the above table parsing step (S122), when the terminal extracts the table through the row merge and column merge properties of each cell, it extracts the row information and column information to which each cell belongs together with the data of each cell, and parses empty cells without data to prevent the row/column structure in the parsed table from being disrupted.

In the above table parsing step (S122), the terminal can maintain consistency and refine outliers or missing values through data normalization of each cell from the extracted table as needed.

At this time, data normalization of each cell may be to unify the notation of the data unit of each cell, outlier cleaning may be to adjust the data unit value of the cell to the average range, and missing value cleaning may be to remove duplicate data among the data of the cell.

In the above formula parsing step (S123), the terminal can recognize the formula and convert it into LaTex format using an artificial intelligence model.

In the above image extraction and tagging step (S124), if a document related to the manufacturing injection molding process is secured or converted into a PDF file through the PDF file parsing step (S121), the terminal can extract and tag reference images and graph images from each page of the PDF file based on OCR and image processing technology.

However, in the image extraction and tagging step (S124), the terminal may extract reference images and graph images from documents related to the manufacturing injection molding process collected from web pages on the Internet through a web crawler and then tag them.

At this time, when a document related to a manufacturing injection molding process is collected from a web page on the Internet through a web crawler, the terminal detects and downloads <img> tags and CSS background images for reference images and graph images from the document related to the manufacturing injection molding process, thereby extracting reference images and graph images from data related to the manufacturing injection molding process.

The terminal of the present invention can automate the following procedures in the image extraction and tagging step (S124) to extract and tag reference images and graph images.

In the above image extraction and tagging step (S124), when extracting a graph image, the terminal can detect specific patterns of the graph, such as axes, data points, and labels in the PDF file, identify the graph, and extract only the identified graph portion to obtain a graph image.

In the above image extraction and tagging step (S124), the terminal can extract and tag metadata of reference images and graph images, such as titles, captions, and sources, and generate tags for the subjects of the reference images and graph images based on deep learning-based image recognition technology.

In the image extraction and tagging step (S124) above, the terminal can store the reference image and graph image and tags for the subject of the reference image and graph image in the manufacturing language database so that the user can easily search and manage the reference image and graph image through the manufacturing language database while maintaining the association with the PDF file or table containing the reference image and graph image.

In the above structured data RDB storage/management step (S125), the terminal extracts structured data such as equipment specifications from text blocks and tables parsed in the PDF file parsing step (S121) and table parsing step (S122), formulas extracted in LaTex format in the formula parsing step (S123), and reference images and graph images extracted and tagged in the image extraction and tagging step (S124), and converts the extracted structured data into the form of a relational database (RDB) so that the search and management of the extracted structured data becomes easy, and then stores the extracted structured data in a manufacturing language database.

At this time, the extracted structured data is converted into the form of a relational database (RDB), which means that the structured data is structured in the form of tables so that it is possible to define relationships between tables.

Referring back to FIG. 1, in the data preprocessing step (S130), the terminal can preprocess a document containing data related to a manufacturing injection molding process for which parsing has been completed, and the detailed process of the data preprocessing step (S130) for preprocessing data related to a manufacturing injection molding process within the document is as illustrated in FIG. 5.

At this time, the document containing data related to the manufacturing injection molding process is not limited, but in one embodiment, it may be a PDF file containing data related to the manufacturing injection molding process through the PDF file parsing step (S121).

Figure 5 is a flowchart illustrating a detailed process of a data preprocessing step according to one embodiment of the present invention.

Referring to FIG. 5, the data preprocessing step (S130) may be performed in the following order: a quality filtering step (S131), a data deduplication step (S132), a personal information scrubbing step (S133), and a toxicity/bias text filtering step (S134) to preprocess a document containing data related to a manufacturing injection molding process for which parsing has been completed.

In the above quality filtering step (S131), the terminal can perform quality filtering of documents related to the manufacturing injection molding process for which parsing has been completed by removing documents containing data of specific keywords from among documents related to the manufacturing injection molding process for which parsing has been completed, or removing documents that are incomplete or contain errors due to low reliability.

In the above quality filtering step (S131), the terminal can perform quality filtering of documents related to the manufacturing injection molding process for which parsing has been completed based on at least one quality filtering method among the methods illustrated in FIGS. 6 and 7.

Figure 6 is a flowchart illustrating the detailed process of the heuristic-based quality filtering step.

Referring to FIG. 6, in the quality filtering step (S131), the terminal may perform quality filtering of documents related to the manufacturing injection molding process for which parsing has been completed through a heuristic method that removes documents that do not conform to rules among documents containing data related to the manufacturing injection molding process for which parsing has been completed based on predefined rules.

In one embodiment, the quality filtering step based on a heuristic method may proceed in the following order: data loading step (S131-1), preprocessing step (S131-2), rule definition step (S131-3), data review step (S131-4), keyword filtering inclusion/exclusion determination step (S131-5), document removal step (S131-6), length check step (S131-7), format check step (S131-8), and special character and space check step (S131-9).

In the above data loading step (S131-1), the terminal can load a document containing data related to a manufacturing injection molding process for which parsing has been completed from a manufacturing language database in order to perform quality filtering of the document related to a manufacturing injection molding process for which parsing has been completed.

In the above preprocessing step (S131-2), the terminal can remove duplicate data or process missing values from data related to the manufacturing injection molding process in the loaded document.

In the above rule definition step (S131-3), the terminal can define a rule for removing a specific document from the document when quality filtering of a document related to a manufacturing injection molding process for which parsing has been completed is performed based on a heuristic.

At this time, the rule defined in the rule definition step (S131-3) may include specific keywords to be removed from the text constituting the document containing data related to the manufacturing injection molding process for which parsing has been completed, the length range of the text within the document, the format (e.g., date format), special characters, and space removal, etc.

In the above data review step (S131-4), the terminal can review whether a document containing data related to a manufacturing injection molding process for which parsing has been completed is a document available for training of a large language model, and additionally, after reviewing whether there are outliers or missing values in the document, the terminal can remove documents with outliers or missing values from the manufacturing language database.

In the step (S131-5) of determining whether the keyword filtering is included, the terminal can determine whether a specific keyword defined as a rule in the rule definition step (S131-3) is included in a document containing data related to the manufacturing injection molding process for which parsing has been completed.

At this time, if a document containing data related to a manufacturing injection molding process for which parsing has been completed contains a specific keyword (S131-5_YES), the terminal can remove the document containing the specific keyword (S131-6).

It means that the documents removed in the above document removal step (S131-6) are removed from the manufacturing language database.

In contrast, if a document containing data related to a manufacturing injection molding process for which parsing has been completed does not contain a specific keyword (S131-5_NO), the terminal determines that the document containing data related to a manufacturing injection molding process for which parsing has been completed is available for training of a large language model, and can examine the text length of the document (S131-7).

In the above length checking step (S131-7), the terminal can remove documents whose text length is less than or exceeds the text length range defined by the rule defined in the above rule definition step (S131-3) from the manufacturing language database.

In the above format inspection step (S131-8), the terminal checks the format of a document containing data related to a manufacturing injection molding process for which parsing has been completed, and, according to the rules defined in the above rule definition step (S131-3), can remove documents that do not conform to the format defined by the rules from the manufacturing language database.

In the above special character and space inspection step (S131-9), the terminal inspects the special characters and space level of a document containing data related to a manufacturing injection molding process for which parsing has been completed, and, according to the rules defined in the above rule definition step (S131-3), documents that contain special characters defined by the rules or exceed the space level defined by the rules can be removed from the manufacturing language database.

Figure 7 is a flowchart illustrating the detailed process of the classifier-based quality filtering step.

Referring to FIG. 7, in the quality filtering step (S131), the terminal can perform quality filtering of documents related to the manufacturing injection molding process for which parsing has been completed through a classifier method that trains a machine learning model and uses the trained machine learning model.

In one embodiment, the quality filtering step based on a classifier method may be performed in the following order: a data loading step (S131a), a preprocessing step (S131b), a feature extraction step (S131c), a training data preparation step (S131d), a model training step (S131e), a confidence filtering level determination step (S131f), a document removal step (S131g), a length checking step (S131h), a format checking step (S131i), and a special character and space checking step (S131j).

In the above data loading step (S131a), the terminal can load a document containing data related to a manufacturing injection molding process for which parsing has been completed from a manufacturing language database in order to perform quality filtering of the document containing data related to a manufacturing injection molding process for which parsing has been completed.

In the above preprocessing step (S131b), the terminal can remove duplicate data or process missing values in documents related to the manufacturing injection molding process from the loaded documents.

In the above feature extraction step (S131c), the terminal can extract features to be used for learning a classification model (machine learning model) by converting data related to the manufacturing injection molding process for which document parsing has been completed into a vector by digitizing the data.

At this time, the terminal can extract features to be used for learning a classification model from the document based on the frequency of specific keywords, the length range of the text within the document, format (e.g., date format), special characters, and the degree of spaces from the text of the document containing data related to the manufacturing injection molding process for which parsing has been completed.

In the above training data preparation step (S131d), the terminal can prepare data for training a classification model by labeling the document containing data related to a manufacturing injection molding process converted into a vector as “good (value is ‘1’)” or “bad (value is ‘0’)” according to the quality criteria.

In the above model training step (S131e), the classification model can be trained to classify the quality of a document containing data related to a manufacturing injection molding process converted into a vector by learning the relationship between the features extracted in the above feature extraction step (S131c) and the labels in the above training data preparation step (S131d).

In the above confidence filtering level determination step (S131f), the trained classification model can predict the likelihood that a document containing data related to a manufacturing injection molding process converted into a vector is of good or bad quality, and determine whether the reliability of the result of the prediction of the likelihood that the document is of good or bad quality is above a preset level.

At this time, if the confidence level of the result predicted by the trained classification model is below a preset level (S131f-NO), the trained classification model can remove documents with a confidence level below a preset level (S131g).

It means that the documents removed in the above document removal step (S131g) are removed from the manufacturing language database.

In contrast, if the reliability of the result predicted by the trained classification model is higher than a preset level (S131f-YES), the trained classification model determines that the document containing data related to the manufacturing injection molding process converted into a vector is available for training the large language model, and the text length of the document can be inspected (S131h).

In the above length inspection step (S131h), the trained classification model can remove documents that are less than or exceed the text length range from the manufacturing language database according to the features extracted in the above feature extraction step (S131c).

In the above format inspection step (S131i), the trained classification model inspects the format of a document containing data related to a manufacturing injection molding process converted into a vector, and, based on the features extracted in the above feature extraction step (S131c), documents in formats not extracted as features can be removed from the manufacturing language database.

In the above special character and space inspection step (S131j), the trained classification model inspects the special characters and space extent of a document containing data related to a manufacturing injection molding process converted into a vector, and, based on the features extracted in the above feature extraction step (S131c), documents in which the special characters extracted as features exist or the space extent extracted as features exceeds the level of features can be removed from the manufacturing language database.

In the above data deduplication step (S132), the terminal performs a deduplication process to remove duplicate data from data related to the manufacturing injection molding process stored in the manufacturing language database, leaving only unique data, thereby preprocessing data related to the manufacturing injection molding process stored in the manufacturing language database.

In the above personal information scrubbing step (S133), the terminal can prevent leakage or infringement of personal identification information (e.g., ID, password, API Key, etc.) included in data related to the manufacturing injection molding process of the manufacturing language database by performing a personal information scrubbing process.

At this time, the terminal may perform an identification process to remove or convert personally identifiable information contained in data related to the manufacturing injection molding process of the manufacturing language database so that the personally identifiable information is not identifiable.

In addition, the terminal can prevent leakage or infringement of personally identifiable information (e.g., ID, password, API key, etc.) included in data related to the manufacturing injection molding process of the manufacturing language database by performing a de-identification or data masking process in addition to identification.

Here, the de-identification process means a process in which the terminal of the present invention converts personal identification information contained in data related to the manufacturing injection molding process of the manufacturing language database into a pseudonym so that the personal identification information is identified differently from the original data.

And the data masking process means a process in which the terminal of the present invention preprocesses personal identification information included in data related to the manufacturing injection molding process of the manufacturing language database by masking it so that the actual value is hidden while maintaining the form of the data.

In the above toxic/biased text filtering step (S134), the terminal may perform a toxic/biased text filtering process to remove harmful and biased texts such as hate/loathing expressions and sexual/racial/religious/political discrimination expressions from among data related to the manufacturing injection molding process of the manufacturing language database, thereby preprocessing data related to the manufacturing injection molding process stored in the manufacturing language database.

Referring back to FIG. 1, in the data conversion step (S140), the terminal can convert data related to the manufacturing injection molding process so that the large language model can learn through data related to the manufacturing injection molding process of the preprocessed manufacturing language database, and the detailed process of the data conversion step (S140) for converting data related to the manufacturing injection molding process is as illustrated in FIG. 8.

Figure 8 is a flowchart illustrating a detailed process of a data conversion step according to one embodiment of the present invention.

Referring to FIG. 8, the data conversion step (S140) may be performed in the following order: a fine tuning format data conversion step (S141) so that the large language model can learn through data related to the manufacturing injection molding process of the manufacturing language database; a large language model API service automation step (S142) that can utilize not only the commercial OpenAI ChatGPT API service but also the open source LLM API service; a tokenization step (S143); and a text embedding/indexing step (S144).

In the above fine-tuning format data conversion step (S141), the terminal can convert data related to the manufacturing injection molding process of the preprocessed manufacturing language database to correspond to the fine-tuning format of the BERT and GPT models.

Here, the BERT model is a model of the Auto-Encoding series that excels in language understanding, and can extract answers by inputting questions and context (paragraphs) together.

Additionally, the GPT model is a model of the Auto-Regressive series with excellent language generation performance, and can generate answers to questions.

In this way, since the data related to the manufacturing injection molding process of the preprocessed manufacturing language database is transformed to correspond to the format of the fine tuning of the BERT and GPT models, it is preferable that the large language model of the present invention is a BERT model or a GPT model.

In the above large language model API service automation step (S142), the terminal can automatically generate questions related to the manufacturing injection molding process and appropriate answers to the questions from data related to the manufacturing injection molding process of the converted manufacturing language database using the service of the large language model API, which is a natural language processing (NLP) model.

At this time, it is desirable that questions and answers related to the manufacturing injection molding process be aligned so that the large language model can generate appropriate answers to the questions.

In the above tokenization step (S143), the terminal can convert data related to the manufacturing injection molding process of the converted manufacturing language database into a text sentence form to secure text data in the text sentence form, and perform a tokenization process to break down the text data into token units (words, subwords, or characters) that the BERT and GPT models can understand.

At this time, the tokenization process can be composed of a token separation step that separates text data into token units, a special token addition step that adds special tokens indicating the beginning and end of sentences in the text data, and a vocabulary dictionary mapping step (indexing) that converts each token into a unique ID in the vocabulary dictionary of the BERT and GPT models.

In the above text embedding/indexing step (S144), the terminal can perform an embedding process to encode the meaning of text in text data decomposed into token units in the tokenization process into numbers in a continuous vector space so that BERT and GPT models can learn and predict.

At this time, the terminal is equipped with an embedding model and can perform the embedding process, and the text is converted into a numerical vector through the text embedding process, and the semantic similarity between words can be expressed as the distance between vectors through embedding.

In the above text embedding/indexing step (S144), the terminal organizes the embedded text data in a specific structured manner on the manufacturing language database, and can build an index so that the embedded text data can be efficiently searched from the manufacturing language database.

At this time, the large language model can quickly retrieve the most relevant documents from the manufacturing language database and then generate an answer when a question about the manufacturing injection molding process with similar meaning is input through the text indexing process.

Meanwhile, after the terminal performs the text embedding/indexing step (S144), it can perform fine tuning of the large language model using open source (SentenceTransformer).

At this time, fine-tuning a large-scale language model using open source is because, in a specific domain such as a manufacturing injection molding process, a pre-trained large-scale language model may not fully understand the terminology or context of the relevant field.

Referring back to FIG. 1, the manufacturing language database update step (S150) may be a step of updating the manufacturing language database by repeating the steps (S110 to S140) rather than a one-time process in which data related to the manufacturing injection molding process is collected, loaded/parsed, preprocessed, and converted through the steps (S110 to S140) described above.

In the above manufacturing language database update step (S150), the terminal can periodically verify data related to the manufacturing injection molding process of the manufacturing language database, and perform a process of removing data unnecessary for learning the large language model during the verification process.

In addition, the terminal stores data related to the manufacturing injection molding process collected from various sources in a unified format in the manufacturing language database whenever the manufacturing language database is updated, thereby constructing the manufacturing language database as an integrated database.

The manufacturing language database constructed through the manufacturing language database construction method (S100) of the present invention constructs a knowledge system (ontology) for the manufacturing injection molding process through continuous updates, and can be utilized for fact-checking as a methodology for verifying illusion/false information of generative AI related to manufacturing injection molding.

Effects according to the present invention

The manufacturing language database construction method (S100) of the present invention automatically collects documents related to a manufacturing injection molding process including structured data as well as unstructured data that were previously dependent on manual work, thereby minimizing repetitive and time-consuming tasks and thereby increasing the efficiency of data processing, thereby reducing labor and operating costs.

In addition, the manufacturing language database construction method (S100) of the present invention reduces the possibility of input errors and duplication of data related to the manufacturing injection molding process due to manual work, thereby increasing the accuracy of data related to the manufacturing injection molding process, ensuring reliability by applying a consistent processing method for data related to the manufacturing injection molding process, and periodically updating the manufacturing language database, thereby increasing the reliability of a large language model and supporting rapid decision-making.

And the manufacturing language database construction method (S100) of the present invention can construct an accurate and rich manufacturing language database through mutual translation between English and Korean of data related to a manufacturing injection molding process collected from a web page on the Internet.

In addition, the manufacturing language database construction method (S100) of the present invention can facilitate learning of a large language model by preventing loss of row/column information in tabular information through parsing of PDF files and tables of data related to a manufacturing injection molding process.

And the manufacturing language database construction method (S100) of the present invention can strengthen data privacy, comply with data protection laws, increase data usability, reduce the risk of data misuse and unauthorized access, and resolve concerns about potential leakage of personal information through a privacy scrubbing process of data related to a manufacturing injection molding process.

In addition, the manufacturing language database construction method (S100) of the present invention can improve user experience, comply with laws and regulations, reduce bias, and maintain fairness through a process of filtering out toxic and biased text from data related to a manufacturing injection molding process.

And the manufacturing language database construction method (S100) of the present invention can be utilized for fact-checking as a manufacturing language database verification methodology for injection molding ontology or manufacturing injection molding-related generative AI against illusion/false information.

The detailed description of the preferred embodiments of the present invention disclosed above has been provided to enable those skilled in the art to implement and practice the present invention. While the above description has been made with reference to preferred embodiments of the present invention, those skilled in the art will appreciate that various modifications and variations can be made to the present invention without departing from the scope of the present invention. For example, those skilled in the art can utilize the individual components described in the above-described embodiments in combination with each other. Accordingly, the present invention is not intended to be limited to the embodiments described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The present invention may be embodied in other specific forms without departing from the technical spirit and essential characteristics thereof. Therefore, the above detailed description should not be construed as limiting in all respects but should be considered illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all changes coming within the equivalent scope of the present invention are intended to be included therein. The present invention is not intended to be limited to the embodiments set forth herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. Furthermore, claims that are not explicitly cited in the claims may be combined to form an embodiment or incorporated into a new claim by post-application amendment.

S100: Method for building a manufacturing language database,
S110: Data collection phase,
S120: Data loading/parsing phase,
S130: Data preprocessing step,
S140: Data conversion step,
S150: Manufacturing Language Database Update Step.

Claims (13)

a) A step in which a code of a web crawler is written from a terminal, and the web crawler is periodically executed by the code to visit web pages on the Internet at specific times and periodically collect documents related to a manufacturing injection molding process to be used for learning a large language model;
b) A step in which documents related to the above manufacturing injection molding process are stored in a manufacturing language database;
c) a loading step in which the terminal loads documents related to the manufacturing injection molding process of the manufacturing language database into memory;
d) a step of parsing data related to a manufacturing injection molding process from a document related to a manufacturing injection molding process loaded in the memory by the terminal;
e) a step of preprocessing a document related to a manufacturing injection molding process for which parsing has been completed by the terminal;
f) a step of converting data related to the manufacturing injection molding process from a document related to the manufacturing injection molding process preprocessed by the terminal so that the large language model can learn; and
g) a step in which the manufacturing language database is updated by repeating steps a) to f);
Step d) above,
d-1) A step of the terminal obtaining or converting a document related to the manufacturing injection molding process loaded in the memory into a PDF file, and then parsing a text block related to the manufacturing injection molding process from the PDF file;
d-2) A step in which the terminal parses tables and formulas related to the manufacturing injection molding process from the PDF file;
d-3) A step in which the terminal extracts and tags reference images and graph images related to the manufacturing injection molding process from the PDF file; and
d-4) A step of extracting structured data from the parsed text block and table, extracted and tagged reference image and graph image of the terminal, converting the structured data into the form of a relational database, and storing it in the manufacturing language database;
Step d-1) above,
d-1-1) When the terminal obtains or converts a document related to the manufacturing injection molding process loaded in the memory into a PDF file, a step of extracting an area of interest containing data related to the manufacturing injection molding process from each page of the PDF file and removing unnecessary areas excluding the area of interest;
d-1-2) A step of parsing the region of interest by distinguishing the left/right of a text block composed of text related to the manufacturing injection molding process among the regions of interest;
d-1-3) A step of tagging after extracting reference images and graph images related to the manufacturing injection molding process from each page of the PDF file using the Fitz library, in which the terminal is equipped with the Fitz library; and
d-1-4) A step in which the terminal tags metadata of the PDF file;
Step d-2) above,
The above terminal detects the border of the table and the location of the cell from each page of the PDF file based on OCR (Optical Character Recognition) and image analysis technology, and extracts the table from each page.
The above large language model is,
In step a) above, in order to structure documents related to the manufacturing injection molding process, which are the latest data collected through a web crawler, by manufacturing subject area, a related table of contents by manufacturing subject area is created and questions and answers are automatically created by detailed table of contents.
The above terminal,
In the above step d-1-3), when applying RAG (Retrieval Augmented Generation), reference images and graph images related to the manufacturing injection molding process are tagged to be used for image retrieval related to questions about the manufacturing injection molding process.
In the above step d-1-4), when applying RAG, the metadata of the PDF file is tagged to be used for document search related to questions about the manufacturing injection molding process.
Step f) above,
f-1) A fine-tuning format data conversion step in which the terminal converts data related to the manufacturing injection molding process of the preprocessed manufacturing language database into a format corresponding to the fine-tuning format of the BERT and GPT models;
f-2) A large language model API service automation step in which the terminal automatically generates questions about the manufacturing injection molding process and answers to the questions from data related to the manufacturing injection molding process in the converted manufacturing language database using the service of the large language model API, which is a natural language processing (NLP) model;
f-3) A tokenization step in which the terminal converts data related to the manufacturing injection molding process of the converted manufacturing language database into a text sentence form, and then decomposes the text data in the text sentence form into token units so that the large language model can understand it;
f-4) A text embedding/indexing step in which the terminal performs an embedding process in which the meaning of the text in the text data decomposed into token units is encoded as numbers in a continuous vector space so that the large language model can learn and predict, and then builds an index so that the embedded text data can be searched from the manufactured language database;
The above terminal,
After performing the above step f-4), fine tuning of the large language model is performed using open source (SentenceTransformer).
The above large language model is,
A method for constructing a manufacturing language database for securing automated manufacturing injection molding data, characterized in that the manufacturing language database is a BERT or GPT model that searches for a document containing text data that is semantically most similar to the question of the manufacturing injection molding process when a question of the manufacturing injection molding process is input through the above step f-4) and generates an answer to the question of the manufacturing injection molding process. In the first paragraph,
Step a) above,
a-1) A step in which the above web crawler searches Wikipedia at specific intervals and performs web crawling to collect documents related to the manufacturing injection molding process from web pages on the Internet;
a-2) A step in which the above web crawler collects domestic and foreign papers related to the manufacturing injection molding process among domestic and foreign papers registered in an academic database at specific times;
a-3) A step in which the above web crawler collects domestic and foreign specialized books related to the manufacturing injection molding process among domestic and foreign specialized books registered on web pages on the Internet at specific times;
a-4) A step in which the web crawler collects manufacturing-specialized newspaper articles related to the manufacturing injection molding process among manufacturing-specialized newspaper articles registered on web pages on the Internet; and
a-5) A method for constructing a manufacturing language database for securing automated manufacturing injection molding data, characterized in that the method comprises a step of mutually translating the languages of documents, domestic and foreign papers, domestic and foreign specialized books, and manufacturing specialized newspaper articles related to the manufacturing injection molding process collected in steps a-1) to a-4) into English and Korean. delete In the first paragraph,
The above structured data is,
A method for constructing a manufacturing language database for securing automated manufacturing injection molding data, characterized in that the manufacturing injection molding process includes structured data such as temperature, pressure, time, speed, and equipment specifications. delete In the first paragraph,
The above areas of interest are:
A method for constructing a manufacturing language database for securing automated manufacturing injection molding data, characterized in that it includes text blocks, tables, formulas, reference images, and graph images that constitute text related to the manufacturing injection molding process. In the first paragraph,
The metadata of the above PDF file is:
A method for constructing a manufacturing language database for securing automated manufacturing injection molding data, characterized in that the PDF file includes the title, author, organization, page number, revision content and date, relationship between documents, and subject (topic). delete In the first paragraph,
Step d-2) above,
A procedure for detecting merged cells from the table extracted by the above terminal, remembering the row merge property and column merge property of each cell, and arranging the cells in the parsed table according to the merge property of each cell when parsing the table so that the row/column positions are accurately maintained;
When extracting a table through the row merge and column merge properties of each cell above, a procedure to extract the row information and column information to which each cell belongs together with the data of each cell, and to parse empty cells without data to prevent the row/column structure from being disrupted in the parsed table;
A procedure for refining outliers or missing values in each cell from the above extracted table,
A method for constructing a manufacturing language database for securing automated manufacturing injection molding data, characterized in that the terminal recognizes the formula and converts the formula into LaTex format. In the first paragraph,
Step d-3) above,
A method for constructing a manufacturing language database for securing automated manufacturing injection molding data, characterized in that the terminal extracts and tags reference images and graph images from each page of the PDF file based on OCR (Optical Character Recognition) and image processing technology. In paragraph 10,
Step d-3) above,
A procedure in which the terminal detects a pattern of a graph in the PDF file to identify the graph, extracts only the identified portion of the graph, and obtains the graph image;
A procedure for extracting and tagging metadata of the above reference images and graph images, and generating tags for the subjects of the above reference images and graph images based on deep learning-based image recognition technology;
A method for constructing a manufacturing language database for securing automated manufacturing injection molding data, characterized in that the procedure for storing the reference image and graph image and the tag for the subject of the reference image and graph image in the manufacturing language database is automated. In the first paragraph,
Step e) above,
e-1) A step in which the terminal performs quality filtering of documents related to the manufacturing injection molding process for which parsing has been completed;
e-2) A step of removing duplicate data from data related to the manufacturing injection molding process of the manufacturing language database by the terminal;
e-3) A step in which the terminal performs a privacy scrubbing process of personal identification information included in data related to the manufacturing injection molding process of the manufacturing language database; and
e-4) A method for constructing a manufacturing language database for securing automated manufacturing injection molding data, characterized in that it comprises a step of filtering out toxic and biased text of data related to the manufacturing injection molding process stored in the manufacturing language database by the terminal. In paragraph 12,
The above step e-1) is,
A method for constructing a manufacturing language database for securing automated manufacturing injection molding data, characterized in that the terminal performs quality filtering of documents related to a manufacturing injection molding process for which parsing has been completed based on a heuristic method or a classifier method.