CN103678697A - Reverse index storage method and system thereof - Google Patents
Reverse index storage method and system thereof Download PDFInfo
- Publication number
- CN103678697A CN103678697A CN201310741178.7A CN201310741178A CN103678697A CN 103678697 A CN103678697 A CN 103678697A CN 201310741178 A CN201310741178 A CN 201310741178A CN 103678697 A CN103678697 A CN 103678697A
- Authority
- CN
- China
- Prior art keywords
- index
- file
- terminal device
- access
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/7867—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/31—Indexing; Data structures therefor; Storage structures
- G06F16/316—Indexing structures
- G06F16/319—Inverted lists
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Library & Information Science (AREA)
- Multimedia (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a reverse index storage method and a system of the method. The method includes the steps that reverse index files of video files are established; a plurality of index servers are provided, the reverse index files are synchronously stored to the index servers, and corresponding index servers are arranged respectively according to an access channel of terminal equipment so that index service can be provided. With the reverse index storage method and the system of the method, the corresponding index servers can be arranged respectively according to the access channel of the terminal equipment so that reverse index information can be retrieved and obtained, and thus access request efficiency and access request rate are improved.
Description
Technical field
The present invention relates to information retrieval technique, relate in particular to a kind of inverted index storage method and system thereof of video resource data.
Background technology
Along with scientific and technological development, increasing user is by internet hunt and watch various videos.The video information providing due to internet is very abundant, and has the feature of continuous variation and renewal, has produced multiple search engine thereupon and has carried out Video Information Retrieval Techniques:.
Current search engine is general uses two kinds of technology to realize information retrieval: the one, use websites collection technology, and tree-shaped classification is carried out in website, the website logging in belongs at least one classification, and each website is had to simple description.The 2nd, use global search technology, global search technology process to as if text, it can for example, set up by word (word) inverted index to document to large volume document (a large amount of webpages on internet), on this basis, when user inquires about document with keyword, system will be returned to the document that contains this keyword to user.The benefit of setting up this inverted index is to check all documents one time for each user's inquiry.
But, terminal device is accessed internet by multiple channel in actual applications, when index service is provided, if do not consider terminal device access channel, to all terminal devices, provide consistent index service, can reduce recall precision, therefore be necessary to propose improved technical scheme and address the above problem.
Summary of the invention
Fundamental purpose of the present invention is to provide a kind of inverted index storage method and system thereof, with solve the not distinguishing terminal equipment that prior art exists access channel, to all terminal devices of access, provide consistent index service to cause reducing the problem of recall precision.
In order to address the above problem, according to an aspect of the present invention, provide a kind of inverted index storage method, it comprises: the inverted index file of setting up video file; A plurality of index servers are provided, described inverted index file synchronization is stored to a plurality of index servers, and according to the access channel of terminal device, corresponding index server is set respectively index service is provided.
Wherein, the described access channel according to terminal device arranges respectively corresponding index server provides index service, comprising: according to the type of terminal device, corresponding index server is set respectively index service is provided; Or the operator's platform using according to terminal device arranges respectively corresponding index server provides index service.
Wherein, described method also comprises: receive the updating file of inverted index file, according to the access channel of terminal device, use the update mode setting in advance that the updating file of described inverted index is distributed to corresponding index server.
Wherein, described method also comprises: the quantity that records the request of access of terminal device; When the quantity of the request of access for same inverted index file surpasses predetermined threshold value, provide the request of access of dilatation index server for receiving terminal apparatus.
Wherein, the described inverted index file of setting up video file, comprising: by default participle mode, video file information is carried out to word segmentation processing and obtain keyword; Set up described keyword and there is the index relative between the video file information of described keyword, thereby set up the inverted index file of video file.
According to a further aspect in the invention, also provide a kind of inverted index storage system, it comprises: set up module, for setting up the inverted index file of video file; Memory module, for being stored to described inverted index file synchronization a plurality of index servers that provide; Module is set, for corresponding index server being set respectively according to the access channel of terminal device, provides index service.
Wherein, the described module that arranges comprises: first arranges module, for corresponding index server being set respectively according to the type of terminal device, provides index service; Second arranges module, for the operator's platform using according to terminal device, corresponding index server is set respectively index service is provided.
Wherein, described system also comprises: update module, and for receiving after the updating file of inverted index file, according to the access channel of terminal device, use the update mode setting in advance that the updating file of described inverted index is distributed to corresponding index server.
Wherein, described system also comprises: logging modle, for recording the quantity of the request of access of terminal device; Administration module, while surpassing predetermined threshold value for the quantity of the request of access when for same inverted index file, provides the request of access of dilatation index server for receiving terminal apparatus.
Wherein, the described module of setting up comprises: keyword acquisition module, carries out word segmentation processing for the participle mode by default to video file information and obtains keyword; Inverted index is set up module, for setting up described keyword and having the index relative between the video file information of described keyword, thereby sets up the inverted index file of video file.
According to technical scheme of the present invention, by inverted index file synchronization is stored to a plurality of index servers, and according to the access channel of terminal device, corresponding index server is set respectively index service is provided, when receiving the request of access of terminal device access inverted index file, can corresponding index server retrieval be set according to the access channel of terminal device and obtain inverted index information, thereby improve efficiency and the speed of request of access.
Accompanying drawing explanation
Accompanying drawing described herein is used to provide a further understanding of the present invention, forms the application's a part, and schematic description and description of the present invention is used for explaining the present invention, does not form inappropriate limitation of the present invention.In the accompanying drawings:
Fig. 1 is according to the process flow diagram of the inverted index storage method of the embodiment of the present invention;
Fig. 2 is the structured flowchart of inverted index storage system according to an embodiment of the invention;
Fig. 3 is the structured flowchart of inverted index storage system according to another embodiment of the present invention.
Embodiment
For making the object, technical solutions and advantages of the present invention clearer, below in conjunction with drawings and the specific embodiments, the present invention is described in further detail.
According to embodiments of the invention, provide a kind of inverted index storage method.
Fig. 1 is according to the process flow diagram of the inverted index storage method of the embodiment of the present invention, and as shown in Figure 1, the method comprises:
Step S102, sets up the inverted index file of video file.
First, by default participle mode, video file information is carried out to word segmentation processing and obtain keyword.Video file information refers to some Word messages such as title that video file comprises, descriptor, brief introduction, obtains the keyword of video file information by word segmentation processing.Usually, word segmentation processing is exactly that continuous word sequence is reassembled into word sequence according to certain standard.The object of participle is exactly that each document analysis is extracted to those words that likely becomes user's query object (word).By default participle mode, file video information is carried out to word segmentation processing, obtain preliminary participle vocabulary.Wherein, default participle mode for example binary is divided morphology, maximum matching method, statistical method scheduling algorithm, does not repeat herein.
Then, set up described keyword and there is the index relative between the video file information of described keyword, thereby set up the inverted index file of video file.
After word segmentation processing obtains keyword, identification information (ID) by keyword together with corresponding file is stored in inverted index file, after All Files is analyzed, by the order of the keyword obtaining to keyword sort, the processing such as merging, add up the probability that each keyword occurs in individual file, and also likely comprise other index informations in index file.For example: number of files, for showing that keyword occurs at how many files; Sum frequency, for the number of times that shows that keyword occurs at All Files; Frequency, for the number of times that shows that keyword occurs at a file.Thereby, set up the incidence relation between keyword and its index information, obtain the inverted index file of video file.
Step S104, provides a plurality of index servers, and described inverted index file synchronization is stored to a plurality of index servers, and according to the access channel of terminal device, corresponding index server is set respectively index service is provided.
The inverted index file synchronization of foundation is stored to a plurality of outside index servers, one or more index servers of corresponding with service are provided according to the access channel setting of terminal device, and in distributed mode, provide index service with a plurality of index servers of the corresponding setting of a class access channel.
During concrete enforcement, after described inverted index file synchronization is stored to a plurality of index servers, this index server can be set at the desired location of each inverted index file provides the access channel information of the terminal device of index service, for when terminal device is initiated request of access, by being arranged on the access channel information of terminal device of the desired location of inverted index file, judge whether current index server provides service for initiating the terminal device of request of access.Or, according to the access channel of different terminal devices, adjust the order of the keyword index result in described inverted index file, for when terminal device is initiated request of access, preferentially provide and the type of described terminal device, the indexed results that channel relevance is large.
Because the channel of each access terminal equipment is different, need to provide differentiated service according to the feature of terminal device.First, terminal device divides and includes by type: mobile terminal, computing machine, intelligent television etc.The required data of these dissimilar terminal devices are different, and also difference to some extent of the service that obtains of expectation.For example, the fault-tolerance that intelligent television allows is minimum, and the fault-tolerance that mobile terminal and computing machine allow is relatively large.Be set to respectively Intelligent television terminal and provide some index servers of index service, some index servers of index service be provided, provide some index servers of index service for terminal for mobile terminal.By providing respectively index service according to the type of terminal device, can improve the speed of request of access, and improve user and experience.
In addition, the access service that terminal device may be used different operator's platforms to provide when accessing Internet, message transmission rate lower (for example, between telecommunications and UNICOM) between different operators, this is especially the most obvious for user's impression of accessing by wide band system.By the index server of multiple operator platform access is provided respectively, and provide respectively index service, make to ask to access fast processing by the user of different operator's platform access, thereby improved the speed of request of access, and improve user and experience.
Pass through above-described embodiment, after receiving the request of access of terminal device access inverted index file, judgement terminal device access channel, and provide corresponding index server that index service is provided according to the access channel of terminal device, thereby make user side obtain inverted index information by the index server corresponding with its access channel, thereby improved efficiency and the speed of request of access.
In one embodiment of the invention, need to carry out at any time the renewal of index information, article one, the new index information inserting will cause all index informations after it in inverted file all will move backward, due to the factor of time, can strengthen the cost of magnetic disc i/o operation when real-time update.In the present invention, according to the access channel of terminal device, corresponding update mode is set, and according to the update mode arranging, the updating file of described inverted index file is distributed to the index server corresponding with the access channel of terminal.For example, the mode of shorter or real-time update update time is set for the minimum intelligent television of fault-tolerance, for the higher computing machine of fault-tolerance or mobile device, longer update mode update time is set.By the update mode of this inverted index file, when meeting user search requirement, reduced operating cost.
In actual applications, when there is accident or focus sheet, show, the sudden increase of visit capacity meeting for these video access, at this moment needs to meet by dilatation server the access of burst.Particularly, record the quantity of the request of access of terminal device, when the quantity of the request of access for identical inverted index file surpasses predetermined threshold value, dilatation index server is provided, and corresponding inverted index file is sent to described dilatation index server, for the request of access of receiving terminal apparatus, the server of the index server of these dilatations and normal work before provides distributed index service.
According to embodiments of the invention, also provide a kind of inverted index storage system.
Fig. 2 is according to the structured flowchart of the inverted index storage system of the embodiment of the present invention, and as shown in Figure 2, described system at least comprises: set up module 10, memory module 20 and module 30 is set, describing structure and the annexation of each module below in detail.
Set up module 10, for setting up the inverted index file of video file.
With reference to figure 3, the described module 10 of setting up further comprises:
Inverted index is set up module 120, for setting up described keyword and having the index relative between the video file information of described keyword, thereby sets up the inverted index file of video file.
The described module 30 that arranges further comprises:
Continuation is with reference to figure 3, and described inverted index storage system also comprises: update module 40, logging modle 50 and administration module 60, wherein:
Logging modle 50, for recording the quantity of the request of access of terminal device;
The operation steps of method of the present invention is corresponding with the architectural feature of system, can cross-reference, repeat no longer one by one.
In sum, according to technical scheme of the present invention, by inverted index file synchronization is stored to a plurality of index servers, and according to the access channel of terminal device, corresponding index server is set respectively index service is provided, when receiving the request of access of terminal device access inverted index file, can corresponding index server retrieval be set according to the access channel of terminal device and obtain inverted index information, thereby improve efficiency and the speed of request of access.
The foregoing is only embodiments of the invention, be not limited to the present invention, for a person skilled in the art, the present invention can have various modifications and variations.Within the spirit and principles in the present invention all, any modification of doing, be equal to replacement, improvement etc., within all should being included in claim scope of the present invention.
Claims (10)
1. an inverted index storage method, is characterized in that, comprising:
Set up the inverted index file of video file;
A plurality of index servers are provided, described inverted index file synchronization is stored to a plurality of index servers, and according to the access channel of terminal device, corresponding index server is set respectively index service is provided.
2. method according to claim 1, is characterized in that, the described access channel according to terminal device arranges respectively corresponding index server provides index service, comprising:
According to the type of terminal device, corresponding index server is set respectively index service is provided; Or
The operator's platform using according to terminal device arranges respectively corresponding index server provides index service.
3. method according to claim 1, is characterized in that, also comprises:
Receive the updating file of inverted index file, according to the access channel of terminal device, use the update mode setting in advance that the updating file of described inverted index is distributed to corresponding index server.
4. method according to claim 1, is characterized in that, also comprises:
Record the quantity of the request of access of terminal device;
When the quantity of the request of access for same inverted index file surpasses predetermined threshold value, provide the request of access of dilatation index server for receiving terminal apparatus.
5. method according to claim 1, is characterized in that, the described inverted index file of setting up video file, comprising:
By default participle mode, video file information is carried out to word segmentation processing and obtain keyword;
Set up described keyword and there is the index relative between the video file information of described keyword, thereby set up the inverted index file of video file.
6. an inverted index storage system, is characterized in that, comprising:
Set up module, for setting up the inverted index file of video file;
Memory module, for being stored to described inverted index file synchronization a plurality of index servers that provide;
Module is set, for corresponding index server being set respectively according to the access channel of terminal device, provides index service.
7. system according to claim 6, is characterized in that, the described module that arranges comprises:
First arranges module, for corresponding index server being set respectively according to the type of terminal device, provides index service; Or
Second arranges module, for the operator's platform using according to terminal device, corresponding index server is set respectively index service is provided.
8. system according to claim 6, is characterized in that, also comprises:
Update module, for receiving after the updating file of inverted index file, is used the update mode setting in advance that the updating file of described inverted index is distributed to corresponding index server according to the access channel of terminal device.
9. system according to claim 6, is characterized in that, also comprises:
Logging modle, for recording the quantity of the request of access of terminal device;
Administration module, while surpassing predetermined threshold value for the quantity of the request of access when for same inverted index file, provides the request of access of dilatation index server for receiving terminal apparatus.
10. system according to claim 6, is characterized in that, the described module of setting up comprises:
Keyword acquisition module, carries out word segmentation processing for the participle mode by default to video file information and obtains keyword;
Inverted index is set up module, for setting up described keyword and having the index relative between the video file information of described keyword, thereby sets up the inverted index file of video file.
Priority Applications (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310741178.7A CN103678697A (en) | 2013-12-26 | 2013-12-26 | Reverse index storage method and system thereof |
| US15/101,698 US20160306811A1 (en) | 2013-12-26 | 2014-12-05 | Method and system for creating inverted index file of video resource |
| PCT/CN2014/093176 WO2015096609A1 (en) | 2013-12-26 | 2014-12-05 | Method and system for creating inverted index file of video resource |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310741178.7A CN103678697A (en) | 2013-12-26 | 2013-12-26 | Reverse index storage method and system thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103678697A true CN103678697A (en) | 2014-03-26 |
Family
ID=50316241
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310741178.7A Pending CN103678697A (en) | 2013-12-26 | 2013-12-26 | Reverse index storage method and system thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103678697A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015096609A1 (en) * | 2013-12-26 | 2015-07-02 | 乐视网信息技术（北京）股份有限公司 | Method and system for creating inverted index file of video resource |
| CN106302608A (en) * | 2015-06-08 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of information processing method and device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101004744A (en) * | 2006-01-17 | 2007-07-25 | 鸿富锦精密工业（深圳）有限公司 | System and method for synchronizing indexes of remote files at multiple points |
| CN101051309A (en) * | 2006-04-06 | 2007-10-10 | 中国科学院计算技术研究所 | Researching system and method used in digital labrary |
| CN101068235A (en) * | 2006-08-18 | 2007-11-07 | 腾讯科技（深圳）有限公司 | Video frequency program prebrowsing method and system |
| CN101071442A (en) * | 2007-06-26 | 2007-11-14 | 腾讯科技（深圳）有限公司 | Distributed indesx file searching method, searching system and searching server |
| US20110016128A1 (en) * | 2004-08-20 | 2011-01-20 | Forman George H | Distributing content indices |
| CN101963999A (en) * | 2010-11-03 | 2011-02-02 | 北京亮点时间科技有限公司 | Music classified search engine system and music classified search method |
| CN102999498A (en) * | 2011-09-08 | 2013-03-27 | 中兴通讯股份有限公司 | Method and device for searching multi-media programs |
-
2013
- 2013-12-26 CN CN201310741178.7A patent/CN103678697A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110016128A1 (en) * | 2004-08-20 | 2011-01-20 | Forman George H | Distributing content indices |
| CN101004744A (en) * | 2006-01-17 | 2007-07-25 | 鸿富锦精密工业（深圳）有限公司 | System and method for synchronizing indexes of remote files at multiple points |
| CN101051309A (en) * | 2006-04-06 | 2007-10-10 | 中国科学院计算技术研究所 | Researching system and method used in digital labrary |
| CN101068235A (en) * | 2006-08-18 | 2007-11-07 | 腾讯科技（深圳）有限公司 | Video frequency program prebrowsing method and system |
| CN101071442A (en) * | 2007-06-26 | 2007-11-14 | 腾讯科技（深圳）有限公司 | Distributed indesx file searching method, searching system and searching server |
| CN101963999A (en) * | 2010-11-03 | 2011-02-02 | 北京亮点时间科技有限公司 | Music classified search engine system and music classified search method |
| CN102999498A (en) * | 2011-09-08 | 2013-03-27 | 中兴通讯股份有限公司 | Method and device for searching multi-media programs |
Non-Patent Citations (1)
| Title |
|---|
| 燕昆: "分布式搜索引擎索引安全及缓存策略研究", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015096609A1 (en) * | 2013-12-26 | 2015-07-02 | 乐视网信息技术（北京）股份有限公司 | Method and system for creating inverted index file of video resource |
| CN106302608A (en) * | 2015-06-08 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of information processing method and device |
| WO2016197839A3 (en) * | 2015-06-08 | 2017-02-09 | 阿里巴巴集团控股有限公司 | Information processing method and device |
| US10491707B2 (en) | 2015-06-08 | 2019-11-26 | Alibaba Group Holding Limited | Information processing using a server group |
| CN106302608B (en) * | 2015-06-08 | 2020-02-04 | 阿里巴巴集团控股有限公司 | Information processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103678694A (en) | Method and system for establishing reverse index file of video resources | |
| CN103488687A (en) | Searching system and searching method of big data | |
| CN102890714B (en) | Method and device for indexing data | |
| CN113609374A (en) | Data processing method, device and equipment based on content push and storage medium | |
| CN102902826B (en) | A kind of image method for quickly retrieving based on reference picture index | |
| CN106294695A (en) | A kind of implementation method towards the biggest data search engine | |
| WO2015096609A1 (en) | Method and system for creating inverted index file of video resource | |
| CN104252536A (en) | Hbase-based internet log data inquiring method and device | |
| CN104239377A (en) | Platform-crossing data retrieval method and device | |
| CN104657435A (en) | Storage management method for application data and network management system | |
| CN103744913A (en) | Database retrieval method based on search engine technology | |
| CN103853838B (en) | A kind of data processing method and device | |
| CN112597348A (en) | Method and device for optimizing big data storage | |
| CN111782663A (en) | Aggregation index structure and aggregation index method for improving aggregation query efficiency | |
| CN103200269A (en) | Internet information statistical method and Internet information statistical system | |
| CN116301656A (en) | Data storage method, system and equipment based on log structure merging tree | |
| CN106021357A (en) | Distribution-based big data paging query method and system | |
| KR101955376B1 (en) | Processing method for a relational query in distributed stream processing engine based on shared-nothing architecture, recording medium and device for performing the method | |
| CN111611222A (en) | Data dynamic processing method based on distributed storage | |
| CN103345527B (en) | Intelligent data statistical system | |
| CN110109870A (en) | A kind of mass data quick retrieval system based on Solr | |
| CN103714158A (en) | Vertical search method and system for video websites | |
| CN113254480A (en) | Data query method and device | |
| CN103678697A (en) | Reverse index storage method and system thereof | |
| CN102831181B (en) | Directory refreshing method for cache files |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C41 | Transfer of patent application or patent right or utility model | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20151225 Address after: Room six, building 19, building 68, No. 100089 South Road, Haidian District, Beijing Applicant after: LETV CLOUD COMPUTING CO., LTD. Address before: Room six, building 19, building 68, No. 100089 South Road, Haidian District, Beijing Applicant before: LeTV Information Technology (Beijing) Co., Ltd. |
|
| AD01 | Patent right deemed abandoned |
Effective date of abandoning: 20180515 |
|
| AD01 | Patent right deemed abandoned |