CN103731369A - Method and system for updating data queue in server - Google Patents
Method and system for updating data queue in server Download PDFInfo
- Publication number
- CN103731369A CN103731369A CN201310741616.XA CN201310741616A CN103731369A CN 103731369 A CN103731369 A CN 103731369A CN 201310741616 A CN201310741616 A CN 201310741616A CN 103731369 A CN103731369 A CN 103731369A
- Authority
- CN
- China
- Prior art keywords
- data file
- queue
- data
- internal queue
- local directory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 23
- 230000002159 abnormal effect Effects 0.000 claims description 16
- 238000001514 detection method Methods 0.000 claims description 13
- 230000004048 modification Effects 0.000 claims description 5
- 238000012986 modification Methods 0.000 claims description 5
- 230000005540 biological transmission Effects 0.000 abstract description 8
- 230000006870 function Effects 0.000 description 5
- 230000008878 coupling Effects 0.000 description 4
- 238000010168 coupling process Methods 0.000 description 4
- 238000005859 coupling reaction Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method and system for updating a data queue in a server. According to the method and system, transmission errors generated when data files are written in a magnetic disk array can be avoided. The method comprises the steps that whether transaction happens to the data files in a local catalog or not is detected; if it is detected that the data files are newly added to the local catalog, an internal queue is established for the newly-added data files, and the internal queue is designated to record path information of the data files; if it is detected that the data files in the local catalog are modified, the path information in the queue is updated according to the path information of the data files; if it is detected that the data files in the local catalog are deleted, the internal queue corresponding to the data files is deleted.
Description
Technical Field
The present application relates to a method and a system for data transmission, and more particularly, to a method and a system for updating a data queue in a server.
Background
With the rapid growth of network traffic, many services that were not easily implemented in the past can be easily accomplished. For example: broadband uploading of multimedia files (multimedia files). The load of the server is still affected due to the massive transmission. Therefore, the known multimedia server also has a Bandwidth management (Bandwidth Manager) function. The server can reduce the data transmission amount at the same time through bandwidth distribution or cache.
In addition, the transmission mode is combined with a plurality of physical disks, so that data can be synchronously written into the disks. In the prior art, the data is written into the disk by a source end in synchronization with the data. The server temporarily stores the data files. Then, the server writes the data files into the physical disk in batches.
This way of writing is cheap and fast to implement. However, if an error occurs during the data writing process, the data can only be rewritten. The data retransmission time is lengthened and the load of the network bandwidth is increased. In addition, the same data file is reloaded from the client and also retransmitted within the server.
And the data file is written into the distributed file device in a buffer mode. All data files are therefore placed in the same cache. This can cause confusion in the retrieval of the data file. In particular, if the pointers of different types of data files are pointing incorrectly, the two data files will be damaged.
Disclosure of Invention
The technical problem to be solved by the present application is to provide a method for updating a data queue in a server, so as to avoid transmission errors occurring when data files are divided or written into a disk array.
In order to solve the above problem, the present application discloses a method for updating a data queue in a server, including: detecting whether the data files of the local directory have abnormal changes or not; if the fact that the data file is newly added to the local directory is detected, an internal queue is created for the newly added data file, and the internal queue is assigned to record path information of the data file; if the data file of the local directory is detected to be modified, updating the path information in the internal queue according to the path information of the data file; and if the data file of the local directory is detected to be deleted, deleting the internal queue corresponding to the data file.
Further, the detecting whether there is a transaction in the data file of the local directory includes: recording the creation information of the internal queue and the data file, wherein the creation information comprises the receiving time of the data file and the file length of the data file; scanning whether the data file is consistent with the creating information at regular time; and if the two are not consistent, determining that the data file of the local directory has transaction.
Further, the detecting whether there is a transaction in the data file of the local directory includes: if the data file read this time does not have the relevant record of the data file, determining that the data file is abnormal; if the receiving time of the data file read this time is not consistent with the receiving time of the data file read last time and/or the file length of the data file, determining that there is a transaction for modifying the data file; and if the data file recorded last time does not exist during the reading of the data file, determining that there is abnormal action for deleting the data file.
Further, if it is detected that the data file of the local directory is modified, searching an internal queue corresponding to the data file according to the path information of the modified data file, and updating the path information in the searched internal queue; and if the data file of the local directory is detected to be deleted, searching an internal queue corresponding to the deleted data file according to the path information of the deleted data file, and deleting the searched internal queue.
In order to solve the above problem, the present application further discloses a data slicing transmission system, including: the detection module is used for detecting whether the data files of the local directory have abnormal changes; the queue management module is connected to the detection module and used for creating an internal queue for the newly added data file and assigning the internal queue to record the path information of the data file if the detection module detects that the local directory has the newly added data file; the queue management module is further configured to update path information in an internal queue according to the path information of the data file if the detection module detects that the data file of the local directory is modified; and if the detection module detects that the data file of the local directory is deleted, deleting the internal queue corresponding to the data file.
Further, the detecting module further comprises: the device comprises a recording unit, a scanning unit and a determining unit; the recording unit is used for recording the internal queue and the creation information of the data file, wherein the creation information comprises the receiving time of the data file and the file length of the data file; the scanning unit is used for scanning whether the data file is consistent with the creating information at regular time; and the determining unit is used for determining that the data file of the local directory has transaction if the two are not consistent.
Further, the determining unit is specifically configured to: if the data file read this time does not have the relevant record of the data file, determining that the data file is updated; if the receiving time of the data file read this time is not consistent with the receiving time of the data file read last time and/or the file length of the data file, determining that there is a transaction for modifying the data file; and if the data file recorded last time does not exist during the reading of the data file, determining that there is abnormal action for deleting the data file.
Further, the determining unit further includes: the internal queue modification subunit is used for searching the internal queue corresponding to the data file according to the path information of the modified data file and updating the path information in the searched internal queue if the data file of the local directory is modified; and the internal queue deleting subunit is used for searching the internal queue corresponding to the deleted data file according to the path information of the deleted data file and deleting the searched internal queue if the data file of the local directory is detected to be deleted.
Compared with the prior art, the application can obtain the following technical effects:
1) according to the method and the device, the transmission process of the data file is improved, and if errors occur during writing of the data file, the data file does not need to be obtained from the client again.
2) If errors occur in the period of transmitting the data file, the method and the device can transmit the error to the alarm server through at least one heartbeat server, so that background personnel can know which component is abnormal in real time.
Of course, it is not necessary for any one product to achieve all of the above-described technical effects simultaneously.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
fig. 1 is a schematic architecture diagram of an embodiment of the present application.
Fig. 2 is a schematic operation flow chart according to an embodiment of the present application.
Detailed Description
Embodiments of the present application will be described in detail with reference to the drawings and examples, so that how to implement technical means to solve technical problems and achieve technical effects of the present application can be fully understood and implemented.
As used in the specification and in the claims, certain terms are used to refer to particular components. As one skilled in the art will appreciate, manufacturers may refer to a component by different names. This specification and claims do not intend to distinguish between components that differ in name but not function. In the following description and in the claims, the terms "include" and "comprise" are used in an open-ended fashion, and thus should be interpreted to mean "include, but not limited to. "substantially" means within an acceptable error range, and a person skilled in the art can solve the technical problem within a certain error range to substantially achieve the technical effect. Furthermore, the term "coupled" is intended to encompass any direct or indirect electrical coupling. Thus, if a first device couples to a second device, that connection may be through a direct electrical coupling or through an indirect electrical coupling via other devices and couplings. The description which follows is a preferred embodiment of the present application, but is made for the purpose of illustrating the general principles of the application and not for the purpose of limiting the scope of the application. The protection scope of the present application shall be subject to the definitions of the appended claims.
Please refer to fig. 1 for a data queue updating system in a server according to the present application. The update system 100 of the present application is composed of the following elements: a memory module 110, a detection module 120 and a queue management module 130. The queue management module 130 is connected to the detection module 120 and the memory module 110. The memory module 110 is configured to receive a data file uploaded by a client. The memory module 110 has a plurality of local directories, and each local directory stores a respective data file. The detecting module 120 is used for detecting whether there is a transaction in the data file of the local directory. The queue management module 130 is configured to create an internal queue for the newly added data file and assign the internal queue to record path information of the data file if the detecting module 120 detects that the local directory has the newly added data file. The queue management module 130 is further configured to update the path information in the internal queue according to the path information of the data file if the detection module 120 detects that the data file of the local directory is modified. If the detecting module 120 detects that the data file of the local directory is deleted, the internal queue corresponding to the data file is deleted.
The detecting module 120 of the present application further includes a recording unit 121, a scanning unit 122, a determining unit 123, an internal queue modifying subunit 1231, and an internal queue deleting subunit 1232. The recording unit 121 is configured to record the internal queue and creation information of the data file, where the creation information includes a receiving time of the data file and a file length of the data file. The scanning unit 122 is used for regularly scanning whether the data file is consistent with the creation information. The determining unit 123 is configured to determine that the data file of the local directory has an abnormality if the two are inconsistent.
Specifically, if the data file read this time does not have a record related to the data file, the determining unit 123 determines that there is a transaction of the newly added data file. If the receiving time of the data file read this time is not consistent with the receiving time of the data file read last time and/or the file length of the data file, the determining unit 123 determines that there is an abnormal change for modifying the data file. If the data file recorded last time does not exist during the current reading, the determining unit 123 determines that there is an abnormal action of deleting the data file.
The internal queue modification subunit 1231 is configured to, if it is detected that the data file of the local directory is modified, search the internal queue corresponding to the data file according to the modified path information of the data file, and update the path information in the searched internal queue. The internal queue deleting subunit 1232 is configured to, if it is detected that the data file of the local directory is deleted, search the internal queue corresponding to the deleted data file according to the path information of the deleted data file, and delete the searched internal queue.
The operation of the above components is represented by the following steps, please refer to fig. 2:
step S210: detecting whether the data files of the local directory have abnormal changes or not;
step S220: if the fact that the data file is newly added to the local directory is detected, an internal queue is created for the newly added data file, and the internal queue is assigned to record path information of the data file;
step S230: if the data file of the local directory is detected to be modified, updating the path information in the internal queue according to the path information of the data file; and
step S240: and if the data file of the local directory is detected to be deleted, deleting the internal queue corresponding to the data file.
The step of detecting whether the data file of the local directory has the transaction further comprises the following steps:
step S310: recording the creation information of the internal queue and the data file, wherein the creation information comprises the receiving time of the data file and the file length of the data file;
step S320: scanning whether the data file is consistent with the creating information at regular time;
step S330: and if the two are not consistent, determining that the data file of the local directory has transaction.
In step S310, the file operation function may be called periodically to read the data file of the local directory, and record the creation information such as the file name, the receiving time of the data file, and the file length of the data file. And the read-write operation of the file operation function on the local directory can be monitored at regular time, so that file transaction information is obtained.
In step S320 and S330, the recorded document creation information is scanned at regular time, and whether there is a document transaction is determined by comparing the previously recorded document creation information with the currently read document creation information. If the file creation information (such as any one of the file name, the data receiving time and the file length of the data) obtained by comparing the two times of reading is inconsistent, the data file transaction is considered to exist.
In the above steps S320 and S330: if the data file read this time does not have the relevant record of the data file, determining that the data file is abnormal;
if the receiving time of the data file read this time is not consistent with the receiving time of the data file read last time and/or the file length of the data file read last time (for example, the receiving time of the data file and/or the file length is changed), determining that there is a transaction for modifying the data file;
if the data file recorded last time does not exist during the reading of this time, it is determined that there is a transaction for deleting the data file (i.e., the name of the data file is recorded during the reading of the last time, and the name of the data file is not found during the comparison in the reading of this time).
When the data file is changed, searching an internal queue corresponding to the data file according to the modified path information of the data file, and updating the path information in the searched internal queue; when the abnormal change that the data file of the local directory is deleted is detected, searching the internal queue corresponding to the deleted data file according to the path information of the deleted data file, and deleting the searched internal queue. In specific implementation, the internal queue corresponding to the data file can be determined by comparing the path information stored in the internal queue; or when the internal queue is created, a queue index file is created at the same time, which is used for storing the corresponding relation between the data file and the internal queue. The path information of the data file may include a path of the data file stored in the local directory, and is generally obtained through an interface function of the system; it may also include the path stored in the end server, and the general name includes the storage path information of the file, such as the IP address of the server and the specific storage disk location on the server.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (8)
1. A method for updating a data queue in a server, comprising the steps of:
detecting whether the data files of the local directory have abnormal changes or not;
if the fact that the data file is newly added to the local directory is detected, an internal queue is created for the newly added data file, and the internal queue is assigned to record path information of the data file;
if the data file of the local directory is detected to be modified, updating the path information in the internal queue according to the path information of the data file;
and deleting the internal queue corresponding to the data file if detecting that the data file of the local directory is deleted.
2. The method for updating a data queue in a server according to claim 1, wherein detecting whether there is a transaction in a data file of a local directory further comprises:
recording the creation information of the internal queue and the data file, wherein the creation information comprises the receiving time of the data file and the file length of the data file;
scanning whether the data file is consistent with the creating information at regular time; and
and if the two are not consistent, determining that the data file of the local directory has transaction.
3. The method for updating a data queue in a server according to claim 2, wherein detecting whether there is a transaction in a data file of a local directory further comprises:
if the data file read this time does not have the relevant record of the data file, determining that the data file is abnormal;
if the receiving time of the data file read this time is not consistent with the receiving time of the data file read last time and/or the file length of the data file, determining that there is a transaction for modifying the data file;
and if the data file recorded last time does not exist during the reading of the data file, determining that there is abnormal action for deleting the data file.
4. The method for updating a data queue in a server according to claim 1, wherein if it is detected that the data file of the local directory is modified, searching an internal queue corresponding to the data file according to the path information of the modified data file, and updating the path information in the searched internal queue;
and if detecting that the data file of the local directory is deleted, searching an internal queue corresponding to the deleted data file according to the path information of the deleted data file, and deleting the searched internal queue.
5. An apparatus for updating a data queue in a server, comprising:
the detection module is used for detecting whether the data files of the local directory have abnormal changes; and the number of the first and second groups,
the queue management module is connected to the detection module and used for creating an internal queue for the newly added data file and assigning the internal queue to record the path information of the data file if the detection module detects that the local directory has the newly added data file;
the queue management module is further configured to update path information in an internal queue according to the path information of the data file if the detection module detects that the data file of the local directory is modified;
and if the detection module detects that the data file of the local directory is deleted, deleting the internal queue corresponding to the data file.
6. The system for updating a data queue in a server of claim 5, wherein the detecting module further comprises: the device comprises a recording unit, a scanning unit and a determining unit; wherein,
the recording unit is used for recording the creation information of the internal queue and the data file, and the creation information comprises the receiving time of the data file and the file length of the data file;
the scanning unit is used for scanning whether the data file is consistent with the creating information at regular time; and the number of the first and second groups,
and the determining unit is used for determining that the data file of the local directory has transaction if the two are not consistent.
7. The system for updating a data queue in a server according to claim 6, wherein the determining unit is specifically configured to:
if the data file read this time does not have the relevant record of the data file, determining that the data file is updated;
if the receiving time of the data file read this time is not consistent with the receiving time of the data file read last time and/or the file length of the data file, determining that there is a transaction for modifying the data file;
and if the data file recorded last time does not exist during the reading of the data file, determining that there is abnormal action for deleting the data file.
8. The system for updating a data queue in a server according to claim 5, wherein the determining unit further comprises:
the internal queue modification subunit is used for searching the internal queue corresponding to the data file according to the path information of the modified data file and updating the path information in the searched internal queue if the data file of the local directory is modified;
and the internal queue deleting subunit is used for searching the internal queue corresponding to the deleted data file according to the path information of the deleted data file and deleting the searched internal queue if the data file of the local directory is detected to be deleted.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310741616.XA CN103731369A (en) | 2013-12-27 | 2013-12-27 | Method and system for updating data queue in server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310741616.XA CN103731369A (en) | 2013-12-27 | 2013-12-27 | Method and system for updating data queue in server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103731369A true CN103731369A (en) | 2014-04-16 |
Family
ID=50455301
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310741616.XA Pending CN103731369A (en) | 2013-12-27 | 2013-12-27 | Method and system for updating data queue in server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103731369A (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH1091508A (en) * | 1996-09-12 | 1998-04-10 | Toshiba Corp | Preserved data updating device |
| US20070061540A1 (en) * | 2005-06-06 | 2007-03-15 | Jim Rafert | Data storage system using segmentable virtual volumes |
| CN101237595A (en) * | 2008-02-25 | 2008-08-06 | 中兴通讯股份有限公司 | Data load method |
| CN101902473A (en) * | 2010-07-22 | 2010-12-01 | 武汉中地数码科技有限公司 | Realization Method of Data Synchronous Update Based on Grid GIS |
| CN102004778A (en) * | 2010-11-19 | 2011-04-06 | 清华大学 | Text index online updating method in cloud environment |
| US8150837B2 (en) * | 2004-06-25 | 2012-04-03 | Apple Inc. | Methods and systems for managing data |
| CN102467412A (en) * | 2010-11-16 | 2012-05-23 | 金蝶软件（中国）有限公司 | Method, device and business system for processing operation request |
| US20120167111A1 (en) * | 2010-12-23 | 2012-06-28 | Microsoft Corporation | Resource deployment based on conditions |
| CN102779185A (en) * | 2012-06-29 | 2012-11-14 | 浙江大学 | High-availability distribution type full-text index method |
| CN103020315A (en) * | 2013-01-10 | 2013-04-03 | 中国人民解放军国防科学技术大学 | Method for storing mass of small files on basis of master-slave distributed file system |
-
2013
- 2013-12-27 CN CN201310741616.XA patent/CN103731369A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH1091508A (en) * | 1996-09-12 | 1998-04-10 | Toshiba Corp | Preserved data updating device |
| US8150837B2 (en) * | 2004-06-25 | 2012-04-03 | Apple Inc. | Methods and systems for managing data |
| US20070061540A1 (en) * | 2005-06-06 | 2007-03-15 | Jim Rafert | Data storage system using segmentable virtual volumes |
| CN101237595A (en) * | 2008-02-25 | 2008-08-06 | 中兴通讯股份有限公司 | Data load method |
| CN101902473A (en) * | 2010-07-22 | 2010-12-01 | 武汉中地数码科技有限公司 | Realization Method of Data Synchronous Update Based on Grid GIS |
| CN102467412A (en) * | 2010-11-16 | 2012-05-23 | 金蝶软件（中国）有限公司 | Method, device and business system for processing operation request |
| CN102004778A (en) * | 2010-11-19 | 2011-04-06 | 清华大学 | Text index online updating method in cloud environment |
| US20120167111A1 (en) * | 2010-12-23 | 2012-06-28 | Microsoft Corporation | Resource deployment based on conditions |
| CN102779185A (en) * | 2012-06-29 | 2012-11-14 | 浙江大学 | High-availability distribution type full-text index method |
| CN103020315A (en) * | 2013-01-10 | 2013-04-03 | 中国人民解放军国防科学技术大学 | Method for storing mass of small files on basis of master-slave distributed file system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9715507B2 (en) | Techniques for reconciling metadata and data in a cloud storage system without service interruption | |
| CN110162525B (en) | B+ tree-based read-write conflict resolution method, device and storage medium | |
| US11687488B2 (en) | Directory deletion method and apparatus, and storage server | |
| US8762353B2 (en) | Elimination of duplicate objects in storage clusters | |
| US9575974B2 (en) | Distributed file system gateway | |
| CN107291710B (en) | Method and device for updating data for distributed database system | |
| CN106951375B (en) | Method and device for deleting snapshot volume in storage system | |
| US10481988B2 (en) | System and method for consistency verification of replicated data in a recovery system | |
| US10013312B2 (en) | Method and system for a safe archiving of data | |
| US20140181395A1 (en) | Virtual tape library system | |
| US20210303566A1 (en) | Data storage using a bi-temporal index | |
| WO2021249201A1 (en) | Monitoring data storage method and apparatus based on shingled magnetic recording disk | |
| US11455117B2 (en) | Data reading method, apparatus, and system, avoiding version rollback issues in distributed system | |
| JP2012089094A5 (en) | ||
| CN103631603A (en) | Software cleaning method and system | |
| CN109947730B (en) | Metadata recovery method, device, distributed file system and readable storage medium | |
| CN118689883A (en) | A method and device for updating supervision object data | |
| US11645333B1 (en) | Garbage collection integrated with physical file verification | |
| US20100010644A1 (en) | System and method for managing applications | |
| CN103731369A (en) | Method and system for updating data queue in server | |
| CN111381774B (en) | System and method for archiving data in a decentralized data protection system | |
| CN103701667A (en) | Method, device and system for monitoring heartbeat of server | |
| CN109766462B (en) | Image file reading method, device and system in power transmission line monitoring system | |
| CN110019233B (en) | Data storage method and system | |
| CN111209304A (en) | Data processing method, device and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C41 | Transfer of patent application or patent right or utility model | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20151225 Address after: Room six, building 19, building 68, No. 100089 South Road, Haidian District, Beijing Applicant after: LETV CLOUD COMPUTING CO., LTD. Address before: Room six, building 19, building 68, No. 100089 South Road, Haidian District, Beijing Applicant before: LeTV Information Technology (Beijing) Co., Ltd. |
|
| AD01 | Patent right deemed abandoned |
Effective date of abandoning: 20180112 |
|
| AD01 | Patent right deemed abandoned |