[go: up one dir, main page]

CN116303275A - Service system and method for distributed files of electric power system industrial internet platform - Google Patents

Service system and method for distributed files of electric power system industrial internet platform Download PDF

Info

Publication number
CN116303275A
CN116303275A CN202211711519.1A CN202211711519A CN116303275A CN 116303275 A CN116303275 A CN 116303275A CN 202211711519 A CN202211711519 A CN 202211711519A CN 116303275 A CN116303275 A CN 116303275A
Authority
CN
China
Prior art keywords
file
library
metadata
verification
upload
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211711519.1A
Other languages
Chinese (zh)
Inventor
陈鹏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NR Electric Co Ltd
NR Engineering Co Ltd
Original Assignee
NR Electric Co Ltd
NR Engineering Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NR Electric Co Ltd, NR Engineering Co Ltd filed Critical NR Electric Co Ltd
Priority to CN202211711519.1A priority Critical patent/CN116303275A/en
Publication of CN116303275A publication Critical patent/CN116303275A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/16File or folder operations, e.g. details of user interfaces specifically adapted to file systems
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/17Details of further file system functions
    • G06F16/172Caching, prefetching or hoarding of files
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/182Distributed file systems
    • GPHYSICS
    • G06COMPUTING OR CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/30Authentication, i.e. establishing the identity or authorisation of security principals
    • G06F21/31User authentication

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Computer Security & Cryptography (AREA)
  • Human Computer Interaction (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The system comprises a basic service module and a progressive service module which adopt a restful interface based on an HTTPS protocol, wherein the basic service module provides file uploading, file downloading, file deleting and file inquiring services; the advanced service module provides file metadata storage, subsection uploading and downloading, file master-slave synchronization, file read-write conflict resolution scheme and file authority management service so as to assist in realizing various services of the basic service module. The mass files of the industrial Internet platform of the power system are managed, so that the requirements of large quantity of the industrial Internet files of the power system, high requirement on real-time file access and attention to system stability and universality can be met.

Description

电力系统工业互联网平台分布式文件的服务系统和方法Service system and method for distributed files of electric power system industrial Internet platform

技术领域technical field

本发明属电力系统文件管理技术领域,涉及电力系统工业互联网平台分布式文件的服务系统和方法。The invention belongs to the technical field of electric power system file management, and relates to a service system and method for distributed files on an electric power system industrial internet platform.

背景技术Background technique

随着电力系统工业互联网的不断发展,与之相关的数据量也日趋庞大,对于海量数据和文件的存储要求也越来越严格。因此亟需一种通用的文件服务,来满足电力系统工业互联网平台海量文件的存储需要,提高电力系统文件存储的稳定性,保证文件数据访问的实时性,同时具备一定的容灾能力。With the continuous development of the power system industrial Internet, the amount of data related to it is also increasing, and the storage requirements for massive data and files are becoming more and more stringent. Therefore, a general-purpose file service is urgently needed to meet the storage needs of massive files on the power system industrial Internet platform, improve the stability of power system file storage, ensure real-time access to file data, and have certain disaster recovery capabilities.

基于FTP协议的传统文件服务,虽然可以提供可靠高效的文件传输服务,但是由于FTP协议的固有缺陷,存在一定的被恶意攻击的可能和风险。而现有开源的分布式文件服务也各有利弊,不完全符合电力系统工业互联网的需要。例如,HDFS具备大文件传输、大吞吐量的优势,但是面对大量小文件、实时文件访问却无能为力;TFS针对小文件量身定做,随机读取性能比较高,但是不适合大文件存储,不支持自定义目录结构与文件权限控制;FastDFS具有优秀的并发处理能力及数据容错恢复能力,支持文件主备存储,但是不适于大文件场景,不具备断点续传功能。Although the traditional file service based on the FTP protocol can provide reliable and efficient file transfer services, due to the inherent defects of the FTP protocol, there is a certain possibility and risk of malicious attacks. The existing open source distributed file services also have their own advantages and disadvantages, which do not fully meet the needs of the industrial Internet of power systems. For example, HDFS has the advantages of large file transfer and high throughput, but it is helpless in the face of a large number of small files and real-time file access; TFS is tailored for small files, and its random read performance is relatively high, but it is not suitable for large file storage. Support custom directory structure and file permission control; FastDFS has excellent concurrent processing capabilities and data fault-tolerant recovery capabilities, and supports primary and backup storage of files, but it is not suitable for large file scenarios and does not have the function of resuming uploads from breakpoints.

发明内容Contents of the invention

为解决现有技术中存在的不足,本发明提供电力系统工业互联网平台分布式文件的服务系统和方法,对电力系统工业互联网平台的海量文件进行管理,具有文件上传、文件下载、文件删除、文件查询、文件冗余备份、文件断点续传、权限管理等文件管理功能,可满足电力系统工业互联网文件数量大、文件访问实时性要求高、关注系统稳定性和通用性的需要,文件上传后由文件服务统一管理,使得平台开发人员不需要关注文件管理相关繁杂的需求,降低工业互联网平台的运维成本。In order to solve the deficiencies in the prior art, the present invention provides a service system and method for distributed files on the power system industrial Internet platform, which manages massive files on the power system industrial Internet platform, and has the functions of file upload, file download, file deletion, file File management functions such as query, file redundancy backup, file breakpoint resume, and authority management can meet the needs of the power system industrial Internet with a large number of files, high real-time file access requirements, and attention to system stability and versatility. Unified management by the file service, so that platform developers do not need to pay attention to the complicated requirements related to file management, reducing the operation and maintenance costs of the industrial Internet platform.

本发明采用如下的技术方案。The present invention adopts the following technical solutions.

电力系统工业互联网平台分布式文件的服务系统,包括采用基于HTTPS协议的restful接口的基础服务模块和进阶服务模块;The distributed file service system of the power system industrial Internet platform, including the basic service module and advanced service module using the restful interface based on the HTTPS protocol;

所述基础服务模块包括文件上传单元、文件下载单元、文件删除单元和文件查询单元,分别提供文件上传、文件下载、文件删除和文件查询服务;The basic service module includes a file upload unit, a file download unit, a file deletion unit and a file query unit, which provide file upload, file download, file deletion and file query services respectively;

所述进阶服务模块包括文件元数据存储单元、分段上传下载单元、文件主备同步单元、文件读写冲突解决方案单元、文件权限管理单元,分别提供文件元数据存储、分段上传下载、文件主备同步、文件读写冲突解决方案、文件权限管理服务,以辅助实现基础服务模块的各项服务。The advanced service module includes a file metadata storage unit, a segment upload and download unit, a file master/slave synchronization unit, a file read/write conflict solution unit, and a file authority management unit, which provide file metadata storage, segment upload and download, File master-backup synchronization, file read-write conflict resolution, file rights management services, to assist in the realization of various services of the basic service module.

优选地,所述文件元数据存储单元,用于将文件存储到磁盘,将文件元数据存储到缓存库和关系库中;Preferably, the file metadata storage unit is configured to store the file on a disk, and store the file metadata in a cache repository and a relation repository;

所述文件分段上传下载单元,用于通过文件分段的方式,实现大文件上传下载、文件并行上传下载、分段断点续传;The file segmentation uploading and downloading unit is used to realize uploading and downloading of large files, parallel uploading and downloading of files, and resumed transmission of segmented breakpoints by means of file segmentation;

所述文件主备同步单元,用于实现主备机同步,主备机冗余双活;The file master-standby synchronization unit is used to realize master-standby machine synchronization, master-standby machine redundancy dual-active;

所述读写冲突解决方案单元,用于采用分布式锁的方案,使用文件信息作为锁秘钥,在文件读写前进行加锁,以解决单机读写问题和主备机间的读写冲突;The read-write conflict solution unit is used to adopt a distributed lock solution, using file information as a lock key, and locking the file before reading and writing, so as to solve the read-write problem of a single machine and the read-write conflict between the master and standby machines ;

所述文件权限管理单元,用于基于token令牌方式进行权限校验。The file authority management unit is used for performing authority verification based on a token token.

优选地,所述文件主备同步单元在文件增删时,主机文件服务通过消息队列发送消息给备机,备机同步更新文件信息,同时基于消息队列消息存储的功能,文件服务具备容灾能力,备机宕机重启后,能从消息队列拉取未同步的文件信息,进行文件数据同步。Preferably, when the file master-standby synchronization unit adds or deletes files, the host file service sends a message to the backup machine through the message queue, and the backup machine synchronously updates the file information. At the same time, based on the message storage function of the message queue, the file service has disaster recovery capabilities. After the standby machine is down and restarted, it can pull unsynchronized file information from the message queue to synchronize file data.

优选地,所述文件权限管理单元为每个用户配置可访问的路径列表,进行权限管理,用户获取token后,在请求文件服务时,请求头携带相应token,由文件服务解析token完成权限校验。Preferably, the file authority management unit configures an accessible path list for each user to perform authority management. After the user obtains the token, when requesting the file service, the request header carries the corresponding token, and the file service parses the token to complete the authority verification .

电力系统工业互联网平台分布式文件的服务方法,包括:The service method of the distributed file of the power system industrial internet platform, including:

基于基础服务模块进行文件上传、文件下载、文件删除和文件查询,同时,进阶服务模块在基础服务模块进行文件上传、文件下载、文件删除和文件查询时进行文件元数据存储、分段上传下载、文件主备同步、文件读写冲突解决方案、文件权限管理辅助。File upload, file download, file deletion, and file query are performed based on the basic service module. At the same time, the advanced service module performs file metadata storage, segmented upload and download when the basic service module performs file upload, file download, file deletion, and file query. , File master and backup synchronization, file read and write conflict resolution, file permission management assistance.

优选地,所述文件上传是指通过接口上传文件相关数据,且在进行文件上传时,首先进行校验,校验通过后,将文件元数据存入库中,并发送同步上传信息至备机,进行主备文件同步;Preferably, the file upload refers to uploading file-related data through an interface, and when uploading a file, first perform verification, after the verification is passed, store the file metadata in the library, and send synchronous upload information to the standby machine , to synchronize the master and backup files;

所述校验包括权限校验、读写冲突校验及参数校验;The verification includes permission verification, read-write conflict verification and parameter verification;

所述将文件元数据存入库中的方式为:先擦除缓存库,后写入关系库的方案,防止出现双库不一致情况。The method of storing the file metadata in the database is as follows: firstly erase the cache database, and then write into the relationship database, so as to prevent the inconsistency between the two databases.

优选地,所述文件上传具体过程包括:Preferably, the specific process of uploading files includes:

(1)波形文件导入:上传电力系统波形文件,选择上传路径、是否覆盖、重命名信息,发送上传请求,请求头携带token;(1) Waveform file import: upload the power system waveform file, select the upload path, whether to overwrite, rename the information, send the upload request, and the request header carries the token;

(2)权限检验:根据请求头携带的token进行解析,判断当前请求是否有权限向相应路径导入波形文件,如果无权限,进入失败流程,打印错误日志,结束流程,否则进入(3);(2) Permission inspection: analyze according to the token carried in the request header, and judge whether the current request has permission to import the wave file to the corresponding path. If there is no permission, enter the failure process, print the error log, and end the process, otherwise enter (3);

(3)读写冲突校验:当前请求尝试获得当前波形文件的分布式锁,如果获取成功,说明没有读写冲突,进入(4),若获取失败,则进入失败流程,打印错误日志,结束流程;(3) Read-write conflict check: The current request tries to obtain the distributed lock of the current waveform file. If the acquisition is successful, it means that there is no read-write conflict. Go to (4). If the acquisition fails, enter the failure process, print the error log, and end process;

(4)参数校验:进行基本参数校验、录波文件格式校验,并且查询库中是否已存在当前波形文件,根据是否允许文件覆盖写,分别进入成功流程和失败流程,查询库时,采用先查缓存库,后查关系库的方式;(4) Parameter verification: perform basic parameter verification, wave recording file format verification, and query whether the current waveform file already exists in the library, and enter the success process and failure process respectively according to whether file overwriting is allowed. When querying the library, Use the method of checking the cache library first, and then checking the relational library;

(5)写入数据库:根据波形文件对应的关键字,删除缓存库文件元数据,之后将文件元数据写入关系库;(5) Write into the database: delete the file metadata of the cache library according to the keyword corresponding to the wave file, and then write the file metadata into the relational library;

(6)发送同步消息:将波形文件元数据作为同步信息,通过消息队列发送文件上传同步消息至备机,备机从消息队列接受同步消息后,根据文件元数据进行同步,实现波形文件的冗余存储。(6) Send synchronization message: use the waveform file metadata as synchronization information, send the file upload synchronization message to the standby machine through the message queue, and after the standby machine receives the synchronization message from the message queue, it will synchronize according to the file metadata to realize the redundancy of the waveform file. remaining storage.

优选地,所述文件下载通过接口发送下载相关数据到服务器,在进行文件下载时,首先进行权限校验,校验通过后,查询库中文件元数据信息,通过元数据定位文件,返回文件信息;Preferably, the file download sends download-related data to the server through the interface, and when the file is downloaded, the authority check is first performed. After the check is passed, the metadata information of the file in the library is queried, the file is located through the metadata, and the file information is returned. ;

其中,查询库的方式为:先查询缓存库,后查询关系库并刷新缓存库。Among them, the method of querying the library is: first query the cache library, then query the relational library and refresh the cache library.

优选地,所述文件删除是指通过接口发送删除文件信息至服务器,在进行文件删除时,首先进行权限校验,校验通过后,查询库中文件元数据信息,通过元数据定位文件,删除对应文件及库中文件元数据信息,删除完成后,发送同步删除信息至备机,完成主备同步删除;Preferably, the file deletion refers to sending deleted file information to the server through an interface. When deleting a file, firstly perform a permission check. After the check is passed, query the metadata information of the file in the database, locate the file through the metadata, and delete the file. Corresponding files and metadata information of files in the library, after deletion is completed, send synchronous deletion information to the standby machine to complete the synchronous deletion of the main and standby machines;

其中,删除对应文件及库中文件元数据信息时,采用先擦除缓存库,后擦除关系库的方案,保证双库一致性。Among them, when deleting the metadata information of the corresponding file and the file in the library, the scheme of erasing the cache library first and then erasing the relational library is adopted to ensure the consistency of the two databases.

优选地,所述文件查询是指通过接口发送查询文件的条件至服务器,在进行文件查询时,首先进行权限校验,校验通过后,采用先查缓存库、后查关系库、刷新缓存库的方式,查询文件元数据信息,将查询到的所有文件信息返回。Preferably, the file query refers to sending the query file conditions to the server through the interface. When performing file query, firstly perform permission verification. After the verification is passed, first check the cache library, then check the relationship library, and refresh the cache library. In this way, the file metadata information is queried, and all the queried file information is returned.

一种终端,包括处理器及存储介质;所述存储介质用于存储指令;A terminal, including a processor and a storage medium; the storage medium is used to store instructions;

所述处理器用于根据所述指令进行操作以执行所述方法的步骤。The processor is configured to operate according to the instructions to perform the steps of the method.

计算机可读存储介质,其上存储有计算机程序,该程序被处理器执行时实现所述方法的步骤。A computer-readable storage medium stores a computer program thereon, and implements the steps of the method when the program is executed by a processor.

本发明的有益效果在于,与现有技术相比:The beneficial effects of the present invention are that, compared with the prior art:

本发明提出了电力系统工业互联网平台分布式文件的服务系统和方法,首先,提供了文件上传、文件下载、文件删除、文件查询等文件管理的基础功能,其次,还提供了文件元数据存储、分段上传下载、文件服务主备同步、文件读写冲突解决方案、文件权限管理等进阶功能。这一系列的功能可以保证系统的实时、稳定、可用,提高了电力系统工业互联网文件服务的实用性。The present invention proposes a service system and method for distributed files on the power system industrial Internet platform. First, it provides basic functions of file management such as file upload, file download, file deletion, and file query. Secondly, it also provides file metadata storage, Advanced functions such as segment upload and download, file service master/standby synchronization, file read/write conflict resolution, file permission management, etc. This series of functions can ensure the real-time, stability and availability of the system, and improve the practicability of the power system industrial Internet file service.

1、本发明采用基于HTTPS协议的restful接口完成功能,而不是采用传统的FTP的方式,使得权限控制更加方便,传输更加安全。1. The present invention uses a restful interface based on the HTTPS protocol to complete the function instead of the traditional FTP method, which makes the authority control more convenient and the transmission more secure.

2、本发明通过文件分段上传、下载,兼顾了大文件、小文件的传输需求,解决了HDFS、TFS等传统文件服务器的痛点,让服务不仅适用于电力系统相关小波形文件快速传输,也适用于大文件的服务器备份。2. The present invention takes into account the transmission requirements of large and small files by uploading and downloading files in segments, and solves the pain points of traditional file servers such as HDFS and TFS, making the service not only suitable for fast transmission of small waveform files related to power systems, but also Suitable for server backups of large files.

3、本发明采用文件元数据存储及缓存库的方案,对比传统文件存库方案,节省数据库资源,提高数据库访问效率,让文件信息的读取变得高效,大大提高了存取性能。3. The present invention adopts the solution of file metadata storage and cache library, compared with the traditional file storage solution, saves database resources, improves database access efficiency, makes file information reading more efficient, and greatly improves access performance.

4、本发明还采用了分布式锁方案,让主备机读写冲突从根本上解决,对多机同时访问、存取具有较高的适应性,方便实现主备机双活。4. The present invention also adopts a distributed lock scheme, which fundamentally solves the read-write conflict of the main and standby machines, and has high adaptability to simultaneous access and access of multiple machines, and facilitates the dual-active implementation of the main and standby machines.

5、本发明通过消息队列方式进行主备机同步,主备机冗余双活,比传统单活方案具有更高的性能,提升了系统的可用性,使系统具备一定的容灾能力。5. The present invention synchronizes the main and standby machines through the message queue method, and the active and standby machines are redundant and active, which has higher performance than the traditional single-active solution, improves the availability of the system, and enables the system to have a certain disaster recovery capability.

综合以上优势,本发明完成了电力系统工业互联网平台所需的文件服务要求,为电力系统提供了实时、稳定、高效、海量存储的文件服务。Based on the above advantages, the present invention fulfills the file service requirements required by the power system industrial Internet platform, and provides real-time, stable, efficient, and mass-storage file services for the power system.

附图说明Description of drawings

图1是本发明电力系统工业互联网平台分布式文件的服务原理图;Fig. 1 is the service schematic diagram of the distributed file of the power system industrial Internet platform of the present invention;

图2是本发明实施例中进行电力系统波形文件上传的流程图。Fig. 2 is a flow chart of uploading power system waveform files in an embodiment of the present invention.

具体实施方式Detailed ways

为使本发明的目的、技术方案和优点更加清楚,下面将结合本发明实施例中的附图,对本发明的技术方案进行清楚、完整地描述。本申请所描述的实施例仅仅是本发明一部分的实施例,而不是全部实施例。基于本发明精神,本领域普通技术人员在没有作出创造性劳动前提下所获得的有所其它实施例,都属于本发明的保护范围。In order to make the object, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The embodiments described in this application are only some embodiments of the present invention, not all embodiments. Based on the spirit of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts all belong to the protection scope of the present invention.

如图1所示,本发明的电力系统工业互联网平台分布式文件的服务系统,在本发明优选但非限制性的实施方式中,所述系统包括采用基于HTTPS协议的restful接口的基础服务模块和进阶服务模块;As shown in Figure 1, the service system for distributed files of the power system industrial Internet platform of the present invention, in a preferred but non-limiting embodiment of the present invention, the system includes a basic service module using a restful interface based on the HTTPS protocol and Advanced service module;

进一步优选地,本发明的文件服务功能模块整体采用restful接口的方式,摒弃传统FTP形式,采用HTTPS方式搭建服务端,HTTPS协议对各种业务类型更为友好,同时也使得文件服务更安全,可以更加方便的进行文件权限管理。Further preferably, the file service function module of the present invention adopts a restful interface as a whole, abandons the traditional FTP form, and adopts the HTTPS method to build the server. The HTTPS protocol is more friendly to various business types, and also makes the file service more secure. More convenient file permission management.

第一方面,所述基础服务模块包括文件上传单元、文件下载单元、文件删除单元和文件查询单元,分别提供文件上传、文件下载、文件删除和文件查询服务,其提供了电力系统工业互联网平台文件管理的基础功能,其具体技术方案如下:In the first aspect, the basic service module includes a file upload unit, a file download unit, a file deletion unit, and a file query unit, which provide file upload, file download, file deletion, and file query services respectively. The basic functions of management, the specific technical solutions are as follows:

文件上传单元,通过接口上传文件相关数据。文件上传时,首先进行权限校验,读写冲突校验及参数校验后,将文件元数据存入库中。存库方案采用先擦除缓存库,后写入关系库的方案,防止出现双库不一致情况。最后,发送同步上传信息至备机,进行主备文件同步。The file upload unit uploads file-related data through an interface. When a file is uploaded, the permission verification, read-write conflict verification and parameter verification are performed first, and then the file metadata is stored in the library. The storage scheme adopts the scheme of erasing the cache library first and then writing to the relational library to prevent the inconsistency between the two databases. Finally, send the synchronous upload information to the standby machine to synchronize the main and standby files.

文件下载单元,通过接口发送下载相关数据到服务器。文件下载时,首先进行权限校验,读写冲突校验及参数校验后,查询库中文件元数据信息,通过元数据定位文件,返回文件信息。查库方案采用先查缓存库、后查关系库、刷新缓存库的方式,提高服务的响应速度。The file downloading unit sends download related data to the server through the interface. When downloading a file, first perform permission verification, read-write conflict verification, and parameter verification, then query the metadata information of the file in the library, locate the file through the metadata, and return the file information. The database query solution adopts the method of first checking the cache library, then checking the relationship library, and refreshing the cache library to improve the response speed of the service.

文件删除单元,通过接口发送删除文件信息至服务器。文件删除时,首先进行权限校验,读写冲突校验及参数校验后,查询库中文件元数据信息,通过元数据定位文件,删除对应文件及库中文件元数据信息。删除采用先擦除缓存库,后擦除关系库的方案,保证双库一致性。删除完成后,发送同步删除信息至备机,完成主备同步删除。The file deletion unit sends the file deletion information to the server through the interface. When deleting a file, first perform permission verification, read-write conflict verification, and parameter verification, query the metadata information of the file in the library, locate the file through the metadata, and delete the corresponding file and the metadata information of the file in the library. The deletion adopts the scheme of erasing the cache library first, and then erasing the relational library to ensure the consistency of the dual databases. After the deletion is completed, send the synchronous deletion information to the standby machine to complete the synchronous deletion of the active and standby machines.

文件查询单元,通过接口发送查询文件的条件至服务器。文件查询时,首先进行权限校验,读写冲突校验及参数校验后,采用先查缓存库、后查关系库、刷新缓存库的方式,查询文件元数据信息,将查询到的所有文件信息返回。The file query unit sends the conditions of file query to the server through the interface. When querying files, first perform permission verification, read-write conflict verification, and parameter verification, and use the method of first checking the cache library, then checking the relationship library, and refreshing the cache library to query the metadata information of the file, and all the queried files will be Information returned.

第二方面,所述进阶服务模块包括文件元数据存储单元、分段上传下载单元、文件主备同步单元、文件读写冲突解决方案单元、文件权限管理单元,分别提供文件元数据存储、分段上传下载、文件主备同步、文件读写冲突解决方案、文件权限管理服务,以辅助实现基础服务模块的各项服务,其提供了电力系统工业互联网平台文件服务的一些进阶功能,来保证系统性能,其具体实现细节如下:In the second aspect, the advanced service module includes a file metadata storage unit, a segmented upload and download unit, a file master/slave synchronization unit, a file read/write conflict solution unit, and a file authority management unit, which respectively provide file metadata storage, distribution Segment upload and download, file master/standby synchronization, file read/write conflict resolution, and file authority management services to assist in the realization of various services of the basic service module, which provides some advanced functions of file services on the power system industrial Internet platform to ensure System performance, its specific implementation details are as follows:

文件元数据存储单元,将文件存储到磁盘,文件元数据存储到缓存库和关系库中,提高文件管理功能的响应速度,保证文件服务的实时性要求。The file metadata storage unit stores files to the disk, and stores the file metadata to the cache library and the relationship library, so as to improve the response speed of the file management function and ensure the real-time requirements of the file service.

文件分段上传下载单元,为了保证大文件管理及断点续传的需要,提高接口性能及吞吐量,文件服务允许文件分段上传和下载。通过文件分段的方式,使得文件服务实现大文件上传、文件并行上传、分段断点续传等功能。The file segment upload and download unit, in order to ensure the management of large files and the need for breakpoint resume, improve interface performance and throughput, the file service allows file segment upload and download. By means of file segmentation, the file service can realize functions such as large file upload, parallel file upload, segmented breakpoint resume upload, etc.

文件主备同步单元,文件服务支持主备机同步,主备机冗余双活。在文件增删时,主机文件服务通过消息队列发送消息给备机,备机同步更新文件信息。同时基于消息队列消息存储的功能,文件服务具备一定的容灾能力。备机宕机重启后,能从消息队列拉取一段时间内未同步的文件信息,进行文件数据同步。File master-standby synchronization unit, file service supports master-standby synchronization, master-standby redundancy active-active. When files are added or deleted, the host file service sends a message to the standby machine through the message queue, and the standby machine updates the file information synchronously. At the same time, based on the message storage function of the message queue, the file service has a certain disaster recovery capability. After the standby machine is down and restarted, the file information that has not been synchronized for a period of time can be pulled from the message queue to synchronize the file data.

读写冲突解决方案单元,文件服务采用分布式锁的方案,使用文件信息作为锁秘钥,在文件读写前尝试进行加锁。不同于单机锁方案,分布式锁不止解决了单机读写问题,也解决了主备机间的读写冲突。In the read-write conflict solution unit, the file service adopts the distributed lock scheme, uses file information as the lock key, and tries to lock the file before reading and writing. Different from the stand-alone lock solution, the distributed lock not only solves the problem of reading and writing on a stand-alone machine, but also solves the read-write conflict between the master and standby machines.

文件权限管理单元,基于token令牌方式,进行权限校验。文件服务为每个用户配置可访问的路径列表,进行权限管理。用户获取token后,在请求文件服务时,请求头携带相应token,由文件服务解析token完成权限校验。The file authority management unit performs authority verification based on the token token method. The file service configures an accessible path list for each user for rights management. After the user obtains the token, when requesting the file service, the request header carries the corresponding token, and the file service parses the token to complete the permission verification.

本发明实施例2提供电力系统工业互联网平台分布式文件的服务方法,基于上述系统实现,所述分布式文件服务方法包括:Embodiment 2 of the present invention provides a service method for distributed files on the power system industrial Internet platform. Based on the above-mentioned system implementation, the distributed file service method includes:

基于基础服务模块进行文件上传、文件下载、文件删除和文件查询,同时,进阶服务模块在基础服务模块进行文件上传、文件下载、文件删除和文件查询时进行文件元数据存储、分段上传下载、文件主备同步、文件读写冲突解决方案、文件权限管理辅助。File upload, file download, file deletion, and file query are performed based on the basic service module. At the same time, the advanced service module performs file metadata storage, segmented upload and download when the basic service module performs file upload, file download, file deletion, and file query. , File master and backup synchronization, file read and write conflict resolution, file permission management assistance.

进一步优选地,所述文件上传是指通过接口上传文件相关数据,且在进行文件上传时,首先进行校验,校验通过后,将文件元数据存入库中,并发送同步上传信息至备机,进行主备文件同步;Further preferably, the file upload refers to uploading file-related data through an interface, and when uploading a file, first perform verification, after the verification is passed, store the file metadata in the library, and send synchronous upload information to the backup machine to synchronize the master and backup files;

所述校验包括权限校验、读写冲突校验及参数校验;The verification includes permission verification, read-write conflict verification and parameter verification;

所述将文件元数据存入库中的方式为:先擦除缓存库,后写入关系库的方案,防止出现双库不一致情况。The method of storing the file metadata in the database is as follows: firstly erase the cache database, and then write into the relationship database, so as to prevent the inconsistency between the two databases.

如图2所示,上传电力系统波形文件的流程具体包括以下步骤:As shown in Figure 2, the process of uploading power system waveform files specifically includes the following steps:

(1)波形文件导入,上传波形文件,选择上传路径、是否覆盖、重命名等信息,请求头携带校验token,发送上传请求。(1) Import the wave file, upload the wave file, select the upload path, whether to overwrite, rename and other information, carry the verification token in the request header, and send the upload request.

(2)权限检验,根据请求头携带的token进行解析,判断当前请求是否有权限向相应路径导入波形文件。如果无权限,进入失败流程,打印错误日志,结束流程。(2) Permission check, analyze according to the token carried in the request header, and judge whether the current request has permission to import wave files to the corresponding path. If there is no permission, enter the failure process, print the error log, and end the process.

(3)读写冲突校验,当前请求尝试获得当前波形文件的分布式锁,如果获取成功,说明没有读写冲突,继续流程。若获取锁失败,进入失败流程,打印错误日志,结束流程。(3) Read-write conflict check. The current request tries to obtain the distributed lock of the current waveform file. If the acquisition is successful, it means that there is no read-write conflict, and the process continues. If the acquisition of the lock fails, enter the failure process, print the error log, and end the process.

(4)参数校验,进行基本参数校验、录波文件格式校验,并且查询库中是否已存在当前波形文件,根据是否允许文件覆盖写,分别进入成功流程和失败流程。查询时,采用先查缓存库,后查关系库的方案。(4) Parameter verification, perform basic parameter verification, wave recording file format verification, and query whether the current waveform file already exists in the library, and enter the success process and failure process respectively according to whether the file overwriting is allowed. When querying, the scheme of checking the cache library first and then checking the relational library is adopted.

(5)写入数据库,分为擦除缓存库和写入关系库两步。根据波形文件对应的关键字,删除缓存库文件元数据,之后将文件元数据写入关系库。(5) Writing to the database is divided into two steps: erasing the cache library and writing to the relational library. According to the keyword corresponding to the wave file, delete the cache library file metadata, and then write the file metadata into the relational library.

(6)发送同步消息,将波形文件元数据作为同步信息,通过消息队列发送文件上传同步消息至备机,备机从消息队列接受同步消息后,根据文件元数据进行同步,实现波形文件的冗余存储。(6) Send a synchronization message, use the waveform file metadata as synchronization information, and send a file upload synchronization message to the standby machine through the message queue. After the standby machine receives the synchronization message from the message queue, it will synchronize according to the file metadata to realize redundancy of the waveform file. remaining storage.

所述文件下载通过接口发送下载相关数据到服务器,在进行文件下载时,首先进行权限校验,校验通过后,查询库中文件元数据信息,通过元数据定位文件,返回文件信息;The file download sends download related data to the server through the interface, and when the file is downloaded, the authority verification is first performed, and after the verification is passed, the metadata information of the file in the library is inquired, the file is located by the metadata, and the file information is returned;

其中,查询库的方式为:先查询缓存库,后查询关系库并刷新缓存库。Among them, the method of querying the library is: first query the cache library, then query the relational library and refresh the cache library.

所述文件删除是指通过接口发送删除文件信息至服务器,在进行文件删除时,首先进行权限校验,校验通过后,查询库中文件元数据信息,通过元数据定位文件,删除对应文件及库中文件元数据信息,删除完成后,发送同步删除信息至备机,完成主备同步删除;The file deletion refers to sending deleted file information to the server through an interface. When deleting a file, first perform permission verification. After the verification is passed, query the metadata information of the file in the database, locate the file through the metadata, and delete the corresponding file and After the metadata information of the file in the library is deleted, the synchronous deletion information is sent to the standby machine to complete the synchronous deletion of the main and standby machines;

其中,删除对应文件及库中文件元数据信息时,采用先擦除缓存库,后擦除关系库的方案,保证双库一致性。Among them, when deleting the metadata information of the corresponding file and the file in the library, the scheme of erasing the cache library first and then erasing the relational library is adopted to ensure the consistency of the two databases.

所述文件查询是指通过接口发送查询文件的条件至服务器,在进行文件查询时,首先进行权限校验,校验通过后,采用先查缓存库、后查关系库、刷新缓存库的方式,查询文件元数据信息,将查询到的所有文件信息返回。The file query refers to sending the query file conditions to the server through the interface. When performing a file query, the authority check is first performed. After the check is passed, the method of first checking the cache library, then checking the relationship library, and refreshing the cache library is adopted. Query the metadata information of the file, and return all the queried file information.

一种终端,包括处理器及存储介质;所述存储介质用于存储指令;A terminal, including a processor and a storage medium; the storage medium is used to store instructions;

所述处理器用于根据所述指令进行操作以执行所述方法的步骤。The processor is configured to operate according to the instructions to perform the steps of the method.

计算机可读存储介质,其上存储有计算机程序,该程序被处理器执行时实现所述方法的步骤。A computer-readable storage medium stores a computer program thereon, and implements the steps of the method when the program is executed by a processor.

本发明的有益效果在于,与现有技术相比:The beneficial effects of the present invention are that, compared with the prior art:

本发明提出了电力系统工业互联网平台分布式文件的服务系统和方法,首先,提供了文件上传、文件下载、文件删除、文件查询等文件管理的基础功能,其次,还提供了文件元数据存储、分段上传下载、文件服务主备同步、文件读写冲突解决方案、文件权限管理等进阶功能。这一系列的功能可以保证系统的实时、稳定、可用,提高了电力系统工业互联网文件服务的实用性。The present invention proposes a service system and method for distributed files on the power system industrial Internet platform. First, it provides basic functions of file management such as file upload, file download, file deletion, and file query. Secondly, it also provides file metadata storage, Advanced functions such as segment upload and download, file service master/standby synchronization, file read/write conflict resolution, file permission management, etc. This series of functions can ensure the real-time, stability and availability of the system, and improve the practicability of the power system industrial Internet file service.

1、本发明采用基于HTTPS协议的restful接口完成功能,而不是采用传统的FTP的方式,使得权限控制更加方便,传输更加安全。1. The present invention uses a restful interface based on the HTTPS protocol to complete the function instead of the traditional FTP method, which makes the authority control more convenient and the transmission more secure.

2、本发明通过文件分段上传、下载,兼顾了大文件、小文件的传输需求,解决了HDFS、TFS等传统文件服务器的痛点,让服务不仅适用于电力系统相关小波形文件快速传输,也适用于大文件的服务器备份。2. The present invention takes into account the transmission requirements of large and small files by uploading and downloading files in segments, and solves the pain points of traditional file servers such as HDFS and TFS, making the service not only suitable for fast transmission of small waveform files related to power systems, but also Suitable for server backups of large files.

3、本发明采用文件元数据存储及缓存库的方案,对比传统文件存库方案,节省数据库资源,提高数据库访问效率,让文件信息的读取变得高效,大大提高了存取性能。3. The present invention adopts the solution of file metadata storage and cache library, compared with the traditional file storage solution, saves database resources, improves database access efficiency, makes file information reading more efficient, and greatly improves access performance.

4、本发明还采用了分布式锁方案,让主备机读写冲突从根本上解决,对多机同时访问、存取具有较高的适应性,方便实现主备机双活。4. The present invention also adopts a distributed lock scheme, which fundamentally solves the read-write conflict of the main and standby machines, and has high adaptability to simultaneous access and access of multiple machines, and facilitates the dual-active implementation of the main and standby machines.

5、本发明通过消息队列方式进行主备机同步,主备机冗余双活,比传统单活方案具有更高的性能,提升了系统的可用性,使系统具备一定的容灾能力。5. The present invention synchronizes the main and standby machines through the message queue method, and the active and standby machines are redundant and active, which has higher performance than the traditional single-active solution, improves the availability of the system, and enables the system to have a certain disaster recovery capability.

综合以上优势,本发明完成了电力系统工业互联网平台所需的文件服务要求,为电力系统提供了实时、稳定、高效、海量存储的文件服务。Based on the above advantages, the present invention fulfills the file service requirements required by the power system industrial Internet platform, and provides real-time, stable, efficient, and mass-storage file services for the power system.

本公开可以是系统、方法和/或计算机程序产品。计算机程序产品可以包括计算机可读存储介质,其上载有用于使处理器实现本公开的各个方面的计算机可读程序指令。The present disclosure can be a system, method and/or computer program product. A computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to implement various aspects of the present disclosure.

计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算机可读存储介质例如可以是――但不限于――电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括:便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、静态随机存取存储器(SRAM)、便携式压缩盘只读存储器(CD-ROM)、数字多功能盘(DVD)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。这里所使用的计算机可读存储介质不被解释为瞬时信号本身,诸如无线电波或者其它自由传播的电磁波、通过波导或其它传输媒介传播的电磁波(例如,通过光纤电缆的光脉冲)、或者通过电线传输的电信号。A computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory), static random access memory (SRAM), compact disc read only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded device, such as a printer with instructions stored thereon A hole card or a raised structure in a groove, and any suitable combination of the above. Computer-readable storage media as used herein is not to be interpreted as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., pulses of light through fiber optic cables), or transmitted electrical signals.

这里所描述的计算机可读程序指令可以从计算机可读存储介质下载到各个计算/处理设备,或者通过网络、例如因特网、局域网、广域网和/或无线网下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和/或边缘服务器。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令,并转发该计算机可读程序指令,以供存储在各个计算/处理设备中的计算机可读存储介质中。Computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or a network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

用于执行本公开操作的计算机程序指令可以是汇编指令、指令集架构(ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代码,所述编程语言包括面向对象的编程语言—诸如Smalltalk、C++等,以及常规的过程式编程语言—诸如“C”语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中,远程计算机可以通过任意种类的网络—包括局域网(LAN)或广域网(WAN)—连接到用户计算机,或者,可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中,通过利用计算机可读程序指令的状态信息来个性化定制电子电路,例如可编程逻辑电路、现场可编程门阵列(FPGA)或可编程逻辑阵列(PLA),该电子电路可以执行计算机可读程序指令,从而实现本公开的各个方面。Computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or Source or object code written in any combination, including object-oriented programming languages—such as Smalltalk, C++, etc., and conventional procedural programming languages—such as the “C” language or similar programming languages. Computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as via the Internet using an Internet service provider). connect). In some embodiments, an electronic circuit, such as a programmable logic circuit, field programmable gate array (FPGA), or programmable logic array (PLA), can be customized by utilizing state information of computer-readable program instructions, which can Various aspects of the present disclosure are implemented by executing computer readable program instructions.

最后应当说明的是,以上实施例仅用以说明本发明的技术方案而非对其限制,尽管参照上述实施例对本发明进行了详细的说明,所属领域的普通技术人员应当理解:依然可以对本发明的具体实施方式进行修改或者等同替换,而未脱离本发明精神和范围的任何修改或者等同替换,其均应涵盖在本发明的权利要求保护范围之内。Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that the present invention can still be Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention shall fall within the protection scope of the claims of the present invention.

Claims (12)

1.电力系统工业互联网平台分布式文件的服务系统,包括采用基于HTTPS协议的restful接口的基础服务模块和进阶服务模块,其特征在于:1. The distributed file service system of the power system industrial Internet platform, including the basic service module and the advanced service module using the restful interface based on the HTTPS protocol, characterized in that: 所述基础服务模块包括文件上传单元、文件下载单元、文件删除单元和文件查询单元,分别提供文件上传、文件下载、文件删除和文件查询服务;The basic service module includes a file upload unit, a file download unit, a file deletion unit and a file query unit, which provide file upload, file download, file deletion and file query services respectively; 所述进阶服务模块包括文件元数据存储单元、分段上传下载单元、文件主备同步单元、文件读写冲突解决方案单元、文件权限管理单元,分别提供文件元数据存储、分段上传下载、文件主备同步、文件读写冲突解决方案、文件权限管理服务,以辅助实现基础服务模块的各项服务。The advanced service module includes a file metadata storage unit, a segment upload and download unit, a file master/slave synchronization unit, a file read/write conflict solution unit, and a file authority management unit, which provide file metadata storage, segment upload and download, File master-backup synchronization, file read-write conflict resolution, file rights management services, to assist in the realization of various services of the basic service module. 2.根据权利要求1所述的电力系统工业互联网平台分布式文件的服务系统,其特征在于:2. The service system of the distributed file of the power system industrial internet platform according to claim 1, characterized in that: 所述文件元数据存储单元,用于将文件存储到磁盘,将文件元数据存储到缓存库和关系库中;The file metadata storage unit is used to store the file to the disk, and store the file metadata in the cache library and the relationship library; 所述文件分段上传下载单元,用于通过文件分段的方式,实现大文件上传下载、文件并行上传下载、分段断点续传;The file segmentation uploading and downloading unit is used to realize uploading and downloading of large files, parallel uploading and downloading of files, and resumed transmission of segmented breakpoints by means of file segmentation; 所述文件主备同步单元,用于实现主备机同步,主备机冗余双活;The file master-standby synchronization unit is used to realize master-standby machine synchronization, master-standby machine redundancy dual-active; 所述读写冲突解决方案单元,用于采用分布式锁的方案,使用文件信息作为锁秘钥,在文件读写前进行加锁,以解决单机读写问题和主备机间的读写冲突;The read-write conflict solution unit is used to adopt a distributed lock solution, using file information as a lock key, and locking the file before reading and writing, so as to solve the read-write problem of a single machine and the read-write conflict between the master and standby machines ; 所述文件权限管理单元,用于基于token令牌方式进行权限校验。The file authority management unit is used for performing authority verification based on a token token. 3.根据权利要求2所述的电力系统工业互联网平台分布式文件的服务系统,其特征在于:3. The service system of the distributed file of the power system industrial Internet platform according to claim 2, characterized in that: 所述文件主备同步单元在文件增删时,主机文件服务通过消息队列发送消息给备机,备机同步更新文件信息,同时基于消息队列消息存储的功能,文件服务具备容灾能力,备机宕机重启后,能从消息队列拉取未同步的文件信息,进行文件数据同步。When the file master-standby synchronization unit adds or deletes files, the host file service sends a message to the standby machine through the message queue, and the standby machine updates the file information synchronously. After the computer is restarted, the unsynchronized file information can be pulled from the message queue to synchronize the file data. 4.根据权利要求2所述的电力系统工业互联网平台分布式文件的服务系统,其特征在于:4. The service system of the distributed file of the power system industrial internet platform according to claim 2, characterized in that: 所述文件权限管理单元为每个用户配置可访问的路径列表,进行权限管理,用户获取token后,在请求文件服务时,请求头携带相应token,由文件服务解析token完成权限校验。The file authority management unit configures an accessible path list for each user, and performs authority management. After the user obtains the token, when requesting the file service, the request header carries the corresponding token, and the file service parses the token to complete the authority verification. 5.电力系统工业互联网平台分布式文件的服务方法,基于权利要求1-4任一项所述系统实现,其特征在于:5. The service method of the distributed file of the electric power system industrial Internet platform, based on the system implementation of any one of claims 1-4, characterized in that: 所述分布式文件服务方法包括:The distributed file service method includes: 基于基础服务模块进行文件上传、文件下载、文件删除和文件查询,同时,进阶服务模块在基础服务模块进行文件上传、文件下载、文件删除和文件查询时进行文件元数据存储、分段上传下载、文件主备同步、文件读写冲突解决方案、文件权限管理辅助。File upload, file download, file deletion, and file query are performed based on the basic service module. At the same time, the advanced service module performs file metadata storage, segmented upload and download when the basic service module performs file upload, file download, file deletion, and file query. , File master and backup synchronization, file read and write conflict resolution, file permission management assistance. 6.根据权利要求5所述的电力系统工业互联网平台分布式文件的服务方法,其特征在于:6. The service method of the distributed file of the power system industrial Internet platform according to claim 5, characterized in that: 所述文件上传是指通过接口上传文件相关数据,且在进行文件上传时,首先进行校验,校验通过后,将文件元数据存入库中,并发送同步上传信息至备机,进行主备文件同步;The file upload refers to uploading file-related data through an interface, and when uploading a file, it first performs verification, and after the verification is passed, the file metadata is stored in the database, and the synchronous upload information is sent to the standby machine for master Backup file synchronization; 所述校验包括权限校验、读写冲突校验及参数校验;The verification includes permission verification, read-write conflict verification and parameter verification; 所述将文件元数据存入库中的方式为:先擦除缓存库,后写入关系库的方案,防止出现双库不一致情况。The method of storing the file metadata in the database is as follows: firstly erase the cache database, and then write into the relationship database, so as to prevent the inconsistency between the two databases. 7.根据权利要求6所述的电力系统工业互联网平台分布式文件的服务方法,其特征在于:7. The service method of the distributed file of the power system industrial Internet platform according to claim 6, characterized in that: 所述文件上传具体过程包括:The specific process of uploading the file includes: (1)波形文件导入:上传电力系统波形文件,选择上传路径、是否覆盖、重命名信息,发送上传请求,请求头携带token;(1) Waveform file import: upload the power system waveform file, select the upload path, whether to overwrite, rename the information, send the upload request, and the request header carries the token; (2)权限检验:根据请求头携带的token进行解析,判断当前请求是否有权限向相应路径导入波形文件,如果无权限,进入失败流程,打印错误日志,结束流程,否则进入(3);(2) Permission inspection: analyze according to the token carried in the request header, and judge whether the current request has permission to import the wave file to the corresponding path. If there is no permission, enter the failure process, print the error log, and end the process, otherwise enter (3); (3)读写冲突校验:当前请求尝试获得当前波形文件的分布式锁,如果获取成功,说明没有读写冲突,进入(4),若获取失败,则进入失败流程,打印错误日志,结束流程;(3) Read-write conflict check: The current request tries to obtain the distributed lock of the current waveform file. If the acquisition is successful, it means that there is no read-write conflict. Go to (4). If the acquisition fails, enter the failure process, print the error log, and end process; (4)参数校验:进行基本参数校验、录波文件格式校验,并且查询库中是否已存在当前波形文件,根据是否允许文件覆盖写,分别进入成功流程和失败流程,查询库时,采用先查缓存库,后查关系库的方式;(4) Parameter verification: perform basic parameter verification, wave recording file format verification, and query whether the current waveform file already exists in the library, and enter the success process and failure process respectively according to whether file overwriting is allowed. When querying the library, Use the method of checking the cache library first, and then checking the relational library; (5)写入数据库:根据波形文件对应的关键字,删除缓存库文件元数据,之后将文件元数据写入关系库;(5) Write into the database: delete the file metadata of the cache library according to the keyword corresponding to the wave file, and then write the file metadata into the relational library; (6)发送同步消息:将波形文件元数据作为同步信息,通过消息队列发送文件上传同步消息至备机,备机从消息队列接受同步消息后,根据文件元数据进行同步,实现波形文件的冗余存储。(6) Send synchronization message: use the waveform file metadata as synchronization information, send the file upload synchronization message to the standby machine through the message queue, and after the standby machine receives the synchronization message from the message queue, it will synchronize according to the file metadata to realize the redundancy of the waveform file. remaining storage. 8.根据权利要求5所述的电力系统工业互联网平台分布式文件的服务方法,其特征在于:8. The service method of the distributed file of the power system industrial Internet platform according to claim 5, characterized in that: 所述文件下载通过接口发送下载相关数据到服务器,在进行文件下载时,首先进行权限校验,校验通过后,查询库中文件元数据信息,通过元数据定位文件,返回文件信息;The file download sends download related data to the server through the interface, and when the file is downloaded, the authority verification is first performed, and after the verification is passed, the metadata information of the file in the library is inquired, the file is located by the metadata, and the file information is returned; 其中,查询库的方式为:先查询缓存库,后查询关系库并刷新缓存库。Among them, the method of querying the library is: first query the cache library, then query the relational library and refresh the cache library. 9.根据权利要求5所述的电力系统工业互联网平台分布式文件的服务方法,其特征在于:9. The service method of the distributed file of the power system industrial Internet platform according to claim 5, characterized in that: 所述文件删除是指通过接口发送删除文件信息至服务器,在进行文件删除时,首先进行权限校验,校验通过后,查询库中文件元数据信息,通过元数据定位文件,删除对应文件及库中文件元数据信息,删除完成后,发送同步删除信息至备机,完成主备同步删除;The file deletion refers to sending deleted file information to the server through an interface. When deleting a file, first perform permission verification. After the verification is passed, query the metadata information of the file in the database, locate the file through the metadata, and delete the corresponding file and After the metadata information of the file in the library is deleted, the synchronous deletion information is sent to the standby machine to complete the synchronous deletion of the main and standby machines; 其中,删除对应文件及库中文件元数据信息时,采用先擦除缓存库,后擦除关系库的方案,保证双库一致性。Among them, when deleting the metadata information of the corresponding file and the file in the library, the scheme of erasing the cache library first and then erasing the relational library is adopted to ensure the consistency of the two databases. 10.根据权利要求5所述的电力系统工业互联网平台分布式文件的服务方法,其特征在于:10. The service method of the distributed file of the power system industrial Internet platform according to claim 5, characterized in that: 所述文件查询是指通过接口发送查询文件的条件至服务器,在进行文件查询时,首先进行权限校验,校验通过后,采用先查缓存库、后查关系库、刷新缓存库的方式,查询文件元数据信息,将查询到的所有文件信息返回。The file query refers to sending the query file conditions to the server through the interface. When performing a file query, the authority check is first performed. After the check is passed, the method of first checking the cache library, then checking the relationship library, and refreshing the cache library is adopted. Query the metadata information of the file, and return all the queried file information. 11.一种终端,包括处理器及存储介质;其特征在于:11. A terminal, comprising a processor and a storage medium; characterized in that: 所述存储介质用于存储指令;The storage medium is used to store instructions; 所述处理器用于根据所述指令进行操作以执行根据权利要求5-10任一项所述方法的步骤。The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 5-10. 12.计算机可读存储介质,其上存储有计算机程序,其特征在于,该程序被处理器执行时实现权利要求5-10任一项所述方法的步骤。12. A computer-readable storage medium, on which a computer program is stored, wherein when the program is executed by a processor, the steps of the method according to any one of claims 5-10 are implemented.
CN202211711519.1A 2022-12-29 2022-12-29 Service system and method for distributed files of electric power system industrial internet platform Pending CN116303275A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211711519.1A CN116303275A (en) 2022-12-29 2022-12-29 Service system and method for distributed files of electric power system industrial internet platform

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211711519.1A CN116303275A (en) 2022-12-29 2022-12-29 Service system and method for distributed files of electric power system industrial internet platform

Publications (1)

Publication Number Publication Date
CN116303275A true CN116303275A (en) 2023-06-23

Family

ID=86831265

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211711519.1A Pending CN116303275A (en) 2022-12-29 2022-12-29 Service system and method for distributed files of electric power system industrial internet platform

Country Status (1)

Country Link
CN (1) CN116303275A (en)

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102420854A (en) * 2011-11-14 2012-04-18 西安电子科技大学 Distributed file system for cloud storage
US20130318242A1 (en) * 2012-05-24 2013-11-28 Scalextreme Inc. Remote Management Of Distributed Datacenters
CN103473337A (en) * 2013-09-22 2013-12-25 北京航空航天大学 Massive catalogs and files oriented processing method in distributed type storage system
CN110401725A (en) * 2019-08-23 2019-11-01 深圳市鹰硕技术有限公司 File uploading method, device, terminal, server, system and storage medium
CN110543464A (en) * 2018-12-12 2019-12-06 广东鼎义互联科技股份有限公司 Big data platform applied to smart park and operation method
CN110679131A (en) * 2018-04-02 2020-01-10 甲骨文国际公司 Data replication conflict detection and resolution scheme for multi-tenant identity cloud service
CN112650729A (en) * 2020-12-17 2021-04-13 深圳点宽网络科技有限公司 Authority management method and system of distributed file system and storage medium
CN112738167A (en) * 2020-12-18 2021-04-30 福建新大陆软件工程有限公司 File service opening method, device, equipment and medium based on API gateway
CN115114294A (en) * 2022-06-30 2022-09-27 深圳市腾讯计算机系统有限公司 Adaptive method, device and computer equipment for database storage mode
CN115118705A (en) * 2022-06-28 2022-09-27 重庆大学 Industrial edge management and control platform based on micro-service

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102420854A (en) * 2011-11-14 2012-04-18 西安电子科技大学 Distributed file system for cloud storage
US20130318242A1 (en) * 2012-05-24 2013-11-28 Scalextreme Inc. Remote Management Of Distributed Datacenters
CN103473337A (en) * 2013-09-22 2013-12-25 北京航空航天大学 Massive catalogs and files oriented processing method in distributed type storage system
CN110679131A (en) * 2018-04-02 2020-01-10 甲骨文国际公司 Data replication conflict detection and resolution scheme for multi-tenant identity cloud service
CN110543464A (en) * 2018-12-12 2019-12-06 广东鼎义互联科技股份有限公司 Big data platform applied to smart park and operation method
CN110401725A (en) * 2019-08-23 2019-11-01 深圳市鹰硕技术有限公司 File uploading method, device, terminal, server, system and storage medium
CN112650729A (en) * 2020-12-17 2021-04-13 深圳点宽网络科技有限公司 Authority management method and system of distributed file system and storage medium
CN112738167A (en) * 2020-12-18 2021-04-30 福建新大陆软件工程有限公司 File service opening method, device, equipment and medium based on API gateway
CN115118705A (en) * 2022-06-28 2022-09-27 重庆大学 Industrial edge management and control platform based on micro-service
CN115114294A (en) * 2022-06-30 2022-09-27 深圳市腾讯计算机系统有限公司 Adaptive method, device and computer equipment for database storage mode

Similar Documents

Publication Publication Date Title
US9697092B2 (en) File-based cluster-to-cluster replication recovery
CN104813321B (en) The content and metadata of uncoupling in distributed objects store the ecosystem
CA2893304C (en) Data storage method, data storage apparatus, and storage device
US11093387B1 (en) Garbage collection based on transmission object models
CN106407040A (en) Remote data copy method and system
CN103797484B (en) The local network storage being connected with remote network storage system
CN107818111B (en) Method for caching file data, server and terminal
CN107967122A (en) A kind of method for writing data of block device, device and medium
CN116467275A (en) Shared remote storage method, device, system, electronic equipment and storage medium
CN115328407A (en) Cache disk replacing method, device, equipment and storage medium
JP6967010B2 (en) Duplication between heterogeneous storage systems
US11341159B2 (en) In-stream data load in a replication environment
CN115658382A (en) Incremental object backup and recovery method, system and storage medium for object storage
CN110515918A (en) A distributed storage platform and construction method based on HDFS
CN116560904A (en) NAS data backup disaster recovery method, system, terminal and storage medium
WO2019109256A1 (en) Log management method, server and database system
CN115905159A (en) Method, device and system for storing business data files based on storage strategy
US11137931B1 (en) Backup metadata deletion based on backup data deletion
CN116303275A (en) Service system and method for distributed files of electric power system industrial internet platform
US11645333B1 (en) Garbage collection integrated with physical file verification
US8281096B1 (en) Systems and methods for creating snapshots
CN111522688A (en) Data backup method and device for distributed system
CN111581015B (en) Continuous data protection system and method for modern application
CN119645733B (en) KVM data storage processing method and system and continuous data point recovery method thereof
US12158821B2 (en) Snappable recovery chain over generic managed volume

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination