CN112650610B - Linux system crash control method, system and medium - Google Patents

Abstract

The invention discloses a Linux system crash control method which comprises the steps of establishing a kernel crash analysis thread, analyzing the cause of system crash, and analyzing the system crash caused by a user or hardware or software; creating a kernel crash evasion thread, carrying out a system crash test experiment caused by software, and triggering the system crash caused by the software; reading a log code generated when the system is crashed, and writing the log code into a suspended task function for protection and shielding; packaging the suspended task function into a Linux kernel, restarting an operating system, and re-entering the Linux kernel; by the mode, the system log can be obtained through the Linux command, the phenomenon of triggering breakdown of hardware problems and software problems can be distinguished by analyzing the system log, and the blocking of the service volume is protected and shielded, so that the service volume is not monitored, when the service access data volume is large, the service access data volume can be properly processed, and the software fault is avoided.

Description

A Linux system crash control method, system and medium

technical field

The invention relates to the field of system abnormality analysis, in particular to a Linux system crash control method, system and medium.

Background technique

When using a server, it is often encountered that the server crashes abnormally or triggers the Linux kernel panic. The cause of the server crash is firstly the hardware problem of the server itself; or the external environment triggers the problem, such as the ambient temperature. If it is too high or too low, the server self-protection threshold is triggered; or it may be the impact of a virus in the external environment, or the server system is down due to task congestion; no matter what kind of phenomenon causes the abnormal restart and downtime, it will affect the Customer experience or customer usage will have an immeasurable impact.

kdump is a tool and service used to dump memory operating parameters when the system crashes, deadlocks or crashes, so that the system will generate a vmcore file under var/crash when the system triggers a kernel panic. Linux engineers based on the generated vmcore -dmesg file and vmcore analysis system causes downtime.

However, at present, the system crash caused by the blockage of Linux tasks and the triggering of kernel panic caused by the large amount of data access in customer use, including banking business, and excessive load cannot be solved well.

Contents of the invention

The technical problem mainly solved by the present invention is to provide a Linux system crash control method, system and medium, which can obtain system logs through Linux commands, and can distinguish the phenomenon of hardware problems and software problems triggering system crashes by analyzing the system logs, and analyze the traffic The blocking of the protection module establishes a barrier to prevent it from being monitored, so that when the amount of business access data is relatively large, it can be handled well and software failures can be avoided.

In order to solve the above-mentioned technical problems, a technical solution adopted by the present invention is: a kind of Linux system crash control method is provided, comprising: comprising creating a kernel crash analysis thread, analyzing the reason that the system produces a crash:

If there is system crash information caused by user behavior in the system log, the cause of the system crash is defined as the system crash caused by the user;

If the information in the system log contains system crash information caused by a hardware failure in the server, the cause of the system crash is defined as a system crash caused by hardware;

If the information in the system log contains fault information caused by system task blocking or soft deadlock error information, the cause of the system crash is defined as a system crash caused by software;

Create a kernel crash avoidance thread, test the system crash caused by the software, trigger the system crash, read the log code generated when the system crashes, and write it into the suspend task function;

Encapsulate the suspending task function into the Linux kernel, restart the operating system, and re-enter the Linux kernel.

Further, the log code generated when the system crashes includes the task process; when the system runs the task process, the system crashes.

Further, the writing suspend task function includes the following steps:

Read the task process and the number of task processes in the log code generated when the system crashes;

Write the task process and the number of task processes into the suspend task function.

Further, said encapsulating the suspending task function into the Linux kernel includes the following steps:

Clear the compilation files and configuration files generated during the Linux kernel compilation process;

Clear object files and executable files generated during Linux kernel compilation;

Use the interface command to turn the kernel configuration interface into a graphic, select the suspend task function, and compile the suspend task function into the Linux kernel;

Compile the Linux kernel by compiling the kernel command;

Use the install command to install the Linux kernel driver module;

Install the Linux kernel.

A Linux system crash control system, comprising: an analysis module, an avoidance module and an encapsulation module;

The analysis module checks the system log, and analyzes the system crash as a system crash caused by the user, a system crash caused by hardware, or a system crash caused by software;

The avoidance module performs a system crash test experiment caused by software, triggers a system crash, then reads the log code generated when the system crashes, and writes it into the suspending task function.

A computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the above-mentioned Linux system crash control method.

The beneficial effects of the present invention are: the present invention can better analyze and locate the cause of the system crash caused by the large amount of business data access, can distinguish the phenomenon of restart triggered by hardware problems and software problems, and establish a protection module for traffic blocking The barrier prevents it from being monitored, so that when the amount of business access data is relatively large, it can be handled well to avoid software failures and system downtime.

Description of drawings

Fig. 1 is a flow chart of a preferred embodiment of a Linux system crash control method of the present invention;

FIG. 2 is a schematic diagram of the architecture of a Linux system crash control system according to the present invention.

detailed description

The preferred embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, so as to define the protection scope of the present invention more clearly.

Embodiments of the invention include:

The first aspect, see Fig. 1, a kind of Linux system crash control method, comprises:

Create a kernel crash analysis thread, and use the Linux command #more/var/log/messages to display the time when the event occurred, the host name of the source system where the event occurred, and the program name that generated the log message, and analyze the cause of the system crash;

If the kernel crash analysis thread checks out the following display in the system log:

shutdown:shutting down for system reboot

init: Switching to runlevel: 6 exiting on signal 15

Got SIGTERM, quitting;

Then the reason for the system crash is the restart behavior initiated by the user himself;

The kernel crash analysis thread looks up the specific command executed by the last user through the command;

The kernel crash analysis thread displays the command information used by the previous user through lastcomm, checks the commands executed by the previous root user through lastcommroot, and analyzes the system log. If the command used or executed by the previous user causes the system to crash; it is caused by the user himself collapse.

View the system log through the kernel crash analysis thread;

If the system log shows:

CPU 1: Machine Check Exception: 4Bank 4: ba00000000070f0f

Kernel panic-not syncing: Machine check

Kernel panic-not syncing: Uncorrected machine check is the system restart caused by the failure of the CPU hardware itself;

If the system log shows:

kernel: CPUX: Temperature above threshold, cpu clock throttled

kernel:CPUX:Core power limit notification(total events=1);

Power Button Pressed received event "button/power PWRF 00000000

00000000", the restart caused by overheating of the server, the kernel crash analysis thread will give a prompt, it is recommended

Check the cooling system of the data center and the fans of the servers;

Then the system crash is the cause of hardware failure;

Causes of hardware failures include bent CPU pins, a large number of irreparable ECC failures in memory, and disk or memory damage.

Analyzing threads through kernel crashes using kdump;

Kernel crash analysis thread starts kdump;

When the system crashes, kdump generates the kernel that captures the current running information, and the kernel will collect all the running status and data information in the memory at this time into the virtual core vmcore file;

Diagnose the system crash problem in the virtual core vmcore file through Kernel Oops Analyzer, and determine the cause of the system crash fault.

View the system log through the kernel crash analysis thread;

If the system log shows:

"kernel:INFO:task:60blocked for more than 120seconds."

It is a fault caused by system task blocking;

If the system log shows:

BUG:soft lockup-CPU#2 stuck for 67s! [vmmemctl:894]

BUG:soft lockup-CPU#5 stuck for 67s! [bdi-default:49]

BUG:soft lockup-CPU#3 stuck for 67s! [irqbalance:1351]

BUG:soft lockup-CPU#4stuck for 67s! [swapper:0]

BUG:soft lockup-CPU#6 stuck for 67s! [watchdog/6:30]

BUG:soft lockup-CPU#5 stuck for 67s! [vmmemctl:894]

BUG:soft lockup-CPU#0stuck for 67s! [events/0:35]

BUG:soft lockup-CPU#7stuck for 67s! [lldpad:1459]

BUG:soft lockup-CPU#6 stuck for 67s! [mpt_poll_0:376]

BUG:soft lockup-CPU#4stuck for 67s! [ksoftirqd/4:21]

If there is a problem with a certain driver of the system, the CPU resources will be insufficient, and the watchdog will not be timely due to being too busy, resulting in the inability to collect the usage data of each logical CPU during operation, and a soft lockup error will be thrown;

Then the system crash is the cause of the software downtime;

In order to solve the faults caused by system task blocking caused by software downtime, a kernel crash avoidance thread is created. In the kernel crash avoidance thread, the system crash test experiment caused by software is first carried out, and the fault caused by system task blocking is triggered to cause the system to crash; then read Get the log code generated when the system crashes,

The log code generated when the system crashes is "kernel:INFO:task xxx:60blocked for morethan 120seconds", and the task process of task xxx shown in the log code is the process that caused the system crash;

Read the task process in the log code, write the task process and the number of task processes into the hung_task function for protection and shielding, so that it will not be monitored;

The kernel crash avoidance thread encapsulates the hung_task function into the Linux kernel, restarts the operating system, re-enters the Linux kernel, and uses the newly compiled Linux kernel to avoid system crashes caused by task blocking.

Encapsulating the hung_task function into the Linux kernel includes the following steps:

#make mrproper Clear all intermediate files generated during the compilation process, including the kernel configuration file ".config" that has been configured in the past, will be cleared, that is, the original old configuration file will be deleted when the new compilation is performed, so as not to affect the new kernel compile;

Through #make clean, clear the object files and executable files with the suffix ".o" generated by the last compilation command;

#make menuconfig#After waiting for a few seconds, the terminal becomes a graphical kernel configuration interface, select the modified function module (hung_task function), and compile the function into the kernel;

#make-j2//Compile the kernel, if the computer is quad-core, use -j4, if the computer is eight-core, you can also use -j8. The larger the number after j, the faster the compilation time will generate the kernel module and vmlinuz, initrd.img, Symtem.map files;

#make modules_install//Install the kernel module. After the compilation is successful, the system will generate a subdirectory under the /lib/modules directory, which stores all the loadable modules of the new kernel (that is, copy the compiled modules to /lib/modules );

make install//Install the kernel, that is, copy .config, vmlinuz, initrd.img, System.map files to the /boot directory, and update grub. For the RedHat system, the following three grub files will be updated automatically, and the new kernel will be started by default.

Among them, the kdump dump memory operation tool is a tool and service used to dump memory operation parameters when the system crashes, deadlocks or crashes.

Kernel Oops Analyzer is a kernel crash analysis tool; the hung task function is a self-protection module used to detect whether there is a process in the system that is in the D state for more than a certain time (the duration can be set), and if it exists, it will Triggering the kernel causes the server to crash and restart; all processes are cyclically detected in the hung_task function.

Linux comes with a watchdog implementation to monitor the operation of the system, including a kernel watchdog module and a user space watchdog program; the Linux kernel watchdog module communicates with the user space through the character device /dev/watchdog, once the user space program Opening the /dev/watchdog device will cause a 1-minute timer to be started in the kernel. After that, the user space program needs to ensure that data is written to this device within 1 minute. Each write operation will cause the timer to be reset , if the user space program has no write operation within 1 minute, the timer will expire and cause a system reboot operation.

In the second aspect, based on the same inventive concept as the Linux system crash control method in the foregoing embodiments, the embodiment of this specification also provides a Linux system crash control system, including: an analysis module, an avoidance module, and a package module;

The analysis module checks the system log, and analyzes the system crash caused by the user, the system crash caused by the hardware or the system crash caused by the software;

Avoid the system crash test experiment caused by the software, trigger the system crash caused by the software; then read the log code generated when the system crashes, read the task process in the log code, write the task process and the number of task processes into the hung_task function Protect and shield from monitoring;

The encapsulation module encapsulates the hung_task function in the avoidance module into the Linux kernel.

In the third aspect, based on the same inventive concept as the Linux system crash control method in the foregoing embodiments, the embodiment of this specification also provides a computer-readable storage medium on which a computer program is stored, and the computer program is executed by the processor Execute the steps of the above-mentioned method for controlling a Linux system crash.

The above is only an embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the description of the present invention and the contents of the accompanying drawings, or directly or indirectly used in other related technologies fields, are all included in the scope of patent protection of the present invention in the same way.

Claims (5)

1. A Linux system crash control method is characterized by comprising the steps of establishing a kernel crash analysis thread and analyzing the cause of system crash; the reasons for system crash include system crash caused by user, system crash caused by hardware and system crash caused by software;

creating a kernel crash avoidance thread, carrying out a test experiment on system crash caused by software, triggering the system crash, reading a log code generated when the system crashes, and writing the log code into a suspended task function;

packaging the suspension task function into a Linux kernel, restarting an operating system, and reentering the Linux kernel;

the log code generated when the system crashes comprises a task process; causing the system to crash when the system runs a task process.

2. The Linux system crash control method of claim 1, wherein: the writing suspension task function comprises the following steps:

reading task processes and the number of the task processes in a log code generated when a system is crashed;

and writing the task processes and the number of the task processes into a suspended task function.

3. The Linux system crash control method of claim 1, wherein: the step of packaging the task suspending function into the Linux kernel comprises the following steps:

clearing a compiling file and a configuration file generated in the Linux kernel compiling process;

clearing object files and executable files generated in the Linux kernel compiling process;

changing a kernel configuration interface into a graphical mode by using an interface command, selecting a suspension task function, and compiling the suspension task function into a Linux kernel;

compiling the Linux kernel through a compiling kernel command;

installing a Linux kernel driver module by using an installation command;

and installing a Linux kernel.

4. A Linux system crash control system, comprising: the device comprises an analysis module, an avoidance module and a packaging module;

the analysis module checks the system log, and analyzes the system crash caused by a user, hardware or software;

the evasion module performs a system crash test experiment caused by software, triggers system crash, reads a log code generated when the system crashes, and writes the log code into a suspended task function;

the encapsulation module encapsulates the suspension function in the avoidance module into a Linux kernel;

the log code generated when the system crashes comprises a task process; causing a system crash when the system runs a task process.

5. A computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to perform the steps of the Linux system crash control method of any one of claims 1-3.

Images