Aix Pstack Core File
Abstract: Abnormal termination of a process will trigger a core dump file. A core dump file is very helpful to programmers or support engineers for determining the root cause of abnormal termination, because it provides invaluable information about the runtime status at crash time. This article provides information about core dumps, as well as features and analysis tools in the Solaris Operating System that can be used to manage core dumps. Note: The information provided in this article is mainly for the Solaris 10 OS. Contents:. When an application process receives a specific signal and terminates, the system generates a core dump and stops the process.
In most cases, the signal leading to the application crash is SIGSEGV or SIGBUS. SIGSEGV indicates that the application is accessing an invalid memory address. This situation often occurs in C/C programs if there are code errors in pointer manipulation. On the Solaris OS, you can use the libumem(3LIB) library as the user-mode memory allocator instead of libc. The libumem library can help find memory leaks, buffer overflows, attempts to use freed data, and many other memory allocation errors. Also, its memory allocator is very fast and scalable with multithreaded applications.
Getting a backtrace from a coredump¶ The best way to get this information from a core dump is by using the ServerDoc tool, described here. Unless there is a problem. Jul 28, 2008 Hi Please let me know if there is any command AIX 5.3 provides, which gives stack trace from Core dump, like pstack on.
SIGBUS indicates that the application is accessing a memory address that does not conform to CPU memory alignment rules. This usually happens to a system with an UltraSPARC processor.
Systems with x86/x64 CPUs can handle unaligned memory addresses, but there is a performance impact. The Sun Studio C/C compiler has the -xmemalign option, which can be used to adjust the behavior of the UltraSPARC CPU when there are unaligned memory addresses that can be determined at compile time. The -xmemalign option causes the compiler to generate additional load/store instructions for unaligned memory access. However, the -xmemalign option cannot handle unaligned memory access during runtime. If unaligned memory access happens during runtime, the developer needs to change the source code.
There are other signals whose default disposition is to create a core dump, for example, SIGFPE, which indicates a floating point exception. The Signal(3HEAD) man page provides more details.
The Solaris OS attempts to create up to three core dump files for each abnormally terminated process. One of the core dump files, which is called the per-process core file, is located in the current directory. Another core dump file, which is called the global core file, is created in the system-wide location. If the process is running in a local zone, a third core file is created in the global zone's location. You can use the coreadm(1M) command to manage the core dumps.
All the settings are saved in the /etc/coreadm.conf configuration file. Below is a typical scenario, which shows the current system configuration for core dumps: -bash-3.00# coreadm global core file pattern: global core file content: default init core file pattern: core init core file content: default global core dumps: disabled per-process core dumps: enabled global setid core dumps: disabled per-process setid core dumps: disabled global core dump logging: disabled In the previous output:. The global core dumps: disabled line indicates no global core dump will be generated. The per-process core dumps: enabled line indicates a per-process core dump will be generated for each abnormal process.
The init core file pattern line indicates the contents will be gathered from the live process to the per-process core dump. You can also use the coreadm command to control the core dump file name: -bash-3.00# coreadm -i core.%f.%p This command causes the per-process core file name to be appended with the program file name (%f) and the runtime process ID (%p). A core dump file will be generated in the current working directory of the process.bash-3.00# coreadm -g /globalcore/core.%f.%p -e global By default, the global core dump is disabled. You need to use the coreadm command with the -e global option to enable it. The -g option causes the command to append the program name (%f) and the runtime process ID (%p) to the core file name. As indicated previously, coreadm can specify the parts of the process that will be saved to the core file. Previously, when you performed a post-mortem analysis, you needed to obtain all the specific versions of the dependent libraries and runtime modules, because the core dump file does not contain this text information.
It is quite a headache for programmers to recreate the environment from the original machine. With the default configuration, the Solaris OS applies the 'default' pattern to each process core dump, which means the process core dump contains stack, heap, text, shared memory (SHM), intimate shared memory (ISM), and dynamic intimate shared memory (DISM) information, plus other information. The text part of the process core dump also contains a partial symbol table (dynsm), which will help you get a readable stack trace directly from one core file without any other boring dependent libraries. If the dynsm is insufficient, you can use coreadm to include all symbol tables, as follows: -bash-3.00# coreadm -G all -I all This previous command makes both the global core file ( -G) and the per-process core file ( -I) contain all the parts of the process.
Here's how to use coreadm to verify the changes: -bash-3.00# coreadm global core file pattern: /globalcore/core.%f.%p global core file content: all init core file pattern: core.%f.%p init core file content: all global core dumps: enabled per-process core dumps: enabled global setid core dumps: disabled per-process setid core dumps: disabled global core dump logging: disabled The coreadm command is used to edit the configuration file of the coreadm service, which is managed by the Service Management Facility (SMF) with this service identifier: svc:/system/coreadm:default. The Solaris OS provides the gcore(1) command in case you need to create a core dump manually for a live process for analysis purposes: -bash-3.00# echo $$ 2770 -bash-3.00# gcore $$ gcore: core.2770 dumped The live process ID is appended automatically to the name of the generated core dump. In the previous example, the process of the current shell is dumped and its process ID is 2770. Note: There are other constraints you need take into account while generating the core dump, for example, the write permissions on the destination directory, the existence of the destination directory, the file system mount option, and process resource limitation. For resource limitation information, refer to the man pages for setrlimit(2) and ulimit(1).
Another useful tool called is available. It automatically collects diagnostic and debugging information when any application crashes under the Solaris OS. This article does not address its usage. For more information on using AppCrash, refer to Greg Nakhimovsky's. There are lots of tools in the Solaris OS for analyzing core dump files: dbx(1), mdb(1), and pstack(1).
The most convenient method is to use the pstack tool to determine the process stack. This tool helps show multithreaded programs as well: -bash-3.00# pstack core.2580 more core 'core.2580' of 2580: javavm - lwp# 1 / thread# 1 - fef40a27 read (b, 804280c, 1) feb11ba8 1cDhpiEread6FipvII (b, 804280c, 1) + a8 feb11aef JVMRead (b, 804280c, 1) + 2f fe77045e???????? (80685b8, 8042864, 22). Feb1d55c jniCallStaticVoidMethod (80685b8, 8069020, 80e8355, 0) + 14c 080516c2 main (2, 8047168, 8047174) + 50c 08050daa????????
(2, 80472cc, 80472d4, 0, 80472d5, 8047301) - lwp# 2 / thread# 2 - fef40d27 lwpcondwait (8067ae8, 8067ad0, fb3a9c08, 0) fef2de3f lwpcondtimedwait (8067ae8, 8067ad0, fb3a9c50) + 35. Fef3fc32 thrsetup (fef82400) + 4e fef3ff20 lwpstart (fef82400, 0, 0, fb3a9ff8, fef3ff20, fef82400) - lwp# 3 / thread# 3 - fef40d27 lwpcondwait (8116588, 8116570, 0, 0) feab737c 1cCosHSolarisFEventEpark6Mv (8116548) + 4c. In general, if the program's symbol table is not stripped and its runtime stack trace is available, you can expect almost 50 percent of the problems to be resolved.
Dbx is a free source-level debug tool provided by Sun Studio software. Sun Studio software includes free, optimizing C, C, and Fortran compilers that can be used on both the Solaris OS and Linux. Dbx not only helps you inspect the state of your program, but it also collects the program performance data. Here is a typical scenario for analyzing the core file using dbx. For more details on dbx, please refer to the document called. When detecting whether the integrity of data was corrupted or whether a fatal error in hardware occurred, the Solaris OS invokes panic.
The panic routine interrupts all processes as if the OS is suspended. Then it generates a system core dump, which is a copy of OS in the memory, and saves it to the dump device. After a crash, the OS use savecore(1) to retrieve the core dump from the dump device to the savecore directory during the next boot. The savecore routine generates two files.
Give More Feedback
One file is unix., which is an OS symbol table list, and the other is vmcore., which is the core dump data file. By default, the dump device is a swap disk partition and the savcore directory is set to /var/crash/. The trailing in the file names is an integer that grows every time savecore runs. You can use dumpadm(1M) to manage dump devices and the savecore directory: -bash-3.00# dumpadm -d /dump -s /savecore Dump content: kernel pages Dump device: /dump (dedicated) Savecore directory: /savecore Savecore enabled: yes To verify this or see the current configuration, you can run only dumpadm: -bash-3.00# dumpadm Dump content: kernel pages Dump device: /dump (dedicated) Savecore directory: /savecore Savecore enabled: yes You can also use dumpadm to set the dump content and enable savecore(1) operation during the boot. All the configuration information is saved in the /etc/dumpadm.conf configure file. The system crash dump service is also managed by SMF with this service identifier: svc:/system/dumpadm:default. In some cases, you need to save a core dump manually to take a snapshot of the live system.
In the Solaris OS, there are several means you can use. For example, you can use reboot -d to force the generation of a core dump with reboot. Or you can use savecore -L to create a live OS core dump.
If you want to use savecore(1M) to create a live core dump, you must use dumpadm to set a non-swap device as the dump device, because live core dumps take a swap device as a part of virtual memory, which is to be dumped. Sometimes, the system will hang without crashing. If you are using a Sun UltraSPARC processor-based machine, you can press Stop-A to run in OpenBoot PROM (OBP) mode, and then use the sync OBP command to force a crash core dump. On x86 platforms, there is no corresponding OBP unit. However, you can use kmdb(1M). To use kmdb to create a core dump, you need load its module during system booting. Here are the steps for the Solaris 10 1/06 OS or later.
Edit the /boot/grub/menu.lst file and append the -k string to the initrd line, as follows:. title Solaris 10 11/06 s10xu3wos10 X86 root (hd0,1,a) kernel /platform/i86pc/multiboot -k module /platform/i86pc/bootarchive This will make the OS boot with kmdb. Then restart the machine manually. Alternatively, if you are using the Solaris 10 GA OS, just enter b -k when you see the Select (B)oot or (I)nterpreter: system prompt during the system boot stage. After performing these steps, press F1-A to break the system to kmdb. This action must be performed in console mode, because kmdb suspends the system and GUI applications. If you are using a desktop system, the Solaris OS will fail to switch to console mode and your desktop will appear to hang.
See More On Stackoverflow
However kmdb is running and you can still type commands. This article cannot provide solutions for fixing a system core dump, because such an analysis requires much low-level computing knowledge of the OS kernel and also of the hardware. However here are some basic guidelines for your reference:. Check the system console and the /var/adm/messages file, because they contain valuable information for identifying the problem that the system encountered.
Use the strings(1) command to process the core dump file. This command prints out the ASCII strings in any binary file, including a core dump file. You need to look at these ASCII strings. Check the error you encounter on the, or use the free Solaris Crash Analysis Tool (CAT) to help you investigate further.