The gnu debugger (gdb) is probably the best tool for looking into core files. It also isn't overly complex to use to get some basic starting information. So this is a quick guide to getting some debug information eg variable's and stack traces from a core dump which is formed when an application crashes in linux.

If an application crashes and doesn't produce a core file it is probably because of the limit settings you can check and enable core dumps by using the following "ulimit -c" if it outputs a 0 it will not produce a core. You can use "ulimit -c unlimited" to make the core dump file size unlimited. Be aware though that if you have a lot of crashes it can use a significant amount of disk space.

Let start with finding out what made a core file in the first place. In order to debug it at all you need to know exactly what program crashed. You can determin this using the following command.

cat core |strings |grep -E '^_='
_=./willcore.exe

In this case the "./willcore.exe" made the core dump. Another way to find out what core'd is to use gdb. The example is below

gdb --core core

Core was generated by `./willcore.exe'.
Program terminated with signal 11, Segmentation fault.
#0  0x080483d4 in ?? ()
(gdb)

Something to notice at this point is that is shows where the last execution point was. In this case it was running at memory location 0x080483d4. However since there are no debugging symbols loaded in gdb yet it shows ?? because it cannot translate the raw address to a function.

You can get gdb to load the executable and debugging symbols (assuming they are compiled into the executable) using the "file" command. I have also added the "bt" command to produce a back trace so show the execution stack.

(gdb) file ./willcore.exe
Reading symbols from /home/james/CVS-Root/linux/misc/willcore.exe...done.
(gdb) bt
#0  0x080483d4 in main (argc=1, argv=0xbf877ef4) at willcore.c:8

In the example above the symbols now resolve and will show a lot more information. If they do not show after loading the executable into gdb it will be a problem with debugging symbols. You will need to go and compile the program with debugging switched on eg the "-ggdb" flag in gcc and g++.

Now that things are loaded you can move between stack frame's using "frame <number>" where number is the part beside the # in the stack output and you can also print and inspect other parts of memory as well as list the source code from the program assuming the source still exists in the original location that it was compiled from.

(gdb) p argc
$1 = 1
(gdb) p argv[0]
$2 = 0xbf87988c "./willcore.exe"
(gdb) p argv[1]
$3 = 0x0
(gdb) list
1
2
3       #include <stdio.h>
4
5       int main(int argc, char **argv) {
6               char *tmp = 0;
7
8               *tmp = '0';
9
10
(gdb) p tmp
$4 = 0x0

Note: the above example shows how to produce a core file by dereferencing a null pointer in c which in its self can be useful to fore a core dump which is what I used for this tutorial.

As a final example the following will dump all the stacks from all threads that were running in a process at the time it crashed. Though in the sample program there only is one thread. In a large application this could provide several pages of output.

(gdb) thread apply all bt

Thread 1 (Thread 12519):
#0  0x080483d4 in main (argc=1, argv=0xbf877ef4) at willcore.c:8
(gdb)

If you have an application that crashes with a "segment fault" or other error error message like SIGBUS or SIGILL. There are an indication that a serious problem has occured inside the application that was running. eg a segment fault means that the application tried to access memory that is not allocated to that process. Or a sig ill is an illegal instruction and this occurs when a program jumps into random memory and the processor cannot understand the instruction at the current address that it is running at.

If the program didn't actually dump a core there can be a number of reasons for this. Typically because the ulimit value is set to 0. You can check this by running the command "ulimit -c" and it will show you the maximum size a core dump can be. You can set this to unliminted by using the command "ulimit -c unlimited" this will typically cause a core to be dumped. There are some other security releated reason why a core will not be dumped. An example is if an executable is marked suid it won't dump a core by default.

Many people always wonder what a core file is and a really simple way to explain it is that it contains the core memory of the application that crashed. Though it is in a format simalar the elf executable file format. This provides information about memory region and their contents. You can show the contents of a core file using the objdump utility.

eg the output of the command "objdump -x core" will show something like this.

core:     file format elf32-i386
core
architecture: i386, flags 0x00000000:

start address 0x00000000

Program Header:
    NOTE off    0x00000294 vaddr 0x00000000 paddr 0x00000000 align 2**0
         filesz 0x0000022c memsz 0x00000000 flags ---
    LOAD off    0x00001000 vaddr 0x08048000 paddr 0x00000000 align 2**12
         filesz 0x00001000 memsz 0x00001000 flags r-x
    LOAD off    0x00002000 vaddr 0x08049000 paddr 0x00000000 align 2**12
         filesz 0x00001000 memsz 0x00001000 flags rw-
    LOAD off    0x00003000 vaddr 0xb75f9000 paddr 0x00000000 align 2**12
         filesz 0x00002000 memsz 0x00002000 flags rw-
    LOAD off    0x00005000 vaddr 0xb75fb000 paddr 0x00000000 align 2**12
         filesz 0x00001000 memsz 0x00140000 flags r-x
    LOAD off    0x00006000 vaddr 0xb773b000 paddr 0x00000000 align 2**12
         filesz 0x00000000 memsz 0x00001000 flags ---
    LOAD off    0x00006000 vaddr 0xb773c000 paddr 0x00000000 align 2**12
         filesz 0x00002000 memsz 0x00002000 flags r--
    LOAD off    0x00008000 vaddr 0xb773e000 paddr 0x00000000 align 2**12
         filesz 0x00001000 memsz 0x00001000 flags rw-
    LOAD off    0x00009000 vaddr 0xb773f000 paddr 0x00000000 align 2**12
         filesz 0x00003000 memsz 0x00003000 flags rw-
    LOAD off    0x0000c000 vaddr 0xb7742000 paddr 0x00000000 align 2**12
         filesz 0x00001000 memsz 0x00015000 flags r-x
    LOAD off    0x0000d000 vaddr 0xb7757000 paddr 0x00000000 align 2**12
         filesz 0x00001000 memsz 0x00001000 flags r--
    LOAD off    0x0000e000 vaddr 0xb7758000 paddr 0x00000000 align 2**12
         filesz 0x00001000 memsz 0x00001000 flags rw-
    LOAD off    0x0000f000 vaddr 0xb7759000 paddr 0x00000000 align 2**12
         filesz 0x00002000 memsz 0x00002000 flags rw-
    LOAD off    0x00011000 vaddr 0xb7774000 paddr 0x00000000 align 2**12
         filesz 0x00002000 memsz 0x00002000 flags rw-
    LOAD off    0x00013000 vaddr 0xb7776000 paddr 0x00000000 align 2**12
         filesz 0x00001000 memsz 0x00001000 flags r-x
    LOAD off    0x00014000 vaddr 0xb7777000 paddr 0x00000000 align 2**12
         filesz 0x00001000 memsz 0x0001b000 flags r-x
    LOAD off    0x00015000 vaddr 0xb7792000 paddr 0x00000000 align 2**12
         filesz 0x00001000 memsz 0x00001000 flags r--
    LOAD off    0x00016000 vaddr 0xb7793000 paddr 0x00000000 align 2**12
         filesz 0x00001000 memsz 0x00001000 flags rw-
    LOAD off    0x00017000 vaddr 0xbfe74000 paddr 0x00000000 align 2**12
         filesz 0x00016000 memsz 0x00016000 flags rw-

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 note0         0000022c  00000000  00000000  00000294  2**0
                  CONTENTS, READONLY
  1 .reg/12315    00000044  00000000  00000000  000002f0  2**2
                  CONTENTS
  2 .reg          00000044  00000000  00000000  000002f0  2**2
                  CONTENTS
  3 .auxv         000000a0  00000000  00000000  000003dc  2**2
                  CONTENTS
  4 load1         00001000  08048000  00000000  00001000  2**12
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  5 load2         00001000  08049000  00000000  00002000  2**12
                  CONTENTS, ALLOC, LOAD
  6 load3         00002000  b75f9000  00000000  00003000  2**12
                  CONTENTS, ALLOC, LOAD
  7 load4a        00001000  b75fb000  00000000  00005000  2**12
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  8 load4b        00000000  b75fc000  00001000  00006000  2**12
                  ALLOC, READONLY, CODE
  9 load5         00000000  b773b000  00000000  00006000  2**12
                  ALLOC, READONLY
 10 load6         00002000  b773c000  00000000  00006000  2**12
                  CONTENTS, ALLOC, LOAD, READONLY
 11 load7         00001000  b773e000  00000000  00008000  2**12
                  CONTENTS, ALLOC, LOAD
 12 load8         00003000  b773f000  00000000  00009000  2**12
                  CONTENTS, ALLOC, LOAD
 13 load9a        00001000  b7742000  00000000  0000c000  2**12
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
 14 load9b        00000000  b7743000  00001000  0000d000  2**12
                  ALLOC, READONLY, CODE
 15 load10        00001000  b7757000  00000000  0000d000  2**12
                  CONTENTS, ALLOC, LOAD, READONLY
 16 load11        00001000  b7758000  00000000  0000e000  2**12
                  CONTENTS, ALLOC, LOAD
 17 load12        00002000  b7759000  00000000  0000f000  2**12
                  CONTENTS, ALLOC, LOAD
 18 load13        00002000  b7774000  00000000  00011000  2**12
                  CONTENTS, ALLOC, LOAD
 19 load14        00001000  b7776000  00000000  00013000  2**12
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
 20 load15a       00001000  b7777000  00000000  00014000  2**12
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
 21 load15b       00000000  b7778000  00001000  00015000  2**12
                  ALLOC, READONLY, CODE
 22 load16        00001000  b7792000  00000000  00015000  2**12
                  CONTENTS, ALLOC, LOAD, READONLY
 23 load17        00001000  b7793000  00000000  00016000  2**12
                  CONTENTS, ALLOC, LOAD
 24 load18        00016000  bfe74000  00000000  00017000  2**12
                  CONTENTS, ALLOC, LOAD
SYMBOL TABLE:
no symbols

The above information isn't really that useful but does provide a memory map with the permissions of the different regions of the application when it crashed. This can become useful later when debugging things. Like in C/C++ when things are declared "const" they are loaded into a read-only section of memory so any attempt to write to that area will fail and cause a "segment fault"

Something that is more usful is with the following command you can determine which application crashed by simply reading all the string for the executable and looking for the '_=' string.

cat core |strings |grep -E '^_='
_=./willcore.exe

You may also be interested in my next post which uses gdb to read a core file