Retrieving stack traces/backtraces from Apache/PHP crashes on RedHat Linux

This post is effectively an amalgamation of 3 documents:

• How can I download or install debuginfo packages for RHEL systems? 
• Installing Debuginfo Packages 
• PHP - Generating a gdb backtrace

The notes above deal with pieces of the puzzle to generate stack traces/backtraces of crashing PHP code; and this note effectively takes the more holistic view to getting it done.

Evidence that crashes occur is typically located in the /var/log/httpd/error_log* files - e.g. entries like:

[timestamp] [core:notice] [pid 7528] AH00052: child pid 11172 exit signal Segmentation fault (11)

[timestamp] [core:notice] [pid 7528] AH00052: child pid 11063 exit signal Segmentation fault (11)

In summary, it is possible to retrieve stack traces of PHP code crashes that cause the Apache process to terminate (e.g. Segmentation fault). This is achieved with the GNU Debugger (gdb). Note that the one piece of the puzzle that is not covered is the requirement for the yum-utils package, which contains the debuginfo-install command.

To generate the stack trace we use gdb to start/run httpd in debug mode (-X).

First things first, we need to install various packages - please do so only on a desktop or dev/test server that can suffer a period of outage:

sudo yum install gdb yum-utils kernel-debuginfo kernel-debuginfo-common

When you first run gdb it will tell you if it has the requisite debug information - and if not it will give you instructions to install:

In our case, when running "gdb /sbin/httpd" as per below we were notified to run:


sudo debuginfo-install httpd-2.4.6-67.el7_4.2.x86_64

We exit'ed gdb (q + ENTER), installed above and then ran gdb again as per below.


sudo -s

# Stop apache (e.g. on a dev/test environment)
service httpd stop

# Run gdb
gdb /sbin/httpd

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/sbin/httpd...Reading symbols from /usr/lib/debug/usr/sbin/httpd.debug...done.
done.

(gdb) run -X
Starting program: /sbin/httpd -X
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Detaching after fork from child process 10363.
Detaching after fork from child process 10364.
Detaching after fork from child process 10365.

# Now run the request through the client to the Apache server, which causes the crash:

Program received signal SIGSEGV, Segmentation fault.
0x00007fffe8b36c49 in _zval_ptr_dtor () from /usr/local/zend/lib/apache2/libphp7.so
 
# Produce the backtrace:
(gdb) bt
#0  0x00007fffe8b36c49 in _zval_ptr_dtor () from /usr/local/zend/lib/apache2/libphp7.so
#1  0x00007fffd9ada5b1 in php_oci_bind_hash_dtor () from /usr/local/zend/lib/php_extensions/oci8.so
#2  0x00007fffe8b57bb2 in zend_hash_destroy () from /usr/local/zend/lib/apache2/libphp7.so
#3  0x00007fffd9ae552b in php_oci_statement_free () from /usr/local/zend/lib/php_extensions/oci8.so
...

As can be seen above, the culprit is the function zval_ptr_dtor, which is called from the OCI driver/interface (Oracle Call Interface) and is related to freeing up the OCI statement.

If you have support with e.g. Zend/Roguewave it is now possible to raise a support request with them and include the stack trace.

Alternatively, one could do a search on the crashing function via https://bugs.php.net for any outstanding bugs and/or file a new bug request.