Foreword

One should note that X is responsible for VT switching, meaning switching between an X session and console terminals. In other words, Ctrl+Alt+Fn is handled by X. If X is stopped, for example because it’s running under gdb, one can no longer switch to another VT. That’s why we’re recommending using a second machine to debug X. Nevertheless, here are some instructions to attempt debugging X with a single machine.

One-machine approach

This is a post-mortem approach. The idea is to run X with the -core option. Once it dies, a core file (/etc/X11/core) is generated, and can be loaded from gdb.

Follow these steps:

  1. Getting a core file.

  2. Loading a core file.

  3. Displaying/saving a backtrace.

Two-machine approach

You pay the “need a second machine” price, but that buys you interactivity. Just log into the first machine from the second one, using ssh.

Follow these steps:

  1. Attaching/Starting X from gdb.

  2. Displaying/saving a backtrace.

Needed packages

Obviously, gdb is needed; xserver-xorg-core-dbg contains the debugging symbols for the server itself. Other needed packages can be determined by looking at the backtrace. FIXME: More info about that.

Actual gdb work

Getting a core file

  • Using gdm3 3.4.1 and above: uncomment Enable = true in the [debug] section of the /etc/gdm3/daemon.conf file.

  • Using an older gdm3 package: The idea is to tweak the daemon’s LocalXserverCommand setting, adding the -core option. As of gdm3 2.30, the defaults can be found in /usr/share/gdm/gdm.schemas. Sample /etc/gdm3/daemon.conf excerpt: 

[daemon]
LocalXserverCommand=/usr/bin/Xorg -br -verbose -audit 0 -novtswitch -core

  • Using kdm: One should look for the ServerArgsLocal variable in the /etc/kde4/kdm/kdmrc file, and add -core there. Example: 

ServerArgsLocal=-br -nolisten tcp -core

  • Using xdm: It’s sufficient to add -core to the command configured through /etc/X11/xdm/Xservers, for example: 

:0 local /usr/bin/X :0 vt7 -nolisten tcp -core

Loading a core file

That’s trivial, one just needs to pass both the core file and the path to the binary:

# gdb -c /etc/X11/core /usr/bin/Xorg

Now gdb is ready to display backtraces.

Attaching X from gdb

The way of starting X doesn’t really matter, as gdb makes it possible to attach a running process. If there’s a single X instance running, that will do the job:

# gdb attach $(pidof X)
[---GDB starts---]
(gdb) handle SIGPIPE nostop
(gdb) cont

If there are several instances, one can use ps aux to determine the PID of the appropriate instance (2nd column → $pid), and then attach it:

# gdb attach $pid
[---GDB starts---]
(gdb) handle SIGPIPE nostop
(gdb) cont

Starting X from gdb

In case X crashes at start-up, one can start X from gdb in the following way. In this example, the only parameter is the display, but more parameters can be added.

# gdb --args Xorg :0
[---GDB starts---]
(gdb) handle SIGPIPE nostop
(gdb) run

What is SIGPIPE?

SIGPIPE is a signal that can reach the X server quite easily, especially when performing a VT switch, or refreshing large parts of the screen. That’s why we ask gdb not to stop when such a signal is caught, thanks to the handle SIGPIPE nostop command.

How to display a backtrace?

Once X is crashed, for example because it received a SIGSEGV (segmentation fault, usually because of a NULL pointer dereference), or a SIGBUS, one gets back to the gdb prompt. One can then request a backtrace (bt) or a full backtrace (bt full). The latter is what developers are usually interested in, because variable values are also available.

(gdb) bt
(gdb) bt full

How to save a backtrace?

To save a recording of the gdb session to a file (gdb.txt by default):

(gdb) set logging on

To save in a different file, use this instead:

(gdb) set logging file my-file.txt
(gdb) set logging on

Once logging is enabled, you can request a (full) backtrace using the previous commands.