Postfix Debugging Howto (2024)

Purpose of this document

This document describes how to debug parts of the Postfix mailsystem when things do not work according to expectation. The methodsvary from making Postfix log a lot of detail, to running some daemonprocesses under control of a call tracer or debugger.

The text assumes that the Postfix main.cf and master.cfconfiguration files are stored in directory /etc/postfix. You canuse the command "postconf config_directory" to find out theactual location of this directory on your machine.

Listed in order of increasing invasiveness, the debuggingtechniques are as follows:

Look for obvious signs of trouble
Debugging Postfix from inside
Try turning off chroot operation inmaster.cf
Verbose logging for specific SMTPconnections
Record the SMTP session with a networksniffer
Making Postfix daemon programs more verbose
Manually tracing a Postfix daemon process
Automatically tracing a Postfix daemonprocess
Running daemon programs with the interactiveddd debugger
Running daemon programs with the interactivegdb debugger
Running daemon programs under a non-interactivedebugger
Unreasonable behavior
Reporting problems to postfix-users@postfix.org

Look for obvious signs of trouble

Postfix logs all failed and successful deliveries to a logfile.

When Postfix uses syslog logging (the default), the fileis usually called /var/log/maillog, /var/log/mail, or somethingsimilar; the exact pathname is configured in a file called/etc/syslog.conf, /etc/rsyslog.conf, or something similar.
When Postfix uses its own logging system (see MAILLOG_README),the location of the logfile is configured with the Postfix maillog_fileparameter.

When Postfix does not receive or deliver mail, the first orderof business is to look for errors that prevent Postfix from workingproperly:

% grep -E '(warning|error|fatal|panic):' /some/log/file | more

Note: the most important message is near the BEGINNING of theoutput. Error messages that come later are less useful.

The nature of each problem is indicated as follows:

"panic" indicates a problem in the software itselfthat only a programmer can fix. Postfix cannot proceed until thisis fixed.
"fatal" is the result of missing files, incorrectpermissions, incorrect configuration file settings that you canfix. Postfix cannot proceed until this is fixed.
"error" reports an error condition. For safetyreasons, a Postfix process will terminate when more than 13 of thesehappen.
"warning" indicates a non-fatal error. These areproblems that you may not be able to fix (such as a broken DNSserver elsewhere on the network) but may also indicate localconfiguration errors that could become a problem later.

Debugging Postfix from inside

Postfix version 2.1 and later canproduce mail delivery reports for debugging purposes. These reportsnot only show sender/recipient addresses after address rewritingand alias expansion or forwarding, they also show information aboutdelivery to mailbox, delivery to non-Postfix command, responsesfrom remote SMTP servers, and so on.

Postfix can produce two types of mail delivery reports fordebugging:

What-if: report what would happen, but do not actuallydeliver mail. This mode of operation is requested with:
```
% /usr/sbin/sendmail -bv address...Mail Delivery Status Report will be mailed to <your login name>.
```
What happened: deliver mail and report successes and/orfailures, including replies from remote SMTP servers. This modeof operation is requested with:
```
% /usr/sbin/sendmail -v address...Mail Delivery Status Report will be mailed to <your login name>.
```

These reports contain information that is generated by Postfixdelivery agents. Since these run as daemon processes that cannotinteract with users directly, the result is sent as mail to thesender of the test message. The format of these reports is practicallyidentical to that of ordinary non-delivery notifications.

For a detailed example of a mail delivery status report, seethe debuggingsection at the end of the ADDRESS_REWRITING_README document.

Try turning off chroot operation in master.cf

A common mistake is to turn on chroot operation in the master.cffile without going through all the necessary steps to set up achroot environment. This causes Postfix daemon processes to faildue to all kinds of missing files.

The example below shows an SMTP server that is configured withchroot turned off:

Verbose logging for specific SMTPconnections

In /etc/postfix/main.cf, list the remote site name or addressin the debug_peer_list parameter. For example, in order to makethe software log a lot of information to the syslog daemon forconnections from or to the loopback interface:

/etc/postfix/main.cf: debug_peer_list = 127.0.0.1

You can specify one or more hosts, domains, addresses ornet/masks. To make the change effective immediately, execute thecommand "postfix reload".

Record the SMTP session with a network sniffer

This example uses tcpdump. In order to record a conversationyou need to specify a large enough buffer with the "-s"option or else you will miss some or all of the packet payload.

# tcpdump -w /file/name -s 0 host example.com and port 25

Older tcpdump versions don't support "-s 0"; in that case,use "-s 2000" instead.

Run this for a while, stop with Ctrl-C when done. To view thedata use a binary viewer, ethereal, or good old less.

Making Postfix daemon programs more verbose

Append one or more "-v" options to selected daemondefinitions in /etc/postfix/master.cf and type "postfix reload".This will cause a lot of activity to be logged to the syslog daemon.For example, to make the Postfix SMTP server process more verbose:

/etc/postfix/master.cf: smtp inet n - n - - smtpd -v

To diagnose problems with address rewriting specify a "-v"option for the cleanup(8) and/or trivial-rewrite(8) daemon, and todiagnose problems with mail delivery specify a "-v"option for the qmgr(8) or oqmgr(8) queue manager, or for the lmtp(8),local(8), pipe(8), smtp(8), or virtual(8) delivery agent.

Manually tracing a Postfix daemon process

Many systems allow you to inspect a running process with asystem call tracer. For example:

# trace -p process-id (SunOS 4)# strace -p process-id (Linux and many others)# truss -p process-id (Solaris, FreeBSD)# ktrace -p process-id (generic 4.4BSD)

Even more informative are traces of system library calls.Examples:

# ltrace -p process-id (Linux, also ported to FreeBSD and BSD/OS)# sotruss -p process-id (Solaris)

See your system documentation for details.

Tracing a running process can give valuable information aboutwhat a process is attempting to do. This is as much information asyou can get without running an interactive debugger program, asdescribed in a later section.

Automatically tracing a Postfix daemonprocess

Postfix can attach a call tracer whenever a daemon processstarts. Call tracers come in several kinds.

System call tracers such as trace, truss,strace, or ktrace. These show the communicationbetween the process and the kernel.
Library call tracers such as sotruss and ltrace.These show calls of library routines, and give a better idea ofwhat is going on within the process.

Append a -D option to the suspect command in/etc/postfix/master.cf, for example:

/etc/postfix/master.cf: smtp inet n - n - - smtpd -D

Edit the debugger_command definition in /etc/postfix/main.cfso that it invokes the call tracer of your choice, for example:

/etc/postfix/main.cf: debugger_command = PATH=/bin:/usr/bin:/usr/local/bin; (truss -p $process_id 2>&1 | logger -p mail.info) & sleep 5

Type "postfix reload" and watch the logfile.

Running daemon programs with the interactiveddd debugger

If you have X Windows installed on the Postfix machine, thenan interactive debugger such as ddd can be convenient.

Edit the debugger_command definition in /etc/postfix/main.cfso that it invokes ddd:

Running daemon programs with the interactivegdb debugger

If you have the screen command installed on the Postfix machine, thenyou can run an interactive debugger such as gdb as follows.

Edit the debugger_command definition in /etc/postfix/main.cfso that it runs gdb inside a detached screen session:

/etc/postfix/main.cf: debugger_command = PATH=/bin:/usr/bin:/sbin:/usr/sbin; export PATH; HOME=/root; export HOME; screen -e^tt -dmS $process_name gdb $daemon_directory/$process_name $process_id & sleep 2

Be sure that gdb is in the command search path.

Append a -D option to the suspect daemon definition in/etc/postfix/master.cf, for example:

/etc/postfix/master.cf: smtp inet n - n - - smtpd -D

Execute the command "postfix reload" and wait until adaemon process is started (you can see this in the maillog file).

Then attach to the screen, and debug away:

# HOME=/root screen -rgdb) continuegdb) where

Running daemon programs under a non-interactivedebugger

If you do not have X Windows installed on the Postfix machine,or if you are not familiar with interactive debuggers, then youcan try to run gdb in non-interactive mode, and have itprint a stack trace when the process crashes.

Edit the debugger_command definition in /etc/postfix/main.cfso that it invokes the gdb debugger:

/etc/postfix/main.cf: debugger_command = PATH=/bin:/usr/bin:/usr/local/bin; export PATH; (echo cont; echo where; sleep 8640000) | gdb $daemon_directory/$process_name $process_id 2&gt&1 >$config_directory/$process_name.$process_id.log & sleep 5

Append a -D option to the suspect daemon in/etc/postfix/master.cf, for example:

/etc/postfix/master.cf: smtp inet n - n - - smtpd -D

Type "postfix reload" to make the configuration changeseffective.

Whenever a suspect daemon process is started, an output fileis created, named after the daemon and process ID (for example,smtpd.12345.log). When the process crashes, a stack trace (withoutput from the "where" command) is written to its logfile.

Unreasonable behavior

Sometimes the behavior exhibited by Postfix just does not match thesource code. Why can a program deviate from the instructions givenby its author? There are two possibilities.

The compiler has erred. This rarely happens.
The hardware has erred. Does the machine have ECC memory?

In both cases, the program being executed is not the programthat was supposed to be executed, so anything could happen.

There is a third possibility:

Bugs in system software (kernel or libraries).

Hardware-related failures usually do not reproduce in exactlythe same way after power cycling and rebooting the system. There'slittle Postfix can do about bad hardware. Be sure to use hardwarethat at the very least can detect memory errors. Otherwise, Postfixwill just be waiting to be hit by a bit error. Critical systemsdeserve real hardware.

When a compiler makes an error, the problem can be reproducedwhenever the resulting program is run. Compiler errors are mostlikely to happen in the code optimizer. If a problem is reproducibleacross power cycles and system reboots, it can be worthwhile torebuild Postfix with optimization disabled, and to see if optimizationmakes a difference.

In order to compile Postfix with optimizations turned off:

% make tidy% make makefiles OPT=

This produces a set of Makefiles that do not request compileroptimization.

Once the makefiles are set up, build the software:

% make% suPassword:# make install

If the problem goes away, then it is time to ask your vendorfor help.

Reporting problems to postfix-users@postfix.org

The people who participate on postfix-users@postfix.orgare very helpful, especially if YOU provide them with sufficientinformation. Remember, these volunteers are willing to help, buttheir time is limited.

When reporting a problem, be sure to include the followinginformation.

A summary of the problem. Please do not just send somelogging without explanation of what YOU believe is wrong.
Complete error messages. Please use cut-and-paste, or useattachments, instead of reciting information from memory.
Postfix logging. See the text at the top of the DEBUG_READMEdocument to find out where logging is stored. Please do not frustratethe helpers by word wrapping the logging. If the logging is morethan a few kbytes of text, consider posting an URL on a web or ftpsite.
Consider using a test email address so that you don't haveto reveal email addresses or passwords of innocent people.
If you can't use a test email address, please anonymizeemail addresses and host names consistently. Replace each letterby "A", each digitby "D" so that the helpers can still recognize syntactical errors.
Command output from:
- "postconf -n". Please do not send your main.cf file,or 1000+ lines of postconf command output.
- "postconf -Mf" (Postfix 2.9 or later).
Better, provide output from the postfinger tool.This can be found at https://github.com/ford--prefect/postfinger.
If the problem is SASL related, consider including theoutput from the saslfinger tool. This can be found athttps://packages.debian.org/search?keywords=sasl2-bin.
If the problem is about too much mail in the queue, considerincluding output from the qshape tool, as described in theQSHAPE_README file.
If the problem is protocol related (connections time out,or an SMTP server complains about syntax errors etc.) considerrecording a session with tcpdump, as described in the DEBUG_README document.