     [file version.texi not included]
     [file rendition.texi not included]
     [file macros.texi not included]
     [file values.texi not included]

Published by the Free Software Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA

Copyright © 2005, 2006, 2007 Sergey Poznyakoff

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, with the Front-Cover texts being ``Mailfromd Manual'', and with the Back-Cover Texts as in (a) below.  A copy of the license is included in the section entitled ``GNU Free Documentation License''.

(a) The FSF's Back-Cover Text is: ``You have freedom to copy and modify this GNU Manual, like GNU software.  Copies published by the Free Software Foundation raise funds for GNU development.''

     Dedico aquest treball a Lluis Llach, per obrir els nous horitzons.

Mailfromd
*********

This edition of the Mailfromd Manual, last updated UPDATED, documents mailfromd Version VERSION.

Preface
*******

Simple Mail Transfer Protocol (SMTP) which is the standard for e-mail transmissions across the internet was designed in the good old days when nobody could even think of the possibility of e-mail being abused to send tons of unsolicited messages of dubious contents.  Therefore it lacks mechanisms that could have prevented this abuse (spamming), or at least could have made it difficult.  Attempts to introduce such mechanisms (such as SMTP-AUTH extension (http://tools.ietf.org/html/rfc2554)) are being made, but they are not in wide use yet and, probably, their introduction will not be enough to stop the e-mail abuse.  Spamming is today's grim reality and developers spend lots of time and efforts designing new protection measures against it. Mailfromd is one of such attempts.

The utility is designed to work with any MTA supporting Milter protocol, such as Sendmail or Postfix.  It allows you to:

  * Control whether messages come from trustworthy senders, using so called callout or Sender Address Verification (see SAV) mechanism.

  * Prevent emails coming from forged addresses by use of SPF mechanism (see SPF Functions).

  * Limit connection and/or sending rates (see Rate Limit).

  * Use black-, white- and greylisting techniques.

  * Invoke external programs or other mail filters.

Short history of mailfromd.
===========================

The idea of the utility appeared in 2005, and its first version appeared soon afterwards.  Back then it was a simple implementation of Sender Address Verification (see SAV) for Sendmail (hence its name -- mailfromd) with rudimentary tuning possibilities.

After a short run on my mail servers, I discovered that the utility was not flexible enough.  It took less than a month to implement a configuration file that allowed to control program and data flow during the envfrom SMTP state.  The new version, 1.0, appeared in June, 2005.

The next major release, 1.2 (1.1 contained mostly bugfixes), appeared two months later, which introduced mail sending rate control (see Rate Limit).

The program evolved during the next year, which led to the release of version 2.0 in September, 2006.  This version was a major change in the main idea of the program.  Configuration file become a flexible filter script allowing to control almost all SMTP states.  The program supplied in the script file was compiled into a pseudo-code at startup, this code being subsequently evaluated each time the filter was invoked.  This caused a considerable speed-up in comparison with the previous versions, where the run-time evaluator was traversing the parse tree.  This version also introduced (implicitly, at the time), two separate data types for the entities declared in the script, which also played its role in the speed improvement (in the previous versions all data were considered strings).  Lots of improvements were made in the filter language (MFL, see MFL) itself, such as user-defined functions, switch statement, catch statement that allows to handle run-time errors, etc.  The set of built-in functions extended considerably.  A testsuite (using DejaGNU) was introduced in this version.

During this initial development period the limitations imposed by libmilter implementation became obvious.  Finally, I felt they were stopping further development, and decided that mailfromd should use its own Milter implementation.  This new library, libgacopyz was the main new feature of the 3.0 release, that was released in November, 2006. Another major feature was the --dump-macros option and macros to rc.mailfromd script, that were intended to facilitate the configuration on Sendmail side.

The development of 3.x (more propery, 3.1.x) series concentrated mainly on bug-fixes, while the main development was done on the next branch.

The version 4.0 appeared on May 12, 2007.  The full list of changes in this release is more than 500 lines long, so it is impractical to list them here.  In particular, this version introduced lots of new features in MFL syntax and the library of useful MFL functions.  The runtime engine was also improved, in particular, stack space become expandable which eliminated many run-time errors.  This version also introduced the beginnings of the MFL module system.  The code generation was re-implemented to facilitate the introduction of object files in future versions.  Another new features in this release include SPF support and mtasim utility, an MTA simulator designed for testing mailfromd scripts (see mtasim).  The test suite in this version was made portable by rewriting it in Autotest.

Typographical conventions
=========================

This manual is written using Texinfo, the GNU documentation formatting language.  The same set of Texinfo source files is used to produce both the printed and online versions of the documentation. This section briefly documents the typographical conventions used in this manual.

Examples you would type at the command line are preceded by the common shell primary prompt, $.  The command itself is printed in this font, and the output it produces in this font, for example:

     $ mailfromd --version
     mailfromd (mailfromd VERSION)

In the text, the command names are printed like this, command line options are displayed in this font.  Some notions are emphasized like this, and if a point needs to be made strongly, it is done this way.  The first occurrence of a new term is usually its definition and appears in the same font as the previous occurrence of ``definition'' in this sentence. File names are indicated like this: /path/to/ourfile.

The variable names are represented LIKE THIS, keywords and fragments of program text are written in this font.

Acknowledgments
===============

Many people need to be thanked for their assistance in developing and debugging mailfromd.  After S. C. Johnson, I can say that this program ``owes much to a most stimulating collection of users, who have goaded me beyond my inclination, and frequently beyond my ability in their endless search for "one more feature".  Their irritating unwillingness to learn how to do things my way has usually led to my doing things their way; most of the time, they have been right.''

A real test for a program like mailfromd cannot be done but in conditions of production environment.  A decision to try it in these conditions is by no means an easy one, it requires courage and good faith in the intentions and abilities of the author.  To begin with, I would like to thank my contributors for these virtues.

Jan Rafaj has intrepidly been using mailfromd since its early releases and invested lots of efforts in improving the program and its documentation.  He is the author of many of the MFL library functions, shipped with the package.  Some of his ideas are still waiting in my implementation queue, while new ones are consistently arriving.

Peter Markeloff patiently tested every mailfromd release and helped discover and fix many bugs.

Zeus Panchenko contributed many ideas and gave lots of helpful comments.  He offered invaluable help in debugging and testing mailfromd on FreeBSD platform.

Sergey Afonin proposed many improvements and new ideas.  He also invested a lot of his time in finding bugs and testing bugfixes.

The following people (in alphabetical order) provided bug reports and helpful comments for various versions of the program: Alan Dobkin, Brent Spencer, Jeff Ballard, Nacho González López, Simon Christian.

Introduction to mailfromd
*************************

Mailfromd is a general-purpose mail filtering daemon for Sendmail (See http://www.sendmail.org) and Postfix (See http://www.postfix.org). It is able to filter both incoming and outgoing messages using a filter program, written in mail filtering language (MFL).  The daemon interfaces with the MTA using Milter protocol.

The program name -- mailfromd -- stems from the fact that the original implementation was a simple filter implementing the sender address verification technique.  Since then the program has changed dramatically, and now it is actually a language translator and run-time evaluator providing a set of built-in and library functions for filtering electronic mail.

The first part of this manual is an overview, describing the features mailfromd offers in general.

The second part is a tutorial, which provides an introduction for those who have not used mailfromd previously.  It moves from topic to topic in a logical, progressive order, building on information already explained.  It offers only the principal information needed to master basic practical usage of mailfromd, while omitting many subtleties.

The other parts are meant to be used as a reference for those who know mailfromd well enough, but need to look up some notions from time to time.  Each chapter presents everything that needs to be said about a specific topic.

The manual assumes that the reader has a good knowledge of the SMTP protocol and Sendmail mail transport system.

Mailfromd at a first glance
===========================

In contrast to the most existing milter filters, mailfromd does not implement any default filtering policies.  Instead, it depends entirely on a filter script, supplied to it by the administrator.  The script, written in a specialized and simple to use language, called MFL (see MFL), is supposed to run a set of tests and to decide whether the message should be accepted by the MTA or not. To perform the tests, the script can examine the values of Sendmail macros, use an extensive set of built-in and library functions, and invoke user-defined functions.

Sender Address Verification.
============================

Sender address verification is the basic mail verification technique, implemented by mailfromd.  It consists in probing each MX server for the given address, until one of them gives a definite (positive or negative) reply.  Using this technique you can block a sender address if it is not deliverable, thereby cutting off a large amount of spam.  It can also be useful to block mail for undeliverable recipients, for example on a mail relay host that does not have a list of all the valid recipient addresses.  This prevents undeliverable junk mail from entering the queue, so that your MTA doesn't have to waste resources trying to send MAILER-DAEMON messages back.

Let's illustrate how it works on an example:

Suppose that the user <jsmith@somedomain.net> is trying to send mail to one of your local users.  The remote machine connects to your MTA and issues MAIL FROM: <jsmith@somedomain.net> command.  However, your MTA does not have to take its word for it, so it uses mailfromd to verify the sender address validity.  Mailfromd strips the domain name from the address (somedomain.net) and queries DNS about MX records for that domain.  Suppose, it receives the following list

10   relay1.somedomain.net
20   relay2.somedomain.net

It then connects to first MX server, using SMTP protocol, as if it were going to send a message to <jsmith@somedomain.net>.  This is called sending a probe message.  If the server accepts the recipient address, the mailfromd accepts the incoming mail.  Otherwise, if the server rejects the address, the mail is rejected as well.  If the MX server cannot be connected, mailfromd selects next server from the list and continues this process until it finds the answer or the list of servers is exhausted.

The probe message is like a normal mail except that no data are ever being sent. The probe message transaction in our example might look as follows (S: meaning messages sent by remote MTA, C: meaning those sent by mailfromd):

     C: HELO mydomain.net
     S: 220 OK, nice to meet you
     C: MAIL FROM: <>
     S: 220 <>: Sender OK
     C: RCPT TO: <jsmith@somedomain.net>
     S: 220 <jsmith@remote.net>: Recipient OK
     C: QUIT

Probe messages are never delivered, deferred or bounced; they are always discarded.

The described method of address verification is called standard method throughout this document.  Mailfromd also implements a method we call strict.  When using strict method, mailfromd first resolves IP address of sender machine to a fully qualified domain name.  Then it obtains MX records for this machine, and then proceeds with probing as described above.

So, the difference between the two methods is in the set of MX records that are being probed: standard method queries MXs based on the sender email domain, strict method works with MXs for the sender IP address.

Strict method allows to cut off much larger amount of spam, although it does have many drawbacks.  Returning to our example above, consider the following situation: <jsmith@somedomain.net> is a perfectly normal address, but it is being used by a spammer from some other domain, say otherdomain.com.  Standard method is not able to cope with such cases, whereas strict one is.

An alert reader will ask: what happens if mailfromd is not able to get a definite answer from any of MX servers?  Actually, it depends entirely on how you will instruct it to act in this case, but the general practice is to return temporary failure, which will urge the remote party to retry sending their message later.

After receiving a definite answer, mailfromd will cache it in its database, so that next time your MTA receives a message from that address (or from the sender IP/email address pair, for strict method), it will not waste its time trying to reach MX servers again.  The records remain in the cache database for a certain time, after which they are discarded.

Limitations of Sender Address Verification
------------------------------------------

Before deciding whether and how to use sender address verification, you should be aware of its limitations.

Both standard and strict methods suffer from the following limitations:

  * The sender verification methods will perform poorly on highly loaded sites.  The traffic and/or resource usage overhead may not be feasible for you.  However, you may experiment with various mailfromd options to find an optimal configuration.

  * Some sites may blacklist your MTA if it probes them too often.  This drawback can to a certain extent be eliminated by raising the expiration timeout in your cache database.

  * When verifying the remote address, no attempt to actually deliver the message is made.  If MTA accepts the address, mailfromd assumes it is OK.  However in reality, mail for a remote address can bounce after the nearest MTA accepts the recipient address.

     This drawback can often be avoided by combining sender address verification with greylisting (see Greylisting).

  * If the remote server rejects the address, no attempt is being made to discern between various reasons for rejection (client rejected, HELO rejected, MAIL FROM rejected, etc.)

  * Unfortunately, some major sites such as yahoo.com do not reject unknown addresses in reply to the RCPT TO command, but report a delivery failure in response to end of DATA after a message is transferred.  Of course, sender address verification does not work with such sites.  However, a combination of address verification and greylisting (see Greylisting) may be a good choice in such cases.

In addition, strict verification breaks forward mail delivery.  This is obvious, since mail forwarding is based on delivering unmodified message to another location, so the sender address domain will most probably not be the same as that of the MTA doing the forwarding.

Controlling Mail Sending Rate.
==============================

Mail Sending Rate for a given identity is defined as number of messages with this identity received within a predefined interval of time.

MFL offers a special function rate (see rate) that computes the sending rate for the given identity.

Building the Package
********************

This chapter contains a detailed list of steps you need to undertake in order to configure and build the package.

 1. Make sure you have the necessary software installed.

     To build mailfromd you will need to have following packages on your machine:

      A. GNU mailutils version 1.0 or newer.

          It is available from http://www.gnu.org/software/mailutils.

      B. A DBM library. Mailfromd is able to link with two flavors of DBM libraries: Berkeley DB or gdbm.  It will refuse to build without DBM.  By default, configure will try to find the best implementation installed on your machine (preference is given to Berkeley DB) and will use it.  You can, however, explicitly specify which implementation you want to use.

          The following table will help you do that.  The column DB type lists types of DBM databases supported by mailfromd. The column confMAPDEF lists the value of confMAPDEF Sendmail configuration macro corresponding to that database type.  It is here because it is usually wise to configure mailfromd to use the same database type as your Sendmail (If you don't know what confMAPDEF is, you probably are wasting your time reading this.  Refer to Sendmail configuration guide first).  Finally, the column configure option lists the option you should give to configure to enable using this database.  N/A in any column means there is no support for this database in Sendmail or mailfromd.

          DB type   confMAPDEF   configure option
          NDBM          -NNDBM       N/A
          Berkeley DB   -NNEWDB      --with-berkeley-db
          GDBM          N/A          --with-gdbm

          The --with-berkeley-db option needs a special note.  By default it will try to determine the version of Berkeley DB installed on your machine.  If, however, this autodetection fails, you can explicitly specify the version or the library name to use as the argument to the option.  Unless the argument begins with a digit, it is taken as a library name, without the lib prefix and library type suffix, so that

               ./configure --with-berkeley-db=db-3.1

          instructs configure to use library libdb-3.1.so (or libdb-3.1.a).  Otherwise, if the argument begins with a digit, it is understood as a version number of the library to link to.  In this case configure assumes a Slackware-like installation layout.  Thus, the option

                 --with-berkeley-db=3.1

          tells configure to use library libdb-3.1.so (or libdb-3.1.a)  and header file /usr/include/db31/db.h.

 2. Decide what user privileges will be used to run mailfromd

     After startup, the program drops the root privileges.  By default, it switches to the privileges of user mail, group mail. If there is no such user on your system, or you wish to use another user account for this purpose, override it using DEFAULT_USER environment variable.  For example for mailfromd to run as user nobody, use

          ./configure DEFAULT_USER=nobody

     The user name can also be changed at run-time (see --user).

 3. Decide where to install mailfromd and where its filter script and data files will be located.

     As usual, the default value for the installation prefix is /usr/local.  If it does not suit you, specify another location using --prefix option, e.g.: --prefix=/usr.

     During installation phase, the build system will install several files.  These files are:

     File   Purpose
     PREFIX/sbin/mailfromd   Main daemon
     PREFIX/etc/mailfromd.rc   Main filter script
     file (The filter script file will be installed only if it is
     not already there. Thus, if you are upgrading to a newer version of
     mailfromd, your old script file will be preserved
     with all your changes.)
     PREFIX/info/mailfromd.info   Documentation

     It is advisable to use the same settings for file name prefixes as those you used when configuring mailutils.  In particular, try to use the same --sysconfdir, since it will facilitate configuring the whole system.

     Another important point is location of local state directory, i.e. a directory where mailfromd will keep its data files (communication socket, pidfile and database files).  By default, its full name is LOCALSTATEDIR/mailfromd.  You can change it by setting DEFAULT_STATE_DIR configuration variable.  This value can be changed at run-time using #pragma option state-directory statement (see pragma state-directory).

 4. Select default communication socket. This is the socket used to communicate with sendmail, in the usual Milter port notation (see milter port specification). If the socket name does not begin with a protocol or directory separator, it is assumed to be a UNIX socket, located in the local state directory.  The default value is mailfrom, which is equivalent to unix:LOCALSTATEDIR/mailfromd/mailfrom.

     To alter this, use DEFAULT_SOCKET environment variable, e.g.:

          ./configure DEFAULT_SOCKET=inet:999@localhost

     The communication socket can be changed at run time using --port command line option (see --port) or #pragma option port statement in the filter script (see pragma port).

 5. Select default expiration interval. Expiration interval defines the period of time during which a record in the mailfromd database is considered valid.  It is described in more detail in Databases.  The default value is 86400 seconds, i.e. 24 hours.  It is OK for most sites.  If, however, you wish to change it, use DEFAULT_EXPIRE_INTERVAL environment variable.

     There are also two variables that allow to control particular expiration intervals: DEFAULT_DNS_NEGATIVE_EXPIRE_INTERVAL sets expiration time for cached negative DNS answers (see DNS Cache Management) (default 3600 seconds) and DEFAULT_EXPIRE_RATES_INTERVAL sets default expiration time for mail rate database (see rate).

     Expiration settings can be changed at run time using #pragma database statement in the filter script file (see database).

 6. Select a syslog implementation to use.

     Mailfromd uses syslog for diagnostics output.  The default syslog implementation on most systems (most notably, on GNU/Linux) uses blocking AF_UNIX SOCK_DGRAM sockets.  As a result, when an application calls syslog(), and syslogd is not responding and the socket buffers get full, the application will hang.

     For mailfromd, as for any daemon, it is more important that it continue to run, than that it continue to log.  For this purpose, mailfromd is shipped with a non-blocking syslog implementation by Simon Kelley.  This implementation, instead of blocking, buffers log lines in memory.  When the buffer log overflows, some lines are lost, but the daemon continues to run.  When lines are lost, this fact is logged with a message of the form:

             async_syslog overflow: 5 log entries lost

     To enable this implementation, configure the package with --enable-syslog-async option, e.g.:

          ./configure --enable-syslog-async

     Additionally, you can instruct mailfromd to use asynchronous syslog by default.  To do so, set DEFAULT_SYSLOG_ASYNC to 1, as shown in example below:

          ./configure --enable-syslog-async DEFAULT_SYSLOG_ASYNC=1

     You will be able to override these defaults at run-time by using --syslog-async or --no-syslog-async command line options (see Logging and Debugging).

 7. Run configure with all the desired options.

     For example, the following command:

          ./configure DEFAULT_SOCKET=inet:999@localhost --with-berkeley-db=3

     will configure the package to use Berkeley DB database, version 2, and inet:999@localhost as the default communication socket.

     At the end of its run configure will print a concise summary of its configuration settings.  It looks like that (with the long lines being split for readability):

          *******************************************************************
          Mailfromd configured with the following settings:

          External preprocessor..................... /usr/bin/m4 -s
          DBM version............................... Berkeley DB v. 3
          Default user.............................. mail
          State directory...........................
                     $(localstatedir)/$(PACKAGE)
          Socket.................................... mailfrom
          Expiration interval....................... 86400
          Negative DNS answer expiration interval... 3600
          Rates expire interval..................... 300
          Default syslog implementation............. blocking
          Readline (for mtasim)..................... yes
          Documentation rendition type.............. PROOF
          *******************************************************************

     Make sure these settings satisfy your needs.  If they do not, reconfigure the package with the right options.

 8. Run make.

 9. Run make install.

 10. Make sure LOCALSTATEDIR/mailfromd has the right owner and mode.

 11. Examine filter script file (SYSCONFDIR/mailfromd.etc) and edit it, if necessary.  If you are upgrading from an older version of mailfromd, see the corresponding section below.

Upgrading from 4.2 to 4.2.1
===========================

The asynchronous syslog implementation was reported to malfunction on some systems (notably on Solaris), so this release does not enable it by default.  The previous meaning of the --enable-syslog-async configuration option is also restored.  Use this option in order to enable asynchronous syslog feature.  To set default syslog implemetation, use DEFAULT_SYSLOG_ASYNC configuration variable (see syslog-async).

Upgrading from 4.1 to 4.2
=========================

Upgrading to this version does not require any special efforts.  You can use your configuration files and filter scripts without any changes. The only difference worth noticing is that starting from this version mailfromd is always compiled with asynchronous syslog implementation. The --enable-syslog-async configuration file option is still available, but its meaning has changed: it sets the default syslog implementation to use (see syslog-async). Thus, it can be used the same way it was in previous versions. You can also select the syslog implementation at run time, see --syslog-async option, for more detailed information.

Upgrading from 4.0 to 4.1
=========================

Upgrading to this version does not require any special efforts.  You can use your configuration files and filter scripts without any changes. Notice only the following major differences between 4.1 and 4.0:

  * Input files are preprocessed before compilation. See Preprocessor, for more information.

  * There is a way to discern between a not-supplied optional parameter, and a supplied one, having null value (see defined).

  * The version 4.1 implements sprintf function (see String formatting) and printf macro (see printf).

  * Support for some obsolete features is withdrawn.  These include:

      1. Using &CODE to specify exception codes

      2. Pragma options: retry, io-retry, and connect-retry.

Upgrading from 3.1.x to 4.0
===========================

Before building this version, please re-read the chapter see Building, especially the section Using non-blocking syslog.

Starting from the version 4.0, MFL no longer uses the predefined symbolic names for exception codes (previous versions used the & prefix to dereference them).  Instead, it relies on constants defined in the include file status.mfh (see status.mfh).

However, the script files from 3.1 series will still work, but the following warning messages will be displayed:

     Warning: obsolete constant form used: &failure
     Warning: remove leading '&' and include <status.mfh>

     Warning: Using built-in exception codes is deprecated
     Warning: Please include <status.mfh>

Another important difference is that pragmatic options ehlo and mailfromd are now deprecated, as well as their command line equivalents --ehlo and --domain.  These options became superfluous after the introduction of mailfrom_address and ehlo_domain built-in variables.  For compatibility with the previous versions, they are still supported by mailfromd 4.0, but a warning message is issued if they are used:

     warning: `#pragma option ehlo' is deprecated,
      consider using `set ehlo_domain "domain.name"' instead

To update your startup scripts for the new version follow these steps:

 1. Change #pragma option mailfrom VALUE to set mailfrom_address VALUE.  Refer to mailfrom_address, for a detailed discussion of this variable.

 2. Change #pragma option ehlo VALUE to set ehlo_domain VALUE.  Refer to ehlo_domain, for a detailed discussion of this variable.

 3. Include status.mfh.  Add the following line to the top of your startup file:

          #include_once <status.mfh>

 4. Remove all instances of & in front of the constants.  You can use the following sed expression: s/&\([a-z]\)/\1/g.

 5. If your code uses any of the following functions: hostname, resolve, hasmx or ismx, add the following line to the top of your script:

          #require dns

     See Modules, for a detailed description of the module system.

 6. Replace all occurrences of next with pass.

 7. If your code uses function match_cidr, add the following line to the top of your script:

          #require match_cidr

     See Modules, for a description of MFL module system.

Upgrading from 3.0.x to 3.1
===========================

 1. The mailfromd binary no longer supports --config-file (-c) option.  To use an alternative script file, give it as an argument, i.e. instead of:

          $ mailfromd --config-file FILE.RC

     write:

          $ mailfromd FILE.RC

     For backward compatibility, the old style invocation still works but produces a warning message.  However, if mailfromd encounters the -c option it will print a diagnostic message and exit immediately.  This is because the semantics of this option will change in the future releases.

 2. If a variable is declared implicitly within a function, it is created as automatic.  This differs from the previous versions, where all variables were global.  It is a common practice to use global variables to pass additional information between handlers (see HELO Domain, for an example of this approach).  If your filter uses it, make sure the variable is declared as global.  For example, this code:

     Figure: Implicit declaration, old style

          prog helo
          do
            # Save the host name for further use
            set helohost $s
          done

     has to be rewritten as follows:

     Figure: Implicit declaration, new style

          set helohost ""

          prog helo
          do
            # Save the host name for further use
            set helohost $s
          done

 3. Starting from version 3.1 the function dbmap takes an optional third argument indicating whether or not to count the terminating null character in key (see dbmap).  If your startup script contained any calls to dbmap, change them as follows:

     in 3.0.x   in 3.1
     dbmap(DB, KEY)   dbmap(DB, KEY, 1)

Upgrading from 2.x to 3.0.x
===========================

Update your startup scripts and/or crontab entries.  The mailfromd binary is now installed in ${prefix}/sbin.

We also encourage you to update the startup script (run cp etc/rc.mailfromd /wherever-your-startup-lives), since the new version contains lots of enhancements.

Upgrading from 1.x to 2.x
=========================

If you are upgrading from version 1.x to 2.0, you will have to do the following:

 1. Edit your script file and enclose the entire code section into:

          prog envfrom
          do
            ...
          done

     See Handlers, for more information about the prog statement.

 2. If your code contained any rate statements, convert them to function calls (see rate), using the following scheme:

          Old statement: if rate KEY LIMIT / EXPR
          New statement: if rate(KEY, interval("EXPR")) > LIMIT

     For example,

             rate $f 180 / 1 hour 25 minutes

     should become

             rate($f, interval("1 hour 25 minutes")) > 180

 3. Rebuild your databases using the following command:

          mailfromd --compact --all

     This is necessary since the format of mailfromd databases has changed in version 2.0: the key field now includes the trailing NUL character, which is also reflected in its length.  This allows for empty (zero-length) keys.  See Database Maintenance, for more information about the database compaction.

Tutorial
********

This chapter contains a tutorial introduction, guiding you through various mailfromd configurations, starting from the simplest ones and proceeding up to more advanced forms.  It omits most complicated details, concentrating mainly on the common practical tasks.

If you are familiar to mailfromd, you can skip this chapter and go directly to the next one, that contains detailed discussion of the configuration language and mailfromd interaction with the Mail Transport Agent.

Start Up
========

The mailfromd utility runs as a standalone daemon program and listens on a predefined communication channel for requests from the Mail Transfer Agent (MTA, for short).  When processing each message, the MTA installs communication with mailfromd, and goes through several states, collecting the necessary data from the sender.  At each state it sends the relevant information to mailfromd, and waits for it to reply.  The mailfromd filter receives the message data through Sendmail macros and runs a handler program defined for the given state.  The result of this run is a response code, that it returns to the MTA.  The following response codes are defined:

continue
     Continue message processing.

accept
     Accept this message for delivery.  After receiving this code the MTA continues processing this message without further consulting mailfromd filter.

reject
     Reject this message.  The message processing stops at this stage, and the sender receives the reject reply (5XX reply code).  No further mailfromd handlers are called for this message.

discard
     Silently discard the message.  This means that MTA will continue processing this message as if it were going to deliver it, but will discard it after receiving.  No further interaction with mailfromd occurs.

tempfail
     Temporarily reject the message. The message processing stops at this stage, and the sender receives the temporary failure reply (4XX reply code).  No further mailfromd handlers are called for this message.

The instructions on how to process the message are supplied to mailfromd in its filter script file.  It is normally called /usr/local/etc/mailfromd.rc (but can be located elsewhere, see invocation) and contains a set of milter state handlers, or subroutines to be executed in various SMTP states.  Each interaction state can be supplied its own handling procedure.  A missing procedure implies continue response code.

The filter script can define up to eight milter state handlers, called after the names of milter states: connect, helo, envfrom, envrcpt, header, eoh, body, and eom.  Two special handlers are available for initialization and clean-up purposes: begin is called before the processing starts, and end is called after it is finished. The diagram below shows the control flow when processing an SMTP transaction.  Lines marked with C: show SMTP commands issued by the remote machine (the client), those marked with => show called handlers with their arguments.  An [R] appearing at the start of a line indicates that this part of the transaction can be repeated any number of times:

Figure: Mailfromd Control Flow

     => begin()
     => connect(HOSTNAME, FAMILY, PORT, IP address)
     C: HELO DOMAIN
     helo(DOMAIN)
     for each message transaction
     do
             C: MAIL FROM SENDER
             => envfrom(SENDER)

     [R]     C: RCPT TO RECIPIENT
             => envrcpt(RECIPIENT)

             C: DATA
     [R]     C: HEADER: VALUE
             => header(HEADER, VALUE)

             C:
             => eoh()

     [R]     C: BODY-LINE
             => /* Collect lines into blocks BLK of
             =>  * at most LEN bytes and call:
             =>  */
             => body(BLK, LEN)

             C: .
             => eom()
     done
     => end()

This control flow is maintained for as long as each called handler returns continue (see Actions).  Otherwise, if any handler returns accept or discard, the message processing continues, but no other handler is called.  In the case of accept, the MTA will accept the message for delivery, in the case of discard it will silently discard it.

If any of the handlers returns reject or tempfail, the result depends on the handler.  If this code is returned by envrcpt handler, it causes this particular recipient address to be rejected.  When returned by any other handler, it causes the whole message will be rejected.

The reject and tempfail actions executed by helo handler do not take effect immediately.  Instead their action is deferred until the next SMTP command from the client, which is usually MAIL FROM.

Simplest Configurations
=======================

The mailfromd script file contains a series of declarations of the handler procedures. Each declaration has the form:

     prog NAME
     do
       ...
     done

where prog, do and done are the keywords, and NAME is the state name for this handler.  The dots in the above example represent the actual code, or a set of commands, instructing mailfromd how to process the message.

For example, the declaration:

     prog envfrom
     do
       accept
     done

installs a handler for envfrom state, which always approves the message for delivery, without any further interaction with mailfromd.

The word accept in the above example is an action. Action is a special language statement that causes the execution of the program to stop and to return a response code to the Sendmail.  There are five actions, one for each response code: continue, accept, reject, discard, and tempfail.  Among these, reject and discard can optionally take one to three arguments.  The first argument is a three-digit RFC 2821 reply code.  It must begin with 5 for reject and with 4 for tempfail.  If two arguments are supplied, the second argument must be either an extended reply code (RFC 1893/2034) or a textual string to be returned along with the SMTP reply.  Finally, if all three arguments are supplied, then the second one must be an extended reply code and the third one must supply the textual string.  The following examples illustrate all possible ways of using the reject statement:

     reject
     reject 503
     reject 503 5.0.0
     reject 503 "Need HELO command"
     reject 503 5.0.0 "Need HELO command"

Please note the quotes around the textual string.

Conditional Execution
=====================

Programs consisting of a single action are rarely useful.  In most cases you will want to do some checking and decide whether to process the message depending on its result.  For example, if you do not want to accept messages from the address <badguy@some.net>, you could write the following program:

     prog envfrom
     do
       if $f = "badguy@some.net"
         reject
       else
         accept
       fi
     done

This example illustrates several important concepts.  First or all, $f in the third line is a Sendmail macro reference.  Sendmail macros are referenced the same way as in sendmail.cf, with the only difference that curly braces around macro names are optional, even if the name consists of several letters.  The value of a macro reference is always a string.

The equality operator (=) compares its left and right arguments and evaluates to true if the two strings are exactly the same, or to false otherwise.  Apart from equality, you can use the regular relational operators: !=, >, >=, < and <=.  Notice that string comparison in mailfromd is always case sensitive.  To do case-insensitive comparison, translate both operands to upper or lower case (see tolower, and see toupper).

The if statement decides what actions to execute depending on the value its condition evaluates to.  Its usual form is:

     if EXPRESSION THEN-BODY [else ELSE-BODY] fi

The THEN-BODY is executed if the EXPRESSION evaluates to true (i.e. to any non-zero value).  The optional ELSE-BODY is executed if the EXPRESSION yields false (i.e. zero).  Both THEN-BODY and ELSE-BODY can contain other if statements, their nesting depth is not limited.  To facilitate writing complex conditional statements, the elif keyword can be used to introduce alternative conditions, for example:

     prog envfrom
     do
       if $f = "badguy@some.net"
         reject
       elif $f = "other@domain.com"
         tempfail 470 "Please try again later"
       else
         accept
       fi
     done

See switch, for more elaborate forms of conditional branching.

Functions and Modules
=====================

As any programming language, MFL supports a concept of function, i.e. a body of code that is assigned a unique name and can be invoked elsewhere as many times as needed.

All functions have a definition that introduces types and names of the formal parameters and the result type, if the function is to return a meaningful value (function definitions in MFL are discussed in detail in see User-Defined Functions).

A function is invoked using a special construct, function call:

      NAME (ARG-LIST)

where NAME is the function name, and ARG-LIST is a comma-separated list of expressions.  Each expression in ARG-LIST is evaluated, and its type is compared with that of the corresponding formal argument.  If the types differ, the expression is converted to the formal argument type.  Finally, a copy of its value is passed to the function as a corresponding argument.  The order in which the expressions are evaluated is not defined.  The compiler checks that the number of elements in ARG-LIST matches the number of mandatory arguments for function NAME.

For compatibility with previous versions, if ARG-LIST consists of a single expression,  the surrounding parentheses can be omitted, i.e. the following forms are equivalent:

     hostname($client_addr)
     hostname $client_addr

However, such syntax creates several ambiguities, so use it sparingly if at all.  We recommend to always use parentheses when calling a function; see Cautions, for a detailed analysis of of this syntax.

When a function does not deliver a result, it should only be called as a statement.

Functions may be recursive, even mutually recursive.

Mailfromd comes with a rich set of predefined functions for various purposes.  There are two basic function classes: built-in functions, that are implemented by the MFL runtime environment in mailfromd, and library functions, that are implemented in MFL.  The built-in functions are always available and no preparatory work is needed before calling them.  In contrast, the library functions are defined in modules, special MFL source files that contain functions designed for a particular task.  In order to access a library function, you must first require a module it is defined in. This is done using #require statement.  For example, the function hostname looks up in the DNS the name corresponding to the IP address specified as its argument.  This function is defined in module dns.mf, so before calling it you must require this module:

     #require dns

The #require statement takes a single argument: the name of the requested module (without the .mf suffix).  It looks up the module on disk and loads it if it is available.

For more information about the module system see Modules.

Domain Name System
==================

Site administrators often do not wish to accept mail from hosts that do not have a proper reverse delegation in the Domain Name System. In the previous section we introduced the library function hostname, that looks up in the DNS the name corresponding to the IP address specified as its argument.  If there is no corresponding name, the function returns its argument unchanged.  This can be used to test if the IP was resolved, as illustrated in the example below:

     #require dns

     prog envfrom
     do
       if hostname($client_addr) = $client_addr
         reject
       fi
     done

The #require dns statement loads the module dns.mf, after which the definition of hostname becomes available.

An orthogonal function, resolve, which resolves the symbolic name to the corresponding IP address is provided in the same dns.mf module.

Checking Sender Address
=======================

The main purpose of mailfromd is verification of the sender address, therefore a special construct is provided for this purpose:

     on poll $f do
     when success:
       accept
     when not_found or failure:
       reject 550 5.1.0 "Sender validity not confirmed"
     when temp_failure:
       tempfail 450 4.1.0 "Try again later"
     done

The on poll construct runs standard verification (see standard verification) for the email address specified as its argument (in the example above it is the value of the Sendmail macro $f).  The check can result in the following conditions:

success
     The address exists.

not_found
     The address does not exist.

failure
     Some error of permanent nature occurred during the check.  The existence of the address cannot be verified.

temp_failure
     Some temporary failure occurred during the check.  The existence of the address cannot be verified at the moment.

The when branches of the on poll statement introduce statements, that should be executed upon each of these conditions.  If any condition occurs that is not handled within the on block, the run-time evaluator will signal an exception (For more information about exceptions and their handling, please refer to Exceptions.) and return temporary failure, therefore it is advisable to always handle all four conditions.  In fact, the condition handling shown in the above example is preferable for most normal configuration: the mail is accepted if the sender address is proved to exist and rejected otherwise.  If a temporary failure occurs, the remote party is urged to retry the transaction some time later.

The poll statement itself has a number of options that control the type of the verification.  These are discussed in detail in poll.

It is worth noticing that there is one special email address which is always available on any host, it is the null address <> used in error reporting.  It is best to not verify this address:

     prog envfrom
     do
       if $f == ""
         accept
       else
         on poll $f do
         when success:
           accept
         when not_found or failure:
           reject 550 5.1.0 "Sender validity not confirmed"
         when temp_failure:
           tempfail 450 4.1.0 "Try again later"
         done
       fi
     done

SMTP Timeouts
=============

When using polling functions, it is important to take into account possible delay, which can occur in SMTP transactions.  Most often such delays are due to low network bandwidth, but sometimes remote sites impose them willingly, as a spam-fighting measure (My private opinion is that such practice is completely lame.)

Mailfromd polling functions implement three distinct timeout values:

Connection timeout
     Maximum time for establishing the initial TCP connection to the remote host.  If the connection is not established within this time interval, the polling returns temp_failure.

Initial response timeout
     Maximum time before receiving initial SMTP response from the remote host.

I/O timeout
     Maximum amount of time for finishing an input/output operation with the remote SMTP server.

These three timeouts can be set using the following pragmatic comments (see Pragmatic comments, for a detailed description of pragmatic comments.) in the script file:

     #pragma option connect-timeout INTERVAL
     #pragma option initial-response-timeout INTERVAL
     #pragma option io-timeout INTERVAL

Here, INTERVAL is the time interval, expressed in usual time units (see time interval specification, for its detailed description.), for example:

     #pragma option initial-response-timeout 1 minute 30 seconds

The default values are:

     #pragma option connect-timeout CONNECT-TIMEOUT
     #pragma option initial-response-timeout INITIAL-RESPONSE-TIMEOUT
     #pragma option io-timeout IO-TIMEOUT

You will most certainly encounter some servers that deliberately delay issuing the initial SMTP reply (in particular, I have noted that most CommuniGate Pro servers are guilty of this lame practice).  The default initial-response-timeout value should be enough to cope with most of them.  If you encounter some server that delays more than 30 seconds, you can raise the initial-response-timeout value.  However, in this case, I'd rather recommend informing its admin that his server settings are not tolerable.

Avoiding Verification Loops
===========================

An envfrom program consisting only of the on poll statement will work smoothly for incoming mails, but will create infinite loops for outgoing mails.  This is because, upon sending an outgoing message mailfromd will start the verification procedure, which will initiate an SMTP transaction with the same mail server that runs it.  This transaction will in turn trigger execution of on poll statement, etc. ad infinitum.  To avoid this, any properly written filter script should not run the verification procedure on the email addresses in those domains that are relayed by the server it runs on.  This can be achieved using relayed function.  The function returns true if its argument is contained in one of the predefined domain list files.  These files correspond to Sendmail plain text files used in F class definition forms (see Sendmail Installation and Operation Guide, chapter 5.3), i.e. they contain one domain name per line, with empty lines and lines started with # being ignored. The domain files consulted by relayed function are defined using the special #pragma option relay statement:

     #pragma option relay "/etc/mail/local-host-names"
     #pragma option relay "/etc/mail/relay-domains"

The above example declares two domain list files, most commonly used in Sendmail installations to keep hostnames of the server (class w, see Sendmail Installation and Operation Guide, chapter 5.2.) and names of the domains, relayed by this server (class R).

Given all this, we can improve our filter program:

     #require dns
     #pragma option relay "/etc/mail/local-host-names"
     #pragma option relay "/etc/mail/relay-domains"

     prog envfrom
     do
       if $f == ""
         accept
       elif relayed(hostname(${client_addr}))
         accept
       else
         on poll $f do
         when success:
           accept
         when not_found or failure:
           reject 550 5.1.0 "Sender validity not confirmed"
         when temp_failure:
           tempfail 450 4.1.0 "Try again later"
         done
       fi
     done

If you feel that your Sendmail's relayed domains are not restrictive enough for mailfromd filters (for example you are relaying mails from some third-party servers), you can use a database of trusted mail server addresses.  If the number of such servers is small enough, a single or statement can be used, e.g.:

       elif ${client_addr} = "10.10.10.1"
            or ${client_addr} = "192.168.11.7"
         accept
       ...

otherwise, if the servers' IP addresses fall within one or several CIDRs, you can use the match_cidr function (see Internet address manipulation functions), e.g.:

       elif match_cidr (${client_addr}, "199.232.0.0/16")
         accept
       ...

or combine both methods.  Finally, you can keep a DBM database of relayed addresses and use dbmap or dbget function for checking (see Database functions).

       elif dbmap(__statedir__ "/relay.db", ${client_addr})
         accept
       ...

HELO Domain
===========

Some of the mail filtering conditions may depend on the value of helo domain name, i.e. the argument to the SMTP EHLO (or HELO) command.  If you ever need such conditions, take into account the following caveats.  Firstly, although Sendmail passes the helo domain in $s macro, it does not do this consistently.  In fact, the $s macro is available only to the helo handler, all other handlers won't see it, no matter what the value of the corresponding Milter.macros.HANDLER statement.  So, if you wish to access its value within an other handler, you will have to store it in a variable in the helo handler and then use this variable value in the other handler.  This brings us to the concept of variables in mailfromd scripts.

A variable is declared using the following syntax:

     TYPE NAME

where VARIABLE is the variable name and TYPE is string, if the variable is to hold a string value, and number, if it is supposed to have a numeric value.

A variable is assigned a value using set statement, as shown below:

     set NAME EXPR

where EXPR is any valid MFL expression.

To get the value of a variable, the following notation is used: %NAME.  The set statement can occur within handler declarations as well as outside of them.

There are two kinds of Mailfromd variables: global variables,  that are visible to all handlers and functions, and automatic variables, that are available only within the handler or function where they are declared.  For our purpose we need a global variable (see Variable classes, for detailed descriptions of both kinds of variables).

The following example illustrates the approach that allows to use the HELO domain name in any handler:

     # Declare the helohost variable
     string helohost

     prog helo
     do
       # Save the host name for further use
       set helohost $s
     done

     prog envfrom
     do
       # Reject hosts claiming to be localhost
       if %helohost = "localhost"
         reject 570 "Please specify real host name"
       fi
     done

Notice, that for this approach to work, the Sendmail statement Milter.macros.helo must contain s (see Sendmail Configuration).  This requirement can be removed by using the handler argument of helo.  Each mailfromd handler is given one or several arguments.  The exact number of arguments and their meaning are handler-specific and are described in Handlers, and milter-control-flow. The arguments are referenced by their ordinal number, using the notation $N.  The helo handler takes one argument, whose value is the helo domain.  Using this information, the helo handler from the example above can be rewritten as follows:

     prog helo
     do
       # Save the host name for further use
       set helohost $1
     done

Controlling Number of Recipients
================================

The simplest way to control the number of recipients per message in Sendmail is using the option MaxRecipientsPerMessage (Sendmail (tm) Installation and Operation Guide, chapter 5.6, O -- Set Option.).  However, it has its limitations and is not flexible, so you are often better off using mailfromd for this purpose.

Mailfromd keeps the number of recipients collected so far in variable %rcpt_count, which can be controlled in envrcpt handler as shown in the example below:

     prog envrcpt
     do
       if %rcpt_count > 10
         reject 550 5.7.1 "Too many recipients"
       fi
     done

This filter will accept no more than 10 recipients per message. You may achieve finer granularity by using additional conditions.  For example, the following code will allow any number of recipients if the mail is coming from a domain relayed by the server, while limiting it to 10 for incoming mail from other domains:

     prog envrcpt
     do
       if not relayed hostname $client_addr and %rcpt_count > 10
         reject 550 5.7.1 "Too many recipients"
       fi
     done

There are three important features to notice in the above code. First of all, it introduces two boolean operators: and, which evaluates to true only if both left-side and right-side expressions are true, and not, which reverses the value of its argument.

Secondly, the scope of an operation is determined by its precedence, or binding strength.  Not binds more tightly than and, so its scope is limited by the next expression between it and and.  Using parentheses to underline the operator scoping, the above if condition can be rewritten as follows:

         if (not (relayed (hostname ($client_addr)))) and (%rcpt_count > 10)

Finally, it is important to notice that all boolean expressions are computed using shortcut evaluation.  To understand what it is, let's consider the following expression: X and Y.  Its value is true only if both X and Y are true.  Now suppose that we evaluate the expression from left to right and we find that X is false.  This means that no matter what the value of Y is, the resulting expression will be false, therefore there is no need to compute Y at all. So, the boolean shortcut evaluation works as follows:

X and Y
     If X => false, do not evaluate Y and return false.

X or Y
     If X => true, do not evaluate Y and return true.

Thus, in the expression not relayed hostname $client_addr and %rcpt_count > 10, the value of the rcpt_count variable will be compared with 10 only if the relayed function yielded false.

To further enhance our sample filter, you may wish to make the reject output more informative, to let the sender know what the recipient limit is.  To do so, you can use the fact that in mailfromd adjacent string expressions concatenate, so the enhanced filter version could look like:

     set max_rcpt 10
     prog envrcpt
     do
       if not relayed hostname $client_addr and %rcpt_count > 10
         reject 550 5.7.1 "Too many recipients, max=" %max_rcpt
       fi
     done

When evaluating the third argument to reject, mailfromd will first convert %max_rcpt to string and then concatenate both strings together, producing string Too many recipients, max=10.

Sending Rate
============

We have introduced the notion of mail sending rate in Rate Limit.  Mailfromd keeps the computed rates in the special rate database (see Databases).  Each record in this database consists of a key, for which the rate is computed, and the rate value, in form of a double precision floating point number, representing the average number of messages per second sent by this key within the last sampling interval.  In the simplest case, the sender email address can be used as a key, however we recommend to use a conjunction EMAIL-SENDER_IP instead, so the actual EMAIL owner won't be blocked by actions of some spammer abusing his/her address.

To control and update sending rates, the rate function is provided.  It takes two mandatory arguments: key, whose meaning is described above, and interval, or the number of seconds, to which the actual sending rate value is converted.  Remember, that it is stored internally as a floating point number, and thus cannot be used in mailfromd filters, which operate only on integer numbers. To use the rate value, it is first converted to messages per given interval, which is an integer number.  For example, the rate 0.138888 brought to 1-hour interval gives 500 (messages per hour).

Wherever the rate function is called, it recomputes and updates the rate record for the given KEY, and returns its value, converted to messages per interval.  For example, the following code limits the mail sending rate for each email address-IP combination to 180 per hour.  If the actual rate value exceeds this limit, the sender is returned a temporary failure response:

     prog envfrom
     do
       if rate($f "-" ${client_addr}, 3600) > 180
         tempfail 450 4.7.0 "Mail sending rate exceeded. Try again later"
       fi
     done

Notice argument concatenation, used to produce the key.

It is often inconvenient to specify intervals in seconds, therefore a special interval function is provided, which converts its argument, which is a textual string representing time interval in English, to the corresponding number of seconds.  Using this function, the function invocation would be:

          rate($f "-" ${client_addr}, interval("1 hour"))

The interval function is described in interval, and time intervals are discussed in time interval specification.

The rate function begins returning non-zero value as soon as it has enough data to compute the rate. By default, it needs at least two mails. Since this may lead to a big number of false positives (i.e. overestimated rates) at the beginning of sampling interval, there is a way to specify a minimum number of samples rate must collect before starting to actually compute rates. This number of samples is given as the optional third argument to the function. For example, the following call will return 0 unless at least 10 mails with the given key value were detected:

          rate($f "-" ${client_addr}, interval("1 hour"), 10)

For additional information about rate function, see rate.

Greylisting
===========

Greylisting is a simple method of defending against the spam proposed by Evan Harris.  In few words, it consists in recording the sender IP-sender email-recipient email triplet of mail transactions.  Each time the unknown triplet is seen, the corresponding message is rejected with tempfail code.  If the mail is legitimate, this will make the originating server retry the delivery later, at which time the destination will accept it.  If, however, the mail is a spam, it will probably never be retried, so the users will not be bothered by it.  Even if the spammer will retry the delivery, the greylisting period will give spam-detection systems, such as DNSBLs, enough time to detect and blacklist it, so by the time the destination host starts accepting emails from this triplet, it will already be blocked by other means.

You will find the detailed description of the method in The Next Step in the Spam Control War: Greylisting (http://projects.puremagic.com/greylisting/whitepaper.html), the original whitepaper by Evan Harris.

The mailfromd implementation of greylisting is based on greylist function.  The function takes two arguments: the key, identifying the greylisting triplet, and the interval.  The function looks up the key in the greylisting database.  If such a key is not found, a new entry is created for it and the function returns true.  If the key is found, greylist returns false, if it was inserted to the database more than interval seconds ago, and true otherwise. In other words, from the point of view of the  greylisting algorithm, the function returns true when the message delivery should be blocked.  Thus, the simplest implementation of the algorithm would be:

     prog envrcpt
     do
      if greylist(${client_addr} "-" $f "-" ${rcpt_addr}, interval("1 hour"))
        tempfail 451 4.7.1 "You are greylisted"
      fi
     done

However, the message returned by this example, is not informative enough.  In particular, it does not tell when the message will be accepted.  To help you produce more informative messages, greylist function stores the number of seconds left to the end of the greylisting period in the global variable greylist_seconds_left, so the above example could be enhanced as follows:

     prog envrcpt
     do
       set gltime interval("1 hour")
       if greylist(${client_addr} "-" $f "-" ${rcpt_addr}, %gltime)
         if %greylist_seconds_left = %gltime
           tempfail 451 4.7.1
              "You are greylisted for " %gltime " seconds"
         else
           tempfail 451 4.7.1
              "Still greylisted for " %greylist_seconds_left " seconds"
         fi
       fi
     done

In real life you will have to avoid greylisting some messages, in particular those coming from the <> address and from the IP addresses in your relayed domain.  It can easily be done using the techniques described in previous sections and is left as an exercise to the reader.

One special case is whitelisting, which is often used together with greylisting.  To implement it, mailfromd provides the function dbmap, which takes two mandatory arguments: dbmap(FILE, KEY) (it also allows an optional third argument, see dbmap, for more information on it).  The first argument is the name of the DBM file where to search for the key, the second one is the key to be searched.  Assuming you keep your whitelist database in file /var/run/whitelist.db, a more practical example will be:

     prog envrcpt
     do
       set gltime interval("1 hour")

       if not ($f = "" or relayed hostname ${client_addr}
              or dbmap("/var/run/whitelist.db", ${client_addr}))
         if greylist(${client_addr} "-" $f "-" ${rcpt_addr}, %gltime)
           if %greylist_seconds_left = %gltime
             tempfail 451 4.7.1
                "You are greylisted for " %gltime " seconds"
           else
             tempfail 451 4.7.1
                "Still greylisted for " %greylist_seconds_left " seconds"
           fi
         fi
       fi
     done

Local Account Verification
==========================

In your filter script you may need to verify if the given user name is served by your mail server, in other words, to verify if it represents a local account.  Notice that in this context, the word local does not necessarily mean that the account is local for the server running mailfromd, it simply means any account whose mailbox is served by the mail servers using mailfromd.

The validuser function may be used for this purpose.  It takes one argument, the user name, and returns true if this name corresponds to a local account.  To verify this, the function relies on libmuauth, a powerful authentication library shipped with GNU mailutils.  More precisely, it invokes a list of authorization functions.  Each function is responsible for looking up the user name in a particular source of information, such as system passwd database, an SQL database, etc.  The search is terminated when one of the functions finds the name in question or the list is exhausted.  In the former case, the account is local, in the latter it is not.  This concept is discussed in detail in see Authorization and Authentication Principles (GNU Mailutils Manual)).  Here we will give only some practical advices for implementing it in mailfromd filters.

The actual list of authorization module available depends on your mailutils installation.  Usually it includes, apart from traditional UNIX passwd database, the functions for verifying PAM, RADIUS and SQL database accounts. Each of the authorization methods is configured using special command line options.  Run mailfromd --help and you will get the listing of those under Authentication options section.  For the exact meaning of each particular option, refer to Authentication and Authorization Options (GNU Mailutils Manual).  It is often inconvenient to specify all required options in the command line, so you are advised to use mailutils system configuration file. See Mailutils Configuration File (GNU Mailutils Manual), for its description.  Its use boils down to placing in it all the necessary authorization options, prefixed by the label :mailfromd.  For example, the following mailutils.rc statement:

     :mailfromd --authorization=pam:system \
                --pam-service=mailfromd

sets up the authorization using PAM and system passwd database.  The name of PAM service to use is mailfromd.

The function validuser is often used together with dbmap, as in the example below:

     if dbmap("/etc/mail/aliases.db", localpart($rcpt_addr), 1)
        and validuser localpart $rcpt_addr
       ...
     fi

For more information about dbmap function, see dbmap.

Databases
=========

Some mailfromd functions use DBM databases to save their persistent state data.  Each database has a unique identifier, and is assigned several pieces of information for its maintenance: the database file name and the expiration period, i.e. the time after which a record is considered expired.

To obtain the list of available databases along with their preconfigured settings, run mailfromd --show-defaults.  You will see a similar output:

     version:             VERSION
     script file:         /etc/mailfromd.rc
     user:                mail
     statedir:            /var/run/mailfromd
     socket:              unix:/var/run/mailfromd/mailfrom
     pidfile:             /var/run/mailfromd/mailfromd.pid
     default syslog:      blocking
     database format:     Berkeley DB 2.x
     dns database:        /var/run/mailfromd/dns.db
     dns negative expiration: 3600
     cache database:      /var/run/mailfromd/mailfromd.db
     cache expiration:    86400
     cache negative expiration: 43200
     greylist database:      /var/run/mailfromd/greylist.db
     greylist expiration:    86400
     rate database:      /var/run/mailfromd/rates.db
     rate expiration:    86400

The text below database format line describes all available databases.  Notice that the cache database, in contrast to the rest of databases, has two expire periods associated with it.  This is explained in the next subsection.

Database Formats
----------------

The version VERSION runs the following database types (or formats):

dns
     DNS database caches DNS lookups.  The key consists of the lookup type and the actual lookup key, separated by a single space.  The type can be one of A, PTR, MX.  The value is an ASCII string built up of one or more fields, separated by a single space.  The first field is always the expiration date for this record in seconds since the Epoch (00:00:00 UTC, January 1, 1970).  The meaning of the rest of the fields depends on the lookup type as described in the following table:

     A
          Each field contains the next IP address corresponding to the lookup key.  Notice, that currently (version VERSION) there can be at most one field here, but it may change in the future.

     PTR
          Each field contains a host name corresponding to the lookup key.  Notice, that currently (version VERSION) there can be at most one field here, but it may change in the future.

     MX
          Each field contains a host name of an MX record for the lookup key.

     See DNS Cache Management, for more information of DNS cache database and its management.

cache
     Cache database keeps the information about external emails, obtained using sender verification functions (see Checking Sender Address).  The key entry to this database is an email address or EMAIL:SENDER-IP string, for addresses checked using strict verification.  The data its stores for each key are:

      1. Address validity.  This field can be either success or not_found, meaning the address is confirmed to exists or it is not.

      2. The time when the entry was entered into the database.  It is used to check for expired entries.

     The cache database has two expiration periods: a positive expiration period, that is applied to  entries with the first field set to success, and a negative expiration period, applied to entries marked as not_found.

rate
     The mail sending rate data, maintained by rate function (see rate).  A record consists of the following fields:

      1. Timestamp.  The time when the entry was entered into the database.  It is used to check for expired entries.

      2. Interval during which the rate was measured (seconds).

      3. Number of mails sent during this interval.

      4. Actual mail sending rate.

      5. Expected rate, i.e. the mail sending rate that would have been achieved if this sender had sent an email right now.

greylist
     This database is maintained by greylist function (see Greylisting).  Each record holds only the timestamp.

Basic Database Operations
-------------------------

To list the contents of a database, use --list option. When used without any arguments it will list the cache database:

     $ mailfromd --list
     abrakat@mail.com           success Thu Aug 24 15:28:58 2006
     baccl@EDnet.NS.CA          not_found Fri Aug 25 10:04:18 2006
     bhzxhnyl@chello.pl       not_found Fri Aug 25 10:11:57 2006
     brqp@aaanet.ru:24.1.173.165  not_found Fri Aug 25 14:16:06 2006

You can also list data for any particular key or keys.  To do so, give the keys as arguments to mailfromd:

     $ mailfromd --list abrakat@mail.com brqp@aaanet.ru:24.1.173.165
     abrakat@mail.com           success Thu Aug 24 15:28:58 2006
     brqp@aaanet.ru:24.1.173.165  not_found Fri Aug 25 14:16:06 2006

To list another database, give its format identifier with the --format (-H) option.  For example, to list the rate database:

     $ mailfromd --list --format=rate
     sam@mail.net-62.12.4.3 Wed Sep  6 19:41:42 2006  139   3 0.0216 6.82e-06
     axw@rame.com-59.39.165.172 Wed Sep  6 20:26:24 2006  0  1  N/A  N/A

The --format option can be used with any database management option, described below.

Another useful operation you can do while listing rate database is the prediction of estimated time of sending, i.e. the time when the user will be able to send mail if currently his mail sending rate has exceeded the limit.  This is done using --predict option.  The option takes an argument, specifying the mail sending rate limit, e.g. (the second line is split for readability):

     $ mailfromd --predict="180 per 1 minute"
     ed@fae.net-21.10.1.2 Wed Sep 13 03:53:40 2006  0 1 N/A N/A; free to send
     service@19.netlay.com-69.44.129.19 Wed Sep 13 15:46:07 2006 7 2
        0.286   0.0224; in 46 sec. on Wed Sep 13 15:49:00 2006

Notice, that there is no need to use --list --format=rate along with this option, although doing so is not an error.

To delete an entry from the database, use --delete option, for example: mailfromd --delete abrakat@mail.com.  You can give any number of keys to delete in the command line.

Database Maintenance
--------------------

There are two principal operations of database management: expiration and compaction.  Expiration consists in removing expired entries from the database.  In fact, it is rarely needed, since the expired entries are removed in the process of normal mailfromd work.  Nevertheless, a special option is provided in case an explicit expiration is needed (for example, before dumping the database to another format, to avoid transferring useless information).

The command line option --expire instructs mailfromd to delete expired entries from the specified database.  As usual, the database is specified using --format option.  If it is not given explicitly, cache is assumed.

While removing expired entries the space they occupied is marked as free, so it can be used by subsequent inserts.  The database does not shrink after expiration is finished.  To actually return the unused space to the file system you should compact your database.

This is done by running mailfromd --compact (and, optionally, specifying the database to operate upon with --format option).  Notice, that compacting a database needs roughly as much disk space on the partition where the database resides as is currently used by the database.  Database compaction runs in three phases. First, the database is scanned and all non-expired records are stored in the memory.  Secondly, a temporary database is created in the state directory and all the cached entries are flushed into it.  This database is named after the PID of the running mailfromd process.  Finally, the temporary database is renamed to the source database.

Both --compact and --expire can be applied to all databases by combining them with --all.  It is useful, for example, in crontab files.  For example, I have the following monthly job in my crontab:

     0 1 1 * * /usr/local/sbin/mailfromd --compact --all

Testing Filter Scripts
======================

It is important to check your filter script before actually starting to use it.  There are several ways to do so.

To test the syntax of your filter script, use the --lint option.  It will cause mailfromd to exit immediately after attempting to compile the script file.  If the compilation succeeds, the program will exit with code 0.  Otherwise, it will exit with error code 78 (configuration error).  In the latter case, mailfromd will also print a diagnostic message, describing the error along with the exact location where the error was diagnosed, for example:

     mailfromd: /etc/mailfromd.rc:39: syntax error, unexpected ACT_REJECT,
     expecting OR or AND or ')'

For complex scripts you may wish to obtain a listing of variables used in the script.  This can be achieved using --xref command line option:

The output it produces consists of four columns:

Variable name

Data type
     Either number or string.

Offset in data segment
     Measured in words.

References
     A comma-separated list of locations where the variable was referenced.  Each location is represented as FILE:LINE. If several locations pertain to the same FILE, the file name is listed only once.

Here is an example of the cross-reference output:

     $ mailfromd --xref
     Cross-references:
     -----------------
     cache_used               number 5   /etc/mailfromd.rc:48
     clamav_virus_name        string 9   /etc/mailfromd.rc:240,240
     db                       string 15  /etc/mailfromd.rc:135,194,215
     dns_record_ttl           number 16  /etc/mailfromd.rc:136,172,173
     ehlo_domain              string 11
     gltime                   number 13  /etc/mailfromd.rc:37,219,220,222,223
     greylist_seconds_left    number 1   /etc/mailfromd.rc:220,226,227
     last_poll_host           string 2

If the script passes syntax check, the next step is often to test if it works as you expect it to.  This is done with --test (-t) command line option.  This option runs the envfrom handler (or another one, see below) and prints the result of its execution.

When running your script in test mode, you will need to supply the values of Sendmail macros it needs.  You do this by placing the necessary assignments in the command line.  For example, this is how to supply initial values for f and client_addr macros:

     $ mailfromd --test f=gray@gnu.org client_addr=127.0.0.1

You may also need to alter initial values of some global variables your script uses.  To do so, use -v (--variable) command line option.  This option takes a single argument consisting of the variable name and its initial value, separated by an equals sign.  For example, here is how to change the value of ehlo_domain global variable:

     $ mailfromd -v ehlo_domain=mydomain.org

The --test option is often useful in conjunction with options --debug, --trace and --transcript (see Logging and Debugging.  The following example shows what the author got while debugging the filter script described in Filter Script Example:

     $ mailfromd --test --debug=50 f=gray@gnu.org client_addr=127.0.0.1
     MX 20 mx20.gnu.org
     MX 10 mx10.gnu.org
     MX 10 mx10.gnu.org
     MX 20 mx20.gnu.org
     getting cache info for gray@gnu.org
     found status: success (0), time: Thu Sep 14 14:54:41 2006
     getting rate info for gray@gnu.org-127.0.0.1
     found time: 1158245710, interval: 29, count: 5, rate: 0.172414
     rate for gray@gnu.org-127.0.0.1 is 0.162162
     updating gray@gnu.org-127.0.0.1 rates
     SET REPLY 450 4.7.0 Mail sending rate exceeded. Try again later
     State envfrom: tempfail

To test any handler, other than envfrom, give its name as the argument to --test option.  Since this argument is optional, it is important that it be given immediately after the option, without any intervening white space, for example mailfromd --test=helo, or mailfromd -thelo.

This method allows to test one handler at a time.  To test the script as a whole, use mtasim utility.  When started it enters interactive mode, similar to that of sendmail -bs, where it expects SMTP commands on its standard input and sends answers to the standard output.  The --port=auto command line option instructs it to start mailfromd and to create a unique socket for communication with it.  For the detailed description of the program and the ways to use it, see mtasim.

Logging and Debugging
=====================

Depending on its operation mode, mailfromd tries to guess whether it is appropriate to print its diagnostics and informational messages on standard error or to send them to syslog.  By default standard error is used if the program is given one of the database management options (see Databases, or --test option (see Testing Filter Scripts.  Otherwise, syslog is used.  To alter these defaults, two command line options are provided: --stderr to print everything to standard error and --syslog to output all diagnostics to syslog.

The data can be sent to syslog using two ways: the syslog function from the system libc library, which is a blocking implementation in most cases, and the internal, asynchronous syslog implementation. See Using non-blocking syslog, for more information on these implementations and for information on how to select the default one.  Two options are provided to select the implementation to use at run time: the --syslog-async option instructs mailfromd to use asynchronous version, and the --no-syslog-async option instructs it to use the blocking version.

The default syslog facility is mail, and it can be changed by --log-facility option.  Argument to this option is a valid syslog facility name, i.e. one of: user, daemon, auth, authpriv (As of mailutils 1.0, authpriv is not supported), mail, and local0 through local7.  The argument can be given in upper, lower or mixed cases, and it can be prefixed with log_

Another syslog-related parameter that can be configured is the tag used to identify mailfromd messages.  By default the program name is used.  The --log-tag option can be used to change it.

As all UNIX utilities, mailfromd is very quiet unless it has something important to communicate, for example an error condition or something similar.  A set of command line options is provided for controlling the verbosity of its output.

The --trace option enables tracing the Sendmail actions executed during the message verification.  When this option is given, any accept, discard, continue, etc. triggered during the execution of your filter program will leave their traces in the log file.  Here is an example of how it looks like (syslog time stamp, tag and PID removed for readability):

     k8DHxvO9030656: /etc/mailfromd.rc:45: reject 550 5.1.1 Sender validity
     not confirmed

This shows that while verifying the message with ID k8DHxvO9030656 the reject action was executed by filter script /etc/mailfromd.rc at line 45.

The appearance of the message ID in the log deserves a special notice. The program will always identify its log messages with the Message-Id, when it is available.  Your responsibility as an administrator is to make sure it is available by configuring Sendmail to export the macro i to mailfromd. The rule of thumb is: make i available to the very first handler mailfromd executes.  It is not necessary to export it to the rest of the handlers, since mailfromd will cache it.  For example, if your filter script contains envfrom and envrcpt handlers, export i for envfrom. See Sendmail Configuration, for instructions on how to ensure it.

To push the log verbosity further, use #pragma option debug (see pragma debug) or its command line equivalent, --debug (-d, see --debug).  It takes a numeric argument specifying the relative verbosity level in the range between 0 and 100.  Zero means disabling verbose output, while 100 produces impractically verbose output, suitable only for the developers of mailfromd.

There is a more sophisticated form of the --debug option, that allows to set debugging levels individually for a set of source modules.  In this form, the argument to the option consists of a comma-separated list of debug specifications, each of which has the following form: SOURCE[=LEVEL]. Here, SOURCE is the mailfromd source name, without the suffix, and optional LEVEL is the debugging level (an integer between 0 and 100), that you wish to assign to this module.  The default level is 100.

For example, the following invocation sets the global debugging level to 1, the level for functions from prog.c to 10, and for engine.c to 100:

     $ mailfromd --debug=100,prog=10,engine

You need to have sufficient knowledge about mailfromd internal structure to use this form of the --debug option.

To control the execution of the sender verification functions (see Polling functions), you may use --transcript (-X) command line option which enables the transcript of SMTP sessions in the logs.  Here is an example of the output produced running mailfromd --debug=1 --transcript:

     k8DHxlCa001774: RECV: 220 spf-jail1.us4.outblaze.com ESMTP Postfix
     k8DHxlCa001774: SEND: HELO mail.gnu.org.ua
     k8DHxlCa001774: RECV: 250 spf-jail1.us4.outblaze.com
     k8DHxlCa001774: SEND: MAIL FROM: <>
     k8DHxlCa001774: RECV: 250 Ok
     k8DHxlCa001774: SEND: RCPT TO: <t1Kmx17Q@malaysia.net>
     k8DHxlCa001774: RECV: 550 <>: No thank you rejected: Account
      Unavailable: Possible Forgery
     k8DHxlCa001774: poll exited with status: not_found; sent
      "RCPT TO: <t1Kmx17Q@malaysia.net>", got "550 <>: No thank you
      rejected: Account Unavailable: Possible Forgery"
     k8DHxlCa001774: SEND: QUIT

Runtime Errors
==============

A runtime error is a special condition encountered during execution of the filter program, that makes further execution of the program impossible.  There are two kinds of runtime errors: fatal errors, and uncaught exceptions.  Whenever a runtime error occurs, mailfromd writes into the log file the following message:

     RUNTIME ERROR near FILE:LINE: TEXT

where FILE:LINE indicates approximate source file location where the error occurred and TEXT gives the textual description of the error.

Fatal runtime errors
--------------------

Fatal runtime errors are caused by a condition that is impossible to fix at run time:

Not enough memory
     There is not enough memory for the execution of the program.  Try to make more memory available for mailfromd or to reduce its memory requirements by rewriting your filter script.

Out of stack space; increase #pragma stacksize
Heap overrun; increase #pragma stacksize
memory chunk too big to fit into heap
     These errors are reported when there is not enough space left on stack to perform the requested operation, and the attempt to resize the stack has failed.  Usually mailfromd expands the stack when the need arises (see automatic stack resizing).  This runtime error indicates that there were no more memory available for stack expansion.  Try to make more memory available for mailfromd or to reduce its memory requirements by rewriting your filter script.

Stack underflow
     Program attempted to pop a value off the stack but the stack was already empty.  This indicates an internal error in the MFL compiler or mailfromd runtime engine.  If you ever encounter this error, please report it to bug-mailfromd@gnu.org.ua.  Include the log fragment (about 10-15 lines before and after this log message) and your filter script.  See Reporting Bugs, for more information about bug reporting.

pc out of range
     The program counter is out of allowed range.  This is a severe error, indicating an internal inconsistency in mailfromd runtime engine.  If you encounter it, please report it to bug-mailfromd@gnu.org.ua.  Include the log fragment (about 10-15 lines before and after this log message) and your filter script. See Reporting Bugs, for more information about how to report a bug.

Programmatic runtime errors
---------------------------

These indicate a programmatic error in your filter script, which the MFL compiler was unable to discover at compilation stage:

Invalid exception number: N
     The throw statement used a not existent exception number N. Fix the statement and restart mailfromd.  See throw, for the information about throw statement and see Exceptions, for the list of available exception codes.

No previous regular expression
     You have used a back-reference (see Back references), where there is no previous regular expression to refer to.  Fix this line in your code and restart the program.

Invalid back-reference number
     You have used a back-reference (see Back references), with a number greater than the number of available groups in the previous regular expression.  For example:

            if $f matches "(.*)@gnu.org"
              # Wrong: there is only one group in the regexp above!
              set x \2
            ...

     Fix your code and restart the daemon.

Uncaught exceptions
-------------------

Another kind of runtime errors are uncaught exceptions, i.e. exceptional conditions for which no handler was installed (see Exceptions, for information on exceptions and on how to handle them).  These errors mean that the programmer (i.e. you), made no provision for some specific conditions.  For example, consider the following code:

     prog envfrom
     do
       if $f mx matches "yahoo.com"
         foo()
       fi
     done

It is syntactically correct, but it overlooks the fact that mx matches generates temp_failure exception, if the underlying DNS query has timed out (see Special comparisons).  If this happens, mailfromd has no instructions on what to do next and reports an error.  This can easily be fixed using a catch statement, e.g.:

     prog envfrom
     do
       # Catch DNS errors
       catch temp_failure, failure
       do
         tempfail 451 4.1.1 "MX verification failed"
       done

       if $f mx matches "yahoo.com"
         foo()
       fi
     done

Another common case are undefined Sendmail macros.  In this case the macroundef exception is generated:

     RUNTIME ERROR near foo.c:34: Macro not defined: {client_adr}

These can be caused either by misspelling the macro name (as in the example message above) or by failing to export the required name in Sendmail milter configuration (see exporting macros).  This error should be fixed either in your source code or in sendmail.cf file, but if you wish to provide a special handling for it, you can use the following catch statement:

     catch macroundef
     do
       ...
     done

Sometimes the location indicated with the runtime error message is not enough to trace the origin of the error.  For example, an error can be generated explicitly with throw statement (see throw):

     RUNTIME ERROR near match_cidr.mf:30: invalid CIDR (text)

If you look in module match_cidr.mf, you will see the following code (line numbers added for reference):

     23 func match_cidr(string ipstr, string cidr) returns number
     24 do
     25   number netmask
     26
     27   if %cidr matches '^(([0-9]{1,3}\.){3}[0-9]{1,3})/([0-9][0-9]?)'
     28     return inet_aton(%ipstr) & len_to_netmask(\3) = inet_aton(\1)
     29   else
     30     throw invcidr "invalid CIDR (%cidr)"
     31   fi
     32   return 0
     33 done

Now, it is obvious that the value of cidr argument to match_cidr was wrong, but how to find the caller that passed the wrong value to it?  The special command line option --stack-trace is provided for this.  This option enables dumping stack traces when a fatal error occurs.  The traces contain information about function calls.  Continuing our example, using the --stack-trace option you will see the following diagnostics:

     RUNTIME ERROR near match_cidr.mf:30: invalid CIDR (127%)
     mailfromd: Stack trace:
     mailfromd: 0077: match_cidr.mf:30: match_cidr
     mailfromd: 0096: test.mf:13: bar
     mailfromd: 0110: mailfromd.rc:18: foo
     mailfromd: Stack trace finishes
     mailfromd: Execution of the configuration program was not finished

Each trace line describes one stack frame.  The lines appear in the order of most recently called to least recently called.  Each frame consists of:

 1. Value of program counter at the time of its execution

 2. Source code location, if available

 3. Name of the function called

Thus, the example above can be read as: the function match_cidr was called by function bar in file test.mf at line 13.  This function was called from function bar, in file test.mf at line 13.  In its turn, bar was called by function foo, in file mailfromd.rc at line 18.

Examining the caller functions will help you localize the source of the error and fix it.

You can also request a stack trace any place in your code, by calling the stack_trace function.  This can be useful for debugging, or in your catch statements.

Warnings about some slippery places in MFL
==========================================

     It seemed like a good idea at the time.

     Brian Kernighan

There are some features of MFL which, when used improperly, may lead to subtle, hard identifiable errors.  These are: concatenation operation (see Concatenation) and passing arguments to one-argument functions without parentheses (see Function call syntax).

Since there is no explicit operator for concatenation, it is often necessary to ensure that it happens at the right time by using parentheses to enclose the items to concatenate.  Consider the following example:

     echo toupper "some" "thing"

Should it print SOMETHING or just SOMEthing?  The correct answer is the former, but it is difficult to deduce unless you are well acquainted with the MFL precedence rules (see Precedence).  Therefore, the rule of thumb is: whenever in doubt, parenthesize:

     echo toupper("some" "thing") => "SOMETHING"
     echo toupper("some") "thing" => "SOMEthing"

Quoteless literals (see Literals) are yet another dangerous feature.  Just as the features mentioned above, it stems from the good old days when MFL was small and sweet and using literals without quotes indeed ``seemed a good idea at the time.''  It ceased to seem so after the introduction of user-defined functions, though.  Consider the following entire program text:

     prog envfrom
     do
       if hostname($client_addr) = $client_addr
         reject
       fi
     done

The intent was obviously to reject any mail if it comes from an address without a proper PTR record (see hostname function).  There is a serious error, however: hostname is not a built-in function as it used to be in previous releases (Up to the version 1.3.91.), and therefore it needs to be defined or required prior to using.  Otherwise it is no more than a literal, and the whole construct hostname($client_addr) is regarded by MFL compiler as a concatenation of the string hostname and the value of client_addr Sendmail variable.  It is easy to see using --dump-tree option:

     $ mailfromd --dump-tree test.mf
     State handlers:
     ---------------
     envfrom:
     COND:
     EQ
       CONCAT:
         STRING: "hostname"
         SYMBOL: {client_addr}
       SYMBOL: {client_addr}
     IFTRUE
       reject
     IFFALSE

In effect, the comparison is always false and reject is never called.

That is why starting from version 3.0 mailfromd warns about any occurrence of an unquoted identifier.  In fact, running --lint on the above program, gives:

     $ mailfromd --lint test.mf
     mailfromd: test.mf:3: warning: unquoted identifier `hostname'

Whenever you see such a message, be sure to inspect the source and to place quotes around the suspicious string, if it is intended to be used as a literal, or to require the corresponding module (see Modules) (or include the source file directly, see include), if it is indeed a function name.

Another place to execute special caution are format strings used with sprintf (see String formatting) and strftime (see strftime) functions.  They use % as a character introducing conversion specifiers, while the same character is used to expand a MFL variable within a string.  To prevent this misinterpretation, enclose format specification in single quotes (see singe-vs-double).  To illustrate this, let's consider the following example:

     echo sprintf ("Mail from %s", $f)

If a variable s is not declared, this line will produce the Variable s is not defined error message, which will allow you to identify and fix the bug.  The situation is considerably worse if s is declared.  In that case you will see no warning message, as the statement is perfectly valid, but at the run-time the variable s will be interpreted within the format string, and its value will replace %s.  To prevent this from happening, single quotes must be used:

     echo sprintf ('Mail from %s', $f)

This does not limit the functionality, since there is no need to fall back to variable interpretation in format strings.

Mail Filtering Language
***********************

The mail filtering language, or MFL, is a special language designed for writing filter scripts.  It has a simple syntax, similar to that of Bourne shell.  In contrast to the most existing programming languages, MFL does not have any special terminating or separating characters (like, e.g. newlines and semicolons in shell).  All syntactical entities are separated by any amount of white-space characters (i.e. spaces, tabulations or newlines).

The following sections describe MFL syntax in detail.

Comments
========

Two types of comments are allowed: C-style, enclosed between /* and */, and shell-style, starting with # character and extending up to the end of line:

     /* This is
        a comment. */
     # And this too.

There are, however, several special cases, where the characters following # are not ignored.

If # is followed by word include (with optional whitespace between them), this statement requires inclusion of the specified file, as in C.  There are two forms of the #include statement:

 1. #include <FILE>

 2. #include "FILE"

The quotes around FILE in the second form quotes are optional.

Both forms are equivalent if FILE is an absolute file name. Otherwise, the first form will look for FILE in the include search path.  The second one will look for it in the current working directory first, and, if not found there, in the include search path.

The default include search path is:

 1. PREFIX/share/mailfromd/VERSION/include

 2. PREFIX/share/mailfromd/include

 3. /usr/share/mailfromd/include

 4. /usr/local/share/mailfromd/include

     Where PREFIX is the installation prefix.

New directories can be appended in front of it using -I (--include) command line option.

For example, invoking

     $ mailfromd -I/var/mailfromd -I/com/mailfromd

creates the following include search path

 1. /var/mailfromd

 2. /com/mailfromd

 3. PREFIX/share/mailfromd/VERSION/include

 4. PREFIX/share/mailfromd/include

 5. /usr/share/mailfromd/include

 6. /usr/local/share/mailfromd/include

Along with #include, there is also a special form #include_once, that has the same syntax:

     #include_once <FILE>
     #include_once "FILE"

This form works exactly as #include, except that, if the FILE has already been included, it will not be included again. As the name suggests, it will be included only once.

This form should be used to prevent re-inclusions of a code, which can cause problems due to function redefinitions, variable reassignments etc.

Another special construct is #require.  It instructs the compiler, that the code below requires functions from the module given as an argument to this statement:

     #require MODULE

Modules are MFL source files that contain a collection of code and data serving a special purpose.  They are named MODNAME.mf and placed in one of the directories comprising the include path (see include search path).  The #require statement looks up the named module in the path and attempts to compile it.  If the module does not exist, a syntax error is reported.

It is not an error to require the same module several times.  After the first successful load, subsequent #require statements with the same MODNAME will be ignored.  In this regard, this statement is similar to #include_once.

You will find more information regarding MFL modules in see Modules.

A line in the form

     #line NUMBER "IDENTIFIER"

causes the MFL compiler to believe, for purposes of error diagnostics, that the line number of the next source line is given by NUMBER and the current input file is named by IDENTIFIER. If the identifier is absent, the remembered file name does not change.

If # is immediately followed by word pragma (with optional whitespace between them), such a construct introduces a pragmatic comment, i.e. an instruction that controls some configuration setting.  Following types of pragmatic comments are currently implemented: #pragma option changes a value of a run-time option, #pragma database configures database settings, #pragma stacksize sets the initial stack size, and #pragma regex controls the compilation of regular expressions.

These pragma types are described in the following subsections.

Pragma option
-------------

The syntax of #pragma option is:

     #pragma option NAME VALUE

where NAME is the name of the command line option (without leading dashes) to set, and VALUE is its new value.  The effect of this statement is the same as that of the command line option --NAME=VALUE.

The VALUE is specific for each particular option.  The supported value types are:

number
     A decimal number.

string
     A string of characters.  It must be enclosed in a pair of quotes, if it contains whitespace characters (use either single or double quotes, at your option).

boolean
     A boolean value: yes, true or t to indicate a true value, and no, false or nil to indicate a false value.

address
     An IP address in ``dotted-quad'' notation or a fully-qualified host name.

interval
     The time interval specification is a string that defines an interval, much the same way we do this in English: it consists of one or more pairs number-time unit.  For example, the following are valid interval specifications:

          1 hour
          2 hours 35 seconds
          1 year 7 months 2 weeks 2 days 11 hours 12 seconds

     The pairs can occur in any order, however unusual it may sound to a human ear, e.g. 2 days 1 year.  If the time unit is omitted, seconds are supposed.

     When used in a #pragma option statement, interval values need not be quoted.

The following options control various database expiration intervals.  The use of these options is deprecated, please use the corresponding #pragma database option.

 -- pragma option: expire-interval INTERVAL
     Sets expiration interval for all databases.

 -- pragma option: positive-expire-interval INTERVAL
     Sets expiration interval for positive cache entries.  Use #pragma database cache positive-expire-interval instead.

 -- pragma option: negative-expire-interval INTERVAL
     Sets expiration interval for negative cache entries.  Use #pragma database cache negative-expire-interval instead.

 -- pragma option: rates-expire-interval INTERVAL
     Sets expiration interval for entries in the rates database.  Use #pragma database rate expire-interval instead.

The following options control I/O operations over TCP.

 -- pragma option: source ADDRESS
     Sets source address for TCP connections.

 -- pragma option: connect-timeout INTERVAL
     Sets initial connection timeout.  If the connection is not established within this time, the corresponding probing function returns temporary failure.  The default value is CONNECT-TIMEOUT. See SMTP Timeouts, for a detailed description.

 -- pragma option: initial-response-timeout INTERVAL
     Sets the time to wait for the initial SMTP response.  Default is INITIAL-RESPONSE-TIMEOUT.  See SMTP Timeouts, for the detailed description.

 -- pragma option: io-timeout INTERVAL
 -- pragma option: timeout INTERVAL
     Sets I/O operation timeout in seconds. Mailfromd will wait the given amount of time for the success of each I/O operation with the remote MX.  Default timeout is IO-TIMEOUT. See SMTP Timeouts, for a detailed description.

     The form timeout is retained for backward compatibility and is considered deprecated.

The following options provide the default settings for poll statement (see Polling).

 -- pragma option: mailfrom STRING
     Sets default email addresses used in MAIL FROM: SMTP command when probing the remote host.  This value can be overridden by as parameter to poll command (see poll).

     This pragma is deprecated (see 31x-400).  Instead of it, use assignment to mailfrom_address variable:

          set mailfrom_address STRING

     See mailfrom_address, for more information about it.

 -- pragma option: ehlo STRING
     Sets default domain used in EHLO (or HELO) SMTP command when probing the remote host.  This value can be overridden by from parameter to poll command (see poll).

     This pragma is deprecated (see 31x-400).  Instead of it, use assignment to ehlo_domain variable:

          set ehlo_domain STRING

     See ehlo_domain, for more information about it.

The following options control debugging and logging facilities:

 -- pragma option: debug STRING
     Sets debugging level.  STRING is a decimal number in the range 0 -- 100.  Level 0 effectively disables debugging, while level 100 enables the most verbose debugging output.

     You can also selectively set different debug levels for different source code modules.  In this case STRING is a comma-separated list of debug specifications, each of which has the following syntax: MODULE[=LEVEL].  See Logging and Debugging, for the detailed description of this.

 -- pragma option: source-info BOOLEAN
     When used together with #pragma option debug, this option includes source information into the debugging output.  Use this for debugging mailfromd.

 -- pragma option: stack-trace BOOLEAN
     Enables dumping stack traces on runtime errors.  This feature is useful for locating the source of an error, especially in complex scripts.  See tracing runtime errors, for a detailed description.

These options control program privileges after startup:

 -- pragma option: user STRING
     Switch to this user's privileges after startup. See Starting and Stopping, for a discussion of the privileges mailfromd runs under and the options that affect them.  See also group below.

 -- pragma option: group STRING
     Retain the supplementary group STRING when switching to user privileges.  By default mailfromd clears the list of supplementary groups when switching to user privileges, but this option allows to retain the given group.  It can be specified multiple times to retain several groups.  Use of this option can be necessary to maintain proper access rights for various files.  See Starting and Stopping.

The following options control interaction with Sendmail:

 -- pragma option: port STRING
     Sets the communication port data.  The STRING value is a port setting in Milter notation:

     unix:FILE
     local:FILE
          A named pipe (socket).

     inet:PORT@ADDRESS
          An IPv4 connection to host ADDRESS at port PORT.  Port must be specified either as a decimal number or a string representing the port name in /etc/services.

     inet6:PORT@ADDRESS
          An IPv6 connection to host ADDRESS at port PORT.  This port type is not yet supported.

 -- pragma option: milter-timeout INTERVAL
     Set the timeout value for connection between the filter and the MTA.  Default value is MILTER-TIMEOUT.  You normally do not need to change this value.

The following options are used to tune database file locking:

 -- pragma option: lock-retry-count NUMBER
     Set maximum number of attempts to acquire the lock.  The time between each two successive attempts is given by lock-retry-timeout pragma (see below).  After the NUMBER of failed attempts, mailfromd gives up.

 -- pragma option: lock-retry-timeout INTERVAL
     Set the time span between the two locking attempts.

Other options:

 -- pragma option: relay STRING
     Specifies the file containing names of email domains, relayed by this machine.  Multiple pragmas accumulate.  The constructed list of domains is analyzed by relayed function (see relayed).  For example:

          #pragma option relay "/etc/mail/sendmail.cw"
          #pragma option relay "/etc/mail/relay-domains"

 -- pragma option: state-directory STRING
     Changes the file name of the program state directory.

          #pragma option state-directory "/var/run/mailfromd"

 -- pragma option: pidfile STRING
     Introduces the full name of a file where mailfromd will store its PID number.  The file must be writable for the user mailfromd runs as (see pragma user).

Pragma database
---------------

The #pragma database statement is used to control file names and expiration periods used for various DBM databases.  Its syntax is:

#pragma database DBNAME file FILENAME
     Set the database file name for the database DBNAME.

#pragma database DBNAME expire-interval INTERVAL
     Set the expiration interval for the database DBNAME. (see time interval specification, for more information on INTERVAL syntax).

The parameter DBNAME can be one of the following: cache, dns, rate and greylist for main cache, DNS lookup, sending rate and greylisting databases, correspondingly.

The first two databases merit special attention.  The cache database implements two different timeouts, therefore there are two special forms of this statement for cache database:

#pragma database cache positive-expire-interval INTERVAL
     Set expiration interval for positive (success) cache entries.

#pragma database cache negative-expire-interval INTERVAL
     Set expiration interval for negative (not_found) cache entries.

The DNS lookup cache uses DNS TTL as expiration interval for records representing positive lookups.  The only value that can be altered is negative expiration interval:

#pragma database dns negative-expire-interval INTERVAL
     Set expiration interval for negative DNS cache entries.

See DNS Cache Management, for more information of DNS cache database and its management.

Pragma stacksize
----------------

The stacksize pragma sets the initial size of the run-time stack size, in words.  The default size is STACK-SIZE.  You may wish to increase this number if your configuration program uses recursive functions or does an excessive amount of string manipulations.  Example syntax:

     #pragma stacksize 7168

Improve the description: When the MFL engine notices that there is no more stack space available, it attempts to expand the stack.  If this attempt succeeds, the operation continues.  Otherwise, a runtime error is reported and the execution of the filter stops.

If you are concerned about the execution time of your script, you may wish to avoid such reallocations.  To help you find out the optimal stack size, each time the stack is expanded, mailfromd issues a similar warning in its log file:

     warning: stack segment expanded, new size=8192

You can use these messages to adjust your stack size configuration settings.

Pragma regex
------------

Another pragmatic comment, #pragma regex, controls compilation of expressions.  You can use any number of such pragma directives in your mailfromd.rc.  The scope of #pragma regex extends to the next occurrence of this directive or to the end of the script file, whichever occurs first.

The syntax of this pragma is:

     #pragma regex [push|pop] [LIST-OF-FLAGS]

where push and pop are optional sub-commands discussed in detail below, and LIST-OF-FLAGS is a whitespace-separated list of regex flags.  Each regex-flag is a word specifying some regex feature.  It can be preceded by + to enable this feature (this is the default), by - to disable it or by = to reset regex flags to its value.  Valid regex-flags are:

extended
     Use POSIX Extended Regular Expression syntax when interpreting regex.  If not set, POSIX Basic Regular Expression syntax is used.

icase
     Do not differentiate case.  Subsequent regex searches will be case insensitive.

newline
     Match-any-character operators don't match a newline.

     A non-matching list ([^...]) not containing a newline does not match a newline.

     Match-beginning-of-line operator (^) matches the empty string immediately after a newline.

     Match-end-of-line operator ($) matches the empty string immediately before a newline.

For example, the following pragma enables POSIX extended, case insensitive matching (a good thing to start your mailfromd.rc with):

     #pragma regex +extended +icase

Optional modifiers push and pop can be used to maintain a stack of regex flags.  The statement

     #pragma regex push [FLAGS]

saves current regex flags on stack and then optionally modifies them as requested by FLAGS.

The statement

     #pragma regex pop [FLAGS]

does the opposite: restores the current regex flags from the top of stack and applies FLAGS to it.

This statement is useful in include files to avoid disturbing user regex settings.  E.g.:

     #pragma regex push +extended +icase
      .
      .
      .
     #pragma regex pop

Data Types
==========

The mailfromd filter script language operates on entities of two types: numeric and string.

The numeric type is represented internally as a signed long integer.  Depending on the machine architecture, its size can vary. For example, on machines with Intel-based CPUs it is 32 bits long.

A string is a string of characters of arbitrary length. Strings can contain any characters except ASCII NUL.

Numbers
=======

A decimal number is any sequence of decimal digits, not beginning with 0.

An octal number is 0 followed by any number of octal digits (0 through 7), for example: 0340.

A hex number is 0x or 0X followed by any number of hex digits (0 through 9 and a through f or A through F), for example: 0x3ef1.

Literals
========

A literal is any sequence of alpha-numeric characters that is not a reserved word (see Reserved Words), or any sequence of characters enclosed in double or single quotation marks.

Literals consisting of only numeric characters represent integer numbers and were discussed in the previous section.

Quoted literals and literals beginning with a letter or underscore represent strings.

After tempfail and reject actions two special kinds of literals are recognized: three-digit numeric values represent RFC 2821 reply codes, and literals consisting of tree digit groups separated by dots represent an extended reply code as per RFC 1893/2034.  For example:

     510   # A reply code
     5.7.1 # An extended reply code

Double-quoted strings
---------------------

String literals enclosed in double quotation marks (double-quoted strings) are subject to backslash interpretation, macro expansion, variable interpretation and back reference interpretation.

Backslash interpretation is performed at compilation time.  It consists in replacing the following escape sequences with the corresponding single characters:

Table: Backslash escapes

Sequence   Replaced with
\a   Audible bell character (ASCII 7)
\b   Backspace character (ASCII 8)
\f   Form-feed character (ASCII 12)
\n   Newline character (ASCII 10)
\r   Carriage return character (ASCII 13)
\t   Horizontal tabulation character (ASCII 9)

In addition, the sequence \NEWLINE has the same effect as \n, for example:

     "a string with\
      embedded newline"
     "a string with\n embedded newline"

Any escape sequence of the form \xHH, where H denotes any hex digit is replaced with the character whose ASCII value is HH.  For example:

     "\x61nother" => "another"

Similarly, an escape sequence of the form \0OOO, where O is an octal digit, is replaced with the character whose ASCII value is OOO.

Macro expansion and variable interpretation occur at run-time.  During these phases all Sendmail macros (see Sendmail Macros), mailfromd variables (see Variables), and constants (see Constants) referenced in the string are replaced by their actual values.  For example, if the Sendmail macro f has the value postmaster@gnu.org.ua and the variable last_ip has the value 127.0.0.1, then the
string (Implementation note: actually, the references
are not interpreted within the string, instead, each such string is
split at compilation time into a series of concatenated atoms.  Thus,
our sample string will actually be compiled as:)

     $f " last connected from " %last_ip ";"

See Concatenation, for a description of this construct.  You can easily see how various strings are interpreted by using --dump-tree option (see --dump-tree).  For our sample string it will produce:

       CONCAT:
         CONCAT:
           CONCAT:
             SYMBOL: f
             CONSTANT: " last connected from "
           VARIABLE last_ip (13)
         CONSTANT: ";"

     "$f last connected from %last_ip;"

will be expanded to

     "postmaster@gnu.org.ua last connected from 127.0.0.1;"

A back reference is a sequence \D, where D is a decimal number.  It refers to the Dth parenthesized subexpression in the last matches statement (The subexpressions are numbered by the positions of their opening parentheses, left to right.).  Any back reference occurring within a double-quoted string is replaced by the value of the corresponding subexpression.  See Special comparisons, for a detailed description of this process.  Back reference interpretation is performed at run time.

Single-quoted strings
---------------------

The characters enclosed in single quotation marks are read unmodified.

The following examples contain pairs of equivalent strings:

     "a string"
     'a string'

     "\\(.*\\):"
     '\(.*\):'

Notice the last example.  Single quotes are particularly useful in writing regular expressions (see Special comparisons).

Notice also, that although unquoted literals are allowed, their use is not encouraged, since they create syntactic ambiguities that can make discovering real syntax errors difficult.  See Cautions.  Therefore mailfromd will always warn about such literals.

Here Documents
==============

Here-document is a special form of a string literal is, allowing to specify multiline strings without having to use backslash escapes.  The format of here-documents is:

     <<[FLAGS]WORD
     ...
     WORD

The <<WORD construct instructs the parser to read all the following lines up to the line containing only WORD, with possible trailing blanks.  The lines thus read are concatenated together into a single string.  For example:

     set str <<EOT
     A multiline
     string
     EOT

The body of a hear-document is interpreted the same way as double-quoted strings (see Double-quoted strings).  For example, if Sendmail macro f has the value jsmith@some.com and the variable count is set to 10, then the following string:

     set s <<EOT
     <$f> has tried to send %count mails.
     Please see docs for more info.
     EOT

will be expanded to:

     <jsmith@some.com> has tried to send 10 mails.
     Please see docs for more info.

If the WORD is quoted, either by enclosing it in single quote characters or by prepending it with a backslash, all interpretations and expansions within the document body are suppressed.  For example:

     set s <<'EOT'
     The following line is read verbatim:
     <$f> has tried to send %count mails.
     Please see docs for more info.
     EOT

Optional FLAGS in the here-document construct control the way leading white space is handled.  If FLAGS is - (a dash), then all leading tab characters are stripped from input lines and the line containing WORD.  Furthermore, if - is followed by a single space, all leading whitespace is stripped from them.  This allows here-documents within configuration scripts to be indented in a natural fashion.  Examples:

     <<- TEXT
         <$f> has tried to send %count mails.
         Please see docs for more info.
     TEXT

Here-documents are particularly useful with reject actions (see reject.

Sendmail Macros
===============

Sendmail macros are referenced exactly the same way they are in sendmail.cf configuration file, i.e. $NAME, where NAME represents the macro name.  Notice, that the notation is the same for both single-character and multi-character macro names. For consistency with the Sendmail configuration the ${NAME} notation is also accepted.

Sendmail macros evaluate to string values.

Notice, that to reference a macro, you must properly export it in sendmail.cf using Milter.macros statement (see Sendmail Configuration).  Attempt to reference a not exported macro will result in raising a macroundef exception at the run time (see uncaught exceptions).

Constants
=========

A constant is a symbolic name for an MFL value. Constants are defined using const statement:

     const NAME EXPR

where NAME is an identifier, and EXPR is any valid MFL expression evaluating immediately to a constant literal or numeric value.

After defining, any appearance of NAME in the program text is replaced by its value.  For example:

     const x 10/5
     const text "X is "

defines the numeric constant x with the value 5, and the literal constant text with the value X is .

Constants can also be used in literals.  To expand a constant within a literal string, prepend a percent sign to its name, e.g.:

     echo "New %text %x" => "New X is 2"

Built-in constants
------------------

Several constants are built into the MFL compiler (In mailfromd versions prior to 1.3.91, they were called macros.).  To discern them from user-defined ones, their names start and end with two underscores (__).

The following constants are defined in mailfromd version VERSION:

 -- Built-in constant: string __file__
     Expands to the name of the current source file.

 -- Built-in constant: string __function__
     Expands to the name of the current lexical context, i.e. the function or handler name.

 -- Built-in constant: number __line__
     Expands to the current line number in the input source file.

 -- Built-in constant: number __major__
     Expands to the major version number.

     The following example uses __major__ constant to determine if some version-dependent feature can be used:

          if __major__ > 2
            # Use some version-specific feature
          fi

 -- Built-in constant: number __minor__
     Expands to the minor version number.

 -- Built-in constant: string __package__
     Expands to the package name (mailfromd)

 -- Built-in constant: number __patch__
     For alpha versions and maintenance releases expands to the version patch level.  For stable versions, expands to 0.

 -- Built-in constant: string __preproc__
     Expands to the external preprocessor command line, if the preprocessor is used, or to an empty string if it cannot, e.g.:

          __preproc__ => "/usr/bin/m4 -s"

     See Preprocessor, for information on preprocessor and its features.

 -- Built-in constant: string __version__
     Expands to the textual representation of the program version (e.g. 3.0.90)

 -- Built-in constant: string __statedir__
     Expands to the default state directory (see statedir). Should have a way to determine the current value of statedir, as set by state-directory pragma, or, better yet, have a variable reflecting it (r/w).

Built-in constants can be used as variables, this allows to expand them within strings or here-documents.  The following example illustrates the common practice used for debugging configuration scripts:

     func foo(number x)
     do
       echo "%__file__:%__line__: foo called with arg %x"
       ...
     done

If the function foo were called in line 28 of the script file /etc/mailfromd.rc, like this: foo(10), you will see the following string in your logs:

     /etc/mailfromd.rc:28: foo called with arg 10

Variables
=========

Variables represent regions of memory used to hold variable data. These memory regions are identified by variable names.  A variable name must begin with a letter or underscore and must consist of letters, digits and underscores.

Each variable is associated with its lexical scope, i.e. the part of source code where it can be used.  Depending on the scope we discern two main classes of variables: global and automatic ones.

Global variables have indefinite lexical scope, so they may be referred to anywhere in the program.  Automatic or local variables are visible only within the given function or handler.

The two variable classes occupy separate namespaces, so that an automatic variable can have the same name as an existing global one. In this case this variable is said to shadow its global counterpart.  All references to such a name will refer to the automatic variable until the end of its scope is reached, where the global one becomes visible again.

A variable is declared using the following syntax:

     TYPE NAME

where NAME is the variable name and TYPE is the type of the data it is supposed to hold.  It is string for string variables and number for numeric ones.

If a variable declaration occurs within a function (see User-defined) or handler (see Handlers), it declares an automatic variable, local to this function or handler. Otherwise, it declares a global variable.

A variable is assigned a value using set statement:

     set NAME EXPR

where NAME is the variable name and EXPR is a mailfromd expression (see Expressions).  The effect of this statement is that the EXPR is evaluated and the value it yields is assigned to the variable NAME.

If the set statement is located outside a function or handler definition, the EXPR must be a constant expression, i.e. the compiler should be able to evaluate it immediately. -xrefoptimizer

It is not an error to assign a value to a variable that is not declared.  In this case the assignment first declares a global or automatic variable having the type of EXPR and then assigns a value to it. Automatic variable is created if the assignment occurs within a function or handler, global variable is declared if it occurs at topmost lexical level.  This is called implicit variable declaration. (Notice, that this differs from the way variables were handled in versions up to 3.0.2, which declared all variables as global.)

Variables are referenced using the notation %NAME.  The variable being referenced must have been declared earlier (either explicitly or implicitly).

Predefined Variables
--------------------

Several variables are predefined.  In mailfromd version VERSION these are:

 -- Variable: Predefined Variable number cache_used
     This variable is set by stdpoll and strictpoll built-ins (and, consequently, by the on poll statement).  Its value is 1 if the function used the cached data instead of directly polling the host, and 0 if the polling took place. See Polling functions.

     You can use this variable to make your reject message more informative for the remote party.  The common paradigm is to define a function, returning empty string if the result was obtained from polling, or some notice if cached data were used, and to use the function in the reject text, for example:

          func cachestr() returns string
          do
            if %cache_used
              return "[CACHED] "
            else
              return ""
            fi
          done

     Then, in prog envfrom one can use:

          on poll $f
          do
          when not_found or failure:
            reject 550 5.1.0 cachestr() "Sender validity not confirmed"
          done

 -- Predefined Variable: string clamav_virus_name
     Name of virus identified by ClamAV.  Set by clamav function (see ClamAV).

 -- Predefined Variable: number greylist_seconds_left
     Number of seconds left to the end of greylisting period.  Set by greylist function (see Special test functions).

 -- Predefined Variable: string ehlo_domain
     Name of the domain used by polling functions in SMTP EHLO or HELO command.  Default value is the fully qualified domain name of the host where mailfromd is run. See Polling.

 -- Variable: Predefined Variable string last_poll_host
     Polling functions (see Polling functions) set this variable before returning.  It contains the host name or IP address of the last polled host.

 -- Variable: Predefined Variable string last_poll_recv
     Polling functions (see Polling functions) set this variable before returning.  It contains the last SMTP reply received from the remote host.  In case of multi-line replies, only the first line is stored.  If nothing was received the variable contains the string nothing.

 -- Variable: Predefined Variable string last_poll_send
     Polling functions (see Polling functions) set this variable before returning.  It contains the last SMTP command sent to the polled host.  If nothing was sent, last_poll_send contains the string nothing.

 -- Predefined Variable: string mailfrom_address
     Email address used by polling functions in SMTP MAIL FROM command (see Polling.).  Default is <>.  Here is an example of how to change it:

          set mailfrom_address "postmaster@my.domain.com"

     You can set this value to a comma-separated list of email addresses, in which case the probing will try each address until either the remote party accepts it or the list of addresses is exhausted, whichever happens first.

     It is not necessary to enclose emails in angle brackets, as they will be added automatically where appropriate.  The only exception is null return address, when used in a list of addresses.  In this case, it should always be written as <>.  For example:

          set mailfrom_address "postmaster@my.domain.com, <>"

 -- Predefined Variable: number sa_code
     Spam score for the message, set by sa function (see sa).

 -- Predefined Variable: number rcpt_count
     The variable rcpt_count keeps the number of recipients given so far by RCPT TO commands.  It is defined only in envrcpt handlers.

 -- Predefined Variable: number sa_threshold
     Spam threshold, set by sa function (see sa).

 -- Predefined Variable: string sa_keywords
     Spam keywords for the message, set by sa function (see sa).

Back references
===============

A back reference is a sequence \D, where D is a decimal number.  It refers to the Dth parenthesized subexpression in the last matches statement (The subexpressions are numbered by the positions of their opening parentheses, left to right.).  Any back reference occurring within a double-quoted string is replaced with the value of the corresponding subexpression.  For example:

     if $f matches '.*@\(.*\)\.gnu\.org\.ua'
       set host \1
     fi

If the value of f macro is smith@unza.gnu.org.ua, the above code will assign the string unza to the variable host.

Notice, that each occurrence of matches will reset the table of back references, so try to use them as early as possible.  The following example illustrates a common error, when the back reference is used after the reference table has been reused by another matching:

     # Wrong!
     if $f matches '.*@\(.*\)\.gnu\.org\.ua'
       if $f matches 'some.*'
         set host \1
       fi
     fi

This will produce the following run time error:

     mailfromd: RUNTIME ERROR near file.rc:3: Invalid back-reference number

because the inner match (some.*) does not have any parenthesized subexpressions.

See Special comparisons, for more information about matches operator.

Handlers
========

Milter stage handler (or handler, for short) is a subroutine responsible for processing a particular milter state. There are eight handlers available.  Their order of invocation and arguments are described in milter-control-flow.

A handler is defined using the following construct:

     prog HANDLER-NAME
     do
       HANDLER-BODY
     done

where HANDLER-NAME is the name of the handler (see handler names), HANDLER-BODY is the list of filter statements composing the handler body.  Some handlers take arguments, which can be accessed within the HANDLER-BODY using the notation $N, where N is the ordinal number of the argument.  Here we describe the available handlers and their arguments:

 -- Handler: connect (string $1, number $2, number $3, string $4)

     Invocation:
          This handler is called once at the beginning of each SMTP connection.

     Arguments:

           1. string; The host name of the message sender, as reported by MTA.  Usually it is determined by a reverse lookup on the host address.  If the reverse lookup fails, $1 will contain the message sender's IP address enclosed in square brackets (e.g. [127.0.0.1]).

           2. number; Socket address family.  Include status.mfh to get symbolic definitions for the address families.  Supported families are:

               Constant   Value   Meaning
               FAMILY_STDIO    0      Standard input/output (the MTA is
               run with -bs option)
               FAMILY_UNIX     1      UNIX socket
               FAMILY_INET     2      IPv4 protocol

           3. number; Port number if $2 is FAMILY_INET.

           4. string; Remote IP address if $2 is FAMILY_INET or full file name of the socket if $2 is FAMILY_UNIX.  If $2 is FAMILY_STDIO, $4 is an empty string.

     The actions (see Actions) appearing in this handler are handled by Sendmail in a special way.  First of all, any textual message is ignored.  Secondly, the only action that immediately closes the connection is tempfail 421.  Any other reply codes result in Sendmail switching to nullserver mode, where it accepts any commands, but answers with a failure to any of them, except for the following: QUIT, HELO, NOOP, which are processed as usual.

     The following table summarizes the Sendmail behavior depending on the action used:

     tempfail 421 EXCODE MESSAGE
          The caller is returned the following error message:

               421 4.7.0 HOSTNAME closing connection

          Both EXCODE and MESSAGE are ignored.

     tempfail 4XX EXCODE MESSAGE
          (where XX represents any digits, except 21) Both EXCODE and MESSAGE are ignored.  Sendmail switches to nullserver mode.  Any subsequent command, excepting the ones listed above, is answered with

               454 4.3.0 Please try again later

     reject 5XX EXCODE MESSAGE
          (where XX represents any digits).  All arguments are ignored.  Sendmail switches to nullserver mode.  Any subsequent command, excepting ones listed above, is answered with

               550 5.0.0 Command rejected

     Regarding reply codes, this behavior complies with RFC 2821 (section 3.9), which states:

          An SMTP server must not intentionally close the connection except:

          [...]

          -  After detecting the need to shut down the SMTP service and returning a 421 response code.  This response code can be issued after the server receives any command or, if necessary, asynchronously from command receipt (on the assumption that the client will receive it after the next command is issued).

     However, the RFC says nothing about textual messages and extended error codes, therefore Sendmail's ignoring of these is, in my opinion, absurd.  My practice shows that it is often reasonable, and even necessary, to return a meaningful textual message if the initial connection is declined.  The opinion of mailfromd users seems to support this view.  Bearing this in mind, mailfromd is shipped with a patch for Sendmail 8.13.7 which makes it honor both extended return code and textual message given with the action.  The patch is in the file etc/sendmail-8.13.7.connect.diff.  It applies cleanly to versions of Sendmail up to 8.14.

 -- Handler: helo (string $1)

     Invocation:
          This handler is called whenever the SMTP client sends HELO or EHLO command.  Depending on the actual MTA configuration, it can be called several times or even not at all.

     Arguments:

           1. string; Argument to HELO (EHLO) commands.

     Notes:
          According to RFC 28221, $1 must be domain name of the sending host, or, in case this is not available, its IP address enclosed in square brackets.  Be careful when taking decisions based on this value, because in practice many hosts send arbitrary strings. We recommend to use heloarg_test function (see heloarg_test) if you wish to analyze this value.

 -- Handler: envfrom (string $1, string $2)

     Invocation:
          Called when the SMTP client sends MAIL FROM command, i.e. once at the beginning of each message.

     Arguments:

           1. string; First argument to the MAIL FROM command, i.e. the email address of the sender.

           2. string; Rest of arguments to MAIL FROM separated by space character.  This argument can be "".

     Notes

           1. $1 is not the same as $f Sendmail variable, because the latter contains the sender email after address rewriting and normalization, while $1 contains exactly the value given by sending party.

           2. When the array type is implemented, $2 will contain an array of arguments.

 -- Handler: envrcpt (string $1, string $2)

     Invocation:
          Called once for each RCPT TO command, i.e. once for each recipient, immediately after envfrom.

     Arguments:

           1. string; First argument to the RCPT TO command, i.e. the email address of the sender.

           2. string; Rest of arguments to RCPT TO separated by space character.  This argument can be "".

     Notes:
          When the array type is implemented, $2 will contain an array of arguments.

 -- Handler: header (string $1, string $2)

     Invocation:
          Called once for each header line received after SMTP DATA command.

     Arguments:

           1. string; Header field name.

           2. string; Header field value. The content of the header may include folded white space, i.e., multiple lines with following white space where lines are separated by LF (ASCII 10). The trailing line terminator (CR/LF) is removed.

 -- Handler: eoh

     Invocation:
          This handler is called once per message, after all headers have been sent and processed.

     Arguments:
          None.

 -- Handler: body (string $1, number $2)

     Invocation:
          This header is called zero or more times, for each piece of the message body obtained from the remote host.

     Araguments:

           1. string; Piece of body text. See Notes below.

           2. number; Length of $1, in bytes.

     Notes:
          As of version VERSION, $1 cannot be processed with usual string functions, because it is not null-terminated.  This limitation will be removed in future versions.

 -- Handler: eom

     Invocation:
          This handler is called once per message, when the terminating dot after DATA command has been received.

     Arguments:
          None

     Notes:
          This handler is useful for calling message capturing functions, such as sa or clamav.  For more information about these, refer to Interfaces to Third-Party Programs.

For your reference, the following table shows each handler with its arguments:

Table: State Handler Arguments

Handler   $1   $2   $3   $4
connect   Hostname   Socket Family   Port   Remote address
helo   HELO domain   N/A   N/A   N/A
envfrom   Sender email address   Rest of arguments   N/A   N/A
envrcpt   Recipient email address   Rest of arguments   N/A   N/A
header   Header name   Header value   N/A   N/A
eoh   N/A   N/A   N/A   N/A
body   Body segment (string)   Length of the segment
(numeric)   N/A   N/A
eom   N/A   N/A   N/A   N/A

The begin and end special handlers
==================================

Apart from the milter handlers described in the previous section, MFL defines two special handlers, called begin and end, which supply startup and cleanup instructions for the filter program.

The begin special handler is executed once for each SMTP session, after the connection has been established but before the first milter handler has been called.  Similarly, an end handler is executed exactly once, after the connection has been closed.  Neither of them takes any arguments.

The two handlers are defined using the following syntax:

     # Begin handler
     begin
     do
       ...
     done

     # End handler
     end
     do
       ...
     done

where ... represent any MFL statements.

An MFL program may have multiple begin and end definitions.  They can be intermixed with other definitions.  The compiler combines all begin statements into a single one, in the order they appear in the sources.  Similarly, all end blocks are concatenated together. The resulting begin is called once, at the beginning of each SMTP session, and end is called once at its termination.

Multiple begin and end handlers are a useful feature for writing modules (see Modules), because each module can thus have its own initialization and cleanup blocks.  Notice, however, that in this case the order in which subsequent begin and end blocks are executed is not defined.  It is only warranted that all begin blocks are executed at startup and all end blocks are executed at shutdown.  It is also warranted that all begin and end blocks defined within a compilation unit (i.e. a single abstract source file, whith all #include and #include_once statements expanded in place) are executed in order of their appearance in the unit.

Due to their special nature, the startup and cleanup blocks impose certain restrictions on the statements that can be used within them:

 1. return cannot be used in begin and end handlers. It could be a useful feature, though.

 2. The following Sendmail actions cannot be used in them: accept, continue, discard, reject, tempfail.  They can, however, be used in cache statements, declared in begin blocks (see example below).

 3. Header manipulation actions (see header manipulation) can be used only in begin header.

The begin handlers are the usual place to put global initialization code to.  For example, if you do not want to use DNS caching, you can do it this way:

     begin
     do
       db_set_active("dns", 0)
     done

Additionally, you can set up global exception handling routines there.  For example, the following begin statement disables DNS cache and, for all exceptions not handled otherwise, installs a handler that logs the exception along with the stack trace and continues processing the message:

     begin
     do
       db_set_active("dns", 0)
       catch *
       do
         echo "Caught exception $1: $2"
         stack_trace()
         continue
       done
     done

Functions
=========

A function is a named mailfromd subroutine, which takes zero or more parameters and returns certain value. Depending on the return value, functions can be subdivided into string functions and number functions.  The number of parameters is fixed for each function.  When invoked, the function must be supplied exactly as many actual arguments as it has parameters.

The function invocation syntax is:

       NAME (ARGS)

where NAME is the function name and ARGS is a comma-separated list of variables.  Single-argument functions compose a special subclass: for convenience they can be invoked without parentheses around their argument.  Summarizing, following are valid function calls:

     # Single-argument functions:
       foo 10
       interval "1 hour"
       interval("1 hour")
     # Multi-argument functions:
       greylist("/var/my.db", 180)

The number of parameters a function takes and their data types compose the function signature.  When actual arguments are passed to the function, they are converted to types of corresponding formal parameters.

There are two major groups of functions: built-in functions, that are implemented in the mailfromd binary, and user-defined functions, that are written in MFL.  The invocation syntax is the same for both groups.

Mailfromd is shipped with a rich set of library functions.  In addition to these you can define your own functions.

In subsequent sections we will discuss the library and built-in functions first, and then we will explain how to write your functions.

Built-in and Library Functions
------------------------------

This subsection describes built-in and library functions available in version VERSION.

String Manipulation Functions
.............................

 -- Built-in Function: string domainpart (string STR)
     Returns the domain part of STR, if it is a valid email address, otherwise returns STR itself.

          domainpart "gray" => "gray"
          domainpart "gray@gnu.org.ua" => "gnu.org.ua"

 -- Built-in Function: number index (string S, string T)

     Returns the index of the first occurrence of the string T in the string S, or -1 if T is not present.

          index("string of rings", "ring") => 2

     To find the last occurrence of a substring, use the function RINDEX (see rindex).

 -- Built-in Function: number interval (string STR)
     Converts STR, which should be a valid time interval specification (see time interval specification), to seconds.

 -- Built-in Function: number length (string STR)
     Returns the length of the string STR in bytes.

          length "string" => 6

 -- Built-in Function: string localpart (string STR)
     Returns the local part of STR if it is a valid email address, otherwise returns STR unchanged.

          localpart "gray" => "gray"
          localpart "gray@gnu.org.ua" => "gray"

 -- Built-in Function: string revstr (string S)
     Returns the string composed of the characters from S in reversed order:

          revstr("foobar") => "raboof"

 -- Built-in Function: number rindex (string S, string T)

     Returns the index of the last occurrence of the string T in the string S, or -1 if T is not present.

          rindex("string of rings", "ring") => 10

     See also String manipulation.

 -- Built-in Function: string substr (string STR,
     number START)
 -- Built-in Function: string substr (string STR,
     number START, number LENGTH)

     Returns the at most LENGTH-character substring of STR starting at START.  If LENGTH is omitted, the rest of STR is used.

     If LENGTH is greater than the actual length of the string, the range exception is signalled.

          substr("mailfrom", 4) => "from"
          substring("mailfrom", 4, 2) => "fr"

 -- Built-in Function: string substring (string STR,
     number START, number END)
     Returns a substring of STR between offsets START and END, inclusive.  To obtain a substring from START to the end of the string, use substring(STR, START, -1):

          substring("mailfrom", 0, 3) => "mail"
          substring("mailfrom", 2, 5) => "ilfr"
          substring("mailfrom", 4, -1 ) => "from"
          substring("mailfrom", 4, length("mailfrom") - 1) => "from"

     This function signals range exception if either START or END are outside the string length.

 -- Built-in Function: string tolower (string STR)

     Returns a copy of the string STR, with all the upper-case characters translated to their corresponding lower-case counterparts. Non-alphabetic characters are left unchanged.

          tolower "MAIL" => "mail"

 -- Built-in Function: string toupper (string STR)

     Returns a copy of the string STR, with all the lower-case characters translated to their corresponding upper-case counterparts. Non-alphabetic characters are left unchanged.

          toupper "mail" => "MAIL"

 -- Library Function: string strip_domain_part (string DOMAIN,
     number N)

     Returns at most N last components of the domain name DOMAIN. If N is 0 the function is equivalent to domainpart.

     This function is defined in strip_domain_part.mf module (see Modules).

     Examples:

          #require strip_domain_part
          strip_domain_part("puszcza.gnu.org.ua", 2) => "org.ua"
          strip_domain_part("puszcza.gnu.org.ua", 0) => "gnu.org.ua"

 -- Library Function: boolean is_ip (string STR)

     Returns true if STR is a valid IPv4 address.  This function is defined in is_ip.mf module (see Modules).

     For example:

          #require is_ip

          is_ip("1.2.3.4") => 1
          is_ip("1.2.3.x") => 0
          is_ip("blah") => 0
          is_ip("255.255.255.255") => 1
          is_ip("0.0.0.0") => 1

 -- Library Function: string revip (string IP)

     Reverses octets in IP, which must be a valid string representation of an IPv4 address.

     Example:

     revip("127.0.0.1") => "1.0.0.127"

String formatting
.................

 -- Built-in Function: string sprintf (string FORMAT, ...)
     The function sprintf formats its argument according to FORMAT (see below) and returns the resulting string.  It takes varying number of parameters, the only mandatory one being FORMAT.

Format of the format string
...........................

The format string is a simplified version of the format argument to C printf-family functions.

The format string is composed of zero or more directives: ordinary characters (not %), which are copied unchanged to the output stream; and conversion specifications, each of which results in fetching zero or more subsequent arguments. Each conversion specification is introduced by the character %, and ends with a conversion specifier.  In between there may be (in this order) zero or more flags, an optional minimum field width, and an optional precision.

Notice, that in practice that means that you should use single quotes with the FORMAT arguments, to protect conversion specifications from being recognized as variable references (see singe-vs-double).

No type conversion is done on arguments, so it is important that the supplied arguments match their corresponding conversion specifiers. By default, the arguments are used in the order given, where each * and each conversion specifier asks for the next argument.  If insufficiently many arguments are given, sprintf raises range exception.  One can also specify explicitly which argument is taken, at each place where an argument is required, by writing %M$, instead of % and *M$ instead of *, where the decimal integer M denotes the position in the argument list of the desired argument, indexed starting from 1. Thus,

         sprintf('%*d', %width, %num);

and

          sprintf('%2$*1$d', width, num);

are equivalent.  The second style allows repeated references to the same argument.

The flag characters
...................

The character % is followed by zero or more of the following flags:

#
     The value should be converted to an alternate form.   For o conversions, the first character of the output string is made zero (by prefixing a 0 if it was not zero already).  For x and X conversions, a non-zero result has the string 0x (or 0X for X conversions) prepended to it. Other conversions are not affected by this flag.

0
     The  value  should be zero padded.  For d, i, o, u, x, and X conversions, the converted value is padded on the left with zeros rather than blanks.  If the 0 and - flags both appear, the  0 flag is ignored.  If a precision is given, the 0 flag is ignored. Other conversions are not affected by this flag.

-
     The converted value is to be left adjusted on the field boundary. (The default is right justification.)  The converted value is padded on the right with blanks, rather than on the left with blanks or zeros.  A  -  overrides a 0 if both are given.

' ' (a  space)
     A blank should be left before a positive number (or empty string) produced by a signed conversion.

+
     A sign (+  or -) always be placed before a number produced by a signed conversion.  By default a sign is used only for negative numbers. A + overrides a space if both are used.

The field width
...............

An optional decimal digit string (with nonzero first digit) specifying a minimum field width.  If the converted value has fewer characters than the field width, it will be padded with spaces on the left (or right, if the left-adjustment flag has been given).  Instead of a decimal digit string one may write * or *M$ (for some decimal integer M) to specify that the field width is given in the next argument, or in the M-th argument, respectively, which must be of numeric type.  A negative field width is taken as a - flag followed by a positive field width.  In no case does a non-existent or small field width  cause truncation of a field; if the result of a conversion is wider than the field width, the field is expanded to  contain the conversion result.

The precision
.............

An optional precision, in the form of a period (.)  followed by an optional decimal digit string.  Instead of a decimal digit string one may write * or *M$ (for some decimal integer M) to specify that the precision is given in the next argument, or in the M-th argument, respectively, which must be of numeric type.  If the precision is given as just ., or the precision is negative, the precision is taken to be zero.  This gives the minimum number of digits to appear for d, i, o, u, x, and X conversions, or the maximum  number of characters to be printed from a string for the s conversion.

The conversion specifier
........................

A character that specifies the type of conversion to be applied.  The conversion specifiers and their meanings are:

d
i
     The numeric argument is converted to signed decimal notation.  The precision, if any, gives the minimum number of digits that must appear; if the converted value requires fewer digits, it is padded on the left with zeros.  The default precision is 1.  When 0 is printed with an explicit precision 0, the output is empty.

o
u
x
X
     The numeric argument is converted to unsigned octal (o), unsigned decimal (u), or unsigned hexadecimal (x and X) notation.  The letters abcdef are used for x conversions; the letters ABCDEF are used for X conversions.  The precision, if any, gives the minimum number of digits that must appear; if the converted value requires fewer digits, it is padded on the left with zeros.  The default precision is 1.  When 0 is printed with an explicit precision 0, the output is empty.

s
     The string argument is written to the output.  If a precision is specified, no more than the number specified of characters are written.

%
     A %  is  written.  No argument is converted. The complete conversion specification is %%.

Mail Header Functions
.....................

 -- Built-in Function: string message_header_encode
     (string TEXT, [string ENC, string CHARSET])
     Encode TEXT in accordance with RFC 2047.  Optional arguments:

     enc
          Encoding to use.  Valid values are quoted-printable, or Q (the default) and base64, or B.

     charset
          Character set.  By default UTF-8.

     If the function is unable to encode the string, it raises the exception failure.

     For example:

          set string "Keld Jørn Simonsen <keld@dkuug.dk>"
          message_header_encode(%string, "ISO-8859-1")
            => "=?ISO-8859-1?Q?Keld_J=F8rn_Simonsen?= <keld@dkuug.dk>"

 -- Built-in Function: string message_header_decode
     (string TEXT, [string CHARSET])
     TEXT must be a header value encoded in accordance with RFC 2047.  The function returns the decoded string.  If the decoding fails, it raises failure exception.  The optional argument CHARSET specifies the character set to use (default -- UTF-8).

          set string "=?ISO-8859-1?Q?Keld_J=F8rn_Simonsen?= <keld@dkuug.dk>"
          message_header_decode(%string)
           => "Keld Jørn Simonsen <keld@dkuug.dk>"

 -- Built-in Function: string unfold (string TEXT)
     If TEXT is a ``folded'' multi-line RFC 2822 header value, unfold it.  If TEXT is a single-line string, return its unchanged copy.

     For example, suppose that the message being processed contained the following header:

          List-Id: Sent bugreports to
            <some-address@some.net>

     Then, applying unfold to its value (For example:)

          prog header
          do
            echo unfold($2)
          done

      will produce:

          Sent bugreports to <some-address@some.net>

Polling Functions
.................

We have described the base notions about sender address verification (or polling, for short) in Checking Sender Address.  Here we will describe the functions mailfromd offers for this purpose.

 -- Built-in Function: number _pollhost
     (string IP, string EMAIL, string DOMAIN,
     string MAILFROM)
     Poll SMTP host IP for email address EMAIL, using DOMAIN as EHLO domain and MAILFROM as MAIL FROM.  Returns 0 or 1 depending on the result of the test. In contrast to strictpoll function, this function does not use cache database and does not fall back to polling MX servers if the main poll tempfails.  The function can throw one of the following exceptions: failure, temp_failure.

 -- Built-in Function: number _pollmx
     (string IP, string EMAIL, string DOMAIN,
     string MAILFROM)
     Poll MXs of the DOMAIN for email address EMAIL, using DOMAIN as EHLO domain and MAILFROM as MAIL FROM address.  Returns 0 or 1 depending on the result of the test. In contrast to stdpoll function, _pollmx does not use cache database and does not fall back to polling the IP if the poll fails.  The function can throw one of the following exceptions: failure, temp_failure.

 -- Built-in Function: number stdpoll
     (string EMAIL, string DOMAIN, string MAILFROM)

     Performs standard poll for EMAIL, using DOMAIN as EHLO domain and MAILFROM as MAIL FROM address. Returns 0 or 1 depending on the result of the test.  Can raise one of the following exceptions: failure, temp_failure.

     In on statement context, it is synonymous to poll without explicit HOST.  more details and references

 -- Built-in Function: number strictpoll
     (string HOST, string EMAIL,
     string DOMAIN, string MAILFROM)

     Performs strict poll for EMAIL on host HOST. See the description of stdpoll for the detailed information.

     In on context, it is synonymous to poll host HOST.

The MAILFROM argument can be a comma-separated list of email addresses, which can be useful for servers that are unusually picky about sender addresses.  It is advised, however, that this list always contain the <> address.  For example:

     _pollhost($client_addr, $f, "domain", "postmaster@my.net,<>")

See also pragma mailfrom.

Before returning, all described functions set the following built-in variables:

Table: Variables set by polling functions

Variable   Contains
last_poll_host   Host name or IP address of the last polled
host.
last_poll_send   Last SMTP command, sent to this
host.  If nothing was sent, it contains literal string nothing.
last_poll_recv   Last SMTP reply received from this
host.  In case of multi-line replies, only the first line is stored.  If
nothing was received the variable contains the string nothing.
cache_used   1 if cached data were used instead of
polling, 0 otherwise.  This variable is set by stdpoll
and strictpoll.  If it equals 1, none of the above
variables are modified.  See cache_used example, for an example.

Internet address manipulation functions
.......................................

Following functions operate on IPv4 addresses and CIDRs.

 -- Built-in Function: number ntohl (number N)
     Converts the number N, from host to network byte order. The argument N is treated as an unsigned 32-bit number.

 -- Built-in Function: number htonl (number N)
     Converts the number N, from network to host byte order. The argument N is treated as an unsigned 32-bit number.

 -- Built-in Function: number ntohs (number N)
     The argument N is treated as an unsigned 16-bit number. The function converts this number from network to host order.

 -- Built-in Function: number htons (number N)
     The argument N is treated as an unsigned 16-bit number. The function converts this number from host to network order.

 -- Built-in Function: number inet_aton (string S)
     Converts the Internet host address S from the standard numbers-and-dots notation into the equivalent integer in host byte order.

          inet_aton("127.0.0.1") => 2130706433

     The numeric data type in MFL is signed, therefore on machines with 32 bit integers, this conversion can result in a negative number:

          inet_aton("255.255.255.255") => -1

     However, this does not affect arithmetical operations on IP addresses.

 -- Built-in Function: string inet_ntoa (number N)
     Converts the Internet host address N, given in host byte order to string in standard numbers-and-dots notation:

          inet_ntoa(2130706433) => "127.0.0.1"

 -- Built-in Function: number len_to_netmask (number N)
     Convert number of masked bits N to   IPv4 netmask:

          inet_ntoa(len_to_netmask(24)) => 255.255.255.0
          inet_ntoa(len_to_netmask(7)) => 254.0.0.0

     If N is greater than 32 the function raises range exception.

 -- Built-in Function: number netmask_to_len (number MASK)
     Convert IPv4 netmask MASK into netmask length (number of bits preserved by the mask):

          netmask_to_len(inet_aton("255.255.255.0")) => 24
          netmask_to_len(inet_aton("254.0.0.0")) => 7

 -- Library Function: boolean match_cidr (string IP,
     string CIDR)

     This function is defined in match_cidr.mf module (see Modules).

     It returns true if the IP address IP pertains to the IP range CIDR.  The first argument, IP, is a string representation of an IP address.  The second argument, CIDR, is a string representation of a IP range in CIDR notation, i.e. "A.B.C.D/N", where A.B.C.D is an IPv4 address and N specifies the prefix length -- the number of shared initial bits, counting from the left side of the address.

     The following example will reject the mail if the IP address of the sending machine does not belong to the block 10.10.1.0/19:

          if not match_cidr(${client_addr}, "10.10.1.0/19")
            reject
          fi

DNS Functions
.............

Most DNS-related functions cache their results in the database dns, so no matter how many times you use a particular function in your startup program, it will result in at most one actual DNS lookup.  See below (see DNS Cache Management) for the description of the database and its management.

The functions are implemented in two layers: primitive built-in functions which raise exceptions if the lookup fails, and library calls that are warranted to always return meaningful value without throwing exceptions.

The built-in layer is always available.  The library calls become available after requesting dns.mf module (see Modules):

     #require dns

 -- Built-in Function: string dns_getaddr (string DOMAIN)
     Returns a whitespace-separated list of IP addresses (A records) for DOMAIN.  At most 64 addresses are returned. This limit should be configurable.

     This function does not use the DNS cache.

 -- Built-in Function: string dns_getname (string IPSTR)
     Returns a whitespace-separated list of domain names (PTR records) for the IPv4 address IPSTR.  At most 64 names are returned. This limit should be configurable.

     This function does not use the DNS cache.

 -- Built-in Function: string getmx (string DOMAIN
     [, number RESOLVE])
     Returns a whitespace-separated list of MX names (if RESOLVE is not given or if it is 0) or MX IP addresses (if RESOLVE!=0)) for DOMAIN.  Within the returned strings, items are sorted lexicographically.  If DOMAIN has no MX records, an empty string is returned.  If the DNS query fails, getmx raises an appropriate exception.

     Examples:

          getmx("mafra.cz") => "smtp1.mafra.cz smtp2.mafra.cz relay.iol.cz"
          getmx("idnes.cz") => "smtp1.mafra.cz smtp2.mafra.cz relay.iol.cz"
          getmx("gnu.org")  => "mx10.gnu.org mx20.gnu.org"
          getmx("org.pl") => ""

     Notes:

      1. The getmx function returns at most 32 MX names or IP addresses. This limit should probably be configurable.

      2. The number of items returned by getmx(DOMAIN) can differ from that obtained from getmx(DOMAIN, 1), e.g.:

               getmx("aol.com")
                 => mailin-01.mx.aol.com mailin-02.mx.aol.com
                           mailin-03.mx.aol.com mailin-04.mx.aol.com
               getmx("aol.com", 1)
                 => 64.12.137.89 64.12.137.168 64.12.137.184
                           64.12.137.249 64.12.138.57 64.12.138.88
                           64.12.138.120 64.12.138.185 205.188.155.89
                           205.188.156.185 205.188.156.249 205.188.157.25
                           205.188.157.217 205.188.158.121 205.188.159.57
                           205.188.159.217

      3. This interface will change in future releases, when array data types are implemented.

 -- Built-in Function: boolean primitive_hasmx (string DOMAIN)
     Returns true if the domain name given by its argument has any MX records.

     If the DNS query fails, this function throws failure or temp_failure.

 -- Library Function: boolean hasmx (string DOMAIN)
     Returns true if the domain name given by its argument has any MX records.

     Otherwise, if DOMAIN has no MXs or if the DNS query fails, hasmx returns false.

 -- Built-in Function: string primitive_hostname (string IP)
     The IP argument should be a string representing an IP address in dotted-quad notation.  The function returns the canonical name of the host with this IP address obtained from DNS lookup.  For example

          primitive_hostname (${client_addr})

     returns the fully qualified domain name of the host represented by Sendmail variable client_addr.

     If there is no PTR record for IP, primitive_hostname raises the exception not_found.

     If DNS query fails, the function raises failure or temp_failure, depending on the character of the failure.

 -- Library Function: string hostname (string IP)
     The IP argument should be a string representing an IP address in dotted-quad notation.  The function returns the canonical name of the host with this IP address obtained from DNS lookup.

     If there is no PTR record for IP, or if the lookup fails, the function returns IP unchanged.

     The previous mailfromd versions used the following paradigm to check if an IP address resolves:

            if hostname(%ip) != %ip
              ...

 -- Built-in Function: boolean primitive_ismx (string DOMAIN,
     string HOST)
     The DOMAIN argument is any valid domain name, the HOST is a host name or IP address.

     The function returns true if HOST is one of the MX records for the DOMAIN.

     If DOMAIN has no MX records, primitive_ismx raises exception not_found.

     If DNS query fails, the function raises failure or temp_failure, depending on the character of the failure.

 -- Library Function: boolean ismx (string DOMAIN,
     string HOST)
     The DOMAIN argument is any valid domain name, the HOST is a host name or IP address.

     The function returns true if HOST is one of the MX records for the DOMAIN.  Otherwise it returns false.

     If DOMAIN has no MX records, or if the DNS query fails, the function returns false.

 -- Built-in Function: string primitive_resolve (string HOST,
     [string DOMAIN])
     Reverse of primitive_hostname.  The primitive_resolve function returns the IP address for the host name specified by HOST argument.  If HOST has no A records, the function raises the exception not_found.

     If DNS lookup fails, the function raises failure or temp_failure, depending on the character of the failure.

     If the optional DOMAIN argument is given, it will be appended to HOST (with an intermediate dot), before querying the DNS.  For example, the following two expressions will return the same value:

          primitive_resolve("puszcza.gnu.org.ua")
          primitive_resolve("puszcza", "gnu.org.ua")

     There is a considerable internal difference between one-argument and two-argument forms of primitive_resolve: the former queries DNS for an A record, whereas the latter queries it for any record matching HOST in the domain DOMAIN and then selects the most appropriate one.  For example, the following two calls are equivalent:

          primitive_hostname("213.130.0.22")
          primitive_resolve("22.0.130.213", "in-addr.arpa")

     This makes it possible to use primitive_resolve for querying DNS black listing domains.  See match_dnsbl, for a working example of this approach.  See also match_rhsbl, for another practical example of the use of the two-argument form.

 -- Library Function: string resolve (string HOST,
     [string DOMAIN])
     Reverse of hostname.  The resolve function returns IP address for the host name specified by HOST argument.  If the host name cannot be resolved, or a DNS failure occurs, the function returns "0".

     This function is entirely equivalent to primitive_resolve (see above), except that it never raises exceptions.

All the functions above use DNS cache database for caching DNS answers.  The database is normally named STATEDIR/dns.db (see Local state directory, for the description of STATEDIR), but its actual name and location can be set at the run time using #pragma database dns file statement (see database).  Each record in the database consists of expiration time in seconds since the Epoch and the actual answer, which depends on the type of the query (see below).

When called, any DNS function first looks if the corresponding query is in the dns database.  It does so by building a lookup key, which consists of query type and query argument separated by a single whitespace.  The query type is:

A     for resolve
PTR   for hostname
MX    for hasmx and ismx

If the lookup key is present in the database and its expiration time is not yet reached, the coresponding answer from the database is returned.  Otherwise, the function performs actual DNS lookup, stores the obtained data in the database and returns it.

The expiration date for each new record is obtained by increasing the current system timestamp by the value of TTL obtained from the DNS reply.  If the latter carried several TTLs (e.g. if it was an MX request), the smallest of them is used.

If the DNS lookup failed, the result is stored in the database as well.  In this case, the value part of the record consists only of the expiration time (negative caching).  It is computed by adding to the current timestamp the negative expiration period, which defaults to 3600 seconds.  The negative expiration period can be configured using #pragma database dns negative-expire-interval statement.

Thus, there are two statements in the startup file that control the settings of the DNS databases.  The following example illustrates them:

     #pragma database dns file "/var/run/mailfromd/dns.db"
     #pragma database dns negative-expire-interval 7200

The contents of the DNS database can be listed using the following command:

     $ mailfromd --list --format=dns
     # or:
     $ mailfromd --list -Hdns

You can look up a particular entry as well.  For example:

     $ mailfromd --list -Hdns "A mx10.gnu.org"
     A mx10.gnu.org: Sun Dec  3 10:56:12 2006 199.232.76.166

The above example shows that the IP address of mx10.gnu.org and that it expires on Sunday, December 3d, at 10:56:12.

Of course, the rest of database management options are also valid for DNS database.  See Database Maintenance, for more information on these.

Database Functions
..................

 -- Built-in Function: boolean dbmap (string DB,
     string KEY, [number NULL])
     Looks up KEY in the DBM file DB and returns true if it is found.

     If the third argument is given and it is not zero, dbmap will include trailing null character into the key.  This is necessary, for example, for searching in /etc/mail/aliases.db.

     In general, if the database DB was created with makemap -N hash command, then you need to use non-zero third argument in the invocation of dbmap.

     See whitelisting, for an example of using this function.

 -- Built-in Function: string dbget (string DB,
     string KEY [, string DEFAULT, number NULL])
     Looks up KEY in the database DB and returns the value associated with it.  If the key is not found returns DEFAULT, if specified, or empty string otherwise.

     If the optional NULL argument is given and is not zero, the terminating null character will be included in KEY length.  It is useful if DB was created using makemap -N hash command.

 -- Library Function: string safedbget (string DB,
     string KEY [, string DEFAULT, number NULL])

     This is an exception-safe interface to dbget.  If some database error occurs while attempting to retrieve the record, safedbget returns empty string.

     To use this function, request the safedb.mf module:

          #require safedb

     See Modules, for a description of MFL module system.

 -- Built-in Function: void dbput (string DB,
     string KEY, string VALUE [, number NULL])
     Inserts in the database a record with the given KEY and VALUE.  If a record with the given KEY already exists, its value is replaced with the supplied one.

     If the optional NULL argument is given and is not zero, the terminating null character will be included in KEY length.

 -- Library Function: void safedbput (string DB,
     string KEY, string VALUE [, number NULL])
     This is an exception-safe interface to dbput.  If a database error occurs while attempting to retrieve the record, the function returns without reporting an error.

     To use this function, request the safedb.mf module:

          #require safedb

     See Modules, for a description of MFL module system.

 -- Built-in Function: void dbdel (string DB,
     string KEY)
 -- Built-in Function: void dbdel (string DB,
     string KEY, number NULL)
     Delete from the database the record with the given KEY.  If there are no such record, return without signalling error.

     The optional NULL argument has the same meaning as for the rest of database functions (see above).

 -- Built-in Function: number db_expire_interval (string FMT)
     The FMT argument is a database format identifier (see Database Formats).  If it is valid, the function returns the expiration interval for that format. How to obtain negative expiration?? Otherwise, db_expire_interval raises the not_found exception.

 -- Built-in Function: string db_name (string FMTID)
     The FMT argument is a database format identifier (see Database Formats).  The function returns the file name for that format.  If FMTID does not match any known format, db_name raises the not_found exception.

 -- Built-in Function: number db_get_active (string FMTID)
     Returns the flag indicating whether the cache database FMTID is currently enabled.  If FMTID does not match any known format, db_name raises the not_found exception.

 -- Built-in Function: void db_set_active (string FMTID, number ENABLE)
     Enables the cache database FMTID if ENABLE is not null, or disables it otherwise.  For example, to disable DNS caching, do:

          db_set_active("dns", 0)

 -- Built-in Function: boolean relayed (string DOMAIN)
     Returns true if the string DOMAIN is found in one of relayed domain files (see pragma relay, and see option relay). The usual construct is:

          relayed hostname ${client_addr}

     which yields true if the IP address from Sendmail variable client_addr is relayed by the local machine.

I/O functions
.............

MFL provides a set of functions for writing to disk files or pipes and reading from them.  The idea behind them is the same as in most other programming languages: first you open the resource with a call to open which returns a descriptor i.e. an integer number uniquely identifying the resource.  Then you can write or read from it using this descriptor.  Finally, when the resource is no longer needed, you can close it with a call to close.

 -- Built-in Function: number open (string NAME)
     The NAME argument specifies the name of a resource to open and the access rights you need to have on it.  The function returns a descriptor of the opened stream, which can subsequently be used as an argument to other I/O operations.

     First symbols of NAME determine the type of the resource to be opened and the access mode:

     >
          The rest of NAME is a name of a file.  Open the file for read-write access.  If the file exists, truncate it to zero length, otherwise create the file.

     >>
          The rest of NAME is a name of a file.  Open the file for appending (writing at end of file).  The file is created if it does not exist.

     |
          Treat the rest of NAME as the command name and its arguments. Run this command and open its standard input for writing.

     |&
          Treat the rest of NAME as the command name and its arguments. Run this command and set up for two-way communication with it, i.e writes to the descriptor returned by open will send data to the program's standard input, reads from the descriptor will get data from the program's standard output.

     If none of these prefixes is used, NAME is treated as a name of an existing file and open will attempt to open this file for reading.

     The open function will signal exception failure if it is unable to open the resource or get the required access to it.

 -- Built-in Function: void close (number RD)
     The argument RD is a resource descriptor returned by a previous call to open.  The function close closes the resource and deallocates any memory associated with it.

     close will signal range exception if RD lies outside of allowed range of resource descriptors. More info on it

Notice that you are not required to close resources opened by open. Any unclosed resource will be closed automatically upon the termination of the filtering program.

The following functions provide basic read/write capabilities.

 -- Built-in Function: void write (number RD, string STR)
     Writes the string STR to the resource descriptor RD.

     The function will signal range exception if RD lies outside of allowed range of resource descriptors, and ioerr exception if an I/O error occurs.

 -- Built-in Function: string getline (number RD)
     Read and return the next newline-terminated string from the resource descriptor RD.  The terminating newline character will be removed from the return value.

     The function will signal range exception if RD lies outside of allowed range of resource descriptors, and ioerr exception if an I/O error occurs.

The following example shows how mailfromd I/O functions can be used to automatically add IP addresses to an RBL zone:

     set nsupdate_cmd
       "/usr/bin/nsupdate -k /etc/bind/Kmail.+157+14657.private"

     func block_address(string addr)
     do
       number fd
       string domain

       set fd open "|%nsupdate_cmd"

       set domain revip(%addr) ".rbl.myzone.come"
       write(%fd, "prereq nxrrset %domain A\n"
                  "update add %domain 86400 A %addr\n\n"
     done

The function revip is defined in revip.

System functions
................

 -- Built-in Function: number time ()
     Return the time since the Epoch (00:00:00 UTC, January 1, 1970), measured in seconds.

 -- Built-in Function: string strftime (string FMT,
     number TIMESTAMP)
 -- Built-in Function: string strftime (string FMT,
     number TIMESTAMP, boolean GMT)
     Formats the time TIMESTAMP (seconds since the Epoch) according to the format specification FORMAT.   Ordinary characters placed in the format string are copied to the output without conversion. Conversion specifiers are introduced by a % character. See Time and Date Formats, for a detailed description of the conversion specifiers.  We recommend using single quotes around FMT to prevent % specifiers from being interpreted as Mailfromd variables (see Literals, for a discussion of quoted literals and variable interpretation within them).

     The TIMESTAMP argument can be a return value of time function (see above).

     For example:

          strftime('%Y-%m-%d %H:%M:%S %Z', 1164477564)
           => 2006-11-25 19:59:24 EET
          strftime('%Y-%m-%d %H:%M:%S %Z', 1164477564, 1)
           => 2006-11-25 17:59:24 GMT

 -- Built-in Function: number system (string STR)
     The function system executes a command specified in STR by calling /bin/sh -c string, and returns -1 on error or the return status of the command otherwise.

Interfaces to Third-Party Programs
..................................

A set of functions is defined for interfacing with other filters via TCP.  Currently implemented are interfaces with SpamAssassin spamd daemon and with ClamAV antivirus.

These functions can be used only in eom handler.

Both interfaces work much the same way: the remote filter is connected and the message is passed to it.  If the remote filter confirms that the message matches its requirements, the function returns true.  Notice that in practice that means that such a message should be rejected or deferred.

The address of the remote filter is supplied as the first argument in the form of a standard URL:

     PROTO://PATH[:PORT]

The PROTO part specifies the connection protocol.  It should be tcp for the TCP connection and file or socket for the connection via UNIX socket.  In the latter case the PROTO part can be omitted.  When using TCP connection, the PATH part gives the remote host name or IP address and the optional PORT specifies the port number or service name to use.  For example:

     # connect to remote.filter.net on port 3314:
     tcp://remote.filter.net:3314

     # the same, using symbolic service name (must be defined in
     # /etc/services):
     tcp://remote.filter.net:spamd

     # Connect via a local UNIX socket (equivalent forms):
     /var/run/filter.sock
     file:///var/run/filter.sock
     socket:///var/run/filter.sock

The description of the interface functions follows.

 -- Built-in Function: boolean sa (string URL, number PREC)
     Pass the message to the SpamAssassin daemon (spamd) at URL.  Return true if SpamAssassin considers it a spam, false otherwise.  The second arguments, PREC, gives the precision, in decimal digits, to be used when converting SpamAssassin diagnostic data and storing them into mailfromd variables. The floating point SpamAssassin data are converted to the integer mailfromd variables using the following relation:

          VAR = int(SA-VAR * PREC)

     where SA-VAR stands for the SpamAssassin value and VAR stands for the corresponding mailfromd one.  int() means taking the integer part.

     The function returns additional information via the following variables:

     sa_score
          The spam score, converted to integer as described above.

     sa_threshold
          The threshold, converted to integer.

     sa_keywords
          A string of comma-separated SpamAssassin keywords identifying this message.

     The sa function can signal the following exceptions: failure if the connection fails, url if the supplied URL is invalid and range if the supplied port number is out of the range 1--65535.

     The simplest way to use the function is:

          prog eom
          do
            if sa("tcp://192.168.10.1:3333", 3)
              reject 550 5.7.0
                   "Spam detected, score %sa_score with threshold %sa_threshold"
            fi
          done

     Here is a more advanced example:

          /* Convert a SpamAssassin code back to fixed point form
             using the given number of digits (prec) */
          func to_double_str(string code, number prec) returns string
          do
            if length(%code) > %prec
              return substring(%code, 0, length(%code)-%prec-1) '.'
                     substring(%code, length(%code)-%prec, -1)
            else
              return $1
            fi
          done

          prog eom
          do
            set prec 3
            if sa("tcp://192.168.10.1:3333", %prec)
              add "X-Spamd-Status" "SPAM"
            else
              add "X-Spamd-Status" "OK"
            fi
            add "X-Spamd-Score" to_double_str(%sa_score, %prec)
            add "X-Spamd-Threshold" to_double_str(%sa_threshold, %prec)
            add "X-Spamd-Keywords" %sa_keywords
          done

 -- Built-in Function: boolean clamav (string URL)
     Pass the message to the ClamAV daemon at URL.  Return true if it detects a virus in it.  Return virus name in clamav_virus_name global variable.

     The clamav function can signal the following exceptions: failure if connection failed, url if the supplied URL is invalid and range if the supplied port number is out of the range 1--65535.

     An example usage:

          prog eom
          do
            if clamav("tcp://192.168.10.1:6300")
              reject 550 5.7.0 "Infected with %clamav_virus_name"
            fi
          done

Special Test Functions
......................

 -- Built-in Function: number greylist
     (string KEY, number INTERVAL)

     Returns true if the KEY is found in the greylist database (controlled by #pragma database greylist pragma, see database).  The argument INTERVAL gives the greylisting interval in seconds.  The function stores the number of seconds left to the end of greylisting period in the internal variable greylist_seconds_left.  See Greylisting, for a detailed explanation.

     The function greylist can signal dbfailure exception.

 -- Built-in Function: boolean listens (string HOST, [number PORT])
     Returns true if the IP address or host name given by HOST argument listens on the port number PORT (default 25).

 -- Built-in Function: number rate (string KEY,
     number SAMPLE-INTERVAL, [number THRESHOLD])

     Returns the mail sending rate for KEY per SAMPLE-INTERVAL.  Optional THRESHOLD, if supplied, specifies the minimal number of mails needed to obtain the statistics.  The default is 2.  See Sending Rate, for a detailed discussion.

 -- Built-in Function: boolean validuser (string NAME)
     Returns true if authenticity of the user NAME is confirmed using mailutils authentication system.  See Local Account Verification, for more details.

 -- Library Function: boolean valid_domain (string DOMAIN)

     Returns true if the domain name DOMAIN has a corresponding A record or if it has any MX records, i.e. if it is possible to send mail to it.

     To use this function, require valid_domain.mf module (see Modules):

          #require valid_domain

 -- Library Function: number heloarg_test (string ARG,
     string REMOTE_IP, string LOCAL_IP)

     Verify if an argument of HELO (EHLO) command is valid.  To use this function, load heloarg_test.mf module (see Modules).

     Arguments:

     arg
          HELO (EHLO) argument.  Typically, the value of $s Sendmail macro;

     remote_ip
          IP address of the remote client.  Typically, the value of $client_addr Sendmail macro;

     local_ip
          IP address of this SMTP server;

     The function returns a number describing the result of the test, as described in the following table.

     Code   Meaning
     HELO_SUCCESS   ARG successfully passes all tests.
     HELO_MYIP   ARG is our IP address.
     HELO_IPNOMATCH   ARG is an IP, but it does not match
     the remote party IP address.
     HELO_ARGNORESOLVE   ARG is an IP, but it does not resolve.
     HELO_ARGNOIP   ARG is in square brackets, but it is not an IP
     address.
     HELO_ARGINVALID   ARG is not an IP address and does not
     resolve to one.
     HELO_MYSERVERIP   ARG resolves to our server IP.
     HELO_IPMISMATCH   ARG does not resolve to the remote
     client IP address.

Mail Sending Functions
......................

The mail sending functions are new interfaces, introduced in version 3.1.

The underlying mechanism for sending mail, called mailer, is specified by --mailer command line option.  This global setting can be overridden using the last optional argument to a particular function.  In any case, the mailer is specified in the form of a URL.

Mailer URL begins with a protocol specification. Two protocol specifications are currently supported: sendmail and smtp.  The former means to use a sendmail-compatible program to send mails.  Such a program must be able to read mail from its standard input and must support the following options:

-oi
     Do not treat '.' as message terminator.

-f ADDR
     Use ADDR as the address of the sender.

-t
     Get recipient addresses from the message.

These conditions are met by most existing MTA programs, such as exim or postfix (to say nothing of sendmail itself).

Following the protocol specification is the mailer location, which is separated from it with a colon.  For the sendmail protocol, the mailer location sets the full file name of the sendmail-compatible MTA binary, for example:

     sendmail:/usr/sbin/sendmail

A special form of a sendmail URL, consisting of protocol specification only (sendmail:) is also allowed.  It means ``use the sendmail binary from the _PATH_SENDMAIL macro in your /usr/include/paths.h file''.  This is the default mailer.

The smtp protocol means to use an SMTP server directly. In this case the mailer location consists of two slashes, followed by the IP address or host name of the SMTP server, and, optionally, the port number.  If the port number is present, it is separated from the rest of URL by a colon. For example:

     smtp://remote.server.net
     smtp://remote.server.net:24

 -- Built-in Function: void send_mail (string MSG,
     string TO [, string FROM, string MAILER])

     Sends message MSG to the email address TO.  The value of MSG must be a valid RFC 2822 message, consisting of headers and body.  The TO argument can contain several email addresses.  In this case the message will be sent to each recipient specified in TO.

     Optional arguments are:

     from
          Sets the sender address.  By default <> is used.

     mailer
          The URL of the mailer to use

     Sample usage:

              set message <<- EOT
                    Subject: Test message
                    To: Postmaster <postmaster@gnu.org.ua>
                    From: Mailfromd <devnull@gnu.org.ua>
                    X-Agent: %__package__ (%__version__)

                    Dear postmaster,

                    This is to notify you that our /etc/mailfromd.rc
                    needs a revision.
                    --
                    Mailfromd filter administrator
              EOT
              send_mail(%message, "postmaster@gnu.org.ua")

 -- Built-in Function: void send_text (string TEXT,
     string HEADERS, string TO
     [, string FROM, string MAILER])
     A more complex interface to mail sending functions.

     Mandatory arguments:

     text
          Text of the message to be sent.

     headers
          Headers for the message.

     to
          Recipient email address.

     Optional arguments:

     from
          Sender email address.

     mailer
          URL of the mailer to use.

     The above example can be rewritten using send_text as follows:

              set headers << -EOT
                    Subject: Test message
                    To: Postmaster <postmaster@gnu.org.ua>
                    From: Mailfromd <devnull@gnu.org.ua>
                    X-Agent: %__package__ (%__version__)
              EOT
              set text <<- EOT
                    Dear postmaster,

                    This is to notify you that our /etc/mailfromd.rc
                    needs a revision.
                    --
                    Mailfromd filter administrator
              EOT
              send_text(%text, %headers, "postmaster@gnu.org.ua")

 -- Built-in Function: void send_dsn (string TO,
     string SENDER, string RCPT, string TEXT
     [, string HEADERS, string FROM, string MAILER])
     This is an experimental interface which will change in the future versions.  It sends a message disposition notification (RFC 2298, RFC 1894), of type deleted to the email address TO. Arguments are:

     to
          Recipient email address.

     sender
          Original sender email address.

     rcpt
          Original recipient email address.

     text
          Notification text.

     Optional arguments:

     headers
          Message headers

     from
          Sender address.

     mailer
          URL of the mailer to use.

Blacklisting Functions
......................

The functions described in this subsection allow to check whether the given IP address is listed in certain black list DNS zone.

 -- Library Function: boolean match_dnsbl (string ADDRESS,
     string ZONE, string RANGE)

     This function looks up ADDRESS in the DNS blacklist zone ZONE and checks if the return falls into the given RANGE of IP addresses.

     It is intended as a replacement for the Sendmail macros dnsbl and enhdnsbl.

     To use match_dnsbl, require match_dnsbl.mf module (see Modules).

     Arguments:

     address
          IP address of the SMTP server to be tested.

     zone
          FQDN of the DNSbl zone to test against.

     range
          The range of IP addresses in CIDR notation or the word ANY, which stands for 127.0.0.0/8.

     The function returns true if dns lookup for ADDRESS in the zone DNSBL yields an IP that falls within the range, specified by CIDR.  Otherwise, it returns false.  If any of ADDRESS or CIDR is invalid, match_dnsbl returns false.

 -- Library Function: boolean match_rhsbl (string EMAIL,
     string ZONE, string RANGE)

     This function checks if the IP address, corresponding to the domain part of EMAIL is listed in the RHS DNS blacklist zone ZONE, and if so, whether its record falls into the given range of IP addresses RANGE.

     It is intended as a replacement for the Sendmail macro rhsbl by Derek J. Balling.

     To use this function, require match_rhsbl.mf module (see Modules).

     Arguments:

     email
          E-mail address, whose domain name should be tested (usually, it is $f)

     zone
          Domain name of the RHS DNS blacklist zone.

     range
          The range of IP addresses in CIDR notation.

SPF Functions
.............

Sender Policy Framework, or SPF for short, is an extension to SMTP protocol that allows to identify forged identities supplied with the MAIL FROM and HELO commands.  The framework is explained in detail in RFC 4408 (http://tools.ietf.org/html/rfc4408) and on the SPF Project Site (http://www.openspf.org/).  The following description is a short introduction only, and the users are encouraged to refer to the original specification for the detailed description of the framework.

The domain holder publishes an SPF record -- a special DNS resource record that contains a set of rules declaring which hosts are, and are not, authorized to use a domain name for HELO and MAIL FROM identities.  This resource record is usually of type TXT. (Although RFC 4408
introduces a special SPF record type for this purpose, it is
not yet widely used.  As of version VERSION, MFL
does not support SPF DNS records.)

The MFL script can verify if the identity matches the published SPF record by calling check_host function and analyzing its return code.  The function can be called either in helo or in envfrom handler.  Its arguments are:

ip
     The IP address of the SMTP client that is emitting the mail. Usually it is $client_addr.

domain
     The domain that provides the sought-after authorization information; Normally it is the domain portion of the MAIL FROM or HELO identity.

sender
     The MAIL FROM identity.

helo_domain
     The HELO identity.

my_domain
     The SMTP domain served by the local server.

The function returns a numeric result code.  For convenience, all possible return values are defined as macros in module spf.mf. The table below describes each value along with the recommended actions for it:

None
     A result of None means that no records were published by the domain or that no checkable sender domain could be determined from the given identity.  The checking software cannot ascertain whether or not the client host is authorized.  Such a message can be subject to further checks that will decide about its fate.

Neutral
     The domain owner has explicitly stated that he cannot or does not want to assert whether or not the IP address is authorized.  This result must be treated exactly like None; the distinction between them exists only for informational purposes

Pass
     The client is authorized to send mail with the given identity.  The message can be subject to further policy checks with confidence in the legitimate use of the identity or it can be accepted in the absence of such checks.

Fail
     The client is not authorized to use the domain in the given identity.  The proper action in this case can be to mark the message with a header explicitly stating it is spam, or to reject it outright.

     If you choose to reject such mails, we suggest to use reject 550 5.7.1, as recommended by RFC 4408.  The reject can return either a default explanation string, or the one supplied by the domain that published the SPF records, as in the example below:

            reject 550 5.7.1 "SPF check failed:\n%spf_explanation"

     (see below for the description of spf_explanation variable.)

SoftFail
     The domain believes the host is not authorized but is not willing to make that strong of a statement.  This result code should be treated as somewhere in between a Fail and a Neutral.  It is not recommended to reject the message based solely on this result.

TempError
     A transient error occurred while performing SPF check.  The proper action in this case is to accept or temporarily reject the message.  If you choose the latter, we suggest to use SMTP reply code of 451 and DSN code 4.4.3, for example:

            tempfail 451 4.4.3
                     "Transient error while performing SPF verification"

PermError
     This result means that the domain's published records could not be correctly interpreted.  This signals an error condition that requires manual intervention to be resolved, as opposed to the TempError result.

The following example illustrates the use of SPF verification in envfrom handler:

     #include_once <status.mfh>
     #require spf

     prog envfrom
     do
       switch check_host($client_addr, domainpart($f), $f, $s)
       do
       case Fail:
         string text ""
         if %spf_explanation != ""
           set text %text "\n" %spf_explanation
         fi
         reject 550 5.7.1 "SPF MAIL FROM check failed" %text

       case Pass:
         accept

       case TempError:
         tempfail 451 4.4.3
                  "Transient error while performing SPF verification"

       default:
         on poll $f do
         when success:
           accept
         when not_found or failure:
           reject 550 5.1.0 "Sender validity not confirmed"
         when temp_failure:
           tempfail 450 4.7.0 "Temporary failure during sender verification"
         done
       done
     done

The SPF support is implemented in MFL in two layers: a built-in layer that provides basic support, and a library layer that implements result caching.

The library layer is implemented in spf.mf module (see Modules).

The rest of this node describes available SPF functions and variables.

 -- Built-in Function: number spf_check_host (string IP,
     string DOMAIN, string SENDER, string HELO_DOMAIN,
     string MY_DOMAIN)

     This function is the basic implementation of the check_host function, defined in RFC 4408, chapter 4.  It fetches SPF records, parses them, and evaluates them to determine whether a particular host (IP) is or is not permitted to send mail from a given email address (SENDER).  The function returns an SPF result code.

     Arguments are:

     ip
          The IP address of the SMTP client that is emitting the mail. Usually it is $client_addr.

     domain
          The domain that provides the sought-after authorization information; Normally it is the domain portion of the MAIL FROM or HELO identity.

     sender
          The MAIL FROM identity.

     helo_domain
          The HELO identity.

     my_domain
          The SMTP domain served by the local server.

     Before returning the spf_check_host function stores additional information in global variables:

     spf_explanation
          If the result code is not Pass, this variable contains the explanation string as returned by the publishing domain, prefixed with the value of the global variable spf_explanation_prefix.

          For example, if spf_explanation_prefix contains The domain %{o} explains: , and the publishing domain example.com returns the explanation string Please see http://www.example.com/mailpolicy.html, than the value of spf_explanation will be:

               The domain example.com explains:
               Please see http://www.example.com/mailpolicy.html

          (see RFC 4408 (http://tools.ietf.org/html/rfc4408), chapter 8, for the description of SPF macro facility).

     spf_mechanism
          Set to the name of a SPF mechanism that decided about the result code of the SPF record.

     spf_ttl
          Set to the minimum interval of time, in seconds, during which it is safe to cache the result value.  This value is used by check_host function.

 -- Built-in Function: number spf_test_record (string RECORD,
     string IP, string DOMAIN, string SENDER,
     string HELO_DOMAIN, string MY_DOMAIN)

     Evaluate SPF record RECORD as if it were published by DOMAIN.  The rest of arguments are the same as for spf_check_host above.

     This function is designed primarily for testing and debugging purposes.  You would hardly need to use it.

     The spf_test_record function sets the same global variables as spf_check_host.

 -- Library Function: number check_host (string IP,
     string DOMAIN, string SENDER, string HELO)

     This function implements the check_host function, defined in RFC 4408, chapter 4.  It fetches SPF records, parses them, and evaluates them to determine whether a particular host (IP) is or is not permitted to send mail from a given email address (SENDER).  The function returns an SPF result code.

     This function differs from the built-in spf_check_host in that it implements caching of the results, thereby reducing network traffic.

     The arguments are:

     ip
          The IP address of the SMTP client that is emitting the mail. Usually it is $client_addr.

     domain
          The domain that provides the sought-after authorization information; Normally it is the domain portion of the MAIL FROM or HELO identity.

     sender
          The MAIL FROM identity.

     helo
          The HELO identity.

     Before returning, the check_host function stores additional information in the same global variables, as spf_check_host.  See the list, for details. In addition, it sets the variable spf_cached to 1 if cached data were used, or to 0 otherwise.

 -- Library Function: string spf_status_string (number CODE)
     Converts numeric SPF result CODE to its string representation.

 -- Variable: Built-in variable string spf_explanation
     If the result code of check_host (or spf_check_host or spf_test_record function is not Pass, this variable contains the explanation string as returned by the publishing domain, prefixed with the value of the global variable spf_explanation_prefix.

     For example, if spf_explanation_prefix contains The domain %{o} explains: , and the publishing domain example.com returns the explanation string Please see http://www.example.com/mailpolicy.html, than the value of spf_explanation will be:

          The domain example.com explains:
          Please see http://www.example.com/mailpolicy.html

 -- Variable: Built-in variable string spf_mechanism
     Set to the name of a SPF mechanism that decided about the result code of the SPF record.

 -- Variable: Built-in variable number spf_ttl
     Set to the minimum interval of time, in seconds, during which it is safe to cache the result value.  This value is used by check_host function.

 -- Variable: Built-in variable string spf_explanation_prefix
     The prefix to be appended to the explanation string before storing it in the spf_explanation variable.  This string can contain valid SPF macros (see RFC 4408 (http://tools.ietf.org/html/rfc4408), chapter 8), for example:

          set spf_explanation_prefix "%{o} explains: "

     The default value is "" (an empty string).

 -- Variable: Library variable string spf_database
     Sets the full name of  DBM file used to cache the results of the SPF functions.  By default is is %__statedir__/spf.db.

 -- Variable: Library variable number spf_negative_ttl
     Sets the interval of time during which negative replies (result code None) will be stored in the database.  By default it is 1 day.

 -- Variable: Library variable number spf_cached
     Set by check_host function to 1 if the function made the decision based on the cached data, or to 0 otherwise.

National Language Support Functions
...................................

The National Language Support functions allow you to write your scripts in such a way, that any textual messages they display are automatically translated to your native language, or, more precisely, to the language required by your current locale.

This section assumes the reader is familiar with the concepts of program internationalization and localization.  If not, please refer to The Purpose of GNU gettext (GNU gettext manual), before reading further.

In general, internationalization of any MFL script follows the same rules as described in the GNU gettext manual. First of all, you select the program message domain, i.e. the identifier of a set of translatable messages your script contain. This identifier is then used to select appropriate translation. The message domain is set using textdomain function.  For the purposes of this section, let's suppose the domain name is myfilter.  All NLS functions are provided by nls.mf module, so you must require this module prior to using any of them.

To find translations of textual message to the current locale, the underlying gettext mechanism will look for file DIRNAME/LOCALE/LC_MESSAGES/DOMAINNAME.mo, where DIRNAME is the message catalog hierarchy name, LOCALE is the locale name, and DOMAINNAME is the name of the message domain.  By default DIRNAME is /usr/local/share/locale, but you may change it using bindtextdomain function.  The right place for this initial NLS setup is in the begin block (see begin/end). To summarize all the above, the usual NLS setup will look like:

     #require nls

     begin
     do
       textdomain("myfilter")
       bindtextdomain("myfilter", "/usr/share/locale");
     done

For example, given the settings above, and supposing the environment variable LC_ALL is set to pl, translations will be looked in file /usr/share/locale/pl/LC_MESSAGES/myfilter.mo.

Once this preparatory work is done, you can request each message to be translated by using gettext function, or _ (underscore) macro.  For example, the following statement will produce translated textual description for 450 response:

     tempfail 450 4.1.0 _("Try again later")

Of course it assumes that the appropriate myfile.mo file already exists.  If it does not, nothing bad happens: in this case the macro _  (as well as gettext function) will simply return its argument unchanged, so that the remote party will get the textual message in English.

The mo files are binary files created from po source files using msgfmt utility, as described in Producing Binary MO Files (GNU gettext manual).  In turn, the format of po files is described in The Format of PO Files (GNU gettext manual).

 -- Built-in Function: string bindtextdomain (string DOMAIN,
     string DIRNAME)
     This function sets the base directory of the hierarchy containing message catalogs for a given message domain.

     DOMAIN is a string identifying the textual domain.  If it is not empty, the base directory for message catalogs belonging to domain DOMAIN is set to DIRNAME.  It is important that DIRNAME be an absolute pathname; otherwise it cannot be guaranteed that the message catalogs will be found.

     If DOMAIN is "", bindtextdomain returns the previously set base directory for domain DOMAIN.

The rest of this section describes the NLS functions supplied by the nls.mf module.

 -- Built-in Function: string dgettext (string DOMAIN,
     string MSGID)
     dgettext attempts to translate the string MSGID into the currently active locale, according to the settings of the textual domain DOMAIN.  If there is no translation available, dgettext returns MSGID unchanged.

 -- Built-in Function: string dngettext
     (string DOMAIN, string MSGID, string MSGID_PLURAL,
     number N)
     The dngettext functions attempts to translate a text string into the language specified by the current locale, by looking up the appropriate singular or plural form of the translation in a message catalog, set for the textual domain DOMAIN.

     See Additional functions for plural forms (GNU gettext utilities), for a discussion of the plural form handling in different languages.

 -- Library Function: string textdomain (string DOMAIN)
     The textdomain function sets the current message domain to DOMAIN, if it is not empty.  In any case the function returns the current message domain.  The current domain is mailfromd initially.  For example, the following sequence of textdomain invocations will yield:

          textdomain("") => "mailfromd"
          textdomain("myfilter") => "myfilter"
          textdomain("") => "myfilter"

 -- Library Function: string gettext (string MSGID)
     gettext attempts to translate the string MSGID into the currently active locale, according to the settings of the current textual domain (set using textdomain function).  If there is no translation available, gettext returns MSGID unchanged.

 -- Library Function: string ngettext
     (string MSGID, string MSGID_PLURAL, number N)
     The ngettext functions attempts to translate a text string into the language specified by the current locale, by looking up the appropriate singular or plural form of the translation in a message catalog, set for the current textual domain.

     See Additional functions for plural forms (GNU gettext utilities), for a discussion of the plural form handling in different languages.

Debugging Functions
...................

These functions allow to enable debugging and tracing for a certain piece of code.

 -- Built-in Function: void debug (string SPEC)
     Enable debugging.  The value of SPEC sets the debugging level. See debugging level specification, for a description of its format.

 -- Built-in Function: number debug_level ([string SRCNAME])
     This function returns the debugging level currently in effect for the source module SRCNAME, or the global debugging level, if called without arguments.

     For example, if the program was started with --debug=20,engine=8 option, then:

          debug_level() => 20
          debug_level("engine") => 8
          debug_level("db") => 0

 -- Built-in Function: string debug_spec ([string SRCNAMES])
     Returns the current debugging level specification, as given by --debug command line option or by #pragma option debug statement.

     If the argument SRCNAMES is specified, it is treated as a comma-separated list of source names for which the debugging specification is to be returned.

     For example, if mailfromd was started with --debug=20,spf=50,engine=8,db=30 option, then:

          debug_spec("all,engine") => "20,engine=8"
          debug_spec("engine,db") => "db=30,engine=8"

     When called without arguments, debug_spec returns full debugging level specification, as shown in the example below:

          debug_spec()
          => "20,debug=0,cache=0,dnscache=0,db=30,dns=0,dnsbase=0,engine=8,
          gram=0,lex=0,main=0,mf-status=0,mu_dbm=0,optab=0,prog=0,
          spf=50,stack=0,symtab=0,rate=0,bi_db=0,bi_dns=98,bi_io=0,
          bi_ipaddr=0,bi_mail=0,bi_poll=0,bi_sa=0,bi_spf=0,bi_string=7,
          bi_system=0,bi_other=0,bi_vars=0"

These three functions are intended to complement each other.  The calls to debug can be placed around some piece of code you wish to debug, to enable specific debugging information for this code fragment only.  For example:

         /* Save debugging level for dns.c source */
         set dlev debug_level("dns")
         /* Set new debugging level */
         debug("dns=80")
         .
         .
         .
         /* Restore previous level */
         debug("dns=%dlev")

You can also change debugging levels for several modules simultaneously:

         /* Save debugging specifications for the sources we are
          * interested in:
          */
         set dspec debug_spec("dns,dnsbase,db")
         /* Set new debugging spec */
         debug("dns=80,dnsbase=100,db=1")
         .
         .
         .
         /* Restore previous debug specification */
         debug("%dspec")

 -- Built-in Function: void program_trace (string MODULE)
     Enable tracing for a set of modules given in MODULE argument. See --trace-program, for a description of its format.

 -- Built-in Function: void cancel_program_trace (string MODULE)
     Disable tracing for given modules.

This pair of functions is also designed to be used together in a bracket-like fashion.  They are useful for debugging mailfromd, but are not advised to use otherwise, since tracing slows down the execution considerably.

 -- Built-in Function: void stack_trace ()
     Generate a stack trace in this point.  See tracing runtime errors, for the detailed description of stack traces.

User-Defined Functions
----------------------

Complicated filter scripts can often be simplified and made more manageable by defining new functions.

The function definitions can appear anywhere between the handler declarations in a filter program.  The only requirement is that the function definition occurs before the place where the function is invoked.

The definition of a function named NAME looks like this:

     func NAME (PARAM-DECL) returns DATA-TYPE
     do
       FUNCTION-BODY
     done

NAME is the name of the function to define, PARAM-DECL is a comma-separated list of parameter declarations.  The syntax of the latter is the same as that of variable declarations (see Variable declarations), i.e.:

     TYPE NAME

declares the parameter NAME having the type TYPE.  The TYPE is string or number.

For example, the following declares a function sum, that takes two numeric arguments and returns a numeric value:

     func sum(number x, number y) returns number

Parameters are referenced in the FUNCTION-BODY by their name, using the syntax for variable references: %NAME.  The value of a parameter can be altered using set statement, the same way as for variables.

A function can be declared to take a certain number of optional arguments.  In function declaration, optional abstract arguments must be placed after the mandatory ones, and must be separated from them with a semicolon.  The following example is a definition of function foo, which takes two mandatory and two optional arguments:

     func foo(string msg, string email; number x, string pfx)

Mandatory arguments are: msg and email.  Optional arguments are: x and pfx.  The actual number of parameters supplied to the function is returned by a special construct $#.  In addition, the special construct @ARG evaluates to the ordinal number of variable ARG in the list of formal parameters (the first argument has number 0).  These two constructs can be used to verify whether an argument is supplied to the function.

The actual parameter for argument n is supplied if the number of actual parameters ($#) is greater than its ordinal number in the declaration list (@N).  This, to check if PFX is given:

     func foo(string msg, string email; number x, string pfx)
     do
       if $# > @pfx
         ...
       fi

The default mailfromd installation provides a special macro for this purpose: see defined.  Using it, the example above will be rewritten as:

     func foo(string msg, string email; number x, string pfx)
     do
       if defined(pfx)
         ...
       fi

Within a function body, optional arguments are referenced exactly the same way as the mandatory ones.  Attempt to dereference an optional argument for which no actual aparameter was supplied, results in an undefined value, so be sure to chech whether a parameter is passed before dereferencing it.

The FUNCTION-BODY is any list of valid mailfromd statements.  In addition to the statements discussed below (see Statements) it can also contain the return statement, which is used to return a value from the function.  The syntax of the return statement is

       return VALUE

As an example of this, consider the following code snippet that defines the function sum to return a sum of its two arguments:

     func sum(number x, number y) returns number
     do
             return %x + %y
     done

The returns part in the function declaration is optional.  A declaration lacking it defines a procedure, or void function, i.e. a function that is not supposed to return any value. Such functions cannot be used in expressions, instead they are used as statements (see Statements).  The following example shows a function that emits a customized temporary failure notice:

     func stdtf()
     do
       tempfail 451 4.3.5 "Try again later"
     done

A variable declared within a function becomes a local variable to this function.  Its lexical scope ends with the terminating done statement.

Parameters, local variables and global variables are using separate namespaces, so a parameter name can coincide with the name of a global, in which case a parameter is said to shadow the global.  All references to its name will refer to the parameter, until the end of its scope is reached, where the global one becomes visible again.  Consider the following example:

     number x

     func foo(string x)
     do
       echo "foo: %x"
     done

     prog envfrom
     do
       set x "Global"
       foo("Local")
       echo %x
     done

Running mailfromd --test with this configuration will display:

     foo: Local
     Global

For compatibility with previous versions, mailfromd also supports the legacy format of function declarations:

     func NAME (PARAM-TYPES) returns RETTYPE

Here, PARAM-TYPES is a comma-separated list of parameter types. The following abbreviations can be used: s for string and n for numeric.  The same holds true for RETTYPE as well.

The parameters declared this way are called positional parameters.  The function can access its actual arguments using the notation $N, where N is the ordinal number of the argument (Well, to tell you the truth, you can access named arguments using positional notation as well.  But such a mixing of styles is not recommended.).  Arguments are counted from left to right.  The first argument is $1.

Using this syntax, the above definition of sum would look like:

     func sum(n, n) returns n
     do
       return $1 + $2
     done

Some Useful Functions
.....................

To illustrate the concept of user-defined functions, this subsection shows definitions of some of the library functions shipped with mailfromd.  These functions are contained in modules installed along with the mailfromd binary.  To use any of them in your code, require the appropriate module as described in #require statement, e.g. to use the revip function, do #require revip.

Functions and their definitions:

 1. revip

     The function revip (see revip) is implemented as follows:

          func revip(string ip) returns string
          do
            return inet_ntoa(ntohl(inet_aton(%ip)))
          done

     Previously it was implemented using regular expressions.  Below we include this variant as well, as an illustration for the use of regular expressions:

          #pragma regex push +extended
          func revip(string ip) returns string
          do
            if %ip matches '([0-9]+)\.([0-9]+)\.([0-9]+)\.([0-9]+)'
              return "\4.\3.\2.\1"
            fi
            return %ip
          done
          #pragma regex pop

 2. strip_domain_part

     This function returns at most N last components of the domain name DOMAIN (see strip_domain_part).

          #pragma regex push +extended

          func strip_domain_part(string domain, number n) returns string
          do
            if %n = 0
              return domainpart %domain
            elif domainpart(%domain) matches '.*((\.[^.]+){' $2 '})'
              return substring(\1, 1, -1)
            else
              return %domain
            fi
          done
          #pragma regex pop

 3. valid_domain

     See valid_domain, for a description of this function.  Its definition follows:

          #require dns

          func valid_domain(string domain) returns number
          do
            return not (resolve(%domain) = "0" and not hasmx(%domain))
          done

 4. match_dnsbl

     The function match_dnsbl (see match_dnsbl) is defined as follows:

          #require dns
          #require match_cidr
          #pragma regex push +extended

          func match_dnsbl(string address, string zone, string range)
              returns number
          do
            string rbl_ip
            if %range = 'ANY'
              set rbl_ip '127.0.0.0/8'
            else
              set rbl_ip %range
              if not %range matches '^([0-9]{1,3}\.){3}[0-9]{1,3}$'
                return 0
              fi
            fi

            if not (%address matches '^([0-9]{1,3}\.){3}[0-9]{1,3}$'
                    and %address != %range)
              return 0
            fi

            if %address matches
                  '^([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})$'
              if match_cidr (resolve ("\4.\3.\2.\1", %zone), %rbl_ip)
                return 1
              else
                return 0
              fi
            fi
            # never reached
          done

Expressions
===========

Expressions are language constructs, that evaluate to a value, that can subsequently be echoed, tested in a conditional statement, assigned to a variable or passed to a function.

Constant Expressions
--------------------

Literals and numbers are constant expressions.  They evaluate to string and numeric types.

Function Calls
--------------

A function call is an expression.  Its type is the return type of the function.

Concatenation
-------------

Any two adjacent expressions are converted to string type and concatenated, producing a new string.  Thus, if $f is smith, and $client_addr is 10.10.1.1, then:

     $f "-" $client_addr => "smith-10.10.1.1"

This feature is often used in filter scripts to generate key values for database lookup functions.

Arithmetic Operations
---------------------

The filter script language offers the common arithmetic operators: +, -, * and /.  All of them follow usual precedence rules and work as you would expect them to.

Relational Expressions
----------------------

Relational expressions are:

Table: Relational Expressions

Expression   Result
X < Y   True if X is less than Y.
X <= Y   True if X is less than or equal to Y.
X > Y   True if X is greater than Y.
X >= Y   True if X is greater than or equal to Y.
X = Y   True if X is equal to Y.
X != Y   True if X is not equal to Y.

The relational expressions apply to string as well as to numbers. When a relational operation applies to strings, case-sensitive comparison is used, e.g.:

     "String" = "string" => False
     "String" < "string" => True

Special Comparisons
-------------------

In addition to the traditional relational operators, described above, mailfromd provides two operators for regular expression matching:

Table: Regular Expression Matching

Expression   Result
X matches Y   True if the string X matches the
regexp denoted by Y.
X fnmatches Y   True if the string X matches the
globbing pattern denoted by Y.

The type of the regular expression used by matches operator is controlled by #pragma regex (see pragma regex).  For example:

     $f => "gray@gnu.org.ua"
     $f matches '.*@gnu\.org\.ua' => true
     $f matches '.*@GNU\.ORG\.UA' => false
     #pragma regex +icase
     $f matches '.*@GNU\.ORG\.UA' => true

The fnmatches operator compares its left-hand operand with a globbing pattern (see glob(7)) given as its right-hand side operand.  For example:

     $f => "gray@gnu.org.ua"
     $f fnmatches "*ua" => true
     $f fnmatches "*org" => false
     $f fnmatches "*org*" => true

Both operators have a special form, for MX pattern matching. The expression:

       X mx matches Y

is evaluated as follows: first, the expression X is analyzed and, if it is an email address, its domain part is selected.  If it is not, its value is used verbatim.  Then the list of MXs for this domain is looked up.  Each of MX names is then compared with the regular expression Y.  If any of the names matches, the expression returns true.  Otherwise, its result is false.

Similarly, the expression:

       X mx fnmatches Y

returns true only if any of the MXs for (domain or email) X match the globbing pattern Y.

Both mx matches and mx fnmatches can signal the following exceptions: temp_failure, failure.

The value of any parenthesized subexpression occurring within the right-hand side argument to matches or mx matches can be referenced using the notation \D, where D is the ordinal number of the subexpression (subexpressions are numbered from left to right, starting at 1).  This notation is allowed in the program text as well as within double-quoted strings and here-documents, for example:

     if $f matches '.*@\(.*\)\.gnu\.org\.ua'
       set message "Your host name is \1;"
     fi

Remember that the grouping symbols are \( and \) for basic regular expressions, and ( and ) for extended regular expressions.  Also make sure you properly escape all special characters (backslashes in particular) in double-quoted strings, or use single-quoted strings to avoid having to do so (see singe-vs-double, for a comparison of the two forms).

Boolean Expressions
-------------------

A boolean expression is a combination of relational or matching expressions using the boolean operators and, or and not, and, eventually, parentheses to control nesting:

table: Boolean Operators

Expression   Result
X and Y   True only if both X and
Y are true.
X or Y   True if any of X or Y
is true.
not X   True if X is false.

Binary boolean expressions are computed using shortcut evaluation:

X and Y
     If X => false, the result is false and Y is not evaluated.

X or Y
     If X => true, the result is true and Y is not evaluated.

Operator Precedence
-------------------

Operator precedence is an abstract value associated with each language operator, that determines the order in which operators are executed when they appear together within a single expression. Operators with higher precedence are executed first.  For example, * has a higher precedence than +, therefore the expression a + b * c is evaluated in the following order: first b is multiplied by c, then a is added to the product.

When operators of equal precedence are used together they are evaluated from left to right (i.e., they are left-associative), except for comparison operators, which are non-associative (these are explicitly marked as such in the table below).  This means that you cannot write:

     if 5 <= %x <= 10

Instead, you should write:

     if 5 <= %x and %x <= 10

The precedences of the mailfromd operators where selected
so as to match that used in most programming languages. (The
only exception is not, whose precedence in MFL is
much lower than usual (in most programming languages it has the same
precedence as unary -).  This allows to write conditional
expressions in more understandable manner.  Consider the following
condition:)

     if not %x < 2 and %y = 3

It is understood as ``if x is not less than 2 and y equals 3'', whereas with the usual precedence for not it would have meant ``if negated x is less than 2 and y equals 3''.

The following table lists all operators in order of decreasing precedence:

(...)
     Grouping

$ %
     Sendmail macros and mailfromd variables

* /
     Multiplication, division

+ -
     Addition, subtraction

< <= >= >
     Relational operators (non-associative)

= != matches fnmatches
     Equality and special comparison (non-associative)

&
     Logical (bitwise) AND

^
     Logical (bitwise) XOR

|
     Logical (bitwise) OR

not
     Boolean negation

and
     Logical and.

or
     Logical or

String Concatenation
     There is no special operator, the operands are simply written side by side (see Concatenation).

Type Casting
------------

When two operands on each side of a binary expression have different type, mailfromd evaluator has to coerce them to a common type.  This is known as implicit type casting.  The rules for implicit type casting are:

 1. Both arguments to an arithmetical operation are cast to numeric type.

 2. Both arguments to the concatenation operation are cast to string.

 3. Both arguments to `match' or `fnmatch' function are cast to string.

 4. The argument of the unary negation (arithmetical or boolean) is cast to numeric.

 5. Otherwise the right-hand side argument is cast to the type of the left-hand side argument.

There is no special syntactic sugar for explicit type casting. However you can use the latter rule to make sure an expression is of desired type: to cast an expression to string, concatenate an empty string to it, e.g.:

       "" %var

Similarly, to cast an expression to numeric, add zero to it:

       0 + %var

Statements
==========

Statements are language constructs, that, unlike expressions, do not return any value.  Statements execute some actions, such as assigning a value to a variable, or serve to control the execution flow in the program.

Action Statements
-----------------

An action statement instructs mailfromd to perform a certain action over the message being processed.  There are two kinds of actions: return actions and header manipulation actions.

Return actions tell Sendmail to return given response code to the remote party.  There are five such actions:

accept
     Return an accept reply.  The remote party will continue transmitting its message.

reject [CODE] [EXCODE] [MESSAGE]
     Return a reject reply.  The remote party will have to cancel transmitting its message.  The three arguments are optional, their usage is described below.

tempfail [CODE] [EXCODE] [MESSAGE]
     Return a temporary failure reply. The remote party can retry to send its message later.  The three arguments are optional, their usage is described below.

discard
     Instructs Sendmail to accept the message and silently discard it without delivering it to any recipient.

continue
     Stops the current handler and instructs Sendmail to continue processing of the message.

Two actions, reject and tempfail can take up to three optional parameters.  The first argument is a three-digit RFC 2821 reply code.  It must begin with 5 for reject and with 4 for tempfail.  If two arguments are supplied, the second argument must be either an extended reply code (RFC 1893/2034) or a textual string to be returned along with the SMTP reply.  Finally, if all three arguments are supplied, then the second one must be an extended reply code and the third one must give the textual string.  The following examples illustrate the possible ways of using the reject statement:

     reject
     reject 503
     reject 503 5.0.0
     reject 503 "Need HELO command"
     reject 503 5.0.0 "Need HELO command"

Header manipulation actions allow you to add, delete or modify message RFC 2822 headers.

add NAME STRING
     Add the header NAME with the value STRING. E.g.:

          add "X-Seen-By" "Mailfromd VERSION"

     (notice argument quoting)

replace NAME STRING
     The same as add, but if the header NAME already exists, it will be removed first, for example:

          replace "X-Last-Processor" "Mailfromd VERSION"

delete NAME
     Delete the header named NAME:

          delete "X-Envelope-Date"

Variable Assignments
--------------------

An assignment is a special statement that assigns a value to the variable.  It has the following syntax:

     set NAME VALUE

where NAME is the variable name and VALUE is the value to be assigned to it.

Assignment statements can appear in any part of a filter program. If an assignment occurs outside of function or handler definition, the VALUE must be a literal value (see Literals).  If it occurs within a function or handler definition, VALUE can be any valid mailfromd expression (see Expressions).  In this case, the expression will be evaluated and its value will be assigned to the variable.  For example:

     set delay 150

     prog envfrom
     do
       set delay %delay * 2
       ...
     done

The pass statement
------------------

The pass statement has no effect.  It is used in places where no statement is needed, but the language syntax requires one:

     on poll $f do
     when success:
       pass
     when not_found or failure:
       reject 550
     done

The echo statement
------------------

The echo statement concatenates all its arguments into a single string and sends it to the syslog using the priority info.  It is useful for debugging your script, in conjunction with built-in constants (see Built-in constants), for example:

     func foo(number x)
     do
       echo "%__file__:%__line__: foo called with arg %x"
       ...
     done

Conditional Statements
======================

Conditional expressions, or conditionals for short, test some conditions and alter the control flow depending on the result.  There are two kinds of conditional statements: if-else branches and switch statements.

The syntax of an if-else branching construct is:

       if CONDITION THEN-BODY [else ELSE-BODY] fi

Here, CONDITION is an expression that governs control flow within the statement.  Both THEN-BODY and ELSE-BODY are lists of mailfromd statements.  If CONDITION is true, THEN-BODY is executed, if it is false, ELSE-BODY is executed.  The else part of the statement is optional.  The condition is considered false if it evaluates to zero, otherwise it is considered true.  For example:

     if $f = ""
       accept
     else
       reject
     fi

This will accept the message if the value of the Sendmail macro $f is an empty string, and reject it otherwise.  Both THEN-BODY and ELSE-BODY can be compound statements including other if statements.  Nesting level of conditional statements is not limited.

To facilitate writing complex conditional statements, the elif keyword can be used to introduce alternative conditions, for example:

     if $f = ""
       accept
     elif $f = "root"
       echo "Mail from root!"
     else
       reject
     fi

Another type of branching instruction is switch statement:

     switch CONDITION
     do
     case X1 [or X2 ...]:
       STMT1
     case Y1 [or Y2 ...]:
       STMT2
       .
       .
       .
     [default:
       STMT]
     done

Here, X1, X2, Y1, Y2 are literal expressions; STMT1, STMT2 and STMT are arbitrary mailfromd statements (possibly compound); CONDITION is the controlling expression. The vertical dotted row represent another eventual case branches.

This statement is executed as follows: the CONDITION expression is evaluated and if its value equals X1 or X2 (or any other X from the first case), then STMT1 is executed.  Otherwise, if  CONDITION evaluates to Y1 or Y2 (or any other Y from the second case), then STMT2 is executed.  Other case branches are tried in turn.  If none of them matches, STMT (called the default branch) is executed.

There can be as many case branches as you wish.  The default branch is optional.  There can be at most one default branch.

An example of switch statement follows:

     switch %x
     do
     case 1 or 3:
       add "X-Branch" "1"
       accept
     case 2 or 4 or 6:
       add "X-Branch" "2"
     default:
       reject
     done

If the value of mailfromd variable x is 2 or 3, it will accept the message immediately, and add a X-Branch: 1 header to it.  If x equals 2 or 4 or 6, this code will add X-Branch: 2 header to the message and will continue processing it.  Otherwise, it will reject the message.

The controlling condition of a switch statement may evaluate to numeric or string type.  The type of the condition governs the type of comparisons used in case branches: for numeric types, numeric equality will be used, whereas for string types, string equality is used.

Loop Statements
===============

The loop statement allows for repeated execution of a block of code, controlled by some conditional expression.  It has the following form:

     loop [LABEL]
          [for STMT1,] [while EXPR1,] [STMT2]
     do
       STMT3
     done [while EXPR2]

where STMT1, STMT2, and STMT3 are statement lists, EXPR1 and EXPR2 are expressions.

The control flow is as follows:

 1. If STMT1 is specified, execute it.

 2. Evaluate EXPR1.  If it is zero, go to 6.  Otherwise, continue.

 3. Execute STMT3.

 4. If STMT2 is supplied, execute it.

 5. If EXPR2 is given, evaluate it.  If it is zero, go to 6. Otherwise, go to 2.

 6. End.

Thus, STMT3 is executed until either EXPR1 or EXPR2 yield a zero value.

The loop body -- STMT3 -- can contain special statements:

break [LABEL]
     Terminates the loop immediately.  Control passes to 6 (End) in the formal definition above.  If LABEL is supplied, the statement terminates the loop statement marked with that label.  This allows to break from nested loops.

     It is similar to break statement in C or shell.

next [LABEL]
     Initiates next iteration of the loop.  Control passes to 2 in the formal definition above.  If LABEL is supplied, the statement starts next iteration of the loop statement marked with that label.  This allows to request next iteration of an upper-level loop from a nested loop statement.

The loop statement can be used to create iterative statements of arbitrary complexity.  Let's illustrate it in comparison with C.

The statement:

     loop
     do
       STMT-LIST
     done

creates an infinite loop.  The only way to exit from such a loop is to call break (or return, if used within a function), somewhere in STMT-LIST.

The following statement is equivalent to while (EXPR1) STMT-LIST in C:

     loop while EXPR
     do
       STMT-LIST
     done

The C construct for (EXPR1; EXPR2; EXPR3) is written in MFL as follows:

     loop for STMT1, while EXPR2, STMT2
     do
       STMT3
     done

For example, to repeat STMT3 10 times:

     loop for set i 0, while %i < 10, set i %i + 1
     do
       STMT3
     done

Finally, the C do loop is implemented as follows:

     loop
     do
       STMT-LIST
     done while EXPR

As a real-life example of a loop statement, let's consider the implementation of function ptr_validate, which takes a single argument IPSTR, and checks its validity using the following algorithm:

Perform a DNS reverse-mapping for IPSTR, looking up the corresponding PTR record in in-addr.arpa.  For each record returned, look up its IP addresses (A records).  If IPSTR is among the returned IP addresses, return 1 (true), otherwise return 0 (false).

The implementation of this function in MFL is:

     #pragma regex push +extended

     func ptr_validate(string ipstr) returns number
     do
       loop for string names dns_getname(%ipstr) " "
                number i index(%names, " "),
            while %i != -1,
            set names substr(%names, %i + 1)
            set i index(%names, " ")
       do
         loop for string addrs dns_getaddr(substr(%names, 0, %i)) " "
                  number j index(%addrs, " "),
              while %j != -1,
              set addrs substr(%addrs, %j + 1)
              set j index(%addrs, " ")
         do
           if %ipstr == substr(%addrs, 0, %j)
             return 1
           fi
         done
       done
       return 0
     done

Exceptional Conditions
======================

When the running program encounters a condition it is not able to handle, it signals an exception.  To illustrate the concept, let's consider the execution of the following code fragment:

       if primitive_hasmx domainpart $f
         accept
       fi

The function primitive_hasmx (see primitive_hasmx) tests whether the domain name given as its argument has any MX records.  It should return a boolean value.  However, when querying the Domain Name System, it may fail to get a definite result.  For example, the DNS server can be down or temporary unavailable.  In other words, primitive_hasmx can be in a situation when, instead of returning yes or no, it has to return don't know.  That is exactly then when it signals an exception.

Exceptions are identified by exception types, the decimal numbers associated with them.  The header file status.mfh defines symbolic exception names for each of these.  The following table summarizes all the exception types implemented by mailfromd version VERSION:

dbfailure
     General database failure.  For example, the database cannot be opened.  This exception can be signaled by any function that queries any DBM database.

divzero
     Division by zero.

failure
     A general failure has occurred.  In particular, this exception is signaled by DNS lookup functions when any permanent failure occurs. This exception can be signaled by any DNS-related function (hasmx, poll, etc.) or operation (mx matches).

invcidr
     Invalid CIDR notation.  This is signaled by match_cidr function when its second argument is not a valid CIDR.

invip
     Invalid IP address.  This is signaled by match_cidr function when its first argument is not a valid IP address.

invtime
     Invalid time interval specification.  It is signaled by interval function if its argument is not a valid time interval (see time interval specification).

ioerr
     An error occurred during the input-output operation.  See I/O functions, for a description of functions that can signal this exception.

macroundef
     A Sendmail macro is undefined.

noresolve
     The argument of a DNS-related function cannot be resolved to host name or IP address.  Currently only ismx (see ismx) raises this exception.

range
     The supplied argument is outside the allowed range.  This is signalled, for example, by substring function (see substring).

regcomp
     Regular expression cannot be compiled.  This can happen when a regular expression (a right-hand argument of a matches operator) is built at the runtime and the produced string is an invalid regex.

ston_conv
     String-to-number conversion failed.  This can be signaled when a string is used in numeric context which cannot be converted to the numeric data type.  For example:

           set x "10a"
           if %x / 2
             ...

     The if condition will signal ston_conv, since 10a cannot be converted to a number.

temp_failure
     A temporary failure has occurred.  This can be signaled by DNS-related functions or operations.

url
     The supplied URL is invalid. See Interfaces to Third-Party Programs.

In addition to these, two symbols are defined that are not exception types in the strict sense of the world, but are provided to make writing filter scripts more convenient.  These are success, meaning successful return from a function, and not_found, meaning that the required entity (e.g. domain name or email address) was not found.  See figure-poll-wrapper, for an illustration on how these can be used.

Normally when an exception is signalled, the program execution is terminated and the MTA is returned a tempfail status.  Additional information regarding the exception is then output to the logging channel (see Logging and Debugging).  However, the user can intercept any exception by installing his own exception-handling routines.

An exception-handling routine is introduced by catch statement, which has the following syntax:

     catch EXCEPTION-LIST
     do
       HANDLER-BODY
     done

where EXCEPTION-LIST is the list of exception types, separated by the word or.  A special form *, which stands for all exceptions,  is allowed as well.  The HANDLER-BODY is the list of statements comprising the handler body.  For example, the code below installs a handler for exceptions 2 (failure) and 3 (temp_failure):

     catch 2 or 3
     do
       ...
     done

However, using numeric exception codes is not convenient and is not in fact recommended.  We recommend to use exception symbolic names instead.  These are declared in the header file status.mfh, which should be included in your startup script.  Using this approach, the above code snippet will look like:

     #include <status.mfh>

     catch failure or temp_failure
     do
       ...
     done

The catch statement can appear anywhere inside a function or a handler, but it cannot appear outside of them.  It can also be nested within another catch.  Each exception redefined by a catch remains in force until either another catch appears further in the same function or handler that redefines it or an end of the containing block is encountered, whichever occurs first.  Upon exit from a function or milter handler, all exceptions are restored to the state they had when it has been entered.

Upon entry to a catch code, it receives two positional arguments, which can be referenced in HANDLER-BODY as $1 and $2.  The first argument gives the numeric code of the exception that has occurred.  The second argument is a textual string containing a human-readable description of the exception.  A catch defined within a function must return from it by executing return statement.  If it does not do that explicitly, the default value of 1 is returned.  A catch defined within a milter handler must end execution with any of the following actions: accept, continue, discard, reject, tempfail.  By default, continue is used.

The following example shows an envfrom handler that catches eventual exceptions, signaled by primitive_hasmx.  The handler accepts the message if the sender domain has any MX records, rejects them if it does not, and continues execution if an exception occurs. In the latter case, a log message is also issued:

     #include <status.mfh>

     prog envfrom
     do
       catch failure or temp_failure
       do
         echo "Caught exception $1: $2"
         continue
       done

       if primitive_hasmx domainpart $f
         accept
       else
         reject
       fi
     done

The following example defines the function hasmx that returns true if the domain part of its argument has any MX records, and false if it does not or if an exception occurs  (This function is part of the mailfromd library, see hasmx.).

     func hasmx (s) returns n
     do
       catch *
       do
         return 0
       done
       return primitive_hasmx domainpart $1
     done

All variables remain visible within catch body, with the exception of positional arguments of the enclosing function or handler.  So, if you wish to access function arguments from a catch, define your function using new style (see User-defined) declaration.  To access positional arguments of a handler, you will have to assign them to local variables prior to installing a catch, e.g.:

     prog header
     do
       string hname $1
       string hvalue $2
       catch *
       do
         echo "Exception $1 while processing header %hname: %hvalue"
         echo $2
         tempfail
       done
       ...

You can also generate (or raise) exceptions explicitly in the filter code, using throw statement:

     throw EXCODE DESCR

The arguments correspond exactly to the positional arguments of the catch statement: EXCODE gives the numeric code of the exception, DESCR gives its textual description.  This statement can be used in complex scripts to create non-local exits from deeply nested statements. Elaborate on that.

Notice several limitations of throw: first, the EXCODE argument must be an immediate value, you cannot use an expression in its place.  Secondly, its range is limited to exception codes, described in status.mfh.  In other words, you cannot (yet) invent your own exception types.  This should be fixed in future releases.

Sender Verification Tests
=========================

The filter script language provides a wide variety of functions for sender address verification or polling, for short.  These functions, which were described in Polling functions, can be used to implement any sender verification method.  The additional data that can be needed is normally supplied by two global variables: ehlo_domain, keeping the default domain for the EHLO command, and mailfrom_address, which stores the sender address for probe messages (see Predefined variables).

For example, a simplest way to implement standard polling would be:

     prog envfrom
     do
       if stdpoll($1, %ehlo_domain, %mailfrom_address) == 0
         accept
       else
         reject 550 5.1.0 "Sender validity not confirmed"
       fi
     done

However, this does not take into account exceptions that stdpoll can signal.  To handle them, one will have to use catch, for example thus:

     #include <status.mfh>

     prog envfrom
     do
       catch failure or temp_failure
       do
         switch $1
         do
         case failure:
           reject 550 5.1.0 "Sender validity not confirmed"
         case temp_failure:
           tempfail 450 4.1.0 "Try again later"
         done
       done

       if stdpoll($1, %ehlo_domain, %mailfrom_address) == 0
         accept
       else
         reject 550 5.1.0 "Sender validity not confirmed"
       fi
     done

If polls are used often, one can define a wrapper function, and use it instead.  The following example illustrates this approach:

Figure: Building Poll Wrappers

     func poll_wrapper(string email) returns number
     do
       catch failure or temp_failure
       do
         return %email
       done
       return stdpoll(%email, %ehlo_domain, %mailfrom_address)
     done

     prog envfrom
     do
       switch poll_wrapper($f)
       do
       case success:
         accept
       case not_found or failure:
         reject 550 5.1.0 "Sender validity not confirmed"
       case temp_failure:
         tempfail 450 4.1.0 "Try again later"
       done
     done

Notice the way envfrom handles success and not_found, which are not exceptions in the strict sense of the word.

The above paradigm is so common that mailfromd provides a special language construct to simplify it: the on statement. Instead of manually writing the wrapper function and using it as a switch condition, you can rewrite the above example as:

Figure: Standard poll example

     prog envfrom
     do
       on stdpoll($1, %ehlo_domain, %mailfrom_address)
       do
       when success:
         accept
       when not_found or failure:
         reject 550 5.1.0 "Sender validity not confirmed"
       when temp_failure:
         tempfail 450 4.1.0 "Try again later"
       done
     done

As you see the statement is pretty similar to switch.  The major syntactic differences are:

 1. Use of the keyword when to introduce conditional branches;

 2. The ampersand before exception names is optional;

General syntax of the on statement is:

     on CONDITION
     do
       when X1 [or X2 ...]:
         STMT1
       when Y1 [or Y2 ...]:
         STMT2
         .
         .
         .
     done

The CONDITION is either a function call or a special poll statement (see below).  The values used in when branches are normally symbolic exception names (see exception symbolic names).

When the compiler processes the on statement it does the following:

 1. Builds a unique wrapper function, similar to that described in figure-poll-wrapper;  The name of the function is constructed from the CONDITION function name and an unsigned number, called exception mask, that is unique for each combination of exceptions used in when branches;  To avoid name clashes with the user-defined functions, the wrapper name begins and ends with $ which normally is not allowed in the identifiers;

 2. Translates the on body to the corresponding switch statement;

The special form of the CONDITION is poll keyword, whose syntax is:

     poll [for] EMAIL
          [host HOST]
          [from DOMAIN]
          [as EMAIL]

The order of particular keywords in the poll statement is arbitrary, for example as EMAIL can appear before EMAIL as well as after it.

The simplest form, poll EMAIL, performs the standard sender verification of email address EMAIL.  It is translated to the following function call:

       stdpoll(EMAIL, %ehlo_domain, %mailfrom_address)

The construct poll EMAIL host HOST, runs the strict sender verification of address EMAIL on the given host. It is translated to the following call:

       strictpoll(HOST, EMAIL, %ehlo_domain, %mailfrom_address)

Other keywords of the poll statement modify these two basic forms.  The as keyword introduces the email address to be used in the SMTP MAIL FROM command, instead of %mailfrom_address.  The from keyword sets the domain name to be used in EHLO command.  So, for example the following construct:

       poll EMAIL host HOST from DOMAIN as ADDR

is translated to

       strictpoll(HOST, EMAIL, DOMAIN, ADDR)

To summarize the above, the code described in figure-stdpoll can be written as:

     prog envfrom
     do
       on poll $f do
       when success:
         accept
       when not_found or failure:
         reject 550 5.1.0 "Sender validity not confirmed"
       when temp_failure:
         tempfail 450 4.1.0 "Try again later"
       done
     done

Modules
=======

A module is an MFL source file containing a collection of conceptually united functions and data.  Currently, there are no special syntax rules that make an arbitrary source file a module, it suffices to make sure the module name ends in standard suffix .mf and place it in one of the directories comprising the include source path (see include search path).  However, such syntax rules will appear in future versions.  It is planned that modules will be compiled and stored on disk in form of object files or combined into libraries.

In order to make definitions from module MODNAME.mf available to your filter program, use the following statement:

     #require MODNAME

The #require statement looks up the module MODNAME.mf in the include path and attempts to compile it (see require).  It is not an error to require the same module several times.  However, requiring a non-existing module results in compilation error.

MFL Preprocessor
================

Before compiling the script file, mailfromd preprocesses it.  The built-in preprocessor handles only file inclusion (see include), while the rest of traditional facililities, such as macro expansion, are supported via m4, which is used as an external preprocessor.

The detailed description of m4 facilities lies far beyond the scope of this document.  You will find a complete user manual in GNU M4 (GNU M4 macro processor).  For the rest of this section we assume the reader is sufficiently acquainted with m4 macro processor.

The external preprocessor is invoked with -s flag, instructing it to include line synchornizattion information in its output, which is subsequently used by MFL compiler for purposes of error reporting.  The initial set of macro definitions is supplied in file pp-setup, located in the library search path (It is
usually located in
/usr/local/share/mailfromd/VERSION/include/pp-setup.), which is fed to the preprocessor input before the script file itself. The default pp-setup file renames all m4 builtin macro names so they all start with the prefix m4_ (This is similar to GNU m4 --prefix-builtin options.  This approach was chosen to allow for using non-GNU m4 implementations as well.).  It changes comment characters to /*, */ pair, and leaves the default quoting characters, grave (`) and acute (') accents without change.  Finally, pp-setup defines the following macros:

 -- M4 Macro: boolean defined (IDENTIFIER)
     The IDENTIFIER must be the name of an optional abstract argument to the function.  This macro must be used only within a function definition.  It expands to the MFL expression that yields true if the actual parameter is supplied for IDENTIFIER. For example:

          func rcut(string text; number num)
          do
            if (defined(num))
              return substr(%text, length(%text) - %num)
            else
              return %text
            fi
          done

     This function will return last NUM characters of TEXT if NUM is supplied, and entire TEXT otherwise, e.g.:

          rcut("text string") => "text string"
          rcut("text string", 3) => "ing"

     Invoking the defined macro with the name of a mandatory argument yields true

 -- M4 Macro: printf (FORMAT, ...)
     Provides a printf statement, that formats its optional parameters in accordance with FORMAT and sends the resulting string to the current log output (see Logging and Debugging). See String formatting, for a description of FORMAT.

     Example usage:

          printf('Function %s returned %d', %funcname, %retcode)

 -- M4 Macro: string _ (MSGID)
     A convenience macro.  Expands to a call to gettext (see NLS Functions).

 -- M4 Macro: N_ (MSGID)
     A convenience macro, that expands to MSGID verbatim.  It is intended to mark the literal strings that should appear in the .po file, where actual call to gettext (see NLS Functions) cannot be used.  For example:

          /* Mark the variable for translation: cannot use gettext here */
          string message N_("Mail accepted")

          prog envfrom
          do
            ...
            /* Translate and log the message */
            echo gettext(%message)

You can obtain the preprocessed output, without starting actual compilation, using -E command line option:

     $ mailfromf -E file.mf

The output is in the form of preprocessed source code, which is sent to the standard output.  This can be useful, among others, to debug your own macro definitions.

The following two options are supplied mainly for debugging purposes:

--no-preprocessor
     Disables the external preprocessor.

--preprocessor=COMMAND
     Use COMMAND as external preprocessor.  Be especially careful with this option, because mailfromd cannot verify whether COMMAND is actually some kind of a preprocessor or not.

Example of a Filter Script File
===============================

This section will show a working example of the filter script file. For the ease of illustration, it is divided in several sections. Each section is prefaced with a comment explaining its function.

The first part defines the configuration settings for this host:

     #pragma option relay /etc/mail/sendmail.cw
     #pragma option relay /etc/mail/relay-domains
     #pragma option timeout 33
     #pragma regex +extended +icase

     #pragma database cache negative-expire-interval 1 day
     #pragma database cache positive-expire-interval 2 weeks

     set mailfrom_address "<>"
     set ehlo_domain "gnu.org.ua"

The second part includes the necessary header files and loads required source modules:

     #include <status.mfh>
     #require dns

Next we define envfrom handler.  In the first two rules, it accepts all mails coming from the null address and from the machines which we relay:

     prog envfrom
     do
       if $f = ""
         accept
       elif relayed hostname $client_addr
         accept
       elif hostname $client_addr = $client_addr
         reject 550 5.7.7 "IP address does not resolve"

Next rule rejects all messages coming from hosts with dynamic IP addresses.  A regular expression used to catch such hosts is not 100% fail-proof, but it tries to cover most existing host naming patterns:

        elif hostname $client_addr matches
              ".*(adsl|sdsl|hdsl|ldsl|xdsl|dialin|dialup|\
     ppp|dhcp|dynamic|[-.]cpe[-.]).*"
          reject 550 5.7.1 "Use your SMTP relay"

Messages coming from the machines whose host names contain something similar to an IP are subject to strict checking:

        elif hostname $client_addr matches
        ".*[0-9]{1,3}[-.][0-9]{1,3}[-.][0-9]{1,3}[-.][0-9]{1,3}.*"
          on poll host $client_addr for $f do
          when success:
            pass
          when not_found or failure:
            reject 550 5.1.0 "Sender validity not confirmed"
          when temp_failure:
            tempfail
          done

If the sender domain is relayed by any of the yahoo.com or nameserver.com MXs, no checks are performed.  We will greylist this message in envrcpt handler:

        elif $f mx fnmatches "*.yahoo.com"
             or $f mx fnmatches "*.namaeserver.com"
          pass

Finally, if the message does not meet any of the above conditions, it is verified by the standard procedure:

        else
          on poll $f do
          when success:
            pass
          when not_found or failure:
            reject 550 5.1.0 "Sender validity not confirmed"
          when temp_failure:
            tempfail
          done
        fi

At the end of the handler we check if the sender-client pair does not exceed allowed mail sending rate:

        if rate($f "-" $client_addr, interval("1 hour 30 minutes")) > 100
          tempfail 450 4.7.0 "Mail sending rate exceeded. Try again later"
        fi
     done

Next part defines the envrcpt handler.  Its primary purpose is to greylist messages from some domains that could not be checked otherwise:

     prog envrcpt
     do
       set gltime 300
       if $f mx fnmatches "*.yahoo.com"
          or $f mx fnmatches "*.namaeserver.com"
          and not dbmap("/var/run/whitelist.db", $client_addr)
         if greylist($client_addr "-" $f "-" $rcpt_addr, %gltime)
           if %greylist_seconds_left = %gltime
             tempfail 450 4.7.0
                    "You are greylisted for " %gltime " seconds"
           else
             tempfail 450 4.7.0
                    "Still greylisted for "
                    %greylist_seconds_left " seconds"
           fi
         fi
       fi
     done

Reserved Words
==============

For your reference, here is an alphabetical list of all reserved words:

  * __file__

  * __function__

  * __line__

  * __major__

  * __minor__

  * __package__

  * __patch__

  * __preproc__

  * __version__

  * accept

  * add

  * and

  * begin

  * break

  * case

  * catch

  * const

  * continue

  * default

  * delete

  * discard

  * do

  * done

  * echo

  * end

  * elif

  * else

  * fi

  * fnmatches

  * for

  * func

  * if

  * loop

  * matches

  * next

  * not

  * number

  * on

  * or

  * pass

  * prog

  * reject

  * replace

  * return

  * returns

  * set

  * string

  * switch

  * tempfail

  * throw

  * when

  * while

Several keywords are context-dependent: mx is a keyword if it appears before matches or fnmatches.  Following strings are keywords in on context:

  * as

  * from

  * host

  * poll

Using the GNU Emacs MFL Mode
****************************

MFL sources are usual ASCII files and you may edit them with any editor you like.  However, the best choice for this job (as well as for many others) is, without doubt, GNU Emacs.  To ease the work of editing script files, the mailfromd package provides a special Emacs mode, called MFL mode.

The elisp source file providing this mode, mfl-mode.el, is installed automatically, provided that GNU Emacs is present on your machine.  To enable the mode, add the following lines to your Emacs setup file (either system-wide site-start.el, or your personal one, ~/.emacs):

     (autoload 'mfl-mode "mfl-mode")
     (setq auto-mode-alist (append auto-mode-alist
                            '(("/etc/mailfromd.rc" . mfl-mode)
                              ("\\.mf$" . mfl-mode))))

The first directive loads the MFL mode, and the second one tells Emacs to apply it to any file whose name ends in /etc/mailfromd.rc (This will match most existing installations.  In the unlikely case that your $sysconfdir does not end in /etc, you will have to edit the directive accordingly.) or in a .mf suffix.

MFL mode provides automatic indentation and syntax highlighting for MFL sources.  The default indentation setup is the same as the one used throughout this book:

  * Handler and function definitions start at column 1;

  * A block statement, i.e. do, done, if, else, elif and fi, occupies a line by itself, with the only exception that do after an on statement is located on the same line with it;

  * A do statement that follows function or handler definition is placed in column 1.

  * Each subsequent level of nesting is indented two columns to the right (see mfl-basic-offset).

  * A closing statement (done, else, elif, fi) is placed at the same column as the corresponding opening statement;

  * Branch statements (case and when) are placed in the same column as their controlling keyword (switch and on, correspondingly (see mfl-case-line-offset).

The mode provides two special commands that help navigate through the complex filter scripts:

C-M-a
     Move to the beginning of current function or handler definition.

C-M-e
     Move to the end of current function or handler definition.

Here, current function or handler means the one within which your cursor currently stays.

You can use C-M-e repeatedly to walk through all function and handler definitions in your script files.  Similarly, repeatedly pressing C-M-a will visit all the definitions in the opposite direction (from the last up to the very first one).

Another special command, C-c C-c, allows to verify the syntax of your script file.  This command runs mailfromd in syntax check mode (see Testing Filter Scripts) and displays its output in a secondary window, which allows to navigate through eventual diagnostic messages and to jump to source locations described by them.

All MFL mode settings are customizable.  To change any of them, press M-x customize and visit Environment/Unix/Mfl customization group.  This group offers two subgroups: Mfl Lint group and Mfl Indentation group.

Mfl Lint group controls invocation of mailfromd by C-c C-c.  This group contains two variables:

 -- MFL-mode setting: mfl-mailfromd-command
     The mailfromd to be invoked.  By default, it is mailfromd.  You will have to change it, if mailfromd cannot be found using PATH environment variable, or if you wish to pass it some special options.  However, do not include --lint or -I options in this variable.  The --lint option is given automatically, and include paths are controlled by mfl-include-path variable (see below).

 -- MFL-mode setting: mfl-include-path
     A list of directories to be appended to mailfromd include search path (see include search path).  By default it is empty.

Mfl Indentation group controls automatic indentation of MFL scripts.  This group contains the following settings:

 -- MFL-mode setting: mfl-basic-offset
     This variable sets the basic indentation increment.  It is set to 2, by default, which corresponds to the following indentation style:

          prog envfrom
          do
            if $f = ""
              accept
            else
              ...
            fi
          done

 -- MFL-mode setting: mfl-case-line-offset
     Indentation offset for case and when lines, relative to the column of their controlling keyword.  The default is 0, i.e.:

          switch %x
          do
          case 0:
            ...
          default:
            ...
          done

Configuring mailfromd
*********************

In the simplest case you will be able to startup mailfromd without any additional command line options: the default values provided when compiling the package and the default script file are enough.

If you need to alter any configuration settings, there are three ways of doing so.  First of all, and this method should always be preferred, you can use various #pragma directives in the filter script.  These were described in Comments.

We advise to use this method because with it all the program settings are kept in a single file, which simplifies the maintenance and ensures they remain valid across program restarts.

If you wish to temporarily override the configuration settings, use the command line.  It is the second configuration method.

Finally, there are several configuration options that are handled by libmailutils itself, and that cannot currently be configured in mailfromd.rc.  We recommend to store these settings in the mailutils system-wide configuration file (The use of mailutils configuration file is covered in see Mailutils Configuration File (GNU Mailutils Manual).). This approach is described in Local Account Verification.

In this chapter we will discuss mailfromd command line options and the ways of starting and shutting down the program.

Mailfromd command line syntax
=============================

The mailfromd binary is started from the command line using the following syntax (brackets indicate optional parts):

     $ mailfromd [OPTIONS] [ASGN] [SCRIPT]

The meaning of each invocation part is described in the table below:

options
     The command line options (see options).

asgn
     Sendmail macro assignments.  These are currently meaningful only with the --test option (see test mode), but this may change in the future.  Each assignment has the form:

          VAR=VALUE

     where VAR is the name of a Sendmail macro and VALUE is the value to be assigned to it.

script
     The file name of the filter script, if other than the default one.

Command Line Options.
=====================

Operation Modifiers
-------------------

     compact

--compact
     Compact database (see compaction).  By default, cache database is compacted.  To specify another database, use --format option (see --format).  Full database name can be given in the command line (see --file option below), if it differs from the one specified in the script file.

     Use with the option --all (see --all) to compact all databases.

     This option implies --stderr (see --stderr).

     daemon

--daemon
     Run in daemon mode (default).

     delete

--delete
     Delete given entries from the database (see deleting from databases).  By default, cache database is assumed.  To specify another database, use --format option (see --format).

     This option implies --stderr (see --stderr).

     expire

--expire
     Delete all expired entries from the database (see Database Maintenance).  By default, cache database is assumed.  To specify another database, use --format option (see --format).  Full database name can be given in the command line (see --file option below), if it differs from the one specified in the script file.

     Use with the option --all (see --all) to expire all databases.

     This option implies --stderr (see --stderr).

     file

-f FILENAME
--file=FILENAME
     Set the name of the database to operate upon (for --compact, --delete, --expire, and --list options). Useful if, for some reason, you need to operate on a database whose file name does not match the one mailfromd is configured to use.

     list

--list
     List the database.  By default, cache database is assumed.  To list another database, use --format option (see --format).

     This option implies --stderr (see --stderr).

     See Basic Database Operations.

     show-defaults

--show-defaults
     Show compilation defaults. See Databases.

     test

-t[STATE]
--test[=STATE]
     Run in test mode.  See Testing Filter Scripts.  Default STATE is envfrom.  This option implies --stderr (see --stderr).

General Settings
----------------

     all

--all
     When used with --compact or --expire option, applies the action to all available databases. See compact cronjob.

     domain

-D STRING
--domain=STRING
     Set default SMTP domain.  Overrides #pragma option ehlo (see pragma ehlo).  This option is deprecated (see 31x-400, for a detailed description of why it is deprecated.): use -v ehlo_domain=STRING instead.

     ehlo
--ehlo=STRING
     Same as --domain.

     expire-interval

-e INTERVAL
--expire-interval=INTERVAL
     Set expiration intervals for all databases to the specified interval. See time interval specification, for a description of INTERVAL format.  The option overrides #pragma option expire-interval (see pragma expire-interval), which you are advised to use instead.

     foreground

--foreground
     Stay in foreground.  When given this option, mailfromd will not disconnect itself from the controlling terminal and will run in the foreground.

     format

-H DBFORMAT
--format=DBFORMAT
     Use database of the given format, instead of the default cache.  See Basic Database Operations.

     group

-g NAME
--group=NAME

     Retain the group NAME when switching to user privileges.  See Starting and Stopping.

     This option overrides #pragma option group (see pragma group).

     ignore-failed-reads

--ignore-failed-reads
     Ignore records that cannot be retrieved while compacting the database.  Without this option, mailfromd will abort the compaction if any such error is encountered.

     include

--include=DIR
-I DIR
     Add the directory DIR to the list of directories to be searched for header files.  This will affect the functioning of #include statement.  See include, for a discussion of file inclusion.

     lock-retry-count

--lock-retry-count=NUMBER
     Set maximum number of attempts to acquire the lock on a database. See the description of #pragma option lock-retry-count, pragma lock-retry-count, for more information about the database locking.

     lock-retry-timeout

--lock-retry-timeout=INTERVAL
     Set the time span between the two locking attempts.  See the description of #pragma option lock-retry-count, pragma lock-retry-count, for more information about database locking.

     This option overrides the value set by #pragma option lock-retry-timeout (see pragma lock-retry-timeout.

     See time interval specification, for a description of valid INTERVAL formats.

     mailer

--mailer=URL
-M URL
     Set the URL of the mailer to use.  See Mail Sending Functions.

     mailfrom

--mailfrom=EMAIL
     Set postmaster email address.  Overrides #pragma option ehlo, which you are advised to use instead (see pragma ehlo).  The default is null return address ("").  This option is deprecated (see 31x-400, for a detailed description of why it is deprecated.): use -v mailfrom_address=STRING instead.

     mtasim

--mtasim
     This option is reserved for use by mtasim (see mtasim).

     no-preprocessor

--no-preprocessor
     Do not run the preprocessor.  See Preprocessor.

     optimize

-O[LEVEL]
--optimize[=LEVEL]
     Set optimization level for code generator.  Two levels are implemented: 0, meaning no optimization, and 1, meaning full optimization.

     port

-p STRING
--port=STRING
     Set communication socket.  Overrides #pragma option port, which you are advised to use instead (see pragma port).

     postmaster-email

--postmaster-email=EMAIL
     See --mailfrom.

     pidfile

--pidfile=FILE
     Set pidfile name.  Overrides #pragma option pidfile, which you are advised to use instead (see pragma pidfile).

     predict

--predict=RATE-LIMIT
     Used with --list enables printing of the estimated times of sending along with the rate database dump.  Implies --list --format=rate.  See estimated time of sending.

     preprocessor

--preprocessor=COMMAND
     Use COMMAND as the external preprocessor instead of the default m4.  See Preprocessor.

     relayed-domain-file

--relayed-domain-file=FILE
     Read relayed domains from FILE.  Overrides #pragma option relay, which you are advised to use instead (see option relay). See Avoiding Verification Loops, and the description of relayed function (see relayed) for more information.

     remove

-r
--remove
     Force removing local socket file, if it already exists.  Unless this option is specified, mailfromd will refuse to start if this file exists.

     state-directory

--state-directory=DIR
     Set new program state directory.  See Local state directory, for the description of this directory and its purposes.  This option overrides the settings of #pragma option state-directory, described in pragma state-directory.

     source

-S
--source
     Set source address for TCP connections.  Overrides #pragma option source, which you are advised to use instead (see pragma source).

     time-format

--time-format=FORMAT
     Set format to be used for timestamps in listings, produced by --list.  The FORMAT is any valid strftime format string, see Time and Date Formats, for a detailed description. The default FORMAT is %c (see %c time format).  To analyze mailfromd --list output using text tools, such as awk or grep, the following format might be useful: %s (see %s time format).  Another format I find useful is %Y-%m-%d_%H:%M:%S.

     user

-u NAME
--user NAME
     Switch to this user privileges after startup. Overrides #pragma
     option user, which you are advised to use instead (see pragma user).  Default user is DEFAULT-USER.

     variable

-v VAR=VALUE
--variable VAR=VALUE
     Assign VALUE to the global variable VAR.  The variable must be declared in your startup script.  See overriding initial values, for a detailed discussion of this option.

Timeout Control
---------------

See time interval specification, for information on INTERVAL format.

     milter-timeout

--milter-timeout=INTERVAL
     Set MTA connection timeout.  Overrides #pragma option milter-timeout, which you are advised to use instead (see pragma milter-timeout.

     timeout

--timeout=NUMBER
     Set I/O operation timeout (seconds).  Overrides #pragma option timeout, which you are advised to use instead (see pragma io-timeout).

Logging and Debugging Options
-----------------------------

     debug

-d STRING
--debug=STRING
     Set debugging level.  Overrides #pragma option debug.  See Logging and Debugging.

     dump-code

--dump-code
     Parse and compile the script file and dump the disassembled listing of the produced code to the terminal.  See Logging and Debugging.

     dump-grammar-trace

--dump-grammar-trace
     Enable debugging the script file parser.  While parsing the file, the detailed dump of the parser states and tokens seen will be output.

     dump-lex-trace

--dump-lex-trace
     Enable debugging the lexical analyzer.  While parsing the script file, the detailed dump of the lexer states and matched rules will be output.

     dump-macros

--dump-macros
     Show Sendmail macros used in the script file.  The macro names are displayed as comma-separated lists, grouped by handler names. See Sendmail Configuration, for a detailed description of this option and its usage.

     dump-tree

--dump-tree
     Parse and compile the script file and dump the parse tree in a printable form to the terminal.

     dump-xref

--dump-xref
     Print a cross-reference of variables used in the filter script. See Testing Filter Scripts.

-E
     Stop after the preprocessing stage; do not run the compiler proper. The output is in the form of preprocessed source code, which is sent to the standard output.  See Preprocessor.

     lint

--lint
     Check script file syntax and exit.  If the file is OK, return 0 to the shell, otherwise print appropriate messages to stderr and exit with code 78 (configuration error).

     single-process

--single-process
     Do not fork sub-processes to serve requests.  This option is meant to assist in debugging mailfromd.  Don't use it for anything else but for debugging, as it terribly degrades performance!

     stack-trace

--stack-trace
     Add MFL stack trace information to runtime error output. Overrides #pragma option stack-trace, which you are advised to use instead (see pragma stack-trace).  See tracing runtime errors, for more information on this feature.

     gacopyz-log

--gacopyz-log=LEVEL
     Set desired logging level for gacopyz library (see Gacopyz).  There are five logging levels.  The following table lists them in order of decreasing priority:

     fatal
          Log fatal errors.

     err
          Log error messages.

     warn
          Log warning messages.

     info
          Log informational messages.  In particular, this enables printing messages on each subprocess startup and termination, which look like that:

               Apr 28 09:00:11 host mailfromd[9411]: connect
                   from 192.168.10.1:50398
               Apr 28 09:00:11 host mailfromd[9411]: finishing
                   connection

          This level can be useful for debugging your scripts.

     debug
          Log debugging information.  This level prints huge amounts of information, in particular it displays dumps of each Milter packet sent and received.

     Although it is possible to set these levels independently of each other, it is seldom practical.  Therefore, the option --gacopyz-log=LEVEL enables all logging levels from LEVEL up.  For example, --gacopyz-log=warn enables log levels warn, err and fatal.  It is the default.  If you need to trace each subprocess startup and shutdown, set --gacopyz-log=info.  Setting the logging level to debug can be needed only for Gacopyz developers, to debug the protocol.

     See Testing Filter Scripts.

     log-facility

--log-facility=FACILITY
     Output logs to syslog FACILITY.

     log-tag

--log-tag=STRING
     Tag syslog entries with the given STRING, instead of the program name.

     no-syslog-async

--no-syslog-async
     Use system libc syslog implementation. See Logging and Debugging, for more information about two syslog flavors, and see Using non-blocking syslog, for information on how to set default syslog implementation at compile time.

     To inspect the default syslog implementation, use the --show-defaults command line option (see show-defaults).

     stderr

-s
--stderr
     Log to stderr (by default logging goes to syslog). See Logging and Debugging.

     This option is turned on implicitly by any of the following options: --compact, --delete, --expire, --test, and --list.

     source-info

--source-info
     Include C source information in debugging messages.  You do not need this option, unless you are developing and debugging mailfromd.

     syntax-check

--syntax-check
     Synonym for --lint.

     trace

--trace
     Enable action tracing.  With this option mailfromd will log all executed actions.  See Logging and Debugging.

     trace-program

--trace-program[=STRING]
     Enable program instruction tracing.  With this option mailfromd will log execution of every instruction in the compiled filter program.  The optional arguments allows to specify a comma-separated list of source code modules for which the tracing is to be enabled, for example --trace-program=bi_io,bi_db enables tracing for functions from modules bi_io.c and bi_db.c (notice, that you need not give file suffixes in STRING).

     This option is useful for debugging mailfromd, but is not advised to use otherwise, since it is very time-costly.

     transcript

-X
--transcript
     Enable transcript of the SMTP sessions to the log channel.  See Logging and Debugging.

     syslog

--syslog
     Output logs to syslog.  This is the default, unless any of the following options is used: --compact, --delete, --expire, --test, and --list.

     syslog-async

--syslog-async
     Use asynchronous syslog implementation. See Logging and Debugging, for more information about two syslog flavors, and see Using non-blocking syslog, for information on how to set default syslog implementation at compile time.

     To inspect the default syslog implementation, use the --show-defaults command line option (see show-defaults).

     xref

--xref
     Same as --dump-xref.  See Logging and Debugging.

Informational Options
---------------------

-?
--help
     Give a short help summary.

--usage
     Give a short usage message.

-V
--version
     Print program version.

Starting and Stopping
=====================

Right after startup, when mailfromd has done the operations that require root privileges, it switches to the privileges of the user it is configured to run as (see default user privileges) or the one given in its script file (see pragma user).  During this process it will drop all supplementary groups and switch to the principal group of that user.

Such limited privileges of the daemon can cause difficulties if your filter script needs to access some files (e.g. Sendmail databases) that are not accessible to that user and group.  For example, the following fragment using dbmap function:

     if dbmap("/etc/mail/aliases.db", $f, 1)
       ...
     fi

will normally fail, because /etc/mail/aliases.db is readable only to the root and members of the group smmsp.

In such situations you need to instruct mailfromd to retain the privileges of one or several supplementary groups when switching to the user privileges.  This is done using #pragma option group directive in the script file (see pragma group).  In example above, you need to use the following directive:

     #pragma option group smmsp

(The same effect can be achieved with --group command line option: mailfromd --group=smmsp).

To stop a running instance of mailfromd use one of the following signals: SIGQUIT, SIGTERM, SIGINT. All three signals have the same effect: the program cancels handling any pending requests, uninitializes the communication socket (if it is a UNIX socket, the program unlinks it) and exits.

To restart the running mailfromd instance, send it SIGHUP.  For restart to be possible, two conditions must be met: mailfromd must be invoked with the full file name, and the configuration file name must be full as well.  If either of them is not met, mailfromd displays a similar warning message:

     warning: script file is given without full file name
     warning: restart (SIGHUP) will not work

or:

     warning: mailfromd started without full file name
     warning: restart (SIGHUP) will not work

The reaction of mailfromd on SIGHUP in this case is the same as on the three signals described previously, i.e. cleanup and exit immediately.

The PID of the master instance of mailfromd is kept on the pidfile, which is named mailfromd.pid and is located in the program state directory.  Assuming the default location of the latter, the following command will stop the running instance of the daemon:

     kill -TERM `head -n1 /usr/local/var/mailfromd/mailfromd.pid`

The default pidfile location is shown in the output of mailfromd --show-defaults (see Databases), and can be changed at run time using #pragma option pidfile statement (see pragma pidfile).

To facilitate the use of mailfromd, it is shipped with a shell script that can be used to launch it on system startup and shut it down when the system goes down.  The script, called rc.mailfromd, is located in the directory /etc of the distribution.  It takes a single argument, specifying the action that should be taken:

start
     Start the program.

stop
     Shut down the program

reload
     Reload the program, by sending it SIGHUP signal.

restart
     Shut down the program and start it again.

status
     Display program status.  It displays the PID of the master process and its command line, for example:

          $ /etc/rc.d/rc.mailfromd status
          mailformd appears to be running at 26030
          26030 /usr/local/sbin/mailfromd --remove --group smmsp

     If the second line is not displayed, this most probably mean that there is a stale pidfile, i.e. the one left though the program is not running.

     An empty rc.mailfromd status output means that mailfromd is not running.

configtest [FILE]
     Check the script file syntax, report any errors found and exit.  If FILE is given it is checked instead of the default one.

macros [-c] [FILE]
     Parse the script file (or FILE, if it is given, extract the names of Sendmail macros it uses and generate corresponding export statements usable in the Sendmail configuration file.  By default, mc statements are generated.  If -c (--cf) is given, the statements for sendmail.cf are output.  See the next chapter for the detailed description of this mode.

You can pass any additional arguments to mailfromd by editing ARGS variable near line 22.

The script is not installed by default.  You will have to copy it to the directory where your system start-up scripts reside and ensure it is called during the system startup and shut down.  The exact instructions on how to do so depend on the operating system you use and are beyond the scope of this manual.

Configuring Sendmail to use mailfromd
*************************************

This chapter assumes you are familiar with Sendmail configuration in general and with Milter configuration directives in particular.  It concentrates only on issues, specific for mailfromd.

To prepare Sendmail to communicate with mailfromd you need first to set up the milter port. This is done with INPUT_MAIL_FILTER statement in your Sendmail file.  Use the same value you have used in #pragma option port statement of mailfromd.rc file (see pragma port):

     INPUT_MAIL_FILTER(`mailfrom', `S=unix:/usr/local/var/mailfromd/mailfrom')

If you prefer to fiddle directly with sendmail.cf file, use this statement instead:

     Xmailfrom, S=unix:/usr/local/var/mailfromd/mailfrom

Then, you need to make sure all Sendmail macros used in your mailfromd.rc file are properly exported.  The simplest way to do so is using rc.mailfromd script, introduced in the previous chapter.  Run it with macros command line argument and copy its output to your sendmail.mc configuration file:

     $ rc.mailfromd macros

If you prefer to work with sendmail.cf directly, use -c (--cf) command line option:

     $ rc.mailfromd macros -c

Finally, if you use other mailfromd script file than that already installed (for example, you are preparing a new configuration while the old one is still being used in production environment), give its name in the command line:

     $ rc.mailfromd macros newconfig.rc
     # or:
     $ rc.mailfromd macros -c newconfig.rc

If you use this method, you can skip the rest of this chapter. However, if you are a daring sort of person and prefer to do everything manually, follow the instructions below.

First of all you need to build a list of macros used by handlers in your mailfromd.rc file.  You can obtain it running mailfromd --dump-macros.  This will display all macros used in your handlers, grouped by handler name, for example:

     envfrom i, f, {client_addr}
     envrcpt f, {client_addr}, {rcpt_addr}

Now, modify confMILTER_MACROS_HANDLER macros in your mc file.  Here, HANDLER means the uppercase name of the mailfromd handler you want to export macros to, i.e. the first word on each line of the above mailfromd --dump-macros output.  Notice, that in addition to these macros, you should also export the macro i for the very first handler (rc.mailfromd macros takes care of it automatically, but you preferred to do everything yourself...)  It is necessary in order for mailfromd to include Message-ID in its log messages (see Message-ID).

For example, given the above macros listing, which corresponds to our sample configuration (see Filter Script Example), the sendmail.mc snippet will contain:

     define(`confMILTER_MACROS_ENVFROM',dnl
     confMILTER_MACROS_ENVFROM `, i, f, {client_addr}')
     define(`confMILTER_MACROS_ENVRCPT',dnl
     confMILTER_MACROS_ENVRCPT `, f, {client_addr}, {rcpt_addr}')

Special attention should be paid to s macro (HELO domain name).  In Sendmail versions up to 8.13.7 (at least) it is available only to helo handler.  If you wish to make it available elsewhere you will need to use the method described in HELO Domain

Now, if you are a really daring person and prefer to do everything manually and to hack your sendmail.cf file directly, you certainly don't need any advices.  Nonetheless, here's how the two statements above could look in this case:

     O Milter.macros.envfrom=i, {auth_type}, {auth_authen}, \
        {auth_ssf}, {auth_author}, {mail_mailer}, {mail_host}, \
        {mail_addr} ,{mail_addr}, {client_addr}, f
     O Milter.macros.envrcpt={rcpt_mailer}, {rcpt_host}, {rcpt_addr} ,i,
     f, {client_addr}

mtasim --- a testing tool
*************************

     [file mtasim.texi not included]

How to Report a Bug
*******************

     Documentation is like sex: when it is good, it is very, very
     good; and when it is bad, it is better than nothing.

     Dick Brandon

Although the author has tried to make this documentation as detailed as is possible and practical, he is well aware that the result is rather ``better than nothing'', than ``very good''.  So, if you find that some piece of explanation is lousy or if you find anything that should have been mentioned here, but is not, please report it to bug-mailfromd@gnu.org.ua.

Similarly, if the program itself fails to meet your expectations, or does not do what is described in this document; if you have found a bug or happen to have any suggestion... or have written a useful function you wish to share with the rest of mailfromd users, or wish to express your thanks, email it to the same address, bug-mailfromd@gnu.org.ua.

If you think you've found a bug, please be sure to include maximum information needed to reliably reproduce it, or at least to analyze it.  The information needed is:

  * Version of the package you are using.

  * Compilation options used when configuring the package.

  * Run-time configuration (mailfromd.rc file and the command line options used).

  * Conditions under which the bug appears.

     [file gacopyz.texi not included]

     [file strftime.texi not included]

     [file fdl.texi not included]

Concept Index
*************

This is a general index of all issues discussed in this manual