IMAP-Agent

Configuration: Site.XML and the external [user].XML files.

All configuration of IMAP-Agent is performed with as little as a single XML file or can be split up into as many files as suits the administrator.  Splitting up the configuration can be done at any time and the additional files can contain information for one or more users and can be placed anywhere in the filesystem that IMAP-Agent has access to open them.  Site.xml is the one required file. 

In smaller installations it makes sense to have everything in the Site.xml file but in larger more complex installations that becomes cumbersome and it can be simpler to split the configuration up into smaller files.

 

To simplify maintenance of the configuration file(s), many attributes are optional or conditional with default values.  Some attributes have a range of possible values, generally it is not acceptable to add attributes and leave them blank.  Optional tags should either have a value or not be used.

SITE.XML Sample File

Here is a sample Site.xml document showing a reasonable configuration. 

The example has been color coded to show the main sections.

Red text identifies the config tag.

Yellow hightlight:  Everything in this section has site-wide scope defining mailservers and site-wide rules.

Blue hightlight:  The user information section defining user specific rules.

 

The purpose of the ADDITIONAL.XML file:

You should also notice the ADDITIONAL.XML file below the Site.XML file.  In the Site.XML sample file there are two users whose information is stored in the additional.xml file.  If you look closely, you will see the names of those two highlighted in green and purple.  

One or more users information can be located in secondary xml files such as this.  Look for the gray highlighted text in the Site.xml and you will see the name of the secondary file.  If the gray highlighted text was removed, IMAP-Agent would assume the name of the secondary xml file was identical to the user’s name attribute.  So for example, if the gray highlighted text was gone for the user named lister then IMAP-Agent would look for a file named lister.xml.

Note: Secondary xml files cannot refer to tertiary xml files.   The Site.xml file can optionally refer to secondary.xml files but those files must contain the users information.

 

 

 

<?xml version="1.0" encoding="UTF-8" ?>

<main>

<config run="y" loglevel="1" maxlogsize="1000" sleepseconds="3" version="1">

                <comment>            

                                                The pwd attribute can be an actual password or can identify

                                                any console type program to run that will look up the

                                                password.  See the password lookup plugin documentation for

                                                more details.

                </comment>

                <all_users >

                                <comment>These site wide rules will be appended to any rules in the user's section at runtime.</comment>

                                <mailserver id="1" host="192.168.0.1" port="143" />

                                <mailserver id="fastmail" host="www.fastmail.fm" port="143" />

                                <mailserver id="holly" host="www.domain1.net" port="143" pwd="[plugin]c:\\MercuryMailBoxes\\mercury_pw_plugin.exe"/>

                                <mailserver id="queeg" host="www.domain2.net" port="143" pwd="[plugin]mercury_remote_plugin.exe"/>

                                <rules Note="folder tags are processed in the same order you see here">

                                                <folder name="INBOX">

                                                                <rule note="Move spam rated 25 or greater to Junk folder." to="INBOX.Junk" searchscope="header"><![CDATA[^X-Header:.*(([2-9][5-9]\d*)|[3-9]\d+).*$]]></rule>

                                                </folder>

                                </rules>

                                <comment>These aging values will be used site wide unless overridden in the users section.</comment>

                                <aging>

                                                <folder name="INBOX.JUNK" age="45" maxcount="100"/>

                                                <folder name="INBOX.Trash" age="14"/>

                                                <folder name="INBOX.Sent" maxcount="100"/>

                                </aging>

                </all_users>

 

                <users >

                                <user mailserverid="holly" name="lister" file="y" >additional.xml</user>

                                <user mailserverid="queeg" name="rimmer" file="n" run="n">

                                                <rules Note="folder tags are processed in the same order you see here">

                                                                <folder name="INBOX">

                                                                                <rule to="lister" searchscope="from" ><![CDATA[^lister@domain1.net$]]></rule>

                                                                </folder>

                                                </rules>

                                                <aging>

                                                                <folder name="INBOX.Trash" age="15" maxcount="100"/>

                                                </aging>

                                </user>

                                <user mailserverid="holly" name="cat" logon="cat"  file="y">additional.xml</user>

                                <user mailserverid="fastmail" name="someuser" logon="someuser@fastmail.fm" pwd="likeiwouldknow" file="n">

                                                <rules Note="folder tags are processed in the same order you see here">

                                                                <folder name="INBOX">

                                                                                <rule note="Move spam to Junk folder." to="INBOX.Junk" searchscope="header"><![CDATA[^From:.*(rimmer@domain2.net).*$]]></rule>

                                                                </folder>

                                                </rules>

                                </user>

                                <user run="y" mailserverid="holly" name="kryton" file="n">

                                                <rules Note="folder tags are processed in the same order you see here">

                                                                <folder name="INBOX">

                                                                                <rule to="Logs" searchscope="header"><![CDATA[From: Fluffy the SMTPGuardDog]]></rule>

                                                                                <rule to="Logs" searchscope="subject" smtb="100" smwa="0"><![CDATA[^Holly.*logs]]></rule>

                                                                                <rule to="Logs" searchscope="subject"><![CDATA[Holly.*Motherboard.*Monitor]]></rule>

                                                                                <rule to="dotNET" searchscope="from"><![CDATA[Xtras.*Net.*Developer.*Network]]></rule>

                                                                                <rule to="dotNET" searchscope="from"><![CDATA[DevX]]></rule>

                                                                                <rule to="jobs" searchscope="from"><![CDATA[Monster]]></rule>

 

                                                                </folder>

                                                </rules>

                                                <aging>

                                                                <folder name="logs" maxcount="100"/>

                                                                <folder name="dotNET" maxcount="100"/>

                                                </aging>

                                </user>

                </users>

 

 

</config>

 

</main>

 

 

[ADDITIONAL].XML Sample File

<?xml version="1.0" encoding="UTF-8" ?>

<main>

<config  run="y" loglevel="4" version="1">

 

                <users >

                                <user name="lister" run="y">

                                                <rules Note="folder tags are processed in the same order you see here">

                                                                <folder name="INBOX">

                                                                                <rule to="Logs" searchscope="header"><![CDATA[From: Fluffy the SMTPGuardDog]]></rule>

                                                                                <rule to="Logs" searchscope="subject" smtb="100" smwa="0"><![CDATA[^Holly.*logs]]></rule>

                                                                                <rule to="Logs" searchscope="subject"><![CDATA[Holly.*Motherboard.*Monitor]]></rule>

                                                                                <rule to="dotNET" searchscope="from"><![CDATA[Xtras.*Net.*Developer.*Network]]></rule>

                                                                                <rule to="dotNET" searchscope="from"><![CDATA[DevX]]></rule>

                                                                                <rule to="jobs" searchscope="from"><![CDATA[Monster]]></rule>

 

                                                                </folder>

                                                </rules>

                                                <aging>

                                                                <folder name="logs" maxcount="100"/>

                                                                <folder name="dotNET" maxcount="100"/>

                                                </aging>

                                </user>

                                <user name="cat" logon="cat"  >

                                                <rules Note="folder tags are processed in the same order you see here">

                                                                <folder name="INBOX">

                                                                                <rule testtype="custom" to="Keep" searchscope="message">G:\\cvs\\plugins\\CustomRule\\custom_rule_plugin\\bin\\release\\plugin_custom_rule.exe</rule>

                                                                </folder>

                                                                <folder name="Keep">

                                                                                <rule  to="Keep2" searchscope="from" smwa="0"><![CDATA[^lister@domain1.net$]]></rule>

                                                                </folder>

                                                </rules>

                                                <aging>

                                                                <folder name="Keep" age="3" maxcount="15"/>

                                                                <folder run="n" name="Keep2" age="5" maxcount="5"/>

                                                </aging>

                                </user>

 

                </users>

 

 

</config>

 

</main>

 


 

 

 

Element and Attribute Tags

Some of the element tags are can appear in the [additional].xml files.  They are shown in the sample above. 

<main>

Required.  No attributes.

<config>

Required tag.

<config run="y" loglevel="4" maxlogsize="1000" sleepseconds="3" version="1">

 

Attribute

Description

value

Scope

Run

Run IMAP-Agent rules.

“y”, “n”

Optional.  Default = “y”

Loglevel

Set loglevel.

1=error

2=warn

3=info

4=verbose

Optional.  Default = “1”

Maxlogsize

Set log file max size.

Kbytes.

Optional but advisable.

Default = “1000” which is 1Mbyte.  When the log file exceeds the maxlogsize then 10% of the file size is cut with oldest entries being removed.

Sleepseconds

IMAP-Agent can sleep between folders.   If you are worried IMAP-Agent is hogging the mail server.

Integer. 

Optional.

Default = “0”

Version

For versioning of the xml file. 

Numeric.

Optional.

Default = “1”.

 

<all_users>

Required tag.

<all_users smtb="20" smwa="true">

 

Attribute

Description

value

Scope

Smtb

 

Skip message if it’s too big.

Numeric.  Kbytes.

Optional.  Applies to all users.

Default = “0”

Smwa

Skip messages with attachments.

1=True, 0=false

Optional.  Applies to all users.

Default = “0”

 

 

 

 

 

<mailserver>

Required tag.  There must be at least one mailserver.

<mailserver id=”anything” host=”domain or IP” port=”143”>

           

attribute

Description

value

Additional

id

 

Each mailserver must have a unique id

Any string value. 

Required.

host

IMAP host identifier.

Domain name or IP address

Required

port

IMAP port number

 (usually 143).

Required

Pwd

Password tag.  [plugin] indicates a script to be run for retrieving passwords.  See Password plugin section for more details.

String

Optional.  Provides a default password for all child <user> elements using this mailserverid.

Extra

The extra tag is passed as one of the parameters in the Password Lookup plugin.

Usage is user defined.

String

Optional.  Default is empty string.  Provides a default extra string for all child <user> elements using this mailserverid. 

 

 

<rules>

Scope: Optional as a child tag under <all_users>, but required as a child tag under <users>.

Any rules under the <all_users> tag are taken to be site wide rules.  Site wide rules are run for each user after that user’s specific rules have been run. 

 

<folder>

 

<folder name=”INBOX” smtb="20" smwa="true">

<folder name=”INBOX” age=”10” maxcount=”100” totalsize=”500”>

 

Attribute

Description

value

Scope

name

 

Name of the mailbox folder to run the rules on.

String.  Must be a valid mailbox.

Required.  Applies to all child rule tags.

Age

Age of the message in days.

Numeric.

Optional.  This attribute applies when the folder tag is a child of the <aging> tag.  It only applies to aging.

Maxcount

Max count of messages in the mailbox.

Numeric

Optional.  This attribute applies when the folder tag is a child of the <aging> tag.  It only applies to aging.

Totalsize

Total size quota for the folder.

Numeric.  This means Kbytes.

Optional.  This attribute applies when the folder tag is a child of the <aging> tag.  It only applies to aging.

Smtb

 

Skip message if it’s too big.

Numeric.  Kbytes.

Optional.  Applies to all rules in this folder.

Default = “0” which means ignore.

Overrides site wide (see “all_users” tag).

Smwa

Skip messages with attachments.

1=True, 0=false

Optional.  Applies to all rules in this folder.

Default = “0”

Overrides site wide (see “all_users” tag).

Run

Run IMAP-Agent rules.

“y”, “n”

Optional.  Default = “y”

 

 

<rule>

A rule to be performed against new mail.

<rule action=”move” to=”Junk” searchscope=”header” testtype=”regex” smtb=20” smwa=”true”>tag value</rule>

 

Note: Rule tags located under the <all_users> tag are appended to the end of each users set of rules such that user rules get run and then site wide rules get run.

 

Attribute

Description

value

Scope

Action

Action to perform if the rule test is successful.

“move”, “copy”, “delete”

Optional.  Default is “move”.

To

The destination mailbox folder.

String.  If folder doesn’t exist it is created.

Required when action is “move” or “copy”.

Searchscope

What part of message to apply rule test against.

“header”, “subject”, “from”, “body”, “message”

---------------------

“message” means headers and body but does not include attachments.

Optional.  Default is “message”

Testtype

Test type to perform.

“regex” is currently the only test type.

Optional.  Default is “regex”.  

To create a custom rule you set testtype to anything else buy “regex”.  See Value attribute below.

Smwa

Skip messages with attachments.

1=True, 0=false

Optional.  Applies to this rule.

Default = “0”

Overrides site wide and folder settings.

Smtb

 

Skip message if it’s too big.

Numeric.  Kbytes.

Optional.  Applies to this rule.

Default = “0” which means ignore.

Overrides site wide and folder settings.

Value

Regex pattern.  Rules are fired when regex evaluates to success.

Example:

<![CDATA[^X-Header:.*(([2-9][5-9]\d*)|[3-9]\d+).*$]]>

Required.

If Testtype != “regex” then this value contains the path to the custom rule plugin.

Example:

C:\\foo\\plugin_custom_rule.exe

For more information see the section on Custom Rules.

 

 

<users>

Parent tag where individual user parameters are described.

<users pwd=”%script%e:\bin\pwlookup.exe” extra=”anything”>

Attribute

Description

value

Scope

Pwd

Password tag.  [plugin] indicates a script to be run for retrieving passwords.  See Password plugin section for more details.

String

Optional.  Provides a default password for all child <user> elements.

Extra

The extra tag is passed as one of the parameters in the Password Lookup plugin.

Usage is user defined.

String

Optional.  Default is empty string.  Provides a default extra string for all child <user> elements.

 

 

 

 

 

 

<user>

<user mailserverid=”1” logon=”mailaccount” pwd=”password” run=”y” file=”n”>  <![CDATA[dlewis.xml]]></user>

Attribute

Description

value

Scope

Name

Unique identity for this user.

String

Required.  If file=”y” then this is used to locate the same user in the additional xml file.  See Value field below.

See

See special “Name and Logon Attribute Notes” below.

Mailserverid

Matches the id attribute for a mailserver tag.

String

Required.

Logon

Users mail account code.

String

Optional.  Uses name attribute if omitted.

See special “Name and Logon Attribute Notes” below.

Pwd

Password tag.  [plugin] indicates a script to be run for retrieving passwords.  See Password plugin section for more details.

String.

Optional.  This may be specified in the <users> tag.  Overrides any defaults – see <users> and <mailserver> tags.

Extra

The extra tag is passed as one of the parameters in the Password Lookup plugin.

Usage is user defined.

String

Optional.  Default is empty string.  Overrides any defaults – see <users> and <mailserver> tags.

Run

Run rules for this user when IMAP-Agent runs.

“y”, “n”

Optional.  Default = “y”

File

This user’s parameters are located in a different xml file.

“y”, “n”

Optional.  Default = “n”

Value

The text value for the <user> tag.

String

If file=”y” then this value is the filename where this users data is located. 

 

Name and Logon Attribute Notes:

The “name” and “logon” attributes have some special characteristics. 

1.      The name attribute is used as an id in the <users…> tag to uniquely identifying each user.  For that reason it needs to be unique.

2.      The name attribute is also used as a filename to store that users history.  Again, for that reason it needs to be unique.

3.      The name attribute can double as the users logon account if the logon attribute is omitted.  This was done to simplify configuration setup.

4.      You might need to use both the name and logon attribute if IMAP-Agent will be managing more than a single mailserver.  In that case, it is possible to have an identical logon account on both mailservers and since the name attribute MUST be unique, at least one of the name attributes would have to be different than it’s logon attribute.

Special Note:  If you ever want to change the contents of a “name” attribute, be sure to rename the history (“name”.dat) file to the new name.  That will preserve the users history. 

 

File and Value attribute Notes:

If file = “y” and Value = “” then the name attribute will be used as the filename.

<aging>

Aging tags are parent tags containing folder tags.  <aging> and <rules> tags are at the same level.

They are separated because aging is only performed periodically such as once per day while the other rules are performed many times per day, perhaps many times per hour and this makes for easier maintenance.

Any combination of the three quotas can be used in the same folder.

The aging process evaluates in this order:

1.      totalsize

2.      maxcount

3.      age 

TotalSize and maxcount are the most efficient.  Totalsize must fetch the message before deleting it.  Maxcount doesn’t need any information from the message and can delete until the maxcount is satisfied.

Age requires fetching each message in the folder to get the message date.

All messages are fetched and/or deleted using their UID.   Messages are assigned UID’s when they are added to a folder.  UID order of the messages may not coincide with date order.   For this reason, it might be better to use the age quota even though it uses more resources.

 

Note: To clarify something about the age quota and the MIME Date: header in the mail messages, I will describe this specific scenario. 

Scenario:  Aging using the age quota is set for 15 days on the “keep” folder.  A message received now at the INBOX folder is moved to the folder named “keep” by IMAP-Agent based on a rule.  So, now the message (less than one day old) that was just received is in the “keep” folder.  Then the user manually moves a 20 day old message from some folder into the “keep” folder.  When Aging is run that night, the 20 day old message will be deleted.  The reason is that the MIME “Date:” header doesn’t get changed by moving the messages around in folders.  In fact, there is no reliable way to determine how long a message has been in any given folder because some (perhaps most) IMAP servers preserve INTERNALDATE when copying messages.  So, the age quota concerns the age of the messages since they were sent by the sender (which should be minutes before they are received by you if you have an always connected mail server) , not how long they have been in the folder.  [I will point out that using maxcount or totalsize, messages are deleted in the order they were added to the folder.]

 

Note: Subject: IMAP - message date question: 

http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&threadm=a8a0ea.0404021022.369e44cc%40posting.google.com&rnum=3&prev=/groups%3Fhl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26q%3Dspamtrap%26meta%3Dgroup%253Dcomp.mail.imap

Note: In all aging quota cases, only the message header is fetched.  The body and any attachments are never fetched.

 

Note: Deleted messages are not moved to the mailbox Trash folder, but are immediately expunged from the mailbox.  There is no ability to retrieve deleted messages. 

 

Attribute

Description

value

Scope

Totalsize

This means the total size quota for the folder in Kbytes.

String

Optional.

A value of zero “0” means ignore.

Maxcount

The maximum number of emails quota.

String

Optional.

A value of zero “0” means ignore.

Age

Meaning number of days old.

All email older than (Age) days is deleted.  The MIME “Date:” header is where the email date is taken from.

String

Optional.

A value of zero “0” means ignore.