CommuniGate Pro

Cloudmark Plugin for CommuniGate Pro


Cloudmark Plugin Overview

The Cloudmark spam filtering Plugin runs as an External Filter and calculates a spam "score" for each message being processed. Unlike tools with statically defined patterns for spam messages, the Cloudmark Plugin dynamically retrieves new patterns from Cloudmark Network thus providing greater accuracy for new spam messages.

The score ranges from 0 to 100; the higher the message score the more likely the message is spam. The score info is added to message headers so it can be processed by Server-Wide, Domain-Wide and Account Rules.

By default the added header lines look like this:

X-Junk-Score:  92 [XXXX]
X-Cloudmark-Score:  92 [XXXX]
X-Alert: possible spam!
X-Color: red
Besides the digital score value, the header field contains a "bar score" to simplify automated message processing: the more 'X' characters the higher the score. The following ratios between the digital and bar scores are used by default:
Digital score rangeBar score
0[]
1-39[X]
40-80[XX]
81-90[XXX]
91-95[XXXX]
96-99[XXXXX]
100[XXXXXX]

Every day at midnight the Plugin generates a report message about the number of mails processed and their spam scores. By default the report message is mailed to postmaster address from the CommuniGate main domain.

Note: The Cloudmark Plugin is available only for some platforms supported with the CommuniGate Pro server software. Before you order the Cloudmark Plugin License, make sure the plugin is available for your CommuniGate Pro Server platform.

Note: The Cloudmark Plugin requires CommuniGatePro version 5.3.15 or later.

Hardware requirements:Pentium-II or above (UltraSparc for Solaris), Multi-processor systems and multi-core CPUs are encouraged (but not Intel HyperThreading). 512MB of RAM, 200MB of hard disk space.

Note:The Cloudmark plugin requires outbound TCP port 80 network access to Cloudmark's external 208.83.136.0/22 CIDR range to access lvc.cloudmark.com and microupdates.cloudmark.com servers. It downloads about 100MB of updates daily.


Download the Cloudmark Plugins

Cloudmark plugins are available for certain platforms only.
Operating System CPU Download
via
http
via
ftp
Linux (RedHat, Debian, SuSE) x86
x86_64
Sun Solaris Sparc
x86_64
FreeBSD 7.0+ x86
x86_64
Microsoft Windows XP/2003/Vista/7
Microsoft Windows 95/98/ME
x86
x86_64

The current version of the Plugin is 2.4


Installing on a Linux System.

Note: The following libraries are required for correct operation of the Cloudmark Authority Engine on Linux-x86 platform.
Library Name Location
libpthread.so.0 /lib
libstdc++.so.5 /usr/lib
libm.so.6 /lib
libgcc_s.so.1 /lib
libc.so.6 /lib
libdl.so.2 /lib
librt.so.1 /lib
ld-linux.so.2 /lib
Note: The following libraries are required for correct operation of the Cloudmark Authority Engine on Linux-x86_64 platform.
Library Name Location)
libpthread.so.0 /lib64
libstdc++.so.6 /usr/lib64
libm.so.6 /lib64
libgcc_s.so.1 /lib64
libc.so.6 /lib64
libdl.so.2 /lib64
librt.so.1 /lib64
ld-linux-x86-64.so.2 /lib64


Installing on a FreeBSD System.

Note: The following libraries are required for correct operation of the Cloudmark Authority Engine on FreeBSD 7 platforms.
Library Name Location
libthr.so.3 /lib
libc.so.7 /lib
libm.so.5 /lib
libgcc_s.so.1 /lib
libstdc++.so.6 /usr/lib


Installing on Solaris System.

Note: The following libraries are required for correct operation of the Cloudmark Authority Engine on Solaris-Sparc platform.
Library Name Location
libm.so.2 /usr/lib
libnsl.so.1 /usr/lib
libc.so.1 /usr/lib
libsocket.so.1 /usr/lib
libresolv.so.2 /usr/lib
libdl.so.1 /usr/lib
librt.so.1 /usr/lib
libpthread.so.1 /usr/lib
libCstd.so.1 /usr/lib
libmp.so.2 /usr/lib
libaio.so.1 /usr/lib
libCrun.so.1 /usr/lib
libthread.so.1 /usr/lib
libc_psr.so.1 /usr/platform/$(uname -i)/lib
libCstd_isa.so.1 /usr/lib/cpu/sparcv8plus

Note: The following libraries are required for correct operation of the Cloudmark Authority Engine on Solaris-x86_64 platform.
Library Name Location
libnsl.so.1 /lib/64
libc.so.1 /lib/64
libsocket.so.1 /lib/64
libresolv.so.2 /lib/64
libdl.so.1 /lib/64
librt.so.1 /lib/64
libpthread.so.1 /lib/64
libCstd.so.1 /usr/lib/64
libmp.so.2 /lib/64
libmd5.so.1 /lib/64
libscf.so.1 /lib/64
libaio.so.1 /lib/64
libCrun.so.1 /usr/lib/amd64
libm.so.2 /usr/lib/amd64
libdoor.so.1 /lib/64
libuutil.so.1 /lib/64


Installing on a MS Windows System.

Note: The following libraries are required for correct operation of the Cloudmark Authority Engine on Windows platforms.
Library Name Location
MSVCR71.DLL {SystemDir}\system32
MSVCP71.DLL {SystemDir}\system32


Updating the Cloudmark Plugin.

When updating the Cloudmark Plugin to a newer version, do the following steps:


Testing the Cloudmark Plugin.

On a Unix System:

On a MS Windows System:

Note: the distributive contains no spam definition databases and on the first launch the Engine will download them and store some files into micro_updates subdirectory; and while the databases are not loaded completely the accuracy may be degraded. To change this behavior you can switch the "download micro-updates before init" in cartridge.cfg file to "yes", however it will make the engine initialization to take quite long time.

Note: the test spam message from test.msg file may become obsolete and can be removed from spam definitions databases; so this is not a malfunction if it's scored 0. Try replacing it with a fresh spam.


Integrating the Cloudmark Plugin with CommuniGate Pro.

Step #1: Create the Helper

Please check the External Filters section of the CommuniGate Pro manual.

Open the General page in the Settings section of the WebAdmin Interface and click the Helpers link. Create a Helper for the Cloudmark Plugin:

Content Filtering
Log Level: Program Path:
Time-out: Auto-Restart:

Note: For a MS Windows system the Program Path should be CGPCloudmark\CGPCloudmark.exe with the back slash "\" as the path separator.

Note: On some versions of FreeBSD system you may need to specify the full path to the program, i.e. /var/CommuniGate/CGPCloudmark/CGPCloudmark

Step #2: Create the Scanning Rule

To invoke the SpamCathcer Helper you should create a Server-Wide Rule with "ExternalFilter Cloudmark" action. The Scanning Rule will apply Cloudmark to the message and the spam score will be added to the message header.
Note: It must be a Server-Wide Rule, not Domain-Wide or Account-level.

The recommended Scanning Rule is as follows:

Data Operation Parameter
Action Parameter

This rule skips messages from the MAILER-DAEMON address (such as non-delivery reports, return-receipts, etc.), skips messages from Client IP Addresses and from authenticated senders, and includes only messages for local accounts and mailing lists.

Note: The unlicensed installation of Cloudmark Plugin is limited to 5 messages per hour. If the E-mail traffic exceeds the limit, the Plugin will let the messages go through unrated.

Step #3: Dealing with the Rated Messages

Cloudmark by itself doesn't block spam, it only assigns a spam score to the messages. To actually block spam you need to create yet another Rule which blocks messages according to their spam score. There are many scenarios possible:

Scenario #1: suitable for small companies where you can assign one person (e.g. postmaster) to look through the spam messages daily to check for false positives, and if any false positives found - redirect them to the appropriate persons.

Create a Server-Wide Rule with the following contents:

Data Operation Parameter
Action Parameter

This Rule moves the incoming messages with score 96 and greater to the "spam_box" mailbox of the postmaster@domain.com account. The "Discard" action is required to prevent the message from going to the initially intended destination (INBOX mailbox). Note in the example above, the "*" in [XXXXX* is necessary to filter all messages scored above 5 X's. Without it, the rule will only filter out messages with 5 X's.
Note: The priority of this Server-Wide Rule must be lower than the priority of the Scanning Rule.

Scenario #2: suitable for large companies and ISPs. Let users to deal with spam on their own.

Create one Domain-Wide rule or many Account-level rules for each account with the following contents:

Data Operation Parameter
Action Parameter

This Rule moves the incoming messages with score 96 and greater to the "Junk" mailbox of the original recipeint account. The users should regularly check their "Junk" mailboxes and purge them. The "Discard" action is required to prevent the message from going to the initially intended destination (INBOX mailbox). Note in the example above, the "*" in [XXXXX* is necessary to filter all messages scored above 5 X's. Without it, the rule will only filter out messages with 5 X's.

The "Junk" mailbox from the above example must exist in every account in the domain. Otherwise the Rule will fail and the message will be delivered into the user's INBOX.

Alternatively, you can use "Junk Mail Control" simplified Rule on domain or account level:

Junk Mail Control
High probability: Medium probability: Low probability:

Scenario #3: suitable for large companies and ISPs for users who don't have access to mailboxes other than INBOX, e.g. POP3 users.

Create one Domain-Wide rule or many Account-level rules for each account with the following contents:

Data Operation Parameter
Action Parameter

This Rule marks subjects of spam messages with [SPAM] prefix.

Scenario #4:suitable for companies with relatively small input traffic, available from CommuniGate Pro version 5.1 and greater.

In CommuniGate Pro version 5.1 and greater you can enqueue messages synchronously. Use the WebAdmin Interface to configure the Enqueuer component. Open the Queue page in the Settings->Mail realm. Clear off the checkbox of the "Enqueue Asynchronously" option:
Message Enqueuer
Log Level: Processors:
Hop Counter Limit:   Enqueue Asynchronously

Please see the details in CommuniGate Manual.

Create a Server-Wide Rule with the following contents: <

Data Operation Parameter
Action Parameter

When enqueueing synchronously, when a message is rejected with a Server-Wide Rule it is rejected on SMTP level with 5xx error code, rather than accepted and bounced.

In any scenario it's not recommend to discard spam messages blindly without saving them because of the possible false positives. It's either highly not recommended to automatically reject spam (unless you're in synchronous mode using scenario#4) because usually the return addresses are forged and the rejection notice message will go to an innocent person or a spamtrap, which may result in your server to become blacklisted. When rejecting in syncronous mode, the sending host will get an error during SMTP transaction and there will be no bounce message generated by your server.

The recommended threshold (the score you start treating messages as spam) is 96. If not enough spam is caught then lower the threshold to 90; if there too many false positives, raise the threshold to 100.


The Plugin Configuration File

On startup the Cloudmark Plugin reads the contents of the CGCloudmark.cfg file from the current directory. The format of the file data elements is described in http://www.communigate.com/CommuniGatePro/Data.html. The description of the data elements you may find in the CGCloudmark.cfg file. The default CGPCloudmark.cfg is available here.

The default CGPCloudmark.cfg has the following contents:

Header="X-Junk-Score: ^1 [^2]";
This line defines the header to be added to the rated messages.
The ^1 combination is replaced with the digital message score.
The ^2 combination is replaced with the bar score.
To create a multi-line header use the \e combination as a line breaker. Make sure each line is a RFC-compliant header, it would be best if you start each with the "X-" prefix. Example: Header="X-Score: ^1\eX-Bar-Score: ^2"

AlertLevel=96;
This line defines the score which triggers the AlertHeader to be inserted into the message, and the messages whose source and destination addresses will be listed in the daily reports as Spam Sources and Targets.

AlertHeader="X-Alert: possible spam!\eX-Color: red";
This line defines the header to be added to the rated messages if its score is equal or greater than the value of AlertLevel. The "X-Color: red" combination changes the message color when viewed via CommuniGate Pro WebMail interface.
Note: To dispatch spam via Rules you may check for the AlertHeader presence instead of checking the message scores, but this method is not flexible because different users may want to use different scores as a threshold.

SubmittedDirectory = "Submitted";
This line defines the CommuniGatePro Submitted directory required for submitting the reports via PIPE module. There can be relative or absolute path, e.g. "/var/CommuniGate/Submitted"

OnLicenseLimitReached=Pass;
This line defines the behaviour of the Plugin when the number of messages exceeds the licensed limit. When it is set to "Delay" the Plugin suspends the CommuniGate Pro Queue processing module until next window, when it is set to "Pass" the Plugin lets extra messages to go through unrated. Messages not scored will not have any X-Cloudmark-Score headers. You will also be notified in CommuniGate log when your license has reached its limit.


The Cloudmark Engine Configuration File

In initialization time the Cloudmark Engine reads configuration options specified in the cartridge.cfg file and the whitelisting options in whitelist.cfg file.

Micro-Updates

Micro-updates is the mechanism that allows the Cloudmark Cartridge to regularly download the latest fingerprint data used to identify abusive messages. The fingerprint data is provided by highly trusted reporters and analysis by the Trust Evaluation System in the Cloudmark Collaborative Security Network in near real-time. Therefore, proper use of the micro-update mechanism is critical in maintaining cartridge accuracy as new types of spam, phishing, and virus messages are reported.

See the details in the default cartridge.cfg file.

Network interaction

Micro-updates are downloaded using standard HTTP requests. If an HTTP proxy is not enabled, then at the specified interval, a download will be attempted over port 80. If this fails, the most recent offline version of the data file in the micro_updates directory will be used until the next micro-updates interval before attempting to download again.

Data file integrity and security

The micro-update files containing data are compressed and encrypted. Data that is not encrypted with the correct key will be ignored. As a result, DNS or IP spoofing is not a concern. The contents of the data files are read into memory at startup, as well as each time a new file is downloaded.

Cartridge statistics

By default, the Cloudmark Cartridge sends cartridge configuration information and message scanning statistics back to Cloudmark. By collecting this information, Cloudmark can more effectively detect potential accuracy issues and proactively address them before there is a need for the customer to contact Cloudmark. If your organization has special privacy concerns, you have the option to disable statistics reporting.

Statistics are always collected at each cartridge installation. Collected statistics are not written to disk and they are not accessible at an installation. By default, statistics are reported to Cloudmark at the following URL: http://<configured micro-update hostname>/cmstats The default hostname is microupdates.cloudmark.com. If the cartridge is configured to use a HTTP proxy for micro-update downloads, statistics will be sent using the same HTTP proxy. By default, the cartridge communicates statistics back to Cloudmark every hour. The statistic reporting interval is managed by Cloudmark; Cloudmark may adjust the frequency of statistic reporting interval when assisting customers with accuracy issues.

In the cartridge.cfg file in "customer id = communigate-customername@feedback.cloudmark.com" please change the "customername" to your company name.

Whitelisting

A whitelist is a list of trusted senders from whom you always accept email, or email characteristics which indicate a trusted message. This feature of the Cloudmark Cartridge minimizes the filtering of legitimate messages and allows system administrators to conveniently manage the receipt of messages from known safe senders.

See the details in the default whitelist.cfg file.


Launching the Plugin from the Command Shell.

The Plugin is so-called text-only application which can be launched from a command shell, and it accepts some command line options. When used with CommuniGate Pro as a helper applicaiton the Plugin does not require any command line options.

Rating Message Files.

The Plugin can be used to calculate the spam scores of a number of message files in a directory.

The syntax of the program is: CGPCloudmark RATE [options] <directory>

The <directory> must contain message files in RFC822 format or in CommuniGate format (RFC822 with the envelope info), there must be one message per file. You can use messages form CommuniGate mailbxes in .mdir format. Messages in .EML, .MSG, .mbox and other formats must be converted to the RFC822 format. Be careful not to alter the messages. For example, "Received" headers are often accidentally removed. Note that mail clients very often modify the messages when they save them.

Example: ./CGPCloudmark RATE /var/CommuniGate/Queue


Reporting misclassified messages to the Cloudmark Network Feedback System

The Cloudmark Network Feedback System is an automatic feedback mechanism that allows Cloudmark ISP and enterprise customers (or CNFS users) to submit misclassified messages directly into the Cloudmark Service. Cloudmark Network Feedback System enables CNFS users to immediately become members of the Cloudmark Collaborative Security Network, joining the millions of users currently providing real-time feedback to the Cloudmark Service on the latest email threats. Cloudmark will immediately begin tracking the reputation of each reporter and gauge their feedback appropriately. Feedback from trusted reporters will lead to near real-time accuracy improvement at the gateway where Cloudmark is filtering spam.

The technical requirements when submitting misclassified messages to Cloudmark:

The feedback messages should be mailed to one of the following addresses:

communigate-legit@feedback.cloudmark.com - for false positives
communigate-spam@feedback.cloudmark.com - for false negatives
communigate-phishing@feedback.cloudmark.com - for phishing false negatives

Feedback messages that are not submitted as an RFC822 attachment will not be forwarded into the Cloudmark Service for evaluation. However, these feedback messages will be tracked for statistical analysis purpose.

To use Microsoft Outlook to submit feedback:

  1. Launch Outlook
  2. Open a new message window by clicking on the New button on the Outlook toolbar or choose File > New > Message from the menu options.
  3. Drag the misclassified message(s) onto the new message window to attach them.
  4. Send the new message containing the attachments to one of the above listed feedback addresses.

To use CommuniGate WebMail interface to submit feedback:

  1. Open the misclassified message from list to a separate window
  2. Click "Forward" link (or icon, depending on the skin you use) to compose a feedback message
  3. Enter one of the above listed feedback addresses into "To:" input field
  4. Click "Send" button (or icon).

CommuniGate® Pro Guide. Copyright © 1998-2013, Stalker Software, Inc.