Server Security Services

 

For additional information and pricing on the installation of these tools, email sysadmin@mis-outsource.com.

 

An excerpt from the FAQ about the security of the Plesk control panel program:

 

PSA itself is secure within the context of its usage; however, a standard operating system installation is inherently insecure and requires manual user intervention to ensure the security of a server. Administrators must take security precautions within their operating system. Specifically, you should consult documentation on the appropriate Linux or UNIX OS running on your server, especially those documents relating to remote access and firewall security.

 

Addressing those operating system and access related issues mentioned above is an important task, as without addressing these issues, no server is safe. Many web hosts lack the staff to deal with the “manual intervention” required, or would rather concentrate more on developing their hosting business than on playing “system administrator”, which is where MIS Outsource can help out.

 

The internet is NOT a friendly place – every server connected to the Internet will experience scans and other attempts to compromise it on a regular basis – on average of once per hour or more depending on the network segment you’re connected through. Most of these attacks are automated scans looking for common exploits, and many others are worms / viruses / Trojan horses performing automated queries for other servers to infect. Some may be specifically targeted at your server. While ANY server can be successfully attacked by a determined hacker, most attacks are relatively unsophisticated, and can be countered by simply making it difficult for an attacker to gain entry so that they move on from your server to an easier target.

 

To make your server one of those that are difficult to attack, MIS Outsource can install a number of programs and packages. These packages are designed to work together with the elimination of known exploits through Linux updates and configuration of each package to work with each other package to create a consolidated barrier to entry, while at the same time allowing all legitimate server traffic to pass unaffected. All packages are time-tested on our own servers, so you’re not experimenting. It does have to be mentioned that each server is somewhat different, and these packages have to be tailored to each server and its applications, tested, and to a certain extent, debugged.

 

Red Hat Linux / Plesk Security Package

 

This package consists of:

 

1. All current errata & updates to Red Hat 7.1/7.2/7.3, including the latest OpenSSH, Bind, and Apache vulnerability fixes as needed. Update to Plesk version 2.5.3 (part of the Apache update).

2. Installation of chkrootkit and automated monitoring / email results to the system administrator. Chkrootkit monitors and detects the presence of root kits that give attackers control over your server - and typically hide their existence.

3. Installation of Portsentry - Intrusion Detection system / automatic blocking of attackers. This includes implementation of selected honeypots to attract and lock out attackers. Portsentry will constantly monitor common exploits and shut down and firewall would-be attackers in about 1 second from their initial attack.

4. Installation of Logsentry - monitors log files and report suspicious activity to the system administrator. Training and support on when to worry and when not to is included.

 

Special Note: Psionic Software has recently been acquired by Cisco Systems, and these two wonderful programs are no longer available as open source. We can still install them, either as part of an overall security system or as a standalone as we have archived copies.

 

5. Installation of a proven IPTABLES-based kernel firewall to prevent attacks to known vulnerable ports and survivability to most DOS attacks. Our own servers have shrugged off a concerted 30 minute ICMP attack, a synflood attack that lasted about 15 minutes, and a mail bomb attack that lasted several hours. Normal service of websites was unaffected. This firewall includes scripts to make it easy for the system administrator to manage lists of blocked sites, quickly block a site while an attack is in progress, and lock down the server totally (Panic mode) in the event extreme measures would be necessary.

6. Review of services running currently and permanent shut down of unneeded vulnerabilities.

7. Configuration of proftpd (the FTP server daemon) to maximize security of FTP services.

8. OpenSSH security enhancements - in its current configuration, OpenSSH on a server is vulnerable to several known exploits which can be addressed through an update and configuration changes. Integration of these changes to OpenSSH and intrusion detection / prevention makes shell access very safe. An optional (but valuable) addition to OpenSSH is the configuration to use public key / private key RSA Authentication. For the user, this can really simplify login as the client "appears" to login automatically, but if fact login is handled by a private key that's as secure as you can keep your local PC. No password is actually sent. Only the PuTTY SSH client is  supported at this time.

 

Available at an additional cost is Tripwire. Tripwire establishes a database of all the installed programs on your server, and can be configured to scan for changes on any frequency desired (daily is suggested) and email the system administrator if any files have been changed. The enhancement to system security is pretty obvious, but Tripwire is tedious to configure on a system (thus the added cost), and quite sensitive, so you have to be able to understand and interpret it’s reports. If you update any programs (Plesk updates, Red Hat up2date updates, etc.) then you also have to rebuild the Tripwire database or it will scream at you every time it runs. It’s not for everyone, but it is a good security tool.

 

Email Virus Scanning

 

This package installs a virus scanning engine in the Linux server, configuration to update new virus definitions (daily) to insure protection is current, and configuration to scan each email message that is sent to a user on the server. Note that this does NOT scan outgoing messages at the current time, nor will it scan redirects (email messages sent through Plesk / Qmail but not actually received into an email box on the Plesk / Qmail server). As new email users are added, scanning is automatically integrated to their account. From an administrator's standpoint, installation of this package does not change any of the current tasks.

 

If a virus is sent to any user on your server, a warning message will be sent to the sender, the intended recipient, and the server administrator informing them of the attempt. The email header of the message is included in the warning so that you can determine who sent it, how, and what virus they sent. Here is an example of the message that would be sent to the intended recipient of a virus-infected email:

 

To: recipient@yourdomain.com

Subject: VIRUS WARNING : EICAR_Test_File
 

You were sent a message with a virus. For your security, this message was
deleted from the system. The subject of this message contains the name of
the virus. The message header can be seen below so that you can see who
sent you the virus - they have already been notified of this problem:

Received: (qmail 23140 invoked from network); 22 Jul 2002 11:26:41 -0000
Received: from speedlink-xx-yyy-zz-aaa.senders-domain.com (HELO host) (xx.yyy.zz.aaa)
  by personalcopy.com with SMTP; 22 Jul 2002 11:26:41 -0000
Message-ID: <000801c23172$a6b41e10$6ec8a8c0@senders_machine_id>
Reply-To: "Any One" <anyone@sendersdomain.com>
From: "Any One" <anyone@sendersdomain.com>
To: <recipient@yourdomain.com>
Subject: hope you like this file i sent u
Date: Mon, 22 Jul 2002 07:26:40 -0400
Organization: Your Company
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000

 

Messages sent to the sender and the system administrator are similar but the explanation describes why they were sent the message. Messages can be customized and multi-lingual.
  

Email - SMTP Security & tarpitting

 

Users of domains hosted on a web server should be not only allowed but encouraged to use their domain name not only for receiving but also for sending email. If your customer's domain is "progolfer.com", then email sent by them should be sent through that domain and reflect the user’s domain name in all reply-to and email headers instead of their local ISP’ domain - it is more professional and all major companies have their own mail servers and their email messages reflect their domain names. Coming from a business, an email address of AOL.COM or HOTMAIL.COM looks, candidly, “Mickey Mouse”. Your customers can also have this benefit - in fact, they already do but may not be using it. This package includes setting up Plesk so that it is not an open relay and subject to spammers sending through your server, documentation for your customers (PDF Manual with pictures and explanations) on how to configure Outlook Express to send and receive email through their domain), and protecting your server from abuse by an "internal spammer" - i.e. one of your own users.

 

A technique to protect the server is called "tarpitting" - in simple terms, if a user tries to send large quantities of an email message (you determine how many emails you'll let a user send - this number is typically between 20 and 60) the server will introduce a delay - typically a long one (30 seconds to 5 minutes) before it will allow any messages over this number limit to be sent. The would-be spammer will see his email client appear to freeze as it waits for the messages to be sent - and typically will give up. For message quantities less than the count, there is no effect on mail performance. Most ISPs using Qmail also do tarpitting.

 

(NOTE: for Plesk users, tarpitting currently works only if you have an RPM install of Plesk - the standard install puts qmail in a different location, and we don't yet have tarpitting working for the standard install.)

 

It is important to you as a hosting provider to protect your server from unauthorized use by a spammer – many servers now check email messages against IP address known to be used by spammers and block them automatically. If this should happen to your server’s IP address, your customers would be unable to send email to their important contacts – and you’d be left trying to get the problem fixed. It’s far easier to prevent it than to fix it after the fact!

 

This package requires recompiling portions of Qmail to incorporate the tarpit code, and instructions on how to turn it off if desired.

 

For those REALLY concerned about spammers using your server to send spam, or if you already have an existing problem, we can install monitoring so that if an unusual number of messages are sent from your server, the server admin gets a notification of this so further investigation can be made. If desired, we can configure this monitor to shut down all SMTP services so that no further emails can be sent until investigation - the email to the system admin is sent prior to disabling email.

 

Spam detection and elimination - Spamassassin

 

Spamassassin is a GPL program that utilizes heuristic content scanning algorithms to scan not only the email headers, but the subject and body of incoming email messages destined for a user on your server. It is highly configurable and can be set to whitelist (allow) or blacklist (reject) email from any sender in addition to scanning all messages from known or unknown senders. With this package, you no longer have to manage "blacklists" on your server to try and reduce spam. Spam received can be deleted automatically, sent to a central email account that collects all spam for the server, sent to a common spam email account for each domain, or sent to a spam email account for each user. These options require setting up additional email accounts, but even a limited domain Plesk license does not limit email accounts.

 

Activation of spam protection is not automatic - the server administrator may configure it through a shell account, but there is a simple shell program to simplify this task. Since some users actually want spam, it requires a conscious decision by the system administrator to activate.

 

In addition to the Administrator option to implement spam scanning, MIS Outsource has a new web-based user interface where your users (any user with an email account can log-in with their email address and password) can turn spam filtering on or off and change the properties of how it's implemented, including sensitivity, mode of operation, and user-level whitelists and blacklists. This interface is independent of Plesk or any other service and installs on a domain of your choosing. If we install this interface, we also configure Spamassassin to support user level preferences via an SQL database.

 

This package does alter the content of a spam message, and can also add a header line showing its inspection results to all messages; since altering of an email header should only be done with the customer’s knowledge and approval, this is another reason why implementation requires a conscious decision by the system administrator, or the user interface where users can make their own choices about spam filtering, and change them at any time.

 

This system is highly effective at reducing spam – since implementing it on our own email accounts, we have received only one spam message that was not rejected. If it has a fault, it is that it rejects messages you might actually want to see. The reason is simple – most companies information messages and newsletters are automatically generated and sent to mailing lists – spammers use the same techniques to send to their own mailing lists, and make their messages look just like legitimate newsletters, so any program that will catch spam will catch some email as well – this is why the program includes a whitelist so that you can allow messages that fail all the tests and are detected as spam to still be delivered.

 

Your degree of involvement in this process is up to you – most commercial sites that offer spam filtering do not give you any control over this.

 

Spamassassin works extremely well with Qmail but does require the installation of several programs in addition to the base program – most are written by Dan Bernstein, the author of Qmail. The Spamassassin installation by MIS Outsource includes a newly written script that can handle email forwards / redirects, autoresponders, and mail groups without reconfiguring and yet still delivers all emails correctly. It also supports a full "tag & release" mode, in which all email to a user account is annotated as spam or non-spam, and forwarded to a local mailbox or a email forward WITH the tags so that the user can configure Rules in Outlook or Outlook Express to separate normal email from spam email.

 

One commonly used technique to reduce the load on servers that are hit with a lot of spam is to implement rblsmtpd and one of the Real Time Blackhole lists. An advantage with this technique is that messages from a blacklisted IP address are rejected before they even enter your system and are bounced back to the sender - a fitting reward for spammers. But unfortunately this may bounce emails that you or your customers really need. MIS Outsource can configure your server to allow an override so that you can still receive messages from an important blacklisted IP address but reject others by overriding the RBL block for that IP address.

 

Spam detection and elimination - greylisting

 

In the continuing struggle against spam, new tools are always being sought. One relative newcomer is "greylisting" - discussed here. Greylisting capitalizes on how spammers attempt to be efficient in sending spam - by using mail servers that "fire and forget" - these servers send a message to the intended recipient but don't queue a failed delivery and retry later on - they send too many messages for this to be practical. But "normal" mail servers will queue a message and attempt multiple subsequent deliveries of failed messages.

 

A mail server which "greylists" will always refuse the first attempt to deliver a message from a particular sender to a particular recipient from a particular mail server's IP address. If the message is retried later on (a characteristic of legitimate mail servers) the message will be accepted - but if the mail server is a "fire and forget" server, the message will never be resent and thus the spam never accepted.

 

As MIS Outsource implements greylisting, we combine it with tarpitting so protection against sending spam is implemented along with protection against receiving spam.

 

Greylisting can be combined with other spam reduction tools discussed on this page (Spamassassin filtering and RBL filtering) to significantly reduce the amount of spam mail users have to deal with.

 

MRTG Server Monitoring

 

Multi Router Traffic Grapher is installed by default when Red Hat installs, and though it runs all the time, it's setup with a "vanilla" configuration that fails to graph anything.

 

The best way to see what MRTG can do is to visit:

 

https://secure.personalcopy.net/mrtg/mrtg-rrd.cgi/

 

This is a protected directory - use the username: visitor and the password: invited to log on and explore some of the options.

 

Certain things we’re graphing (Cooling Fan Speed and CPU / Mainboard Temperature) depend on a hardware monitor chip on the server's motherboard, which most new servers should have, but can't be done without this.

 

We also graph things like E-Mail messages sent / received - a great way to find out if your email is working correctly - or if you have a spammer on your server! If you think this last is a trivial issue, wait until you try to get your server's main IP unlisted by SPEWS so your customers can send email again. Other "special" log data can be graphed as well. We do a lot of FTP transfers and limit the number of connections that a single user can make. We graph the number of connections that are refused by these rules so we can detect excessive "abuse" of our FTP server.

 

MRTG can monitor just about anything that a server logs or reports - custom graphs are available at some surcharge which depends on how difficult it might be to obtain the correct data. This implementation includes the latest implementation of a Round Robin database for the data, and on-the-fly generation of the graphs to reduce load on the server (pages are only generated or updated when a user requests the page).

 

This can be extremely useful as a monitoring tool - you can just leave the MRTG page open and it will refresh automatically each time there is data added to the graphs (server configured so it’s user / browser independent) to keep tabs on what your server is doing.

 

If you want a real-world benefit from this (besides having something pretty to look at) - consider these graphs from my own server:

 

 

I had been monitoring Mainboard and CPU temp, and seen it climb from around 25°C to over 35°C with no signs of stabilizing. I also monitor cooling fan speed, and knew it was unchanged during this time, and load averages (also monitored) didn't explain the increases.

I emailed the co-location center and gave them access to these graphs and asked the question - what they were able to do once they saw graphically in these graphs is also reflected, graphically. I saved what could have been serious overheating problems because I knew what my server was doing!

Nagios monitoring

Nagios takes a different approach to server monitoring in that it can run on any server, and monitor any other server or group of servers; in addition it can be configured to monitor the same server that acts as its "host". It primarily monitors services such as HTTP, DNS, FTP, MySql, POP and SMTP, PING, but it can also monitor SWAP memory usage, server load, disk space, and even software RAID. It will send email notifications to individuals or groups if services reach WARNING or CRITICAL error levels, which can be customized to "fit" individual servers.

If you are interested in a Nagios monitoring solution, email me at jimroe@mis-outsource.com and I can supply you with a username and password to see Nagios in action.

This can be a time consuming installation and configuration process, so anticipate this when asking for a quote on installation; but it's an extremely valuable tool for monitoring the health of your network - especially if you have multiple servers and you'd like to know "at a glance" that they're all running as they should be.

bwbar monitoring

Sometimes MRTG may be not what you need. There is another very simple bandwidth monitoring tool we're using that shows bandwidth usage in "near real-time" - it updates every 5 seconds in our implementation, but can be set to update more or less frequently. You can see what it does by visiting:

https://www.personalcopy.net/bandwidth/

FTP Log analysis

 

Plesk installs and uses Webalizer - a very capable package for analyzing web page activity for each domain so your customers can see their own information (but not anyone else’s if this is setup correctly). But, Webalizer uses a common configuration file for all the domains (it's modified on-the-fly by Plesk to run individually against each domain's log file) and it’s difficult to set it up to monitor FTP server logs without breaking it (and the FTP logs are different from the HTTP logs).

 

Analog is another more-configurable log analyzer that can be configured to run independently of Webalizer and analyze the performance of the FTP server. If you don't use FTP for anything but web site maintenance, this may not be important, but if you DO allow FTP transfers this is an invaluable tool.

 

You can look at a report for an FTP site by visiting:

 

http://www.personalcopy.net/ftpstat/report.html

 

This is a protected directory: username: visitor password: invited

 

EMail Usage

 

A lot of hosting providers would like to know how much email is being sent / received by their server. This is vital if you believe that one of your users might be using your server so send spam but aren't sure who. Also, since Plesk doesn't report bandwidth usage for email, but as a hosting provider you pay for all bandwidth without regard to how it was used, this could be the tool you need.

 

We install and configure Isoqlog in a protected directory on your server and set it to update mail traffic to and from your server hourly.

 

If you want to see a sample of this program, visit:

 

http://www.enderunix.org/isoqlog/output/

 

PhpMyAdmin

 

This MySql database management tool is installed and used by Plesk to manage user MySql databases, but is only available through the Plesk control panel to either the system administrator, or a client. It is limited also in that it cannot access the Plesk database (psa) or the MySql server itself – only user data bases. While EXTREME caution must be exercised when accessing the Plesk database (as it easy to break something in Plesk), there are times when this tool is really needed (e.g. to immediately reset a disconnected admin session instead of waiting for the automatic reset, or checking passwords you might have forgotten and can’t normally view). This package installs and configures a copy of PhpMyAdmin that runs in a protected SSL directory (and so is not only secure, but you don’t have to worry about your password being sniffed when you login). After installation, PhpMyAdmin can manage any MySql database on the server.

 

Qmail restart

 

Some users have problems, on busy servers, with qmail "hanging" - messages are no longer sent, and new messages aren't received. This update consists of a number of changes to "reconfigure" qmail to better handle heavy loads so that these problems are less frequent, and a monitoring program to make sure qmail is processing messages on a timely basis. If messages are not being processed, qmail is restarted in such a way the processing resumes. A log file is kept of the last 24 hours of performance so the server admin can keep a close eye on qmail.

 

Apache Restart Control

 

If changes are made to the Apache web server on ANY server running Plesk, Apache must be restarted to incorporate these changes. On a busy server with many resellers, Apache can spend more time restarting than it does serving pages. The frequent restarts also interrupt http downloads which is inconvenient for site users.

 

The Apache Restart Control changes the behavior in several ways:

FTP Downloading

 

Anonymous FTP can be an excellent and efficient approach to providing visitors access to large files from your site. If you use Plesk, it's very easy to control the number of users and the bandwidth for each user so you can control bandwidth costs. But anonymous FTP is subject to a lot of abuse, and if you want to use it you should take some steps to control the abuse. At MIS Outsource we have several approaches to this, but one of the more effective is how we can configure your server to deal with users of "download managers". A download manager attempts to establish multiple connections to your server to thwart your attempts to limit bandwidth - at the client's computer data downloaded from these multiple connections can be recombined to create a complete file. You can limit the number of simultaneous connections from any single user, but the download manager will just fill up your log files with rejected connection attempts. We have software that will detect this, and disable FTP access to your server on a per-IP basis to any abusive user of download managers. This is a unique and proprietary solution from MIS Outsource.