To get the current day statistics simply run the following:

• sar — CPU statistics
• sar -r — RAM statistics
• sar -b — I/O statistics
• sar -n DEV — network devices statistics
• sar -A — all available statistics!

If you want historical data, use sar with the following switches:

• -s to define starting time of date
• -e to define ending time of date
• -f /var/log/sa01 to use the data for the first day of the current month

Introduction

Linux implements discretionary access control. The controls are discretionary (voluntary and selective) in the sense that a subject is capable of transferring permissions to other subjects – for example, a process can change security properties of resources. Main features of this traditional access control in Linux are:

• processes are running under UID and GID (real and effective)
• object access is based on ownership and permissions
• users define access control for their objects

The kernel has hard-coded access policy. The main problem is that the access is based on users’ access. Kernel can’t distinguish users from processes. For example, a compromised web browser can read all of the user’s files, including ssh private keys!  A compromised email client can make the users email files world readable. Furthermore, root user can bypass all the restrictions and it’s not easy to set minimum privileges.

To address these problems NSA initially developed SELinux (Security-Enchanced Linux) which is a set of kernel modifications to provide mandatory access control to Linux. SELinux is now developed mostly by RedHat (Dan Walsh is the lead SELinux developer).

So basically SELinux provides an additional security layer:

• processes and resources have new security contexts
• policy defines relationship between those contexts
• users can’t bypass the policy

Features of SELinux are:

• it is mandatory
• it is modular
• it is granular
• there is a possibility to define minimum privileges
• root isn’t almighty
• it is transparent for applications

Benefits of SELinux are:

• access is based on user and application function
• processes run with least-privilege
• privilege escalation tightly controlled (compromise of Apache is limited by the policy)

Three forms of access control

There are 3 forms of access control:

1. Type enforcement (primary mechanism)
2. Role-based access control
3. Multi-level security

Access control is configurable via policy language (targeted, strict, mls). SELinux follows the model of least-privilege: access is denied by default.

Type enforcement overview

Security contexts are stored in extended file system attributes (xattr, metadata). Security context is assigned by labeling of the file system. Look at the following security contexts for httpd process and /var/www/html directory:

$ sudo ps -Z -C httpd | head -2
LABEL                             PID TTY          TIME CMD
system_u:system_r:httpd_t:s0     1404 ?        00:00:00 httpd
$ sudo ls -dZ /var/www/html/
drwxr-xr-x. root root system_u:object_r:httpd_sys_content_t:s0 /var/www/html/

‘system_u:system_r:httpd_t:s0′ and ‘system_u:object_r:httpd_sys_content_t:s0′ are security contexts. The fields represent user:role:type:level respectively. For type enforcement the key field is type. Access is allowed solely by type. The policy defines that httpd_t can read httpd_sys_content_t, while everything else is denied.

Files and directories are assigned type by the configuration file which specifies default context. New files in folders inherit the security contexts of directories. Process types are inherited from the parent process or are set by policy (type transition rule) or by application. Transition must be allowed by policy – e.g. apache cannot start processes in init_t.

Targeted policy

Targeted policy is default in RHEL. It is important to have in mind that with targeted policy processes are unconfined by default – the policy confines only targeted services. Unconfined processes behave almost the same as SELinux is disabled. By default user processes run in unconfined_t, while system processes run in initrc_t domain. Transition of unconfined_t into the type that is defined by the policy is automatic upon starting up of the process.

SELinux configuration

SELinux stores its configuration files in /etc/selinux:

$ ls -l /etc/selinux/
total 20
-rw-r--r--. 1 root root  449 Apr 28 19:05 config
-rw-r--r--. 1 root root  113 Feb 14 21:49 restorecond.conf
-rw-r--r--. 1 root root   76 Feb 14 21:49 restorecond_user.conf
-rw-r--r--. 1 root root 1766 Dec  7 23:44 semanage.conf
drwxr-xr-x. 5 root root 4096 Apr 28 19:11 targeted

/etc/selinux/config defines enforcing mode and policy type and /etc/sysconfig/selinux is a symlink to this file on RHEL systems. /etc/selinux/targeted directory contains targeted policies and following subdirectories:

• contexts – object context definitions
• modules – modules used to build the policy
• policy – compiled (binary) policy file

The /etc/selinux/targeted/contexts/files includes the following:

• file_contexts – basic contexts
• file_contexts.local – local rewrites of contexts
• file_contexts.homedir – user files
• media – removable devices

There are man pages for SELinux policy documentation for several confined services:

$ man -k _selinux
abrt_selinux         (8)  - Security-Enhanced Linux Policy for the ABRT daemon
ftpd_selinux         (8)  - Security-Enhanced Linux policy for ftp daemons
git_selinux          (8)  - Security Enhanced Linux Policy for the Git daemon
httpd_selinux        (8)  - Security Enhanced Linux Policy for the httpd daemon
kerberos_selinux     (8)  - Security Enhanced Linux Policy for Kerberos
mysql_selinux        (8)  - Security-Enhanced Linux Policy for the MySQL daemon
named_selinux        (8)  - Security Enhanced Linux Policy for the Internet Name server (named) daemon
nfs_selinux          (8)  - Security Enhanced Linux Policy for NFS
pam_selinux          (8)  - PAM module to set the default security context
rsync_selinux        (8)  - Security Enhanced Linux Policy for the rsync daemon
samba_selinux        (8)  - Security Enhanced Linux Policy for Samba
squid_selinux        (8)  - Security-Enhanced Linux Policy for the squid daemon
ypbind_selinux       (8)  - Security Enhanced Linux Policy for NIS

Packages and utilities

Several standard utilities are modified to support SELinux:

• ls -Z
• id -Z
• ps auxZ
• lsof -Z
• netstat -Z
• find / -context

Other modified programs are:

• Login programs, PAM (sshd, login, xdm)
• Password utilites (passwd, useradd, groupadd)
• rpm

Backup and disk management support:

• tar, zip (both have extended attribute support)
• rsync (-X, -xattrs)
• star
• amanda
• tar –selinux
• tar xv | restorecon -f – (good option)

libselinux package:

• getenforce – check whether machine is enforcing or permissive
• setenfroce 1/0 – sets the machine in enforcing/permissive
• selinuxenabled – used in scripts
• matchpathcon – shows the default context of file/directory
• avcstat – display SELinux AVC statistics

coreutils package:

• chcon – changes security context of the file

policycoreutils package:

• sestatus – shows SELinux status
• semodule – loads the module into the policy
• setsebool – sets booleans
• genhomedircon – generates file_contexts. homedir
• setfiles – executes initial file system labeling
• restorecon – sets default context defined by the policy
• fixfiles – correct contexts database on file systems

checkpolicy package:

• checkmodule – compiles module
• checkpolicy – compiles policy

policycoreutils-python package:

• semanage – makes queries and changes the policy
• audit2allow – generates the rules based on activity logs

policycoreutils-gui package:

• system-config-selinux – SELinux administration
• selinux-polgengui – wizard for making new policies

setroubleshoot  and setroubleshoot-server packages:

• setroubleshootd – searches and processes AVC messages
• sealert – shows description of AVC messages
• seapplet – pops SELinux realted messages to the user in X

File labeling

File context can be changed with the fundamental utility chcon (which is similar to chmod):

$ chcon -R -t httpd_sys_content_t /srv/www

This change will be active until next relabeling. restorecon command sets contexts that are defined by policy rules. If there is /.autorelabel file, the file system will be relabeled during the next system boot. To make the change permanent use semanage utility to set the context.

The policy defines contexts in /etc/selinux/targeted/contexts/files/file_contexts file, by using regular expressions to indicate the paths. “–” in the second column indicates regular files, “-d”directories, “-b” block devices, “-c” character devices, “-l” links, etc. Special context “<<none>>” indicates that the context should not be changed.

Log files

SELinux messages are in the form of AVC – access vector cache. You can usually find them in /var/log/messages or /var/log/audit/audit.log (if auditd is running):

May  1 19:46:55 www kernel: type=1400 audit(1335901615.836:28093): avc:  denied  { open } for  pid=13595 comm="clamscan" name="parts" dev=xvdj ino=7823 scontext=system_u:system_r:clamscan_t:s0 tcontext=system_u:object_r:amavis_var_lib_t:s0 tclass=dir
May  1 19:46:55 www kernel: type=1400 audit(1335901615.837:28094): avc:  denied  { getattr } for  pid=13595 comm="clamscan" path="/var/amavis/tmp/amavis-20120501T141957-11544/parts/p002" dev=xvdj ino=7822 scontext=system_u:system_r:clamscan_t:s0 tcontext=system_u:object_r:amavis_var_lib_t:s0 tclass=file
May  1 19:46:55 www kernel: type=1400 audit(1335901615.837:28095): avc:  denied  { read } for  pid=13595 comm="clamscan" name="p002" dev=xvdj ino=7822 scontext=system_u:system_r:clamscan_t:s0 tcontext=system_u:object_r:amavis_var_lib_t:s0 tclass=file
May  1 19:46:55 www kernel: type=1400 audit(1335901615.837:28096): avc:  denied  { open } for  pid=13595 comm="clamscan" name="p002" dev=xvdj ino=7822 scontext=system_u:system_r:clamscan_t:s0 tcontext=system_u:object_r:amavis_var_lib_t:s0 tclass=file
May  1 19:46:55 www kernel: type=1400 audit(1335901615.868:28097): avc:  denied  { write } for  pid=13595 comm="clamscan" name="tmp" dev=xvdj ino=7515 scontext=system_u:system_r:clamscan_t:s0 tcontext=system_u:object_r:amavis_var_lib_t:s0 tclass=dir
May  1 19:46:55 www kernel: type=1400 audit(1335901615.868:28098): avc:  denied  { add_name } for  pid=13595 comm="clamscan" name="clamav-2e86f23064829c5cae67d7e8bc85d569" scontext=system_u:system_r:clamscan_t:s0 tcontext=system_u:object_r:amavis_var_lib_t:s0 tclass=dir
May  1 19:46:55 www kernel: type=1400 audit(1335901615.868:28099): avc:  denied  { create } for  pid=13595 comm="clamscan" name="clamav-2e86f23064829c5cae67d7e8bc85d569" scontext=system_u:system_r:clamscan_t:s0 tcontext=system_u:object_r:amavis_var_lib_t:s0 tclass=dir

AVC messages appear because of the:

• mislabeled files (file_t, default_t, mv’ed files, solution: restorecon)
• process running under the wrong context
• bug in policy
• intrusion

Explanation of AVC messages can be shown by using the sealert command. If you don’t have the sealert command, install the following package:

$ sudo yum install setroubleshoot-server

setroubleshootd analyzes the log messages and can send email alerts. The configuration for it is at /etc/setroubleshoot/setroubleshoot.conf. sealert is a setroubleshoot client that is used to diagnose SELinux denials and attempts to provide user friendly explanations and recommendations for how one might adjust the system to prevent the denial in the future.

AVC messages could also be suppressed by the policy rule dontaudit. Sometimes this could be problematic when troubleshooting and to disable dontaudit rules run the following command:

$ sudo semanage dontaudit off

Another useful tools for troubleshooting log files can be found in policycoreutils-python package. audit2allow generates policy allow rules from logs of denied operations. Let’s see whats the problem with clamscan on my system:

$ sudo grep clamscan /var/log/messages | audit2allow 

#============= clamscan_t ==============
#!!!! The source type 'clamscan_t' can write to a 'dir' of the following type:
# clamscan_tmp_t

allow clamscan_t amavis_var_lib_t:dir { write search read create open getattr add_name };
allow clamscan_t amavis_var_lib_t:file { read getattr open };

This tells me that clamav policy is missing allow rules for allowing clamscan to access amavis’ directory to scan the files. I will use this specification to compile policy module:

$ sudo grep clamscan /var/log/messages | audit2allow -M clamscan
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i clamscan.pp

audit2allow created type enforcement file clamscan.te and clamscan.pp policy package. Next step involves loading the policy package file:

$ sudo semodule -vi clamscan.pp
Attempting to install module 'clamscan.pp':
Ok: return value of 0.
Committing changes:
Killed

Whoops what was that? It turns out that the kernel killed the process because it made the system ran out of memory:

May  2 12:56:57 www kernel: Out of memory: Kill process 3375 (semodule) score 312 or sacrifice child
May  2 12:56:57 www kernel: Killed process 3375, UID 0, (semodule) total-vm:309008kB, anon-rss:292076kB, file-rss:104kB

I freed some of the reserved memory and compiled the module successfully:

$ sudo semodule -l | grep clamscan
clamscan	1.0

This change is permanent and survives a restart. If I ever wanted to remove that policy I would execute the following:

$ sudo semodule -r clamscan

Policy customization

Policies usually have booleans associated that enable run time customization of the policy. Booleans are if/then/else statements in the policy.

• getsebool – shows variable value
• togglesebool – changes variable value
• setsebool – by using -P parameter you can change the value permanently

Remember to check the particular policy man page for variables description.

semanage utility is pretty important and it can list and change policy parameters:

• semanage user -l — list SELinux users
• semanage port -l –list SELinux ports
• semanage fcontext -a -t httpd_sys_content_t “/web(/.*)?” — allow Apache to serve content from /web
• semanage port -a -t http_port_t -p tcp 81 — allow Apache to listen on non-standard port 81
• semanage permissive -a httpd_t — disable httpd protection
• semanage permissive -d httpd_t — enable httpd protection

Apache httpd

Versions

As of 21th February 2012 the new major release of Apache httpd is 2.4, while the previous one was 2.2. There are numerous welcoming enhancements in the latest major version, many of them being well suited for cloud environment. The primary goal of new release is to deliver more performance, directly aiming at nginx (which has the reputation of providing better performance and lower memory footprint).

New features include:

• Improved performance (lower resource utilization and better concurrency)
• Reduced memory usage
• Dynamic reverse proxy configuration
• Performance on par, or better, than pure event-driven Web servers
• More granular timeout and rate/resource limiting capability
• More finely-tuned caching support, tailored for high traffic servers and proxies
• Dynamic loadable MPMs
• mod_ssl with OCSP support
• Event MPM is now fully supported

For the overview of all of the new features visit this link.

Architecture

Instead of implementing a single architecture, Apache provides several Multi-Processing Modules (MPMs) which allow Apache to run in a process based, hybrid based (processes and threads) and event based modes, to better match different demands. These modules are responsible for basic web server operation: binding to network ports on the machine, accepting requests and dispatching children to handle requests.

In the following paragraphs different Apache MPMs that are available for GNU/Linux are described.

prefork – safer choice, traditional, non-threaded, forking

A single control process is responsible for spawning child processes which listen for connections and serve them when they arrive. The control process is started as root to bind to port 80, while child processes are launched as less-privileged users (User and Group directives). Apache always tries to maintain a number of spare (idle) processes which stand ready to serve incoming requests.  Prefork is the best choice if Apache has to use non-thread-safe libraries (such as mod_php) and it is ideal for request (process) isolation. It is process per request model.

worker – hybrid multi-process, multi-threaded

A single process is responsible for spawning child processes. Each child process creates a fixed number of threads, as well as listener thread which listens for connections and passes them to server thread for processing. Basically worker uses threads to serve requests so it is being able to serve a large number of request by using less system resources than a prefork, process-based web server. It retains much of the stability of the process-based server by maintaining multiple processes available, each with many threads. Apache always tries to maintain a pool of spare or idle server threads, which stand ready to serve incoming requests. The main process is started as root, while the child processes and threads are launched as a less-privileged user (User and Group directives). Though, keep in mind that PHP’s thread safety is highly disputed.

event – consuming threads only for connections with active processing, based on worker

Event MPM is designed to allow more requests to be served simultaneously by passing some processing work to supporting threads, freeing up the main threads to work on new processes. Event MPM is experimental in 2.2, but stable and supported in 2.4.

I will also mention two more MPMs that are written to address the privilege separation problem (running virtual hosts under different UID and GID): mpm-itk and mpm-peruser. I won’t go into describing these as each of these have their own shortcomings, such as peruser MPM is considered not to be production ready, while mpm-itk processes request headers as root, switches to the target UID, and then kills the httpd process when finished serving the connection. Processing headers as root is very dangerous and opens the web server up to many potential security problems.

So to conclude MPMs – prefork is a safer choice, while threading should have better performance with less overhead (should be more effective within multiprocessor environments).

Apache allows the use of filters to process incoming or outgoing data in a configurable manner. Some examples of filter modules include mod_ssl for https, mod_deflate for compression/decompression on the fly and mod_ext_filter for running external programs as filters.

Apache also uses handlers which define action to be performed when a file is called. Handlers may be configured based on either file type, filename extensions or on location, without relation to file type. Some types of built-in handlers are server-status and server-info.

Installation and optimization

Since I want everything to install quickly and easily for the purpose of this article, I will not go for the newest version, but will instead install httpd version 2.2.15 from CentOS 6.2 repositories:

[pawwa@www2 ~]$ sudo yum install httpd

Query the set of the modules that are compiled directly into the binary:

[pawwa@www2 ~]$ httpd -l
Compiled in modules:
  core.c
  prefork.c
  http_core.c
  mod_so.c

The httpd version from the repositories has the common prefork MPM compiled in. If we needed the worker MPM we would have to compile httpd from source with enabled support for the worker module.

Next, ensure that the service starts on boot:

[pawwa@www2 ~]$ chkconfig --list httpd
httpd          	0:off	1:off	2:off	3:off	4:off	5:off	6:off
[pawwa@www2 ~]$ sudo chkconfig --level 35 httpd on
[pawwa@www2 ~]$ chkconfig --list httpd
httpd          	0:off	1:off	2:off	3:on	4:off	5:on	6:off

Start the service with the default configuration and check its memory footprint and number of processes:

[pawwa@www2 ~]$ sudo service httpd start
Starting httpd:                                            [  OK  ]
[pawwa@www2 ~]$ sudo netstat -tlnp | grep httpd
tcp        0      0 :::80                       :::*                        LISTEN      4408/httpd
[pawwa@www2 ~]$ sudo ps -C httpd -O user,ppid,vsz,rss --forest
  PID USER      PPID    VSZ   RSS S TTY          TIME COMMAND
 4408 root         1 175524  3796 S ?        00:00:00 /usr/sbin/httpd
 4410 apache    4408 175524  2412 S ?        00:00:00  \_ /usr/sbin/httpd
 4411 apache    4408 175524  2412 S ?        00:00:00  \_ /usr/sbin/httpd
 4412 apache    4408 175524  2412 S ?        00:00:00  \_ /usr/sbin/httpd
 4413 apache    4408 175524  2412 S ?        00:00:00  \_ /usr/sbin/httpd
 4414 apache    4408 175524  2412 S ?        00:00:00  \_ /usr/sbin/httpd
 4415 apache    4408 175524  2412 S ?        00:00:00  \_ /usr/sbin/httpd
 4416 apache    4408 175524  2412 S ?        00:00:00  \_ /usr/sbin/httpd
 4417 apache    4408 175524  2412 S ?        00:00:00  \_ /usr/sbin/httpd

This confirms that we are using prefork MPM: there is one parent httpd process that is listening on http port which launched several child processes. 8 server processes are started because of the ‘StartServers 8′ directive in httpd.conf. The listing shows virtual memory size of 175524 KB (VSZ is entire virtual memory of a process, pretty much irrelevant, VmLib + VmExe + VmData + VmStk) and resident set size of 2413 of individual child processes (RSS is the non-swapped physical memory that a task has used,  including code, data and stack segments). Keep in mind that GNU/Linux ps utility does not report the real memory usage of the process, only the approximation, since it assumes that the single process is the only one running on the system, while there are of course dozen of running processes at any given time that share single copies of referenced libraries (such as libc). The more realistic representation of memory usage can be acquired with pmap -d PID command but its output is somewhat hard to interpret. To make it short, “r-x” are code segments, while “rw-” are data segments. If you factor out the shared libraries code segments, you end up with wirtable/private figure that is shown at the end of the pmap output, which in my example shows 1996K for httpd child process:

[pawwa@www2 ~]$ sudo pmap -d 4417 | tail -1
mapped: 175524K    writeable/private: 1996K    shared: 580K

This is different and less than approximated 2412K reported by ps. This information becomes crucial for configuring MaxClients directive which determines how many concurrent requests the web server can actually handle. Another trick to get detailed stats is to check  /proc/PID/status of a process.

Next step is to comment LoadModule directives in httpd.conf for the dynamicly shared modules (DSOs) that are not being used. When I have cleaned my configuration out of unnecessary modules I reduced the size of a single httpd child process by ~500K. Here’s a dump of modules at this point:

[pawwa@www2 ~]$ httpd -M
Loaded Modules:
 core_module (static)
 mpm_prefork_module (static)
 http_module (static)
 so_module (static)
 auth_basic_module (shared)
 authn_file_module (shared)
 authn_default_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 authz_groupfile_module (shared)
 authz_default_module (shared)
 log_config_module (shared)
 logio_module (shared)
 setenvif_module (shared)
 mime_module (shared)
 autoindex_module (shared)
 negotiation_module (shared)
 dir_module (shared)
 actions_module (shared)
 alias_module (shared)
Syntax OK

Later on I will be adding modules such as mod_rewrite, mod_deflate and mod_ssl, which will increase httpd’s memory footprint for about 500K.

Next, move-on to prefork MPM optimization. The most important directive is MaxClients, which defines the number of simultaneous requests that can be served. It defines the hard limit on the number of running child httpd processes. The default value of 256 sounds like a lot and should be reduced on smaller systems. A too high setting can pre-fork a lot of httpd processes which could use all of the servers available memory. This could cause the system to come to an OOM state and start thrashing (performing more memory management and paging in and out then any real work). One should furthermore carefully tune this parameter if the server is running DBMS and any other subsystems. The more memory you leave for the operating system, the more space it will have to perform file system caching. Choosing values for MaxClients involves some trial and error, and measuring number of processes at peak times. Basically, the figure could be calculated as:

( total memory – operating system memory – DBMS memory ) / memory footprint of httpd

Eliminate all the extra checks the Web server must do. Some well known optimization techniques are to disable hostname lookups, enable FollowSymLinks (saves a lot on disk activity), disable .htaccess checks on every directory (move .htaccess configuration to httpd.conf whenever possible), avoid wild cards as parts of directives, experiment on keep alive values for handling persistent connections, and similar.

From the hardware perspective, RAM is crucial for performance (caching, mod_mem_cache for dynamic content), while heavily dynamic websites have higher CPU requirements. RAID arrays with striping modes of operation increase file serving performance.

PHP: Hypertext Preprocessor

PHP is a general purpose scripting language that is most frequently embedded into HTML. Ultimately, the code is interpreted by a web server containing a PHP processor module which generates a web page.

Versions

PHP 5 is the current generation of PHP. At the time of writing this article the supported versions are 5.3.10 and 5.4.0. One can run several versions on a single system and contain debug builds.

PHP handling

There are different ways to integrate PHP with Apache httpd. Each handler affects web server performance and features that can be used by employing them. The basics of a few popular PHP handlers are presented below.

mod_php runs as Apache DSO module. This means excellent performance, but not so flexible, and poor security as all PHP code is run as the user that has launched httpd process (defined with the User directive within httpd.conf). This is the most usual way to use PHP tough.

suPHP runs as a CGI module, but executes scripts as the user who owns them. It consists of an Apache module (mod_suphp) and a setuid root binary (suphp) that is called by the Apache module to change the uid of the process executing the PHP interpreter. It is more secure as the scripts not owned by a particular user will not be executable. Also, the files that have permissions set to world writable will likewise be non-executable. You can have a custom php.ini per site and run PHP 4 and PHP 5 at the same time. The drawback is that suPHP runs higher on CPU making it a slower solution than mod_php (mod_php is about 20-30% faster) and it does not support opcode caching.

FastCGI is high performance CGI. It has the security benefits of user separation, very good performance and support for opcode caching. The caveat: high memory usage. This is because rather then creating a process per PHP request, it keeps a persistent session open in the background. One benefit is that you can have FastCGI running on different machine.

PHP-FPM (FastCGI Process Manager) is an alternative PHP FastCGI implementation with some additional features useful for sites of any size, especially busier sites. It is good in security and performance terms, but I am interested in the stability.

There is also mod_ruid2, which isn’t a PHP handler but can work with one (except with FastCGI). It is an Apache extension that allows all requests to a domain to run as the owner of that domain. It is usually deployed in conjunction with mod_php to leverage security of user separation through POSIX capabilities, while having excellent performance. It also supports opcode caching, as oposed to suPHP, but allows only one php.ini for all web sites. I am interested in how stable this module really is, as one benchmark marked it as bad in terms of stability.

In terms of security, almost all of these modules are subject to a server compromise if Apache was hacked, and because of that I would recommend employing additional security layers, such as mandatory access control mechanisms.

Installation

To deploy PHP to prefork-based Apache for a VPS that is tight on CPU and RAM, I choose mod_php and mod_ruid2 for user-based security. Follow installation instructions from mod_ruid2′s README file, while a simple ‘yum install php’ will install mod_php and all the necessary files such as php.ini, few PHP modules, php.conf configuration for httpd and such.

If you need additional PHP modules for your applications, search for their corresponding package names by executing ‘yum search php. module’. For example, if you will be connecting to MySQL database from your PHP scripts you will need to install php-mysql package. Keep in mind that you should install only the modules you will actually need!

PHP files don’t need to be executable, since they are handled by the module directly (AddHandler Apache directive). The php.ini configuration file is read when PHP starts up, which happens only once  – when the web server has started. Remember to copy the recommended php.ini-production file over /etc/php.ini for production environment. You can set PHP configuration directives in httpd.conf and .htaccess files also as far as you have ‘AllowOverride Options’ or ‘AllowOverride All’ privileges.

Keep an eye on the following PHP resource settings:

• max_execution_time – how much CPU seconds a script can use (30)
• max_input_time – How long (seconds) a script can wait for input data (60)
• memory_limit – How much memory (bytes) a script can consume before being killed (32M)
• output_buffering – How much data (bytes) to buffer before sending out to the client (4096)

Since my VPS has a small amount of RAM I will not employ opcode caching. If you have more RAM, install opcode cache such as eAccelerator, APC or ionCube Accelerator. Opcode is a binary representation of the code to be executed. An opcode cache saves compiled opcode and reuses it the next time the page is called. This saves a considerable amount of time.

MySQL

MySQL is a popular database for use in web applications. It is also used in many large-scale web sites such as wikipedia, google, facebook, twitter, youtube and others.

Versions

Versions 5.0.x and below are no longer actively developed. Current stable release is 5.5.23. As of version 5.1 there are two offerings that have a common code base: the open source MySQL Community Server and commercial MySQL Enterprise Server.

Since Sun Microsystems was acquired by Oracle, MySQL database had gotten a community-developed fork under the name of MariaDB. The aim of the project is to provide a drop-in replacement to MySQL while being licensed under GNU/GPL.

Features and database engines

MySQL had grown into a full featured RDBMS. Among other features, it supports transactions, stored procedures, triggers, SSL, query caching, ACID compliance, replication, clustering, … It supports several different storage engines which provide CRUD (create, read, update, delete) functions on the database. The most popular ones are:

MyISAM – for read speed, but no transactions. Table-level locking (performance can suffer with high profile applications). Full-text indexes and searches. They crash relatively often. In the case of a crash, it has to rescan whole indexes and possibly tables to recover. There is a myisamchk utility to repair the database in the case of data corruption, but it is not guaranteed to work. Many hosting providers only support MyISAM.  Each table is presented as three files on disk with the following extensions: frm (definition), MYD (data) and MYI (index data).

InnoDB – default as of 5.5, ACID compliant transactional features, referential integrity through foreign keys, higher concurrency. Row-level locking. In the case of a crash it recovers faster from the transactional log.

Installation and tuning

Install mysql server and client packages:

[pawwa@www2 ~]$ sudo yum install mysql mysql-server

Start mysqld service and run mysql_secure_installation to set the root password, remove test database and anonymous users:

[pawwa@www2 ~]$ sudo mysql_secure_installation

If the goal of database tuning is to reduce the memory footprint of the database, eliminating various buffers will certainly help, at the expense of query speed and application performance. Instead, one of the metrics should be application response time, which opens up tuning possibilities other than just the database’s memory usage. Always look at optimizing your queries first though – the most dramatic benefits usually come from proper indexing and carefully written queries.

Here are some parameters to tune:

• key_buffer_size – the most useful single variable to tweak (some rough suggestions are to set it to at least a quoter of available memory). The larger you set it, the more of your MyISAM table indexes you store in memory.
• innodb_buffer_pool_size – while the key_buffer_size is the variable to target for MyISAM tables, for InnoDB tables it is innodb_buffer_pool_size.
• table_open_cache – each time MySQL accesses a table, it places it in the cache. If your system accesses many tables, it is faster to have these in the cache. A good way to see whether your system needs to increase this is to examine the value of open_tables at peak times. Variables to watch are Open_tables and Opened_tables.
• sort_buffer – it can be useful if performing large numbers of sorts.
• read_rnd_buffer_size – if you use many queries with ORDER BY, increasing this parameter can improve performance.
• tmp_table_size – This variable determines the maximum size for a temporary table in memory. If the table becomes too large, a MyISAM table is created on disk. Try to avoid temporary tables by optimizing the queries where possible, but where this is not possible, try to ensure temporary tables are always stored in memory. Watching the processlist for queries with temporary tables that take too long to resolve can give you an early warning that tmp_table_size needs to be increased.
• …

This is the end of this longish article As a final tip remember that constant measurement of performance really helps in tuning and optimizing. Watch CPU, IO, bandwidth, etc. Spot trends. React.

I had the “fortunate” opportunity of becoming a student in early 2000s – a period in which Bologna Process was forming academic degree standards in Europe which is today known as Bologna Declaration document. When I graduated college in year 2007, I was expecting a degree that was promised by the school – a bachelor’s degree, but I didn’t get one as Bologna Process wasn’t finished. What I had got was an academic title of a Bachelor of Engineering that couldn’t be easily recognized in the rest of the world I suppose.

As in the meanwhile the Bologna Declaration was signed, I decided to claim the bachelor’s title that belongs to me. What I had to do was to put together an extra one more bachelor thesis document, so I decided to write about cloud computing. Although it is completely written in Serbian language, here’s an English abstract that describes the main goals so you can get the impression of what’s it about:

This paper describes the cloud computing model and technology.

The first section presents virtualization technology, whose introduction and wider acceptance enabled architecting of optimized data centers which can nowadays support the cloud computing model. In addition to general definitions, terms and types of virtualization, also being presented are a number of leading open-source virtualization projects.

The second chapter deals with the topic of cloud computing. It interprets the architecture, presents the characteristics and describes deployment and service models.

The last chapter presents a number of popular open-source software projects for implementing the infrastructure as a service cloud model. It also specifies the typical and general Service License Agreement conditions, and gives a comprehensive real-world implementation example on the case of Amazon Web Services.

If you can read in Serbian language feel free to download and check my work:

• Bachelor thesis: Cloud Computing Concepts and Technologies - main document
• Bachelor thesis: Cloud Computing Concepts and Technologies – a presentation

Note that the presentation PDF is more like a quick personal reminder than a real presentation (lots of small sized fonts ).

Wireless problems

I use cable broadband and a home WLAN for providing Internet access to the connecting clients, and thus wanted to perform a network installation via wireless connection. This is where it had all begun.

I know – Equi donati dentes non inspiciuntur, but this PCI wireless NIC that I had in the machine was an unsupported and notorious Broadcom chip. When the d-i (debian-installer) tried to load a driver for it, it failed, and the console output complained about missing firmware. This device needs proprietary firmware to be loaded onto the chip before it can operate. Ok, so I performed a couple of reboots to search the net for the right firmware for my WiFi device, and when I finally found one, d-i had loaded the firmware but it could not sense the link. I tried a couple of different firmwares I had found on the Internet, but without luck. Finally, I though my wireless card was dying.

Luckily enough, I have had another USB powered WiFi NIC. Since this was a RealTek, it had no problems with missing firmware, but again it could not sense the link. So another half an hour of troubleshooting, and I had found out what was the problem – the Debian installer does not support encrypted WPA! This was a shock. When I reconfigured my wireless router to use WEP, it worked.

Overlapping partitions problem

Debian installer could not recognize my existing and working partition table (this machine had a working setup consisted of Windows XP and Ubuntu). I tried several moves to resolve this, such as writing the partition table again with fdisk and similar, but at the end I had found that I had overlapping partitions problem. It was resolved by using a powerful and free data recovery utility testdisk.

Audio playing faster problem

On this machine I have a professional audio interface, E-MU 0404 PCI variant. The lack of driver support for it on Linux was the main reason why this machine wasn’t used primarily as a Linux box. Since ALSA did include the support for this interface, I wanted to give it a try. I installed the module (snd_emu10k1) and it worked out of the box but the playback was slightly faster This was resolved by changing the sampling rate to 48kHz by creating the file /etc/asound.conf with the following configuration:

pcm.!default {
        type plug
        slave {
                pcm "hw:0,0"
                rate 48000
        }
}

Aaaand, this machine was finally usable I am not saying that this is all Debian‘s fault of course (although WPA support during install time and some hints regarding partition table recognition would be nice), but I had just remembered how Linux installation used to be a pain in the ass a long time ago Yet this kind of working-my-way-through was sure one of the things that attracted me to GNU/Linux in the first place.

First, check lspci output for let’s say a wireless card:

$ lspci | grep -i wireless
01:00.0 Network controller: Atheros Communications Inc. AR928X Wireless Network Adapter (PCI-Express) (rev 01)

Note the number 01:00.0 in lspci output, in which 01 represents the bus number the device is attached to, 00 is the device number and final 0 is PCI device function. To get more information on that device we could list the entries in /sys/bus/pci/devices/0000:01:00.0 directory of sysfs (a RAM file system that exports kernel structures, attributes and their inner links. udev uses sysfs also to create dynamic device files). lspci actually reads sysfs.

Next, we print raw (-n option) PCI identification data that includes the previous numbers:

$ lspci -n | grep 01:00.0
01:00.0 0280: 168c:002a (rev 01)

These numbers are from /usr/share/misc/pci.ids file. Lets brake down the numbers:

• 01:00.0 – 01 is bus number, 00 device number, 0 device function
• 0280 – device class
• 168c – vendor ID
• 002a – device ID

We use the vendor ID and device ID numbers and compare them to a modinfo of kernel drivers:

$ find /lib/modules/$(uname -r)/kernel/drivers -type f -exec modinfo "{}" \; | grep -B 200 -i 168c | grep -B 50 -i 002a | grep filename
filename:       /lib/modules/2.6.31-23-generic/kernel/drivers/net/wireless/ath/ath5k/ath5k.ko
filename:       /lib/modules/2.6.31-23-generic/kernel/drivers/net/wireless/ath/ath9k/ath9k.ko

So, my wireless card is supported by the ath9k driver. If I didn’t got the output from modinfo, it would probably mean that the hardware is not supported and that I need to get the driver from the device vendor. I could also search only for the vendor ID to get some results but that could yield some unexpected results I suppose.

No mater if you have a 19″ rack, some servers might not fit in. There is a standard named EIA-310-D which defines “standard rack” and specifies design features for 19″ racks.

The problem is that this standard does not define some details such as how deep is rack’s mounting width, rack holes (threaded, rounded, square…). For example, rack holes are the number one problem for server and rack incompatibility. Things should be good if you have square holes on your rack – you can always add threads with a cage nut if you need them.

In December of 1995 the EIA-310-D standard was updated. The changes made were mostly grammatical. The mechanical requirements were left unchanged. The updated documentation was originally known as EIA-310-E. It is currently referred to as EIA/ECA-310-E

So, be careful when choosing new servers for your racks, always double check the compatibility between them.

It is released under LGPL license and is MINEX-certified (fingerprint template interoperability standard). It’s written in C++ and can run on Linux, Windows, Android and some other operating systems. FingerJetFX runs well on embedded chips, desktop computers and servers. They say that it is easy to use – it can be used with as little as one function call.

More information @ http://digitalpersona.com/fingerjetfx

Introduction

The new server was meant to be placed in our DMZ and should run a few virtual machines providing services to external clients.

Some basic hardware requirements were:

• 1U rack mount
• 1 x Xeon based CPU
• 8 GB RAM
• 2 x 500 GB
• Hardware RAID
• 2 x Intel NIC, 1Gbps

The software and services which are planned to be installed or implemented:

• Debian Squeeze
• KVM
• MySQL
• NTP
• JBoss
• EJBCA
• VPN
• …

Some virtualization notes

There are different performance goals:

• Single guest performance
• Aggregate performance
• Density (as much guests running on single host)

Virtualization optimization depends on the needs. Experiment with transparent huge pages, try different IO schedulers, try hyperthreading… High performance virtualization is hard. Some things just cannot be emulated efficiently. Timekeeping has always been a virtualization headache. It is good to have a tickless kernel, and pvclock so guests can ask the host what time it is.

KVM

KVM (Kernel-based Virtual Machine) is virtualization hypervisor. It is integrated in kernel, lightweight and great in performance. The host machines need to be running either Intel VT or AMD-V chipsets that support hardware-assisted virtualization.

Hardware

CPU

Since Xeon was on the requirements list, I was looking for ones with the following capabilities:

• VT-x
• Hyperthreading (?)
• ECC

From the Xeon-5500 series, probably the best model for virtualization is E5520 (2.26GHz) in price/performance estimation. We have chosen quad-core Intel® Xeon® E5620. It is a quad core CPU that supports VT-x, ECC memory and hyperthreading.

By the way, the main difference between some cheaper consumer CPU’s such as Intel’s I7 and enterprise targeted CPU’s such as Intel’s Xeon, is the support for ECC memory which is a must for mission critical applications.

VT

Nowadays, Virtualization Technology instruction set is implemented inside the CPU’s, which makes hypervisors simpler, thus providing better performance over software-only virtualization solutions.

Hyperthreading

Is Hyperthreading an advantage or can it have even a negative impact on the system? Because HT takes advantage of unknown variables (cache misses for example), it is hard to take advantage of HT. It seems that nobody has a real idea of how HT impacts on application performances. Depending on application internal architecture HT execution of threads can be a benefit or a real pain. Some suggest switching HT off.

Hyperthreading can reduce scheduling latencies, which reduces spinlock worst case overhead.

Memory

Registered on unbuffered?

In enterprise server systems, it is a question of registered or unbuffered memory modules. Registered (also called buffered) memory modules have a register between the DRAM modules and the system’s memory controller. They place less electrical load on the memory controller and allow single systems to remain stable with more memory modules than they would have otherwise.

The difference between registered memory and unbuffered memory is whether there are registers on the memory module. Almost all system memory in today’s PCs is unbuffered memory. For those who need to utilize more than 4GB of memory (maybe more like 16GB or 32GB) in a system, registered memory is absolutely a must-have. Registered memory is all about scalability and stability. A small performance hit is generally incurred.

ECC

ECC stands for Error Checking and Correction. ECC detects and corrects memory errors so it is highly advisable to use this type of modules in servers that utilize multi-gigabytes of memory and usually run 24/7, and have increased probability of soft errors.

Storage adapter (RAID)

Beware of fakeraid controllers! Check driver support for your operating system or support in Linux vanilla kernel, and decide whether to use battery backed cache. The cache memory in RAID controllers improves performance to some extent by storing information that was recently used, or that the controller predicts will be used in the future, so it can be supplied to the system at high speed if requested instead of necessitating reads from the slow hard disk platters. Battery backed cache is for data protection from unexpected power outage. In every case, the goal of the cache is the same: to provide a temporary storage area that allows a faster device to run without having to wait for a slower one.

One area where caching can impact performance significantly is write caching, sometimes also called write-back caching. When enabled, on a write, the controller tells the system that the write is complete as soon as the write enters the controller’s cache; the controller then “writes back” the data to the drives at a later time. The reason that write-back caching is so important with RAID is while writes are slightly slower than reads for a regular hard disk, for many RAID levels they are much slower.

Read performance under mirroring is far superior to write performance.

Chosen controller for our system is ServeRAID M5014 SAS/SATA Controllers because it is supported in Linux vanilla kernel (driver name is megaraid_sas), and it provides additional performance advantages of an adequate amount of cache (256MB) + we ordered a standard battery backup unit.

Hard drives

We have chosen 2 x IBM 500GB 7200 6Gbps NL SAS 2.5″ SFF Slim-HS HDD.

SAS vs. SATA

• SAS is full duplex
• SATA uses the ATA command set; SAS uses the SCSI command set
• SAS hardware allows multipath I/O to devices while SATA (prior to SATA 3Gb/s) does not
• SATA is more consumer, SAS targets critical server applications
• SAS error-recovery and error-reporting use SCSI commands which have more functionality than the ATA SMART commands used by SATA drives

Network adapters

We have chosen Intel Ethernet Dual Port Server Adapter I340-T2 for IBM System x, as it is based on 82580 chip that is supported by Linux in igb driver.:

# dpkg -S "igb.ko"
linux-image-2.6.31-19-generic: /lib/modules/2.6.31-19-generic/kernel/drivers/net/igb/igb.ko
linux-image-2.6.31-20-generic: /lib/modules/2.6.31-20-generic/kernel/drivers/net/igb/igb.ko
linux-image-2.6.31-14-generic: /lib/modules/2.6.31-14-generic/kernel/drivers/net/igb/igb.ko
linux-image-2.6.31-22-generic: /lib/modules/2.6.31-22-generic/kernel/drivers/net/igb/igb.ko
linux-image-2.6.31-23-generic: /lib/modules/2.6.31-23-generic/kernel/drivers/net/igb/igb.ko

...
01:00.0 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
        Subsystem: Intel Corporation Ethernet Server Adapter I340-T2
...

When choosing network adapter always be careful to look if the device is supported by Linux (for example, for some Broadcom NeXtreme NIC’s you don’t have a driver for Debian).

Software

Operating system

KVM supports both 32 and 64 bit guests. According to KVM’s guest list, Debian Squeeze is supported.

References

1. http://www.linux-kvm.org/page/Main_Page
2. http://www.linux-kvm.org/page/HOWTO
3. http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/index.jsp?topic=/liaai/kvminstall/liaaikvminstallstart.htm
4. http://searchservervirtualization.techtarget.com/answer/Hyperthreading-in-virtualized-environments
5. http://www.redhat.com/promo/summit/2010/presentations/summit/in-the-weeds/thurs/riel-420-kernel/summit2010-kvm-optimizations.pdf
6. http://www.pcguide.com/ref/hdd/perf/raid/conf/advCaching-c.html
7. http://blog.fastmail.fm/2009/10/19/ibm-x3550-m2-or-x3650-m2-and-debianubuntu/