Your Inner Sysadmin

Better Development Through Understanding Your Servers

Chris Tankersley

MadisonPHP 2014

Who Am I?

  • PHP Developer for around 10 years
  • Using Linux longer than I remember
  • Sysadmin professionaly for about 8 years

Traditional LAMP Stack

The Server

Where Stuff Is

  • /bin - Essential user executable files
  • /boot - Stuff that makes the OS boot up!
  • /dev - Special device stuff you probably won't touch
  • /etc - Configuration files
  • /home - User home directories
  • /sbin - System binaries
  • /usr - Multi-user apps and utilities
  • /var - Data usually lives here

Installing Software

  • Compile software from scratch
  • Use the package manager (yum/apt)

Learn to love the command line

Authentication and Authorization


You can give admin access to users (or groups of users) without giving them root.

# Add sudo access to a single user to run as root
dragonmantank ALL=(ALL) ALL

# Add sudo access to a full group
%admin ALL=(ALL) ALL

You can even restrict what commands the users can run

# Restrict web developers to only restart Apache and MySQL
%webdevs NOPASSWD:/usr/sbin/service apache2 restart, /usr/sbin/service mysql restart

Jailing Users

Keeps people from getting to things they shouldn't. Protects the users from themselves.

Jailed Shells

Gives users a full shell but not the entire file system. You can pick and choose what programs the user can have access too. Jailkit makes this incredibly easy to set up.

Jailed SFTP

Locks the user to a specific base path, but doesn't give them a shell, much like FTP. You get the security of SSH though! Does require a system user though.


If you do Docker the 'non-Docker' way

Scripting Languages


Most servers use bash as the default shell. Most shells understand bash's syntax. If you find yourself running the same commands over and over, throw it in a bash script.


Ships with most distros. Great for when you need more power than what bash has.


Leverage your PHP skills to write shell scripts.

Locking Down Your Code

Running Apache as a Different User


<IfModule mpm_itk_module>
    AssignUserId [user] [user]


RMode config
RUidGid myuser mygroup
RDocumentChRoot /var/www/vhosts/ www/public


user = myuser
group = mygroup
chroot = /path/to/my/chroot



Rotates logs out for organization (or other) reasons

rotate 4
include /etc/logrotate.d
/var/log/wtmp {
    minsize 1M
    create 0664 root utmp
    rotate 1


Script that runs every so often and scans a bunch of logs so you get a pretty e-mail with a summary of the events
--------------------- httpd Begin ------------------------

 0.17 MB transferred in 792 responses  (1xx 0, 2xx 786, 3xx 0, 4xx 6, 5xx 0)
    199 Content pages (0.09 MB),
    593 Other (0.09 MB)

 Requests with error response codes
    400 Bad Request
       / 1 Time(s)
    404 Not Found
       /MyAdmin/scripts/setup.php: 1 Time(s)
       /phpmyadmin/scripts/setup.php: 1 Time(s)
       / 1 Time(s)
       /webdav/: 2 Time(s)

 ---------------------- httpd End -------------------------


Actually a Host Intrusion Detection System, but it does this by watching logs. Will alert you immediately to problems, and even shut down attacks if it notices them!
OSSEC HIDS Notification.
2012 Oct 24 11:38:10

Received From: maple->/var/log/auth.log
Rule: 5712 fired (level 10) -> "SSHD brute force trying to get access to the system."
Portion of the log(s):

Oct 24 11:38:09 maple sshd[1062]: Failed password for invalid user alias from port 59988 ssh2
Oct 24 11:38:07 maple sshd[1062]: Invalid user alias from
Oct 24 11:38:06 maple sshd[1059]: Failed password for invalid user recruit from port 59884 ssh2
Oct 24 11:38:04 maple sshd[1059]: Invalid user recruit from
Oct 24 11:38:03 maple sshd[1057]: Failed password for invalid user sales from port 59763 ssh2
Oct 24 11:38:00 maple sshd[1057]: Invalid user sales from
Oct 24 11:37:59 maple sshd[1055]: Failed password for invalid user staff from port 59661 ssh2
Oct 24 11:37:58 maple sshd[1055]: Invalid user staff from

Preventing Intruders


Set of files to allow or deny access to the machine or certain apps/ports on the machine


A firewall that is generally available on Linux machines that can be configured many different ways to allow or block or mangle traffic


IDS that was logs and will use hosts.deny and iptables to block stuff automatically for you!

Configuration Management

Process by which you figure out what goes on your servers, how you want them set up, and keeping track of that information. Files are usually stored in source control on one server and pushed to clients.

Why do you need it?

  • Ever needed to keep track of when files get changed?
  • Ever needed to roll back a change?
  • Ever needed to push the same change to a bunch of servers
  • Ever needed to set up a server exactly the same way as another server?


Sample Manifest File

user { 'ctankersley':
        ensure => present,
package {'postfix':
        ensure => present,
node '' {
        $mysql_rootpw = 'notreallytherootpassword'
        include mysql::server
        include mysql::client

        class {'apache':}
        class {'dotdeb':}
        class {'siteadmin':}

        include monit
                pidfile => '/var/run/mysqld/',
                start   => '/etc/init.d/mysql start',
                stop    => '/etc/init.d/mysql stop',

Server Monitoring

Is your server up right now? Is the web server and the database services up and running?

Host Monitoring


Service Monitoring


Software Tools!

Lots of software on the server to help with development


Command line multiplexer


Watch log files as they are generated


Command line program to for transferring data via a URL.
# Download a single file
$ curl

# Gives nify stats
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 66 1215k   66  805k    0     0  33060      0  0:00:37  0:00:24  0:00:13 45900
100 1215k  100 1215k    0     0  39474      0  0:00:31  0:00:31 --:--:-- 68987

# Download multiple files
$ curl -O -O

# View headers!
$ curl -v
Date: Wed, 24 Oct 2012 19:36:38 GMT
Server: Apache
Set-Cookie: PHPSESSID=baac274c99fd7023dabd439df624a885; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-Type: text/html; charset=iso-8859-1
X-Cache: MISS from localhost
X-Cache-Lookup: MISS from localhost:3128
Via: 1.0 localhost:3128 (squid/2.6.STABLE5)
Connection: close
[and then the output of the request]


Displays a breakdown of bandwidth usage by host


Allows you to view and record data transmitted over the network. Couple this with wireshark and you can inspect the packets!

Servers For Hackers

Chris Fidao