Your Inner Sysadmin

Better Development Through Understanding Your Servers

Chris Tankersley
@dragonmantank
http://ctankersley.com

LonestarPHP 2014

Who Am I?

  • PHP Developer for around 10 years
  • Using Linux longer than I remember
  • Sysadmin professionaly for about 8 years

Traditional LAMP Stack

The Server

Where Stuff Is

  • /bin - Essential user executable files
  • /boot - Stuff that makes the OS boot up!
  • /dev - Special device stuff you probably won't touch
  • /etc - Configuration files
  • /home - User home directories
  • /sbin - System binaries
  • /usr - Multi-user apps and utilities
  • /var - Data usually lives here

Installing Software

  • Compile software from scratch
  • Use the package manager (yum/apt)

Learn to love the command line

Authentication and Authorization

Sudo

You can give admin access to users (or groups of users) without giving them root.


# Add sudo access to a single user to run as root
dragonmantank ALL=(ALL) ALL

# Add sudo access to a full group
%admin ALL=(ALL) ALL
							

You can even restrict what commands the users can run


# Restrict web developers to only restart Apache and MySQL
%webdevs 192.168.1.0/255.255.225.0=(root) NOPASSWD:/usr/sbin/service apache2 restart, /usr/sbin/service mysql restart
							

Jailing Users

Keeps people from getting to things they shouldn't. Protects the users from themselves.

Jailed Shells

Gives users a full shell but not the entire file system. You can pick and choose what programs the user can have access too. Jailkit makes this incredibly easy to set up.

Jailed SFTP

Locks the user to a specific base path, but doesn't give them a shell, much like FTP. You get the security of SSH though! Does require a system user though.

Docker

Scripting Languages

Bash

Most servers use bash as the default shell. Most shells understand bash's syntax. If you find yourself running the same commands over and over, throw it in a bash script.

Python

Ships with most distros. Great for when you need more power than what bash has.


http://xkcd.com/353/

PHP

Leverage your PHP skills to write shell scripts.

Helping Apache Along

Running Apache as a Different User

mpm-itk

<IfModule mpm_itk_module>
    AssignUserId [user] [user]
</IfModule>

mod_ruid2

RMode config
RUidGid myuser mygroup
RDocumentChRoot /var/www/vhosts/domain.com/ www/public

php-fpm

user = myuser
group = mygroup

Logs

logrotate

Rotates logs out for organization (or other) reasons

weekly
rotate 4
create
include /etc/logrotate.d
/var/log/wtmp {
    monthly
    minsize 1M
    create 0664 root utmp
    rotate 1
}

Logwatch

Script that runs every so often and scans a bunch of logs so you get a pretty e-mail with a summary of the events
--------------------- httpd Begin ------------------------

 0.17 MB transferred in 792 responses  (1xx 0, 2xx 786, 3xx 0, 4xx 6, 5xx 0)
    199 Content pages (0.09 MB),
    593 Other (0.09 MB)

 Requests with error response codes
    400 Bad Request
       /w00tw00t.at.ISC.SANS.DFind:): 1 Time(s)
    404 Not Found
       /MyAdmin/scripts/setup.php: 1 Time(s)
       /phpmyadmin/scripts/setup.php: 1 Time(s)
       /w00tw00t.at.blackhats.romanian.anti-sec:): 1 Time(s)
       /webdav/: 2 Time(s)

 ---------------------- httpd End -------------------------

ossec

Actually a Host Intrusion Detection System, but it does this by watching logs. Will alert you immediately to problems, and even shut down attacks if it notices them!
OSSEC HIDS Notification.
2012 Oct 24 11:38:10

Received From: maple->/var/log/auth.log
Rule: 5712 fired (level 10) -> "SSHD brute force trying to get access to the system."
Portion of the log(s):

Oct 24 11:38:09 maple sshd[1062]: Failed password for invalid user alias from 199.167.138.44 port 59988 ssh2
Oct 24 11:38:07 maple sshd[1062]: Invalid user alias from 199.167.138.44
Oct 24 11:38:06 maple sshd[1059]: Failed password for invalid user recruit from 199.167.138.44 port 59884 ssh2
Oct 24 11:38:04 maple sshd[1059]: Invalid user recruit from 199.167.138.44
Oct 24 11:38:03 maple sshd[1057]: Failed password for invalid user sales from 199.167.138.44 port 59763 ssh2
Oct 24 11:38:00 maple sshd[1057]: Invalid user sales from 199.167.138.44
Oct 24 11:37:59 maple sshd[1055]: Failed password for invalid user staff from 199.167.138.44 port 59661 ssh2
Oct 24 11:37:58 maple sshd[1055]: Invalid user staff from 199.167.138.44

Preventing Intruders

hosts.deny/hosts.allow

Set of files to allow or deny access to the machine or certain apps/ports on the machine

iptables

A firewall that is generally available on Linux machines that can be configured many different ways to allow or block or mangle traffic

ossec

IDS that was logs and will use hosts.deny and iptables to block stuff automatically for you!

Configuration Management

Process by which you figure out what goes on your servers, how you want them set up, and keeping track of that information. Files are usually stored in source control on one server and pushed to clients.

Why do you need it?

  • Ever needed to keep track of when files get changed?
  • Ever needed to roll back a change?
  • Ever needed to push the same change to a bunch of servers
  • Ever needed to set up a server exactly the same way as another server?

Puppet

Sample Manifest File

user { 'ctankersley':
        ensure => present,
}
package {'postfix':
        ensure => present,
}
node 'server.mydomain.com' {
        $mysql_rootpw = 'notreallytherootpassword'
        include mysql::server
        include mysql::client

        class {'apache':}
        class {'dotdeb':}
        class {'siteadmin':}

        $monit_secret='stillnotreallythepassword'
        $monit_alert='chris@tankws.com'
        $monit_mailserver='localhost'
        $monit_enable_httpd='yes'
        include monit
        monit::check::process{"mysql-server":
                pidfile => '/var/run/mysqld/mysqld.pid',
                start   => '/etc/init.d/mysql start',
                stop    => '/etc/init.d/mysql stop',
        }
}

Server Monitoring

Is your server up right now? Is the web server and the database services up and running?

Host Monitoring

Icinga

Service Monitoring

monit

Software Tools!

Lots of software on the server to help with development

tmux/screen

Command line multiplexer

tail

Watch log files as they are generated

cURL

Command line program to for transferring data via a URL.
# Download a single file
$ curl http://www.zendcon.com

# Gives nify stats
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 66 1215k   66  805k    0     0  33060      0  0:00:37  0:00:24  0:00:13 45900
100 1215k  100 1215k    0     0  39474      0  0:00:31  0:00:31 --:--:-- 68987

# Download multiple files
$ curl -O http://www.zendcon.com -O http://www.zend.com

# View headers!
$ curl -v http://www.zendcon.com
Date: Wed, 24 Oct 2012 19:36:38 GMT
Server: Apache
Set-Cookie: PHPSESSID=baac274c99fd7023dabd439df624a885; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-Type: text/html; charset=iso-8859-1
X-Cache: MISS from localhost
X-Cache-Lookup: MISS from localhost:3128
Via: 1.0 localhost:3128 (squid/2.6.STABLE5)
Connection: close
[and then the output of the request]

iftop

Displays a breakdown of bandwidth usage by host

tcpdump

Allows you to view and record data transmitted over the network. Couple this with wireshark and you can inspect the packets!

Servers For Hackers

Chris Fidao
@fideloper http://serversforhackers.com/

Questions?

THE END

  • https://joind.in/10810
  • http://ctankersley.com/talks