Your Inner Sysadmin

Being a Better Developer by Having Better Servers

Chris Tankersley
@dragonmantank (Twitter, app.net)
http://ctankersley.com

ZendCon 2012

Who Am I?

  • PHP Developer for about 9 years
  • Using Linux longer than I remember
  • Sysadmin professionaly for about 5 years

Quick Overview

  • Configuration Management
  • Logs
  • Server Monitoring
  • Command Line Tools
  • Users

Configuration Management

Process by which you figure out what goes on your servers, how you want them set up, and keeping track of that information. Files are usually stored in source control on one server and pushed to clients.

Why do you need it?

  • Ever needed to keep track of when files get changed?
  • Ever needed to roll back a change?
  • Ever needed to push the same change to a bunch of servers
  • Ever needed to set up a server exactly the same way as another server?

Puppet

Sample Manifest File

user { 'ctankersley':
        ensure => present,
}
package {'postfix':
        ensure => present,
}
node 'server.mydomain.com' {
        $mysql_rootpw = 'notreallytherootpassword'
        include mysql::server
        include mysql::client

        class {'apache':}
        class {'dotdeb':}
        class {'siteadmin':}

        $monit_secret='stillnotreallythepassword'
        $monit_alert='chris@tankws.com'
        $monit_mailserver='localhost'
        $monit_enable_httpd='yes'
        include monit
        monit::check::process{"mysql-server":
                pidfile => '/var/run/mysqld/mysqld.pid',
                start   => '/etc/init.d/mysql start',
                stop    => '/etc/init.d/mysql stop',
        }
}

Logs

Everyone loves to watch the logs all day long right?

Logwatch

Script that runs every so often and scans a bunch of logs so you get a pretty e-mail with a summary of the events
--------------------- httpd Begin ------------------------

 0.17 MB transferred in 792 responses  (1xx 0, 2xx 786, 3xx 0, 4xx 6, 5xx 0)
    199 Content pages (0.09 MB),
    593 Other (0.09 MB)

 Requests with error response codes
    400 Bad Request
       /w00tw00t.at.ISC.SANS.DFind:): 1 Time(s)
    404 Not Found
       /MyAdmin/scripts/setup.php: 1 Time(s)
       /phpmyadmin/scripts/setup.php: 1 Time(s)
       /w00tw00t.at.blackhats.romanian.anti-sec:): 1 Time(s)
       /webdav/: 2 Time(s)

 ---------------------- httpd End -------------------------

Ossec

Actually a Host Intrusion Detection System, but it does this by watching logs. Will alert you immediately to problems, and even shut down attacks if it notices them!
OSSEC HIDS Notification.
2012 Oct 24 11:38:10

Received From: maple->/var/log/auth.log
Rule: 5712 fired (level 10) -> "SSHD brute force trying to get access to the system."
Portion of the log(s):

Oct 24 11:38:09 maple sshd[1062]: Failed password for invalid user alias from 199.167.138.44 port 59988 ssh2
Oct 24 11:38:07 maple sshd[1062]: Invalid user alias from 199.167.138.44
Oct 24 11:38:06 maple sshd[1059]: Failed password for invalid user recruit from 199.167.138.44 port 59884 ssh2
Oct 24 11:38:04 maple sshd[1059]: Invalid user recruit from 199.167.138.44
Oct 24 11:38:03 maple sshd[1057]: Failed password for invalid user sales from 199.167.138.44 port 59763 ssh2
Oct 24 11:38:00 maple sshd[1057]: Invalid user sales from 199.167.138.44
Oct 24 11:37:59 maple sshd[1055]: Failed password for invalid user staff from 199.167.138.44 port 59661 ssh2
Oct 24 11:37:58 maple sshd[1055]: Invalid user staff from 199.167.138.44

Server Monitoring

Is your server up right now? Is the web server and the database services up and running?

Host Monitoring

Icinga

Service Monitoring

monit

Software Tools!

Lots of software on the server to help with development

If I say Zend Studio 10 helps developers, do I get a free license upgrade?

screen

There is also tmux, but I haven't switched to that. Supposedly it's better.

tail

Watch log files as they are generated

cURL

Command line program to for transferring data via a URL.
# Download a single file
$ curl http://www.zendcon.com

# Gives nify stats
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 66 1215k   66  805k    0     0  33060      0  0:00:37  0:00:24  0:00:13 45900
100 1215k  100 1215k    0     0  39474      0  0:00:31  0:00:31 --:--:-- 68987

# Download multiple files
$ curl -O http://www.zendcon.com -O http://www.zend.com

# View headers!
$ curl -v http://www.zendcon.com
Date: Wed, 24 Oct 2012 19:36:38 GMT
Server: Apache
Set-Cookie: PHPSESSID=baac274c99fd7023dabd439df624a885; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-Type: text/html; charset=iso-8859-1
X-Cache: MISS from localhost
X-Cache-Lookup: MISS from localhost:3128
Via: 1.0 localhost:3128 (squid/2.6.STABLE5)
Connection: close
[and then the output of the request]

iftop

Displays a breakdown of bandwidth usage by host

tcpdump

Allows you to view and record data transmitted over the network. Couple this with wireshark and you can inspect the packets!

Users

Sudo

You can give admin access to users (or groups of users) without giving them root.


# Add sudo access to a single user to run as root
ctankersley ALL=(ALL) ALL

# Add sudo access to a full group
%admin ALL=(ALL) ALL
							

You can even restrict what commands the users can run


# Restrict web developers to only restart Apache and MySQL
%webdevs 192.168.1.0/255.255.225.0=(root) NOPASSWD:/usr/sbin/service apache2 restart, /usr/sbin/service mysql restart
							

Jailing Users

Keeps people from getting to things they shouldn't. Protects the users from themselves.

Jailed Shells

Gives users a full shell but not the entire file system. You can pick and choose what programs the user can have access too. Jailkit makes this incredibly easy to set up.

Jailed SFTP

Locks the user to a specific base path, but doesn't give them a shell, much like FTP. You get the security of SSH though! Does require a system user though.

THE END

  • https://joind.in/7030
  • http://ctankersley.com/talks