CVE-2014-3704 Drupal SQL Injection Vulnerability – How can you fix it?

Twitter Facebook

This is a critical vulnerability. If this is the first you are hearing about it and you manage a Drupal website, I would highly recommend that you go read the Drupal PSA and follow the instructions there before doing anything else.

 

CVE-2014-3704

The situation with CVE-2014-3704

On October 15th, 2014 the Drupal core team released the details of a vulnerability, CVE-2014-3704, that was classified as the most severe type of vulnerability: Highly Critical. This CVE-2014-3704 SQL injection attack can allow remote, anonymous users to take control of your Drupal installation and gain access to all of your content. The Drupal team describes it in a bit more technical detail:

Drupal 7 includes a database abstraction API to ensure that queries executed against the database are sanitized to prevent SQL injection attacks.
A vulnerability in this API allows an attacker to send specially crafted requests resulting in arbitrary SQL execution. Depending on the content of the requests this can lead to privilege escalation, arbitrary PHP execution, or other attacks.
This vulnerability can be exploited by anonymous users.

The website firewall company, Sucuri, observed attacks on Drupal sites starting only 8 hours after the vulnerability was disclosed. Those attacks have become systematized and are now widespread. As a result, the Drupal team is advising all website owners to assume “[...] that every Drupal 7 website was compromised unless updated or patched before Oct 15th, 11pm UTC [...].”

What should website owners do about CVE-2014-3704?

Unfortunately, the patch that Drupal provided for CVE-2014-3704 can only prevent a future attack. If your site has already been compromised, as the Drupal team and Sucuri data suggest is very likely, the patch can not undo an attack that has already occurred. The recommendation from Drupal is to restore your website to a time before October 15th to remove the backdoor and then apply the patch to prevent re-infection.

If you have your website backed up with CodeGuard, restoring to a time before October 15th should not be an issue. If you do not have daily backups, CodeGuard or otherwise, you could also check with your hosting provider. However, while they may have a disaster recovery backup of their servers (including your website content) that was taken some time between 24 hours and 7 days ago, that would not help in a case like this where backdoors could have been inserted weeks ago. If you do not have a viable backup, the only other option that Drupal provides is to rebuild your website from scratch. Ouch. Backdoors can be incredibly difficult to find manually, so if you do not have a comprehensive backup, rebuilding is the only other truly safe option if you have been compromised by CVE-2014-3704.

Backups are a necessary part website infrastructure

Just like reliable hosting, a comprehensive backup strategy should be high on the list of priorities when building a new website or maintaining an existing property. In addition to providing some measure of insurance against events like CVE-2014-3704, they can also save you time in cases where files are accidentally deleted or when plugin updates go awry.

In addition to 99.99% levels of reliability and daily backups, CodeGuard’s ChangeAlert feature can also be helpful in bringing unauthorized file content changes to your attention. In the event that the backdoor was inserted in a file or your static content was defaced, CodeGuard can provide you with a notification of the change and the tools to roll back the malicious activity.

In this age of rapidly exploited vulnerabilities like CVE-2014-3704, a comprehensive backup strategy is no longer a nice-to-have option, it’s a must-have for every website owner or administrator.

– Jonathan

Why Do WordPress Websites Go Down?

Twitter Facebook

CodeGuard and WordPress

WordPress is a great tool for creating and managing websites. With over 60 million websites using WordPress, it is by far the most popular content management systems on the web. Not only that, WordPress is also open source which means that it is constantly being updated and improved. Why then, is it so common for WordPress websites to go down?

Unreliable Hosts

Often, the problem is not the user’s fault, but their host’s or even other people on their shared server. Failing hardware, outdated software, or mismanaged resources, can all cause server crashes. When this happens any websites on the server will go down. If you’re lucky, the server will be back up shortly, and you will have access to your website again. Sometimes, though, files and databases can become corrupt. In this case, your website might not be accessible, or it might look or behave differently. It often takes an experienced user to fix this type of problem.

WordPress Vulnerabilities

Problems can also arise within WordPress itself. Because WordPress is so easy to install, users often fail to properly secure their installation against outside attacks, and with so many on the web, WordPress installations are a popular target for hackers. The fact that all of the WordPress bugs and vulnerabilities are posted online makes it even easier for attackers to compromise a WordPress site. CVEDetails.com keeps an up-to-date list with information about many known WordPress vulnerabilities.

Once a hacker has access to your website, you might not even know that anything has changed. Many hackers create networks of compromised websites that they can use for DDOS attacks, click fraud, and adware/spyware/malware distribution.

Plugin Vulnerabilities

Plugin vulnerabilities are possibly the largest contributor to WordPress failures. In mid July, ZDNet published an article that described how vulnerabilities in just four WordPress plugins affected over 20 million WordPress installations. One of the vulnerabilities even allowed “attackers to upload malicious PHP files or backdoors to the target server without needing admin privileges.” Just like WordPress, many plugin bugs and vulnerabilities are published online for everyone to see. On one hand, the fact that everyone has access to this information is good, because it prompts developers to fix the problems, if they haven’t already, and it gives users a heads up that their website may be susceptible. Unfortunately, though, this information also makes it easier for hackers to find and exploit vulnerable plugins.

Besides being vulnerable, WordPress plugins are often buggy, bloated, or poorly maintained. In addition, multiple plugins can conflict with each other. Plugin incompatibility is one of the reasons why we at CodeGuard discontinued support for our WordPress plugin in favor of a more reliable and platform-agnostic website backup strategy.

What can you do?

There are several things you can do to make sure your WordPress website stays up and running.

  1. Choose a reputable host
    If you want your website to available 24/7 make sure you choose a reliable hosting provider. Do a little bit of research beforehand to see if other users have had problems with the host you have in mind.
  2. Secure your WordPress installation
    WordPress has published an excellent guide to help you harden your WordPress installation against outside attacks.
  3. Keep your plugins up-to-date
    Old plugins with known vulnerabilities are easy targets for hackers. Keeping your plugins up-to-date will help ensure that they are bug-free and harder to attack. Limiting the number of plugins you have is also a good idea, because it reduces the number of attack vectors that hackers can potentially exploit.
  4. Have a backup
    If all else fails, it’s important to have a plan for recovery. CodeGuard provides easy backups for your WordPress installation, and we have specific WordPress features that make restoring your site even easier.
    Website_Backup_in_the_Cloud___CodeGuard

– Taylor

Product Development at Scale

Twitter Facebook

We know that our customers see our service as a critical part of their infrastructure, and this is a responsibility that we take seriously. Unlike other software-as-a-service companies, we don’t have the luxury of hoping that a customer will notice an issue and report it or try again later. CodeGuard provides an automated service and for customers that rely on us for daily backups, that means that we have a 24 hour window, 365 times a year where we have to perform at our very best. Our process for developing and maintaining the service has to reflect that reality.

Just as we have scaled our service to accommodate hundreds of thousands of website and database backups on a daily basis, we’ve had to scale our product development process to consistently deliver high-quality results for our customers while maintaining the 99.99% levels of reliability we have achieved. We can’t haphazardly release features or change functionality without thoughtful preparation, focused execution and thorough testing.

What does this look like?

It can vary slightly from project-to-project depending on the components involved, but we generally follow an Agile-based workflow. As a result, the entire team participates in every phase of the project from brainstorming through maintenance.

1. Brainstorming

523001_434980639899033_150466711_n

Brainstorming allows everyone the opportunity to share ideas and get everything out into the open. These discussions are not structured and often cover topics from from high-level deliverables to specific implementation details. Not only are they helpful for getting good (and bad) ideas into the group consciousness, they also serve as an opportunity for everyone to reach a common understanding of the project at hand, any potential complexities and options for execution.

2. Planning

 

Restore_Improvement_Plan_v1_0_-_2Sept14_-_Google_Sheets

After brainstorming, we move on to planning. This is a very important, but often incredibly difficult part of the process. The high-level product goals have to be deconstructed into small enough pieces for an engineer to build. Knowing how small or large to make these units of work and how to estimate their size relative to the other units of work in this project can be a tedious exercise. Make the pieces too small and it’s difficult to adjust the plan as new information arises and divide the work amongst the team. Make the units too large and you may end up waiting for a big piece of functionality to be completed which delays progress on other components. How this process works exactly will vary greatly based on the composition of personalities on your team. For us, it’s usually a spirited discussion that lasts two or three hours every other week. The result is often flow charts of functionality and a plan with the individual units of work.

3. Execution

_D1A0488_v1fs

As an engineer, this is the most fun part of the product development cycle. Everyone on the team starts implementing the items that were defined in the plan. We meet formally every morning to talk about progress and areas that we need help, but there is constant communication within the team during the day. During this process, we often work in ad-hoc pairs to get through the more complex bits of functionality or in cases where we are modifying existing components related to our core backup or reporting systems.

4. Testing

In addition to automated unit testing and functional testing, we also do a lot of team testing ahead of releasing new functionality. Our service connects to a wide variety of hosting providers, platforms and server types. As a result, we are extremely suspicious of the software that we build. Having humans devise test scenarios and directly interact with any new features we have built gives us an extra level of assurance that we did not miss anything in our automated testing. For functionality that requires user interface changes, it also give us the opportunity to make sure that everything works as expected in all of the popular browsers.

5. Deployment and Maintenance

Unlike other development teams, once we’ve built a new feature, thoroughly tested it and agree that it is ready for release, we can’t just give it to the client as say that it’s finished. We have to deploy the changes to all of our customers and maintain all of the functionality that we build. This reality of being both the development and operations team makes us very cautious about how things are implemented. If you know that your phone could be the one ringing at 2am when something breaks, you’re going to think twice before modifying code that you don’t completely understand or writing a super-clever, elegant, one-line and completely unreadable piece of code.

Conclusion

I believe that this process, which represents our intentionality, intense focus and commitment to delivering a high-quality product to customers is one of things that sets us apart from other backup solutions. An open-source plugin may have had more developers looking at the the source code, but the ultimate responsibility for ensuring that a backup runs every day is still up to you, the website owner. And, what about those times that you inevitably need assistance? Community support is a wonderful thing, but is it the best option for mission-critical infrastructure services?

These limitations do not apply to CodeGuard. Our service isn’t free, but that is because we have a team of professionals that are dedicated to the success of your backups. Every CodeGuard customer has a whole team working for them to ensure that their backups happen as scheduled and we have developed a process that allows us to do that while continuing to improve and add features to our service.

– Jonathan

Service-Oriented Architecture: Not just for web services

Twitter Facebook

CodeGuard, like many rapidly growing software as a service businesses with a maturing product, is breaking apart our monolithic software architecture into smaller, modular services, or a Service-Oriented Architecture (SOA). While monolithic architecture has the advantage of avoiding premature architectural decisions when at the experimental stage of a product’s life, it is difficult to scale after this stage and lacks modularity by definition. As monolithic applications grow, it becomes increasingly difficult to reason about the impact of changes and predict or avoid side-effects. For these reasons, new features and products at CodeGuard are built using an SOA approach with modularity in mind. In addition, functionality of current monolithic applications is broken out into separate modular services when possible.

SOA by Command-line Interface

For many developers, the term SOA, or Service-Oriented Architecture, is synonymous with web services. This perception is not only a misconception, but also misses the core ideas of SOA and focuses instead on the method of implementation. SOA, as the term suggests, is instead about software architecture focusing on loosely coupled units that are self-contained and expose well-defined interfaces for the services they implement. While web services are one way to implement SOA, they are not always the best choice, especially when breaking apart monolithic software. When our engineering team decided to build a backup verification service we implemented it as a command-line interface, or CLI.

Reasons for choosing this approach over a web service include:

  •  I/O cost: A web service is typically deployed on separate infrastructure from the client of that service. While this can be an advantage, it can also incur overhead if the service requires access to data such as a backup. For backup verification, this data may already be available on local disk and running a service locally via a command avoids the cost of retrieving that data twice.
  • Everything can shell out: Building a service as a CLI is language-agnostic since every language and framework that runs on linux can shell out. In this respect, it is no more limiting than a web service and possibly less so as some technology stacks lack robust networking or good support for relatively newer technologies like JSON.
  • No data model duplication: In addition to avoiding I/O overhead, a CLI can help avoid the temptation of duplicating data models or relationships from the client application.
  • Well-defined, documented API: Choosing to build a service as a CLI instead of a web service does not mean compromising on the interface for the service. A CLI can explicitly require options and combinations of options, returning an error if the requirements are not met. Documentation tools include help output and man pages and can often be auto-generated from code.
  • Easy deployment: A web service has to be deployed, the clients of the service have to be configured to know where to send requests or use service discovery, and then of course the service must be available at all times that it is needed. A CLI, in contrast, needs only to be deployed (in many cases just a binary) to the servers that call it.
  • CLI -> Web Service: A CLI is just another form of interface and a CLI, if it is built with modularity and separation of the interface from the implementation, can become a web service without rewriting the functional code. Instead of command-line options, the options are specified in a network request, usually an HTTP request containing JSON.

Backup Verification Service

One of the services that the CodeGuard engineering team has built as CLI is our backup verification tool. This tool is used to independently verify that the file backups of our customers’ websites are complete and correct. It does not share any code with our backup system. Given the location of a backup directory structure, it compares the contents to the contents of the remote server and logs any differences it finds, including missing files, files of the wrong size, additional files, and files with differing metadata. This service is built in Go (www.golang.org), which compiles to easily-deployable static binaries with no dependencies and is also simple to cross-compile.

A simplified version of the CLI is included below:

Application Options:
 --path= Path of backup to verify
 --u= User name
 --p= Password for login
 --key= SSH key if using key-based authentication
 --port= Port to connect
 --site= Remote server URI
 --x= Exclusion filter (regex). May be specified multiple times
 --last_backup= Last backup time

To separate the interface from the implementation of the service, we use the excellent go-flags package. Using the backup verification service as a CLI, the backup system can simply shell out to start a verification. Logs of differences found or errors are written to the standard error output and can be retrieved by the client service. The interface clearly documents the parameters required to run a verification and will exit with an error if the required parameters are not included as command-line flags.

At CodeGuard, we also build web services as  part of our SOA, but they are certainly not the only way to build a service-oriented architecture. Often they are not even the best choice as the overhead and complexity incurred can be unacceptable or lead to new problems. As our architecture evolves we may someday need to run our backup verification service as a web service. When that day comes we will need only to rewrite the implementation of the wrapping interface. The encapsulated functionality may not change at all. For now, a CLI gives us all the major benefits of SOA without unnecessary architectural complexity or I/O overhead.

-Randall

A/B Testing and Website Backup

Twitter Facebook

August was an exciting month for the CodeGuard team. Something we decided to focus on was fine-tuning the copy on our marketing website. Were we doing a good job of explaining ourselves? Were we effectively reaching our target audience? How can we increase engagement on our website?

I think it’s healthy for a company to take a step back and ask themselves these questions at least once a year. From one year to another an assortment of policies, features, services offered, prices, or more could have changed with your product. It’s important to make sure that you are still explaining what it is you do in an accurate way, while also using words that will resonate with your target audience.

With these questions in mind, we decided to launch a series of A/B tests on our homepage and a few landing pages, to see how we could improve. For the remainder of the blog post, I’ll be focusing on just our homepage tests. A/B testing is a simple way to test changes to your page against the current design, and determine which ones produce positive results. It’s a way to test which version of the page has the best ‘conversion’ rate, based on a goal that you decide.

The goal could be, “how many times was this video played?”, “How many times was this button clicked?”, or “How long did the website visitor stay on the page?” It’s very important to clearly define what success means for your A/B test, so that you can figure out which page performed better. We wanted to keep these tests as simple as possible, so for us, success meant “Did the website visitor click our ‘Sign Up Free!’ button at the top of the page?”

These are the main things that we tested between our control page, and the variation page:

  1. Headline (h1) copy
  2. Paragraph copy above the fold
  3. Video placement and image placement
  4. Button copy

Before we could begin creating our variation copy for these elements, we had to reflect on the image we wanted to portray of ourselves to prospective customers. To start, building trust with our website visitors from the get-go is extremely important to us. We take security very seriously here at CodeGuard, and we want potential customers to know that their data will be safe with us. That is why the first A/B test we launched was between our homepage and a variation homepage with our “Trust Seals” right at the top.

 

CodeGuard Website Backup Homepage

 

We wanted to know if having our security and business certifications prominently displayed on our website would help visitors realize that we are very concerned with security, and very passionate about the transparency of our business.

In the end, we saw a staggering 56% improved conversion rate with our variation page that had the Trust Seals! We couldn’t believe that a change so simple had such jaw-dropping impact on our goal!

We made a few more minor changes here and there with button copy, and some copy elsewhere on the page. After all of that though, we were itching to launch another A/B test to see if we could get more results like our first major A/B test. This time, we decided to make a much bolder change. We tested our existing homepage (that now had the Trust Seals permanently at the top) with a variation page that changed the headline copy, paragraph copy, and the ‘Hero Image’ of our top section.

For a long time now, we have used our “Get a time machine for your website” tagline to help people understand what our product does. We always thought it was catchy enough to reel in people’s interest, but also descriptive enough to let people know we dealt with website backup. After a few rounds of user-testing, we were able to both confirm and refute this assumption.

For some or our testers, this tagline made perfect sense to them, and invited them to want to explore the page further. Some of our testers though, were confused by the tagline, and it took them a few seconds of further reading before they realized what our product offered. If a potential customer decided to Google search for a way to back up their website, were they entering in queries that had the word ‘time machine’ in them? Most likely not. Our analytics dashboard helped confirm this as well.

So for our next biggest A/B test we decided to change our headline copy to “Reliable, robust, and secure cloud backup for websites.”

 

Screen Shot 2014-09-05 at 9.31.46 AM

Because the ‘time machine’ wording was no longer being used, we also decided to swap out our time machine image for a more prominent placement of our video. Finally, we changed the paragraph copy below our headline in hopes of building even more trust. “We are backing up over 150,000 websites worldwide… Trusted by some of the world’s largest hosting providers…” etc.

The A/B test was launched and we waited eagerly for the results.

After about a six days of patiently waiting, I decided I couldn’t take the anticipation anymore, and so I logged into our Analytics account to see how the test was doing.

Our variation page was performing at a -41% (negative) conversion rate compared to our original homepage. I was shocked! Leading up to the experiment, I was convinced we had done everything right in terms of critically thinking about the variation copy we were writing. We used existing data in our Analytics account, and notes from previous user-testing sessions to inform our decisions, and guide our new copy. So why was our variation page performing so poorly? It was the first time one of our variation pages lost.

We decided it was time to take a step back from the barrage of A/B tests we were launching, and reflect on our tests a little before beginning another one. In the end, I think this A/B test wasn’t as successful because we tried to change too much too soon. Before this test, our tests consisted of changing one thing, and seeing how it performed against our control variable, or original page. With this A/B test, we changed three very major things in the top section of our homepage. We got ahead of ourselves because of our excitement!

I don’t think this is necessarily a bad thing though, because it helped us understand how we can perform a better test in the future. When it comes to A/B testing, there are countless articles out there that tell you how to do it best, and I read many of those articles. This is what we learned from our first-hand experience:

  1. Have clearly defined goals for the experiment
  2. Thoroughly test that your goals can be successfully recorded in your Analytics account before launching the test
  3. Use data to guide your decisions for what to change
  4. Don’t change too many things all at once. Less is more!
  5. Take time to learn from your tests before launching another one

I’m sure that more will be added to this list over time, but for the time being, we are very happy with the results we have been able to accumulate over the month of August. Our website is performing well, we have renewed confidence that we are building trust with our visitors, and we look forward to the next round of improvements!

Whether your company deals with website backup, travel booking, social media, or anything else on the web, I think it’s important to ask yourself once in awhile “Are we reaching our target audience as best as we can?”

The results of your A/B tests may surprise you!

Natalie