The Mission: Mobile Site!

Here at CodeGuard things tend to move pretty quickly. We’re made up of a team of highly driven individuals that like to build things that we know will positively impact our users and build them fast! We soon realized that not only do we move fast, but our users do too. More and more we realized that we were getting a lot of traffic to our website from mobile devices, and our users were increasingly requesting features that were optimized for mobile (if not built to be on mobile first – desktop second).

We have been steadily optimizing the CodeGuard website into a responsive design for a while now, but it became clear to us that it was time for a different strategy: A mobile-first design. It’s not that there is anything seriously wrong with a responsive design. RWD (Responsive Web Design) is an increasingly popular way to serve mobile users a version of your site that can fit their devices and allow them to access the same content that they would on a desktop website. There are some major caveats though. When you serve the same exact content to desktop and mobile visitors in the same exact way, regardless of what device they are on, you risk slow load times and giving your mobile users the content that maybe they don’t want to see.

For example, a restaurant may have large images of their food at the top of their desktop website but a user visiting that same website from their phone may not want to be served large images that are slow to load. Maybe all they wanted was to find that one tiny link that says “Directions” so they could get to that restaurant. If you don’t think about what your users want from you out of your mobile experience then you are bound to give them an experience that is unsatisfactory. One needs to think about why users are visiting your site from a mobile device, and here at CodeGuard we set out to find those answers and create a multi-device mobile-first design that best suited our users needs.

The Timeline/Deadline: Two Weeks?!?!?!?!

As always, we strive to push ourselves to new limits at CodeGuard. We gave ourselves two weeks to plan, research, ideate, wireframe, design, prototype, develop, test, and launch a completely new mobile homepage for CodeGuard.com – all while still taking care of the other on-going projects that we had going on at the time.

The Research/Planning:

So how did I get the idea to try a mobile-first design? A lot of my inspiration for this project came from one spectacular event I went to in Atlanta called “An Event Apart.”

Not only were the speakers amazing at what they presented on, but the community that showed up was really fun to interact with and meet. I highly recommend that you attend one in your city if you get the chance! It was here that I got a lot of inspiration and ideas for how to re-imagine CodeGuard’s mobile experience. I learned that a mobile-first design where you started from the smallest screen size and worked your way up was the best way ensure that your site would look good on any screen size. I learned that you should let the content decide where the layout “breaks” into a new layout and not the device it is on (which is a great way to make sure your design is device agnostic) and I learned a bunch of creative and inventive ways on how to best optimize your content and layout for these small screen sizes.

With this newfound inspiration I took to researching good design and development practices. For my design research I sought out to answer the questions: How is navigation typically laid out on mobile websites? Who does it well? What do I like about this mobile website? What do I not like about this other mobile website?

Development planning was a slightly larger task. Because we were going with a mobile-first design this meant that we were going to be creating a separate mobile website – while still maintaining our desktop website for a time period. I did a lot of research on this topic, and going to An Event Apart helped me reach this decision in the end. The cons involved us supporting two codebases for a short period of time, and figuring out a graceful solution for how we would handle device detection and redirects when they landed on our homepage, but the pros far outweighed the cons in the end. We would be able to start fresh with our content and give our users the content we know they want, instead of just serving them what is the same on the Desktop site. We would also be eliminating a lot of potential “code bloat” by starting over with a new codebase that is purely optimized for mobile from the start, instead of the other way around. It also has the potential to be much faster than our existing website.

In the end we decided to create our mobile homepage as a stand alone static website built on a front-end tool called Middleman while creating our html/css using haml and sass, but I’ll get into more detail on how I developed the site in moment!

The Wireframing/Designing/Prototyping:

So to begin, what was our goal with this new mobile homepage? What did we want to accomplish? What do we want our users to see? What do we think they want to see? To answer these questions, we decided in the end that our goal would be to “start a relationship with the mobile user.”

I started sketching wireframes on my whiteboard at home and broke up the homepage into a few sections that I thought would help us achieve this.

The top section would answer the question that every homepage – mobile or not – should answer, “Who are we, and what do we do?” Right away there were some challenges though, because the way our current homepage does this is by having some text and then a large video. While everyone here at CodeGuard loves our homepage video, it takes up a lot of space when translated directly into a small viewport size. Space is already very precious on a mobile website. I tried to keep the navigation and logo area just a small strip at the top of the screen so that it only takes up about 10% of the screen on load, and then the rest of the 90% of the screen can be all content answering the “Who are we, and what do we do?” question.

Our video could have easily taken up all of that 90% though, so I had to think differently about displaying it. The end result is that it is hidden until requested to be watched. This ended up saving us precious bytes as well, since we are not loading it until it is asked for (but I’ll get to speed optimization and other development wizardry later in the post).

The next section would answer the question, “What can we do for you?” I went back and forth with this section on whether I wanted to show a story of how CodeGuard works, or what our three main features were. I decided to go with our three main features (Backup, Monitor, Restore) in the end, hoping that if people liked what we had to say there, they could find out the “behind the scenes” stuff on a different page.

The third main section (one that cannot be seen in my whiteboard sketches because I added it later) is a client testimonial section. We can tell our mobile users all day long about how perfect we think we are for them, and how important our services are (and they are!), but hearing it a second time from a source other than us is always a good thing too!

The final section of our mobile homepage is a section where the user can enter their email address and get a special discount code emailed to them that they can use on any plan. We liked the idea of our mobile users checking out our website from their mobile device, or wherever they may be, but also getting the opportunity to browse our desktop site at a later time. We thought the email would be a great way to remind the user that they visited us, while also giving them a great incentive to try us out!

After deciding on all of this it was time to hit up Photoshop. Below I’ve included a few screenshots of things that I was designing, scraping, re-designing and polishing throughout this phase.

The Development/Collaboration:

Now that I was ready to develop it was time to start playing with my fun new tool Middleman. Middleman is a static site generator that came with a lot of nice features out of the box for those familiar with Ruby (which is what we primarily wield here at CodeGuard). It already had support for haml and sass, which were two tools I’ve come to love here at CodeGuard, and it had a wicked fast server for building your project and compiling your tools into whatever output you needed (html, css, javascript). It also did a lot of neat compression things for you when you start the project/server up so it would keep our site fast. When it came to the client testimonial section of the homepage I had a lot of cool ideas to involve jQuery mobile swipe functionality and other neat touch interactions, but I was fast approaching on my two week deadline. In the end I decided to save some of the more “bells and whistles” type coding for version 2.0 and focus on the nuts and bolts haml and sass coding that needed to be done. For testing the site as I was developing I used a Ruby gem called localtunnel to be able to view the site on my mobile devices and a neat Chrome plugin called OSX Resizer to let me resize my browser window on my laptop smaller than 400px.

The Testing/Finishing Touches:

I saved the last two to three days of my two week deadline for testing because I wanted to be as thorough as possible. The most popular mobile browser being used today is Safari, followed by Android default browser, and then Chrome. The fourth most popular was Opera Mini (my personal mobile browser of choice at the moment) but the drop off from Chrome to Opera in use was so severe that it was almost not worth testing for (but I did anyway). The Android default browser had the most issues to work out, mainly around the video functionality and pop-up messages surrounding the form, but overall testing was not nearly as painful as it was when I was doing responsive design testing. This made me believe even stronger that I had done things right this time around, and had approached this project the right way.

Launch:

With help from our Director of Engineering Jonathan Manuzak, we were able to solve all server redirect details quickly and get our mobile website out and in front of our eager users. Two weeks went by very quickly, but with all the planning and research that went into this project I was very confident that we were doing things very right. I think that we stayed true to our goal in the end while never forgetting our users in the process. In two weeks we were able to critically think about the content our users want from our homepage and give it to them in a way that is easy to use, fast to access, and fun to interact with. But don’t take my word for it, go see for yourself! Visit codeguard.com on your mobile device of choice and make sure to let us know what you think!

- Natalie

We have reason to believe that a new type of attack – WordPress 404 Redirect Hosted Scam Sites – could be impacting millions of WordPress websites. Several weeks ago we posted on a Yahoo! Mail XSS or CSRF attack that hijacked Yahoo! Mail accounts and sent emails to address books without the user’s knowledge. We explained that the emails were just the first step in an elaborate phishing or scam attack. The crucial second step, we explained, was the landing page on a compromised website that entices the victim.

In attempting to book a dinner reservation, I stumbled across a compromised WordPress Site (3.5) that seems to be a part of the same scam. But this compromise is different, in that any URL input will return the scam landing page. We think that this type of website compromise – the 404 Redirect Hosted Scam Page may be related to the WordPress Brute Force attacks from mid-April. A WordPress site with compromised credentials could be violated at any point in time, and the website owner would not know it.

The worst type of web compromise, or any compromise, is one that you don’t know is occurring. If you are not aware of the compromise, you will not take steps to end it. There is no site scanning system on the market that is catching these hacks. I will repeat this as it is important. McAfee SiteAdvisor, Symantec’s Safe Web, and Google Safe Browsing do not detect and will not help you. You will not know if your site has been compromised. Slightly disconcerting?

Why should you care?

This is important because if you own a website, you are responsible for how it is being used or abused. You are liable for the business asset that is your website. Your website is intended to help your business in some manner, or perhaps it is your personal space on the interwebs to blog and share with the world. Either way, your website could be a part of scam. Right now. Just like a high-end Atlantan restaurant that is redirecting customers to weight loss pills.

But the site is fine if I view it and don’t see anything weird, right? Maybe not. Look at what happens if we add anything to the URL that directs us to a page not explicitly defined by the webserver (meaning the page isn’t there). Normally, we would expect to see a 404 page stating that the page we are looking for can’t be found. When we do the same thing with topflr.com, and enter “www.topflr.com/test” the following screenshot is what is returned. Notice that the title of the page has changed to “ConsumerLifestyles” and the content looks completely foreign and irrelevant to fine dining in Atlanta.

Look carefully at the URL – “http://topflr.com/test”. Hmmmmm. What’s happened? What has happened is that someone has taken advantage of a highly profitable corner of the internet – URL typos and missing pages. John Biggs explains best, referring to the outcry when Network Solutions attempted to monetize these pages circa 2004:

“These DNS servers are absolutely vital to the Internet. When Network Solutions, the arbiters of US websites, changed the nature of DNS translations by adding a ‘search page’ when users typed in incorrect addresses, the tech world protested loudly. This simple change destroyed hundreds of hours of work in the community and commercialized the Internet to a degree that made many gurus uncomfortable. Network Solutions quickly removed the search page, meaning that visitors who typed the URL www. groosble. com by mistake will get an error instead of a Network Solutions-sponsored web page and advertisement.”

John Biggs. Black Hat: Misfits, Criminals, and Scammers in the Internet Age (Kindle Locations 850-853). Kindle Edition.

What is happening is that miscreants are taking advantage of a very known and profitable portion of internet real estate: 404 pages. They are using random chance to aid in driving unsuspecting website visitors to diet pill scam sites. Have you ever typed in the wrong address to a website and seen a 404 page? How can you figure out if your site is infected. Let’s take a look at some of the tools you might try to use.

Garcinia Cambogia: Is your site infected?

Google has some handy functions that allow you to search on content within your site, using the “site:” command. The key here is that these pages have been indexed by Google. Unfortunately, they might not work with this hack, if Google hasn’t indexed a particular page, or if the page does not yet exist – meaning the php file has not rendered it for Google to index, because the unwary site visitor has not yet mistyped the address? Confusing? I’ll explain how the 404 Redirect Hosted Scam Page works in just a moment.

How does it work?

What is known right now, thanks to the Sucuri team, is that two key files are necessary for this hack to work. “Wpppm.php” and “.k”. The wpppm.php file, when any 404 requests are made, loads the content in the “.k” directory. Right now, that content points to Garcinia Cambogia diet pills, but it could change.

What is the impact?

Something has caused the extreme spike in websearches for “Garcinia Cambogia”. Press releases, and Dr. Oz mentions would fuel increased interest, but nothing like what is seen in the Google Trends chart below. Dr. Oz talked about Cambogia first around October 2012. The small bump in the below chart can be most likely attributed to the mention on his show. But what, then, has caused the massive surge in interest? A press release? Anyone who has published a press release can tell you that simply publishing one will not generate the type of interest seen below. It is likely that some type of mass spam email and website hacking has taken place to fuel this surge.

WordPress 404 Redirect Hosted Scam Sites Could Impact Millions

WordPress websites, visited by hundreds of millions of people globally, are being compromised. The exact vector is unknown at this point. I’m using “vector” to mean the way that the site was compromised – if someone guessed your WordPress password & username, for instance, it can be said that they exploited a credentials management vulnerability.

What should you do? Search your wp-content (and the rest of your WordPress installation) via FTP for “wpppm.php” and “.k”. And keep searching for those files everyday. Or you can try CodeGuard, which will notify you via a ChangeAlert if anything on your website has changed.

-David 

CodeGuard Host Server Load Testing
Version: 1.0 / Date: 5 April, 2013

Overview

The goal of this test is to determine what impact the CodeGuard website backup service has on a host server during a backup. The results contained in this document were gathered after a series of four tests (described later). They are by no means conclusive and real-world results will vary based on the composition of the websites being backed up and the hardware and software running the underlying host server.

Test Methodology

There are two phases of a backup that are of particular interest with respect to host server performance: file listing and downloading. The first, file listing, entails inspecting the metadata for each file and folder within the scope of the backup. Practically, this means recursing through all of the folders in the target website and inspecting every file. The downloading phase simply transfers all new or modified files from the host system to the CodeGuard service. For the initial backup, this means downloading every file in the site, but for subsequent backups only new or changed files are downloaded. These two steps are always performed sequentially.

Definitions

Activation: The first backup performed by CodeGuard. All files within the backup scope will be inspected and all files will be downloaded.

Pull: All backups performed after the initial Activation. All files within the backup scope will be inspected, but only new or changed files will be downloaded.

Test Website

Only one website was used for this suite of tests. In cases where concurrent backups were taking place, multiple copies of the site were used to avoid simultaneous requests for the same files on disk.

Website Statistics
Size: 538MB
File and Folder Count: 8,774
Type: Real, active WordPress blog

Test Host
Server: The host used for testing was a RackSpace CloudServer.
OS: CentOS 6.3
FTP Server: Pro-FTP
Memory: 512MB
CPU Cores: 1

Website Backup Testing Results

The graphs below illustrate the results of each test. Following each are notes discussing the findings.

Metrics and Definitions

• CPU usage: System: Percentage of CPU usage by system processes.
• CPU usage: User: Percentage of CPU usage by user processes.
• % Memory Used: Percentage of system memory used.
• eth0 in: Network transfer in from the public network connection in KB/s.
• eth0 out: Network transfer out to the public network connection in KB/s.
• Server Load (Last 5 Minutes): The numeric representation of the load on the system for the last five minutes. This is an amalgamation of different metrics but, for this system, loads less than 1.0 are acceptable. More information can be found here: http://blog.scoutapp.com/articles/2009/07/31/understanding-load-averages
• I/O Average Queue Size: Weighted number of milliseconds spent doing I/Os. This can provide a measure of both I/O completion time and the backlog that may be accumulating.
• I/O Wait: Time in milliseconds spent waiting to perform I/O operations.
• I/O Reads / second: Number of file system reads per second.
• I/O Writes / second: Number of file system writes per second.

1. One Activation – FTP

Figure 1. Server load graph.

Figure 2. Server I/O graph.

Notable Times
File Listing Start/End: 6:50AM/7:05AM
Download Start/End: 7:05AM/8:00AM

As shown in Figure 1, CPU usage is slightly elevated for the duration of the process, but never exceeds 10% for this single core system. Server load oscillates at the beginning of the file listing process as well as during and after the file downloading phase of the backup, but remains relatively low (< .1). We can see that the eth0 out metric corresponds to the file download times, peaking at ~750 KB/s of outbound network traffic. Memory usage increases marginally during both phases, presumably due to the FTP server managing the listing and file transfer operations. The I/O profile in Figure 2 shows an increase in latency during the download phase of the backup. At the peak (7:45am), there is a > 4.5ms I/O wait time. The baseline for this system before the test began was between 0.5ms and 1ms, so there is an obvious increase, but the real-world impact of an increase of this magnitude is unknown. Finally, Reads increase during the download phase, but largely stay below 10 reads/sec, peaking briefly at 20 reads/sec.

2. Five Concurrent Activations – FTP

Figure 1. Server load graph.

Figure 2. Server I/O graph.

Notable Times
File Listing Start/End: 12:15PM/12:30PM
Download Start/End: 12:30PM/1:25PM

The profile for five concurrent FTP activations follows the same general trends as the single FTP activation discussed previously. The metrics peak at slightly higher values, but multiple concurrent FTP activations do not appear to cause a linear increase in resource consumption.

3. One Activation – SFTP

Figure 1. Server load graph.

Figure 2. Server I/O graph.

Notable Times
File Listing Start/End: 7:05AM/7:30AM
Download Start/End: 7:30AM/10:00AM

In Figure 1, we can see that CPU usage is negligible for the duration of the process. Server load oscillates during both the download and listing phases, but remains low overall (< 0.1). As expected, eth0 out is elevated for the majority of the downloading phase, peaking at more than 1,500 KB/s. There is an anomaly from 8:05am to 9:55am, where the download speed is very low. Further investigation is necessary here to determine the root cause of this delay.

In Figure 2, we can see that there is an increase in I/O Wait Time during the file listing process but, like the FTP activation, it stays below 5ms. The Average Queue Size peaks over 200 during the download phase which is higher than the FTP activation tests. It’s difficult to say if this is a result of our testing or if we were competing with another VM on this host for disk resources.

4. Five Concurrent Activations – SFTP

Figure 1. Server load graph.

Figure 2. Server I/O graph.

Notable Times
File Listing Start/End: 2:55PM/3:10PM
Download Start/End: 3:10PM/4:05PM

As shown in Figure 1, the CPU, Memory and server load metrics follow the same profile as a single SFTP activation. Aside from the Average Queue Size, all of the other metrics are right inline with the single SFTP activation. The Average Queue Size for multiple concurrent SFTP activations peaks lower (< 125) than the single activation (which was > 200). It’s difficult to say if this is a result of the backups running slower because of the concurrency or if there was another VM on this same host using more disk resources during this test.

As with the single SFTP activation, it does not appear to cause a linear increase in resource consumption.

Conclusion

These tests produced some interesting results, but they are fairly high-level and were only performed once. As a result, they should be viewed with some amount of skepticism.

The three biggest takeaways that were found during this analysis are:

1. Multiple concurrent activations do not produce a linear increase in server resource consumption for either SFTP or FTP.
2. The profile of SFTP and FTP backups are very similar in terms of network, memory, CPU and I/O load.
3. The most load on the server is during the download phase. For CodeGuard, that should mean the Pulls should not put nearly as much strain on the servers as Activations.

After this round of testing, there are still some other open questions, like:

• What is the real world (i.e. user) impact of the elevated load and I/O wait times?
• Would using dedicated hardware change the outcome?
• Do the performance characteristics change for much larger sites and longer-running jobs?
• Would using a different FTP server (Pure FTP vs Pro FTP) have made a difference?

- Jonathan Manuzak, Director of Engineering

CodeGuard Database Backup Load Testing
Version: 1.1 / Date: 25 April, 2013

Overview

The goal of this test is to determine what impact the CodeGuard service has on a host server during a MySQL database backup. The results contained in this document were gathered after a series of four tests paralleling the tests previously done on FTP and SFTP backups by Jonathan Manuzak (see other document). They are by no means conclusive and real-world results will vary based on the composition of the websites being backed up and the hardware and software running the underlying host server.

Test Methodology

From the perspective of the CodeGuard backup service, there is a single phase that occurs on a remote server: running the ‘mysqldump’ command. Running this command includes using the MySQL client, either remotely or tunneled over SSH, to simultaneously extract the database tables to a flat file format and transfer that file from the remote server to the CodeGuard service. Currently ‘mysqldump’ commands are run with the options ‘–quick’, ‘–single-transaction’, and ‘–skip-extended-insert’. These configure the operations one row at a time, attempt to ensure consistent state of some types of tables, and avoid multiple-row insert statements, respectively. Executing a mysqldump with different options may result in different performance characteristics.

Pull Definition

Pull: All database backups are of this type. The entire database is downloaded in via the ‘mysqldump’ command, committed to a git repository, and uploaded to Amazon S3. Unlike website file backups, incremental downloads are not supported by mysqldump, so the entire database is downloaded from the remote server each time, although the backup is ultimately incremental since it is stored as a commit in a git repository. Test Database Only one database was used for this suite of tests. In cases where concurrent backups were taking place, the same database was downloaded simultaneously. While a somewhat contrived test setup, this to some approximation simulates other processes accessing database tables concurrently with backups.

Test Database

Only one database was used for this suite of tests. In cases where concurrent backups were taking place, the same database was downloaded simultaneously. While a somewhat contrived test setup, this to some approximation simulates other processes accessing database tables concurrently with backups.

Database Statistics
Size: 2723 MB
Row Count: 856262 (21 tables)
Type: Real MySQL database from language-learning website.

Test Host
Server: The host used for testing was a RackSpace CloudServer.
OS: CentOS 6.3
MySQL Server: 5.1.67
Memory: 512MB CPU Cores: 1

Database Backup Testing Results

The graphs below illustrate the results of each test. Following each are notes discussing the findings.

Metrics and Definitions

• CPU usage: System: Percentage of CPU usage by system processes.
• CPU usage: User: Percentage of CPU usage by user processes.
• % Memory Used: Percentage of system memory used.
• eth0 in: Network transfer in from the public network connection in KB/s.
• eth0 out: Network transfer out to the public network connection in KB/s.
• Server Load (Last 5 Minutes): The numeric representation of the load on the system for the last five minutes. This is a unitless amalgamation of different metrics but, for this system with a single core processor, loads less than 1.0 are acceptable. Loads above 1.0 indicates that processes are waiting for CPU access. More information can be found here: http://blog.scoutapp.com/articles/2009/07/31/understanding-load-averages
• I/O Average Queue Size: Weighted number of milliseconds spent doing I/Os. This can provide a measure of both I/O completion time and the backlog that may be accumulating.
• I/O Wait: Time in milliseconds spent waiting to perform I/O operations.
• I/O Reads / second: Number of file system reads per second.
• I/O Writes / second: Number of file system writes per second.

1. One Backup – MySQL Direct Connection

Figure 1a. Server load graph. View on ScoutApp

Figure 1b. Server I/O graph. View on ScoutApp

Notable Times
MySQL Dump Start/End: 6:55AM/7:22AM

As shown in Figure 1, mysqldump is neither CPU-intensive or memory-intensive for a single database backup. Server load remains below 0.1. I/O is affected more significantly, as mysqldump is a read-intensive operation. Reads peak at 55/s with latency topping 7 ms. Ethernet output (eth0 out) averages between 1500 and 2300 kB/s. The entire process completes in 27 minutes.

2. Five Concurrent Backups – MySQL Direct Connection

Figure 2a. Server load graph. View on ScoutApp

Figure 2b. Server I/O graph.  View on ScoutApp

Notable Times
MySQL Dump Start/End: 10:28AM/1:33PM

Concurrent backups show a similar pattern with the main impact of concurrency causing all backups to take more time. Peak latency peaks at 13 ms, higher than the single backup, but peak reads/s are actually lower likely due to inefficiencies introduced by multiple processes competing for I/O access leading to more HD seeking. Network output also varies between 1500 and 2300 kB/s with a sharp break at half an hour into the backup lasting for 5 minutes. I do not yet have a good explanation for the pause in the backup process. Network output also lags the start of the backups by about 20 minutes. This means that transferring of the mysqldump file to CodeGuard is not occurring during this time.

The entire backup process completes in 3:05. This is 1.37 times the length of 5 consecutive 27 minute backups, giving an idea of the performance impact of the competition for the shared database resource. However, server performance does not appear to be significantly impacted.

3. One Backup – MySQL Tunneled over SSH

Figure 3a. Server load graph.View on ScoutApp

Figure 3b. Server I/O graph.View on ScoutApp

Notable Times
MySQL Dump Start/End: 9:28AM/9:53AM

A single backup tunneled over SSH mirrors the performance of a MySQL direct backup closely for all metrics. Completion time is very slightly faster at 25 minutes, although this was not tested for reproducibility. See discussion of the single MySQL direct backups for more information of the overall metrics.

4. Five Concurrent Backups – MySQL tunneled over SSH

Figure 4a. Server load graph. View on ScoutApp

Figure 4b. Server I/O graph.View on ScoutApp

Notable Times
MySQL Dump Start/End: 1:47PM/4:37PM

Multiple concurrent MySQL backups tunneled over SSH perform slightly better than concurrent MySQL direct backups. Notably, the pause in the process seen at 40 minutes in direct backups does not occur.  Peak I/O wait time is 6.3 ms vs 13.2 ms for MySQL direct. Network average transfer speeds are lower, possibly due to compression on the SSH tunnel as well as the lack of a break in the process. Despite the lower network speeds the backups still completed slightly faster at 2:55 vs. 3:05 for MySQL direct.  Similarly to the MySQL direct test, the concurrent backups completed in 1.4x the time of 5 sequential 25-minute backups.

Conclusion

These tests of database backup performance produced solid high-level information on the performance metrics that predict performance of a mysqldump, as well as the load it places on a server, but given the simulated nature of the test system they must be viewed with some skepticism.

Given the above caveat, analysis of the test results shows that:

• MySQL database backups are almost entirely I/O bound with very little CPU or memory usage.
• MySQL direct and SSH tunnel behavior is almost identical for single backups.
• Concurrent MySQL backups tunneled over SSH are slightly faster than MySQL direct backups and appear to have better sustained performance and lower impact on remote server performance.
• There is a slight penalty to running concurrent backups of the same database vs. sequential backups.

Further questions not fully addressed by this testing:

• What is the real world (i.e. user) impact of the elevated load and I/O wait times?
• Would using dedicated hardware change the outcome?
• What is the impact of backups to real world MySQL database performance. Does backing up certain tables cause significant slowdown in database performance that would impact user experience?
• Would different MySQL dump options cause different performance profiles and are there ways to optimize the process further with compression of the data either over SSH tunnel or with MySQL?
• It is possible that the limiting factor is network bandwidth in some phases of the backup. It is unknown what the impact to I/O performance would be in a system with a higher network bandwidth to I/O bandwidth ratio. 

-Randall McPherson, Sr. Engineer

Is Your Website Hacked?

Your website might be hacked right now. “My website hacked?”, you think. Yes, your website. “But I’m not a major media outlet, I don’t process billions of e-commerce transactions, and I’m not a federal agency”, you say. If you have a website, it may have been compromised. “But if my site was hacked, it would show up on Google’s blacklist, right?” Wrong. “But if my site was hacked, the hackers would deface it and I would see it, right?” Wrong. “But if my site was hacked, my web developer would know it, right?” Wrong. “But my host protects me, right?” Wrong.

You are alone and your website is your responsibility. Your website is valuable because of the reputation you have established with online registries and because it sits atop a webserver that can be manipulated. To explain why someone would want to hack your quiet corner of cyberspace, I will use an analogy, comparing your website to a cement plant. I’m not joking.

Your Website is Like a Cement Plant

Think about it this way – your website is like a cement factory connected to a power plant. Please follow along – I promise it won’t be too painful.

The cement factory is known in the community and has established relationships. It is trusted. People have bought cement from it, it is in the yellow pages, and therefore, viewed as a legitimate business entity – because it is one.

Powering the cement factory is a power plant, on the premises. This power plant produces electricity that is used for crushing rocks and keeping the lights on at the plant, but that electricity could be used for anything, if it was accessible.

If a criminal broke into the plant and found a way to harness the electricity of the power plant, would he want to be found? If that criminal figured out a way to sell that electricity on the open market, would he ever want the cement plant to know? Of course not – he would want to utilize those assets for his purposes as long as possible.

Should a cement plant be commandeered, it could look similar to a website hacked. I realize this is a tough analogy – but give me just a bit more to explain. The reputation of the cement plant and the goodwill it had built in the community would allow the criminals to continue to operate it, and as long as they didn’t post signs that they had taken over the plant, no one would know. Contrast this with the criminals building a new fake cement plant. Permitting, construction, time, energy, and money would be poured into the new facility. And even once it was finished, if it didn’t produce cement, people would doubt its authenticity. Building and running a cement plant is hard work, and it is tough to fake!

The crucial first step to repurposing the all important power plant at the cement plant is taking over the plant and tricking everyone in the community into believing the plant has not been taken over. Imagine if cement plant could be overthrown without the employees even knowing. This happens everyday to a website hacked. The server administrator doesn’t know. The webmaster doesn’t know. Website visitors don’t know. The problem is that no one knows it has been compromised other than the criminals.

Now the cement plant has been taken over – but surreptitiously. And now the criminals run cables to the power plant on site, and start selling that energy on the open market. Buyers of the energy use it to power factories to illegally construct weapons, illicit drugs, etc. The point is that if you buy energy from an overthrown cement plant, you are most likely up to no good.

What are the criminals doing with my hacked website?

Right now, millions of websites are compromised. And their owners do not have a clue. This is because the website is valuable only as long as its owner does not realize it has been compromised.

Criminals Are Using Your Reputation

Establishing a trusted reputation for a URL & IP address doesn’t happen overnight, so spammers rely on websites with trusted reputations to elude spam filters and deceive recipients into clicking on the links within them. By using your website to redirect victims of phishing attacks, you become part of the problem. Your reputation is used as part of the scam, and all without you knowing it. The recent WordPress brute force attacks are an example of the first step – compromising your site without your knowledge. Once the brute force attacker figures out your username and password, do you think they are going to change it? Not a chance.

Criminals Are Using Your Webserver

The server that powers your website isn’t that different than the laptop or desktop you use everyday. It is just dedicated for a single purpose, which is responding to requests as visitors to your website make them with clicks of the mouse or form submissions. Your webserver can be used in a multitude of ways; think how many different ways your laptop or desktop can be used – and extend this logic to your webserver. To list a few: your hacked website could be used to execute a DDOS (distributed denial of service) attack on your government, companies, or individuals. DDOS isn’t complex; just imagine Amazon’s website around Christmas. Lots of visitors, and lots of requests. DDOSing isn’t nice. It is an attack, and your hacked website could be part of a DDOS right now. Your hacked website could also be used for URL redirecting.

URL redirecting is when a criminal uses your hacked website to take visitors to another website of the criminal’s choice. An example is below.

This is happening millions of times everyday. HST Egypt is a security products company, and right now it has a hacked website that is a part of a weight-loss diet pill scam.

What Can I Do?

At CodeGuard, we are sick and tired of this virtual blight, and aim to stop it. What can you do? Sign up for CodeGuard, right now. We are the only way you can know if your site has been compromised via our ChangeAlerts. And we don’t leave you high and dry at that point. If your site is compromised, we fix it with the click of a button, and then we will patch whatever security vulnerability your site has. Just. that. simple.

Ignorance is bliss. But now you know. Do your duty, and ensure that your website isn’t part of the problem, but part of the solution.

CodeGuard ChangeAlerts Provide Peace of Mind

If you own a website, you should know if it has been compromised. With CodeGuard ChangeAlerts, you can have that confidence. Over the last three years, we have monitored thousands of websites and observed billions of file changes. Billions. We realized that webmasters cannot easily digest and comprehend the volume of changes to their sites, if that information is provided in a firehose format, with a list of files added, modified, and deleted.

So we conducted focus groups and listened to our valuable customers. They told us what they wanted to see in the ChangeAlerts, and helped us find a better way to display information that would allow them to quickly gain peace of mind that their website was OK.

Many of our customers, however, liked how the information was provided to them. And for those customers, we want them to be able to continue to enjoy ChangeAlerts as they want them. Within the website settings tab, under Email Notifications, the Legacy option provides just that.

We are excited to announce a new format that brings the best of the Legacy option and the best of the new Summary option together.

Extended ChangeAlerts Now Available

Last week, we released our new ChangeAlert emails that allow customers to better stay on top of which files are changing on their website. We got great feedback that while the new emails helped tremendously to provide greater visibility into the important files changing, the old format was helpful to quickly scroll to see *everything* that had changed. So we’ve incorporated this feedback, and now offer three ChangeAlert styles: Summary, Extended, and Legacy.

The Summary format is what I described in the ChangeAlert update blog post, with sections for Totals, Overview, Website Files, Media and Other. The Extended format differs from the Summary in two ways. First, all website files that have changed are listed, instead of just the first ten. Second, media and other files that have changed are listed underneath the Media and Other. In the Summary format, there is no visibility into Media and Other files, due to their decreased importance.

AP Hacked: Twitter Account Compromised at Approximately 1:07PM EST

In a developing story, it seems the Syrian Electronic Army targeted the Associated Press and succeeded with the AP hacked. The Associated Press’s twitter account was compromised and a tweet stated: “Breaking: Two Explosions in the White House and Barack Obama is injured”.

Sam Hananel confirmed this, stating: “Please Ignore AP Tweet on explosions, we’ve been hacked.” But the POTUS is fine and the attacks did not occur. Spokesman Jay Carney says that “The President is fine”.

The DJIA plunged almost 130 points after the tweet went live, but has rebounded since.

Whether it is a media outlet’s twitter account or a website, on an almost daily basis, we are reminded of how vulnerable our assets are. The twitter compromise was most likely a phishing attack directed at AP writers. Whether the writers entered their full credentials or were the victims of session riding, it is too soon to know.

-David

New ChangeAlerts Help You Track File Changes

Last week, we released an update of our ChangeAlerts, a feature many of our customers value more than our secure cloud website backups. ChangeAlerts notify you when something on your website has changed, and are invaluable for detecting if your website has been compromised. URL redirects to scam sites, drive-by-download malware, and Blackhat SEO Spam (“Pharma Hack”) all rely on changes to your files. And with ChangeAlerts, now you will know if you have been victimized.

The ChangeAlert email summary is intended as a quick way to gain insight into what is happening on and to your website. If anything appears unusual, view the detailed information available once you have logged into your website. This is an abbreviated email summary and not exhaustive.

Key Sections: Backup Total, Overview, Website Files, Media Files and Other

Backup Total: High-level for all files

Under the Backup Total section of the ChangeAlert, the summation of files added, modified, and deleted is displayed. This provides a quick snapshot of what is going on with your site.

Overview: static and Dynamic File Granular

The Overview section provides more granular information into important static and dynamic files you should keep your eyes on. Static files are those rendered in the browser, while dynamic require a server to generate the output. Depending on your website and configuration, there are likely other file types that are important to you. This list is not exhaustive, but serves as a starting point for the vast majority of our customers. Html, css, javascript, htaccess, php & ruby files are those we place in the abbreviated overview. If any of these change and you or your developer did not change them, contact us immediately as you may have been hacked.

Website Files: Individual file Listing

In this portion, you can view the names of the files that have changed. The old ChangeAlerts resemble just this portion – pure additions, deletions, and modifications, along with the truncated filenames. We will list up to ten of the file changes here, with the rest viewable upon logging into codeguard.com.

Media Files and Other: The rest of your content

Changes to images and video files are much less likely to be problematic, and therefore, are listed last. In this portion, you can view images, videos, and all other file types, which are grouped under “Other”.

ChangeAlerts provide industry-leading visibility into how the content on your website is changing. Stay on top of your site, and gain peace of mind, knowing that if a site is compromised or a malevolent employee defaces the site, you will be the first to know it.

-David

An attack directed at Yahoo! Mail users is now being utilized to drive traffic to scam and phishing sites. In the middle of it all are innocent and legitimate businesses whose websites have been hacked. The hacked websites serve no malware, do not appear on blacklists, and pass McAfee and Symantec security scanners. Sites built on WordPress and Joomla have been discovered as hosts.

Whether the Yahoo! Mail users were compromised via the XSS (cross site scripting) vulnerability announced on Jan 7, 2013, or via something more sinister, such as a server compromise, the end result is the same: Yahoo! mail accounts are being used to send email to their address books, which greatly increases the likelihood of deliverability. Recipients of the emails click on the links, since they are from a trusted source.  If the recipient is a Yahoo! Mail user, there is a good chance that the website they click on, once rendered, will compromise their account and send emails to their contacts. The recipient, in addition to serving as a distribution vector, is also a target.

The scam: weight-loss products that fly underneath the radar of the FDA, only occasionally getting attention, because, to paraphrase the FDA, since these scams do not represent severe health threats (aka bubonic plague), with the limited time & resources of the FDA, they have more important things to do. How does the scam work?

Step 1: Email from a friend, pure SPAM, or Google Ad

The email shown below is what it would look like after your email service provider had received enough complaints to give you more information. This email was delivered to my inbox because it came from a trusted sender. Gmail is one of the best email providers, and prevents more spam than the leading competitors. This warning did not initially appear when the message arrived, however; it took time for Gmail to gather information and then start reporting the message as potentially harmful.

Another way the exploit could be initiated is that you search for Dr Oz within Google. Three ads pop up, and all appear innocuous. One even says “www.womensdigest.org” – that looks safe, right?

Whether you received an email or you conducted a Google search, the outcome is the same, you end up clicking on something that takes you to a landing page.

Step 2: Fake Health Article Landing Page

You search for the latest Dr. Oz promoted snake oil in google and click on a promoted ad, or click on a link in an email – either from a friend or pure spam. Almost certainly, you are taken to a landing page that appears to be a consumer health or women’s health publication. On this fake landing page resides an article about the latest diet drops or pills, with a video of Dr. Oz. He has promoted the following over the last five years: Acai Berry, Raspberry Ketones, Green Coffee Bean, HCG drops, and most recently, Garcinia Cambogia.

Giveaways: Look at the URL. In the example below, it is actually “http://womenshealthmag.com-most-popular-deal.com/womens_healthgarcinia-a/garcinia1-index.php”. A quick glance at the logo would lead one to believe it is the Women’s Health site. And a quick glance at the URL would reinforce this. This is a subdomain of “com-most-popular-deal.com”. Tricky, huh!

Step 3: Link to e-Commerce Diet Pill site

There will be multiple links on the fake landing page to a webpage where you can place your order for whichever fake product was promoted on the landing page. These days, the rage is Garcinia Cambogia. While the links will have different titles, and seem to reference different articles or sources, they will all go to the same place, the e-Commerce site. Celebrity endorsements are common on these landing pages as well.

Step 4: Buy now at e-Commerce site

The last part of the chain is a website, likely to be laden with fake “trustmarks” (McAfee Secure, BBB Accredited, etc), and fake testimonials, that will collect your credit card information in short order. The landing pages are simple, with a limited form to collect your name, address, phone, and email. On the next page, you will be prompted for credit card information. Either on the first or second page trust marks will appear, and if you right click on them you will see that they are a single image, not a verified trust mark.

Be careful and don’t trust diet products on the web

These scams abound, and the legitimacy of Dr. Oz is a key piece to the scams identified thus far. Fake trust marks from Symantec, McAfee, and GoDaddy are common. Whether you have interest in the weight-loss product or not, if you are referred to a site that resembles a consumer health site, with an article about a new breakthrough product, accompanied by a video of Dr. Oz, the site is a fake intermediary, set up with the goal of directing you to purchase the product. There will always be a link on this page that takes you to a separate site on which you can purchase the product.

Just because you got an email from a friend, or clicked on a Google ad, does not mean the end destination is safe. Even if the product looks appealing, do not enter any personal information, log out of your facebook/email/twitter accounts, and if anything suspicious occurs (if you are a Yahoo! Mail user, reset your password).

What about Yahoo! and CodeGuard?

We came across these scams because websites and webservers are being compromised to make this process work. Sitting inbetween Yahoo! mail recipients and the landing pages are redirecting webpages – with legitimate reputations. These redirecting pages are used so that as the landing pages and e-Commerce sites are reported and disabled, the ruse can continue with new landing pages receiving traffic.

The key question is: when will the FTC and FDA step in? With limited research, it is blatantly apparent that a multi-level marketing scheme based around products marketed through Dr. Oz’s television show, is being used to defraud consumers.

Compromised Websites: WordPress & Joomla

We will explore how the compromised websites are being used. A member of our team received an odd email from a friend, and after firing up a virtual machine and turning off javascript within the browser, pasted the link: http://www.iolcus.gr/kfaiyjg/ddswjet. A rapid redirect occurred, and this is the website that was displayed.

The subject line of the email was “Breaking news”, so the landing page seemed to be appropriate. A break through weight-loss product. If that is real, it is definitely breaking news. One big problem – the URL we input was www.iolcus.gr, not mxxfox.com.  Iolcus.gr’s website looks like this:

This is a WordPress site, and from this screenshot, you cannot tell the site has been compromised. The only way that the redirect could have functioned is if someone had access to the folders on the webserver. There are countless ways for someone to get access, so we won’t spend time on that part now. The important part is that I highly doubt Iolcus had any idea they were a part of a diet-product scam. Another site that we observed to have been compromised was a Joomla! site.

This site was used to redirect to similar landing pages as we have seen already. Some cleverness exists in the process of the redirects that makes it difficult to follow everything that is happening.

Summary: When there’s smoke . . .

The processes used to market and sell these break-through diet products are complicated. Different companies seem to be involved in various aspects, with some sending SPAM emails, others using vulnerabilities to compromise webservers or commandeer user’s accounts without their knowing, and even others using Google Adwords. Customers are driven to landing pages that mislead and deceive. And then customers click and pursue their weight-loss dreams.

At this point, while there is a mountain of evidence that signals something nefarious about not just the marketing techniques used, but the underlying products themselves, we won’t comment more than to offer this one unique tidbit. Raspberry ketones were non-existent before Dr. Oz referenced them on February 6th, 2012. Since then, things seem to have progressed. You can research yourself to see how green coffee beans, garcinia cambogia, hcg drops, african mango, and other miracle products have found their way to the market.

WordPress Brute Force Attacks: Quick Simple Solution

Protect yourself by limiting wp-login.php to your IP address

If you are a WordPress user, this is all you need to know – the WordPress brute force attacks that occurred last week can be mitigated with one simple technique: restricting which IPs can access your wp-login.php page. That’s it. The reason last week’s WordPress brute force attacks were so effective is that rather than one single computer IP address attempting to guess your password, tens of thousands were used, which means that the attack could occur without sounding some of the traditional alarm bells.

But why risk relying on security plugins that may fail you when you can fix things yourself? And why install new software when the fix will take 2 minutes? Lastly, why rely on subpar solutions that can still cause your server to crash, due to the strain of rendering the wp-login page? How about you implement a solution that almost effortlessly rejects the unwanted advances in the most resource-effective way.

This is all that you need to do to protect yourself from a WordPress brute force attack:

1. Identify your IP address (http://www.myipaddress.com/)
2. Log into your server via FTP/SFTP or your hosting control panel’s file manager. HostGator’s file manager is below.
   
3. Navigate to your .htaccess file (If it doesn’t exist, create it with a text editor)
   
4. Add this to the beginning (replacing the xxx.xxx.xxx.xxx with your ip):

   <files wp-login.php>
   order deny,allow
   deny from all
   allow from xxx.xxx.xxx.xxx
   </files>

Verify that your wp-login page cannot be accessed from a computer other than the one you are using. To do this, try using your phone or a friend’s computer. We are not claiming that this will protect you from all of the nefarious characters on the internet. This will, however, protect you completely from a WordPress brute force attack. Thanks to James Dunn from wpmu.org for providing much of the guidance.

Background Information

Last Thursday, April 11th 2013, hundreds of thousands (at least) of website owners across the globe became victims of a sophisticated attempt to gain access to the portions of their webservers controlled by the WordPress Content Management System (CMS). WordPress is installed as a subordinate on the Linux operating system, usually below software used by shared hosting providers to provide control panels. The leading software used is cPanel and Plesk, produced by cPanel, Inc, and Parallels, Inc, respectively. Custom made hosting software is also used by hosting providers. Notable users of each type:

• cPanel: HostGator, Namecheap, WebHostingBuzz, BlueHost, A Small Orange
• Plesk: Hostmysite.com, Media Temple
• Custom-made software (GoDaddy, DreamHost, Endurance International vDeck properties: FatCow, iPage, etc). 

Vulnerabilities exist at each level of software installed: server (Linux), hosting provider (cPanel), and CMS (WordPress). I want to mention each level before focusing on the CMS and WordPress brute force attacks. To address only the CMS does a disservice, since a false sense of security can be created by closing every single window in the house if the front door is left open.

Server/Operating System/Hypervisor Vulnerabilities

Protection at this level is absolutely the responsibility of the hosting provider. WordPress users have no ability to control this, and only through picking a hosting provider focused on security can vulnerabilities at the hardware, operating system, and hypervisor be addressed. This equipment and software serves to create the base layer for hosting provider software to run.

Web Hosting Control Panels: cPanel, Plesk, vDeck, Custom

Locking down the control panel may be the responsibility of the hosting provider, or if you are running a VPS or dedicated server, it’s on you. Vulnerabilities do exist at this level, and have been exploited. Thousands of sites were hacked after a Parallels Plesk exploit was utilized - http://krebsonsecurity.com/2012/07/plesk-0day-for-sale-as-thousands-of-sites-hacked. It is common knowledge that no system is impervious, and that as software increases in usage, the benefit to hacking it increases to miscreants. Software exploits, while possibly indicative of poorly written code, are more often a sign that the software has become popular enough to be a target.

Content Management Systems: WordPress, Joomla, Drupal, etc

All content management systems have vulnerabilities. WordPress brute force attacks, however, just come knocking at the front door, again and again. By simply guessing at a website owner’s username and password, the only restricting factor is the speed the website owner’s computer can respond “Yes” or “No” and the speed at which the attacker can make requests.