We are often asked how well our services perform relative to competitors. The biggest competitor for end customers is “do nothing”. Imagine how many drivers had auto insurance before it was made compulsory by their respective states! So relative to doing nothing, we work infinitely better!
For end website owners – businesses, bloggers, non-profits – depending on the framework used to construct their website, there are many options, assuming that they are doing something. If a customer has a WordPress site, there are many, many, many plugins that they can use to back their site up. If the customer chooses Drupal, there are modules they may rely upon. CodeGuard’s SFTP/MySQL direct/SSH product line has strengths and weaknesses when compared with content management system (CMS) specific solutions. We can explore those specific comparisons later.
What about website hosting companies that offer 3rd-party website backup services? Do they sell plugins across relatively generic hosting plans? We haven’t seen any that do this. What we have seen is that hosting companies like to sell services like ours that are essentially platform-agnostic, since they access file content on servers via FTP/SFTP and either directly or over SSH to MySQL databases.
Until recently, we had not encountered a major competitor in the United States in the website hosting market, and did not know how our service performed compared to others. GoDaddy announced a website backup partnership with Singaporean email, mobile, and website backup company Dropmysite (DMS). So we thought it made sense to see how well their product worked relative to ours.
Advertised Feature Comparison
To start, it will be helpful to take a look at the features that DMS advertises and how those compare to CodeGuard. Here is a chart that illustrates the features of both.
The two services don’t look altogether that different when comparing features. Sure, there are more X’s next to CodeGuard, but the big categories seem to be fulfilled by both services: daily backups, automatic restore, downloadable zips, etc. If we were comparing two restaurants, what we could say at this point is that both have menus that advertise what appear to be similar dishes. Immediately the question then becomes – how good are those dishes? Anyone can print a menu. Preparing delicious food is tougher.
Quantitative and Qualitative Testing
We decided to perform quantitative and qualitative tests. The quantitative tests would involve testing the amount of time for backups, restores, and processing of zip files to complete, as well as testing each system for special case handling (e.g. symlinks). Qualitative testing would involve examining the user interfaces, and comparing how effective the interfaces were in either preventing known issues (e.g. CMS-driven sites without database backups) or assisting customers once they had issues (e.g. credential change causing backups to fail).
Across the six performance tests, CodeGuard performed better than DMS on five of the six. The average performance margin of victory was 1,491%.
The figure above shows the results of the six tests, broken out in various units to make it easier to digest. Under “Initial Backup Results”, one set of tests was completed for each website type: a small WordPress site, a larger WordPress site, and 2GB blog with a number of photos. The “Restore Results” were two tests involving restoring a single file and all content for the 2GB Photo Blog. The “Zip Results” shows the amount of time to complete the zip request for the Photo Blog.
Here is a graphical representation of the results for the Initial Backup testing.
Here is a graph of the results of the restore tests.
And lastly, the zip completion results.
If the results of these six tests are taken in aggregate, this is the output.
Of the 6 tests, CodeGuard performed better in 5 out of 6, with an average margin of performance difference of 1,491%. DMS performed better in the single file restore, likely because they are not storing backups in git repositories, thus making it easier to access small numbers of files.
Is CodeGuard 1500% better than DMS? There are about 40+ other ways we will compare the two products, but the most important set of tests are the ones we could not conduct. CodeGuard might be infinitely better than DMS. If a customer needs a file and a backup service doesn’t have that file, it doesn’t matter how fast backups are, restores are, or zips are processed. To perform valid tests on reliability across the entire DMS storage system would require more access than we have.
In-Depth Usage Comparison
The user experience, restore functionality, and architecture will be evaluated. Within the user experience, we will examine not how easy to navigate each web interface is, but evaluate critical functionality or the lack thereof, that inexperienced customers need to successfully use a website backup service.
CodeGuard’s product has grown since launching in May 2011 at TechCrunch Disrupt, where we won the Audience Choice Award. Many companies have a support arm that interacts with customers who have difficulty or questions about their products or services. At CodeGuard, we call that function “Product Development Feedback”, and not “Support”. We view the word “support” as a reminder that your product is built so poorly that customers actually need further assistance to use it. For enterprise products, this does not hold true. But for many SaaS products, it does.
At CodeGuard, Product Development Feedback (PDF) is part of what fuels our ongoing improvements and enhancements to our product. We have been improving our product in this fashion for over four years now, and the list of differences we are about to expound upon were quite literally not built in a day.
1. No On-demand Backups
Perhaps you just finished important work on your website and do not want to wait until your scheduled back-up time because you don’t want to risk it. Or you want to backup before upgrading a CMS or plugin. DMS does not have an on demand backup feature, while CodeGuard does.
2. No Credential Change Protection
An important aspect of user experience is protecting customers from themselves. Customers don’t spend every waking hour in your software-as-a-service (SaaS) tool, and you should realize that there might be operations they can perform that actually harm their sites. Protocols, hostnames, root folder paths, and credentials can be changed within DMS without any warning as to the ramifications with restores. Do customers want to learn at the point when they are performing a restore that a folder they were backing up before is unaccessible for that very restore? We don’t think so. Furthermore, we were able to verify that changing the backup protocol from SFTP to FTP caused a restore of the initial backup to fail. #FAIL
3. No File-level Exclusions
While excluding directories is helpful, oftentimes customers want to exclude particular files. Perhaps they have a few large files and don’t want to back them up, but desire the rest of the content in that folder to be backed up. CodeGuard allows you to include or exclude individual files within the directory selection structure. DMS does not allow for individual file inclusion or exclusion.
4. No Directory Symlink Handling
Linked directories are just skipped during the backup process with no warning or indication that their contents are not protected. What are symlinks? The simple explanation is that they are tool used by many hosting providers to give customers easy access to content areas outside of their home directories or to make navigating to deeply-nested directory structures more convenient.
5. No File and Database Association
To make restoration of a particular website easier, CodeGuard allows file backups and databases to be associated with each other. Within the user interface, under a single URL, content for both the files and databases can be found. This may not seem significant, but when time is of the essence and a restore needs to be performed, it is one less thing to worry about.
In addition to associating files and databases, CodeGuard provides recommendations via automated detection of your particular CMS, so that you can be sure you are associating the right file structure and database. This prevents the following headache scenario: “I need to restore my blog! Wait, which database belongs to this website?! Was it “mywpsite_mysql1415” or “mywpsite_mysql1451”?’
6. No Changed File List
CodeGuard ChangeAlerts include information about files added, modified, and deleted, every time we perform a daily backup with changed content. DMS provides an email notification when a new backup is taken, but there is no list of the actual changed files, just a number, which is not helpful when customers want to know the details of what is happening with their site: “Was I hacked? Was htaccess modified?” Was there some unintentional or malicious tampering?
CodeGuard provides these notifications via email and our ChangeAlert iOS app. DMS does not appear to have neither actionable ChangeAlert-type notifications nor any apps for iOS or Android for website management.
7. No Combined Database + File Restore
By working directly with customers and closely monitoring restore feedback, we’ve developed a system that helps novice and first-time users through the restore process for their website. Guiding customers through an *entire* restore of both their website and database while providing recommendations around which versions to use reduces website downtime and frustration.
8. No Test Restore
A backup service is only as good as their ability to restore. Wouldn’t it be nice to be able to test the efficacy of the restore process without potentially disrupting your live website? CodeGuard lets you perform a test restore at your leisure to confirm that the permissions on your website are setup properly, as well as to doublecheck credentials, passwords, etc, are all correct.
9. No Pre-restore Backup
Before CodeGuard completes a full-site restore, we backup the content on the live server. Our pre-restore backups allow customers to return to the state immediately before their restore. When customers are performing restores, they often aren’t sure exactly which prior version to restore.
Without either an automatic pre-restore backup or a manual on-demand backup, there could be a significant amount of content that would be destroyed and irretrievable if a restore (which removes the content) was initiated.
10. No Restore Feedback
CodeGuard asks for feedback after every restore. We want to know when we met the customer’s expectations and when we did not so that we can improve the experience, underlying systems, documentation or anything in between. Even if our internal tracking for a restore says that it completed properly, if the customer provides feedback that it didn’t restore to their expectation, we can assist them, and investigate the underlying issues.
The idea of website setup completeness came directly form restore feedback. Customers weren’t aware that they needed to backup databases for their CMS-driven sites. After investigating why customers were reporting restores weren’t successful, we found that we needed to proactively prompt them to add databases – far ahead of when they would need to perform a restore.
And once the restore has been completed, the feedback from the post-restore survey populates a 50 inch monitor in the CodeGuard network operations office.
11. Request Zip Slow
It takes ~10x longer for DMS to provide a zip file that can be downloaded than CodeGuard (DMS: 20 minutes, CG: 2.25 minutes) Note: This is just the time to produce the zip and make it available for download. This does not include the transfer time required to download the file from the service. Our custom tools allow us to access customer data and process it quickly.
12. Full Restore Slow
Restore is the reason backup exists. Backup in and of itself is meaningless if you cannot access the contents. And when it is time to restore, it should be easy and fast. No one cares if Batman is in shape until a disaster strikes. When that disaster strikes, you better believe Gotham will be hoping that not only will Batman show up promptly and take care of things, but that Batman has maintained a stringent workout regiment during the off-period.
Image courtesy of http://www.comicvine.com/images/1300-4810718
Backup is not altogether different. There are intense crises when something has gone awry, and the product must deliver at that moment. And then there is the lull after the disaster, when the service must perform at the same level, but without customer accountability as to the performance. What I mean is that anyone can tell if a restore performed properly – after the restore finishes, refresh the website and see if it is working.
But how can you tell if the daily backups are working properly? That is a much more challenging problem to solve. Download the zip for each day and compare the contents to the live server? Who has time for that? The reason I bring up the importance of the daily backups is that for the restore to work at all, the right data has to have been backed up in the first place, and then properly stored. The slickest interface in the world and the easiest-to-use service loses all value if the actual data needed isn’t there. Let’s assume that the data is there and a restore is attempted.
In many cases, particularly for customers who are used to managing their website through the CMS interface, a restore is a daunting proposition. CodeGuard’s restore wizard guides customers through the restore process and provides suggestions to ensure that they can get the restore started quickly.
In our testing, for a full restore where only one file has changed, CodeGuard was 6,266% faster than DMS. CodeGuard completed the operation in 24 minutes, including taking the pre-restore backup. DMS took nearly 26 hours.
13. No Differential Restore
The full power of differential backup is felt when a restore is needed. We are able to bring simplicity to a complicated situation through our technology, which intelligently and efficiently discerns what content needs to be replaced on the live server. While DMS uses a form of differential backup, unfortunately, the restores do not leverage the metadata to perform intelligent comparison-based restores. Their restores simply push all of the prior content, which is one of the reasons we are 6,000%+ faster when conducting restores.
14. No Evidence of Custom Quadpipes Tool
CodeGuard has developed a number of advanced tools to ensure that our system is secure, reliable, efficient and performant.
The reason our backups are so much faster than DMS is that we transfer data via simultaneous connections, and do so with code written in Go. We call it “Quadpipes” and it has dynamic download capabilities across multiple connections and sessions. This means that we can increase or decrease the number of connections we make to a particular host based on their performance, availability and capacity.
15. No Evidence of Custom S3GOF3R Tool
Part of the reason our restores are so fast is that we use s3gof3r, which was written in-house and then shared with the world. It is used by MariaDB, Basho, Heroku and who knows how many others! It is a streaming S3 client and possibly the fastest in the world. We are not aware of one that is faster.
16. No Evidence of Custom Zipper Tool
The reason our zips are so fast is probably “Zipper”, which is a parallelized zip download creation tool we built. A straightforward approach would be to download a backup, extract the contents, archive the files and then upload the archive to a place where it can be accessed. That process requires moving and copying content several times, which generates a lot of slow I/O operations on the server and, ultimately a delay in delivery for the end-customer.
Zipper leverages the speed of s3gof3r and the parallelized nature of the git-archive tool to produce a zip from a bare git repository without the need to move and copy the content several times. The result is that we’re able to retrieve the repository, produce the zip and place it in an area that a customer can access with minimal delay.
17. No Evidence of Monitoring Systems
CodeGuard has built several monitoring systems to ensure that our backup and restore systems are working at 99.99+% levels of reliability every day. When operating at the scale that CodeGuard does (~8.5M backups per month), we need automation to ensure that no customers or issues are going unnoticed.
We track our daily backup success rate across all product lines as well as our daily restore success rate across all product lines. Perhaps DMS also has monitoring systems but we weren’t able to find any information to indicate this was the case.
18. No Evidence of Redundancy
DMS advertises a single IP that is used for connectivity. This indicates a lack of redundancy in their infrastructure. If something goes wrong whether it’s the fault of DMS, their host or the datacenter that their host uses, it’s likely that customers will be missing backups.
While CodeGuard utilizes load balancers for web requests to our user interface and a virtual private cloud (VPC) for our outbound connections, we still have multiple IP addresses from which we make connections. Providing all of these IPs to customers allows us to change infrastructure, distribute load, route around localized failures and perform routine maintenance on different aspects of our systems in a way that is completely transparent to our customers. While we do rely heavily on cloud infrastructure, this approach helps to ensure that we’re able to maintain our high levels of reliability.
19. Primitive Initial Backup Server Requests
To gain insight into how DMS constructed their solution, we backed up the same six files with DMS and CodeGuard. We utilized log files to record the specific server requests and these are the results: 80 requests for DMS and 191 requests for CodeGuard. Fewer requests isn’t better in this case – CodeGuard is doing some additional work to detect content management systems and additional configuration to ease future restores. All of that and the initial backup times for CodeGuard are still faster than DMS.
CodeGuard’s server requests for the initial backup process contains five major steps:
- Setup through Web UI
- Start of Backup Process Login & FTP Setup
- Testing for CMS Versions & Configuration
The process for DMS has the same first four steps, though they are accomplished in a different fashion. Most notably, “MLSD” is the listing command relied upon by DMS. CodeGuard experimented with MLSD in our prototype architecture in 2011 but has been using “LIST -a” since then, as it is more widely supported and produces more predictable results.
Ultimately, the server requests being made aren’t that different. The performance gains CodeGuard has over DMS must occur after the server requests are made. Here are the 191 requests for CodeGuard.
20. Primitive Restore Server Requests
The server requests for the initial backup are not that different – but what about the restore? CodeGuard has a 6,000% performance difference in the speed of restores – perhaps some reasons for that performance disparity will become evident by examining the server logs.
The 45 requests in CodeGuard’s restore process can be broken into seven major steps:
- Start of Pre-restore Backup Process: Login & FTP Setup
- Connectivity Test
- Ensure root directory path exists and is accessible
- Start of Restore Process: Login & FTP Setup
- Upload needed files
The process for DMS is different. It contains two major steps:
- Login & FTP Setup
- Upload files and change modification time to prior values
The DMS restore has only two major steps and the core of the process is the second step – uploading all website files. Does the website being restored have 10,000 files? That’s not an unreasonable size for a moderate WordPress blog. Well, DropMySite will upload all 10,000 of those files during a restore whether they were missing, changed or identical. Whether the goal of the restore was to repair one configuration file or revert changes to a botched theme, the customer will be spending a few hours waiting for DMS to overwrite perfectly good files.
The CodeGuard restore process has several additional steps which provide a safe and speedy restore. Before overwriting, deleting or modifying any website content, CodeGuard first takes a backup of the current state of the website. In the event that the customer selected the wrong date to restore to or the restore is interrupted by a hosting issue, they can always revert to a point in time immediately before the restore. DMS does not provide this functionality nor do they allow on demand backups for customers to initiate this themselves. This leaves the customer exposed with a gap of hours, days or weeks since their last backup.
The core CodeGuard restore process leverages the power of differential backups, but in reverse. After performing a new backup, CodeGuard will analyze the website, see what *needs* to be restored and modify the running website content to match. This extra analysis often yields a very small subset of changes that need to be made to restore the website to the desired state. For our hypothetical 10,000 file WordPress site, this may mean that only 1, 100 or 1,000+ files need to be uploaded. Any of those cases would complete much faster than uploading 10,000.
Is the DMS approach simpler? Yes, but just like a go kart is more simple than a Ferrari, one is going to get you where you need to go a lot faster and be a lot safer going down the highway.
21. No Evidence of Intellectual Property
We are not aware of any intellectual property possessed by Dropmysite Pte Ltd. CodeGuard filed a patent in 2011 and was awarded the patent on June 30, 2015, USPTO 9,069,885. The patent is titled “Systems and Methods for Automated Retrieval, Monitoring, and Storage of Online Content”. Here is a brief description of the patent:
Systems and methods for automated retrieval, monitoring, and storage of online digital content, wherein such content includes source code and files for hosting websites, audio files, video files, data files, system files, image files, or any other content that is typically stored in third party servers. A content retrieval system hosted on a physical server or a cloud continuously monitors user data hosted on a third party server for changes to the data.
The method involves creation of an index list that is updated periodically to keep track of changes to the metadata associated with the user’s content. Such an approach saves time and valuable resources to individuals and/or organizations enabling them to perform periodic monitoring of their data, and enabling rollback to a previous version of their data whenever needed. The system additionally monitors user content for malicious attacks and hacks, and provides notification alerts relating to the same.
22. Unfocused Product Strategy
CodeGuard offers website backup services. DMS offers website backup, email backup, and phone backup, via its dropmysite, dropmyemail, and dropmymobile brands. Strategically, we have focused all of our resources on delivering the most efficient and effective website backup service in the world. DMS is simultaneously deploying a website backup service, email backup service, and a backup service for Android devices.
23. Incorporated in Singapore
CodeGuard is a Delaware C-corp, subject to the laws of the United States. The Delaware General Corporation Law is one of the most advanced and flexible corporation statutes in the United States. We are unsure of how enforcement of statutes works with Singaporean-domiciled companies.
According to the Singapore website http://www.singaporelaw.sg/sglaw, “The legal system in Singapore has received numerous international accolades for its efficiency and integrity. As a consequence of this, there is now wide recognition of Singapore as a leading legal hub in Asia.”
Earlier I explained how CodeGuard views support and actually calls it Product Development Feedback, but when we work with large hosting providers, we have to be malleable and acknowledge that many companies use the term “support” and have entire “support” departments. For this next portion, then, we will intentionally forget what I stated before, and whole-heartedly adopt the word “support”. Let’s see what we can glean in comparing CodeGuard’s support with that offered by DMS.
24. No Evidence of 30-minute Average for Initial Ticket Response
CodeGuard utilizes Zendesk for all customer support interactions, and through Zendesk, we are able to easily track key metrics, such as new tickets per week/day, open tickets, time to resolution, etc. One of the key metrics we track is the time to first response. On average, CodeGuard responds to tickets submitted between 9am-5pm EST within 30 minutes. Tickets submitted outside of that window take longer, but we do have an escalation path for customers in dire need.
Without privileged access to DMS support metrics, assuming they are being tracked, we have to use external proxy information to try to ascertain what ticket response times may be. Two data points does not necessarily make any type of trend. Here are two recent tweets that do not have responses as of Nov 6th, 2015, and one of the tweets says a response was not received for months.
25. No Evidence of 24×7 Support
CodeGuard offers 24×7 support for our customers via hosting partners. This is not an easy task and requires us to work with 3rd party agencies to route after-hours calls. We work with VoiceNation so that our customers can call anytime – night or day, and speak with a live person. Depending on the nature of the call, VoiceNation can call our on-duty engineer and get immediate assistance for the customer.
We do not know exactly how DMS handles support. A call made at 10:00AM EST to the number listed on their website was not answered. The only number we could find is +65 6779 5131, though on the website there appears to be an office in the United States.
During normal business hours in the United States, phone calls to CodeGuard are answered within three rings. Since we use Grasshopper for our phone system, it may actually be a few more rings for the customer or prospect calling us. Calls to our support line have a dedicated person responsible for answering them each day. Calls to the sales line might be answered or may go to voicemail. Since we can’t do all things at once, right now we are more concerned about delivering for our existing customers than gaining a few new ones.
Let’s take a look at the prices for CodeGuard compared to DMS. Since we don’t have access to the pricing offered to hosting providers, we can use the pricing on their website as of November 5th, 2015.
Clicking on the the last plan name, “100GB”, you can obtain pricing for larger storage plans, up to 1,000 GB. Here is a chart with each of the plans along with the price and price per GB.
To make this easier to digest, here is the graphical output of the monthly price per GB.
Starting with the cheapest plan which provides for 10GB, the monthly price per GB is $.25. The price per GB steadily decreases until it reaches its nadir at $.11. Let’s take a look at CodeGuard’s pricing.
A VERY important clarification needs to be made immediately before any useful analysis can take place. On CodeGuard’s pricing table above, the storage amounts are COMPRESSED storage on our servers, not UNCOMPRESSED. Our plans have storage allotted based upon its COMPRESSED size. On the DMS pricing page, the storage numbers they use are for UNCOMPRESSED storage. It doesn’t really matter whether we use a compressed or uncompressed number, we just have to be consistent.
In the testing that was done, we found that the 2.25GB Photo Blog site was 200MB within the CodeGuard system and 2.25GB within DMS. 2.25GB/200MB = 11.25, and for the purposes of comparison we will use this ratio to convert between the COMPRESSED storage size and UNCOMPRESSED. But what if that is the wrong ratio? It is likely that this is not the right ratio, but we are being conservative since the site possesses a large numbers of photos, which are hard to compress, versus a large number of text files, which are much easier to compress. So it is likely that the ratio is even higher.
For the four major plans, Ninja, Ronin, Samurai, and Shogun, the price per GB of UNCOMPRESSED storage can be seen in the following graph.
OK – so what does the pricing look like when CodeGuard’s plans have been adjusted to UNCOMPRESSED storage? Here is the prior chart, with CodeGuard’s plans now added at the bottom.
Even with this data set, it is slightly challenging to compare pricing, although for plans that are utilized at their maximum storage allotment, CodeGuard’s pricing is much lower than DMS. To produce as close to a true apples to apples comparison, we will compare the four CodeGuard plans and their closest DMS equivalent. With the 11.25 Uncompressed/Compressed ratio, the four CodeGuard plans end up with uncompressed storage allotments of 56, 563, 1406, and 5625 GBs.
The largest plan you can purchase through DMS’s website is 1000 GB, so unfortunately, we cannot compare the Samurai and Shogun plans, as uncompressed Samurai offers over 1,400GB and that number is greater with Shogun. Ninja and Ronin, though, have comparable equivalent plans of 60GB and 600GB of storage. The prices per GB/month for the DMS 60 and 600 plans are $.21 and $.12. CodeGuard’s equivalent pricing is $.09 and $.07, which is more favorable by 134% and 80%.
26. DMS is More Expensive
In the graph above, the top black line is the pricing for DMS for the 60GB plan and for the 600GB plan, compared to the equivalent CodeGuard pricing. DMS is clearly more expensive for the equivalent plans.
Summary: CodeGuard Outperforms
According to the tests we conducted, CodeGuard outperformed DMS in every area but one. The restore time was 63X faster. Not 2% faster. Not a few minutes faster. 6,000% faster.
The initial backups averaged 1,210% faster than DMS.
The zip was 800% faster.
And CodeGuard is cheaper. Anywhere from 80-134% cheaper.
Faster. Cheaper. More fully-featured, and built to last by world-class engineers. Hopefully, this blog post is sufficient to answer the question: “How is DMS different from CodeGuard?”. Below is a summary of the information for quick review.