Product Development at Scale

Twitter Facebook

We know that our customers see our service as a critical part of their infrastructure, and this is a responsibility that we take seriously. Unlike other software-as-a-service companies, we don’t have the luxury of hoping that a customer will notice an issue and report it or try again later. CodeGuard provides an automated service and for customers that rely on us for daily backups, that means that we have a 24 hour window, 365 times a year where we have to perform at our very best. Our process for developing and maintaining the service has to reflect that reality.

Just as we have scaled our service to accommodate hundreds of thousands of website and database backups on a daily basis, we’ve had to scale our product development process to consistently deliver high-quality results for our customers while maintaining the 99.99% levels of reliability we have achieved. We can’t haphazardly release features or change functionality without thoughtful preparation, focused execution and thorough testing.

What does this look like?

It can vary slightly from project-to-project depending on the components involved, but we generally follow an Agile-based workflow. As a result, the entire team participates in every phase of the project from brainstorming through maintenance.

1. Brainstorming


Brainstorming allows everyone the opportunity to share ideas and get everything out into the open. These discussions are not structured and often cover topics from from high-level deliverables to specific implementation details. Not only are they helpful for getting good (and bad) ideas into the group consciousness, they also serve as an opportunity for everyone to reach a common understanding of the project at hand, any potential complexities and options for execution.

2. Planning



After brainstorming, we move on to planning. This is a very important, but often incredibly difficult part of the process. The high-level product goals have to be deconstructed into small enough pieces for an engineer to build. Knowing how small or large to make these units of work and how to estimate their size relative to the other units of work in this project can be a tedious exercise. Make the pieces too small and it’s difficult to adjust the plan as new information arises and divide the work amongst the team. Make the units too large and you may end up waiting for a big piece of functionality to be completed which delays progress on other components. How this process works exactly will vary greatly based on the composition of personalities on your team. For us, it’s usually a spirited discussion that lasts two or three hours every other week. The result is often flow charts of functionality and a plan with the individual units of work.

3. Execution


As an engineer, this is the most fun part of the product development cycle. Everyone on the team starts implementing the items that were defined in the plan. We meet formally every morning to talk about progress and areas that we need help, but there is constant communication within the team during the day. During this process, we often work in ad-hoc pairs to get through the more complex bits of functionality or in cases where we are modifying existing components related to our core backup or reporting systems.

4. Testing

In addition to automated unit testing and functional testing, we also do a lot of team testing ahead of releasing new functionality. Our service connects to a wide variety of hosting providers, platforms and server types. As a result, we are extremely suspicious of the software that we build. Having humans devise test scenarios and directly interact with any new features we have built gives us an extra level of assurance that we did not miss anything in our automated testing. For functionality that requires user interface changes, it also give us the opportunity to make sure that everything works as expected in all of the popular browsers.

5. Deployment and Maintenance

Unlike other development teams, once we’ve built a new feature, thoroughly tested it and agree that it is ready for release, we can’t just give it to the client as say that it’s finished. We have to deploy the changes to all of our customers and maintain all of the functionality that we build. This reality of being both the development and operations team makes us very cautious about how things are implemented. If you know that your phone could be the one ringing at 2am when something breaks, you’re going to think twice before modifying code that you don’t completely understand or writing a super-clever, elegant, one-line and completely unreadable piece of code.


I believe that this process, which represents our intentionality, intense focus and commitment to delivering a high-quality product to customers is one of things that sets us apart from other backup solutions. An open-source plugin may have had more developers looking at the the source code, but the ultimate responsibility for ensuring that a backup runs every day is still up to you, the website owner. And, what about those times that you inevitably need assistance? Community support is a wonderful thing, but is it the best option for mission-critical infrastructure services?

These limitations do not apply to CodeGuard. Our service isn’t free, but that is because we have a team of professionals that are dedicated to the success of your backups. Every CodeGuard customer has a whole team working for them to ensure that their backups happen as scheduled and we have developed a process that allows us to do that while continuing to improve and add features to our service.

– Jonathan

Service-Oriented Architecture: Not just for web services

Twitter Facebook

CodeGuard, like many rapidly growing software as a service businesses with a maturing product, is breaking apart our monolithic software architecture into smaller, modular services, or a Service-Oriented Architecture (SOA). While monolithic architecture has the advantage of avoiding premature architectural decisions when at the experimental stage of a product’s life, it is difficult to scale after this stage and lacks modularity by definition. As monolithic applications grow, it becomes increasingly difficult to reason about the impact of changes and predict or avoid side-effects. For these reasons, new features and products at CodeGuard are built using an SOA approach with modularity in mind. In addition, functionality of current monolithic applications is broken out into separate modular services when possible.

SOA by Command-line Interface

For many developers, the term SOA, or Service-Oriented Architecture, is synonymous with web services. This perception is not only a misconception, but also misses the core ideas of SOA and focuses instead on the method of implementation. SOA, as the term suggests, is instead about software architecture focusing on loosely coupled units that are self-contained and expose well-defined interfaces for the services they implement. While web services are one way to implement SOA, they are not always the best choice, especially when breaking apart monolithic software. When our engineering team decided to build a backup verification service we implemented it as a command-line interface, or CLI.

Reasons for choosing this approach over a web service include:

  •  I/O cost: A web service is typically deployed on separate infrastructure from the client of that service. While this can be an advantage, it can also incur overhead if the service requires access to data such as a backup. For backup verification, this data may already be available on local disk and running a service locally via a command avoids the cost of retrieving that data twice.
  • Everything can shell out: Building a service as a CLI is language-agnostic since every language and framework that runs on linux can shell out. In this respect, it is no more limiting than a web service and possibly less so as some technology stacks lack robust networking or good support for relatively newer technologies like JSON.
  • No data model duplication: In addition to avoiding I/O overhead, a CLI can help avoid the temptation of duplicating data models or relationships from the client application.
  • Well-defined, documented API: Choosing to build a service as a CLI instead of a web service does not mean compromising on the interface for the service. A CLI can explicitly require options and combinations of options, returning an error if the requirements are not met. Documentation tools include help output and man pages and can often be auto-generated from code.
  • Easy deployment: A web service has to be deployed, the clients of the service have to be configured to know where to send requests or use service discovery, and then of course the service must be available at all times that it is needed. A CLI, in contrast, needs only to be deployed (in many cases just a binary) to the servers that call it.
  • CLI -> Web Service: A CLI is just another form of interface and a CLI, if it is built with modularity and separation of the interface from the implementation, can become a web service without rewriting the functional code. Instead of command-line options, the options are specified in a network request, usually an HTTP request containing JSON.

Backup Verification Service

One of the services that the CodeGuard engineering team has built as CLI is our backup verification tool. This tool is used to independently verify that the file backups of our customers’ websites are complete and correct. It does not share any code with our backup system. Given the location of a backup directory structure, it compares the contents to the contents of the remote server and logs any differences it finds, including missing files, files of the wrong size, additional files, and files with differing metadata. This service is built in Go (, which compiles to easily-deployable static binaries with no dependencies and is also simple to cross-compile.

A simplified version of the CLI is included below:

Application Options:
 --path= Path of backup to verify
 --u= User name
 --p= Password for login
 --key= SSH key if using key-based authentication
 --port= Port to connect
 --site= Remote server URI
 --x= Exclusion filter (regex). May be specified multiple times
 --last_backup= Last backup time

To separate the interface from the implementation of the service, we use the excellent go-flags package. Using the backup verification service as a CLI, the backup system can simply shell out to start a verification. Logs of differences found or errors are written to the standard error output and can be retrieved by the client service. The interface clearly documents the parameters required to run a verification and will exit with an error if the required parameters are not included as command-line flags.

At CodeGuard, we also build web services as  part of our SOA, but they are certainly not the only way to build a service-oriented architecture. Often they are not even the best choice as the overhead and complexity incurred can be unacceptable or lead to new problems. As our architecture evolves we may someday need to run our backup verification service as a web service. When that day comes we will need only to rewrite the implementation of the wrapping interface. The encapsulated functionality may not change at all. For now, a CLI gives us all the major benefits of SOA without unnecessary architectural complexity or I/O overhead.


A/B Testing and Website Backup

Twitter Facebook

August was an exciting month for the CodeGuard team. Something we decided to focus on was fine-tuning the copy on our marketing website. Were we doing a good job of explaining ourselves? Were we effectively reaching our target audience? How can we increase engagement on our website?

I think it’s healthy for a company to take a step back and ask themselves these questions at least once a year. From one year to another an assortment of policies, features, services offered, prices, or more could have changed with your product. It’s important to make sure that you are still explaining what it is you do in an accurate way, while also using words that will resonate with your target audience.

With these questions in mind, we decided to launch a series of A/B tests on our homepage and a few landing pages, to see how we could improve. For the remainder of the blog post, I’ll be focusing on just our homepage tests. A/B testing is a simple way to test changes to your page against the current design, and determine which ones produce positive results. It’s a way to test which version of the page has the best ‘conversion’ rate, based on a goal that you decide.

The goal could be, “how many times was this video played?”, “How many times was this button clicked?”, or “How long did the website visitor stay on the page?” It’s very important to clearly define what success means for your A/B test, so that you can figure out which page performed better. We wanted to keep these tests as simple as possible, so for us, success meant “Did the website visitor click our ‘Sign Up Free!’ button at the top of the page?”

These are the main things that we tested between our control page, and the variation page:

  1. Headline (h1) copy
  2. Paragraph copy above the fold
  3. Video placement and image placement
  4. Button copy

Before we could begin creating our variation copy for these elements, we had to reflect on the image we wanted to portray of ourselves to prospective customers. To start, building trust with our website visitors from the get-go is extremely important to us. We take security very seriously here at CodeGuard, and we want potential customers to know that their data will be safe with us. That is why the first A/B test we launched was between our homepage and a variation homepage with our “Trust Seals” right at the top.


CodeGuard Website Backup Homepage


We wanted to know if having our security and business certifications prominently displayed on our website would help visitors realize that we are very concerned with security, and very passionate about the transparency of our business.

In the end, we saw a staggering 56% improved conversion rate with our variation page that had the Trust Seals! We couldn’t believe that a change so simple had such jaw-dropping impact on our goal!

We made a few more minor changes here and there with button copy, and some copy elsewhere on the page. After all of that though, we were itching to launch another A/B test to see if we could get more results like our first major A/B test. This time, we decided to make a much bolder change. We tested our existing homepage (that now had the Trust Seals permanently at the top) with a variation page that changed the headline copy, paragraph copy, and the ‘Hero Image’ of our top section.

For a long time now, we have used our “Get a time machine for your website” tagline to help people understand what our product does. We always thought it was catchy enough to reel in people’s interest, but also descriptive enough to let people know we dealt with website backup. After a few rounds of user-testing, we were able to both confirm and refute this assumption.

For some or our testers, this tagline made perfect sense to them, and invited them to want to explore the page further. Some of our testers though, were confused by the tagline, and it took them a few seconds of further reading before they realized what our product offered. If a potential customer decided to Google search for a way to back up their website, were they entering in queries that had the word ‘time machine’ in them? Most likely not. Our analytics dashboard helped confirm this as well.

So for our next biggest A/B test we decided to change our headline copy to “Reliable, robust, and secure cloud backup for websites.”


Screen Shot 2014-09-05 at 9.31.46 AM

Because the ‘time machine’ wording was no longer being used, we also decided to swap out our time machine image for a more prominent placement of our video. Finally, we changed the paragraph copy below our headline in hopes of building even more trust. “We are backing up over 150,000 websites worldwide… Trusted by some of the world’s largest hosting providers…” etc.

The A/B test was launched and we waited eagerly for the results.

After about a six days of patiently waiting, I decided I couldn’t take the anticipation anymore, and so I logged into our Analytics account to see how the test was doing.

Our variation page was performing at a -41% (negative) conversion rate compared to our original homepage. I was shocked! Leading up to the experiment, I was convinced we had done everything right in terms of critically thinking about the variation copy we were writing. We used existing data in our Analytics account, and notes from previous user-testing sessions to inform our decisions, and guide our new copy. So why was our variation page performing so poorly? It was the first time one of our variation pages lost.

We decided it was time to take a step back from the barrage of A/B tests we were launching, and reflect on our tests a little before beginning another one. In the end, I think this A/B test wasn’t as successful because we tried to change too much too soon. Before this test, our tests consisted of changing one thing, and seeing how it performed against our control variable, or original page. With this A/B test, we changed three very major things in the top section of our homepage. We got ahead of ourselves because of our excitement!

I don’t think this is necessarily a bad thing though, because it helped us understand how we can perform a better test in the future. When it comes to A/B testing, there are countless articles out there that tell you how to do it best, and I read many of those articles. This is what we learned from our first-hand experience:

  1. Have clearly defined goals for the experiment
  2. Thoroughly test that your goals can be successfully recorded in your Analytics account before launching the test
  3. Use data to guide your decisions for what to change
  4. Don’t change too many things all at once. Less is more!
  5. Take time to learn from your tests before launching another one

I’m sure that more will be added to this list over time, but for the time being, we are very happy with the results we have been able to accumulate over the month of August. Our website is performing well, we have renewed confidence that we are building trust with our visitors, and we look forward to the next round of improvements!

Whether your company deals with website backup, travel booking, social media, or anything else on the web, I think it’s important to ask yourself once in awhile “Are we reaching our target audience as best as we can?”

The results of your A/B tests may surprise you!


GoDaddy Website Backup: Are you at risk?

Twitter Facebook

Have you backed up your GoDaddy Website? GoDaddy Website Backup Required

GoDaddy does not guarantee any backups for your website and according to their terms of service, you agree to back-up your own content. This is not a marketing gimmick to promote website backup services; this is the unfortunate truth. Look at their terms of service yourself – Under “5. General Rules of Conduct”, scroll down to number 7. “You agree to back-up all of your User Content so that you can access and use it when needed. Go Daddy does not warrant that it backs-up any Account or User Content, and you agree to accept as a risk the loss of any and all of your User Content.” Sign up for CodeGuard today to comply with the GoDaddy terms of service.

GoDaddy Website Backup, or the Lack Thereof

Today we will examine GoDaddy and the guarantees associated with GoDaddy website backup. Below is a screenshot from GoDaddy’s Legal page. Let’s look at what GoDaddy states about GoDaddy website backup. GoDaddy Website Backup Legal Statement Directly stated in GoDaddy’s terms of service, it says “You agree to back-up all of your User Content so that you can access and use it when needed. Go Daddy does not warrant that it backs-up any Account or User Content, and you agree to accept as a risk the loss of any and all of your User Content.” So the user, which is the website owner, bears complete responsibility for GoDaddy website backup! Go Daddy does not warrant that it backs-up any account or user content, and by purchasing web hosting, the website owner accepts the risk of loss for all content! Astonishing. In light of the discovery that GoDaddy does not guarantee backups of any kind, what should you do?

GoDaddy Website Backup: Manual Options

There are a handful of blog posts across the internet that offer tips for how you can perform GoDaddy website backup – ranging from using homebrew PHP scripts to doing it manually yourself. 1. On GoDaddy’s support site, “metalarcade” inquires how to “download/backup website to hard drive”. He states that he has been running a videogames webiste for a year, and still hasn’t figured out how to back up his site.  He wants peace of mind that the posts on his site won’t be lost, and describes how distraught he would be if something ever went wrong. After six more interactions with the technical support representative from GoDaddy, it can be surmised that metalarcade succeeded in backing up his site to his hard drive.  This exchange occurred over 4 months. 2. The shares a backup utility that can be used to make the manual backup process for GoDaddy website backup easier. But it is . . . still . . . manual. So that means you will need to do it on a regular basis to backup your site. 3. Blogelina provides a tutorial for GoDaddy website backup, specifically, how to perform GoDaddy WordPress backup. She recommends backing up your WordPress site once per month, and this is how:

  1. Go to My Acct
  2. Then click Hosting (in the left)
  3. Click Launch
  4. In the green toolbar hover over Databases and click My SQL
  5. On the far right click the pencil (edit)
  6. On the top left of toolbar click Backup
  7. Then click OK
  8. Confirm OK

GoDaddy Website Backup: Automatic Options

CodeGuard recently announced a super-simple wizard that was released to help GoDaddy customers backup their sites in 2 seconds. Yes, you read that correctly. All CodeGuard needs is your username and password for your GoDaddy account, and we will set up the rest. Learn more here: Other alternatives include building scripts yourself, ensuring they work (and that GoDaddy doesn’t shut them off), and maintaining them. Free options with questionable reliability and effectiveness abound, for those on an extreme budget who don’t value their websites.

Don’t violate your terms of service! Setup automatic or manual website backups today

GoDaddy does not guarantee backups of any kind for its customers. Many customers think that GoDaddy website backups are a sure thing, but this is simply not the case. While GoDaddy may have some type of server backup for disaster recovery, they do not guarantee the safety of your content. What are your options? We’ve looked at manual backups and touched upon a way to automate things yourself. You can simply write a script, upload it to your site, and access the CRON manager in your GoDaddy Hosting Control Panel. From there, you can set a frequency for the backup. If only this automated website backup was as easy as this sounds! But it isn’t.  CodeGuard provides the easiest, most reliable, transparent, and trusted way to accomplish GoDaddy website backup. All you need to know is your username and password. After that, CodeGuard does the rest. -David *Original post: April 2012, updated on August 22, 2014

GoDaddy Backup and 1&1 Website Backup Made Simple

Twitter Facebook

We know that our customers’ time is extremely valuable, and we want to do whatever we can to minimize the amount of time spent configuring and managing backups. To that end, we’ve designed our system to be robust and reliable, so that after websites and databases are added to CodeGuard customers can trust that their backup will be taken care of every day.

GoDaddy Backup Made Easy

For some time we’ve been working on extending those same concepts to our initial setup process. To date, we’ve asked customers to provide us with all of the connection details for their website. For many, gathering their FTP/SFTP server hostname, FTP/SFTP username, FTP/SFTP password, MySQL database hostname, username and password is quite a chore. We’re excited to launch our new website addition process that allows customers of several large hosting providers take advantage of our new automatic account configuration. All you need to do is provide your hosting account credentials and select which websites and databases you would like to be backed up. The entire four step process is as follows:

Step 1: Select your host

The first step when adding a new website to CodeGuard is selecting a host. Currently, we support automated website addition for GoDaddy backup and 1&1 backup. If your websites are not hosted on either of these providers, the “Other” option takes you to the advanced website connect process.


Step 2: Provide your hosting credentials

Next, the customer is prompted to enter credentials for his or her respective hosting provider. These credentials are 100% secure and are only used one time.


Step 3: Select a Hosting Account

Then, the customer is asked to select a hosting account, which has various websites associated with it.


Step 4: Select a website

Finally, the customer is prompted to select a website to add to CodeGuard.



Throughout this entire process, CodeGuard makes secure requests to our new application Viper. This application drives a headless web browser using Watir and PhantomJS to log into the customer’s account and retrieve the necessary information. When a customer selects a website to be backed up, Viper creates a background job which automates all of the required steps to enable website backup. Within a short time, the customer receives an email that his or her website is being successfully backed up with CodeGuard.

Working with a hosting provider in this way can be complex, so Viper has several monitoring tools which allow for a seamless customer experience and quick failure notification. If anything goes wrong during any of the website addition steps, Viper promptly notifies our team, so we can address the issue and inform the customer that the issue has been resolved. We also run automated tests several times per day to ensure that any changes to a hosting provider’s website are immediately noticed and Viper gets updated.

Looking towards the future, our main goal is to ensure that every CodeGuard customer can take advantage of these new tools. For example, Viper currently detects WordPress sites and adds both the website and database to CodeGuard without requiring any extra information. However, this feature is only available for GoDaddy backup customers. In order to improve every CodeGuard customer’s experience, we are working on extending Viper’s functionality to as many hosting providers as possible.

-Tripp Roberts