staging-devopsy.kinsta.cloud

Category: DevOps Toolbox

  • Does Modeling Have a Place in DevOps?

    Does Modeling Have a Place in DevOps?

    As organizations adopt DevOps, they not only release smaller changes sooner, but they also change the way they develop software. Additionally, applications no longer are seen as giant monoliths, but rather as dispersed across dozens of services that may be built on different platforms (from Node.JS to .NET), depending on the team and technology du jour. To complicate matters even more, as technology is changing, so are business requirements.

    This is where an application model comes in. It saves time, ensures quality and increases organizational knowledge of the application and its components that are constantly being changed and rebuilt. This article will explain what an application model is, who is involved in creating one and how to start using it.

    What is an Application Model

    The application model is not a specific set of unified modeling language (UML) diagrams or models used in a direct 1-to-1 relationship with the code. Rather, it is a high-level document that furthers understanding of the application and helps provide insights about that application.

    The model needs to be approachable to personnel from many different business roles, including developers, project managers, IT administrators and even users. The array of people who might use the model should give a rough idea about technical level.

    Kanban2

    What makes the application model so different, so useful and so important is that it defines not just the field of the application but also the specific domain, which is also why it can be called a domain model.

    Field vs. Domain

    For example, a core function of banks is to underwrite loans, but they do many different types of loans. A field of underwriting could be as specific as commercial underwriting for loans over $5 million. This field will have many similarities no matter the bank because underwriting is a regulated practice; therefore, the information used to make a decision is mostly identical.

    However, the field isn’t specific enough to create an application model because it doesn’t include the unique factors of a specific bank—things such as the amount of weight a system gives to any number of loan factors (credit score, cash on hand, etc.), processes that are unique to an individual bank or even specific nomenclature that bank uses. All of these factors go into creating the application model.

    In the end, the team gathered will build a complete model of what the application is and how it currently works. It’s important that the model being created reflects the actual application and not an idealized version of it. Because many business applications are quite large, it often makes sense to have a broad model for the entire application and several more intricate models for specific sections or features.

    Bite-sized chunks

    For example, a bank will have different criteria for commercial underwriting depending on size, income and even type of business. While the overall application covers all the different types of businesses being underwritten, different sections could easily be divided by requirement and modeled individually as part of the whole.

    Lastly, it’s important that the model is reflected in the code base for easy understanding by the development team. If the model is meant to serve as a central hub for conversations about changing and improving the application, the decisions and insights found during those conversations can be relayed and documented for the development team more easily and much better if there is a tight relationship between the model and the code base.

    How to Create the Model

    As mentioned earlier, the model must be approachable and understandable to individuals across many different business roles and with different backgrounds because it is the thing around which conversations about change and improvements happen. So, the first logical step in creating the model is assembling the team.

    The core team should share several attributes for success. They should all (no matter their business title) be very familiar with the application and the current processes; this is not the place to learn about how things work.

    At a minimum, the team should include developers, operations and project managers, but knowledgeable users can also be invaluable. Everyone should also be above average communicators because they will be working to create understanding with the core team and relaying that understanding to non-team members.

    Communication Matters

    Communication is an extremely important skill for any member of the model team to have. For example, there could be an incredibly gifted developer who has vast knowledge of the application being modeled, but who speaks the local language as a second language and without advanced and nuanced proficiency. This is fine when changes that they are working on are documented through a ticket system and can be understood by reading the ticket and looking at the code base; however, it’s not appropriate for the modeling team.

    At best, that developer will slow down the entire modeling process because they will have to ask many questions to fully understand what is being said by non-developers. At worst, that developer will not ask questions to get the understanding needed, and therefore will not be able to share their insights with the modeling team nor adequately explain the model to fellow developers.

    Once a team has been created, the best start is to talk about the application itself as it is. The goal is to create a shared understanding of how the application works and how it’s being used. Some good starting topics are:

    • What does it do well, what doesn’t it?
    • What’s missing from the application?
    • What’s in the app that’s not being used?
    • What other applications does this interact with and which processes?
    • Is the application being misused (e.g.: a field labeled one thing used for another)?
    • What is the process and data flow?
    • How much overlap is there between data and objects?
    • How granular are security requirements?
    • What are the application deployment specifics?
    • Which users will need access?
    • Which APIs will be used?
    • What are the Performance and Health Monitoring requirements?

    Conversing around a whiteboard is often helpful for quick sketches and lists that everyone can see. You’ll note that there are both non-technical and semi-technical topics above, so a good project manager who is familiar with both ends of the application can be invaluable in facilitating discussion.

    Miscommunication

    For example, if an underwriter who works on business loans was describing the process to a group modeling an underwriting application, they might inform the group that a credit rating is one of the factors that is used in determining loan worthiness and needs to be recorded in the application. Credit ratings are something that everyone who has ever had a credit card, home loan or even auto insurance is familiar with. However, it also presents an opportunity for miscommunication.

    Individual loans (for things such as a mortgage) use a credit rating score that can go up to 850. A developer might know this from personal experience and create a field on in the application to record a credit score. Restraints on the field could be numeric only, with a range of 300 to 850 (this would help ensure that only valid scores were entered). However, businesses are given credit ratings with letters and symbols—not numbers—such as AAA, for example.

    In this example, if there were no follow-up questions and no examples used even though multiple people on the team were using the same term and thought they were using it correctly, there is still miscommunication that would result in an error in the final software that would then need to be fixed.

    As discussions progress (and it will take time), what is being developed by the modeling group is a shared understanding of the application, as well as shared knowledge of how to talk about it. This allows words to be used precisely with the assurance that everyone on the team is using them in the same way.

    Sometimes this understanding is referred to as a universal language because through questions, miscommunications, explanations, demonstrations and discussions, what was created by the core group is a way to easily and fluently talk about the application without any misunderstanding.

    Using this shared understanding documenting the application should be achievable. Because the application model isn’t a set series of diagrams or maps, there isn’t a definitive template for creating one, but many teams have found that a combination of data flow, activity, use case, sequence and component diagrams are at a high enough level that everyone can understand and use them, while still being technical enough to be useful for the development team. It can also be helpful for text documents of notes, snippets of sample code and even photos of whiteboard drawings to be part of the model.

    Of course, a series of diagrams and notes can be messy to use and organize, so oftentimes a mind-map can be helpful as a method of keeping track of the disparate parts and diagrams and maintaining ease of use.

    Modeling for Development

    Having a model is all well and good, but how does the model help with development? By defining the application not just from the developer point of view, insights will have been gained about what needs to be tweaked, what needs a major overhaul and what needs to be demolished completely.

    By not just knowing what needs to be changed but why it’s broken and what lead to the break makes changing the software not just an issue on a ticket, but a complete picture. Also, noting application traits such as data overlap, object reuse and layers of abstraction allow the core development team to be able to develop and iterate quickly without worrying about changing functionality in an unplanned way, thereby saving on testing time and costs.

     Modeling to Save Time

    For example, if a new federal regulation were introduced, and several new pieces of information had to be logged for loans of a specific size (or greater), the domain model might show:

    • That some information was already being collected and only needed minor updates.
    • None of the information was being collected at that size loan; but it was for a different size, and that function could easily be added to that section of the application.
    • That data wasn’t being collected on loans of any size, and a conversation might ensue to determine if it was worth including the same data requirements on loans other than newly regulated.
    • That information could be calculated based on existing procedures and would only need to be accurately logged and kept.

    Surely, a model is only useful so long as it’s current. Any changes to the application need to be entered into the model. An outdated model can be worse than having no model at all since conversations that occur around it are started with false data and assumptions. If a model isn’t maintained, then all of the organizational time and effort that went into it and shared knowledge it created are lost.

    Maintaining the model is key to ensuring its usefulness long after it is first “complete.” Once a model is created, it also can be helpful for the members of the modeling team to present and discuss it with their department coworkers. Sharing knowledge beyond just a few people ensures that no matter what personnel changes happen or what development trends and methodologies are implemented in the future, the model can carry on and maintain usefulness as a teaching and documentation tool.

    Modeling for Operations

    As applications and infrastructure become more and more intertwined, it becomes critical that both Developers and Operations are involved with creating an application model. Any changes that will be considered based on the model will have to take into account factors such as security, application monitoring and the host of other criteria that Ops deals with.

    Designing Changes with Devs and Ops

    When evaluating an organization for a loan, the credit rating of that organization will need to be acquired. There are vendors that perform the task of rating other organizations, and individual banks will have preferred vendors for that metric.

    If the task of gathering these ratings were to be automated, a developer might want to implement a simple process wherein the application queries the vendor through its API and waits for a reply. This is simple and straightforward, and only requires two pieces of data: the query and the report. However, Operations would certainly step in because this scenario would be a security risk. It would either involve two-way communication or require leaving the communication channel open for an extended period of time.

    A better solution that could be created with input from operations might involve an additional piece of data: a query ID. The interaction would then consist of multiple steps such as a query being made, a query ID getting returned and, then, at regular intervals, the vendor being sent the query ID until the report is ready.

    Operations will be involved in the application development, either from the start where it can help define practices and procedures that the development team must follow, or before the application is released and it raises red flags and security issues the development team must comply with before the changes are released. It’s much more efficient for Operations to be involved from the start.

    Get Modeling

    Organizations have proprietary software because it is a competitive advantage for them to create their own applications based on their business needs. Because business needs are constantly changing and evolving, the software that encompasses them must change and adapt as well.

    Finding methods that allow for quicker changes while ensuring better change quality should be a goal in any organization. Creating an application model, and the shared understanding that goes with it, can be an invaluable part of any application that faces maintenance or overhaul.

    About the Author / Scott Reece

    ReeceScott Reece is a huge fan of the greater visibility and communication enabled by DevOps. As a Friend of Astah Scott has been successfully closing project communication gaps by using models and team building. With more than 10 years of project management experience, Scott currently serves as a Partner Director at Inedo.

  • 8 Reasons to Go ‘All In’ and Attend Jenkins World

    8 Reasons to Go ‘All In’ and Attend Jenkins World

    Jenkins World 2016, the world’s largest gathering of Jenkins users, is coming to the Santa Clara (California) Convention Center, Sept. 13-15. The lead organizing sponsor is CloudBees, along with a number of premier Jenkins ecosystem vendors including BlazeMeter, Undo and Black Duck. Jenkins World will offer attendees opportunities to learn, explore and network. This year’s theme is “All IN” as Jenkins users, experts and thought leaders prepare to go all in on DevOps.

    Need more convincing? Below are eight reasons to go all in at Jenkins World 2016:

    Hear keynotes from industry leaders — Kohsuke Kawaguchi, founder of the Jenkins project, kicks off the conference with the opening keynote this year. Other keynotes include Sacha Labourey, CEO of CloudBees, and Gary Gruver, former DevOps exec at Macys.com and HP and industry author. Rumor has it that Gene Kim may make a guest appearance, too!

    Attend training/workshop add-on options — Come to Jenkins World as an attendee, leave as a Jenkins master. Join us a day early to attend Jenkins certification training and/or learn the fundamentals of Docker and Jenkins. Additional workshops, covering topics such as plugin development, Jenkins certification and automating pipelines with the Pipeline plugin, are available.

    FREE certification! Your Jenkins World registration also provides you with the option to take a certification exam completely free! Did we mention it was FREE? All you have to do is sign up and study up!

    Brush shoulders with the Jenkins stars — Get access to some of the best Jenkins experts in the world; attend their sessions and network with them. The sessions cover a range of topics including infrastructure as code, security, containers, pipeline automation, best practices, scaling Jenkins and community development projects.

    Visit a variety of vendors — At the expanded Sponsor Expo you can check out a range of technologies and services that help you optimize software delivery with Jenkins.

    Pick up your next read — CloudBees Senior Consultant Viktor Farcic has recently published his book, “The DevOps 2.0 Toolkit: Automating the Continuous Deployment Pipeline with Containerized Microservices.” Be one of 200 lucky attendees to get a free copy and be sure to catch his session Sept. 13. We will even be hosting a book-signing with Viktor.

    Do you follow CommitStrip? They will be onsite and you can help them paint a custom Jenkins-themed mural!

    Meet the Butler and snap a selfie – You may have seen the Butler on @CloudBees engaged in some Xtreme social media adventures to get himself to Jenkins World, but at Jenkins World you’ll have the chance to meet him in person. Don’t forget to snap a pic with him at the social media station and share with your friends.

    All of this and so much more awaits you at Jenkins World. Go “All In” and register now! Use this code JWDEVOPSCUST and get 20 percent off your conference registration.

    See you in Santa Clara!

  • 4 n00b MySQL Mistakes Every Programmer Makes

    4 n00b MySQL Mistakes Every Programmer Makes

    Even experienced programmers can sometimes be thrown for an infinite loop–or other code failure–with these common MySQL mistakes.

    When a database application goes belly up, the cause often can be traced to sloppy coding. Every now and then, the reason for a misbehaving app is an idiosyncrasy in the platform itself. Here’s how to prevent your database from tripping over one of these common MySQL mistakes.

    Handling Numeric Value Overflows

    In an Aug. 11, 2014, article in the Database Journal, Rob Gravelle describes a “gotcha” (not a bug) in the way MySQL handles numeric value overflows. When a value supplied by an automated script or application is outside the range of a column data type, MySQL truncates the value to an entry within the acceptable range.

    Generally, a database system will respond to an invalid value by generating an error instructing the script to proceed or abort, or it will substitute the invalid error with its best guess as to which valid entry was intended. Of course, truncating or substituting the entered value is almost certainly going to introduce an error into the table. Gravelle explains how to override MySQL’s default handling of overflow conditions to ensure it generates an error, which is the standard response to invalid entries in most other databases.

    Misuse of Primary Keys

    According to Justin James on the Tech Republic site, the most common database programming no-no is the misuse of primary keys.

    MySQL-Mistakes

    James insists that primary keys should have nothing at all to do with the application data in a row. If they’re not system values managed by the system, you’re likely to encounter problems when you change the underlying data or migrate the data to another system.

    Using Stored Procedures as a Crutch

    Another frequent cause of program problems is overuse of stored procedures, which James describes as a “maintenance disaster.” There’s no easy way to determine which applications are using a particular stored procedure, so you end up writing a new one when you make a significant change to an app rather than adapting an existing stored procedure. Instead, James recommends that you use advanced object-relational mappers (ORMs).

    Not every developer is sold on ORMs, however. On his Experimental Thoughts blog, Jeff Davis explains why he shies away from ORMs. Because ORMs add more lines of code between the application and the data, they invite more semantic errors. Davis points out that debugging in SQL is simpler when you can query the database as if you were an application. The more lines of code between the application error and the database, the more difficult it is to find the glitch.

    Overuse of BIGINT

    One of the common database errors identified by Thomas Larock on the SQL Rockstar site is playing it safe by overusing the BIGINT data type. If you’re certain no value in a column will exceed 100,000, there’s no need to use the 8-byte BIGINT data type when the 4-byte INT data type will suffice. You may not think a mere 4 bytes is significant, but what if the table ends up with 2 million rows? Then your app is wasting 7.8MB of storage. Similarly, if you know you won’t need calendar dates before the year 1900 or after 2079, using SMALLDATETIME will make our app much more efficient.

    Misuse of GUIDs

    On the SQL Skills site, Kimberly Tripp highlights another common database design error: use of non-sequential globally unique identifiers, or GUIDs. In addition to creating fragmentation in the base table, non-sequential GUIDs are four times wider than an INT-based identity.

    Is your app’s failure to launch due to poor typing skills?

    Maybe the best way to start your hunt for a coding error is via a printout rather than with a debugger. That’s the advice of fpweb.net’s tutorial on troubleshooting SQL errors. The problem could be due to a missing single quote mark, or misuse of double quotes inside a string. If you get the “No value given for one or more required parameters” error message, make sure your column and table names are spelled correctly

    Likewise, if the error states “Data type mismatch in criteria expression,” you may have inserted letters or symbols in a column set for numeric values only (or vice versa). The FromDual site provides a complete list of MySQL error codes and messages, including explanations for many of the codes, possible sources for the errors, and instructions for correcting many of the errors.

    MySQL Error Codes

    The FromDual site explains how to find sources of information about MySQL error messages.

    MySQL error message

    For many of the MySQL error messages, the FromDual index provides an explanation, reasons for the error message’s appearance, and potential fixes.

    With a little work and a lot of sticky notes, these common MySQL mistakes are fairly easy fixes. There’s nothing worse than looking like a nOObie programmer and damaging the integrity of your application. So stay sharp, and take a second look at your MySQL queries.

  • Orchestration: How To Get Everyone in Tune

    Orchestration: How To Get Everyone in Tune

    Automation is key to success in the cloud. When your entire infrastructure is accessible via an API, it doesn’t make sense to configure and install your applications by hand. This is where orchestration tools come into play.

    Simply put: You will not have long-term success in the cloud unless an orchestration tool is a core part of your strategy.

    Orchestration Tool Uses

    Each orchestration tool works in roughly the same manner, taking a set of instructions (Chef uses recipes, Puppet uses manifests and Ansible uses plays) and applying them to one or more systems. The syntax of the instructions and how they are applied changes with each tool, but the principles are consistent.

    This leads to the most common use case: configuring a new server.

    Instead of manually installing all of the required libraries, applications, config files and other tools, you simply run the matching set of instructions. Need a new web server? One command and you’re set. Need a database server? Another command has it up and running quickly.

    That’s the basic idea, but with each of these tools you easily can build out a more complex configuration by nesting sets of instructions. Taking the two examples above, you could have a baseline server configuration with all of the common tools and configurations, then customize each server with a smaller set of configurations specific to web servers and database servers.

    Does a new utility need to be on all of your systems? Do you have a new version of your app? Do you need to push out a new configuration file for your data stores? Send one set of instructions through your orchestration tool and you’re done.

    It doesn’t take long to see the value of these tools and to mandate their use for all of your cloud deployments.

    Not So Fast

    Like most challenges in the DevOps space, orchestration isn’t that simple. You can’t just wave your magic wand and get all of your teams using a centralized orchestration tool. This is not a new challenge, we’ve been trying to get this right since the early ’90s (remember Novell ZenWorks or the first version of Microsoft Systems Manager?) and are just now starting to see wide(r)spread success.

    Culture is the No. 1 blocker when it comes to this category of tool.

    You’ll hear some common objections from teams:

    • The choice of technology (is Chef versus Puppet versus Ansible versus the Emacs versus Vi?)
    • It wasn’t built here.
    • The manual process works just fine.

    These objections rarely hold up to scrutiny, but that doesn’t matter. Any objection raised needs to be addressed in some manner until you’ve got enough people on the target team to sway the rest over to the direction you’re trying to head.

    When you’re facing a cultural challenge, you have to address the people side of the equation in your strategy!

    3 Steps to Take

    No solution is going to work in all cases, as every organization is unique. You know the culture of your organization best, so go with your gut. With that in mind, here are the three approaches that tend to work best for most organizations.

    1. Empathy

    No one likes to be told what to do from up on high. While mandating that a team switch over to this shiny new orchestration tool might get the work done initially, it also will build resentment. That’s the opposite of what you want. You want buy-in.

    Odds are, each team in your organization already has a to-do list a mile long. Regardless of the benefits, your orchestration project probably isn’t very high on the list. Accepting that your priority isn’t their priority is a key step toward understanding and empathy.

    2. Pitch In

    One way to help move up the priority list is to offer to help out and pitch in.

    Learning a new tool and new way of conducting ops doesn’t just happen overnight. Have a strong, step-by-step plan to help bring other teams up to speed. A core part of that plan has to be one of your team working with the new team for as long as it takes to get them comfortable with the new tool.

    Take a patient, hands-on approach. Let other teams learn by doing … and by making mistakes. Your role is to guide them along the path of adopting the new tool. Time and time again, this type of approach has proven to be far more successful than other options.

    3. Duct Tape

    As your efforts progress, you may reach the point where the other teams are comfortable with the tooling, understand the workflow and are ready to adopt the new tool.

    The problem is now either:

    1. Developing new sets of instructions or
    2. Converting old scripts or hacks to the new tool.

    To tackle the first challenge, make sure you’ve set up an easy way for teams to share sets of instructions. A centralized set of git repos work great for this. This will reduce the overall workload and encourage collaboration. It’s better to borrow than to rewrite.

    You’ll also want to make sure there is a regular communication highlighting successes with the new tooling. This will help remind teams of the value of the new way of working.

    To tackle conversion (problem No. 2), plug your nose and use this messy hack. All three major orchestration tools allow you to execute another script or program. If you already have working sets of instructions, keep using them. Just wrap them in the new tool to gain the workflow benefits.

    Eventually you will replace them with native instruction sets, but if it’s working “as is,” that’s one less set that you need to build.

    Smoother Operations

    A lot of operational issues happen because of a manual error. Asking someone to make the same configuration change on six systems is bound to lead to a mistake. By moving to the cloud, you’ve already accepted the ability to program your infrastructure. Manually configuring the assets in that infrastructure doesn’t make any sense.

    Orchestration tools allow you to reduce the number of mistake made, ensure consistent results and amplify the productivity of your teams.

    What’s not to love?

    What have your experiences moving a number of teams to a common tool been like? What tips and tricks did you learn along the way? Share in the comments below or on Twitter, where I’m @marknca.

  • Things to Know When Planning for Cassandra Backup

    Things to Know When Planning for Cassandra Backup

    NoSQL databases such as Apache Cassandra, MongoDB, Amazon DynamoDB, Azure DocumentDB and others are being adopted by enterprises to meet the data requirements of next-generation applications. Specifically, Apache Cassandra database provides linear scalability, a flexible consistency model and shared-nothing architecture that enables enterprises to scale their database footprint according to performance needs.

    As mission-critical applications are being deployed on Cassandra databases, there are enterprise data management requirements also, such as backup and recovery. Today, there are various options available for backup and recovery of Cassandra. These options have pros and cons, which I will elaborate on in this blog post. It is important for a Cassandra database administrator (DBA) to understand the pros and cons of these options before choosing any option and validating products and solutions available to achieve enterprise goals for backup and recovery.

    Let me first point out the important features that DBAs should look for while planning the backup:

    1. Secondary Storage footprint: Efficient storage utilization is key for any backup and recovery design and plan. The backup data set may be much larger than primary setup depending on the retention period of the backup data. Hence, if storage management is not efficient, customer storage capital expenditure could increase multifold, defeating the purpose of data protection.
    2. RPO and RTO: Recovery point objective (RPO) is the maximum targeted period in which data might be lost from an IT service due to a major incident. Recovery time objective (RTO) is the targeted duration of time and a service level within which a business process must be restored after a disaster. Typically, enterprise requirements for these next-generation applications dictate very small RPOs and RTOs.
    3. Backup performance: This is the time spent in backing up the desired data set. Typically, the backup performance should be scalable enough to match the data change rate at the primary cluster. In addition, the time spent in taking the initial full backup of the primary cluster also matters, so periodical backup is not affected.

    Cassandra Backup Options

    Now let’s take a look at four options available for performing Cassandra backup:

    Snapshot-based Backup

    Cassandra provides native node-level snapshot option via the nodetool command. Snapshot creates a hard link to existing Cassandra data files and expects users to delete it after backing up the files. This way, despite changes in Cassandra data, the backup is performed from snapshot files.

    Pros:

    • Simpler to manage: It requires periodical snapshot commands on each of the nodes, triggering backup on snapshot files and removal of snapshot files after backup.

    Cons:

    • Not a true backup: Node-level snapshot doesn’t mean cluster-consistent backup and such a simplistic strategy will result in heavy repairs upon recovery.
    • Potential large RPO: The snapshot operation is not trivial, as it requires flushing of all in-memory data on disk and, hence, frequent invocation of snapshots will impact the production cluster’s performance dearly.
    • Large storage footprint due to data compaction at primary cluster: As part of data compaction, Cassandra merges multiple data files on a node into a new file and, hence, reduces multiple versions of a record to a single and latest version. Compaction not only helps in reduction of data at the primary cluster, it also improves the read performance as it reduces the number of files to look up for a record. As a side effect of compaction, it’s possible that same records will appear in multiple versions, as they are merged into a new compacted file. It isn’t a trivial task to filter out already captured records from a compacted file. Depending on various aspects such as workload type, compaction strategy or versioning interval, compaction may cause multifold more data to be backed up, causing a serious impact on CapEx planning.
    • Snapshot storage management at primary cluster: Cassandra expects users or admins to clean up snapshot files after they are no longer required. It requires a reliable job to clean up the snapshot folders; otherwise, they may affect the primary cluster’s availability due to lack of storage.
    • Siloed solution: Enterprises require multiplatform backup and recovery product and native tools are built specifically for a particular database and don’t scale for multiple databases.

    Incremental Backup

    Cassandra supports incremental backup functionality. If enabled, every time Cassandra flushes memory content as a new file, a hard link for the new file is created in the incremental backup folder. Once the backup of these files is done, the administrator is supposed to delete these hard links. The best part of incremental backup is that the hard link is created only for newly created file due to the flushing of in-memory data contents. Files created due to compaction are not hard-linked.

    Pros:

    • Better storage utilization: There are no duplicate records in the backup, as the compacted files are not backed up.
    • Point-in-time backup: Companies can achieve better RPO, as backing up from the incremental backup folder is a continuous process and, hence, versioning can be done more granularly.

    Cons:

    • Space management at primary cluster: With this option, companies need a reliable space management process on the primary cluster. The incremental backup folder must be emptied after being backed up. Failure to do so may cause serious space issue on the primary cluster.
    • Creates lots of small size file in backup: Since incremental backup facilitates a backup of new files created due to the flush of in-memory data from Cassandra, typically the file size is small. This leads to many small files, making file management and recovery not a trivial task (and fairly expensive compared to the RTO guarantees customers want and expect).

    Incremental Backup in Combination with the Snapshot

    Incremental backup in combination with snapshots is another rudimentary way to take Cassandra point-in-time backups. Periodically, data is backed up from the snapshot, while incremental backup-based files are used for point-in-time needs. This means periodically the incremental backup-based files, which are already covered via snapshot-based backup, must be deleted.

    Pros:

    • Fairly large-size backup files: Only the data between periodical snapshots are from the incremental backup.
    • Point in time: It provides a point-in-time backup and restores.

    Cons:

    • Space management in backup: After every snapshot, the incremental backed-up data needs to be cleaned up. Also, for backed-up files from the snapshot, a large backup storage footprint due to compaction at primary always is a concern.
    • Operationally very heavy: This method requires DBA admins to script solutions; this is not scalable for enterprise scale.

    Commit-Log Backup in Combination with the Snapshot

    This method is similar to incremental backup. Rather than backing up the newly added SSTables, DBA admins archive the commit-log. Periodical snapshot-based backup provides the bulk of backup data, while the archived commit-log is used for the point-in-time backup.

    Pros:

    • Point in time: Snapshot-based backups in the combination of commit-log archival provide a point-in-time backup option.

    Cons:

    • Space management: Again, the commit-log archival space management is admin’s responsibility; this method results in operational heavy solution (OpEx).
    • Restore complexity: The restore is a lot more complex, as part of restore will happen from the commit-log replay.
    • Storage overhead: Snapshot-based backup will have a storage overuse issue because of duplication of data due to compaction. This results in a very large use of secondary storage and, thus, a very high CapEx expenditure.

    Summary

    Overall, Cassandra provides multiple options for backup. Typically, snapshot-based backup in the combination of commit-log archiving/incremental backup is used if the point-in-time restore is a requirement. If point-in-time restore is not a requirement, simple snapshot-based backup is much easier to implement, though it is not a true backup and will result in heavy repairs.

    Most of these options require significant manual effort and are error prone. They work for staging or preproduction and test/dev environments that are smaller in scale and can sustain data loss and high RTO. However, for large enterprise customers and production deployments, there are key requirements such as data reduction via novel compression/de-duplication techniques, various failure handling and faster recovery with desired consistency.

    For mission-critical, always-on applications where minimizing data loss and operational resiliency are important factors,  administrators should consider leveraging next-generation data protection products from companies that specialize in this area.

  • Why IT Monitoring Matters: The State of Monitoring 2016

    Why IT Monitoring Matters: The State of Monitoring 2016

    Times are changing rapidly in the world of IT monitoring. A once stable landscape consisting of just a few monolithic solutions has exploded and fragmented as a new wave of tools better-suited to the scale, complexity and agility of modern infrastructure and applications has emerged.

    Which is why the team at alert correlation platform vendor BigPanda decided to launch the first-ever “State of Monitoring 2016” survey. The goal was to shine a light on what today’s monitoring landscape looks like and, more importantly, delve into the challenges facing IT ops teams and how these issues affect performance.

    In just one week, the survey gathered more than 1,700 responses, indicating that monitoring is clearly a critical, hot-button issue in the IT community. Respondents primarily identified as IT, engineering or operations professionals and represented a wide range of industries. While company size generally skewed larger (1,000+ employees), team size demonstrated the opposite trend (“less than 10” or “10-20” being the most common responses), indicating that organizations are either migrating away from a centralized IT structure or that companies are being forced to do more with less.

    Key findings from the survey include:

    • Almost two-thirds of respondents, including those at large enterprises, reported having production environments in the cloud, indicating that organizations increasingly are taking advantage of the scale and efficiency of cloud computing. Also, 75 percent indicated that cloud migration was currently a challenge for their team.
    • The modern monitoring stack is increasingly diverse and complex. Among the top 10 most popular tools are solutions for a wide range of monitoring specialties, including systems monitoring, APM, web and user monitoring and log management. Furthermore, outside of the 23 tools included in the survey, about a quarter of respondents reported using at least one tool that wasn’t listed or an in-house solution.
    • More tools equates to more noise, and IT teams are feeling the repercussions of alert pain more than ever before. Almost half of those surveyed receive more than 50 alerts per day from their tools, and about a quarter receive more than 100.
    • Alert floods also have a clear and substantial effect on remediation. Of those who receive 100+ alerts per day, only 17 percent are able to investigate and remediate the majority (75 percent to 100 percent) within 24 hours.
    • Eighty percent of respondents agree that a strategic monitoring process is important to their organization, but only 12 percent are very satisfied with their current approach.
    • A good monitoring strategy goes a long way. Respondents who are satisfied with their organization’s monitoring strategy find critical IT issues much easier to handle and report the best rates of remediation.
    • IT performance is increasingly tied to business performance. Customer satisfaction and service level agreement (SLA) compliance take the lead as the most common key performance indicators (KPIs) to measure IT performance, outranking mean time to response (MTTR) and incident volume.
    • Good news for budgets, but not for ROI. Only 16 percent anticipate a decrease to their overall IT budget in 2016, but a dismal 9 percent reported that they were very satisfied with their organization’s monitoring strategy, based on overall investment.

    The findings of the report confirm that while the IT stack of 2016 is one that is incredibly advanced, its complexity introduces a whole new range of challenges for IT teams. There’s no “one-size-fits-all” solution anymore for monitoring. To manage systems in a smart and effective way, organizations need to not only ensure they have the right tools in their kit, but also that each tool works in harmony to support an effective overall monitoring strategy.

    The benefits of an effective monitoring strategy speak volumes: Alerts are easier to handle, service disruptions easier to manage and teams are better equipped to identify root cause and handle future occurrences. Organizations failing in this regard need to take a crucial look at their overall approach: How are alerts being managed? Are your tools automated with your service assurance and ticketing systems? Is anything being done to reduce noise or improve root cause identification?

    Remember: Your monitoring strategy affects not only IT health, but the health of the overall business. The complete “State of Monitoring 2016” report is available to download here.  

  • Will Deduplication Solutions Work for Cassandra?

    Will Deduplication Solutions Work for Cassandra?

    Cassandra is a popular next-generation database system (NoSQL) that powers the back end of high-performance web-scale applications in enterprises. It is database software for cloud applications that accelerates the organizations’ ability to power the growing number of cloud applications that require data distribution across data centers and clouds.

    While Cassandra offers high availability, it also provides significant opportunities for meeting data protection requirements. Creating cluster-consistent and space-efficient backup of a distributed database can be a challenging task. In this blog post, I will highlight existing deduplication solutions for Cassandra, why deduplication matters, and how to achieve deduplication in Cassandra.

    What is Cassandra?

    Here is a brief introduction. You might be curious, “Aren’t existing deduplication solutions enough to save space for Cassandra snapshot files?”  There are multiple deduplication solutions that can eliminate redundant data from the different levels of the storage layer. So why do we need a new one for Cassandra cluster backups?

    Let’s begin with some background information.

    Cassandra is a distributed database that is becoming increasingly popular with the emergence of big data applications such as software as a service (SaaS), internet of things (IoT) and real-time analytics. These applications require high availability and scalability over consistency. Cassandra supports eventual consistency rather than strict consistency, which is provided by traditional database systems such as Oracle, MySQL and IBM DB2. “Eventually consistent” means consistency will be achieved eventually rather than immediately. As well-known in CAP theorem, we cannot have following three properties in a single system: consistency, availability and partition tolerance. In short, Cassandra is an eventually consistent database that provides high performance, high availability and high scalability—but not strong consistency.

    Replication and its Role in Deduplication

    One of the most important mechanisms of distributed scale-out database systems such as Cassandra is data replication. By replicating the same data on different nodes across failure boundaries, distributed database systems can continue to service application requests even with a certain number of node failures. The downside is performance overhead to maintain multiple data copies; both write and read operations will be slower to create multiple copies and to check the consistencies among multiple copies. Although asynchronous data replication technique can be used to minimize the performance overhead of writes, it also would lower the level of guaranteed consistency level, which called is eventual consistency.

    As I just explained, replication plays a very important role in a distributed database system, and therefore we should not remove the redundancy from a live Cassandra cluster to save storage space.

    The situation becomes different when we think about backup files (or, secondary data) from a Cassandra cluster. Like any other database system in an enterprise organization, backups are needed for Cassandra; it is not because Cassandra is not reliable or not available enough, but primarily because people make mistakes (“fat fingers”) and enterprise applications sometimes have to keep the history of their databases. As they say, to err is human!

    Cassandra has a nice node-level snapshot feature, which can persist a complete state of an individual node to snapshot files. One very important point is that a Cassandra snapshot is a “per-node” operation, which does not guarantee anything about the cluster status as shown in the figure below.

    1

    Backing up a Cassandra Cluster

    To create a backup of a Cassandra cluster, we have to trigger a snapshot operation on each node, collect created snapshot files, then claim the set of collected snapshot files as a backup. In this backup, the replicated data exist as is, and the size of the backup will be N times bigger than the size of user data, where “N” is the replication factor. Replication has an important role in a “live” Cassandra cluster to provide high availability and scalability, but what’s the use of replication for backups? If we upload the backup files to an object store such as S3 or Swift, the “already replicated” data will be replicated again by the object store to provide reliability and availability for their own sake. In short, there are redundant data copies in Cassandra backup files (secondary data). If we can eliminate the redundant data copies we will save massive storage space for Cassandra backups without sacrificing retention periods. Saving on storage space directly translates to saving big bucks for the operational cost of maintaining and operationalizing a big data system across an organization.

    But, let’s return to the question of whether existing deduplication solutions would work for Cassandra backup files. You can test this by collecting and placing Cassandra database files to a deduplication system. Will the solution can save storage consumption? No! Existing deduplication solutions wouldn’t work for Cassandra data files for the following two reasons:

    • Cassandra has a masterless peer-peer architecture. Each node receives a different set of rows and, therefore, there are no identical nodes in a cluster, which means data files will look different, as shown in the following figure. Since different combination of rows are stored in Cassandra data files, each data file will be hardly identical even at the chunk level.2
    • Cassandra data files are compressed with 64KB-sized chunks regardless of row boundaries. If you know enough about deduplication algorithms, you can easily understand why Cassandra data files cannot be easily deduplicated. Fixed-length, chunk-based deduplication will not work because of the chunk alignment, and variable-length chunk-based deduplication will not work because of compression.

    Cassandra’s compaction and small record size are other reasons why the existing block- or file-level deduplication solutions will not work for Cassandra backup files. Compaction is an independent operation that merges multiple database files into a new data file; small-sized records can be hardly deduplicated with bigger-sized chunk.

    Summary

    The existing file- and block-level deduplication solutions will not work for Cassandra backup files because:

    1. Each Cassandra data file contains different set of records.
    2. Cassandra data files are compressed regardless of the row boundary.
    3. Cassandra runs compactions independently.
    4. Cassandra can have smaller record size than deduplication chunk size.
  • Using Case Management to Tackle the Unknown

    Using Case Management to Tackle the Unknown

    One of the biggest challenges for an organization is producing IT response models that accurately reflect real-world scenarios. It’s pretty straightforward to establish protocols for predictable situations. We call these “standard” scenarios.

    It gets harder when you have to tackle the unknown.

    “Case management,” as it’s called, involves establishing strategies and methods to approach unpredictable situations. In my recent whitepaper, titled, “How IT Response Models Drive Modern Operations,” I explain the benefits of combining these two approaches to have a complete model for all forms of responses that arise.

    Standard response models establish and standardize processes for certain situations. Companies that I have worked with use standardization to increase performance and improve outcomes through to faster response times, more efficient resource usage and proven methodologies. Agile methods have accelerated the pace of change; to keep up, your organization should pinpoint opportunities for automation and enhanced collaboration. Many companies are using DevOps to do so, boosting workflows by integrating new communication and collaboration tools.

    It’s all great until a situation arises that has no response template to follow. Then organizations fill the gap with case management. Unfortunately, most IT organizations are not even aware that case management discipline exists. Sometimes their business has case managers, but it does not occur to apply the same methods to IT. These organizations perform ad hoc case management without any formal guidance, rules, resources or methods. As a result, they often perpetuate unnecessary inefficiencies and fail to address business-critical problems.

    By applying a level of formality to case management, organizations can confront the unfamiliar head on and maximize their chances of handling incidents successfully. By acquiring and analyzing useful data, and devising ways to improve mechanisms over time, they eventually can create new standard models.

    Embracing the flexibility and agility afforded by DevOps is key to a solid case management strategy. Case workers can choose procedures from any number of sources to assemble a case management process. They can create toolchains to connect the procedures, making team hand-offs and collaboration more efficient and effective. This allows organizations to structure and control the response process as much as possible, even in unpredictable situations. If you have the right tools to help your teams work together, you can better adapt and respond to new information, external events and actions taken. And, as I explain in my white paper, the benefits of case management can extend beyond expectations and lead to greater flexibility in responding to user needs as well as higher customer and user satisfaction.

    Standardization is the goal: Although we can never completely standardize the world, you should do your best to transform the uncharted into the defined. Having a robust case management process allows you to identify new unfamiliar scenarios and ultimately develop mechanisms for standard models of response. By treating cases with the same level of rigor as standardized procedure, your IT department can truly maximize the benefits of DevOps.

    The standard+case approach introduces case management to our thinking for the same reason that we introduced DevOps: to bring in defined practices, mature management, relevant reporting and structured improvement so we can be more effective and efficient at responding to the situations the world presents to us.

    Check out my white paper for more on info on how to implement standard+case.

     

    About the Author/Rob England

    rob_englandRob England B.Sc., MIITP, CITP is an independent IT management consultant, trainer and commentator based in Wellington, New Zealand. Rob is an internationally recognized thought leader in IT Service Management (ITSM) and a published author of seven books and many articles, best known for his controversial blog and alter-ego, the IT Skeptic. He speaks regularly at international conferences. In 2005 Rob founded his company, Two Hills, with the motto “Sensible Business Practices.”

    Books: http://www.twohills.co.nz/books

    Blog: http://www.itskeptic.org/

  • Backup and Recovery of MongoDB, Part 2

    Backup and Recovery of MongoDB, Part 2

    This blog is the second in a two-part series on backup and recovery for MongoDB. In one of my previous blogs, I covered why companies require both replication and backup for enterprise grade data protection. And in the first part of this blog series, I discussed the existing solutions for backup and recovery of MongoDB and their drawbacks. Now in this blog, I will discuss the key requirements for protecting data that resides on MongoDB (deployed either on-premises, on private cloud with an as-a-service model or in public cloud with Amazon AWS or Google Cloud Platform).

    Requirement 1: Online Cluster-Consistent Backups

    Next-generation applications deployed on MongoDB must be always-on. This means pausing the database to make backups is not feasible and the backup operation should not impact the performance of the application. As the application scales, the underlying MongoDB also needs to scale-out to multiple shards. In this case, a backup solution must provide a consistent backup copy across shards without disrupting database and application performance during the backup.

    Advanced Data Management ServicesRequirement 2: Flexible Backup Options

    Depending on the application, data may have different change rate and patterns. For example, in a product catalog, certain items may be refreshed every day (fast-selling goods, for example), while the others may have longer shelf life, such as premium items. Based on the application requirements, some collections may need to be backed up every hour versus others that may be backed up daily. Providing this flexibility to schedule backups at any interval and at collection level granularity is another requirement we have heard from customers using MongoDB. More importantly, these backups always should be stored on the secondary storage in native formats to avoid vendor lock-in.

    Requirement 3: Scalable Recovery

    During its life cycle, data resides in multiple stages such as development, test, preproduction and production, and also may reside in multiple clouds (private cloud and public cloud). The topology of MongoDB clusters at each stage is different. For production, the application could be deployed on a sharded MongoDB cluster on-premises, but the test team might have access only to unsharded MongoDB clusters in the Amazon AWS (public cloud). Hence, the backup solution should allow multiple restore operations such as sharded to sharded (such as from 5×3 cluster to 2×3 sharded cluster) or sharded to unsharded (such as 5×3 cluster to 1×3 unsharded) across such cloud configuration.

    chart

    Requirement 4: Handling Failure

    Failures are a norm in the distributed database world. However, the backup solution should be resilient to database process failures, node failures, network failure and even logical corruption of data during backup and recovery operations. Finally, the backup solution should be able to handle failures of MongoDB config servers that store metadata for sharded clusters.

    Customers are deploying MongoDB in physical servers, private clouds and microservices such as frameworks, and in public cloud.  Backup and recovery should be seamless across these deployment example and the ease of backup and recovery deployment is a big one for MongoDB customers and I will cover this in the next blog. Stay tuned!

  • Using MongoDB for a Product Catalog Application

    Using MongoDB for a Product Catalog Application

    In today’s era of data-driven business, enterprises are continuously adopting new technologies to gain a competitive advantage. Distributed and cloud databases is one such technology. These databases are fundamentally different from relational databases, offering scalability, availability and flexible data models that are beyond the capabilities of traditional relational databases.

    MongoDB is one of the leading next-generation scale-out databases. It is a document-oriented database that provides dynamic schemas, making the integration of data in certain types of applications easier and faster. MongoDB also provides easy scaling through auto-sharding, tunable consistency for reads and built-in replication.

    In this blog, I will describe an application, product catalog management, that distinctly benefits from features MongoDB offers.

    Most e-commerce enterprises need a product catalog with the ability to store varied types of objects with different sets of attributes. These kinds of data collections are quite compatible with MongoDB, which can offer dynamic queries on a constantly changing dataset.  However, it is important to understand the major requirements of industry verticals such as retail, media and entertainment, insurance, health care and e-commerce that are addressed by MongoDB before designing an optimal data model and for your own product catalog.

    For example, Ebay offers several use cases in which MongoDB has been implemented successfully and has delivered much value for its product catalog:

    • Product Availability – Being a JSON document store, not only can MongoDB help in complex querying of a product catalog in real time but it also can keep track of placed orders to accurately calculate the available quantity for popular products. The data is important from customer service point of view, as it keeps them aware of available choices at the time of order placement and prevents order cancellations later due to non fulfillment.
    • Dynamic Pricing – Product prices change quite frequently in the online world. E-commerce enterprises need to keep pace with these fluctuations and dynamically adjust product prices to retain their existing customer base. MongoDB helps tremendously through its ability to update data on live sites without shutting down/restarting any systems. On top of that, ability to run geospatial queries using geospatial index allows the sites to show converted pricing and taxes based on the location of their customers. The example below shows how to find all theatres within 10 miles of a location using a geospatial query.

    db.theatres.find({ location:
    { $geoWithin:
    { $centerSphere: [ [ <longitude> , <latitude> ], 10 / 3963.2 ] } } })

    • Shipping Estimation – Today’s customers enjoy the convenience of shopping online at home but also demand the flexibility to control the delivery of their products. Some customers will pay extra money for expedited delivery while others are content with waiting longer to pay no shipping costs. The ability to provide correct shipping costs and time estimates is a critical functionality for e-commerce applications. Again, MongoDB comes to rescue with its geospatial queries to calculate correct shipping estimates based on the locations of the user and the inventory.
    • Personalized experience – E-commerce enterprises want their customers to enjoy the online shopping experience by understanding their needs as well as their social profile. This allows them to provide every single user with a personalized version of the site, which can be based on their past order history and their interaction with the company via stores, customer service and social media. Not only that, e-commerce enterprises run applications that recommend new products based on customers’ indicated preferences. MongoDB is being used by online retailers such as Otto to consume data from multiple sources and build customized catalog experiences for individual e-commerce customers.
    • Sales/Discounts – To engage their customers at the next level, e-commerce enterprises have built capabilities to develop customized discount offerings. This can be done via “flash sales” that allow customers to buy their coveted products at lower prices within the constraints of quantity, time and minimum spend. MongoDB has been the with technology of choice here as well (for example GILT Groupe) because it can keep track of time-bound and volume-bound parameters in real time to make such deals successful.

    ecommerceIt can be argued that e-commerce industry and dynamic product catalogs have been in existence for a long time. So, how is MongoDB changing the landscape? The answer lies in the rapidly evolving landscape of customer buying patterns. Today, a wait of an extra minute often means losing customers and revenue to the next best online competitor. Always-on applications are a norm and scalable infrastructure is a must-have requirement. Hence, MongoDB is helping enterprises to meet their changing business needs.

    However, product catalog applications generally require consolidating data from disparate sources and present a uniform view to customers. This also makes the underlying database, MongoDB, prone to logical and human errors. Hence, a highly available data protection infrastructure that can guarantee minimal data loss (low recovery point objective and recovery time objective) is required to keep online retail engines running. Further, certain catalog portions could be updated more frequently such as perishable goods; hence, those databases need to be backed up every hour versus the slow-moving goods.

    The data protection system should provide enough flexibility to set backup policies per the business requirements (granularity, etc.).