staging-devopsy.kinsta.cloud

Category: DevOps Toolbox

  • The COVID-19 Pandemic’s Lasting Impact on Tech

    The COVID-19 Pandemic’s Lasting Impact on Tech

    The COVID-19 pandemic created major opportunities for innovation and ushered in new technology solutions, but even as the danger subsides, this tech may still serve a purpose.

    Whether it be face mask detection, temperature checks, facial recognition or AI surveillance, technology crafted to combat COVID-19 is here to stay. This may present a number of benefits, but there are ethical issues that persist. 

    Throughout the pandemic, school districts sought ways to mitigate the risk of outbreaks and ensure students were complying with safety regulations. AI-empowered cameras, for example, were one of the first pieces of technology implemented, allowing schools to monitor which students weren’t wearing masks. 

    “I think it’s helpful for COVID-19 stuff, but it seems a little intrusive,” said Noa Young, a 16-year-old junior from Atlanta’s Fulton County school district. “I think it’s strange that we were not aware of that.”  

    Although students knew cameras were monitoring them, they weren’t informed about the extent of the surveillance. The cameras have the capability to scan for body temperature and can also determine if a student is wearing their mask correctly. 

    In addition to monitoring for compliance, these cameras have the ability to assist with contact tracing. They can scan students’ faces and determine who they’ve been in contact with, enabling schools to retrace the steps of someone who contracts COVID-19. This is just one example of pandemic tech’s continued development, but as adoption and use spreads to other sectors and it’s used for other purposes, security and privacy concerns begin to pile up. 

    Courtenay Brown, 31, works in an Amazon warehouse and explained the scrutiny she is under throughout her workdays. Amazon uses similar AI surveillance to keep tabs on worker productivity, notifying management if someone deviates from their assigned task for too long. Many believe this level of surveillance creates an inhumane work environment.

    “They basically can see everything you do, and it’s all to their benefit,” said Brown. “They don’t value you as a human being. It’s demeaning.” 

     The implementation of this technology is one of the factors pushing Amazon workers to take a stand through unionization efforts. 

    “There’s no way we’re going to stop or let this bring us down,” said Derrick Palmer, co-founder of the Amazon Labor Union. “It’s going to do the complete opposite. We’re going to go 10 times harder.” 

    Another byproduct of the pandemic was the shift to remote work in organizations across the country and in every industry. This shift created the need for remote access to an organization’s tools, information, data and assets, which didn’t come without inherent security risks—especially since such access often needed to happen literally overnight. 

    One way companies adjusted was through adoption of identity and access management (IAM) solutions. IAM can be used to identify, record and grant access in the cloud, an integral part of secure functionality in hybrid or remote workplaces. 

    “In the game of cybercrime cat and mouse, one could argue there is no such thing as being over-resourced. A major breach can bring down a company,” said World Wide Worx CEO Arthur Goldstuck.

    Although IAM proved to be a workable solution to the challenge of remote access management, cybersecurity threats are ever-changing and now cyberattackers have shifted focus—they may be attacking IAM solutions. 

    “Sophisticated threat actors are actively targeting identity and access management infrastructure, and credential misuse is now a primary attack vector,” Gartner described in its Top Security and Risk Management Trends 2022 report.

    To keep data safe, companies will have to be ready for an even wider range of cyberattacks, including threats to IAM. 

    These are just two examples of technological breakthroughs that were originally seen as innovative solutions but which have evolved to have their own set of problems.

  • Update Those Ops Tools, Too

    Update Those Ops Tools, Too

    We’re all pretty well settled on the idea that the rate of change in IT has been on the increase over time. From the advent of VMs to Agile to DevOps to containers, the movement has been faster—and the same is true for internal development.

    I’m going to approach this topic from three perspectives instead of my usual two. Kind of like breaking the fourth wall in cinema. We’ll discuss this movement just a bit from the perspectives of my two preferred bits, DevOps and DevOps. But for this blog, we’ll also mention DevOps—the infrastructure, tools and processes that enable the other two.

    For DevOps, languages—both updates to those currently in use and the addition of new ones—bring with them a nearly automatic update to tools used. Favorite libraries/modules, IDEs, build tools—these are all looked at and updated more frequently today than at any point in history, by my estimation. For teams that do not change languages or libraries much, source code analysis (SCA) scanning is increasingly helping to take a closer look at all those included tools and filter them based upon impact (normally security impact, but other impacts on occasion, too—like project activity and stability).

    For DevOps, the toolchain is still new enough that evolution is causing regular change, and that change is bringing with it new tools and utilities to solve the problems of this existing infrastructure. In fact, I’ve long thought the DevOps part needed a bit of a cooldown on the change and new tools/utilities front, personally.

    And finally, DevOps has gone a bit sideways in this regard. Every person—whether you call them ops, admin or DevOps—responsible for troubleshooting machines and apps has a collection of tools. But the divergence of platforms has created a divergence of tools. It is possible to work without a dedicated set of tools—and, indeed, many DevOps engineers do. Others have a folder. Meanwhile, there is another dedicated set of DevOps engineers that have a PCMCIA or USB full of useful tools that is always with them. Some have both. Of course, what tools are included in all of this varies widely also. It is an interesting artifact of our growth in IT that developers tend to share or borrow the best libraries/modules/APIs across an organization, but DevOps is more independent. As long as DevOps team members are getting the job done, no one is all that worried about which tools they used to troubleshoot. They do tend to share if asked, but with little physical evidence of what tools they used, it doesn’t come up often.

    So, what am I getting at? Look for ways to enable DevOps tool sharing on an ongoing (or at least regular) basis. An internal social media channel dedicated to the vendor your teams use to communicate, brown bags, a maintained rating system “Why I think fdisk rocks” type of thing, etc. The more power there is at the fingertips of every DevOps member you have, the better. Given our history, building an “approved” list is likely a bad idea, but having that central communication location that security can also peruse to look for red flags is not a bad idea. Of course, they are likely too busy, but worst case, such conversations can help in forensic analysis later. Not being cynical, just living IT in the 21st century.

    And keep kicking it. More and better tools will help keep all those sweet apps and servers clipping along, so there is really only upside here. Give your teams the tools and revel in your considerable uptime.

  • Does GraphQL Introduce New Security Risks?

    Does GraphQL Introduce New Security Risks?

    The GraphQL query language is an excellent tool for increasing the ease of data sharing. The premise is that you request the fields you need in a single bundled request, avoiding multiple network calls. Due to its usability, GraphQL is a growing area of development. Yet, as with any new technology, it comes with caveats.

    REST APIs function as a request-and-response paradigm in which the receiver accesses one resource at a time. On the other hand, GraphQL exposes many fields at once, requiring additional rate-limiting and time-outs to avoid nefarious queries. Additionally, the ability to traverse the entire schema could be abused by attackers to expose intricate data linkages. There are also some usability features with GraphQL that could be abused if misconfigured.

    GraphQL is a friendly alternative to REST. But how do we handle the security implications of GraphQL? I recently chatted with Mike Benjamin, vice president of security research, Fastly, to dive deeper into the security repercussions of adopting GraphQL. Below, we’ll consider the unique insecurities associated with GraphQL and suggest specific steps to limit the impact of these risks.

    Four GraphQL Security Vulnerabilities

    1. Introspection

    GraphQL provides built-in documentation that tells developers what the fields are, providing a schema of everything you can query. You’ve essentially given the entire dataset schema to a developer, meaning they can perform introspection to capture attack surface information. Hackers will most commonly look for injection or mutation capabilities that they can exploit, explained Benjamin. Or, they might seek to leverage the interlinkages between data to map a way through to classified information.

    To mitigate this problem, Benjamin cautioned against over-exposing all documentation publicly. Instead, he suggested using a restrictive security model for external viewers. “Provide appropriate user documentation in a different method,” he says. This may end up looking more like the standard REST API view of documentation.

    2. Field Suggestions

    Another usability benefit in GraphQL could also be a potential security concern—field suggestions. Field suggestions occur when you don’t enter the correct field name or enter a partial input and the system returns a hint. You might type in 0 or user, and the system makes a suggestion to fill out the rest of the field.

    This user interaction improves the developer experience, but, unfortunately, it could be abused by a hacker, said Benjamin. Thus, he recommended disabling hints after initial development. Or, if you decide to keep it in, he recommended placing a limit on its use. For example, if you try this more than five times in a second, the system could flag it as a bot.

    Right now, this attack vector is pretty limited, but Benjamin anticipated that, over time, hackers could automate this introspection with mass scanners to divulge more information.

    3. Denial-of-Service-Oriented Attacks

    Without the proper guardrails, all web APIs are prone to denial-of-service (DoS) attacks. Such an attack may pummel a service with a million queries per second to shut it off completely. In the case of GraphQL, these queries can dive very deep in a nested data relationship. A social network, for example, might expose friends of users, friends of users’ friends and so on.

    Or, hackers might utilize GraphQL’s bulk query ability to perform reconnaissance, brute-forcing the enumeration of data to query hundreds of user IDs to see which are valid. If you’re allowing unlimited data retrieval bundled within a single request, you could be inviting hundreds of bundled queries that could expose large swaths of the underlying database in one fell swoop.

    To counter these actions, Benjamin recommended limiting the testing depth. This could involve placing limits on how long a query can run, which would cause a time-out for requests that require suspiciously high degrees of processing. Another option is to limit how deep the bundling can get, said Benjamin. For example, perhaps the system restricts going three levels deep into a graph database, as that could expose millions of interconnected endpoints.

    4. Lack of Object-Level Authorization

    Another vulnerability that GraphQL endpoints may be prone to is a lack of object-level authorization, which hackers could leverage to get at a datapoint through its edges, said Benjamin. Since the idea is that you can access any data from one all-encompassing endpoint, authorization rules are often too abstracted away from the actual data. Queries may be able to access nested data or query relationships between data, too, which may unintentionally divulge information.

    Benjamin recommended placing authorization closer to your objects or data to rectify this issue. Use authorization-level controls that align permissions structure and roles with users. “Everyone needs a common authorization model,” said Benjamin.

    Future Evolution: GraphQL Security Matures

    As I’ve written before, the greatest strength of GraphQL is also its greatest weakness. Using GraphQL with default settings is a security trade-off in favor of usability. But according to Benjamin, we’re seeing more GraphQL servers disabling some introspection features, which he sees as a very positive move and a sign of maturity of the base configuration.

    GraphQL might suffer from similar threats that all APIs are vulnerable to, such as XSS, injection, DDoS attacks and broken authorization. However, Benjamin stressed that the added complexity of GraphQL makes it a unique case. “The most important thing for folks to consider is that GraphQL is more complicated than a REST endpoint—much more complicated,” said Benjamin. “GraphQL is unique in that it has linkages of data which contributes to complexity. It takes a little more time to think about how that’s confirmed, how it’s presented and how it would be attacked.”

    In other words, with great power comes great responsibility.

    Although exploits in public GraphQL APIs are still rare, we should stay vigilant. Benjamin encouraged developers to continue reading about, understanding and thinking about these attack vectors when designing their interfaces, as these risks will become more prominent as GraphQL usage grows and hackers become more familiar with these specific frailties.

  • OpenSSF Adds Open Source Package Analysis Tool Prototype

    OpenSSF Adds Open Source Package Analysis Tool Prototype

    The Open Source Security Foundation (OpenSSF) has made available a prototype of a package analysis tool that has already identified more than 200 malicious packages uploaded to PyPI and npm software components.

    Caleb Brown, an OpenSSF maintainer of the project, said the goal is to understand the behavior and capabilities of packages available on open source repositories. It also tracks how packages behave over time to make it easier to identify when previously safe software began to act in a suspicious manner.

    Most of the malicious packages detected are employing dependency confusion and typosquatting techniques, Brown said. They usually contain a simple script that runs during installation and provides a few details about the host to an external command-and-control system. Most of these packages are not exfiltrating meaningful data except the name of the machine or a username, but they make no attempt to disguise their behavior. The issue is that any one of these packages could have done far more damage, noted Brown.

    Future goals for the project include detecting differences in package behavior over time; automating the processing of the package analysis results; storing the packages themselves as they are processed for long-term analysis and improving the reliability of the pipeline. As such, the OpenSSF is looking for application developers and cybersecurity researchers to contribute to the project. The expectation is that many providers of application security tools will incorporate the package analysis tool within their offerings rather than requiring each vendor to create their own version of the same tool, said Brown.

    The OpenSSF has launched a series of initiatives to improve the overall state of open source security. Most recently, it added an Alpha-Omega Project to automate security testing processes for open source software using a $5 million initial investment provided by Microsoft and Google. The core challenge is that many organizations are dependent on open source software projects created and maintained by just a handful of volunteer maintainers and contributors. The individuals that created those projects don’t always have a lot of cybersecurity expertise. In fact, many of them would argue that the onus for securing open source software is on the organizations that use what amounts to free software. It’s not the responsibility of the contributors and maintainers to, for example, drop everything and immediately create a patch to address a zero-day vulnerability.

    It’s not clear how easily open source software could be compromised by cybercriminals that insert malware into downstream applications that rely on that code. However, in the wake of a series of high-profile breaches, it’s safe to assume that more cybercriminals than ever are trying to compromise open source software projects. While some organizations may opt to limit their dependency on open source code that has not been properly vetted, there is no way to entirely avoid using open source code. The challenge is finding a more efficient way to ensure the integrity of that code for all concerned.

  • Stytch Launches New, Flexibility-First SDK

    Stytch Launches New, Flexibility-First SDK

    Passwordless solutions have been a trend for a while now, improving user experience (UX) while reducing exposure to common attack vectors. Now, Stytch’s new JavaScript SDK aims to make password-free authentication a bit easier by increasing customization and flexibility while offering fully baked authentication out of the box.

    Julianna Lamb, Stytch CTO, said the solution provides dual benefits by doing the heavy lifting of implementation while providing the flexibility to create a powerfully branded UX from the customer’s first sign-on experience. The Stytch JavaScript SDK can offer an all-in-one experience that handles the entire user auth journey with a pre-built frontend component. But it can also give teams complete control over designing a sophisticated and tailored authentication journey. In an era where tech companies chase the ultimate customer experience as their holy grail, this should appeal to teams wanting to improve a customer’s first impression of their brand, Lamb said.

    Multiple Options for a Smarter Front Door

    Like many authentication tools, Stytch’s product suite originally offered two options: One was a standard widget to handle frontend components while backend APIs provided logins. The other option provided direct API integration, in which the developers built their frontend, handled tasks like session management and setting cookies on the frontend and used the API to do things like send a one-time passcode or email a magic link.

    “But we found people really want flexibility when it comes to any solution they’re using,” Lamb said. “They also don’t want to do the heavy lifting when it comes to things like session management. Those things can be complex to reason through and having those out of the box is really compelling.”

    The new SDK decouples drop-in UIs from helper functions, giving teams multiple options. They can place Stytch’s UI components anywhere they want in the product, sculpting the authentication flow to their specifications. As a headless SDK, it can be embedded client-side in the application to handle all the calls to the API, do the authentication, create a session and check its validity. The result: Less backend code to write and less logic to implement.

    Customizable UI flows mean that teams can fully own the UX experience and build a UI that is customized exactly how they want it to look—and they can do this without having to build an internal API to route authentication requests from client to server. Teams looking for ways to make a brand impact right out of the gate can capitalize on this ability by creating a unique user experience.

    “Authentication is the front door to your application,” Lamb said. “Sign-up is the first impression the user has of your app—and the ability to customize it with your brand look and feel so the UI fits your aesthetic is super-critical.”

    Frictionless Flexibility in Identity Verification

    Because teams may have different authentication needs within one product, the SDK can be used in tandem with the core Stytch API to meet shifting needs. Teams could generate embedded magic links via the API while providing Google OAuth and Google One-Tap sign-in options via the SDK.

    The solution also reduces friction by making it easier to not only authenticate users but verify them on return visits to continue the same session. By loading the SDK on the frontend, teams can automatically connect Stytch’s sessions product to the browser cookie, feeding user session data directly into the cookie.

    The Stytch team is currently building out more SDK functionality and frontend components, Lamb said. Given the ongoing evolution of simple authentication tools that meet complex requirements, it will be interesting to see where innovative authentication goes next.

     

  • Twitter/Bluesky ADX Algorithm | Cloud Energy Use ‘Tripled’ | Apple Staff are Revolting

    Twitter/Bluesky ADX Algorithm | Cloud Energy Use ‘Tripled’ | Apple Staff are Revolting

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Bluesky—Twitter’s research spinout—opens its ADX algorithm, Irish data center electricity worries grow, and Apple employees are the latest to rise up against hybrid working. (more…)

  • Internal Developer Portal: What it Is and Why You Need One

    Internal Developer Portal: What it Is and Why You Need One

    One main advantage of the cloud is that it offers developers flexibility and a self-service solution. Instead of having to contact the central infrastructure team whenever a dev team needs a new server or a cluster, the dev team can do it themselves through a simple web UI. This change drastically improves productivity. But there’s a caveat: Most companies don’t actually want to use the default portals provided by cloud providers. There are a few reasons for this: Security, simplicity or the need for custom, company-specific options, to name a few. That’s why we now see a move toward internal developer portals. In this post, you’ll learn what internal developer portals are and how they can make your developers’ lives easier. 

    What is an Internal Developer Portal?

    An internal developer portal is a self-service application and data store that lets developers and managers track and organize everything their engineering teams build and operate.

    Imagine that you need to create a new microservice. Your developers will first need to provision infrastructure for it; for that, they’ll use a cloud provider portal. Next, they’ll need to create new CI/CD pipelines or modify existing ones. Then, they’ll probably need to implement some changes to existing components like the API gateway and database. Lastly, they’ll probably need to add some documentation or make their new microservice visible to the rest of the company.

    Overall, they’ll need to touch a few different UIs or portals. Now, imagine they can do all of that from one place. That’s what we call an internal developer portal. It’s a self-service portal that lets developers catalog and understand relationships, ownership, resources, external service data and anything else they need to understand and operate their software.

     Why Does an Internal Developer Portal Matter?

    Now that we know what an internal developer portal is, let’s take a step back and discuss why internal developer platforms (IDPs) are on the rise. The main reason is because of the increased complexity of both software and infrastructure. Especially when we’re talking about cloud-based microservices environments, it’s challenging even for senior engineers to keep track of all dependencies and things that must happen every time changes are needed. Microservices architectures are heavily decentralized. Although you’ll still see the typical three-tier software architecture, in most cases, it’s no longer easy to tell by looking at a random component where it fits in the bigger picture just by its name or server location.

    Natural Evolution

    The second reason is that it’s simply a natural next step in the evolutionary process for self-service portals. Old-school, on-premises-based environments barely had any self-service capabilities. Do you remember how difficult that was? To request a new server, you had to email the central infrastructure team. If you were lucky, you’d get what you needed in a matter of hours or days. But more often than not, it took a week or two. And if you made some mistake in the initial request, that would cost you an extra week (or more) of waiting. More advanced companies? Even when they created some custom self-service solutions, these were limited. Often, it wasn’t much more than an online form to be processed by the infrastructure team which often became a bottleneck for all dev teams.

    When most companies started moving to the cloud, they began to see the benefits of self-service platforms. The ability to create servers ad hoc within a matter of minutes led to increased developer productivity, faster time-to-market and better quality software. But again, cloud-provider platforms are suitable for managing cloud infrastructure; you need more than that for a complete solution. Your developers can be more effective at creating and changing cloud infrastructure, but everything else was still done separately. And that’s how we naturally evolved into the demand for internal developer portals. To bring even more flexibility and clarity while increasing your dev teams’ productivity, you need to take the next step in the self-service platform evolution and combine all the things developers need into one internal developer portal.

    Clarity 

    Also, from a business perspective, internal developer portals bring clarity and offer a good overview of all your infrastructure and software development efforts. Even if you’re in the cloud and your dev teams can manage the infrastructure as they choose, you still need some control and oversight—for example, to avoid wasting money on test servers that developers forgot to delete. Sure, you can get simple reporting from your standard cloud-provider portal, but what if you’re using multiple cloud providers or have a hybrid-cloud environment? An internal developer portal can benefit both developers and their managers. Long story short: IDP portals are the future of software development.

    What Can an IDP Actually Do?

    How do you create an IDP for your company? You can build your own from scratch, use some OSS offerings or go for ready-to-use products.

    Build Your Own

    The first option would give you the most flexibility and ability to perfectly tailor the solution to your needs and the needs of your developers. However, building your own IDP from scratch is almost like building your own operating system from scratch. It requires a lot of work which could negate the main benefits that IDP is supposed to bring. You want to implement IDP to increase developer productivity and decrease time-to-market, but it would take you months to build it and years to maintain it to keep it working properly with constantly changing cloud APIs.

    Open Source IDP

    The next option is open source IDPs. Since IDPs are, in general, quite complicated pieces of software, there aren’t many options to choose from. The two most popular are Backstage and Gimlet. Backstage is more of a DIY platform for building developer portals, which means you’ll still need to spend quite a lot of time creating something that actually benefits your company. Gimlet is quite popular, too, but it focuses on Kubernetes-based environments. Both are great tools—don’t get me wrong—but if you’re looking for something ready to use and feature-rich, consider the third option.

    Ready-to-Use Solutions

    The last option is to use a ready-to-use product. Ready-to-use solutions combine the IDP functionality with incident response capabilities as well as metrics, dependencies visualization and filtering across multiple clouds.

    Internal developer portals are gaining popularity for good reason. In fact, many companies have been trying to build IDPs for years without even knowing the solution had a name. This is because an IDP-like solution will naturally evolve in any company that cares about developer experience and moving faster. The problem is that it usually takes years to come up with a single solution that fulfills all their needs. Another problem is that most of these organically created internal developer portals focus only on the tech side and skip the business needs. That’s why a purpose-built tool can be much easier and faster to implement while also doing its job much better.

  • Smoothing the Transition From REST to GraphQL

    Smoothing the Transition From REST to GraphQL

    Since its open source release in 2015, GraphQL has taken the development world by storm. GraphQL, the query language originally developed by Facebook, is a big boon for frontend developers, as it enables you to grab the exact data you need with a single request. This solves the problems of over-fetching and under-fetching, which classical REST APIs are prone to. GraphQL also offers a nice developer experience with a consistent schema and documentation. Some are even treating GraphQL as a convenient meta layer to aggregate multiple underlying services and abstract their interfaces with simple commands.

    For these reasons and more, many development teams are considering switching from REST to GraphQL. Even enterprises are beginning to use GraphQL in large production environments. But, how can these teams make the switch without breaking existing integrations? How can we ensure that things like developer experience, security, and access control all remain stable?

    I recently met with Anant Jhingran, the co-founder of StepZen, to discover tips for transitioning from REST to GraphQL. According to Jhingran, most organizations see the benefit in adopting GraphQL but are hesitant to forgo years of investment into RESTful architectures. But according to Jhingran, it’s rarely a case of “either/or”—more often, he sees projects running both styles simultaneously or layering one on top of the other. Below, we’ll explore some things to keep in mind as you invest in GraphQL.

    The Decade of Developers

    We’ve reached ubiquity of web APIs among companies big and small. “APIs have turned out to be phenomenal for software supply chains,” said Jhingran, the former CTO of Apigee. They help share data between partners and can enable platform ecosystems to thrive. Jhingran describes how within the last decade, APIs and API management have peaked to near-ubiquity in terms of adoption.

    But not all APIs behave the same way. Although representational state transfer (REST) is the most popular style, it’s still quite a general architectural guideline for web APIs—how the data comes out of each service is still highly nuanced. The structure of requests and responses are different from service to service, causing developer experience hangups at scale.

    GraphQL, on the other hand, “is a fantastic way of structuring data,” said Jhingran. With a single call, you can return the explicit information that the client requires in an expected format. For an e-commerce application, for example, this might be customer information, order numbers and delivery status. Displaying this all with one call is an awesome experience for frontend developers. And, GraphQL APIs are easy to explore and navigate since the schema is baked in. This improves upon REST APIs, where documentation may not always match the actual production implementation.

    “This decade is the decade of developers,” said Jhingran. Satisfying developer experience has become more mission-critical as enterprises work to increase their footprint in the cloud marketplace and GraphQL is part of this movement. “For people who build apps, it’s just better,” said Jhingran.

    But just because frontend teams clamor for the ease of use of GraphQL doesn’t mean backend teams can easily restructure their entire framework around it. Doing it right requires many skills, and the journey can be arduous, explained Jhingran.

    It’s Rarely REST Vs. GraphQL

    REST and GraphQL are often pitted against one another to the point that they seem like polar opposites. But in practice, teams often use GraphQL to build on top of APIs already in place. Jhingran describes the most common pattern as “GraphQL comes in as a layer that connects the dots between various REST APIs.” Either GraphQL is consumed directly or as a REST layer above GraphQL. “It could go either way,” he said.

    GraphQL is a fantastic data infrastructure tier as it can mix and match database backends. It could, in essence, become the new plumbing upon which applications are constructed. But while GraphQL may be excellent for internal construction, the name of the game is stability when externalizing services to the outside world. With REST, the big goal is to decrease broken clients and ensure high-grade security for each resource. Therefore, Jhingran foresees certain use cases where using a REST wrapper around GraphQL services to deliver fixed JSON responses would be preferred.

    Another point is that GraphQL is not only relegated to appeasing frontend developers—it has some backend potential, too. “GraphQL is an opportunity to reimagine frontend and GraphQL is an opportunity to reimagine backend infrastructure,” said Jhingran. This concept has weight nowadays, as Netflix engineers, for example, use GraphQL as a backend for microservices architecture.

    Making the Transition From REST to GraphQL

    Not all apps are going to be transformed on day one. Teams may encounter cultural resistance to new technologies. Interface redesigns, especially for those external apps which are exposed to third parties, must also require extra precaution to maintain app compatibility to avoid breaking change.

    Thus, it may take time to transition effectively. “Getting it started versus getting it right are not the same thing,” said Jhingran. He said people who get started with GraphQL may struggle for months due to competing best practices and tools. For example, Jhingran identified two main perspectives when approaching GraphQL: Programming and declarative.

    • Programming: Teams use GraphQL libraries such as Apollo to program their way out of your GraphQL endpoint. In this approach, you just start writing. It’s quick to get started, said Jhingran, but you end up describing everything you do and encountering many contingencies along the way.
    • Declarative: The other option is a more declarative approach—instead of telling the system how, you tell it what to do. For example, this subsystem should do X, this subsystem should do Y and so on. According to Jhingran, adopting tooling with declarative snippets to create GraphQL endpoints is “better in the short and long run.”

    GraphQL Security

    A March 2022 report from Salt Labs found a shocking 681% increase in API-based cyberattacks over the previous 12 months. There are many concerns surrounding API security right now—these valuable endpoints are prone to broken access control, data overexposure and over-permissive states.

    As I’ve covered previously, GraphQL security is a bit nuanced compared to REST. Within the REST paradigm, you can easily assign permissions and rate limit calls per resource. But with GraphQL, it’s not that simple. Since you can batch multiple queries, callers have much more power at their fingertips, and hackers might abuse such power if the proper authorization isn’t in place.

    Companies may rightly worry about opening another can of worms with GraphQL. But according to Jhingran, these worries are all solvable. A comprehensive GraphQL endpoint will likely require another layer to uniquely segment authorization to retain a least-privilege model for all resources and backend endpoints.

    Future of GraphQL

    To date, GraphQL has gained modest prominence. But, Jhingran foresees an incoming inflection point in the next two to three years. “The dominoes are beginning to fall,” he said.

    Now that the market is saturated with APIs, characteristics outside of functionality will be what sets a service apart and gives it a competitive edge. This includes things like performance, security and developer experience, the latter of which makes GraphQL an attractive proposition. This, paired with the explosion in GraphQL tools, will undoubtedly help developers perform their jobs better.

    Looking to the future, Jhingran predicts increased usability and abstraction in how developers work with GraphQL. He compared it to how disjointed pages on the web became interconnected and searchable with links. We see similar thinking happening in the world of APIs, he explained, in which the disjointed APIs require a better connection. Similar to how Google simplified search, he foresees GraphQL and the tools that work with it helping to decrease cognitive load by abstracting complexity with declarative commands.

  • vFunction Tool Uses AI to Help Convert Monolithic Java Apps

    vFunction Tool Uses AI to Help Convert Monolithic Java Apps

    vFunction today added the ability to employ artificial intelligence (AI) to automatically assess the level of effort required to convert a monolithic Java application into a set of microservices.

    Bob Quillin, chief ecosystem officer for vFunction, said the goal is to make it simpler for IT teams to more effectively employ the company’s vFunction Application Transformation Engine by using a vFunction Assessment Hub that assesses the level of technical debt involved.

    Legacy monolithic Java applications tend to have a lot of technical debt and dependencies that make them a challenge to convert into microservices, he noted. Most simpler Java applications have either already been converted or are in process, Quillin added.

    The vFunction Assessment Hub enables IT teams to better predict the outcome of refactoring a Java application to better prioritize which Java application should be converted first based on, for example, the level of scalability that can be achieved, said Quillin. Current assessment approaches involve a lot of manual effort and which typically requires organizations to hire external contractors to perform, he added.

    An AI-based approach reduces the total cost of creating a business case and constructing a roadmap for modernizing existing Java applications, said Quillin. The vFunction Assessment Hub measures application complexity based on code modularity and dependency entanglements and then assesses the level of risk changes might have on the stability of the application based on the depth and length of the dependency chains. It then surfaces benchmarks for debt, risk and complexity in addition to identifying aging frameworks that could create security risks and licensing issues.

    Thus far most organizations have been content to lift and shift Java applications into the cloud without modernizing them. The application simply now runs on infrastructure that is managed by an external cloud service provider. That approach does little to improve the scalability of a monolithic Java application, noted Quillin. The goal should be to convert that Java application into a set of microservices that run natively in the cloud, he added.

    The vFunction Assessment Hub is already being used within a Java Refactoring Service for the Azure cloud provided by Microsoft that is based on agent software to analyze code and the vFunction Application Transformation Engine.

    In general, organizations are embracing microservices to build applications that are both more scable and resilient. In the event a microservice becomes unavailable, requests for services are rerouted to another microservice to make sure the application degrades gracefully, rather than simply crashing altogether. The challenge, of course, is that the dependencies that can exist between hundreds of microservices can be more challenging to manage over time than those of the monolithic application being replaced.

    It’s not likely that microservices-based applications will entirely replace monolithic applications any time soon, but the bulk of new applications are being built on top of a microservices-based framework. As such, many DevOps teams are now managing a broad mix of monolithic and microservices-based applications across IT environments that become increasingly more complex to manage with each passing day.

  • Database Migration From Microsoft Azure to Snowflake on AWS: Part 2

    Database Migration From Microsoft Azure to Snowflake on AWS: Part 2

    In part one of this blog series, we looked at our approach to migration by leveraging the Azure data factory, tuning the performance of Data Factory Pipeline, using Parquet files and increasing computing power. 

    In this blog, we’ll look at some of the challenges encountered during the migration, cost-saving actions and our approach to data validation.

    Challenges Encountered 

    • Data Factory export failed for tables if the column name had spaces or special characters. In such cases, aliases were used (replacing spaces with an underscore) for column names and the query method was used as a source.
    • Data Factory export failed for the column data type “Geography” and had to be cast to varchar and then exported to Blob.
    • Some column names in SQL server were keywords in Snowflake, which didn’t allow it to migrate. For example, ‘LOCALTIME’ is a column name in SQL but is also a keyword in Snowflake, and it is addressed using inverted commas as escape characters (“”) while creating the Snowflake table.
    • During ingestion into Snowflake, the flat files were named after tables, listing files based on the name pattern. For example, if we wanted to list the file with table name test_1, Snowflake would list all the files starting with the name test_1. It is essential to make sure that each file name is unique.
    • If the file to be ingested was incorrect with respect to the table, Snowflake did not return any error messages or warnings. It simply ingested the file and put NULL wherever column mapping was not found.
    • While creating the tables, if the column name had spaces, escape characters were used so that the same spaces could be maintained. But for ingestion of such columns, the column mapping from a flat file would be the same aliases that were used in the source query.
    • Triggers and compute columns were not supported in Snowflake and had to be eliminated while creating the tables.
    • Also, data types that did not exist in the Snowflake schema had to be changed, keeping in mind the correctness of data.

    Cost-Saving Actions

    • Use of the Parquet file facilitated a compression rate of up to 75%. The 800GB of data was compressed to 234GB in multiple files which reduced the storage cost of Blobs.
    • Detailed POC and analysis discovered that Snowflake ingestion was optimal with Small for moderate sized tables and Medium for large sized tables which kept Snowflake costs in check.
    • Self-hosted IR saved on the data pipeline operating cost as the compute power of Azure VM was used.

    Cost Estimation 

     

    Data Validation

    The vital part of the project was ensuring that the integrity and correctness of the data were maintained during the migration process. As a part of standard DataOps best practices, a thorough data validation activity was performed to make sure that the data stayed meaningful and remained of use to drive business decisions.

    The approach to data validation was two-fold:

    In the first approach, all the distinct data types in the SQL server that were part of the db migration were listed and sampled randomly across one row per data type. The same row was fetched from Snowflake and matched.

    List of tables based on data type

    select table_catalog,table_schema,table_name,COLUMN_NAME,* from
    INFORMATION_SCHEMA.COLUMNS
    where DATA_TYPE = 'money' and TABLE_NAME not like 'sys%'

    Issues like that shown below were uncovered in this validation process and were corrected for all instances of the data type.

    The second approach involved writing an automated script that accomplished the following results (More details can be found here):

    • Script touched each table of each database
    • A set of five records were randomly sampled from both sides and matched
    • If the tables were empty, it was recorded as tables with zero records
    • If the sample sets did not match the script, it wrangled the sets and sorted them one column at a time. If the sample matched the script, it was recorded. This wrangle took care of the scenario wherein the first n number of columns had duplicate records
    • Any failed validation attempt would be visible in a generated log report:

    Validation Started for Table: Test1

    —-> Validation Passed for table Test1

    Validation Started for Table: Test2

    —-> Validation Passed for table Test2

    Validation Started for Table: Test3

    —-> Table has 0 records

    Validation Started for Table: Test4

    —-> Validation Failed for table Test4

    Validation Started for Table: Test5

    —-> Validation Passed for table Test5

    ============= Report =============

    # Total Number of Tables Scanned: 5

    # Tables with 0 records: 1

    # Passed: 3

    # Failed: 1

    # Percentage 60.5556123