staging-devopsy.kinsta.cloud

Tag: testing

  • Harnessing Generative AI for Feature Management Testing

    Harnessing Generative AI for Feature Management Testing

    When it comes to DevOps, the emerging integration of generative AI into feature management testing marks a significant evolution. We’re going to take a no-nonsense look at how this technology is revolutionizing the way we create testing environments. 

    Using artificial intelligence to generate tests allows us to mimic a vast array of user scenarios and environments, which means we can develop and deploy features that aren’t just good but great—robust, efficient and ready for whatever the real world throws at them. 

    Embracing Generative AI in DevOps Testing

    Artificial intelligence is dramatically transforming the world around us, impacting a wide variety of industries at breakneck speeds. A stunning 83% of companies assert that AI is a critical component of their business strategies, particularly when it comes to software development.

    Generative AI has quickly become a transformative tool when it comes to DevOps, particularly when it comes to feature management testing, by leveraging machine learning algorithms to generate data and test scenarios that closely mimic real-world conditions.

    Traditionally, DevOps teams have relied mostly on manual and a couple of automated testing methods to validate their features. However, these methods often fail to accurately simulate the wide variety of complex and unpredictable user interactions and environmental variables. 

    By creating nuanced and diverse test cases, generative AI can fill this gap, enabling a more thorough evaluation of features under various conditions. Even now, on a smaller scale, WordPress site owners are using predictive analysis and even simple comparisons to properly orchestrate the rollout of new features–with over 60,000 plugins to choose from, generative AI has the potential to prevent lag, which can often be the result of a single shoddy plugin. 

    Simply put, AI is actively improving how features are implemented and tested across various platforms by intelligently crafting diverse test scenarios. 

    Enhancing Test Coverage and Efficiency

    The significant expansion in test coverage is one of the key benefits of integrating generative AI into DevOps testing. Traditional testing methods are typically limited by the scenarios that testers can foresee and manually script.

    In contrast, AI-driven testing tools can generate a multitude of unique, unexpected test scenarios, capturing edge cases that might otherwise go unnoticed, which is crucial in identifying potential bugs and vulnerabilities early in the development cycle, saving time and resources in the long run. 

    Additionally, AI can automate both the creation and implementation of these tests. This simplifies the testing workflow, allowing human testers to dedicate their full attention to more intricate and challenging tasks.

    Simulating Real-World Scenarios

    Another key advantage of generative AI in DevOps is its ability to simulate real-world user behavior and environments. 

    AI models can create realistic and diverse user scenarios by analyzing historical data, user patterns, and environmental variables. This type of functionality is a huge asset in feature management, where understanding how new features will perform in the real world is paramount. 

    To use a couple of examples, an AI model can generate test cases that mimic peak usage times, varying network conditions, or different user demographics. Furthermore, companies like Netflix, Google and IBM are already actively using generative AI to simulate real-world scenarios for various applications, such as content delivery optimization, predictive maintenance, and software testing. 

    Accelerating Feature Development in DevOps

    The integration of generative AI into feature testing also accelerates the feature development and deployment process. 

    A well-constructed prompt chain can set up an environment where DevOps teams can effectively compare different products/services, especially when it comes to more sensitive software, such as cloud management platforms and even find the best ways to escape vendor lock-in without actually wasting time manually comparing and testing everything. 

    Applying generative AI can also drastically cut down on the time it takes to validate a new feature by generating and running through thousands of test scenarios in quick succession. In DevOps environments, where rapid iteration and continuous deployment are key, this is essential. 

    AI-driven testing can also provide developers with immediate feedback, allowing for quick adjustments and improvements. This ensures that features are developed faster while meeting the highest quality standards before proper deployment.

    Potential Challenges and Considerations

    While the potential of generative AI in DevOps testing is immense, there are challenges and considerations that teams must navigate and work to overcome to successfully apply this powerful new tool. 

    One key issue is the quality and relevance of the data used to train AI models; since inaccurate or biased training data can lead to suboptimal or even harmful outputs, carefully selecting and curating data is a crucial step​​​​.

    Incorporating generative AI into established DevOps processes can also present challenges in terms of complexity and resource demands. Organizations may face significant challenges in modifying their existing pipelines and processes to accommodate AI technologies, which often require substantial investments in hardware, software and skilled personnel.

    Lastly, the need to continuously improve and update AI models cannot be overstated. As new technologies and security threats emerge, these models must be regularly refined and adapted to remain effective and relevant, which will require ongoing investment in AI systems and a commitment to staying abreast of evolving technologies and methodologies in what is a rapidly evolving and advancing field.

    Future Directions of Generative AI

    Looking forward, the role of generative AI in DevOps is poised to grow even more prominent, and as AI models become more sophisticated and data-rich, their ability to generate realistic and complex test scenarios will only improve. 

    But let’s look beyond just creating testing environments and isolated testing environments: A well-executed, AI-aided process can reduce costs during SAP staff augmentation and can also streamline processes in other high-level enterprise systems such as Oracle ERP and Salesforce CRM. Overall, ERP software is a perfect candidate for DevOps feature management, as it allows devs to use generative AI for everything from writing code to custom tests. 

    We can also expect to see further integration of AI in other aspects of DevOps, such as monitoring and performance optimization. The ultimate goal is to create a seamless, AI-enhanced pipeline that tests and deploys features efficiently and continuously learns and improves from real-world feedback.

    To better illustrate this transformative impact, some key examples of how generative AI will further impact DevOps in the future include the following: 

    • AI-Augmented Development: Generative will assist developers with code suggestions, bug fixes, and even making architectural decisions, thereby streamlining the development workflow​.
    • Improved Infrastructure Management and Automation: The use of predictive analytics, driven by generative AI, will play a crucial role in anticipating infrastructure requirements, facilitating preemptive scaling, and efficient distribution of resources.
    • Adaptive Problem Resolution and Incident Handling: Generative AI is set to revolutionize the way operational problems are identified, diagnosed, and resolved swiftly, greatly boosting the dependability and steadiness of DevOps practices. This is evident in platforms such as Google Cloud’s operations suite, where such AI capabilities are already making a significant impact. 
    • Generative AI in Proactive DevOps: This emerging trend focuses on leveraging generative AI to anticipate potential issues before they manifest, which enables preemptive maintenance strategies and is aimed at reducing system downtime.
    • AI-Driven Testing and QA: AI-powered testing will become more sophisticated, with capabilities such as automated test case generation, test data creation, and advanced anomaly detection​.

    Harnessing Generative AI in DevOps

    Integrating generative AI into DevOps is a groundbreaking development poised to revolutionize how we approach software development, testing, and infrastructure management. 

    This innovative technology offers immense potential to automate complex tasks, enhance efficiency and improve the quality and reliability of software products. However, as with any emerging technology, its adoption comes with its own set of challenges and considerations.

    Enterprises aiming to leverage generative AI within their DevOps practices must address challenges associated with data integrity, intricate integration processes, adherence to privacy laws and the necessity for ongoing enhancement of their AI models.

    Despite these challenges, the benefits of adopting generative AI in DevOps are quite substantial, ranging from increased efficiency and cost savings to enhanced scalability and improved overall software quality.

  • Embracing the Future: Navigating the Waves of AI in DevOps

    Embracing the Future: Navigating the Waves of AI in DevOps

    In the rapidly evolving digital landscape, the advent of generative AI and large language models (LLMs) has ushered in a new era of innovation and transformation. As IT organizations pivot toward DevOps, DevSecOps and SRE mastery, leaders find themselves at the crossroads of excitement and anxiety. This dual sentiment stems from the vast potential of AI to revolutionize operations and the inherent challenges that accompany generative AI’s integration. In this article, we’ll explore the reasons behind this excitement and anxiety, outline the top use cases for generative AI and provide strategic guidance to ensure a smooth DevOps and digital transformation journey.

    The Catalysts of Excitement

    1. Enhanced Efficiency and Innovation: Generative AI, with its ability to automate complex processes and generate new ideas, promises unprecedented efficiency. It enables IT organizations to streamline development pipelines, reduce manual errors and foster innovation, making the leap toward DevOps and DevSecOps not just a goal but a tangible reality.
    2. Improved Decision-Making: Large language models offer sophisticated data analysis capabilities, turning vast datasets into actionable insights. This empowers leaders to make informed decisions rapidly, optimizing operations and enhancing service reliability in line with SRE principles.
    3. Competitive Advantage: In the digital age, staying ahead means adopting the latest technologies. Generative AI provides a unique competitive edge, allowing organizations to deliver superior digital experiences, innovate products and services and respond to market changes proactively.

    Top Three Generative AI Use Cases in DevOps

    1. Automated Code and Test Generation and Review: AI-driven tools can generate code and test snippets, conduct code reviews and testing and suggest optimizations, accelerating development cycles and improving code quality.
    2. Security Enhancement: By integrating AI into security protocols, organizations can predict and mitigate potential threats more effectively, enhancing their DevSecOps initiatives with proactive rather than reactive measures.
    3. Incident Management and Resolution: AI can automate incident response workflows, predict outages before they occur and suggest remediations, aligning with SRE goals of maintaining high availability and reliability.

    Sources of Anxiety

    1. Ethical and Privacy Concerns: The potential for misuse of AI technologies, including privacy violations and biased decision-making, raises significant ethical questions, creating apprehension among leaders.
    2. Skill Gaps and Workforce Impact: The shift toward AI-centric operations necessitates new skills and roles. Leaders worry about the existing workforce’s ability to adapt and the challenge of filling emerging skill gaps.
    3. Integration and Compatibility Issues: Integrating AI into legacy systems and ensuring compatibility across the digital infrastructure poses technical challenges, complicating the transition to advanced DevOps and SRE practices.

    Navigating the Digital Transformation Journey

    To harness the benefits of AI while mitigating its risks, leaders should consider the following strategic actions:

    1. Foster an AI-Ready Culture: Cultivate a culture of continuous learning and adaptability. Encourage your team to embrace AI tools and methodologies, emphasizing the importance of upskilling and reskilling.
    2. Implement Ethical AI Guidelines: Develop and adhere to ethical guidelines for AI use, focusing on transparency, accountability and fairness. Ensure privacy and security measures are integral to your AI initiatives.
    3. Invest in Talent and Training: Address skill gaps by investing in training programs and partnerships with educational institutions. Consider hiring AI specialists to bridge the gap between traditional IT roles and AI requirements.
    4. Prioritize Seamless Integration: Adopt a phased approach to AI integration, ensuring compatibility with existing systems. Leverage APIs and microservices architectures to facilitate smoother transitions.
    5. Establish Governance and Oversight: Create a governance framework to oversee AI initiatives, ensuring alignment with organizational goals and compliance with regulatory requirements. This should include monitoring AI performance and its impact on operations and workforce dynamics.

    Summary: Seize the AI Opportunity

    As we stand on the brink of a transformative era, the fusion of AI with DevOps, DevSecOps and SRE practices represents a significant leap forward. While the journey is fraught with challenges, the potential rewards are immense. By embracing AI with strategic foresight, ethical consideration and a commitment to continuous improvement, IT leaders can propel their organizations toward unparalleled efficiency, innovation and competitive advantage.

    In conclusion, the integration of AI into DevOps to further accelerate digital transformation is not just an option but a necessity for staying relevant in the digital age. The journey requires careful navigation, but with the right approach, IT organizations can emerge stronger, more agile and better equipped to face the future. Let’s embrace this opportunity to redefine the digital landscape, ensuring our transformations are not just successful but sustainable and responsible.

    Embrace the future; embrace AI.

  • Kobiton Open Sources Test Script Generation Tool

    Kobiton Open Sources Test Script Generation Tool

    Kobiton today announced it is open sourcing a script generation tool for the Appium automation framework that automates the running of test scripts.

    The overall goal is to make the tool, which makes it simpler to create more testing scripts, more widely available and, ultimately, lead to higher-quality applications being deployed.

    Kobiton CTO Frank Moyer said the tool makes it possible to generate tests from a test case that has been captured in a Node.js format. Rather than having to hire hundreds of testers to create those scripts manually, it’s now possible to automate those tasks using an open source tool, he added.

    On average, Kobiton claims the tool it developed can write a 30-step script in 10 minutes, whereas a manual approach could take as long as four hours to construct a similar test.

    Once created, those test scripts can automatically be executed using the Appium framework. The output from that tool can also be used on other testing platforms, including Sauce Labs, BrowserStack and LambdaTest or integrated with low-code testing platforms from Perfecto and Tricentis.

    It’s no secret that the quality of any given application tends to vary widely, not just between organizations but also within them. Most of those issues can be traced back to a lack of testing that usually occurs because there isn’t enough time. As deadlines near, one of the first places organizations look to make up for delays is the time available for testing. The hope is that whatever issues arise will be addressed via the next update. Of course, when the update falls behind schedule, the testing process is again cut back.

    Ideally, application development projects would not be allowed to go forward without adequate testing, but given business pressures, they often do. The only way to limit the impact of application development delays is to reduce the time required to create and run tests. The more automated that process becomes, the less time there will be spent on manually running routine tests. In fact, hopefully, there might even be enough time to run more complex tests to uncover issues that might adversely impact the user experience in ways no one might have otherwise considered.

    One way or another, the quality of the applications being deployed needs to improve if for no other reason than end users have options. Most end users will never give a mobile application a second chance if the initial experience doesn’t meet expectations. At the same time, application security regulations are only going to become more stringent. The tolerance for known vulnerabilities discovered after an application has been deployed is falling. Many organizations will find themselves subject to fines being levied simply because there was not enough adequate testing baked into the software supply chain.

    In fact, application security should, in the final analysis, be an element of a large quality assurance process that begins with adequate testing.

  • SmartBear Acquires Reflect to Gain Generative AI-Based Testing Tool

    SmartBear Acquires Reflect to Gain Generative AI-Based Testing Tool

    SmartBear this week revealed it has acquired Reflect, a provider of a no-code testing platform for web applications that leverages generative artificial intelligence to create and execute tests.

    Madhup Mishra, senior vice president of product marketing at SmartBear, said the platform Reflect created will initially be incorporated into the company’s existing Test Hub platform before Reflect’s generative AI capabilities are added to other platforms.

    Reflect provides access to a natural language interface to create tests using multiple large language models (LLMs) that it is designed to invoke. It can also understand the intent of a test to understand what elements to test regardless of whether, for example, a button has been moved from one part of a user interface to another, said Mishra. Test step definitions, once approved, can also be automatically executed using scripts generated by the platform.

    SmartBear has no plans to build its own LLMs, said Mishra. Rather, the company is focusing its efforts on providing the tools and prompt engineering techniques needed to effectively operationalize them, he added.

    Reflect is the tenth acquisition SmartBear has made as part of an effort to provide lightweight hubs to address testing, the building of application programming interfaces (APIs) and analysis of application performance and user experience. Last year, the company acquired Stoplight to gain API governance capabilities.

    Rather than building a single integrated platform, the company is focused on providing access to lightweight hubs that are simpler to invoke, deploy and maintain, said Mishra. The overall goal is to meet IT teams where they are versus requiring them to adopt any entirely new monolithic platform that requires organizations to rip and replace every tool they already have in place, he said.

    There is little doubt at this point that generative AI will have a profound impact on application testing in a way that should ultimately improve the quality of the applications. As the time required to create tests drops, more tests will be run. Today, it’s all too common for tests not to be conducted as thoroughly as they should be simply because either a developer lacked the expertise to create one or, with a delivery deadline looming, they simply ran out of time.

    Naturally, the rise of generative AI will also change how testing processes are managed. It’s not clear how far left generative AI will push responsibility for testing applications, but as more tests are created and run, they will need to be integrated into DevOps workflows.

    Of course, testing is only one element of a DevOps workflow that is about to be transformed by generative AI. DevOps teams should already be identifying manual tasks that can be automated using generative AI as part of an effort to further automate workflows that, despite commitments to automation, still require too much time to execute and manage. Once identified, DevOps teams can then get a head start on redefining roles and responsibilities as generative AI is increasingly operationalized across those workflows.

  • Sauce Labs Adds Visual Testing Tool to Platform

    Sauce Labs Adds Visual Testing Tool to Platform

    Sauce Labs has added native visual regression testing capabilities to its testing platform to enable DevOps teams to streamline workflows.

    Marcus Merrell, vice president of technology strategy for Sauce Labs, said Sauce Visual provides the ability to test user interfaces without requiring DevOps teams to acquire, maintain and integrate a separate tool.

    The overall goal is to make it possible to test functions and user interfaces in parallel by adding a single line of code to a DevOps workflow, he added. Updates to the visual presentation of a page are automatically highlighted, with any visual inconsistencies automatically surfaced. In comparison, a standalone visual testing tool increases the time it takes to execute a test suite, noted Merrell.

    Sauce Visual natively uses the same tools as the rest of the Sauce test automation platform, and can be integrated into any continuous integration/continuous delivery (CI/CD) workflow. At launch, Sauce Visual is compatible with Java/JUnit, Java/TestNG, WebDriverIO, Selenium, Appium (web) and Cypress testing tools with support for Playwright, Puppeteer, TestCafe, Espresso, and XCUITest forthcoming.

    The Sauce Labs approach also serves to reduce the total cost of application testing at a time when the number of applications organizations are building and deploying only continues to increase exponentially, said Merrell.

    In general, organizations are attempting to strike a balance between shifting responsibility for testing too far left toward developers that typically lack testing expertise and the need to verify application experiences are delivered as expected. Many organizations have shifted more responsibility for testing toward developers in the hope of improving code quality and reducing the number of vulnerabilities that might find their way into a production environment. The challenge is those efforts have also increased the level of cognitive load developers need to carry at the expense of having more time to write application code.

    Of course, in time, artificial intelligence (AI) tools will make it easier to generate test scripts, but there is still going to be a need for testers who were not involved in the writing of code to validate it, noted Merrell. The difference is those testers will not necessarily need to be rocket scientists as the process of creating tests becomes more automated, he added.

    Testing user interfaces is especially problematic for developers because many of them don’t have a lot of expertise in the business functions that the application they are building enables. In addition, there are often minute changes to user interfaces that developers are not always going to discern.

    As always, one of the first things that tends to be reduced whenever a developer falls behind schedule is testing. In a recent Sauce Labs survey found more than two-thirds (67%) admitted they pushed code into a production environment without testing, with more than a quarter (28%) acknowledging they do so on a regular basis.

    The assumption is that if any issues do arise, they can be addressed via the next update to the application. End users, however, are less tolerant of that approach when, in the age of digital services, an alternative application is readily available.

  • 2024: The Year of Testing

    2024: The Year of Testing

    Caveat: I currently cover DevOps – including test – and security – including the *AST. This does give me a viewpoint that others may not have, and also potentially gives me blinders that others may not have. I am not generally a predictor but a reader of direction. Proclamations like this are rare for me and are shorter-term/more tactical than the grab bag of predictions usually seen this time of year.

    I was thinking about this one this morning; the direction of travel of traditional and security testing is very similar at the moment. We are seeing the vast complexity of the overall space and the rapid growth of automation come together to once again make all testing a topic worthy of consideration.

    I loved test-driven development (TDD). I can’t speak for others, but RPC/API is just a cool way to implement centralized functionality, and TDD is almost mandatory to develop APIs and the clients that call them. There is no UI to distract or have separate requirements in API development; there is a stated function and correct or incorrect behavior. To a computer scientist, this is predictable beauty. TDD made it more than predictable—it made it a trail through the woods. “We know it needs to do X, Y and Z, with the only side effects being A,” can be detailed with tests to prove that is what it does, and then look to prove the “It doesn’t have other side effects” part. And you can do it all while developing—no need for dev/test iterations. Like any IT solution, it isn’t a great fit for everything, but even if it isn’t called TDD, it is nearly impossible to develop API-based solutions without some level of TDD.

    The thing is, TDD is absolutely a developer’s tool. Developers and DevOps pretty much must use it to turn out usable APIs. The test team only has a use for a subset of those tests. Because if the dev team did it right, they passed all the TDD tests, and the test team’s role in that scenario is to look closely at what the developers might have missed. Integration is often beyond the control of an individual dev and/or API, for example, because many different clients may be integrating with it. Test teams, in a true TDD shop, fill the gaps with meaningful tests that evaluate the overall application quality.

    Security testing—at least static application security testing (SAST) and increasingly dynamic application security testing (DAST)—are also becoming a core part of the development process. This is as it always should have been. It is far easier to fix a security flaw (or any flaw) at the design/early code stage than in a testing process after deployment. The quality of security testing inputs is also increasing, making it far more viable. And finally, the tools are being effectively integrated. Being a developer who did a lot of security work because we often needed it and had no one to do it, I will say I get a little rush about being told right in my IDE about the security risks in my code. I write, I see, I revise where necessary, and I turn out better code. All without having to pause for security or bang heads over policy.

    To date, all this testing is separate, but I suspect, having vaguely the same location in the process of development, that testing will all occur together and the markets may even start to merge a bit—though merger of security testing and functional/performance testing tools is a bit of a stretch, it is more possible than in the past. In most orgs, even though developers have an increasing stake in both sets of testing, the target purchasers are still two different groups—security versus test—with different priorities. That’s why I think they aren’t likely to merge any time soon.

    While AI in most technology spaces is currently experiencing the same level of FUD as XML, Java, JavaScript or low-code was in the past, all super-hyped tools in tech do have a sweet spot. Security and testing are in that sweet spot for AI. Both in test generation and in results evaluation, it does it better, faster and with more coverage while giving staff a concrete set of results to pore over and improve security posture and/or code quality. I cannot say loudly enough that in terms of enabling the steps in the SDLC that have suffered from a lack of resources and timeline to write, execute and evaluate tests, AI is a game-changer.

    So, all of these things together mean that 2024 is a good time to take a hard look at your testing environment(s) and architecture, both traditional testing and security testing. Now that AI has made advanced automation a fact, it is time to consider implementing the level of testing we always knew we should have but never had the resources. Build it right into the build process and make it part of the organizational DNA—just like Agile is.

    If you could hire a specialist who would paw through your code and give you a list of the 25 things you need to fix first, you would do it. That is, essentially, what modern testing tools are. They can do more tests, evaluate more issues and side effects and filter down to the “No—this is really important!” list without costing the team a ton of time.

    And keep rocking it. We’re all sitting on bugs and vulnerabilities. I do hope we are way past denying that fact. Apps are big old complex monsters; these days, sometimes built from generations of technologies cobbled together. Of course they have issues. Even yours. So let’s take steps to clean it up, just a little bit. And keep clipping along.

  • FAQs for Software Testing: All the Background You Need

    FAQs for Software Testing: All the Background You Need

    No matter what type of application you and your team are building, testing is a critical step in the software development life cycle (SDLC) that can not be overlooked. In this article, I’ll answer frequently asked questions about software testing, why software testing matters and suggest best practices to help ensure your application is ready to delight customers by the time it is deployed.

    Why Do We Need Testing?

    Software testing involves evaluating an application to see if it meets the requirements and functions it was built for. The process includes identifying defects, errors and bugs, along with flaws in user experience. Software testing is so critical because finding and fixing these bugs and issues in the development process means the end product reaching customers works properly and reliably and delivers a high-quality user experience. Additionally, testing during the development process makes bugs easier and less expensive to fix as opposed to making fixes after an application has been deployed.

    Overview of Testing Types

    Let’s quickly define the common types of software testing:

    ● Unit Testing: This process evaluates individual components of software to ensure they are working properly.
    ● System Testing: This involves ensuring an entire application or system meets requirements and functions as it should, end to end.
    ● Integration Testing: This looks at whether or not different modules of an application or system work properly together.
    ● Regression Testing: This process revisits previously tested components of software to make sure they work correctly after a change has been made.
    ● Acceptance Testing: This process determines whether an application meets the needs of end users and is ready to be deployed.
    ● Usability Testing: This process assesses how user-friendly a customer will find the product.

    Creating the Plan

    Any solid software testing plan is going to incorporate multiple testing approaches, including manual, automated, user experience, exploratory and non-functional testing. You’ll start by defining the scope for what parts of your application, integrations and functionality you need to test for. Then, you can decide what type of testing methods are needed for each and document these to ensure consistency.

    You’ll want to combine code coverage and test coverage to ensure the code for your application is tested thoroughly. The type of tests you prioritize will vary based on different products, and your testing strategy will certainly evolve and grow over time.

    Measuring Your Strategy for Thoroughness and Effectiveness

    To ensure your testing strategy is thorough and effective, you will need to make sure to test for code, device coverage and all aspects of user experience. The bottom line for the program should be delivering value to the business by making sure the application works as intended for users. To be effective, testing should take place before an application reaches end users, so issues and bugs can be identified and fixed before deployment. To be thorough, testing strategies must be balanced, end-to-end, and look at functionality, usability, accessibility and more.

    Effective software testing can be measured in a few ways, including:

    ● Code Coverage: The percentage of code that is tested.
    ● Test Case Effectiveness: The percentage of test cases that found defects or bugs.
    ● Time to Detect and Fix Defects: The time it takes to find and fix issues in an application.
    ● Defect Density: The number of defects found per unit of code.

    Improving Your Testing Approach

    Your software testing program should be evolving and improving over time and can improve by:
    ● Including test-driven development
    ● Leveraging a shift-left approach
    ● Using automation where possible
    ● Benchmarking QA for current projects versus previous ones
    ● Conducting a cost-benefit analysis

    Testing Best Practices

    While the below list is not exhaustive, here are some top best practices to keep in mind as you build out a software testing practice at your organization:
    ● Incorporate the voice and feedback of your customer into product design
    ● Make accessibility a priority
    ● Test throughout the entire SDLC, including in pre-production, after changes are made to code, and in-sprint testing
    ● Maintain a matrix for the different devices using your application
    ● Prioritize high-quality user experience
    ● Automate repetitive tests where possible
    ● Review and adjust testing processes regularly, as needed
    ● Track and report results to analyze trends and identify areas for improvement
    ● Balance between leveraging different testing types

    Your organization’s software testing strategy directly impacts the quality of the end application you deliver to your customers. Poor usability, accessibility, and reliability of an application can frustrate users and send them to competitors, while high-quality user experiences and application performance can boost adoption and keep customers happy and loyal to your brand. With so much competition, taking the time to build out, adjust and prioritize software testing is critical for modern businesses.

  • Steady On. We Still Have a Job to Do

    Steady On. We Still Have a Job to Do

    It is all the rage to talk about how significantly large language models (LLMs) and generative AI will change every niche and corner of IT. And change IT it will. But the hype cycle is at its highest right now, and I’d caution against getting too worked up about it. We’re in for a few interesting years, and the spaces I cover (DevOps and security—with a more application-centric perspective but with ops experience) are definitely going to benefit from the various AI bits being worked in. All testing—including AppSec, but all testing—will benefit both in test generation and results analysis/filtering. In fact, I’ll go ahead and say that the question in testing will no longer be, “How much time to we have to devote to it?” and will rapidly become “How many systems’ resources do we have to dedicate to it?” It will still be a long process—AI isn’t speeding up the cumulative time it takes to run thorough testing (be it functional or security). It will make that the bottleneck though, because test generation will at least be LLM-assisted, and will eventually become completely generative AI-created while at the same time, results will get more and more filtering and analysis. A minor result here plus a minor result there will automatically be detected as an extended vulnerability. There are a ton of other possibilities, but these two seem pretty obvious when I look across the testing technology space. More testing and better results analysis are on the horizon—without requiring more staff. Of course, you can do so much more, which would require the organization to add staff to follow up on results, but that’s nothing new; testing finds issues that are interruptive and take man-hours to fix—we all know of bits in the application portfolio that we should probably fix but that aren’t high enough priority to do so, for example. But fixing is different than finding. I have seen demos in security scanners that offer auto-fix options for developers. Run the test during CI, create a ticket and let the developer see suggested code rework right in the IDE. It is early days for this, but if you look, you can find vendors doing it. And that’s really rather cool.

    But we’ve all heard the “this changes everything” storyline over and over. 4GLs, XML, Java and JavaScript were all going to eliminate developers. In fact, they all created more developers. AI has a better chance of actually removing parts of our workload than any of these technologies did, but we’re still in “watch and wait” mode. It seems obvious that grunt coding will fall to LLMs … And aside from keeping staff trained to manage source, that is a good thing. We use open source, modules and libraries to do much the same thing today. I see this as equivalent to entry-level writing jobs. We’ll have to transform from the current model, but the current entry level development model wasn’t ideal anyway, so lets make a better one. Testing capabilities will be greatly enhanced, but testing of all kinds was so far behind new code generation at most orgs that the impact will be all positive as testing catches up. AI-based test generation today can generate thousands of permutations for a given piece of code, and analyze the results to just have analysts look at the truly disconcerting or confusing test results. And the question will become, “How many servers do we want to dedicate to run this?” Which is different, but just means we have more tests running to make a more solid and/or secure product and environment.

    So, in summary, is change coming? Yes. And that is not significantly different than the last couple decades where change has become IT’s constant companion. For most of you, the way you do your job today is not the same as the way you did it even two years ago. ML/GAI/LLMs are just another iteration. The apps still have to be built, deployed and maintained. So until HAL comes along, keep kicking it. You’re keeping it together, and who knows what the future brings.

  • New Relic Adds App Security Testing Tool to Observability Platform

    New Relic Adds App Security Testing Tool to Observability Platform

    New Relic today made available a public preview of an application security testing tool that will be integrated into its observability platform.

    Esteban Gutierrez, CISO and vice president of information security for New Relic, said the New Relic Interactive Application Security Testing (IAST) will provide DevSecOps teams with the context needed to identify the root cause of a cybersecurity issue down to specific lines of code.

    DevSecOps teams will be able to take advantage of IAST to identify vulnerabilities both in code as it is written and after it has been deployed in a production environment without any false positives, he noted.

    New Relic achieves that goal using deterministic testing techniques that surface an actual proof-of-exploit using observability data collected from both applications and the underlying infrastructure upon which they depend, said Gutierrez. DevSecOps teams are then provided with guided remediation suggestions, guardrails and tracking tools to both remediate the issue at hand and prevent it from reoccurring, he noted.

    New Relic IAST takes advantage of the agent software New Relic provides to collect observability data, which reduces the total cost of DevSecOps by providing IT teams that have adopted the New Relic platform with an integrated set of application security testing tools that doesn’t require them to acquire and manage a separate additional platform, added Gutierrez.

    That approach also ensures that application security testing is integrated with the continuous integration/continuous delivery (CI/CD) platforms that New Relic already observes, he added.

    The reason most vulnerabilities are not remediated in a timely fashion is the DevOps teams responsible for creating and deploying the required patches lack any context. They often don’t know, for example, how many instances of a vulnerability there might be or the actual level of severity. Without those insights, it becomes exceedingly difficult to prioritize remediation efforts, noted Guttierrez.

    As application security requirements become more stringent in the wake of forthcoming legislation, it’s now more of a question of when than if organizations will be required to better secure their software supply chains. The issue DevOps teams face now is finding a way to integrate application security testing tools into existing DevOps workflows without slowing down the pace at which applications are being developed.

    Of course, not every vulnerability discovered requires the same level of attention. Depending on its severity and likelihood of being exploited, DevOps teams need to prioritize which vulnerabilities need to be remediated first. There simply are not enough resources available today to fix every vulnerability an application might have. For the foreseeable future, there will remain technical debt involving vulnerabilities that will need to be addressed after an application is deployed in a production environment.

    The issue, of course, is that cybercriminals are getting more adept at exploiting those vulnerabilities. The amount of time any DevOps team has to resolve issues before they result in a breach continues to dwindle.