Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: Google’s full-stack, browser based development environment in the cloud. It’s not just a Copilot clone, but aims to help you “get an app from zero to production.” (more…)
At its recent DASH 2023 conference, Datadog revealed Flex Logs and Intelligent Test Runner, two major new capabilities aimed at improving logging and testing efficiency for engineering teams.
Omri Sass, director of product management at Datadog, said Flex Logs introduces a new middle tier of log storage between real-time querying and cold archives. As Sass explained, this provides a cheaper option for retaining logs beyond the typical two weeks for occasional analytics and investigations. By decoupling storage from compute for reading logs, teams can independently scale each based on their needs. This enables retaining rich log data for months or years to uncover long-term trends, he said.
In addition, Sass noted that Flex Logs helps address common log analysis limitations around storage costs and rehydrating archives. The feature better aligns log pricing with specific use cases beyond just troubleshooting. According to Sass, this evolution in log storage tiers will likely shift more logs from legacy on-premises systems to its cloud platform.
The second capability, Intelligent Test Runner, leverages observability signals to optimize CI test suites. As Sass explained, comprehensive test automation in CI pipelines often increases deployment times and flakes. Intelligent Test Runner inspects code changes to determine associated tests, ignoring irrelevant ones, which helps avoid long wait times for minor changes just to exercise unaffected areas.
By tapping into existing telemetry around code commits, test health and runtimes, the feature adapts testing to each code modification. Sass explained that observability data has matured enough to enable real-time optimization like this. The end result is faster CI validation, so engineering teams can ship updates more rapidly.
Sass highlighted how Flex Logs and Intelligent Test Runner are Datadog’s attempts to address real pain points developers face around managing log data life cycles and CI speed bumps. Sass said he believes this combination of depth in core disciplines like logs and application performance monitoring (APM), along with expanding into adjacent developer workflows, is key to solving these issues through innovation.
Looking to the future, the observability space continues to evolve rapidly as emerging technologies and practices expand monitoring beyond traditional IT infrastructure. Companies handling modern workloads expect their observability platform to provide comprehensive visibility as development toolchains and architectures grow more complex, with the next generation of observability likely incorporating advanced analytics, automation and AI to help engineers tame systems at ever-increasing scale and speed.
The second installment of Salesforce’s 2023 releases provides some great eye candy for DevOps pros. From new admin features to user enhancements and the anticipated Flow updates, a lot is covered in the nearly 700-page Salesforce Summer ‘23 release. Let’s break down the highlights of some appealing features, why these updates will save all Salesforce users time and energy and explore key details of the impending transition to Flows.
First, we’ll break down notable updates from this release that will save you time, help DevOps teams automate faster and enhance the experience for both you and your customers.
There are lots of great adjustments to some tools we know you’ve been waiting for and some we hope will be made available to all users.
● Mass Quick Actions on Related Lists: This is a beta feature that you can opt-in to try. You can now add and update records directly from a related list—a time and tab saver! This feature has been requested many times and will decrease the need for custom-built interfaces. For instance, if you’re on an Account and want to edit the related Contacts, you can add, edit or mass update directly from the Account screen. This saves clicks and stops new tabs from opening and losing your spot, etc. Allowing users to update in bulk is a major win!
● Einstein Search: Advance your search capabilities by having additional field types to search within, such as picklist, which were not previously searchable. For example, if you have a picklist field on Contact, you can search for one of the values and any records with that selected value will appear. Users will get more relevant results in their search. This new enhancement is in beta, so we’d highly suggest opting in to give it a test drive.
● Lightning reports and dashboards improvements, including:
Enhanced reports and dashboards with images and text descriptions let you add visualizations to your dashboards with company logos, helpful imagery or clarifying text. Explain exactly what the Dashboard is providing with helpful messaging or encourage the sales team with positive feedback about their work. This will not only better guide end users throughout the dashboard but will also increase engagement and personalization.
Additional dashboard filters permitted: Dashboard filter allowance jumped from 3 to 5, making dashboards more configurable and customizable for different views and end users.
A new ‘post to Slack’ option on dashboards: share analytics across your Slack workspaces so your colleagues can view, share, and subscribe to dashboard details and view in real-time.
Note: While all of these improvements to reports and dashboards are amazing features, they are only available to unlimited edition users. We hope Salesforce will trickle this down to enterprise users soon.
● MFA auto-enablement continues: Unless you’re using a single sign-on (SSO) solution like Okta, Salesforce mandates the use of multifactor authentication (MFA). Hard enforcement is coming by spring 2024 when Salesforce initiates a full rollout, so we’d suggest getting ahead of it by implementing this now.
With the forthcoming transition to Flows, we find that the more Salesforce does with Flows, the better. In general, it gives users better tools to deliver to customers and offers better experiences overall. These Flow Builder updates are key:
● Send API callouts without using code (this is in beta). This is like having Postman-like capabilities right in Flow!
● Create screens that react to changes from other components on the same screen. This is huge, since you can’t create reactive screens in Flows right now. Workflow optimization is all about reducing clicks, which users can now achieve with reactive forms (this is also in beta).
● Log emails and use email templates with the ‘Send Email’ action. This was rigid before, as users had to pull in their own templates or make very basic emails in Flows. This is a basic but hugely helpful upgrade.
We’re all always hunting for ways to save time and simplify processes. Once again, Salesforce delivered with the Summer ‘23 release. Fewer clicks, bulk actions and more sharing and collaboration capabilities. And this one has a little bit of excitement for everyone–admins, developers and end users.
The best thing you can do is to experiment with ways to optimize. There’s a lot in beta, so roll your sleeves up and dig in! Happy summer!
There is a growing reliance on web APIs. A full 92% of organizations plan to increase their investments into APIs over the next year, the 2023 Postman State of the API Report found. APIs are powering everything from internal microservices strategies to partner strategies and fully-fledged products.
Yet, this newfound API sprawl has consequences; a looming threat that might rise from the grave to haunt you …
I’m talking, of course, about zombie APIs. Zombie APIs are endpoints that are no longer maintained yet are still active. They may be unused endpoints, old features never officially deprecated or forgotten development or testing environments. Regardless, zombie APIs are a type of technical debt that could pose a legitimate threat if left to rot.
The sudden growth of APIs, along with immature inventory management and a lack of internal communication, can contribute to creating countless shadowy endpoints. Below, I’ll review some of the issues that zombie APIs might cause and consider some actionable steps DevOps teams can take to prevent them.
Forgotten APIs can cause a number of issues. For example, suppose an insecure testing environment was accidentally left exposed to the public. An attacker using automated endpoint scanning could find authentication and authorization holes to exploit, potentially exfiltrating sensitive data.
It’s also a bad idea to let any piece of technology go unmaintained, given the daily influx of new exploits and vulnerabilities. This point is accentuated by the connected nature of APIs, which may plug into a vast network of databases and allow both read and write privileges to the backend infrastructure.
Insecure APIs can cause a business both economic harm and damage reputation. Just consider the high number of API-related data breaches in recent years, which occurred at companies like Optus, Dropbox, Twitter and Zendesk. It’s even estimated that somewhere between $12 billion to $23 billion was lost in 2022 due to compromises in web APIs.
Zombie APIs are an easy target for these sorts of cyberattacks because they tend to have insecure designs that are easier to crack. They also tend to use immature monitoring systems, which means hackers can act under the radar more easily than with actively used endpoints.
As they say, you can’t secure what you don’t know about. In fact, OWASP lists improper inventory management as one of the ten top API security risks. Therefore, proper inventory management is a good first step to avoiding zombie APIs. Scan your service portfolio for forgotten endpoints, catalog them and update the catalog regularly.
It’s also a good practice to thoroughly document your services to aid in internal discoverability. If possible, adopting specification-first practices can also ensure the documentation and implementation are based on a single source of truth, thus avoiding mismatched versions and documentation drift.
“The key to preventing zombie APIs is visibility,” said Jeremy Snyder, CEO, FireTail. “Many organizations lack ongoing, up-to-date inventories of all the APIs that are running in either production or staging environments.”
Another method to reduce the chance of zombie APIs is to adopt mature API life cycle management practices. The chosen API versioning strategy should be clearly defined and codified. And versioning changes should be clearly communicated to all stakeholders.
On that note, it’s just as important to plan out a sunsetting, retirement and deprecation strategy from the onset. Then, remove deprecated services when they have reached their end of life! This is not a glamorous task, but it should be baked into someone’s assignments.
“Good cyber hygiene is an often overlooked strategy,” explained Snyder. “A second requirement is to have a policy around versioning and how many versions of an API can exist at any given moment. For instance, that policy may state that only the latest version can exist or the latest plus the previous. However, this relies on having visibility in order to maintain compliance.”
Other areas of proper life cycle management include employing governance around development and deployment and utilizing automated testing in continuous deployment (CI/CD) pipelines. Both of these could help set policies and security requirements, which should also bolster API safety.
Tribal knowledge is the antithesis of DevOps culture. As such, it’s a good practice to de-silo and share internal knowledge of API-powered services to limit the propagation of rogue endpoints. This could be done through private network repositories or even an internal Postman Collection or Workspace.
“Internal processes like organizational coordination and communication are critical,” added Snyder. “If teams don’t tell each other about new APIs being developed and launched into production, security teams can’t assess or audit them. Conversely, if app teams don’t know about network infrastructure resources designed to cover various API security aspects, developers don’t know what security controls need to be baked into the API code.”
A recent CISO survey revealed that API security was the number one concern for CISOs. For many reasons, web APIs are still low-hanging fruit for attackers. This is underscored by the 400% spike in malicious API attack traffic recorded by Salt Labs earlier this year.
Many of these attacks, we’ve also learned, are occurring over authenticated APIs. This shows that greater accountability and visibility are required to begin to stem the tide of API-related security incidents.
To reach an adequate level of visibility, leaders should consider adopting tooling that helps determine if there are outdated APIs or APIs with multiple live versions, said Snyder. “These tools should help display what security live APIs have and provide recommendations around how to fix them, if not providing the necessary code or modules to secure the APIs directly.”
At its DASH 2023 conference, Datadog today launched Bits, a digital assistant that uses generative artificial intelligence (AI) from OpenAI’s ChatGPT platform to provide real-time recommendations to resolve issues surfaced by the company’s observability platform.
At the same time, the company announced it is extending the reach of that platform to enable IT organizations to monitor the large language models (LLMs) many are now embedding within custom applications.
Datadog also released an LLM Observability platform in beta, which brings together data from applications, models and various integrations to help engineers discover and resolve AI issues in applications such as performance degradations, drift and hallucinations in real-time.
Those extensions complement an existing ability to monitor the usage patterns of application programming interfaces (APIs) making calls to the OpenAI ChatGPT platform.
Yrieix Garnier, vice president of product at Datadog, said the generative AI capabilities in beta are enabled by data the company used to extend the core capabilities of OpenAI’s ChatGPT platform.
Bits AI learns from both customer data and other sources to answer queries via a natural language interface that makes the Datadog platform more accessible to a much wider range of IT professionals, noted Garnier.
The overall goal is to reduce the mean-time-to-resolution for issues by, for example, explaining errors and suggesting code fixes that can be easily shared with developers via a Slack communication channel.
Those capabilities extend the functionality of the machine learning algorithms that Datadog has already embedded within its platform, noted Garnier.
While the generative AI capabilities added to the core Datadog platform will make DevOps teams more efficient, it’s not quite clear what role those teams will play in monitoring LLMs created by data science teams. In theory, DevOps teams should be responsible for optimizing the performance of any element of an application, but most DevOps engineers today don’t tend to have much experience with LLMs that are created by data science teams.
At the very least, however, DevOps engineers should be able to share the root cause of an issue that is adversely affecting those LLMs with those data science teams.
It’s not clear how quickly LLMs will be incorporated into applications, but most DevOps teams should expect them to soon be pervasive. In effect, Datadog is now providing DevOps teams with the AI tools they will need to effectively monitor applications infused with AI capabilities. The issue is not so much whether DevOps teams will be taking advantage of AI to better manage application environments but how quickly. Given the overall level of complexity DevOps teams regularly encounter, few are likely to want to take on that challenge without some level of AI augmentation.
The challenge is determining exactly what DevOps tasks might be best handled by a machine that enables software engineers to focus on more complex issues they previously didn’t have the time to address.
Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: Can the Linux desktop installed base ever break the mythical 10% barrier? Google has been refactoring ChromeOS, and GNOME is working on new window manager ideas. (more…)
In my recent blog, Revolutionizing the Nine Pillars of SRE with AI-Engineered Tools, I indicated that AI could assist in shifting the wisdom of production to the left. In this blog, I’ll explain in more detail how AI-engineered tools can be used to improve the pillar that I call SRE culture.
Shifting the “wisdom of production” to the left is all about bringing production considerations into the development process as early as possible. Here are some use cases where AI can help facilitate this shift, along with specific challenges and AI tools that can assist.
Predicting System Behavior: Predicting system behavior under various conditions can be difficult, especially as systems grow more complex. AI can help by analyzing historical data and predicting system behavior. Tools like Splunk can use machine learning to provide these predictions and facilitate earlier decision-making.
Automated Code Reviews: Manually reviewing code for potential production issues can be time-consuming and error-prone. AI can automate this process, identifying potential issues more quickly and accurately. Tools like DeepCode or Codota use AI to analyze code and identify potential issues.
Security Vulnerability Detection: Identifying security vulnerabilities early in the development process can prevent costly and damaging breaches in production. AI can analyze code to identify potential vulnerabilities that might be missed by human reviewers. Tools like Snyk or WhiteSource can help with this.
Performance Optimization: Predicting how code changes will impact performance can be challenging. AI can analyze changes and predict their impact on system performance. Tools like Akamas leverage AI to automate the performance optimization process.
Dependency Analysis: Identifying how changes in one part of a system will impact other parts can be difficult. AI can analyze system dependencies and predict the impact of changes. Tools like FOSSA can help in automating dependency analysis and identifying potential issues.
Resource Allocation: Predicting resource needs for new features or changes can help prevent resource shortages in production. AI can analyze resource usage patterns and predict future needs. Tools like Turbonomic or Densify use AI to optimize resource allocation.
User Experience Optimization: Predicting how changes will impact the user experience can help prevent negative impacts in production. AI can analyze user behavior and predict the impact of changes on user experience. Tools like FullStory or Crazy Egg can provide AI-driven insights into user behavior.
A practical implementation roadmap for introducing AI tools to help shift wisdom to the left may include the following steps.
Identify Needs and Goals: Start by identifying the areas where shifting wisdom to the left could have the most impact. This might include areas where issues are frequently discovered late in the development process or where manual processes are time-consuming or error-prone. Define clear goals for what you hope to achieve by introducing AI in these areas.
Research Tools: Once you’ve identified your needs and goals, research the AI tools available that could help meet those needs. Consider factors like the tool’s capabilities, integration with your existing systems, cost and support options.
Evaluate Tools: Based on your research, select a few tools to evaluate more thoroughly. This could involve setting up trials or demos, speaking with the tool’s developers or other users and assessing how well the tool meets your needs.
Pilot Implementation: Once you’ve selected a tool, start with a pilot implementation on a small scale. This will allow you to learn how to use the tool, identify any potential issues or obstacles and understand the tool’s impact before rolling it out more broadly.
Training: Provide training for your team on the new tool. This could include formal training sessions as well as ongoing support as they start using the tool.
Monitor and Adjust: After the tool has been implemented, monitor its use and impact. This could involve tracking metrics related to your initial goals, as well as soliciting feedback from your team. Use this information to make any necessary adjustments to your use of the tool.
Expand Use: Once you’re satisfied with the results of your pilot implementation, expand the use of the tool to other areas of your organization. Continue to monitor its use and impact and make any necessary adjustments.
Repeat: Technology is constantly evolving, and there may be new tools or features that could further improve your ability to shift wisdom to the left. Periodically revisit your needs and goals, and repeat this process to continuously improve your processes.
Shifting the wisdom of production to the left is an essential strategy for modern IT organizations, but it’s a journey that requires thoughtful planning, the right tools, and a commitment to continuous learning and improvement. As we’ve explored, there are several compelling use cases for leveraging AI in this transformative process. Whether it’s predicting system behavior, automating code reviews, detecting security vulnerabilities, optimizing performance, analyzing dependencies, managing resources or enhancing the user experience, AI tools are helping organizations tackle these challenges more efficiently and effectively.
While embarking on this journey may seem daunting, a well-crafted implementation roadmap can help make the process manageable and rewarding. By identifying specific needs and goals, researching and evaluating the right tools, piloting their implementation, providing comprehensive training and consistently monitoring and adjusting their usage, organizations can successfully adopt AI tools to shift the wisdom of production to the left. With continued effort and commitment, this transformation can lead to more informed decision-making, more efficient processes and, ultimately, better outcomes in production.
A survey of 404 IT professionals conducted by Snyk found the use of artificial intelligence (AI) to write code is creating a security paradox. On one hand, just over three-quarters of respondents (77%) said AI tools improve code security. At the same time, however, 59% are concerned that AI tools trained using general-purpose large language models (LLMs) will introduce security vulnerabilities that first appeared in the code used to train them.
The challenge with general-purpose LLMs is that they are trained using a combination of excellent, mediocre and flawed code. As a result, any recommendation surfaced has the potential to be a suboptimal security result.
Randall Degges, head of developer relations for Snyk, said that as AI tools increase the velocity at which applications can be constructed, it’s probable that DevOps teams will discover more vulnerabilities in their code.
In general, 61% of respondents also noted that automation has increased the number of false positives, with 62% of respondents reporting that at least one out of every four vulnerability alerts they received from automation tools were false positives. Well over a third (35%) said false positives represented over half of the vulnerability alerts.
On the plus side, however, more organizations are paying closer attention to application security issues in the wake of several high-profile breaches of software supply chains and disclosures of vulnerabilities. A full 96% reported their organizations are addressing supply chain security problems on an ad hoc basis, yet only half have a formalized supply chain security strategy in place.
The survey also finds 87% of respondents work for organizations impacted by a software supply chain security issue in the last year, with 61% having implemented new tools and processes as a result.
A total of 62% of respondents work for organizations that have a software life cycle assurance process in place, but only 42% of organizations are using software bills of materials (SBOMs). In addition, only 40% have formal security ratings for the open source packages that developers employ, and 31% ignore risks that arise from indirect dependencies.
However, only 40% of organizations have embedded security testing tools into their integrated development environments (IDEs), with 40% not using software composition analysis (SCA) or static application security testing (SAST) tools at all. In fact, even though 80% of respondents reported working for organizations that ship code daily, only 27% claim to continuously audit their code.
As a result, many vulnerabilities remain unpatched, with Java (42%) and JavaScript (31%) topping the list in terms of code with the most ignored vulnerabilities.
Finally, the survey also noted that vulnerabilities involving open source code are now being remediated faster than proprietary code. Maintainers of open source software are now paying more attention to cybersecurity issues thanks in part to the tools and best practices provided by the Open Source Security Foundation (OpenSSF), said Degges.
There will, of course, come a day when AI is more widely used to analyze vulnerabilities in code. In effect, AI will be used to solve some of the security issues general-purpose LLMs create. In the meantime, however, DevSecOps teams would be well-advised to proceed with caution.
Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.
This week: A big-tech consortium goes toe-to-toe with Google Maps, and a big-tech consortium tries to fool AI regulators. (more…)
Many organizations are experiencing deficits in technical skills and gaps in productivity that ultimately harm their bottom line. To overcome those obstacles and drive growth, businesses can use generative AI tools like ChatGPT and Bard to handle large volumes of data and increase the speed and output of basic tasks without draining valuable human capital. Already, businesses are finding that saving time is one of the most valuable benefits generative AI can deliver. However, there must be oversight of deployments to avoid recurring pitfalls, as the technology contains just as much risk as it does opportunity.
Generative AI provides developers with helpful task automation while simultaneously creating new challenges. Security, bias and IP questions have DevOps teams scratching their heads at how to best ethically and operationally implement generative AI. Developers must embrace these challenges to remain competitive in an increasingly automated and technologically-driven world.
Generative AI’s ability to understand and solve code-related issues makes it an ideal tool to help software developers optimize code for faster feature development and quicker releases. By identifying where it can contribute to tasks, like legacy code conversion or automation of processes, companies can free up valuable developer time for building the tools that can grow the business or drive new initiatives. An added benefit is that development teams can automate the coding of new applications or systems by updating old code or re-using existing examples.
ChatGPT, for example, can also turbocharge development time by scouring swathes of information on GitHub and other forums to find examples of whatever a coder is trying to build. A 2022 study published in the Proceedings of the Association for Computing Machinery on Programming Languages (PACMPL) found that developers could code faster and work more efficiently thanks to generative AI. Tedious searches in tech forums and manual copy-and-paste procedures are replaced by simply asking the right questions (via prompts).
However, generative AI tools are not closed systems. As users enter data into ChatGPT, it is saved and aggregated to continuously train the language model, opening the door for potential leaks of sensitive company information. There is a serious, potential risk of employees using AI tools as sensitive data already makes up 11% of what employees paste into ChatGPT.
As a third-party system, AI tools absorb every query, comment and question without any privacy guardrails for users. If an AI bot’s security is compromised, businesses can unintentionally hand over data to bad actors. Even without a hacker’s involvement, sharing confidential customer or partner information may violate contractual agreements and create long legal battles. Developer teams must remain conscious of AI’s security risks.
It is of utmost importance that businesses make reliable, fair and traceable decisions based on the latest facts. Currently, ChatGPT-4 hasn’t reached the maturity level necessary to be trusted by most companies without significant verification measures. Already, testers have persuaded ChatGPT to make statements that are either inaccurate, biased or both.
One key concern in generative AI is the ingrained and accidental bias in answers due to the human-generated content used to train large language models. ChatGPT works by searching for textual content relating to the question and recombining it for a distillation. In an almost democratic way, the majority opinion will dominate. For instance, if most people believe that the Swedish are hard drinkers, ChatGPT will say so. While the technology provides remarkably accurate responses in an alarmingly convincing way, it’s important to remember that it’s not always the most reliable or balanced source that wins.
Furthermore, OpenAI, the creator of ChatGPT, has explicitly stated that its goal is to create a system where queries can be pre-processed before they are fed to the neural net. Answers can then be post-processed before being shown to users. But this safeguard lacks clarity around its objectives and opens the door for ideological or political manipulation.
Companies considering implementing AI tools must carefully examine their potential applications to ensure they avoid scenarios where bias can result in insights and output that could end up harming the company and its reputation.
Enterprises are looking for tools to integrate generative AI into existing business processes to enhance numerous functions and deliver greater value to their customers. While the technology still isn’t capable of replacing human developers responsible for updating and maintaining largescale software, updates to generative AI tools and integrations are occurring at a rapid pace, leaving businesses to determine which programs are most viable for developer function.
As the economy continues to tumble closer to a recession, companies seek ways to minimize costs. And those looking to use generative AI to work with the source code and technical documentation the company uses to run its own operations—rather than the open source code tools like ChatGPT provide—face budgetary decisions. Implementing generative AI programs into the business model can be extremely costly if done wrong, leaving executives to weigh the benefits of adopting the tech versus the cost of trying to hire a full team of developers.
For businesses looking to build the connected experiences their employees, partners and customers desire, generative AI is a great resource to accomplish this. However, it’s just one piece of the puzzle that is a modern enterprise. These tools can only deliver smart answers if they get smart questions and businesses connect their internal processes and applications to enable machine learning on accurate and holistic datasets.
Although AI poses many benefits for developers, such as increased efficiency, accuracy and the ability to handle large amounts of data, they must be wary of the potential security, ethical and budgetary concerns that impact employees, customers and the enterprise as a whole.
However, once the concerns are overcome or managed, the power of large language models will be leveraged in all kinds of technology and business use cases. Certainly a truly transformational shift and a step change towards a world where technology and software play an unprecedented role in business and society.