Tag: ml

  • How Embedded Analytics Can Transform Future Businesses

    How Embedded Analytics Can Transform Future Businesses

    It’s no secret that big tech companies like Google, Amazon and Netflix are not only leveraging data but also delivering embedded analytics and insights as a standard functionality of their software, services, applications and toolkits. The analytics appear to consumers as recommendations, usage habits and other helpful information. Such insights not only help consumers extract the most value out of the product they are using but also help providers understand how they can provide better services. Businesses that embed analytics to provide a differentiated user experience also have a direct impact on engagement and revenue.

    Have you considered how embedded analytics could differentiate your company’s own software and hardware products? Let’s understand how embedded analytics can help transform technology businesses:

    1. Analytics are expected and welcomed in your products

    By embedding analytics into existing enterprise software and applications, users can drill down into real-time and context-aware data (for example: How busy a location will be at various times of the day) and can benefit from faster data-driven decision-making without worrying about application compatibility. Commonly, software applications provide in-app analytics to delight and serve their customers, while hardware companies can provide helpful stats on a web portal or app.

    2. Embedded analytics is easier than ever today

    Cloud technology has completely disrupted tech-enabled industries. It’s easier than ever for software, apps and hardware to connect to a data repository to save logs that are a rich source of analytics. The invention of the S3 bucket and object store has made it fairly cheap to keep log data. Finally, the transition to platform-as-a-service (PaaS) and software-as-a-service (SaaS) has fundamentally added an “easy button” to tracking consumption patterns and resource utilization.

    3. Embedded analytics helps design better products and services

    Embedded analytics can give businesses more insight into application telemetry (how customers use the data, how they engage with the product) and provide insights into the challenges they face from a usability standpoint. Providers can use advanced analytics to quickly iterate, draw patterns and make predictions, troubleshoot customer challenges, discover new avenues to monetize, detect and mitigate security threats or create entirely new features and applications that put customers on a faster path to improved application experience and better decision-making.

    4. Embedded analytics increases operational efficiencies

    The tech industry is transitioning from traditional monolithic architectures where applications are self-contained to microservices architectures where different components are spun up as needed via API calls. By embedding analytics directly into developer tools, businesses can manage resources without sacrificing availability or performance. Companies can allocate and optimize resources dynamically and understand peaks and valleys in consumption. Businesses must be able to leverage historical and real-time data to build predictive models to proactively identify and mitigate issues with individual components of the application before the customer experiences a problem.

    Embedding Analytics in Your Application

    Before you start embedding analytics in your application, it’s important to choose the right database foundation, as this will support current needs and future goals. Below are the top capabilities that should be standard features in analytics databases:

    1. Handles massive data volumes: The database must scale easily and efficiently with minimal cost using massive parallel processing. Today’s scale might be terabytes, tomorrow it might be petabytes and so on.

    2. Enables speedy data analytics: Speed is a key factor in a positive user experience. Users want instant insights because they don’t like waiting for results. The database should be capable of optimizing load and query performance so that analytics is both timely and relevant.

    3. Embeds machine learning: Machine learning algorithms are critical for predictive analytics. A platform that can embed ML and iteratively learn from new data can make a big difference and add a whole new level of intelligence.

    4. Features native integrations: The more time developers spend figuring out tools and integrations, the less time is spent innovating. The platform must ideally have out-of-the-box integrations with popular applications, data sources (on-premises, cloud, multi-cloud or hybrid), ETL, and visualization tools (e.g., Microstrategy, Looker, Power BI, Qlik, Tableau). It should be flexible enough to support user-defined functions for things that don’t have native integration.

    5. Encourages self-service: The database should make it easy for data scientists to leverage tools and programming languages they already know such as R, Java, C/C++, Python and SQL. Data scientists will also appreciate a platform that has built-in functions for a wide range of analytics (such as geospatial, time-series, event-series, real-time, machine learning, text analytics, pattern matching and regression) and delivers the tools to spot trends, uncover anomalies and anticipate the unexpected.

    6. Supports highly advanced analytics: Many platforms can analyze single tables or do a simple look-up but few have the power to analyze dozens of tables, hundreds of data types, dimensions and attributes from numerous sources over several years of history.

    7. Unified: The database should be able to streamline everything right from data ingestion to exploration to production until delivery. Users should be able to analyze data in its place to always have a single source of truth.

    It’s not hard to visualize that the most disruptive companies of the future will be the ones that harness data as a core asset and embed analytics in their products and services. Embedding analytics requires the right database foundation; to get to the right foundation, IT teams need to cut through the noise and study the fine print. With the proper database, the future is yours to create.

  • Finding Value in AI-Augmented DevOps

    Finding Value in AI-Augmented DevOps

    While artificial intelligence (AI) and machine learning (ML) are emerging technologies, we know they can help an organization parse large data sets and glean actionable insights. But do AI-infused processes actually make a difference in organizations that employ DevOps? The answer is explored in a recent survey by Tricentis. Working in partnership with Techstrong Group, Tricentis surveyed more than 2,600 DevOps practitioners across the globe about their AI-augmented DevOps processes. The results pointed to a bright future in DevOps.

    In this survey, the majority of respondents cited they’re already getting value from AI-powered DevOps processes. A third of survey respondents found AI-augmented DevOps extremely useful today and nearly half found it very useful. Although the technology is still relatively young, a 2021 report by IT infrastructure automation company Puppet Labs found that 83% of organizations are implementing DevOps processes. This rapid adoption speaks to the promise of the technology.

    Interestingly, nearly two-thirds of respondents cited that testing is the area within DevOps where AI will have the most impact. Testing is a major pain point for DevOps organizations because a scaled DevOps environment calls for complex testing scenarios and large amounts of data, and many organizations struggle to scale their automation to the level DevOps requires. Without automation, manual testing can significantly delay releases. In AI-infused DevOps processes, AI can help accelerate the authoring of tests, help testers understand where the risks are, and repair broken tests.

    AI’s Greatest Potential is in Functional Testing

    We typically think of software testing as either functional, where the program enables users to properly execute tasks, or non-functional, where the application is secure, fast and stable. Respondents were asked which disciplines of testing stood to gain most from AI and 65% of them indicated functional testing. Unit testing and UI testing were the sub-categories most often selected.

    The complexity of the functional testing process is where AI shines. Functional testing is open-ended and many permutations are required to ensure proper test coverage. AI steps in and manages those permutations, parsing large amounts of data and providing insights. Additionally, AI can help wrangle production signals that aren’t discovered in a test environment.

    Throughout the survey, respondents specifically noted AI’s promise in regard to UI testing. UI testing is perhaps the most time-intensive and manual type of testing in most organizations. With the many variations of the user experience paths that exist, AI can help mimic the actions of a real end-user, adjusting for application nuances or variations in experience. AI and ML can also boost your build process, increasing the efficiency of repetitive tasks. The ability to quickly identify and resolve issues can improve the effectiveness of your automation.

    While survey respondents see the most potential in testing, they acknowledge that AI-powered DevOps can address other business challenges. More than 40% claim that these processes can reduce the skills gap and empower junior staff to perform complex tasks, improve the customer experience, cut costs, boost innovation and increase efficiency among developers. Nearly 40% also believe that AI-powered DevOps delivers leadership the insights they need for continuous improvement.

    Automation is the Key to Scaling DevOps

    It’s worth noting that among the DevOps organizations that see benefits from AI and ML, most of them are mature organizations; they employ DevOps workflow pipelines, toolchains, automation and cloud technologies, and they’ve automated more than half of their testing. Only 21% of practitioners who are just dipping their toes into DevOps have achieved this level of automation. And even in mature organizations, testing is still the main bottleneck; only 40% of them have automated more than half of their testing.

    The key to scaling DevOps practices is developing test automation that can keep up with an increasing volume and speed of releases. But the road to DevOps isn’t always easy. Practitioners cited obstacles like a lack of tech skills (44%), insufficient budget (25%) and tool selection (19%). Organizations that want to implement DevOps need to rely on external vendors and internal resources to address these challenges and effectively scale.

    Looking to Adapt More DevOps Practices?

    So where does an organization that wants to implement DevOps start? With the big questions. It’s crucial to identify the specific business problem that you’d like to solve. What’s the gain you’re looking for? Where in the software life cycle do you expect to find a benefit? AI and ML can add business value, but they don’t solve every problem across the life cycle.

    After an organization has identified its DevOps goals, its focus should shift to automation. To reap the benefits of AI-powered testing, an organization should automate repetitive functional testing, especially unit and regression testing. Most organizations start by implementing continuous integration and continuous delivery. Once an organization has taken these steps, it will start to amass a large cache of results and artifacts generated from running the test cases, which AI can mine to increase test stability and identify recurring problems. The more you automate, the more you’ll get out of it and the faster you’ll accelerate your delivery.

    Organizations just starting out with DevOps should also understand the distinction between artificial intelligence and ML. Most people refer to ML as AI, but they’re not interchangeable, and most organizations haven’t advanced to the use of AI. A true AI is broadly powerful, while ML technologies are narrow and solve a specific domain problem that a model was trained for.

    When you’re evaluating potential vendors, there are a number of key questions to ask, including:

    • What specific AI technology is the vendor using? Is it an off-the-shelf library or something they built themselves?

    • How are they using AI? Which algorithms are in play and what approach do they take?

    • How do they train their models? Different types of patterns emerge in data sets depending on the industry, the frequency with which data is updated and the application. The data population that the vendor is training on must look like your production data, or the model could take you in the wrong direction.

    • How do they ensure that bias doesn’t come into play? If they train models in a different domain or vertical, for example, you could be an outlier for that vendor.

    • How do they handle failure? How do they monitor for miscategorization and misapplication of the AI in a way that lets you trust the data?

    • Will your data be used to train and improve your learning models? If so, does that raise any privacy concerns? Expectations around data privacy should be explicit and leave no questions.

    • Are there other customers in your vertical with a similar business profile that are getting the benefits you’re looking for? Can the vendor provide examples?

    What Does AI-Infused DevOps Look Like in Practice?

    Consider a financial services organization that has more than 20 applications to manage. With so many applications, it’s necessary to conduct significant UI testing and the tests must be stable. One of the biggest challenges to UI testing lies in locating various web elements; most people start with manual script-based automation, but it can be challenging and time-consuming to create and maintain. Employing AI and smart locators to identify these elements can save time and drastically cut down on UI testing maintenance.

    Survey respondents were asked how they’re currently using AI to augment their testing processes. Thirty-seven percent say they’re accelerating the creation of their automated test cases, while 44% focus their testing on the areas of highest risk. Almost half are able to reduce test case maintenance with self-healing and 43% use AI to identify the root cause of failed tests. Thirty-one percent claim that AI offers insights into test process improvements and 34% use AI to help identify tests to run based on changes within the application.

    Where is AI-Powered DevOps Going Next?

    Going forward, AI will become more and more embedded in testing processes. Great progress with off-the-shelf vendors is being made, and their products are a lot more accessible than they used to be. As the tooling continues to mature, these products will reduce the need for specialized skills. Internal initiatives like data lakes or data warehouses will persist, but there will be less need to build large data science teams to get the advantages of AI.

  • Iterative Adds Registry to GitOps Portfolio for ML Models

    Iterative Adds Registry to GitOps Portfolio for ML Models

    Iterative added a registry for machine learning models to its portfolio of Git-based tools for infusing artificial intelligence into applications.

    Iterative CEO Dmitry Petrov said the Iterative Studio Model Registry is intended to make it easier for application development teams to track which models are being used as the number of applications infused with AI capabilities continues to grow.

    The Iterative Studio Model Registry, accessible via a graphical interface, command line interface or application programming interface (API), makes it simpler to explore models, including history, versions and stages, via a central dashboard. Teams can also identify the experiment that produced the model and track how, when and by whom a model version was created.

    The Data Versioning Control (DVC) platform created by Iterative exposed a Git-like interface to enable organizations to track multiple versions of data sets, models and pipelines across a machine learning operations (MLOps) workflow. That approach enables those models to be stored in a Git repository alongside other software artifacts versus requiring organizations to deploy a separate platform just to store ML models.

    Providers of platforms used by data science teams have been making a case for separate repositories for tracking the artifacts used to construct an AI model. Iterative contends it is more efficient to employ Git repositories where other types of software artifacts are already being stored because, after all, an ML model is just another type of software artifact.

    That approach also makes it simpler for DevOps and data science teams to collaborate because the DevOps team will have more visibility into what AI models will eventually need to be incorporated into an application, noted Petrov.

    The challenge organizations face is the cultural divide between data science and application development teams. Data science teams today typically have defined their own workflow processes using a wide range of graphical tools. However, as it becomes obvious that almost every application is going to be infused with machine learning and deep learning algorithms to some degree, the need to bridge the current divide between DevOps and data science teams will become more acute.

    In fact, DevOps teams should assume that many more AI models are not only on the way but that those models will need to be continuously updated. Each AI model is constructed based on a set of assumptions; however, as more data becomes available, AI models are subject to drift that results in less accuracy over time. Organizations may even determine that an entire AI model needs to be replaced because the business conditions on which assumptions were made are no longer valid. One way or another, the updating and tuning of AI models are likely to soon become just another continuous process being managed via a DevOps workflow.

    No IT organization can, of course, manage what it can’t track. Registries have become an essential component of a DevOps workflow as the volume of types of artifacts that make up application environments has increased. It remains to be seen to what degree ML models become just another software artifact but one way or another someone in the IT organization needs to know what model is being employed in which application for what purpose.

  • VW CEO Fired for Dev Fails | Fiber Shortage Hits | Google Fires Blake Lemoine

    VW CEO Fired for Dev Fails | Fiber Shortage Hits | Google Fires Blake Lemoine

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Herbert Diess is out at VW because software is hard (yo), fiber optic cable is hard to find, and the guy who said LaMDA was sentient is dumped by Google. (more…)

  • Google FLoC is Dead | Meta AI Supercomputer Lives | ARM Deal is Dead

    Google FLoC is Dead | Meta AI Supercomputer Lives | ARM Deal is Dead

    Welcome to The Long View—where we peruse the news of the week and strip it to the essentials. Let’s work out what really matters.

    This week: Google’s FLoC proposal is dead, Meta/Facebook is buying RSC—a huge AI supercomputer, and Arm “will IPO” instead of selling to Nvidia.

    1. Forget ‘Federated Learning of Cohorts’—Long Live ‘Topics’

    First up this week: Google bows to the inevitable. Third-party cookies will soon go away, because people are fed up with being tracked. But Google’s FLoC proposal wasn’t the answer.

    Analysis: FLoC is a flop

    Google doesn’t want to invade your privacy. But Google does want to sell well-targeted ads. The key difference between FLoC and Topics is that FLoC buckets were automatically generated—and hence opaque, risking overlap with “protected categories”—e.g., race or gender. But the topic classifications in the Topics API are curated and have user-visible, descriptive names.

    Frederic Lardinois: Google kills off FLoC, replaces it with Topics

    Google’s controversial project for replacing cookies for interest-based advertising … is dead. In its place, Google today announced a new proposal. … Your browser will learn about your interests as you move around the web.
    …
    When you hit upon a site that supports the Topics API for ad purposes, the browser will share three topics you are interested in. … The site can then share this with its advertising partners to decide which ads to show you. … Users will be able to review and remove topics from their lists [or] turn off the entire Topics API.
    …
    It’ll remain to be seen if other browser vendors will be interested in adding the Topics API. [But] they all quickly turned a cold shoulder to FLoC. … The plan is to start trialing the Topics API at the end of this quarter.


    A minor tweak and rename? “Yes and no,” equivocates mananaysiempre:

    Yes, it still amounts to building user-tracking functionality into user agents. No, it’s not the same approach.
    …
    FLoC was doing unsupervised clustering of users, whereas Topics allocates users into predetermined clusters. This makes it potentially more transparent and also seems designed to work around the “protected categories” objection.


    But Google has a long way to go to convince garett_spencley:

    I’m glad FLoC is being ditched, but as an end-user I’m still not asking for this Topics nonsense. It doesn’t solve any problem that I have. It solves problems that the advertising industry has.
    …
    I’ll choose to use browsers that don’t support this.


    2. Machine Learning at Metaverse Scale is Expensive

    Zuckerberg’s new toy is RSC: A $400 million Nvidia-based AI supercomputer. And that’s just Phase One.

    Analysis: 3.8×1018 integer operations per second is … a lot

    If you rely on Open Compute Project systems for your DevOps, know that it’s no longer the bee’s knees over at Hacker Way. Meta/Facebook is investing eye-watering sums into a custom configuration of off-the-shelf hardware.

    Timothy Prickett Morgan: Meta Buys, Rather Than Builds And Opens, Its Massive AI Supercomputer

    If you thought it took a lot of compute and storage to build Facebook’s social network, you ain’t seen nothing yet. … To build the Metaverse … is going to take an absolutely enormous amount of supercomputing power.
    …
    But what is a surprise, given Facebook’s more than decade long focus on … opening up its hardware designs through the Open Compute Project, is that the … Research Super Computer [RSC] system is built from … commercially available … servers, storage, and switches. [It] probably comes down to … working a deal with some vendors who want the Facebook publicity and are willing to work the price and the parts availability.
    …
    [So Meta bought] 760 DGX A100 systems from Nvidia—which each … have a pair of AMD “Rome” 64-core Epyc 7742 processors … 1 TB of main memory … eight A100 GPU accelerators [and] eight 200 Gb/sec HDR InfiniBand controllers—all-flash storage from Pure Storage, and … caching servers from Penguin Computing. … At list price this machine would cost around $400 million … Phase one of the RSC machine is rated at 59 petaFLOPS … (FP64), and 118.6 petaFLOPS FP64 using the Tensor Core matrix engines. … With INT8 processing on the Tensor Cores, which is needed for AI inference, the phase one machine is rated at 3.79 exaOPS.


    How much??? It doesn’t seem like news to kleiba:

    I used to work at a university where my professor had been in automatic speech recognition for a long time, but basically gave up on that line of research about 10 years ago because he figured that universities simply cannot compete budget wise with the big industry players. I suppose the same will soon be true for most ML-related areas of research sooner or later.
    …
    Already, a substantial amount of research innovation in [Natural Language Processing] and [Computer Vision] has been coming from big companies in recent years. Of course there is a discussion to be had about what that means for society at large.


    But 3.79 exoOPS? That’s the way the wind is blowing, according to sungazer:

    Give it 15 or so years—this level of computation will be on your desktop. If desktops still exist.
    …
    Or desks.


    3. Arm Bid: No Legs

    Speaking of Nvidia, as I’ve said before, the Arm merger really shouldn’t go through. And it’s looking less and less likely. ARM chips are an increasing fixture in the data center—especially those that value “performance per Watt.”

    Analysis: ARM M&A DOA—IPO PDQ

    A veritable hornets’ nest of Bloomberg journalists have put their names to an anonymously-sourced report. The story argues that the proposed Nvidia buyout of Arm from SoftBank Group is dead. Next stop: IPO.

    Ian King, Giles Turner, Peter Elstrom, Dina Bass, David McLaughlin, Ruth David and Dinesh Nair: Nvidia Quietly Prepares to Abandon … Arm Bid

    According to people familiar with the matter … Nvidia … doesn’t expect the transaction to close. … SoftBank, meanwhile, is stepping up preparations for an Arm initial public offering.
    …
    If Nvidia manages to get the deal over the line, it would be a massive coup. … But it will be an uphill fight. … The world’s biggest tech companies rely on Arm technology, and they fear they could lose unfettered access under Nvidia.
    …
    Within SoftBank, there are factions. [And] the ordeal has created divisions within Nvidia. … “We remain hopeful that the transaction will be approved,” a SoftBank spokesperson said in an emailed statement.


    Post-Brexit Britain would no doubt like to keep Arm independent. Here’s simonh:

    Arm has I think ⅔ of its employees in the UK—mainly Cambridge—and they’re expanding the campus there. The Semiconductor Physics Group at the Cavendish Laboratory at the University is at the cutting edge globally with 18 PhD projects currently.
    …
    Guess where scads of Arm’s top scientists are recruited from? Arm and the University are joined at the hip. That company’s not going anywhere.


    More broadly, it’s good news, thinks hatchet:

    Hopefully this means NVIDIA will focus on RISC-V instead. This will be good for everyone.


    The Moral of the Story: Zeal should not outrun discretion.


    You have been reading The Long View by Richi Jennings. You can contact him at @RiCHi or tlv@richi.uk.

    Image: Reed Geiger (via Unsplash)

  • Driving Software Delivery Automation With AI/ML

    Driving Software Delivery Automation With AI/ML

    What if we could use an artificial intelligence (AI) engine to predict the next lines of software code to build? What if machine learning (ML) could trigger the creation of automated test cases to validate our solutions?

    Automation has been a focus for the information and communications technology (ICT) industry for years, and there are some lessons that could apply to the software developer community, too, as well as other verticals. Today, more streamlined execution of operational and software delivery processes, such as continuous integration, continuous delivery and continuous deployment (CI/CD) in app development, continue to drive new levels of cost reductions, allowing products to get to market and update faster. 

    However, with the increased amount of data comes more complex ecosystems and the need for the next level of automation. Hyperautomation, or the use of AI/ML to further streamline processes, could build on these CI/CD processes, revolutionize the way multiple industries deliver software and optimize operational processes.

    Today’s Network Complexities and Challenges

    In the telecommunications industry, disruptions from hyperscalers and other innovative players, as well as increasing levels of investment in 5G technology, are forcing the hand of communications service providers (CSPs). This is causing them to turn to automation to improve their levels of operational efficiency and innovation to keep up with the need to push frequent updates. 

    However, challenges remain in the form of:

    • Total cost of ownership and operating costs: With more frequent needs for upgrading existing networks, additional levels of automation including AI/ML and hyperautomation are required to manage and maintain today’s complex networks. 
    • Time to market: In an increasingly competitive market, the CSP’s ability to react to market needs and/or new marketing ideas and to rapidly launch those needs and ideas to their end users and enterprise customers is a key competitive differentiator. 
    • Customer experience: Digitalizing business processes, providing an increased level of self-service and seeking ways to optimize customers’ overall journey remains an important need.
    • Employee productivity: In an effort to increase revenue, more efficiently drive their networks and improve end user customer experience, CSPs also need to increase employee productivity. They can do this by engaging their employees to solve bigger problems. As a result, the simplest and most repetitive low-level tasks can be automated.

    The Power of AI and ML

    The automation journey started with breaking down complex processes into smaller parts and automating discrete parts using scripts, runbook automation (RBA) and/or robotic process automation (RPA). 

    With the introduction of AI emulating human intelligence through mechanical and computational processes and ML using algorithms that learn from data to make predictions and decisions, CSPs can now bring automation and hyperautomation to the next level. 

    The potential use cases for the delivery and maintenance of telco software solutions are unlimited, and many of these lessons can be applied to other sectors. Some examples include:

    Autonomous Fault and Anomaly Detection for Cloud-Native Solutions

    Today’s network solutions are deployed on containers and microservices. The identification of issues and assurance of service quality becomes increasingly challenging with thousands of workloads running in parallel. Container failure can cause severe service disruptions. In fact, a single service failure or abnormal behavior could have a butterfly effect on the overall system and negatively affect customer satisfaction. 

    A proactive, AI-based monitoring solution can continuously search for anomalous behavior in the environment. Different container metrics and logs can automatically analyze and detect problems proactively and help find the right corrective actions for the application workflows. This is based on identifying anomalies in the container orchestration metrics related to pods, workloads and namespace. These techniques enable a more reliable and predictable application runtime environment with higher service quality and faster identification and resolution of system faults and, ultimately, a better overall customer experience

    Machine Learning-Triggered Closed-Loop Automation

    Large clusters of compute, storage and networking resources are managed throughout a data center to enable the deployment of custom workloads. Due to the inherent nature of applications running on the virtualized and cloud infrastructure, there are usage highs and lows which need constant infrastructure component deployment or reconfiguration of the cluster to ensure cost-efficient service availability. 

    A machine learning-based solution can detect anomalies in the infrastructure. Specifically, with closed-loop automation, enterprises can bring in a layer of intelligence over the current setup to monitor application performance and infrastructure metrics and identify anomalies that require immediate system attention. If those are found, they can trigger the automated addition of capacity to the infrastructure. When anomalies have subsided and once the utilization is capable of being handled by the original capacity, the solution can consistently monitor metrics to roll back the virtual or cloud-native functions. This allows for an efficient and fully automated use of network capacity, minimizing human oversight.

    Smart Code Recommendations

    One of the most clear-cut ways tools can reduce the effort of software development for programmers is through code completion. AI-powered code completion is a convenient way to access descriptions of functions and their parameter lists. These solutions also speed up software development by reducing keyboard input and the necessity of name memorization. They also allow users to refer less frequently to external documentation, as interactive documentation appears dynamically in the form of tooltips. AI-powered code completion can also use an automatically generated in-memory database of classes, variable names and other constructs or references based on machine learning to open code repositories.

    AI-Driven Test Recommendation

    Significant time is spent understanding solution requirements and identifying critical scenarios for testing. This is usually followed by a manual test plan creation and execution. To solve this, AI/ML techniques can be applied to identify what tests will need to be done during each cycle of the software development stage, including automating the analysis of required documentation. This can be used to increase overall test-cycle time and improve productivity. 

    AI-Enabled Test Automation

    API testing is critical for validating application logic and needs to go beyond the GUI layer to validate the core of an application. This implies complexity because the protocols are usually difficult to understand, the API inventory is usually large and sequencing of the API calls is critical to a successful test specification and execution. With AI/ML and natural language processing techniques, the analysis of the solution test plan can be automated to obtain recommendations on what test frameworks to utilize, identify and suggest the parameters for each test case, identify the API call sequence and automate the development of test cases based on reusable functions. This can increase overall quality of service, simplify change management, improve the productivity of the testing team and offer more reliable test cases.

    Test Failure Analytics

    During execution of automated test cases, multiple failures can be observed in various logs. Manually analyzing the logs and correlating them with the application issues can be time-consuming. Also, the identification of unreliable tests impacting system stability is complex. 

    AI/ML techniques such as vectorization, K-means clustering or topic modeling can be applied to automate the collection of test logs across applications and label failures including the type of cluster failed tests. The test subject matter expert (SME) is provided with a failure analysis report summarizing issues and problems automatically from many logs across applications, simplifying the identification of issues in test automation areas or scripts. These techniques can significantly increase test productivity and overall troubleshooting time.

    Accelerated Root Cause Identification and Analysis (RCI&A)

    When a problem occurs in a telco network, it tends to cause a ripple effect, affecting other integrated services and creating an alert storm. In such scenarios, root cause identification can become complex and cumbersome. Event aggregation and synchronization are needed to reduce environment complexity and variance. 

    Machine learning weighted analysis techniques can be used to determine the influence individual IT metrics have in affecting the overall health and performance of hosted applications and services so that pinpointing the root cause of any service impact in real-time can become easier for the team operating the network. Efficient AI/ML techniques for accelerated RCI&A can lead to significant timeline reductions. 

    AI/ML Driving Operational Innovation and Market Agility

    In telco, there is great potential for AI/ML-driven use cases to further improve the cloud-native journey and enable an IT-like software life cycle management process for the solutions of the future.

    More broadly, highly automated and self-healing networks will drive higher levels of efficiency and customer satisfaction. This includes faster testing with less resources; more efficient utilization of network resources and cloud infrastructure; faster time-to-market with more software releases per year with similar capacity as well as an increased quality for faster and more agile monetization of the CSPs capabilities.

  • DevSecOps Implementation: Intrusion Detection

    DevSecOps Implementation: Intrusion Detection

    Originally, this series was just going to be four articles on the DevSec side of DevSecOps. There are many reasons for this, but primarily because that side is cleaner. The other reason is that these topics are beyond the work we were doing at Accelerated Strategies Group.

    But we’ve had a number of requests to continue, and I’ve used many tools in the space, so on we go.

    Intrusion Detection

    Intrusion detection (IDS) started because attackers were getting through firewalls, and, once inside, seemed to have free rein. Intrusion detection wanted to offer another option for detecting and stopping attackers by watching things inside the firewall for indications your organization had been compromised. Relatively quickly, intrusion detection was supplemented by intrusion prevention (IPS). The idea of prevention was to proactively make changes based upon known issues to keep intruders out. Intrusion detection offered information about what paths attackers were exploiting, and intrusion prevention would plug them. Note that IDS and IPS are often further subdivided into host-based (HIDS) or network-based (NIDS) systems. Since their role – detection of intruders – is the same, for this overview we will discuss them together. A serious deep dive into IDS/IPS should look at HIDS and NIDS separately.

    In the DevSecOps space, some have folded these into security monitoring, but they are still performing the same basic tasks – just much better. In fact, if you found intrusion detection to be limited and clunky to work with because of rules maintenance and fingerprinting issues, it’s probably time to take another look. Here’s why:

    Problems With Traditional Intrusion Detection

    The problems with pure rules-based intrusion detection were two-fold.

    First, rules were hard and fast.

    “Look for X and Y and Z. If all are present, raise an incident.” This tended to result in only two types of rules – ones that were too narrow, and thus brittle, or ones that were too broad and that buried IT/security staff in alerts.

    Second, those rules were outdated before they hit the network.

    The old adage “You are always fighting the last war,” (meaning you are doing what worked the last time you won) is more true in IT security than almost anywhere else. For a rule or a fingerprint to be assembled, it has to have been seen. That means we were perpetually protecting against previously successful attacks and doing little against emerging threats. Security vendors took a lot of steps before coming to our current solution, which is a great step in the right direction.

    Attacks against IT infrastructure share a lot of commonalities. Purely rule-based systems counted on tribal knowledge to code those commonalities into rules. It was clunky and imperfect. Templates helped, by adding common items based upon the type of rule you were creating, but, like rules, they tended to be static and old news.

    Enter Machine Learning

    Bright people with an eye on the ever-changing environment knew that we needed something that could adapt to multiple attack forms/vectors and draw the type of conclusions that security analysts were tasked with. The ability to reduce the number of alerts that get raised is pretty close to miraculous. Anyone with a web presence sees thousands of security events a day, as attack bots and scanners automate chunks of the attack matrix. Some kind of solution to weed out the false information is nearly mandatory, as no organization has the manpower to monitor at those levels and reasonably follow up on threats.

    Machine learning (ML) makes connections between alerts – that might not be a big deal in isolation – and other events that, together, make a case for human intervention. Added to fingerprinting and rules, it is a powerful addition. While a successful attack would require going through rules and making certain you covered the behavior/changes found in the new attack, ML can take that information in and adapt automatically.

    This is cranking through volumes of data we couldn’t even imagine a decade ago, and finding anomalous behavior, correlating it and alerting where necessary. How much it reports up the chain is generally configurable, allowing an organization to tighten the flow of alerts that analysts have to deal with.

    Go With the Flow

    More recently, flow evaluation has become a positive step in the monitoring process. If this connection/user moves from A to B to C, it might be an attack. This allows for another way to rate suspicious activity and raise it to an analyst’s awareness.

    An increasing number of vendors are moving the analysis portion of these activities into their hosted environment. This is a “great strength/great weakness” solution – putting the data points gathered from your system into their cloud allows them to correlate events and improve all customers’ responses. The other side of the coin is that you are paying them operational costs each month, and for most vendors, you’re charged based on how much of the service you use. This has budget implications. It also assumes you have connectivity to their service. If you don’t, for any reason, no analysis will be done on collected data. A careful rollout and contingency planning are in order for this scenario to be successful.

    Integration with security information and event management (SIEM) systems is ubiquitous. We’ll talk about SIEM next; I just wanted to include here that IDS/IPS vendors are feeding SIEM.

    IDS isn’t perfect, and there are a ton of other things out there to help, but it’s a step in the process, one you should consider as you gear up your DevSecOps to professional levels. Some of you are using the tools, but don’t have them integrated into your DevOps systems. This is a nudge to do so. If resolution of an attack automatically creates tickets for Dev to make the code (or ops to make the config) more secure, it absolutely is a plus to the organization responding proactively.

    Keep kicking it. These are just tools, and they’re useless without you. Figure out what your organization needs, and update those needs regularly as the environment (both attackers and architecture) continually changes around you.

  • Aspects of Machine Learning on the Edge

    Aspects of Machine Learning on the Edge

    Machine learning (ML) is hard. Making it work within the confined environment of an embedded device can easily become a quagmire unless we consider, and frequently revisit, the design and deployment aspects crucially affected by ML requirements. A bit of upfront planning makes the difference between project success and failure.

    For this article, our focus is on building commercial-grade applications with significant, or even dominant, ML components. Edge devices, especially ML enabled ones, don’t operate in isolation; they form just one element of a complex automated pipeline.

    You have a device, or better yet, an idea for one which will perform complex analytics, usually in something close to real time and deliver results as network traffic, user data displays, machine control or all three. The earlier you are in the design process, the better positioned you’ll be to adjust your hardware and software stack to match the ML requirements. The available tools (especially at the edge) are neither mature, nor general purpose. The more flexible you are, the better your odds of building a viable product.

    Let’s start by describing a hypothetical device and we’ll work through some ML considerations of the design. As we discuss the design, we’ll visit and revisit DevOps automations that go hand in hand with these other engineering processes.

    A Smart Security Camera

    For our design, let’s look at a networked security camera. As an IoT device, it’s continually connected to the internet. We’ll assume our device will have at least 4GB of SDRAM, a 64-bit ARM CPU and run an embedded Linux that supports an Anaconda Python distribution, OpenCV, DLIB and TensorFlow.

    Our ML related goals are first, recording and labeling interesting frames, and second, alerting security personnel to suspicious activity. We are constrained by various physical, environmental and cost factors. To make the best use of the available data, we’ll need to use ML to examine and classify multiple objects from every image frame. We’ll use a cloud service to handle the second goal, so here, ML on the edge concerns image recognition. How should we proceed?

    Process the Images in the Cloud?

    We’ll need to recognize various objects in the frames, such as people, faces, vehicles, etc. Each object set requires execution (inference) of an ML model, which produces a set of bounded, labeled objects. A typical camera will record about 30 frames per second. Couldn’t our camera just send that data to a cloud provider? Ignoring other considerations, this is about 2.5 million images/day; even at a deep discount this would be billed at about $1000 a day, and that’s for each recognition model applied. Clearly, we’ll need to make other choices.

    Let’s start by examining the raw input stream. Each raw 720p (standard HD 1280x720px) frame uses about 5MB, so if we were to send 30 frames/sec over the network we would need an incredible 1.5Gbps connection (about a terabyte every two hours). For full HD and higher, multiply this by four to 10 times. We will not be sending raw, uncompressed video. Our problem is our ML models will only work against individual image frames. Where should we make our trade-offs?

    Video produces vast amounts of data, but very little information in a given frame is new; this is why various compression techniques work extremely well, and raw data reductions of 100-1000x is typical. These reduced rates of 15 down to 1.5 Mbps is fairly reasonable, so perhaps we can do our ML inference using cloud services, as long as we host our own machine instances rather than paying for ML services. Depending on our workloads, we can expect to process 24×7 video for between $50 and $200 per month.

    Having decided to host our ML models in the cloud, we now need to manage a corresponding set of DevOps considerations. Do we have enough compute power to manage the service? How will we scale the workload as the number of edge machines increases? What are the storage and data retention policies? And of course, since we’re dealing with images of people, how will we manage privacy concerns and the associated legal and political ramifications.

    What should bias us toward more cloud processing?

    • Our initial development budget is more limited.
    • Our ML models have not been proven effective in limited processing/memory environments.
    • We need the flexibility to rapidly change the models we use for processing.
    • We know the network connectivity is robust and offline operation is not required.
    • Our budget for cloud services can significantly exceed the cost of our edge hardware.

    The choice between processing in the cloud, or on the edge is not a binary one. Designs can massively benefit from even limited edge computing; in some cases, simple detections (motion/vehicle/human/face) can be implemented on the edge to reduce cloud workloads by orders of magnitude with no loss in utility.

    Can We Do Most of the Work on the Edge?

    If we’re using Python with C library acceleration on our quad core ARM, simple ML models such as face detection (not recognition) may handle 10 to 20 frames/second, but 30/fps on anything like a full sized image will not be possible. Deeper, more complex models won’t fit into our limited RAM, and if they did fit, we could see inference times exceed a second per frame. We’re now faced with the realities of life on the edge: trade-offs.

    Once we’ve committed to placing our models into the edge device, we’ll need continuous integration (CI) and deployment services to support our product. These will let us train and retrain our models in the cloud, automate the process of validating improvements and produce the embedded code that updates the models in our target devices.

    Since we initially used models well suited to GPU based servers, can we just add a GPU to our device? In a word, yes, but with some very real costs. Adopting an alternate SoM with GPU support will provide a magnitude of improvement in inference performance, but will double or treble the unit cost and power consumption will increase. Such changes may incur significant non-recurring engineering as adjustments, rewrites or retraining of the model will be required to take advantage of the new hardware. Non-recurring engineering is a one shot cost associated with a change to a part, assembly or system.

    In some cases we can plug an Intel Movidius Neural Compute Stick (NCS) into a spare USB port and see 5x to 10x boost in per image rates. These are supported by the Intel OpenVINO toolkit, but there will be some development time to get things running and not every model can be made to work.

    The next tool in the box is Field Programmable Gate Arrays (FPGA) development. It’s a chore to incorporate these into our board design, but performance can be equal or better than GPUs (on some models), and power consumption is lower. The Intel OpenVINO Toolkit supports Arria and Stratix FPGAs for DNNs. Pricing can be prohibitive.

    Finally, if our device will be produced in large quantities, it can be worth a few million dollars of NRE to build an Application Specific Integrated Circuit (ASIC). The sunk cost of ASIC design can be huge, but the lower unit costs will make up for it. We’ll go this route only once our design has been proven, locked down and we are sure the market can take the volume needed to break even on these engineering costs. I should note architectural improvements in dataflow processing of DNNs in ASICs, such as Vivienne Sze’s work, show promise for dramatic improvements in weight, power consumption and cost. If you are planning for a longer term, don’t rule out these developments.

    What would make our edge processing easier?

    • Start with low footprint models such as TinyYOLO, MobileNet or SqueezeNet.
    • Accept lower frame rates for image processing in our design.
    • Evaluate the ML models with the hardware specifics and tool sets in mind, up front.
    • Provide automated DevOps services to update models in your devices.

    Building our ML models to run on edge devices will involve significant development costs. Tool chains do not install seamlessly, and it’s common to spend days to weeks tweaking the environments. Development and testing on these platforms tend to be slower and involve more manual steps than we’re used to in our DevOps environments.

    Running models on the edge will increase our DevOps complexity. Our cloud workload will be lower, but now we’ll include cross compilation, embedded code builds, download management of larger images and various additional test and validation stages. These will be on top of the training, load balancing, data retention, privacy and security issues we need to address for a cloud-only system.

    Conclusions

    What else will affect our choices here? The nature of the inputs can really matter. If our camera visualizes traffic on a busy roadway, or people in an airport terminal, we can expect almost every frame will contain something of interest; if we’re monitoring the bottom floor of a parking garage–not so much.

    To recap:

    • Cloud services for video rate ML are cost prohibitive.
    • Integrate DevOps early to complete your product support system.
    • Understand and characterize your input data by volume, scale and importance.
    • Recognize not every model can execute on edge hardware.
    • You have more choices in edge hardware than you thought.

    Although there are still challenges to building more intelligence into devices, it’s important to realize every year these constraints are fewer, the tools get better and the devices become more powerful. A compelling argument can be made that most data processing should occur as close to its source as possible.

    — John Fogarty

  • Are Self-Service Machine Learning Models the Future of AI Integration?

    Are Self-Service Machine Learning Models the Future of AI Integration?

    DevOps teams seeking to step up their mojo in developing cutting-edge artificial intelligence (AI) features are facing a big skills bottleneck when it comes to data analytics and machine learning modeling. As a result, the market is seeing an influx of self-service machine learning models and machine learning-as-a-service offerings designed to help development teams more easily integrate AI capabilities into their software.

    This is coming in direct response to an explosion in demand for AI capabilities in the enterprise. According to Gartner analysts, AI adoption in the enterprise tripled in the past year. A report last fall from MIT Sloan Management Review and Boston Consulting Group found that 91% of enterprises believe that AI will deliver new business growth to them by 2023.

    The trouble is that folding AI functions and predictive analytics into software requires a whole new level of expertise in data science and machine learning from cross-functional DevOps teams. They need added skills to choose the right algorithmic approaches, acquire and manage data, train the models and integrate them into the code base and underlying infrastructure so that everything works properly under the hood.

    “We have heard from customers everywhere that they want to adopt machine learning but struggle to actually get models into production,” said Eric Boyd, vice president of cognitive and AI for Microsoft.

    Ali Farhadi Xnor
    Ali Farhadi, Xnor.ai CEO

    DevOps leaders today already face an uphill battle to keep their developer ranks staffed with well-trained software engineers. Piling on additional requirements for very specialized machine learning and data science expertise further strains those recruitment efforts. According to the most recent Harvey Nash/KPMG CIO Survey, the top one and two technical realms that suffer the biggest skills shortages today are in data science and AI, with 46% and 38% of CIOs respectively reporting recruiting pain in those areas.

    “There’s a huge imbalance between the demand in the market and the supply of the very best AI experience to actually train these models,” said Ali Farhadi, CEO of Xnor.ai, a Seattle firm that just last week released a self-service platform for developers. Called AI2Go, the platform provides pre-trained deep learning models to quickly integrate AI features such as facial recognition and object classification directly into their software.

    “We want to enable anyone who can code to benefit from AI,” Farhadi said.

    The Xnor release comes just a few weeks on the heels of a big launch from Microsoft of several self-service machine learning model tools aimed firmly at MLOps teams—those DevOps teams implementing machine learning capabilities. The new services from Microsoft roll out capabilities to manage the code, data and environments used by MLOps teams throughout the machine learning deployment life cycle.

    “Azure Machine Learning service’s MLOps capabilities provide customers with asset management and orchestration services, enabling effective ML lifecycle management,” explained Jordan Edwards, senior program manager for Azure MLOps, in a blog post on the topic. “With this announcement, Azure is reaffirming its commitment to help customers safely bring their machine learning models to production and solve their business’s key problems faster and more accurately than ever before.”

    Tools such as these are not always going to obviate the need for data scientists and machine learning/AI specialists on DevOps teams. As Farhadi noted, even a simple service like AI2Go requires development teams to choose the right models for the right situations. Nevertheless, this burgeoning market could take the pressure off teams by minimizing the scale of specialized recruiting, while reducing bottlenecks and dreaded re-work.

    “Machine learning projects usually involve several people—data engineers, data scientists, DevOps and others. Conventionally, at each stage of the machine learning life cycle, most, if not all, of these people end up doing an excess of custom coding, redundant work and manual trial and error,” said Craig Stewart, senior vice president of product management and product marketing for SnapLogic, which last fall introduced a self-service machine learning add-on for its integration platform. He noted that one of the primary goals is speed, to help cross-functional teams “build, train, validate and deploy high-performing models faster.”

    — Ericka Chickowski