staging-devopsy.kinsta.cloud

Tag: IoT platform

  • Building a Services and IoT Platform with DevOps, Part 3

    Building a Services and IoT Platform with DevOps, Part 3

    Business needs and IoT demands are taking new dimensions in connecting millions of devices, people, process, tools and systems. With endless possibilities of creating new IoT solutions, DevOps teams are the key players who handle the challenges in successfully rolling out a IoT solution while handling the business, development, operations and infrastructure demands in the background. This article focuses on some of the DevOps best practices to adopt during IoT development, enabling faster time to market.

    DevOps practices to adopt during IoT development

    1. Adopt Security by design – Teams need to be educated and made aware of security threats. Use threat models mitigation techniques such as using authentication; storing secrets separately and protecting those secrets; using tamper-resistant protocols, digital signatures, timestamps, audit trails, encryption, authorization policies, filtering; and running services with least privileges during security design and development.
    2. Shift left security analysis and security testing – Integrate security analysis tools, customize security compliance rules and run security testing as part of continuous integration pipeline, generate security threat reports on vulnerabilities and act on these threat reports to fix issues early in the development phase and minimize the security risks.
    3. Follow security standards during development such as security development life cycle (SDL), which has guidelines and measures for building the software for devices and application components of IoT platform.
    4. Perform code reviews for device software, application software, PaaS rules and configurations, as quality assurance can become a huge risk given the complexity of the infrastructure, millions of devices connecting and dynamic environment configurations. Code review tools need to be integrated to the CI pipeline with pull request code review.
    5. Use the right branching policies in a version control repository, including precommit/preflight builds, pull requests for merging the code changes and access control for maintaining branches and using the right branching strategies, as there will be multiple fragmented development and deployment pipelines for devices and application developments to reduce the overhead. Use distributed repository/central repository with traceability and auditability based on the requirements.
    6. Automate deployments and create deployment pipelines for device development, application development, infra-automation scripts development with build and deployment pipelines targeting for different target endpoints.
    7. Customize the builds and release pipelines to pull the packages from multiple build and deployment pipelines to integrate and deploy as units of deployment.
    8. Plan for frequent release cadences—say, monthly—for application rollouts. Release cadences for firmware update rollouts can be planned for three to six months, as device updates cannot be pushed so often.
    9. Continuously monitor deployments – Use a release management tool for orchestrating the releases and release coordination activities, with gated checks and approvals with a view of what is getting deployed to each environment, as well as traceability and audit.
    10. Enable feedback loops and automated alerts if firmware deployments or application updates fail.
    11. DevOps teams can define key tenets, criteria to be met and standards to be followed for the devices; infrastructure to be provisioned; device protocols usage and standards in connecting to existing/new devices and services. This can reduce the complexity in development and deploying to multiple varied devices by following the same protocols for connectivity and communication.
    12. Train the team of developers, QA and hardware developers on how to do continuous integration for IoT development, follow agile and lean practices and ensure security criteria for all layers of the application and in process.
    13. Adopt user-centric test automation and end-to-end testing approaches and build robust test cases in perspective of testing a complete flow, with devices/services that may not be in your scope but will be part of your overall IoT platform ecosystem.
    14. Create/utilize container based test environments and device simulators coupled with service virtualization for functional testing.
    15. Automated infrastructure provisioning – Use the right configuration management tools to automate the infrastructure along with dependencies to create environments through code instead of provisioning basic VMs.
    16. Use container and cluster management tools such as Docker Swarm or Kubernetes for service discovery, auto-scaling, self-healing and scheduling.
    17. Use RBAC, service accounts and security access controls in the infrastructure templates/scripts that are created instead of individual user credentials to limit the risk of unauthorized users provisioning the environments using scripts. Build security in devices, cloud security, application security and infrastructure automation templates. Use secrets, key vaults, encryption methods and certificates to secure the credentials and access control.
    18. Configure individual build and deployment pipeline jobs for application development and infra-automation development, as there will be a huge development effort involved in automating the infrastructure.
    19. Create infrastructure verification tests with scripting tools such as Python or use testing frameworks such as Pester or any other open-source tools and integrate the infrastructure tests with the CI/CD pipeline. Ensure to run these tests every time the infrastructure is provisioned.
    20. Version-control the infrastructure templates, scripts, user profiles, environment configurations and device configurations apart from source code.
    21. Set up a centralized dashboard of monitoring tools to monitor the clusters, nodes, firewall rules, VMs, applications, devices and services. Feedback should flow back to developers/QA/DevOps teams.
    22. Configure applications to provide logging about exceptions, memory leakages and response times and push the logs to the monitoring tools or the dashboards in monitoring tools.
    23. Use the telemetry from the applications and devices to derive the analytics graphs/trends and set up KPIs/metrics to measure application performance/availability, as well from derive business metrics to set SLAs for DevOps and operations, to improve productivity.
    24. Ensure DevOps teams have close coordination with dev/QA teams of both software and hardware teams, as managing IoT platforms need more skill and expertise in hardware and software in the IoT domain.
    25. Manage legacy and cloud infrastructure with multicloud management platforms, and monitor and optimize the infrastructure usage.
    26. Create a DevOps governance plan/guide, with clearly defined access control policies, access rights for devices, control limits on infrastructure provisioning in different clouds, limits on PaaS/IaaS services to be used, etc., and standardize the process to be followed in DevOps/development/testing/operations, tools, etc.

    Following DevOps practices for application development has become quite streamlined in recent years. However, in IoT, where traditional methodologies and practices are followed in industrial enterprises, adapting to agile methodologies and following DevOps practices coupled with right DevOps-skilled team, tools and expertise can streamline and simplify IoT developments.

    About the Author / Lavanya Subbarayalu

    Lavanya Subbarayalu is Senior Architect working with Technology Office in HCL Technologies. She has expertise in IOT, Azure, DevOps consulting & Microsoft technologies. She is associated with DevOps COE, working on design and Development of DevOps solutions and consulting tools. Connect with her on LinkedIn.

  • Building a Services and IoT Platform with DevOps, Part 2

    Building a Services and IoT Platform with DevOps, Part 2

    BCG’s latest IoT market research finds that predictive maintenance, automated inventory and self-optimizing production are some of the top use cases driving the need to build internet of things (IoT) platforms by 2020.

    As we think of building IoT solutions, it’s important to have a deep understanding of the business-level considerations and architectural design-level considerations before selecting the right IoT platform. This article talks about some of the important factors to consider in selecting an IoT platform.

    Considerations for Selecting the Right IoT Platform

    Following are some of the considerations in selecting a right IoT platform or IoT solution provider.

    Business Need and Requirements: Identify the business needs, estimate the ROI, identify what specific business and architectural use cases needs to be addressed.

    Examine the core differences in different IoT solution providers and analyze their evaluate out-of-the-box features against the defined business/technical limitations for the use cases you have identified before deciding to build custom or off-the shelf solution.

    Global Presence and Business Stability: Identify whether the existing IoT solution providers have a global footprint with datacenters across the geos, which will enable you to scale in that business region as your business grows.

    Device Management: Check if the platform and the hardware is supported by the IoT provider and supports remote monitoring of the devices and maintain over-the-air updates to the devices.

    Data Analytics: Look for options on how data will be stored, the data storage options to store large volumes of data, with segregation and storage of device and telemetry data, in addition to the extraction and reporting tools and dashboards.

    IoT Data Collection: Identify how data can be collected from the devices and how your devices can communicate to the respective platform (cloud or local). Each vendor has different means of collecting the data from the remote devices, For example, Azure IoT uses Azure IoT Hub while AWS uses Rules Engin, to connect and communicate to the devices and collect the data.

    Edge Computing: Identify the network latency, how to filter the data selectively from different devices and relay it to the cloud, the connectivity of devices that are not exposed to internet through a field gateway, the customization of the gateway and protocols and how to achieve quicker response times with different IoT platforms.

    IoT Protocols and Standards: Examine whether the device list supported by the vendor is developed as per the standards defined by IoT consortiums and whether the vendor supports standard protocols or uses any proprietary technologies.

    Compatibility and Extensibility: Examine for the compatibility of the devices to the vendor platforms, support for additional new devices to the platform and support for customization/extensibility of WRT gateway protocols and device rules.

    Storage: Identify platforms/IoT vendors that provide big data storage, data security, compliance and data recovery options.

    Security:  Think security at all levels from ground up—device-level security, device-to-cloud security, cloud security, data security and privacy, user-level security, application security and end-to-end security.

    Pricing and Hosting Models: Finally, compare hosting models and pricing models offered and any hidden costs. Pricing varies from per device to per million message-basis, per message volumes/day, cloud usage and other factors.

    IoT Architectural Considerations

    Hardware and Software Heterogeneity: The platform should support for various hardware and software devices from a wide range of device manufacturers, multiple devices, device operating systems and support multiple protocols including MQTT, web sockets and AMQP. It also should support device management and monitoring of connected/non-connected devices and seamlessly connect with multiple network providers.

    Enhance Security: The system should have security and privacy for device, users, device and user identity, authentication and authorization with policies, data protection (encryption/decryption) for data at rest and in motion, data attestation, SAS-based token per unique publisher, certificates, TLS authentication support and adhering security standards to enable developers to follow security standards during device development and security testing.

    Hot/Cold Data Path Analysis: Determine the network latency, support for high-scale data ingestion, support for high-scale data processing and high data retention options, support for segregating device metadata from telemetry data storage and the ability to perform analytics on the data with customizable dashboards.

    Scale Efficiently: The system should support for hyperscale deployments of individual data processing units such as event hubs, stream analytics and AWS Kinesis and also support millions of device and scale between small to hyperscaled dimensions of devices and telemetry.

    Flexibility: The platform should be extensible to use third-party solutions for individual conceptual components in IoT solutions and should support integration with business applications/workflows and REST APIs/CLIs for communicating to the systems from external systems.

    DevOps Automation: There should be support for the adoption of build, test and deployment automation for device and other solution components in an IoT solution with the right set of tools for DevOps. Testing and monitoring are key in DevOps automation with user-centric end-to-testing and automated releases of firmware updates to the remote devices. The platform you choose or building should be driven with DevOps capability.

    QoS, Cost: Other key factors to consider include maintaining data and devices, monitoring devices, scalability, reliability of systems, backup and disaster recovery of data and devices, performance of the devices, the network and pricing models (based on messages sent/received, per devices, cloud transactions, message size, throttling limitations of cloud based components, cost of ingress/egress/compute/storage/services/business apps/devices/security, etc.).

    Lower Barrier to Entry: The platform should support the ability to evaluate, prototype, deploy and scale quickly; lower the barriers to trying IoT out-of-the-box/preconfigured solutions; and support new development and experimentation with the data to refine and derive useful business metrics from telemetry.

    Building an IoT platform may be challenging, but making critical decisions on business factors and design considerations such as device management, security, storage, analytics and pricing before selecting a platform can help achieve ROI and lead to building a mature IoT platform.

    About the Author / Lavanya Subbarayalu

    Lavanya Subbarayalu is Senior Architect working with Technology Office in HCL Technologies. She has expertise in IOT, Azure, DevOps consulting & Microsoft technologies. She is associated with DevOps COE, working on design and Development of DevOps solutions and consulting tools. Connect with her on LinkedIn.

  • Building a Services and IoT Platform with DevOps, Part 1

    Building a Services and IoT Platform with DevOps, Part 1

    With the growth of the internet of things (IoT), expected to connect more than 50 billion devices (things) in next three years, building intelligent systems in a more connected world is presenting many challenges. Especially in building software for IoT, beyond hardware and network connectivity challenges there are numerous challenges in development, test, release, managing the infrastructure, monitoring the infrastructure and devices (things) and security.

    This article focuses on key components in IoT platform, how to select right set of DevOps tools for building those components, and creating a DevOps pipeline for the IoT platform.

    Building Your Own Services and IoT Platform with DevOps

    The main components in IoT solution comprises of device registration and identity, data ingestion, command and control, rules and actions for hot data analytics and cold data analytics, data storage, alerts and reporting.

    Device Registration and Identity: The devices/edge need to connect to each other and to a cloud gateway to send/receive the data, for which the devices have to be uniquely identified and registered at the gateway so that only authenticated and authorized devices/users can communicate to the cloud. Every IoT platform vendors has its own way of device registration and identity methods for storing device metadata such as AWS Things Shadow and AWS Gateway Rules Engine, Device twin and Azure IOT hub in Azure, GE Predix, etc.

    Data Ingestion: The devices communicate to the cloud/field gateway, which further ingests the data (device info and telemetry info, such as temperature data from sensors) into the cloud. You can use multiple options of PAAS services from Azure/AWS/GE predix, etc. for this component.

    Command and Control: These components cause the device to take an action and send messages to a device from the cloud, which can be built with PaaS services such as AWS Kinesis, Lambda functions, Azure functions, Azure Logic Apps, etc. or open-source technologies.

    Rules and Actions: These components act on specific device-to-cloud data, using the rules defined for processing hot data (where the data needs to be queried and analyzed within specified time interval) and cold data (where the data can be stored for delayed querying and further analysis). These components can also be built with existing PaaS services or using open-source tools such as Apache Storm, Map Reduce, Hive, Data Lake, etc. for data streaming and analytics.

    Data Storage: You need to store multiple types of data ranging from device metadata, telemetry data from specified set of devices, hot data and cold raw data, which needs to be stored and scaled up based on demand.

    Alerts/Notifications: The analyzed data needs to be visualized to derive more useful business metrics and outcomes, which can be integrated to external systems or other business applications to take preventive and predictive actions. Customized dashboards and data monitoring tools needs to be integrated to the solution to enable monitoring, alerts and notifications.

    With increased complexity in IoT development, the large number of devices to be monitored and large volumes of data to be analyzed and to manage complex infrastructure, DevOps becomes essential to the IoT development life cycle.

    Selecting the Right DevOps Tools for an IoT Platform

    Selecting the right DevOps tools for an IoT platform can help overcome challenges in creating/pushing applications and firmware updates to large scale of devices, as well as monitoring and infrastructure provisioning on-demand.

    With PaaS services used for IoT development, there is less code development but more configurations and rules-based development for developing key IoT components.

    Also, the teams developing device software and application development may be distributed, but the releases need to be streamlined with complete traceability and auditability. You need to manage a scalable infrastructure to support an IoT platform.

    Given below are some of the DevOps Tools used in most of IoT developments to help DevOps teams develop, build and release an IoT platform seamlessly.

    • Version Control: Git, for distributed teams and maintaining repositories for device development, application development and infrastructure automation development. Other tools are Clear case, SVN or TFVC.
    • Continuous Integration: Jenkins, VSTS (Visual Studio Team Services) and Bamboo to create jobs to do build, unit test and code quality analysis for the application, device software and infrastructure scripts, with the support of large collection of tools plugins and usage of different build tools such as gradle, maven, msbuild, etc., as well as supporting multiple technology stacks.
    • Infrastructure Provisioning and Automation: Ansible, ARM (Azure Resource Manager) and AWS cloud formation templates to automate the infrastructure in multiple cloud vendors and use tools such as Terraform or Spinnaker to automate the provisioning of infrastructure across hybrid clouds and maintain a common infrastructure blueprint, which enables the porting/migration of infrastructure workloads across clouds.
    • Infrastructure Testing: When PaaS services are used for developing certain components in an IoT platform, testing those PaaS components is a bit different—PaaS services have less code but more configurations and endpoints from dev/test team perspectives. Testing these components will involve endpoint testing, whether rules are created properly, how data flows between the different components and robust infrastructure verification. These infrastructure testing scripts can be written using Python/groovy or Vagrant to simulate the environment and test, or by using testing frameworks such as Pester for PowerShell scripts. You can extend your test solutions to leverage the REST APIs/CLI commands, exposed by different PaaS services, and these test scripts can test the PaaS services too, with simulated test data and generate the test result reports.
    • Continuous Deployment: VSTS Release manager, Jenkins, Capistrano, CA Nolio, etc. can be used to create the deployment workflows for deploying device firmware updates, application updates separately with control to release on multiple environments, gated checks and streamlined deployment process, with feedback enabled.
    • Test Automation: Selenium, Jmeter, security testing tools such as Veracode, EMMA, CA Lisa, Inspec, Robot framework and performance test tools are used for test automation. Testing in IiT is complex, as it needs an end-to-end testing of multiple devices interconnected. Most of the other device development may not be in your control, except your own device development, so to test an end-to-end functionality you need to have robust test cases and simulated test devices and test environments. This is where containers can help in test automation—a simulated device as a Docker container can run in one test environment and testers can use service virtualization tools such as IBM Rational Test Virtualization Server, CA Service Virtualization, Parasoft Virtualize or HP Service Virtualization to test their end-to end functionality when other dependent components/devices are being developed or not available.
    • Security and Code Quality Analysis and Security Testing is key in IoT development. As more and more devices are interconnected, the risk of data breach/data privacy is high; hence, a shift left of security testing during the development phase as part of the CI pipeline is inevitable. Tools such as Veracode, EMMA and Inspec can be used to run security analysis and identify security vulnerabilities and compliance issues. Unit testing tools such as junit, xunit, nunit or mstest can be used based on the technology stack. Code quality analysis tools such as SonarQube, coverity, cobertura and Fxcop are widely used for code quality analysis in IoT application components development.
    • Containerization and Container Orchestration: You can deploy your application as Docker containers, which have less footprint in the physical/hosted infrastructure and runs only with required application framework, libraries and dependencies. These containers/cluster of containers are managed with container orchestration tools such as Kubernetes, Docker Swarm, AWS Container Services, Azure Container Services, etc. As mentioned above, simulated test devices can run in docker containers, which enables shipping/moving the containers across any cloud and making the test environments ready within minutes.
    • Monitoring: Monitoring in IoT involves monitoring the devices; infrastructure VMs/clusters; telemetry data; the build, test and releases; and billing, among others. Metrics such as cluster usage, CPU usage, network availability, response time, number of devices connected and devices updated/not updated can be configured and viewed in monitoring tools including AWS Cloudwatch, Azure Application Insights & LogAnalytics, Stream Analytics, New Relic, Prometheus (open source), Nagios, Coscale, SysDig, Dynatrace and GE Predix Monitoring. The tools can be selected depending on the target platform on different clouds or on-premises, technology stack and other factors.
    • Cloud and Cloud Management: Cloud providers including AWS, GE Predix, Azure and GCP provide out-of-the-box features and support for IoT platforms with starter kits, IoT device SDKs, protocols, gateways customization and more. Cloud management tools such as Right Scale cloud management platform, OneOps, etc., can be used for managing the infrastructure workloads; bringing in central governance, access control and security; monitoring billing and optimization; managing security and standards; and streamlining and orchestration of IoT platform development.

    Depending on the technology stack, the DevOps requirements and business needs, the right tool needs to be selected to streamline IoT platform development with feedback flowing between the different stream of teams involved in IoT development, and to bring in standardization in security, governance and tools, metrics to measure and to better manage IoT platform.

    Release, Rollout Firmware/Application Updates

    Rollout of firmware updates for an IoT platform is more challenging compared to application releases; with device updates, we have to deal with network latency, issues with connectivity of devices, security issues, different sets of devices, different deployment channels, deployments done over the air and different sets of protocols used by different manufacturers.

    There is additional complexity involved when there are new additions to devices or when supporting legacy devices, and this needs incorporating integrated end-to-end testing as part of deployment pipeline, to test the devices and applications from a user-centric approach.

    In IoT development, there are multiple development and deployment pipelines for firmware updates, server-side code, mobile apps and multiple devices delivering to multiple endpoints. Hence, the release cadence of IoT components varies, as application code updates can be released monthly, whereas firmware updates to devices can happen every three to six months—hence, a well-defined process must be followed for regularly integrating the code.

    Continuous testing involves infrastructure testing, performance testing, end user testing and functional testing, all of which must be integrated with the release pipeline, where deployments can happen to simulated test environments, provisioned as environments as code (environment with all runtime dependencies needed to run the application like Docker containers), rather than simply infrastructure as code.

    DevOps teams need to keep track of releases and monitor all these deployments, ensure environments are stable, trace back the releases with the device firmware updates and track the defects. The DevOps teams should pull the release packages of different components from multiple pipelines and prepare main and customized builds to integrate and deploy the components together.

    The following are the main steps in CI/CD pipeline for IoT platform development.

    1. Code checked into version control: Check in the code for application development, device/firmware development, PaaS services configurations, infra automation scripts and templates to the version control, pointing to the defined repositories.
    2. Continuous Integration: Build jobs created in a CI server such as Jenkins/VSTS/Bamboo executes build compilation, unit testing, code quality analysis, automated reviews, environment configuration preparation, running security testing analysis, creating deployment packages or baked-in container images, for deployment.
    3. Triggers Deployment: The build agents generate the packages for firmware updates application updates and notifies release agent to trigger deployment.
    4. The deployment workflow can call a service/worker job, which in turn can connect to the cloud gateway services, to notify firmware updates.
    5. The cloud gateway picks up the firmware updates and notifies devices for devices updates. The devices will the pull the updates and installs the updates and further send telemetry data.
    6. Multiple testing environments in cloud can be integrated to the release pipeline to simulate the firmware updates.
    7. In case of application rollouts, there will be canary releases to gradually roll out to different environments in different regions after functional testing/regression testing.
    8. Testing: Security testing, firmware updates targeting to specific devices, performance of the device during the updates, device identity and memory usage, etc., needs to be addressed during deployments in IoT. Once the application is deployed to the test environment, functional/regression testing occurs and generates the test reports.
    9. Monitoring metrics such as memory usage, performance of the device, network traffic, latency, devices connected, reliability, etc., needs to be monitored, as these are of prime importance to decide on how to implement remote updates and monitoring without affecting the performance of the devices and these metrics has to be notified to the Dev/QA/DevOps teams as part of release pipeline execution.
    10. On-demand environment provisioning scripts/templates must be integrated with deployment pipeline to provision the test environments, add more devices as per the scaling metrics.

    Establishing a CI CD pipeline well ahead at the start of development and following mature DevOps practices such as security testing, infrastructure testing, containerization, monitoring, cloud and DevOps governance and feedback-driven development are the keys to help address the challenges in building your own services and IoT solutions.

    About the Author / Lavanya Subbarayalu

    Lavanya Subbarayalu is Senior Architect working with Technology Office in HCL Technologies. She has expertise in IOT, Azure, DevOps consulting & Microsoft technologies. She is associated with DevOps COE, working on design and Development of DevOps solutions and consulting tools. Connect with her on LinkedIn.