Companies from all industries constantly face challenges to deliver software faster, with higher quality, and reduced risk and costs. This talk will cover how Continuous Quality (CQ) processes and automation along with an emerging set of best practices can enable both product owners and partner companies meet rapid release timelines with quality outcomes.

VMware offers a wide variety of partner programs for hardware and software development. Most programs provide dev kits, certifications, tools, and resources partners need to integrate, test and go-to-market with VMware. These partner programs are geared towards device manufacturers and OEMs that often require physical devices, specific hardware to execute the required testing. Certification kits representing different technology areas are offered to partners through a flexible and scalable SaaS platform that enables VMware Business Groups to offer validation and certification programs for both hardware and cloud-based software applications. The goal of these tests in certification is to validate the VMware integration and interoperability standards for partner products and software solutions thereby ensuring VMware product health in any Integration at customer environments.

Partners are expected to setup required testbed and run certification tests. Customer and Partner facing listings of VMware certified products and extensions are listed on the VMware Compatibility Guide. VMware Ready designates VMware’s highest level of endorsement for products created by our established partners. Products that meet VMware Ready requirements will display the VMware Ready logo. These products are jointly supported by both the Partner and VMware.

VMware and partner products undergo constant changes driven by innovation and to deliver better customer value. VMware Partner Engineering team does readiness assessments for new VMware releases, product stack changes, new features, design changes etc. and work closely with partners to enable them on latest product releases. VMware uses Continuous Quality (CQ) as a reliable mechanism to ensure faster release cadence. The process includes execution of Certification Suites continuously against development builds to ensure VMware/partner product changes do not break ecosystem hardware compatibility. The certification programs have disciplined change management and lifecycle policies and practices, which provide a stable baseline for checking vSphere and partner product compatibility. If a vSphere release breaks compatibility with the certification kit on known good hardware, it will likely have compatibility risks with partner products and/or certification kits. Identifying and resolving compatibility issues early in the development cycle is critical to ensure release of the product and certification suite on time.

vSphere is the industry leading cloud and virtualization platform. Hundreds of thousands of customers depend on vSphere to run their operations. Ensuring high quality of hardware and solution is crucial that leads to better customer experience and lower support costs. It is of paramount importance for VMware and ecosystem partners to ensure vSphere and partner solutions are compatible. VMware vSphere release cadence is every six months and typically there is a new vSphere development builds every week that are published through an API endpoint. The CQ auto-trigger process monitors the service and once a new build is detected, the CQ auto-trigger pipeline invokes the Job Manager which then kicks off a series of pipelines for various certification tests. The architecture supports parallel as well as sequential execution to manage and enable pipeline dependencies. Test pipelines execute on different server platforms. CQ results are part of the overall vSphere Product Readiness dashboard, which is reviewed regularly by vSphere leadership team.

The pipeline is configured to monitor test failures and trigger re-runs for failing tests to handle any flakiness. Any consistent failures after the re-run are tracked and marked as ‘Release Blockers’ until addressed by the relevant product/test teams. vSphere and the corresponding certification kits are released synchronously. Tests failures would lead to process interruption and slow down. Continuous quality process effectiveness and velocity is constantly enhanced through the following processes and best practices:

1. A variety of servers, hardware from different vendors is used to provide adequate ecosystem coverage.
2. Test pipelines are structured so that all required tests can be completed within one week.
3. Testbed setup and test execution are fully automated; no user intervention is necessary. The testbed setup and requirements are complex as it often requires specific hardware/devices to perform the testing.
4. Automatic test retry is used to overcome unexpected infrastructure related flaky failures.

VMware also enables partners to integrate our certification suites into their automated DevOps workflows and pre-release processes. This helps left shift certification validation during partner development, providing lead time to partners to act on issues thereby ensuring release readiness with quality at a reduced cost.

The continuous quality process ensures consistency through the reuse of robust certification tests in both internal and external partner environments. This approach benefits not only the product owners, partner development teams, but also the extended teams that develop solutions/products that integrate with VMware. Using the common set of certification tests allows the ecosystem to avoid redundant investment in developing quality tools and processes and helps establish a consistent quality standard. Consequently, stakeholders in the ecosystem can have more autonomy when they develop its products and solutions. Through continuous quality efforts in the entire ecosystem, companies can deliver innovative products with agility and high quality that delights customers.