Data science is now a first-class citizen in the HPC data center, in the cloud, and on the workstation. The converse is also true, High Performance Computing (HPC) is now a high priority for data scientists who confront issues of time-to-model when training, inference latency, and of course, the ever present need to manipulate large amounts of data. HPC clusters must support these combined workloads – or more specifically “converged” HPC, AI, and High-Performance Data Analytic (HPC-AI-HPDA) workloads.

This convergence of HPC-AI-HPDA reflects a golden age for data science as people expect to run with supercomputer performance on their HPC systems be they workstations, on their local clusters, and in the cloud. The reason is that workload optimized servers for AI provide optimized software stacks that run industry-standard tools like TensorFlow* with support for vectorized low-precision arithmetic along with vector AVX-512 floating point operations with high performance. Higher core count processors deliver greater parallelism in high memory bandwidth configurations. These capabilities deliver performance and eliminate the need for accelerators for many organizations.

AI and cloud innovation have helped stimulate a golden age of HPC as well. The mass adoption of AI has introduced new thinking where AI is part of a simulation and modeling process along with a new ecosystem of industry-standard tools and workload-optimized, high performance hardware. New 2nd Generation Intel® Xeon® Scalable processors with Intel® Deep Learning Boost demonstrate a real convergence at the hardware level while facilitating better convergence for workloads.

The AI and Data Analytics mass driver

In a nutshell, the huge growth in electronically analyzable data coupled with a large, accessible open-source ecosystem of industry-standard AI and data analytics tools lets users work with data in ways that has transformed the computer industry and created a new era in HPC:

The supercomputer as an economic driver

The impact of AI on computing has been substantial. Narrative Science reports that 61% of study respondents are currently implementing AI, with predictive analytics as the most widely used type of AI-powered solution.[i] Similarly, AI continues to become more important to the enterprise; MarketWatch* estimates growth of the global AI market at a compound annual growth rate of 36% through 2024.[ii]

The need for workload-optimized servers

AI has made a large impact on HPC as discussed in this white paper, The Case for Running AI and Analytics on HPC Clusters . The data intensive nature of AI workloads and need to run them in HPC environments mean that vendors now speak in terms of “building blocks” that include processors, accelerators, fast storage and networking building blocks plus simulation and modeling-focused tools and optimization.

Many vendors now sell workload-optimized servers as building blocks that simplify cluster configuration and ensure performance. Intel® Select Solutions, for example, offers servers verified for optimized performance to ensure efficient operation including:

• Processor and accelerator configuration
• Scalable networking capability that supports AI and HPC
• Scalable storage for data intensive workloads
• Provide upgrade paths to new technologies including accelerators, dedicated neural network hardware (like the Intel Neural Processor), as well as FPGAs.

Summary

Succinctly, running AI and HPDA on HPC systems has clear advantages, which means both data science and HPC organizations must now support HPC-AI-HPDA workloads. Intel also supports the vision to pool multiple HPC-AI-HPDA clusters together inside a unified data environment to save money and user pain.

Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration.  No product or component can be absolutely secure. Check with your system manufacturer or retailer or learn more at intel.com.

Intel and the Intel logo are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries.

*Other names and brands may be claimed as the property of others.

© Intel Corporation

---

[i] Outlook on Artificial Intelligence in the Enterprise 2018. Narrative Science https://medium.com/@narrativesci/2018-outlook-on-artificial-intelligence-b1b63a7386f4

[ii] Artificial Intelligence Market to Rise at Spectacular CAGR of 36.10% During 2016-2024, Players in End-use Industries Leverage its Potential to Automate Processes – TMR. MarketWatch, August 16, 2018, https://www.marketwatch.com/press-release/artificial-intelligence-market-to-rise-at-spectacular-cagr-of-3610-during-2016-2024-players-in-end-useindustries-leverage-its-potential-to-automate-processes—tmr-2018-08-16