NEC SX-Aurora TSUBASA

SX-Aurora TSUBASA Architecture

NEC Vector Engine Processor

The NEC Vector Engine Processor of the new system will have eight independent vector cores.

There will be 64 fully functional vector registers per core, which can feed the functional units or which receive results from those. This is the first significant and important enhancement compared to the previous SX-systems, which only had eight vector registers, plus 64 so-called vector-data-registers used for intermediate results, but lacking the full functionality of a vector-register, namely to act as an input to the functional units. So in the future the compiler will have a lot more options to optimize the register usage and consequently this will enhance the efficiency of system.The vector registers will have 256 entries of 8-Byte-width, thus being able to handle double-precision data at full speed. So the full set of registers represents 128kByte of data readily available for computations at the speed of a register! The new system will have three FMA-units, plus one for divide and sqrt, per core. FMA stands for “floating-point multiply add”, and reflects an expression like a * b + c.
The importance of the pipe for divide and sqrt is often underestimated. Indeed such operations are not dominating in normal scientific applications. But if there are cases where they are needed, they often prove to be limiting, or at least contribute significantly to the calculations. Other architectures emulate such operations by a software-approximation, which can be time consuming.
In order to provide a high computing capability those pipes are designed with a 32-way parallelism, means, each pipe produces 32 results per cycle. Traditionally only add- and multiply-operations are counted when computing the peak-performance of an architecture. For the new system this is then 3 * 2 * 32 = 192 double precision floating point operations per cycle
There are more pipes for fixed-point arithmetic and logical operations. And all together they are fully connected to the vector-registers by a fully non-blocking crossbar.

NEC SX-Aurora TSUBASA Memory

The NEC Vector Engine Processor has a newly developed shared "Last-Level-Cache" (LLC), the first shared vector cache ever. This shared LLC serves all cores simultaneously, and has a "write-back" policy, which means coherency between different cores, LLC and memory is always easily ensured. At the same time this kind of architecture lends itself easily to a shared memory parallelization, by autoparallelization or OpenMP, while MPI would be used to parallelize over different Engines. The last level cache has a line-size of 128 bytes, and some additional features are implemented to increase the efficency for strided stores or scatter operations.

NEC utilizes the second generation of the so called "High Bandwidth Memory" standard, HBM2. A HBM2 memory block is realized by stacking four or eight dies together, and it achieves up to 200 GB/s bandwidth while providing either 4 or 8 GB of capacity. Six of these memory blocks and the Vector Engine Processor are connected by means of a so-called "Silicon interposer", a special die to mount on and that connects memory and processor. They provide a total of 24 GB to 48 GB per Vector Engine and an outstanding 1.53 TB/s bandwidth, the best bandwidth in the market.

NEC SX-Aurora TSUBASA Interconnect

The NEC Vector Engine can communicate with other Vector Engines or x86 CPUs over shared memory, PCI Express or a high speed network.

NEC SX-Aurora TSUBASA Software

NEC SX-Aurora TSUBASA Software Environment

With the NEC SX-Aurora TSUBASA release a new concept when it comes to software have been developed. The previous generation of NEC SX-series machines was running the Super-UX Operating System, that provided optimal performance for the dedicated set of hardware supplied with the NEC SX machines. Combining this knowledge with the functionality of the current Linux kernel stack enables a larger variety of hardware and software concepts for SX-Aurora TSUBASA.

The NEC SX-Aurora TSUBASA system takes advantage of the versatility of Linux. The users have access to all hardware and software that is supported by Linux, but for the applications the Aurora system offers a well-tuned and optimized compiler and libraries. The Vector Host takes care of all OS-related tasks as well as of tasks like scheduling or monitoring without disturbing the application. The application runs dedicated on the Vector Engine.

The interface between the Vector Host and the Vector Engine is controlled by a Linux Kernel module so that applications can be run transparently for the user on the Vector Engine.

The user will work in a complete Linux environment, including optimized compilers and highly tuned libraries. Execution of the application can be started from within the Linux environment either interactively or via a batch system. Everything that compiles for Linux can also be compiled for the Vector Engine. The compilers support Fortran 2003, with extensions from Fortran 2008, as well as C++14. The compilers are able to vectorize and auto-parallelize loops. For general parallelization OpenMP and MPI will be supported.

Applications

Computational chemistry

VASP (Vienna Ab initio Simulation Package)

Application Overview

Computational chemistry software for performing first-principles quantum mechanical calculations.

It has been widely used in more than 1,400 research institutes worldwide for the design of materials such as batteries, ceramics, catalysts, metals and semiconductors.

Usage

• • Electronic Minimization
  • Semiconductor Characterization
  • Structural Optimization
  • Molecular Dynamics
  • Characteristic Analysis of the Electrode
  • Magnetic Calculation
  • Transition State Calculations etc.

Acceleration

The basic algorithms of first-principles calculations are vector-oriented, as they are matrix operations and fast Fourier transforms.

The analysis calculation time is reduced by about “45%" compared to the calculation on the Xeon server and up to 1.78 times higher computing performance.

QuantumESPRESSO

Application Overview

An integrated suite of Open-Source computer codes for materials modeling based on electronic structure calculations of PWscf (Plane-Wave Self-Consistent Field).

Usage

• • Estimation of the electronic state of the crystal, electrical conduction properties and optical excitation spectra
  • Material structure exploration for material development
  • Modeling of various materials by molecular simulation of crystals, liquids, etc.

Acceleration

The algorithms underlying first-principles calculations are suited for vectors, as they are matrix operations and fast Fourier transforms.

The analysis calculation time is reduced by about “a half" compared to the calculation on the Xeon server.

Structural analysis

CalculiX

Application Overview

CalculiX is an open source software for finite element analysis.

It was developed by a team of employees at MTU Aero Engines, a German aviation engine manufacturer.

You can perform engineering calculations such as structural analysis, strength analysis, heat transfer analysis, and coupled analysis.

It has been developed to be compatible with the Abaqus input file format (INP), a well-known finite element analysis software.

Usage

• • Model building and computational processing for small aircraft such as jet engines
  • Falling Impact Analysis of equipment
  • Evaluation of warpage and stress due to thermal stress on electronic circuit boards, etc.
  • Prediction of Fatigue Life of Brazed Parts
  • Its design simulation can be used for modeling and analysis of various types of structures, and is used in architecture, manufacturing, and material development.

About the SX-Aurora TSUBASA version of CalculiX ("CalculiX for Aurora")

• • We have incorporated our NLC (Numeric Library Collection) HeteroSolver and SBLAS into the CalculiX solver
  • The implemented solver is available for Static analysis, Dynamic analysis, Thermal analysis and so on.
  • In this public version, it is possible to build a working environment using PrePoMax* as a pre-post processing tool
  • This “CalculiX for Aurora” version is the result of joint research with Professor Ryoichi Shibata of Gifu National College of Technology. In addition, Industry Supercomputer Promotion Center (ISCPC) cooperated with us in accelerating and creating a tool (builder) for building the usage environment

About PrePoMax

PrePoMax is a free structural analysis tool available exclusively for Windows that incorporates the traditionally highly evaluated open source "structural analysis CAE solver" called "CalculiX" and it is also extremely easy to use, approaching the level of commercial CAE.

Acceleration

The Aurora version of CalculiX uses a hybrid calculation model that combines VH and VE, which is one of the features of SX-Aurora TSUBASA.

We performed calculation in VH for those operations which are not suitable for vectorization such as pre-processing and post-processing parts of CalculiX, and embedded our NLC (Numerical Library Collection) HeteroSolver (direct method) and SBLAS (iterative method (CG method)) in the analysis solver to execute them in VE.

By this method, we achieved high speed calculation.

Comparing to the original SPOOLES solver on Xeon server, the analysis calculation is approximately 5 times faster for piston-stress static analysis and about 8 times faster for Contact Analysis of Nail Joints in Wooden Construction.

Library

Super Matrix Solver

Overview

• Super Matrix Solver is a collection of libraries originally developed by VINAS designed to solve simultaneous linear equations. It can be incorporated into numerical analysis applications in various fields.
• “Super Matrix Solver-BEM” for dense matrices and “Super Matrix Solver-MF” for sparse matrices was tuned for vector processors. This library provides quick and stable solution for various simultaneous linear equations.

Purpose

• Super Matrix Solver-BEM … to solve simultaneous linear equations formulated and discretized by boundary element method. Applicable in fields such as wave propagation analysis, ground vibration analysis, acoustic problem analysis, electromagnetic field analysis, etc.
• Super Matrix Solver-MF … to solve simultaneous linear equations discretized by methods such as finite element, finite volume and finite difference . Applicable in fields such as structure analysis, fluid analysis, electromagnetic field analysis, etc.

Performance

• Realizes High-speed computation because solving simultaneous linear equations mostly consists of matrix operations, which high performance of SX-Aurora TSUBASA contributes. Significant time saving can be achieved specifically for dense matrices.
• Shared memory parallelization with OpenMP is built in.
• Computational time can be reduced to approximately maximum 1/15 compared to Xeon CPUs.

Programming Languages, Tools

Programming Languages and Compilers

The Fortran compiler conforms to the Fortran-2003 standard (ISO/IEC 1539-1:2004) and supports many features from Fortran-2008 (ISO/IEC 1539-1:2010).

The C compiler conforms to C11 (ISO/IEC 9899:2011), the C++ compiler supports the C++14-standard (ISO/IEC 14882:2014) and supports some features of C++17 (specification in progress, ISO/IEC 14882:2017). The C and C++ compilers are highly compatible with GNU C and C++, and will support a lot of the related extensions.

• Supporting the same compiler options as the GNU C and C++

• Supporting many useful GNU C preprocessor functions

• Allowing for GCC-specific pragmas

• __attribute__

• Support for inline assembly language

NEC has more than 30 years’ experience in compiler technology for high-end vector machines, and these newly developed compilers will be based on this experience and support all the optimization strategies that NEC compilers are known for.
In particular these compilers will all support automatic vectorization on the high level for which NEC compilers were famous in the past. A set of directives is supplied to provide additional information to the compiler and to tune the vectorization further.

NEC MPI

The MPI implementation of the SX-Aurora TSUBASA works across high speed network, over PCI Express and shared memory regions inside the Vector Engine Processor. This way low latency and high throughput data transfer are possible. The MPI implementation supports the latest MPI 3.1 standard and is available for both Fortran and C/C++.

Powerful software development aid tools

Performance analysis tool, “NEC Ftrace Viewer”

NEC Ftrace Viewer is a performance analysis tool for Fortran/C/C++ programs. It works for parallel codes which use MPI and OpenMP and displays not only information about the execution time of MPI processes or threads but also information about the communication time between MPI processes in a graphical manner to identify performance of bottlenecks and load imbalance. The NEC Ftrace Viewer supports a wide variety of performance analysis ranging from a single thread program to a large scale parallel program.

Debugger, “NEC Parallel Debugger”

NEC provides a parallel debugger which enables interactive debugging from a user terminal. It can handle Fortran/C/C++ and provide various functionalities including debugging of applications based on MPI and shared memory parallelization with OpenMP.

※ OpenMP ：OpenMP is a registered trademark of OpenMP Architecture Review Board.