GE Architecture Introduction【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/geSystem Architecture OverviewGE (Graph Engine) is a high-performance graph mode implementation for Ascend series chips in the CANN ecosystem, composed of frontend adaptation layer, offline compilation toolchain, graph compiler, and graph executor components. The entire system forms a complete chain from frontend frameworks or model files, to AscendIR, to compilation artifacts model/OM, and finally execution on devices.The logical structure of GE is as follows:Frontend Framework Adaptation LayerThe frontend framework adaptation layer is responsible for using GE as thebackendfor mainstream deep learning frameworks. When users write and execute models in frameworks like PyTorch and TensorFlow, the adaptation layer completes IR conversion within the framework and calls GE. In this mode, GE is directly driven by the framework without independent invocation, called theonline scenario.Currently CANN provides official PyTorch and TensorFlow adaptation components:TorchAir: Converts PyTorchs AtenIR to AscendIRTF Adapter: Converts TensorFlows GraphDef to AscendIRAdaptation layers typically evolve synchronously with their respective frameworks and are maintained in separate repositories, integrated with GE through public interfaces. This ensures adaptation layers can evolve independently while keeping GE input boundary stable.atc (Ascend Tensor Compiler)atcis GEs standalone compilation toolchain for model compilation inoffline scenarios. In offline scenarios, GE does not participate in execution flow as a frontend framework backend, but directly compilesmodel files: users provide exported models (such as onnx, pb, etc.), and atc converts them to AscendIR and calls GE Compiler to generate OM files.Characteristics of offline scenarios:No Ascend device required(pure host-side compilation)No frontend framework runtime required(no dependency on PyTorch/TF graph execution)Artifacts can be deployed independently(OM files can be directly loaded and executed on devices)atc supports multiple frontend framework exported model formats, such as TensorFlow exported.pb, PyTorch exported.onnx.GE Compiler (Graph Compiler)GE Compiler is GEs core component, responsible for compiling AscendIR into binary models (model/OM) executable on Ascend devices. Its main work includes:Graph Optimization: Performs general compiler optimization, operator fusion, etc. at AscendIR levelOperator Compilation: Combines shape information inferred on the graph to perform online compilation of operators for better execution performanceStream Planning: Identifies parallel relationships in the graph, assigns concurrent operators to different streams to improve execution efficiencyMemory Planning: In static shape graphs, plans and reuses tensor memory from a whole-graph perspective to achieve better memory reuse rates and lower peak memory usageModel Serialization: Serializes compiled models into OM files (for offline scenarios)GE Executor (Graph Executor)GE Executor is responsible for model execution control on Ascend devices, including:Model Loading: Loads resources required for model execution onto Ascend devices, such as operator binaries (bin) and weights. For sink models, the complete execution sequence of the model is pre-loaded to the device side, enabling execution to be triggered with a single model launch without per-operator dispatch, with internal scheduling handled by hardware.Model Execution: Executes according to model semantics. GE Executor provides control logic required for execution, such as necessary branch jumps and stream synchronization.Terminology: In GE concepts, AscendIRs compilation artifact is called Model, so strictly speaking it should be called model execution. However, in practice, graph execution is also commonly used as a semantically equivalent expression.AscendIR GraphAscendIR is the core IR (Intermediate Representation) used in GE compilation flow, expressing model computation logic and data dependency structure using static computation graphs. AscendIR is also abbreviated as AIR, with the Air in TorchAir derived from this. AscendIR belongs to the high-level graph representation (HLO level) at the same abstraction level as ONNX, AtenIR, StableHLO, with operators and tensors as basic building blocks for describing model computation semantics and graph structure.In GE architecture, AscendIR has the following positioning and characteristics:Unified Compilation EntryRegardless of whether input comes from frontend frameworks (through Adapter) or from model files (through atc), all input is converted to AscendIR before entering GE Compiler. AscendIR thus constitutes the unified compilation entry for GE Compiler.Static Computation Graph StructureAscendIR represents static graphs, with graph structure fixed at compile time and not dynamically changing during execution. Static graph design enables GE to perform graph-level optimization, memory planning, and scheduling sink from a whole-graph perspective, achieving better compilation and execution efficiency.Distinguish between static graph and static Shape graph concepts.Static Shape graph: All tensor shapes are fixed across multiple executions.Dynamic Shape graph: Tensor shapes may change across different executions.AscendIR can express both static and dynamic Shape graphs, and GE supports compilation and execution of both. Since GE does not support dynamic graphs (i.e., graph structure dynamically changes during runtime), static graph / dynamic graph in communication is often shorthand for static Shape graph / dynamic Shape graph.Core Graph ElementsAscendIR graph is a directed acyclic graph (DAG), mainly composed of the following elements:Graph: Carries nodes, edges, input/output descriptions, etc., and is the basic processing unit for compilation.Node: Represents operator-level computation units, containing operator type, references to input/output tensors, and attributes.Tensor: Operator input/output data entities, including shape, dtype, format, and other metadata.Attribute: Operator additional information determined during graph construction, such as modes, configurations, or fixed parameters.Data Edge: Represents tensor producer-consumer relationships, directed from src node to dst node.Control Edge: Represents pure dependency relationships without data transfer; used to explicitly constrain execution order, ensuring src node executes before dst node.Implementation Note:In GEs actual implementation, there are no independent Edge objects in the graph, but rather edge relationships are described through Anchors.DataAnchorused to represent data edges (data flow)CtrlAnchorused to represent control edges (execution order only)Each anchor maintains its peer anchors, thus expressing connection relationships between nodes.Operator Definition SystemIn GEs overall architecture, AscendIRs basic structures (including Graph, Node, Tensor, Attribute, etc.) are maintained in the GE repository; however,specific operator definitions are not located in the GE repository, but are maintained in independent operator repositories(such as ops-math, ops-transformer, etc.). This design enables operator implementations to remain unified across both graph entry (Graph) and aclnn (native API call) scenarios.Why Operator Definitions Are Not in GE RepositoryGE is a graph compiler and executor, with main responsibilities:Graph-level optimizationCompleting scheduling, memory planningGenerating and serializing executable models (model/OM)GEdoes not define each operators semantics and implementation, so externalizing operators enables better decoupling. Both custom operators and built-in operators are external to GE and integrated through unified interfaces.Role of Operator RepositoriesOperator repositories (such asops-math,ops-transformer, etc.) assume the following responsibilities:Provideoperator definitions, includingoperator type, inputs, outputs, attributesImplement operators, includingShape inference rules, legality checks, Kernels, etc.Used simultaneously in two major scenarios:aclnn: Operator native API implementation and invocationGraph entry: GE references same operators during graph construction and compilationOperator repositories provide a unified operator source for the CANN ecosystem, with GE and aclnn sharing the same operator definitions and implementations, achieving consistency in operator semantics and precision.GE and Operator Repository CollaborationDuring graph compilation, GE checks Ascend graph legality based on operator definitions provided by operator repositories.During graph compilation, GE relies on operator implementations (such as shape inference, compilation) to complete partial optimizations.Operator repositories are released and upgraded independently from GE, enabling the operator system to evolve independently while GE only needs to maintain compatibility with operator specifications.This layered structure keeps GEs responsibilities clear as a graph compiler while ensuring the operator system is unified across the entire stack.Compilation OptimizationGE Compiler takes AscendIR as input and generates models (model/OM) executable efficiently on Ascend devices through multi-stage compilation and optimization flows. These optimizations cover graph-level, operator-level, scheduling, memory, and other dimensions, aiming to achieve better execution performance and lower resource usage while maintaining semantic correctness.Important optimizations in GE include the following categories:Graph-Level OptimizationGraph-level optimization performs structural transformations on the entire computation graph to improve execution efficiency, reduce redundant computation, or eliminate invalid operations. Overall, they fall into two major categories:general compiler optimizationsandfusion optimizations.General Graph Structure OptimizationThese optimizations are common graph transformation techniques in general compilers, such as:Common Subexpression Elimination (CSE)Retains only one occurrence of equivalent computations that appear repeatedly in the graph, reducing redundant computation.Constant FoldingEvaluates constant expressions at compile time, eliminating unnecessary runtime computation costs.Dead Code EliminationDeletes nodes and edges that have no effect on final outputs, making graph structure more compact and execution more efficient.Fusion OptimizationFusion optimization improves performance by merging multiple operators for execution, reducing kernel scheduling frequency, lowering intermediate tensor read/write, and in some scenarios reducing actual computation. Fusion optimization in GE is mainly divided into two categories:Pattern-Based Handwritten FusionPerform matching and fusion through handwritten pattern rules. The advantage of this approach is controllability and predictability, suitable for high-quality specialized handling of typical model structures.Classification-Based Automatic Fusion (Autofusion)Automatically analyze fusion opportunities based on operator computation formulas, input-output dependencies, and operator classifications, then generate fused operator computation code through codegen techniques and perform online compilation to generate kernels. This approach requires no handwritten patterns and can explore potential fusion combinations in a larger operator space.Operator-Level OptimizationGE uses tensor shape, dtype, and other information on the graph to deeply optimize operators, including:Operator Online CompilationCustomized compilation for operator input shapes, significantly improving operator execution performance.The goal of operator-level optimization is to improve single-operator execution efficiency without changing graph structure.Scheduling OptimizationScheduling optimization aims to generate efficient execution order and execution plans, reduce scheduling overhead during graph execution, improve parallelism, and maximize hardware pipeline utilization. GE mainly includes the following scheduling-related optimizations:Sink SchedulingSerialize the graphs operator sequence into a complete execution sequence directly schedulable on the device side. In this mode, execution only needs to trigger one launch, with operator scheduling inside the model automatically completed by the device side, significantly reducing host-side dispatch overhead, providing significant benefits for Host-bound models.Weight MergingMerge scattered weight data into contiguous memory regions, making the loading phase more efficient.Stream AllocationAnalyze concurrency opportunities in the graph and assign operators that can execute concurrently to different streams to improve overall pipeline parallelism. Reasonable stream allocation can significantly reduce critical path length and improve throughput.SuperKernel (Experimental Feature)Use online compilation capability to automatically merge a group of kernels executing sequentially on the same stream into one large kernel, reducing scheduling and switching overhead between kernels. This approach is suitable for scenarios with fine operator granularity, many kernels, and significant scheduling costs.Plugin and Extension MechanismGE supports extending compilation capabilities through plugins, includingAscendC custom operatorsandcustom Passes. Developers can add new operators or new optimization rules for specific business scenarios without modifying GE main code.Refer to the following resources for detailed development guides:AscendC custom operator graph entry: Search Operator Graph Entry (GE) Development on Documentation PortalCustom pass development: Custom Fusion PassProject Structure├── api # API interface implementation ├── base # Basic utility methods, module definitions ├── cmake # CMake common scripts ├── compiler # GE Compiler graph compilation module ├── dflow # DataFlow executor, providing heterogeneous model description and chained execution capabilities, will be decoupled from GE in the future and operate as independent repository ├── docs # Related documentation directory ├── examples # Usage examples directory ├── graph_metadef # Graph-related data structure definitions ├── inc # Header files ├── parser # Industry frontend framework IR to AscendIR implementation, currently supporting tensorflow/onnx/caffe/mindspore ├── runtime # GE Executor graph execution module ├── scripts # Packaging script file directory ├── tests # Developer test directory【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/ge创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考