{"id":16826,"date":"2026-09-17T17:10:01","date_gmt":"2026-09-17T10:10:01","guid":{"rendered":"https:\/\/pmac.asia\/cau-truc-chip-ai-hbm-interposer\/"},"modified":"2026-09-18T17:48:13","modified_gmt":"2026-09-18T10:48:13","slug":"what-inside-an-ai-chip","status":"publish","type":"post","link":"https:\/\/pmac.asia\/en\/what-inside-an-ai-chip\/","title":{"rendered":"What\u2019s Inside an AI Chip? From GPUs and HBM to Interposers and Package Substrates"},"content":{"rendered":"

From the outside, an AI GPU is often viewed as a single chip. In reality, many modern AI chips<\/strong> are highly integrated systems composed of processing dies, HBM, interposers, high-density interconnect layers, and package substrates placed in close proximity. Each layer serves a distinct role in computation, data transfer, power delivery, and overall system stability.<\/p>\n

This architecture is becoming increasingly important as AI demands greater transistor counts and higher memory bandwidth. Chip performance therefore depends not only on GPU compute capability, but also on how data, power, and heat are managed across the components within the package.<\/p>\n

I. Core Architecture of a Modern AI Chip<\/strong><\/h2>\n

1. Processing Dies and Compute Functions<\/strong><\/h3>\n

At the heart of an AI chip is the processing die<\/strong>, which contains the transistors and logic circuits responsible for most computational operations. In a GPU, this includes parallel processing cores, cache, controllers, and specialized units designed for matrix operations used in AI training and inference.<\/p>\n

A GPU does not necessarily correspond to a single die. NVIDIA Blackwell Ultra<\/a>, for example, uses two dies approaching the reticle size limit, connected through the NVIDIA High-Bandwidth Interface with 10 TB\/s of bandwidth. Software still recognizes these two dies as a unified accelerator, illustrating how the physical boundaries of silicon increasingly differ from how a system defines a \u201cGPU.\u201d<\/p>\n

AMD<\/a> also adopts an architecture that separates functions across specialized silicon dies. CDNA 5 distributes compute, cache, and I\/O across different dies before integrating them within an advanced package. This approach allows each block to be optimized independently for performance, power consumption, and manufacturing technology.<\/p>\n

PMAC has explored multi-die architectures in greater detail in What Are Chiplets? Why Is the Semiconductor Industry Moving Toward Chiplet Architectures?<\/strong><\/a>. In this article, chiplets are considered within a broader system architecture to clarify how each layer fits into a complete AI chip.<\/p>\n

\"die-xu-ly-chip-ban-dan\"

Figure 1. Semiconductor processing die with an array of contact points<\/p><\/div>\n

2. HBM and Memory Bandwidth<\/strong><\/h3>\n

High compute capability can only be fully utilized when the processor receives data fast enough. If memory bandwidth cannot keep pace with the GPU\u2019s data requirements, compute units may remain idle and part of the processor\u2019s potential performance is left unused.<\/p>\n

HBM \u2013 High Bandwidth Memory<\/strong> addresses this challenge by vertically stacking multiple DRAM dies and placing them close to the processor. The DRAM layers are connected through TSVs, while the entire HBM stack communicates with the GPU through wide interconnects on the interposer. PMAC explains this architecture in more detail in What Is HBM? Why Do AI Chips Need High Bandwidth Memory?<\/strong><\/a>.<\/p>\n

HBM capacity is increasing rapidly alongside AI chip performance. AMD Instinct MI455X integrates 432 GB of HBM4 across 12 stacks, delivering up to 23.3 TB\/s of bandwidth<\/strong>, illustrating how memory has become a direct part of AI accelerator architecture rather than a separate component located far from the processor.<\/p>\n

As the number of HBM stacks increases, more area is required for connections between memory and logic. This places greater demands on the interposer, routing density, and overall package architecture.<\/p>\n

\"cau-truc-hbm-stack-tsv\"

Figure 2. HBM architecture with stacked DRAM dies, TSVs, micro-bumps, and a buffer die in an AI chip<\/p><\/div>\n

II. Interposers and Interconnect Networks Inside AI Chips<\/strong><\/h2>\n

1. Interposers in AI Chip Architecture<\/strong><\/h3>\n

Although GPUs and HBM may reside within the same package, they still require thousands of electrical connections to exchange data. An interposer<\/strong> is an intermediate layer that provides high-density interconnects between nearby dies, particularly between processing dies and HBM stacks.<\/p>\n

Compared with conventional circuit boards, interposers can support much finer routing dimensions and spacing, allowing more connections within a limited area. Shorter transmission paths also help control signal loss and power consumption as data moves continuously between memory and the processor.<\/p>\n

TSMC<\/a> applies this principle in its CoWoS<\/strong> platform, where multiple SoCs and HBM stacks can be integrated within a 2.5D system. According to TSMC, CoWoS is designed for AI and high-performance computing applications that require both high compute capability and substantial memory bandwidth.<\/p>\n

The interposer can therefore be understood as a high-speed interconnect network positioned directly beneath the dies rather than simply as a passive supporting layer.<\/p>\n

\"interposer-trong-chip-ai\"

Figure 3. Interposer architecture connecting the die to the package substrate<\/p><\/div>\n

2. RDL and Redistribution Layers<\/strong><\/h3>\n

Another important component is the RDL \u2013 Redistribution Layer<\/strong>. RDL uses fine metal traces to reroute signals from connection points on a die to different locations that align with the interposer or package substrate.<\/p>\n

RDL becomes especially useful when the contact density on a die exceeds what can be routed directly to the layer below. Connections can be redistributed across multiple layers, creating a network between logic, memory, and other structures within the system.<\/p>\n

Not every interposer is made entirely from silicon. TSMC CoWoS-R uses an RDL-based interposer, while CoWoS-L combines an RDL structure with localized silicon bridges to provide higher connection density where required. TSMC currently specifies a minimum RDL pitch of 4 \u00b5m for CoWoS-R.<\/p>\n

This development shows that interposers are evolving into a family of architectures rather than a single structure. Manufacturers can choose silicon, RDL, or hybrid configurations depending on requirements for interconnect density, package size, and cost.<\/p>\n

\"rdl-interposer-chip-ai\"

Figure 4. RDL interposer connecting SoCs and HBM to the PCB substrate<\/p><\/div>\n

3. Die-to-Die Interconnects Between Processors<\/strong><\/h3>\n

When a system is divided across multiple dies, data must cross the physical boundaries between different pieces of silicon. Die-to-die interconnects<\/strong> therefore need to deliver very high bandwidth while maintaining low latency and reasonable power consumption.<\/p>\n

Blackwell Ultra uses a 10 TB\/s interface between its two GPU dies. AMD CDNA 5, meanwhile, uses Infinity Fabric to connect compute, cache, I\/O, and HBM across its multi-die architecture. Although the implementations differ, both reflect the same principle: the quality of die-to-die connectivity increasingly affects overall AI chip performance.<\/p>\n

As the number of silicon components grows, system design must therefore address both raw computing capability and how efficiently individual blocks communicate with one another.<\/p>\n

III. Package Substrates in AI Chip Systems<\/strong><\/h2>\n

1. How Are Interposers Different from Package Substrates?<\/strong><\/h3>\n

Interposers and package substrates<\/strong> often appear in the same architectural diagram, but they serve different functions. The interposer focuses on high-density connections between nearby dies, while the package substrate expands those connections to a larger scale so the chip system can interface with the system board.<\/p>\n

The package substrate also provides mechanical support for the overall structure, assists with power distribution, and must maintain dimensional stability during manufacturing and operating temperature cycles. Interconnect dimensions therefore gradually increase as signals move from the die and interposer down through the substrate and onto the circuit board.<\/p>\n

The interposer can be viewed as the AI chip\u2019s internal interconnect network, while the package substrate serves as the interface between the packaged chip system and the rest of the server. PMAC discusses this broader role in What Is Semiconductor Packaging and Why Do Semiconductor Chips Need It?<\/a><\/strong>.<\/p>\n

2. Package Substrates and Increasing Package Size<\/strong><\/h3>\n

AI is driving larger package sizes as chip designers seek to integrate more logic and HBM within the same system. TSMC<\/a> has stated that, in 2026, it is manufacturing CoWoS at approximately 5.5 times the reticle size<\/strong>, while also outlining a roadmap toward even larger configurations for future generations of AI systems.<\/p>\n

Larger package dimensions increase requirements for flatness, warpage control, coefficient of thermal expansion, and interconnect accuracy. Even very large packages must maintain precise alignment across thousands or tens of thousands of extremely small connection points.<\/p>\n

These challenges are driving research into new substrate materials. PMAC has explored this direction in What Is a Glass Core Substrate? Why Is Glass Being Considered for Next-Generation AI Chips?<\/strong><\/a>.<\/p>\n

IV. 2.5D and 3D Integration in AI Chips<\/strong><\/h2>\n

1. 2.5D Integration in AI Chips<\/strong><\/h3>\n

In 2.5D integration<\/strong>, logic dies and HBM are typically positioned side by side on an interposer. The distance between them is much shorter than when components are placed separately on a circuit board, while each die still retains sufficient surface area for interconnection and thermal management.<\/p>\n

This architecture is widely used in AI accelerators because it enables very high bandwidth between the GPU and HBM without requiring all dies to be stacked directly on top of one another. TSMC\u2019s CoWoS is a representative example and includes variants based on silicon interposers, RDL interposers, and hybrid structures to accommodate different package sizes and connection densities.<\/p>\n

From the outside, such a system appears to be a single AI chip. Internally, however, it may consist of multiple processing dies and memory stacks distributed across a large silicon area.<\/p>\n

2. 3D Integration in AI Chips<\/strong><\/h3>\n

3D integration<\/strong> places certain dies vertically on top of others, shortening interconnect distances and increasing component density within the same footprint. HBM already follows this principle by stacking multiple DRAM dies vertically and connecting them through TSVs.<\/p>\n

At the logic level, direct silicon-to-silicon bonding technologies continue to advance. TSMC SoIC uses high-density bonding to integrate dies vertically, and its 3 nm stacking technology entered volume production in 2025.<\/p>\n

While 3D integration reduces transmission distances, it also creates additional challenges in thermal management, power delivery, and testing. For this reason, 2.5D and 3D integration are selected or combined according to the requirements of each architecture rather than being viewed as technologies that simply replace one another.<\/p>\n

\"tich-hop-2-5d-3d-chip-ai\"

Figure 5. Comparison of 2.5D and 3D integration with dies, interposers, TSVs, micro-bumps, and package substrates<\/p><\/div>\n

V. From a GPU to a Complete AI Chip System<\/strong><\/h2>\n

When the layers are considered together, the structure of a modern AI chip can be summarized as follows:<\/p>\n\n\n\n\n\n\n\n\n\n\n
Layer \/ Component<\/strong><\/td>\nPrimary Role<\/strong><\/td>\nKey Technical Requirements<\/strong><\/td>\n<\/tr>\n
GPU die \/ accelerator<\/strong><\/td>\nPerforms AI computations<\/td>\nTransistor density, performance, power efficiency<\/td>\n<\/tr>\n
HBM<\/strong><\/td>\nSupplies high-speed data<\/td>\nBandwidth, TSVs, stack stability<\/td>\n<\/tr>\n
Die-to-die interconnects<\/strong><\/td>\nTransfers data between silicon components<\/td>\nHigh bandwidth, low latency, low power<\/td>\n<\/tr>\n
Interposer \/ RDL<\/strong><\/td>\nConnects logic, HBM, and other dies<\/td>\nRouting density, signal integrity<\/td>\n<\/tr>\n
Package substrate<\/strong><\/td>\nRoutes signals and power to the system board<\/td>\nFlatness, dimensional stability, reliability<\/td>\n<\/tr>\n
Power delivery network<\/strong><\/td>\nSupplies power throughout the system<\/td>\nVoltage drop, current density, power stability<\/td>\n<\/tr>\n
Thermal system<\/strong><\/td>\nRemoves heat from the dies<\/td>\nThermal resistance and heat distribution<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

This architecture shows that a modern AI chip is closer to a miniature electronic system integrated within a single package<\/strong> than to a single piece of silicon. GPU compute capability remains central, but the ability to deliver data, power, and connectivity ultimately determines how much of that performance can be realized in practice.<\/p>\n

This also explains why chip architecture, HBM, and advanced packaging increasingly need to be developed together. A change in one layer can create new requirements across the others, from the interposer and package substrate to thermal management.<\/p>\n

VI. Power Delivery and Thermal Management in AI Chips<\/strong><\/h2>\n

1. Power Delivery Networks in AI Chips<\/strong><\/h3>\n

A high-performance AI chip must deliver significant electrical power from the system board to multiple silicon blocks. Current flows through the package substrate, power distribution layers, and connection points before reaching the GPU, HBM, and other components.<\/p>\n

As power density increases, even very small resistances along the current path can cause voltage drops and additional heat generation. Power architecture must therefore account for connection placement, the number of power paths, and the ability to respond to rapid changes in computational load.<\/p>\n

Advanced packages are increasingly bringing power-support components closer to the processor. TSMC CoWoS-L, for example, can integrate deep trench capacitors within the package to help stabilize power delivery near the processing die.<\/p>\n

Power delivery is therefore becoming part of the physical architecture of the AI chip rather than something addressed only at the server level after chip design is complete.<\/p>\n

2. Thermal Management in AI Chip Architecture<\/strong><\/h3>\n

Placing GPUs and HBM close together improves bandwidth, but it also increases heat density within a relatively small area. Heat generated by the dies must travel through thermal interface materials, heat spreaders, and ultimately the server\u2019s cooling system.<\/p>\n

The challenge becomes more significant in 3D structures, where one die may be positioned above another. Lower dies have a longer thermal path, while different regions of the package may develop hotspots with uneven power density.<\/p>\n

Die placement, HBM configuration, and power distribution therefore need to be considered together with thermal design from the architectural stage. Transistor density is not the only scaling limit; the ability to deliver power and remove heat also determines how far AI chips can continue to scale.<\/p>\n

VII. Materials Behind AI Chip Architecture<\/strong><\/h2>\n

1. Thin Films and Metallization<\/strong><\/h3>\n

Before a die is integrated into a package, it already contains numerous material layers with different thicknesses and functions. Conductive metals, diffusion barriers, dielectrics, and other functional films must be tightly controlled in terms of composition, thickness, and uniformity.<\/p>\n

PVD, CVD, and ALD<\/strong> address different requirements in thin-film deposition. PVD is widely suited to metallic layers, CVD supports a broad range of materials, while ALD is particularly useful when extremely thin films or highly conformal coatings are required on three-dimensional structures.<\/p>\n

PMAC explains the mechanisms and applications of these technologies in How Are PVD, CVD, and ALD Different?<\/a><\/strong>. Connecting thin-film deposition with AI chip architecture highlights how final device performance begins with the ability to control materials at very small scales.<\/p>\n

2. Metal Interconnects and Plating Technologies<\/strong><\/h3>\n

Copper and other metal systems continue to play important roles throughout semiconductor packaging, from RDL and TSVs to bumps and bonding surfaces. The thickness, composition, uniformity, and surface quality of these metallic layers influence electrical resistance and package reliability.<\/p>\n

As connection pitches shrink, process tolerances become increasingly narrow. Small variations in a metal layer can affect bonding performance or create differences in electrical resistance across densely packed interconnects.<\/p>\n

PMAC discusses the relationship between materials and interconnect quality in Semiconductor Plating Materials<\/strong><\/a>, particularly in processes involving Cu, Ni, Au, and Pd for surface treatment and electrical interconnection.<\/p>\n

3. Materials for Package Substrates<\/strong><\/h3>\n

Package substrates must simultaneously meet mechanical, electrical, and thermal requirements. As AI packages grow larger, warpage and differences in coefficient of thermal expansion between materials can create stress on bumps, interconnects, and bonding regions.<\/p>\n

Substrate materials therefore directly affect the ability of the package to maintain dimensional stability during heating, cooling, and operation. This is one reason technologies such as glass core substrates are receiving increasing attention as the industry seeks to scale package dimensions while maintaining very tight geometric tolerances.<\/p>\n

The evolution of AI chips is therefore influencing the entire materials chain, from wafers and thin films to interconnect layers and package substrates.<\/p>\n

VIII. Analysis and Metrology in AI Chips<\/strong><\/h2>\n

As the internal architecture of AI chips becomes increasingly dense, manufacturing requires measurement capabilities across multiple scales. Film thickness, material composition, bump height, substrate flatness, bonding structures, and internal defects can all affect device performance or reliability.<\/p>\n

Analytical technologies are selected according to the property that needs to be controlled. XRF can support composition analysis and thickness measurement for suitable coating systems; X-Ray can reveal many structures inside the package; SEM can examine morphology at very small scales; while profiling and film-thickness measurement systems help monitor process stability.<\/p>\n

The value of metrology lies in feeding measurement data back into manufacturing. As process tolerances become tighter, detecting deviations only after a product is complete can be too late and too costly. Measurement data can instead be used to monitor trends, detect process drift, and adjust equipment before deviations develop into large-scale defects.<\/p>\n

AI chip manufacturing is therefore becoming increasingly dependent on the ability to produce, measure, and control<\/strong> every material layer and structure within the system.<\/p>\n

IX. AI Chips as Multi-Layer Integrated Systems<\/strong><\/h2>\n

The development of AI is changing how the semiconductor industry defines a \u201cchip.\u201d A modern accelerator may combine multiple processing dies, several HBM stacks, and high-density interconnect networks on an interposer, with the entire structure then mounted on a package substrate and connected to the server\u2019s power and thermal systems.<\/p>\n

Scaling therefore takes place across multiple layers simultaneously. More powerful GPUs require greater memory bandwidth; additional HBM stacks require larger interposers and RDL networks; larger packages, in turn, increase demands on substrates, power delivery, thermal management, and process control. TSMC\u2019s CoWoS and SoIC platforms illustrate how the industry is advancing through both horizontal expansion and vertical stacking to integrate more computing resources within a single system.<\/p>\n

For the semiconductor supply chain, this shift is bringing materials, manufacturing equipment, and analytical technologies closer to the chip performance equation. This is also the direction PMAC is pursuing by connecting materials \u2013 equipment \u2013 processes \u2013 analysis and metrology<\/strong> around the practical requirements of individual manufacturing stages.<\/p>\n

X. Frequently Asked Questions <\/strong><\/h2>\n

Is an AI chip simply a GPU?<\/h3>\n

No. A GPU or accelerator performs the main computational workload, but many high-end AI chips also integrate HBM, interposers, RDL layers, package substrates, and power delivery networks within the same system. Real-world performance depends on how effectively these layers work together, particularly in transferring data between memory and logic.<\/p>\n

Where is HBM located inside an AI chip?<\/h3>\n

HBM is typically positioned very close to the processing die and connected through an interposer or another high-density interconnect structure. The short distance and wide interface allow memory to supply large volumes of data to the GPU at much higher bandwidth than memory placed farther away on a circuit board.<\/p>\n

Are interposers and package substrates the same thing?<\/h3>\n

No. An interposer provides high-density connections between nearby dies such as GPUs and HBM, while the package substrate routes signals and power from the chip assembly to the system board. It also provides mechanical support for the overall package.<\/p>\n

Why do AI chips use 2.5D or 3D integration?<\/h3>\n

These approaches bring logic and memory closer together, reducing transmission distances and increasing interconnect density. In 2.5D architectures, dies are typically positioned side by side on an interposer, while 3D architectures stack certain dies vertically. The choice depends on bandwidth, thermal constraints, package area, and manufacturability.<\/p>\n

Contact PMAC<\/strong><\/a> to discuss materials, equipment, and analytical and metrology technologies for semiconductor manufacturing, from thin-film deposition and metallization to surface treatment, material control, and technical requirements in advanced packaging.<\/strong><\/p>\n

PMAC Joint Stock Company<\/span><\/h3>\n

Ho Chi Minh City Office:<\/strong>
4th Floor, HUTECH Building, D1 Road, Saigon Hi-Tech Park, Tang Nhon Phu Ward, Ho Chi Minh City, Vietnam<\/p>\n

Hanoi Office:<\/strong>
22B O2, Linh Dam Peninsula, Hoang Liet Ward, Hanoi, Vietnam<\/p>\n

Hotline:<\/strong>\u00a0+84 387 235 878<\/p>\n

Facebook:<\/strong>\u00a0PMAC<\/a><\/p>\n

\"PMAC

Author:<\/strong> PMAC – Nguyen Quoc Dung \u2013 Lab & Maintenance Center Chief<\/p><\/div>\n","protected":false},"excerpt":{"rendered":"

From the outside, an AI GPU is often viewed as a single chip. In reality, many modern AI chips are<\/p>\n","protected":false},"author":5,"featured_media":16828,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[76,75],"tags":[600,601,515,602,591,592,603,81,469,604,487,514,605,606],"class_list":["post-16826","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-tech-news","category-news-media-en","tag-2-5d-en","tag-3d-en","tag-advanced-packaging-en","tag-chip-ai-en","tag-gpu-en","tag-hbm-en","tag-interposer-en","tag-pmac","tag-pmac-2-en","tag-rdl-en","tag-semiconductor","tag-semiconductor-2-en","tag-substrate-en","tag-wafer-en"],"_links":{"self":[{"href":"https:\/\/pmac.asia\/en\/wp-json\/wp\/v2\/posts\/16826","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pmac.asia\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pmac.asia\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pmac.asia\/en\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/pmac.asia\/en\/wp-json\/wp\/v2\/comments?post=16826"}],"version-history":[{"count":2,"href":"https:\/\/pmac.asia\/en\/wp-json\/wp\/v2\/posts\/16826\/revisions"}],"predecessor-version":[{"id":16832,"href":"https:\/\/pmac.asia\/en\/wp-json\/wp\/v2\/posts\/16826\/revisions\/16832"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/pmac.asia\/en\/wp-json\/wp\/v2\/media\/16828"}],"wp:attachment":[{"href":"https:\/\/pmac.asia\/en\/wp-json\/wp\/v2\/media?parent=16826"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pmac.asia\/en\/wp-json\/wp\/v2\/categories?post=16826"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pmac.asia\/en\/wp-json\/wp\/v2\/tags?post=16826"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}