The first time I ever wondered what a TPU was happened years ago, when I was experimenting with Google Colab. I noticed that alongside the usual CPU and GPU options, there was this mysterious third choice: TPU. Back then, I had no idea what made it different or why Google offered it as an alternative engine for running machine learning code.
Fast forward to today, and the landscape has changed dramatically. Google’s latest Gemini 3 Pro large model was trained entirely on Google’s custom Tensor Processing Units (TPUs), without relying on NVIDIA GPUs. That decision highlights not only Google’s confidence in its in-house hardware but also the rising importance of TPUs in modern AI development.
Both GPUs and TPUs are high-performance accelerators that are crucial to machine learning, yet they come from very different origins. GPUs began life as chips optimized for rendering graphics and were later repurposed for parallel computation, which made them invaluable for deep learning. TPUs, on the other hand, were designed by Google from the ground up specifically to accelerate neural network workloads.
In this article, we will explore the differences between GPUs and TPUs in the context of training and inference, and examine how each integrates into frameworks like TensorFlow and PyTorch, and when one might be a better choice than the other.
Architecture and Design Differences
GPUs consist of thousands of smaller cores optimized for parallel processing. This makes them highly versatile — they can execute many operations simultaneously, which is ideal for the matrix and vector math in deep learning. Modern GPUs (e.g. NVIDIA’s A100/H100) also include specialized units like Tensor Cores to accelerate mixed-precision matrix multiplications, on top of very high on-board memory bandwidth for shuttling data. In practice, GPUs offer a flexible architecture that can handle a wide range of computations (not just neural networks), which is a legacy of their graphics and general-compute roots.
TPUs, on the other hand, are application-specific integrated circuits (ASICs) designed from the ground up for machine learning tasks. Instead of general-purpose CUDA cores, a TPU chip contains large matrix multiplication units (e.g. 128×128 systolic array multipliers) alongside a few vector and scalar units. This highly specialized architecture means TPUs excel at tensor operations — the core math of neural networks — and can often achieve higher throughput and efficiency on those tasks than a comparably priced GPU. However, TPUs have fewer general-purpose cores and rely on feeding large batches of computations into their matrix units. As a result, they outperform GPUs in certain AI workloads that are well-structured for batch tensor operations, but are less flexible for arbitrary or highly dynamic processing.
Another key architectural difference is how these accelerators scale. GPU-based systems often use high-speed interconnects like NVLink or InfiniBand to enable multi-GPU training across servers. TPUs, in contrast, are designed to scale in “TPU pods” with a dedicated high-bandwidth interchip interconnect (e.g. a 2D torus network on TPU v4/v5) for tightly coupled parallel training. Google’s TPU pods can connect hundreds of TPU chips, enabling near-linear scaling for very large models. This tight integration is one reason Google can train enormous models like Gemini cost-effectively on TPUs — they control the full stack of custom silicon and networking. By comparison, GPUs can certainly be scaled to clusters as well, but typically with a bit more overhead in setting up distributed training (and often with more power per chip to get similar aggregate throughput).
Memory is another consideration: high-end GPUs today often have large on-board VRAM (e.g. 80 GB in an NVIDIA A100) with extremely high memory bandwidth. TPUs tend to have a more distributed memory design — each TPU core has its local high-speed memory (e.g. tens of GBs of HBM), but large models are spread across many chips. This means GPUs can sometimes fit a larger model in a single device, whereas TPUs assume you will partition the model across multiple cores. Google’s approach with TPUs has been to “scale out” with many chips, rather than rely on a single chip’s memory. In practice, for very large models, both architectures require splitting across devices, but it’s worth noting that GPU setups might use fewer, more memory-rich cards, while TPU setups use more chips with fast interconnects.
Training: GPUs vs TPUs
GPUs have been the workhorse for training deep learning models in most of the industry. Frameworks like PyTorch and TensorFlow were initially developed with GPU acceleration in mind, and the vast majority of open-source models (from CNNs to today’s LLMs) are trained on NVIDIA GPU hardware. One reason is the ecosystem: GPUs support a broad range of ML libraries and developer tools, and almost all research code, tutorials, and debugging tools are GPU-centric. If you’re fine-tuning a model with PyTorch or using advanced libraries (DeepSpeed, Megatron, etc.), the software expects GPU by default. GPUs are also readily available across cloud providers (AWS, Azure, GCP, etc.) or for purchase, giving practitioners flexibility to train on-premise or in various cloud environments.
TPUs in training shine when you have extremely large models or datasets and you can take advantage of Google’s infrastructure. Google itself uses TPUs to train and deploy massive models like PaLM and Gemini on vast TPU pod clusters. These chips are optimized for throughput: you feed them large batches of data and they crank through matrix multiplications with high efficiency. In scenarios like large-scale image classification or transformer model training, a TPU pod can complete training significantly faster (and often with lower energy use) than an equivalently priced GPU cluster. In one comparison, for example, training a ResNet-50 model for a set number of epochs took only 15 minutes on a Cloud TPU v3 versus ~40 minutes on an NVIDIA V100 GPU (with the same batch size). This isn’t to say TPUs always beat GPUs, but for well-optimized models, TPUs can offer superior training speed per dollar at scale.
That said, not every model is a good fit for TPU training. To fully utilize TPUs, your model architecture and input pipeline need to be XLA compiler-compatible. In practice, this means your code should avoid unsupported or very custom operations and ideally use static shapes. TPUs tend to struggle with dynamic shapes or control-flow heavy models, as well as certain custom ops that aren’t part of the TensorFlow/XLA supported set. For example, a research model with lots of Python-side logic, or a model that requires high-precision arithmetic throughout, might run slower on TPU or not run at all without modification. In contrast, GPUs can run almost anything you throw at them — from dynamic PyTorch models with conditionals to custom CUDA kernels — making them more flexible for experimentation. Debugging is also generally easier on GPUs (you can use tools like NVIDIA Nsight or simply print intermediate tensors) whereas on TPUs you’re more constrained by the compiled graph execution.
Another factor is cost and availability. GPUs have been expensive and in high demand (the so-called “NVIDIA tax” in recent AI booms), whereas Google offers TPU access that can be cost-competitive for the compute you get. In fact, Google’s Gemini 3 model card suggests that using TPUs allowed training longer and larger at a lower overall cost than would be possible with GPUs. However, TPU access is largely limited to Google Cloud. If you want to train on your own servers or another cloud, TPUs are not an option — you’d be using GPUs in those cases. Thus, the decision often comes down to ecosystem and scale: inside Google or for very large-scale TensorFlow projects, TPUs can be incredible; for most others (especially using PyTorch or needing multi-cloud/on-prem), GPUs remain the go-to standard.
Inference: GPUs vs TPUs
When it comes to inference (deploying trained models to make predictions), GPUs have been the default choice for high-performance serving in the industry. Modern GPUs are highly optimized for the matrix and vector operations in model inference as well — for instance, NVIDIA’s latest GPUs include features like Transformer Engines and support for lower precision (FP8/INT8) to speed up large language model inference. The tooling around GPU inference is very mature: frameworks like TensorRT, ONNX Runtime, or Hugging Face’s text-generation libraries all target GPUs to achieve low latency. As a result, many popular AI services run on GPU instances. For example, OpenAI’s ChatGPT service runs on NVIDIA A100/H100 GPUs, which are well-suited to transformer model inference.
TPUs also play a significant role in inference, particularly within Google’s own products. Because TPUs are designed from scratch for tensor math, they can be extremely fast and efficient for model inference at scale. Google has publicly shared that TPUs are used in Search, Google Photos, Google Maps, and other services to power ML models in production. For large models, a TPU v4 or v5 cluster can serve many requests in parallel with high throughput and low latency. In fact, TPUs were built with both training and inference in mind — the first-generation TPU (v1) was deployed just for inference in services like Translate, and later generations added training capabilities. Today’s TPUs (like TPU v5e) are optimized for high-performance inference, with improvements in throughput per dollar specifically for serving models efficiently.
Outside of Google’s cloud, however, TPU usage in inference is not common. Most organizations use GPUs or sometimes specialized inferencing GPUs (or other accelerators like AWS Inferentia, etc.) because of the convenience and broader support. If you are deploying a model on AWS/Azure or on-prem, you’re almost certainly using a GPU or CPU, as Google’s TPUs are not available there. That said, if you are in the Google Cloud ecosystem and your model is built with TensorFlow, you could deploy a TensorFlow Serving instance backed by a Cloud TPU for potentially higher efficiency on large batches. TPU inference especially shines when you need to serve massive models or high volumes of requests and want to minimize cost-per-query — Google has noted that TPU-based inference can be more cost-effective at scale, even if raw latency might be similar to a GPU’s, due to TPUs’ specialized hardware and Google’s datacenter optimization.
For edge or mobile inference, it’s worth mentioning that the term “TPU” also appears in Google’s Edge TPU (a small accelerator for IoT devices) and the NPUs found in smartphones (like Google’s Tensor chipset or Apple’s Neural Engine). These are specialized for low-power inference on-device. GPUs, conversely, have variants like NVIDIA Jetson for edge AI. While edge scenarios are beyond our main scope, it underscores a trend: specialized neural accelerators (TPUs, NPUs) tend to offer better efficiency for inference, whereas GPUs offer flexibility and are used as a general solution when specialized hardware isn’t available.
