Skip to main content
PyTorch Essentials
CHAPTER 06 Intermediate

PyTorch Tensors and Tensor Operations

Updated: May 16, 2026
6 min read

# CHAPTER 6

PyTorch Tensors and Tensor Operations

1. Introduction

The fundamental building block of PyTorch is the Tensor. A tensor is a multi-dimensional array of numbers. If you read Chapter 4, you might be thinking: *"Wait, that is exactly what a NumPy ndarray is!"* You are correct. Tensors and NumPy arrays are almost identical, but Tensors have two mathematical superpowers: they can run on GPUs, and they can automatically calculate calculus gradients. In this chapter, we master Tensor manipulation.

2. Learning Objectives

By the end of this chapter, you will be able to:
  • Create PyTorch Tensors from scratch or from NumPy arrays.
  • Understand Tensor dimensions (Ranks) and shapes.
  • Perform basic tensor operations (addition, multiplication).
  • Move tensors between the CPU and the GPU.
  • Understand the basics of the Autograd engine.

3. Creating Tensors

Let's import torch and create some basic tensors.
python
123456789101112131415161718192021 
import torch
import numpy as np

# Creating a 0D Tensor (Scalar)
scalar = torch.tensor(42)
print(f"Scalar: {scalar}")

# Creating a 1D Tensor (Vector)
vector = torch.tensor([1.0, 2.0, 3.0])
print(f"Vector Shape: {vector.shape}")

# Creating a 2D Tensor (Matrix)
matrix = torch.tensor([
    [10, 20],
    [30, 40]
])
print(f"Matrix Dimensions: {matrix.ndim}") # Output: 2

# Converting NumPy to Tensor (Very common in real-world workflows!)
numpy_array = np.array([1, 2, 3])
tensor_from_numpy = torch.from_numpy(numpy_array)

4. Tensor Operations

PyTorch includes heavily optimized functions for all standard mathematical operations.
python
12345678910111213141516 
tensor_a = torch.tensor([1, 2, 3])
tensor_b = torch.tensor([4, 5, 6])

# Addition
added = tensor_a + tensor_b

# Multiplication (Element-wise)
multiplied = tensor_a * tensor_b

# Matrix Multiplication (Crucial for Neural Networks)
# Note: Matrix multiplication requires specific shapes!
mat_a = torch.tensor([[1, 2], [3, 4]])
mat_b = torch.tensor([[5, 6], [7, 8]])
dot_product = torch.matmul(mat_a, mat_b)

print("Dot Product:\n", dot_product)

5. GPU Tensors

A massive advantage of PyTorch over NumPy is GPU acceleration. By default, tensors are created on the CPU. We must explicitly move them to the GPU.
python
12345678910 
# Check if GPU is available
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using device: {device}")

# Create a tensor directly on the GPU
gpu_tensor = torch.tensor([1, 2, 3], device=device)

# Or move an existing CPU tensor to the GPU
cpu_tensor = torch.tensor([4, 5, 6])
moved_tensor = cpu_tensor.to(device)

6. The Magic of Autograd

The second superpower of Tensors is Autograd. If you tell a Tensor requires_grad=True, PyTorch will quietly record every single mathematical operation done to that Tensor. When you call .backward(), PyTorch will instantly calculate the calculus derivative (gradient) for Backpropagation.
python
1234567891011 
# Create a tensor and tell PyTorch to track its gradients
x = torch.tensor(2.0, requires_grad=True)

# Perform a math operation: y = x^2
y = x ** 2

# Calculate the gradient (dy/dx of x^2 is 2x. If x=2, the gradient is 4)
y.backward()

print("Gradient of x:", x.grad) 
# Output: tensor(4.) -> PyTorch did the calculus for you!

7. Mini Project: Tensor Operations Playground

Let's generate some random tensors and inspect them, a very common task when initializing neural network weights.
python
1234567891011121314 
import torch

# 1. Create a tensor filled with random numbers from a normal distribution
# Shape: (3, 3) - A 3x3 matrix
random_tensor = torch.randn(3, 3)
print("Random Tensor:\n", random_tensor)

# 2. Find the maximum value in the tensor
max_val = torch.max(random_tensor)
print("\nMax Value:", max_val.item())

# 3. Find the index of the maximum value
max_index = torch.argmax(random_tensor[0]) # Checking the first row
print("Index of max value in row 1:", max_index.item())

*Notice .item()? If you have a Tensor that contains exactly ONE number (like maxval), you call .item() to extract the raw Python float out of it.*

8. Common Mistakes

  • Shape Mismatch in Matrix Multiplication: You cannot use torch.matmul on two tensors unless the inner dimensions match. E.g., A (3, 2) matrix can multiply with a (2, 5) matrix, but NOT a (3, 5) matrix. This is a strict rule of linear algebra!
  • Mixing Devices: You cannot add a CPU Tensor to a GPU Tensor. PyTorch will crash. Both tensors must be on the same device before performing math.

9. Best Practices

  • Use torch.float32: Deep learning relies on decimals. Ensure your input tensors are usually float32. Using float64 takes up twice as much RAM on your GPU and rarely improves accuracy.

10. Exercises

  1. 1. Create a PyTorch 2D matrix filled entirely with the number 1 (Hint: research torch.ones).
  1. 2. Write code to move a tensor named mytensor to the GPU using the .to() method.

11. MCQ Quiz with Answers

Question 1

What are the two primary advantages a PyTorch Tensor has over a standard NumPy array?

Question 2

If you try to add Tensor A (which is on the CPU) to Tensor B (which is on the GPU), what happens?

12. Interview Questions

  • Q: Explain what the requiresgrad=True parameter does when initializing a PyTorch Tensor.
  • Q: Explain what "Rank" means in the context of a Tensor and give an example of a Rank-2 Tensor.

13. FAQs

Q: Do I need to manually track gradients in production (e.g., when the model is making predictions for a user)? A: No! During production/inference, you don't need to do Backpropagation, so tracking gradients is a waste of memory. You should wrap your prediction code in a with torch.nograd(): block.

14. Summary

Tensors are the lifeblood of PyTorch. By storing multidimensional data in these specialized objects, we unlock the ability to process massive matrices on GPU hardware and automatically calculate the calculus required for AI learning using the Autograd engine.

15. Next Chapter Recommendation

We know the theory of neural networks, and we know the data structures (Tensors) that power them. It is finally time to put it all together. In Chapter 7: Building Your First Neural Network in PyTorch, we will build a model that can recognize handwritten numbers!

Finish this Chapter

Save your progress on your learning path and prepare for coding interview challenges.

Previous Chapter Understanding Neural Networks
Next Chapter Building Your First Neural Network in PyTorch

Discussion

Join the discussion

Log in or create a free account to participate.

Sort: ·

Explore More

📖 Related Tutorials 4