Python Virtual Environments: venv and Dependency Management

After completing this topic, you will:

Understand why virtual environments are necessary, be able to create project-specific isolated environments with venv, and know how to manage dependencies using requirements.txt.

Why are virtual environments necessary?

In Python, pip install requests installs the package globally on your system. This becomes problematic when you have two projects that require different versions of the same package.

Project A: requires requests==2.28.0
Project B: requires requests==2.31.0
System:    Only one requests can be installed → Conflict!

Virtual environments create independent Python environments for each project. Each project has its own set of packages, eliminating conflicts.

Project A/venv: requests==2.28.0 (isolated)
Project B/venv: requests==2.31.0 (isolated)
System Python:  Unaffected

Creating and activating a venv

# Create a virtual environment
python3 -m venv venv

# Activate (macOS/Linux)
source venv/bin/activate

# Activate (Windows)
venv\Scripts\activate

# The prompt changes
(venv) $ python --version
Python 3.11.5

In python3 -m venv venv, the second venv is the folder name. It's common to use venv or .venv.

Verify Activation

# Which Python are you using?
(venv) $ which python
/home/user/project/venv/bin/python

# Deactivate
(venv) $ deactivate
$ which python
/usr/bin/python3

When activated, python and pip point to the ones inside the virtual environment. Deactivating returns you to the system Python.

Installing packages within a virtual environment

(venv) $ pip install requests flask

(venv) $ pip list
Package    Version
---------- -------
Flask      3.0.0
requests   2.31.0
...

These packages are installed inside the venv/lib/ folder and do not affect the system Python.

requirements.txt: Recording dependencies

# Record currently installed packages
(venv) $ pip freeze > requirements.txt

# requirements.txt
Flask==3.0.0
Jinja2==3.1.2
MarkupSafe==2.1.3
Werkzeug==3.0.1
click==8.1.7
requests==2.31.0
urllib3==2.1.0
certifi==2023.11.17
charset-normalizer==3.3.2
idna==3.6

pip freeze outputs all installed packages and their exact versions. Storing this in requirements.txt allows others (or your server) to recreate the same environment.

Installing dependencies

# Install the same packages in another environment
(venv) $ pip install -r requirements.txt

Manually writing vs. pip freeze

# Manually written (minimal)
Flask>=3.0
requests>=2.31

# pip freeze (complete)
Flask==3.0.0
Jinja2==3.1.2
... (including all dependencies)

pip freeze includes all transitive dependencies. Manually writing allows you to specify core packages and version ranges for more flexibility. Choose based on project size.

.gitignore: Don't commit the venv

# .gitignore
venv/
.venv/
__pycache__/
*.pyc

The virtual environment folder should never be included in git. Reasons:

1. Size: The venv folder can be tens to hundreds of MB.
2. OS Dependent: A venv created on macOS might not work on Linux.
3. Reproducibility: The requirements.txt file allows you to recreate it at any time.

# Start a project in a new environment
git clone project
cd project
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Real-world workflow

# 1. Start a project
mkdir my-project && cd my-project
python3 -m venv venv
source venv/bin/activate

# 2. Install packages
pip install flask requests pandas

# 3. Develop...

# 4. Save dependencies
pip freeze > requirements.txt

# 5. Git commit
git add .
git commit -m "Add requirements"

# 6. Someone else checks it out and runs
git clone <repo>
cd my-project
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
python app.py

Handling multiple Python versions

# If you have both Python 3.10 and 3.12 installed on your system
python3.10 -m venv venv310
python3.12 -m venv venv312

# Each venv is tied to the Python version used when creating it
source venv310/bin/activate
python --version  # Python 3.10.x

source venv312/bin/activate
python --version  # Python 3.12.x

venv uses the Python version at the time of creation. If you need different Python versions for different projects, create a venv with the corresponding version.

venv vs. conda

# conda environment creation (for comparison)
conda create -n myenv python=3.11
conda activate myenv
conda install numpy pandas scikit-learn

Conda is preferred in data science because packages like NumPy and SciPy depend on C/Fortran libraries. Conda manages these binary dependencies. However, for web development or general Python projects, venv is sufficient.

Modern Dependency Management: pyproject.toml

In Python 3.11+ projects, there's a trend to use pyproject.toml instead of requirements.txt.

# pyproject.toml
[project]
name = "my-project"
version = "1.0.0"
requires-python = ">=3.10"
dependencies = [
    "flask>=3.0",
    "requests>=2.31",
]

[project.optional-dependencies]
dev = [
    "pytest>=7.0",
    "black>=23.0",
]

# Install based on pyproject.toml
pip install .           # Production dependencies
pip install ".[dev]"    # Includes development dependencies

requirements.txt is a simple list, while pyproject.toml includes project metadata (name, version, Python version constraints). If you plan to distribute a library, pyproject.toml is the standard.

Common Mistakes

Key Takeaways

A virtual environment is "an independent Python for each project." Once you get into the habit, you'll eliminate package conflicts, the "it works on my machine" problem, and deployment environment inconsistencies. Make python3 -m venv venv the first command you run when starting a new Python project.