Python — Decoding Virtual Environment

This article explains what is inside a Python virtual environment.

What is Virtual Environment

A Python virtual environment is like a sandbox where you can work on your Python projects separately from your system-installed Python.

It provides an isolated environment where you can set up your own libraries and dependencies without affecting the system-wide Python installation.

Think of it as a self-contained Python universe tailored to your project’s needs.

Python virtual environment can be created using venv,virtualenv, conda etc. tools.

The below listed steps are required for Creating and Managing Virtual Environments:

• Create: Use tools like venv, virtualenv, or conda to create a virtual environment.

• Activate: Activate the environment to work within it.

• Install Packages: Install project-specific packages into the environment.

• Deactivate: Exit the environment when done.

Why virtual environment

Python is not very good at dependency management. Hence, pip will install all packages in the default Python installation (System-wide installation). All packages are installed in the folder site-packages/ in your default Python installation. On Windows, the default installation is in the folder.

C:\Users\<YourUserName>\AppData\Local\Programs\Python\Python<version>\

For example, on my Windows machine, the Python system-wide installation (default) is in the below folder:

C:\Users\MCA\AppData\Local\Programs\Python\Python311\

If you do not know, the default Python installation directory:

1 — Open the Command Prompt (CMD).

2 — Type the following command and press Enter:

where python

This will display the path to the Python interpreter. For example:

C:\Users\MCA\AppData\Local\Programs\Python\Python311\python.exe

Alternatively, you can also follow the below steps (Windows 10):

1 — Type ‘Python’ in the Windows Search Bar.

2 — Right-click on the Python App and select “Open file location.”

3 — Right-click again on the Python shortcut and select “Open File Location.”

4 — You’ll now get the location/path where your Python is installed on Windows.

On Linux, you can use the which python command. This command will display the path to the default Python installation, such as /usr/bin/python3. You can also use the whereis command to find the location of specific versions of Python interpreters, such as

whereis python2.7  or 

whereis python3.2

Installing all packages to the same folder will create dependency issues if you are working on different projects requiring different versions of the same package.

A virtual environment solves the problem of dependency issues, that’s why almost every matured Python developer will recommend you to use a virtual environment.

On Debian-based systems such as Ubuntu, you will have dist-packages instead of site-packages.

The concept of “purlib” and “pathlib”

In Python, both purelib and platlib folders are related to the installation of Python packages and modules. Let’s explore what each of these folders means:

“purelib”

• The purelib directory is where pure Python modules (those without any platform-specific binary components) are installed.

• It contains Python packages and modules that can be run on any platform without modification.

• These modules are architecture-independent and do not rely on any specific system libraries.

• For example, if you install a Python package using “pip”, the pure Python files (“.py” files) will be placed in the purelib directory.

“platlib”

• The platlib directory is where platform-specific binary extensions (such as compiled C extensions) are installed.

• It contains modules that have platform-specific components or dependencies.

• These modules may include compiled code (e.g., “.pyd” files on Windows or “.so ” files on Unix-like systems).

• The “platlib” directory allows binary extensions to follow the system’s conventions.

You can summarize it like — “purelib” directorycontains pure Python modules, while “platlib” directiry holds platform-specific binary extensions.

Both “purelib” and “platlib” are sub-directories inside the site-packages directory on most platforms, if any exist on your system.

However, on 64-bit Red Hat-based platforms (like CentOS 7), the paths may be as follows:

• ‘purelib’: ‘$base/lib/python$py_version_short/site-packages’

• ‘platlib’: ‘$platbase/lib64/python$py_version_short/site-packages’

• This separation makes sure that 64-bit binaries go into a “lib64 ”directory, while architecture-independent files live in a “lib” directory.

• The path to the platlib directory depends on the specific platform and system configuration.

It is importe to note here that on most of the default installations, the directories — “purelib” and “platlib”may not be available. The reason is that the default Python configuration makes “purelib” and “platlib”identical, which somehow defeats the purpose of the distinction in the first place.

On Windows, you can check the path for “purelib” and “platlib” directories using sysconfig as shown below.

Code: (Save it in a “.py” file and execute)

import sysconfig

print(sysconfig.get_path("purelib"))

print(sysconfig.get_path("platlib"))

Output:

C:\Users\MCA\AppData\Local\Programs\Python\Python311\Lib\site-packages
C:\Users\MCA\AppData\Local\Programs\Python\Python311\Lib\site-packages

OR

# Run the below lines of code directly on Python Interpreter 
>>> import sysconfig
>>> sysconfig.get_path("purelib")
'C:\\Users\\MCA\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages'
>>> sysconfig.get_path("platlib")
'C:\\Users\\MCA\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages'

The output is in the format

C:\Users\<user-name>\AppData\Local\Programs\Python\Python311\Lib\site-packages

On other platforms, the complete path may be different but still the same for “purelib” and ‘platlib’.

So, with most installations, you will have a directory “site-packages” where package‌s are installed except Debian and its derivatives where you will have “/dist-packages”.

The conclusion is that the same single directory site-pakages will not let you install different versions of the same package. Further, a project may not require all installed packages.

Even if you see two separate paths for “puerlib” and “platlib” i.e. your operating system distinguishes between the two. The dependency conflicts will occur because all “purelib” modules will be stored in a single location, as will “platlib” modules.

The main issue is that you cannot have a project-specific install of packages.

The solution to all such problems is the virtual environment.

Virtual Environment — The Structure

When you create a new virtual environment with the “venv” module. Python makes a directory (folder) structure all on its own. Then, it either copies or creates symbolic links of the Python executable files inside that directory structure.

Execute the tree command to get the contents (structure) of the virtual environment, i.e. the structure of the directory having the name of the virtual environment.

Assume that the virtual environment name is “my_env”. So, navigate to the directory, which has “my_env”; then run the below command on Windows PowerShell.

tree my_env /F

On Linux, type $ tree my_env. But you may need to install “tree” first, may using sudo apt install tree.

The output of the tree ​command, i.e. the content of your virtual environment directory — the my_envdirectory is a very long tree structure. It is so, since a virtual environment directory contains a lot of files and sub-directories. Take a look at the outputted tree structure, you will observe that the long tree structure is mostly due to the contents inside the site-packages directory. The below image shows a trimmed-down version of the tree command’s output on Windows platform.

virtual environment directory structure

On Linux or Mac systems, you will have “/bin” instead of “/Scripts” sub-directory within your virtual environment directory.

Here is a brief description of each sub-directory within the virtual directory.

Include\

Python uses this directory to store C header files for installed packages that rely on C extensions. Initially it is an empty folder.

Lib\

This directory has the site-packages\ subdirectory, which is responsible for most of the virtual environment. Python will install all external packages intosite-packages\ directory. By default, your virtual environment has two packages — “pip” and “setuptools”. Using pip is the recommended way to install packages in Python, and “setuptools”is a dependency for pip.

Scripts\

Consists of your virtual environment’s executable files. The Python interpreter (python.exe), the pip executable (pip.exe), and the activation script for your virtual environment. All these executable files are available in several formats to allow you to work with different shells.

For example, “activate” is used for the activation of your virtual environment on most ‌“Windows” shells, such as PowerShell.

pyvenv.cfg

It’s an important file for your virtual environment. It includes a few key value pairs that Python needs to configure variables in the sys module. These configurations control which Python interpreter and site-packages directory; the current Python session will use.

On Linux platform, the bin/ contains the executable files of the virtual environment instead of Scripts\. Again, which activation script you will use to activate your virtual environment depends on what shell you use. For example, you can execute “activate”, which works for the Bash and Zsh shells.

One last that deserves explanation is that there is a pkg_resources/ sub-directory which helps applications discover plugins automatically and allows Python packages to access their resource files. It’s distributed together with “setuptools”.

The virtual environment is composed of three key pieces:

1 — A copy or a symlink of the Python binary

2 — A pyvenv.cfg file

3 — A site-packages directory

Recapitulate

This article delivers a superficial idea of the structure of a virtual environment. Remember that a virtual environment created with “venv” module only takes care of installed packages. So you still have to manage the Python version manually. More on this in a different article; till then, keep reading.