Requirements#

Understanding Construbot’s Python dependencies and how to manage them.

Overview#

Construbot uses a structured approach to dependency management with separate requirement files for different environments:

  • base.in / base.txt - Core dependencies (shared across all environments)

  • local.in / local.txt - Development tools and utilities

  • test.in / test.txt - Testing frameworks and tools

  • production.txt - Production dependencies (inherits from base)

Note

`.in` files: Source files with high-level dependencies

`.txt` files: Compiled files with pinned versions (generated by pip-compile)

Requirements Structure#

File Organization#

requirements/
├── base.in          # Source: core dependencies
├── base.txt         # Compiled: pinned core dependencies
├── local.in         # Source: development dependencies
├── local.txt        # Compiled: pinned development dependencies
├── test.in          # Source: testing dependencies
├── test.txt         # Compiled: pinned testing dependencies
└── production.txt   # Production (just includes base.txt)

Workflow:

  1. Edit .in files to add/remove packages

  2. Run make pipcompile to generate .txt files

  3. Install from .txt files with pip install -r requirements/<env>.txt

Core Dependencies (base.txt)#

Django and Extensions#

Django 3.2.19:

Django==3.2.19

Long-term support (LTS) version with security updates until April 2024.

Django Extensions:

django-environ==0.9.0        # Environment variable management
django-model-utils==4.2.0    # Model utilities and mixins
django-allauth==0.51.0       # Authentication and registration
django-crispy-forms==1.14.0  # Form rendering
django-bootstrap4==22.1      # Bootstrap 4 integration
django-treebeard==4.5.1      # Tree structures (hierarchical contracts)
django-compressor==4.0       # Static file compression
django-autocomplete-light==3.9.4  # Autocomplete widgets

Why these extensions?

  • django-environ: Manage settings via environment variables (12-factor app)

  • django-allauth: Complete authentication with email, social auth

  • django-treebeard: Hierarchical contract structure (materialized path)

  • django-autocomplete-light: User-friendly select widgets for foreign keys

Database and Caching#

psycopg2-binary==2.9.6       # PostgreSQL adapter
redis==4.3.4                 # Redis client
hiredis==2.0.0               # C parser for Redis (performance)

Why PostgreSQL?

  • ACID compliance for financial data

  • Advanced features (JSON fields, full-text search)

  • Production-ready and scalable

Why Redis?

  • Caching layer for performance

  • Celery message broker

  • Session storage (optional)

REST API#

djangorestframework==3.13.1           # REST framework
djangorestframework-simplejwt==5.2.0  # JWT authentication
django-cors-headers==3.13.0           # CORS support
drf-yasg==1.21.3                      # API documentation

Features:

  • JWT token authentication (stateless)

  • Browsable API for development

  • Automatic API documentation with Swagger

Task Queue#

celery==5.2.7                # Distributed task queue
django-celery-beat==2.4.0    # Periodic tasks
flower==1.2.0                # Celery monitoring

Use cases:

  • Sending emails asynchronously

  • Generating PDF reports in background

  • Scheduled data synchronization

Document Generation#

reportlab==3.6.12            # PDF generation
openpyxl==3.0.10             # Excel file handling
Pillow==9.3.0                # Image processing

Features:

  • Generate construction estimates as PDF

  • Import catalog data from Excel

  • Process uploaded images

Email#

django-anymail[mailgun]==9.0  # Email service integration

Supported providers:

  • Mailgun

  • SendGrid

  • Postmark

  • Amazon SES

Utilities#

python-slugify==6.1.2        # Slug generation
python-dateutil==2.8.2       # Date utilities
pytz==2022.6                 # Timezone support
argon2-cffi==21.3.0          # Password hashing

Why Argon2?

  • More secure than PBKDF2 or bcrypt

  • Recommended by OWASP

  • Resistant to GPU/ASIC attacks

Development Dependencies (local.txt)#

Debugging Tools#

django-debug-toolbar==3.7.0     # Debug panel
django-extensions==3.2.1        # Shell_plus, runserver_plus, etc.
ipython==8.5.0                  # Enhanced Python shell
Werkzeug==2.2.2                 # Debugger for runserver_plus

Usage:

# Enhanced Django shell with auto-imports
python manage.py shell_plus

# Run server with Werkzeug debugger
python manage.py runserver_plus

Code Quality#

pylint==2.15.5                  # Linting
pylint-django==2.5.3            # Django-specific linting
black==22.10.0                  # Code formatting
isort==5.10.1                   # Import sorting
flake8==5.0.4                   # Style enforcement

Pre-commit hooks:

# Format code
black .

# Sort imports
isort .

# Check linting
pylint construbot/

Documentation#

Sphinx==6.2.1                   # Documentation generator
sphinx-rtd-theme==1.3.0         # ReadTheDocs theme
sphinx-autobuild==2021.3.14     # Auto-rebuild docs

Build docs:

cd docs
make html
make livehtml  # Auto-rebuild on changes

Testing Dependencies (test.txt)#

Test Frameworks#

pytest==7.2.0                   # Test framework
pytest-django==4.5.2            # Django integration
pytest-cov==4.0.0               # Coverage plugin
pytest-sugar==0.9.6             # Better test output
coverage==6.5.0                 # Coverage measurement
django-coverage-plugin==2.0.4   # Django template coverage

Why pytest over Django’s built-in?

  • Better fixtures and parametrization

  • Cleaner test syntax

  • Rich plugin ecosystem

  • Better output formatting

Test Utilities#

factory-boy==3.2.1              # Test data factories
faker==15.3.4                   # Fake data generation
freezegun==1.2.2                # Time mocking

Example usage:

# factories.py
import factory
from faker import Faker

fake = Faker()

class UserFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = User

    email = factory.Sequence(lambda n: f'user{n}@example.com')
    first_name = factory.LazyAttribute(lambda _: fake.first_name())

# In tests
user = UserFactory()

Production Dependencies (production.txt)#

-r base.txt                     # Includes all base dependencies

# Production-specific
gunicorn==20.1.0                # WSGI server
whitenoise==6.2.0               # Static file serving
sentry-sdk==1.11.1              # Error tracking
django-storages[boto3]==1.13.1  # S3 storage backend

Why these additions?

  • gunicorn: Production WSGI server (not runserver)

  • whitenoise: Efficient static file serving

  • sentry-sdk: Error tracking and monitoring

  • django-storages: Store media files in S3

Managing Dependencies#

Installing Dependencies#

Development:

pip install -r requirements/local.txt

Testing:

pip install -r requirements/test.txt

Production:

pip install -r requirements/production.txt

Adding New Dependencies#

Step 1: Add to appropriate .in file

# Edit requirements/base.in
echo "requests>=2.28.0" >> requirements/base.in

Step 2: Compile requirements

make pipcompile

This runs pip-compile on all .in files and generates .txt files with pinned versions.

Step 3: Install new requirements

pip install -r requirements/local.txt

Step 4: Commit both files

git add requirements/base.in requirements/base.txt
git commit -m "Add requests library"

Updating Dependencies#

Update specific package:

# Edit .in file with new version constraint
# Then recompile
make pipcompile

Update all packages:

# Install pip-tools
pip install pip-tools

# Upgrade all packages
pip-compile --upgrade requirements/base.in
pip-compile --upgrade requirements/local.in
pip-compile --upgrade requirements/test.in

Sync environment:

pip-sync requirements/local.txt

Removing Dependencies#

Step 1: Remove from .in file

# Edit requirements/base.in and remove the package

Step 2: Recompile

make pipcompile

Step 3: Uninstall package

pip uninstall <package-name>

# Or sync to ensure clean state
pip-sync requirements/local.txt

Pinning Versions#

Why Pin Versions?#

Benefits:

  • Reproducibility: Same environment every time

  • Stability: Avoid breaking changes

  • Security: Control when to upgrade

Approach:

  • Pin exact versions in .txt files (Django==3.2.19)

  • Use ranges in .in files (Django>=3.2,<4.0)

Version Constraints#

# Exact version (compiled .txt files)
Django==3.2.19

# Minimum version (.in files)
Django>=3.2

# Version range (.in files)
Django>=3.2,<4.0

# Compatible release (.in files)
Django~=3.2.0  # Equivalent to >=3.2.0,<3.3.0

Security Updates#

Check for vulnerabilities:

# Install safety
pip install safety

# Check dependencies
safety check

# Or use pip-audit
pip install pip-audit
pip-audit

Update vulnerable packages:

# Update constraint in .in file
# Recompile and test
make pipcompile
make test

Dependency Analysis#

View Dependency Tree#

# Install pipdeptree
pip install pipdeptree

# Show dependency tree
pipdeptree

# Show reversed dependencies
pipdeptree -r -p django

# Find conflicts
pipdeptree --warn

Find Unused Dependencies#

# Install pip-check
pip install pip-check

# Check for unused packages
pip-check

Check for Updates#

# List outdated packages
pip list --outdated

# Show latest versions
pip-outdated

Common Issues#

Dependency Conflicts#

Error: “Cannot install package-a and package-b”

Solution: Check dependency constraints

# Find conflicting requirements
pipdeptree --warn

# Update constraints in .in files
# Recompile
make pipcompile

Compilation Errors#

Error: “Could not find a version that satisfies the requirement”

Cause: Incompatible version constraints

Solution:

# Check .in files for version conflicts
# Relax constraints if needed
# Try compiling with --resolver=backtracking
pip-compile --resolver=backtracking requirements/base.in

Installation Failures#

Error: “Failed building wheel for psycopg2”

Cause: Missing PostgreSQL development headers

Solution:

# macOS
brew install postgresql

# Ubuntu/Debian
sudo apt-get install libpq-dev python3-dev

Error: “Failed building wheel for Pillow”

Cause: Missing image libraries

Solution:

# macOS
brew install libjpeg libtiff

# Ubuntu/Debian
sudo apt-get install libjpeg-dev libtiff-dev

Virtual Environment Issues#

Problem: Dependencies installed globally

Solution: Always use virtual environment

# Create venv
python3 -m venv venv

# Activate
source venv/bin/activate

# Verify
which python  # Should show venv path

Best Practices#

  1. Always use .in files - Don’t edit .txt files manually

  2. Pin exact versions in .txt - For reproducibility

  3. Use ranges in .in - Allow flexibility during compilation

  4. Regular updates - Monthly dependency review

  5. Security scanning - Use safety or pip-audit

  6. Test after updates - Run full test suite

  7. Document reasons - Add comments for unusual constraints

Example Workflow#

# 1. Add new dependency
echo "requests>=2.28.0" >> requirements/base.in

# 2. Compile requirements
make pipcompile

# 3. Review changes
git diff requirements/base.txt

# 4. Install
pip install -r requirements/local.txt

# 5. Test
make test

# 6. Commit
git add requirements/
git commit -m "Add requests library for API integration"

See Also#