988 lines
28 KiB
Markdown
988 lines
28 KiB
Markdown
|
|
# PostgreSQL Migration Guide
|
||
|
|
|
||
|
|
This document describes how to migrate the Tablet Management System from SQLite to PostgreSQL when the database grows beyond SQLite's capabilities.
|
||
|
|
|
||
|
|
## When to Migrate
|
||
|
|
|
||
|
|
Consider migrating to PostgreSQL when you encounter any of these scenarios:
|
||
|
|
|
||
|
|
| Metric | SQLite Limit | PostgreSQL | Migration Trigger |
|
||
|
|
|--------|--------------|------------|-------------------|
|
||
|
|
| Database Size | ~10GB max | Unlimited | >1GB |
|
||
|
|
| Concurrent Writers | 1 | Thousands | >50 simultaneous |
|
||
|
|
| Active Users | <100 | Millions | >200 |
|
||
|
|
| Transactions/min | <100 | 100K+ | >500 |
|
||
|
|
| Servers | Single machine | Cluster | Multiple servers |
|
||
|
|
| High Availability | No | Yes | Required |
|
||
|
|
| Backup Strategy | Manual | Automated | Automated needed |
|
||
|
|
|
||
|
|
## Architecture Overview
|
||
|
|
|
||
|
|
The migration uses a **Repository Pattern** to abstract the database layer, allowing both SQLite and PostgreSQL to work seamlessly.
|
||
|
|
|
||
|
|
```
|
||
|
|
project/
|
||
|
|
├── backend/
|
||
|
|
│ ├── config/
|
||
|
|
│ │ ├── __init__.py
|
||
|
|
│ │ ├── settings.py # Database configuration
|
||
|
|
│ │ └── database.py # Repository factory
|
||
|
|
│ ├── repositories/
|
||
|
|
│ │ ├── __init__.py
|
||
|
|
│ │ ├── base_repository.py # Abstract base classes
|
||
|
|
│ │ ├── sqlite_repo.py # SQLite implementation
|
||
|
|
│ │ └── postgres_repo.py # PostgreSQL implementation
|
||
|
|
│ └── app.py # Main application (unchanged)
|
||
|
|
├── migrations/ # Alembic migrations
|
||
|
|
│ └── versions/
|
||
|
|
│ └── initial_schema.py
|
||
|
|
├── scripts/
|
||
|
|
│ └── migrate_to_postgres.py # Migration script
|
||
|
|
└── docker-compose.yml # Optional Docker setup
|
||
|
|
```
|
||
|
|
|
||
|
|
## Step 1: Install Dependencies
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# For development
|
||
|
|
pip install psycopg2-binary sqlalchemy alembic
|
||
|
|
|
||
|
|
# For production (more efficient)
|
||
|
|
pip install psycopg2 sqlalchemy alembic
|
||
|
|
```
|
||
|
|
|
||
|
|
## Step 2: Set Up PostgreSQL
|
||
|
|
|
||
|
|
### Option A: Local Installation
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Ubuntu/Debian
|
||
|
|
sudo apt update
|
||
|
|
sudo apt install postgresql postgresql-contrib
|
||
|
|
|
||
|
|
# Create database and user
|
||
|
|
sudo -u postgres psql
|
||
|
|
```
|
||
|
|
|
||
|
|
In PostgreSQL shell:
|
||
|
|
```sql
|
||
|
|
CREATE DATABASE tablet_management;
|
||
|
|
CREATE USER tablet_user WITH PASSWORD 'your_secure_password';
|
||
|
|
GRANT ALL PRIVILEGES ON DATABASE tablet_management TO tablet_user;
|
||
|
|
ALTER USER tablet_user CREATEDB;
|
||
|
|
\q
|
||
|
|
```
|
||
|
|
|
||
|
|
### Option B: Docker (Recommended for Development)
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Start PostgreSQL container
|
||
|
|
docker run --name tablet-db -e POSTGRES_PASSWORD=your_password -e POSTGRES_USER=tablet_user -e POSTGRES_DB=tablet_management -p 5432:5432 -d postgres:16-alpine
|
||
|
|
|
||
|
|
# Or use docker-compose (see docker-compose.yml)
|
||
|
|
docker-compose up -d postgres
|
||
|
|
```
|
||
|
|
|
||
|
|
## Step 3: Configure Environment
|
||
|
|
|
||
|
|
Create a `.env` file:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Database configuration
|
||
|
|
DB_TYPE=postgres # or 'sqlite'
|
||
|
|
DB_URL=postgresql://tablet_user:your_password@localhost:5432/tablet_management
|
||
|
|
|
||
|
|
# For SQLite (fallback)
|
||
|
|
DB_PATH=tablets.db
|
||
|
|
```
|
||
|
|
|
||
|
|
Or set environment variables:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
export DB_TYPE=postgres
|
||
|
|
export DB_URL=postgresql://tablet_user:your_password@localhost:5432/tablet_management
|
||
|
|
```
|
||
|
|
|
||
|
|
## Step 4: Create Repository Abstraction
|
||
|
|
|
||
|
|
### Base Repository (Abstract Interface)
|
||
|
|
|
||
|
|
```python
|
||
|
|
# backend/repositories/base_repository.py
|
||
|
|
from abc import ABC, abstractmethod
|
||
|
|
from typing import Optional, List
|
||
|
|
from datetime import datetime
|
||
|
|
|
||
|
|
|
||
|
|
class BaseTabletRepository(ABC):
|
||
|
|
@abstractmethod
|
||
|
|
def get_by_id(self, tablet_id: int) -> Optional[dict]:
|
||
|
|
pass
|
||
|
|
|
||
|
|
@abstractmethod
|
||
|
|
def get_by_serial(self, serial: str) -> Optional[dict]:
|
||
|
|
pass
|
||
|
|
|
||
|
|
@abstractmethod
|
||
|
|
def get_all(self, status: Optional[str] = None) -> List[dict]:
|
||
|
|
pass
|
||
|
|
|
||
|
|
@abstractmethod
|
||
|
|
def add(self, brand: str, model: str, serial_number: str, notes: Optional[str] = None) -> dict:
|
||
|
|
pass
|
||
|
|
|
||
|
|
@abstractmethod
|
||
|
|
def update_status(self, tablet_id: int, status: str) -> bool:
|
||
|
|
pass
|
||
|
|
|
||
|
|
@abstractmethod
|
||
|
|
def delete(self, tablet_id: int) -> bool:
|
||
|
|
pass
|
||
|
|
|
||
|
|
|
||
|
|
class BaseUserRepository(ABC):
|
||
|
|
@abstractmethod
|
||
|
|
def get_by_id(self, user_id: int) -> Optional[dict]:
|
||
|
|
pass
|
||
|
|
|
||
|
|
@abstractmethod
|
||
|
|
def get_by_identification(self, identification: str) -> Optional[dict]:
|
||
|
|
pass
|
||
|
|
|
||
|
|
@abstractmethod
|
||
|
|
def get_all(self) -> List[dict]:
|
||
|
|
pass
|
||
|
|
|
||
|
|
@abstractmethod
|
||
|
|
def add(self, name: str, email: Optional[str], phone: Optional[str], identification: str) -> dict:
|
||
|
|
pass
|
||
|
|
|
||
|
|
|
||
|
|
class BaseLoanRepository(ABC):
|
||
|
|
@abstractmethod
|
||
|
|
def get_by_id(self, loan_id: int) -> Optional[dict]:
|
||
|
|
pass
|
||
|
|
|
||
|
|
@abstractmethod
|
||
|
|
def get_active_by_tablet(self, tablet_id: int) -> Optional[dict]:
|
||
|
|
pass
|
||
|
|
|
||
|
|
@abstractmethod
|
||
|
|
def get_by_user(self, user_id: int) -> List[dict]:
|
||
|
|
pass
|
||
|
|
|
||
|
|
@abstractmethod
|
||
|
|
def get_all(self, status: Optional[str] = None) -> List[dict]:
|
||
|
|
pass
|
||
|
|
|
||
|
|
@abstractmethod
|
||
|
|
def create(self, tablet_id: int, user_id: int) -> dict:
|
||
|
|
pass
|
||
|
|
|
||
|
|
@abstractmethod
|
||
|
|
def return_loan(self, loan_id: int) -> bool:
|
||
|
|
pass
|
||
|
|
|
||
|
|
|
||
|
|
class BaseNonLoanableDeviceRepository(ABC):
|
||
|
|
@abstractmethod
|
||
|
|
def get_by_id(self, device_id: int) -> Optional[dict]:
|
||
|
|
pass
|
||
|
|
|
||
|
|
@abstractmethod
|
||
|
|
def get_all(self) -> List[dict]:
|
||
|
|
pass
|
||
|
|
|
||
|
|
@abstractmethod
|
||
|
|
def add(self, brand: str, model: str, serial_number: str, device_type: str,
|
||
|
|
location: Optional[str] = None, notes: Optional[str] = None,
|
||
|
|
purchase_date: Optional[str] = None, purchase_cost: Optional[float] = None) -> dict:
|
||
|
|
pass
|
||
|
|
|
||
|
|
@abstractmethod
|
||
|
|
def update(self, device_id: int, **kwargs) -> bool:
|
||
|
|
pass
|
||
|
|
|
||
|
|
@abstractmethod
|
||
|
|
def delete(self, device_id: int) -> bool:
|
||
|
|
pass
|
||
|
|
```
|
||
|
|
|
||
|
|
### SQLite Implementation
|
||
|
|
|
||
|
|
```python
|
||
|
|
# backend/repositories/sqlite_repo.py
|
||
|
|
import sqlite3
|
||
|
|
from typing import Optional, List
|
||
|
|
from .base_repository import (
|
||
|
|
BaseTabletRepository, BaseUserRepository,
|
||
|
|
BaseLoanRepository, BaseNonLoanableDeviceRepository
|
||
|
|
)
|
||
|
|
|
||
|
|
|
||
|
|
class SQLiteTabletRepository(BaseTabletRepository):
|
||
|
|
def __init__(self, db_path: str = 'tablets.db'):
|
||
|
|
self.db_path = db_path
|
||
|
|
self._init_db()
|
||
|
|
|
||
|
|
def _init_db(self):
|
||
|
|
conn = sqlite3.connect(self.db_path)
|
||
|
|
cursor = conn.cursor()
|
||
|
|
cursor.execute('''
|
||
|
|
CREATE TABLE IF NOT EXISTS tablets (
|
||
|
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||
|
|
brand TEXT NOT NULL,
|
||
|
|
model TEXT NOT NULL,
|
||
|
|
serial_number TEXT UNIQUE NOT NULL,
|
||
|
|
status TEXT DEFAULT 'available',
|
||
|
|
notes TEXT
|
||
|
|
)
|
||
|
|
''')
|
||
|
|
conn.commit()
|
||
|
|
conn.close()
|
||
|
|
|
||
|
|
def get_by_id(self, tablet_id: int) -> Optional[dict]:
|
||
|
|
conn = sqlite3.connect(self.db_path)
|
||
|
|
conn.row_factory = sqlite3.Row
|
||
|
|
cursor = conn.cursor()
|
||
|
|
cursor.execute("SELECT * FROM tablets WHERE id = ?", (tablet_id,))
|
||
|
|
row = cursor.fetchone()
|
||
|
|
conn.close()
|
||
|
|
return dict(row) if row else None
|
||
|
|
|
||
|
|
def get_by_serial(self, serial: str) -> Optional[dict]:
|
||
|
|
conn = sqlite3.connect(self.db_path)
|
||
|
|
conn.row_factory = sqlite3.Row
|
||
|
|
cursor = conn.cursor()
|
||
|
|
cursor.execute("SELECT * FROM tablets WHERE serial_number = ?", (serial,))
|
||
|
|
row = cursor.fetchone()
|
||
|
|
conn.close()
|
||
|
|
return dict(row) if row else None
|
||
|
|
|
||
|
|
def get_all(self, status: Optional[str] = None) -> List[dict]:
|
||
|
|
conn = sqlite3.connect(self.db_path)
|
||
|
|
conn.row_factory = sqlite3.Row
|
||
|
|
cursor = conn.cursor()
|
||
|
|
query = "SELECT * FROM tablets"
|
||
|
|
params = ()
|
||
|
|
if status:
|
||
|
|
query += " WHERE status = ?"
|
||
|
|
params = (status,)
|
||
|
|
cursor.execute(query, params)
|
||
|
|
results = [dict(row) for row in cursor.fetchall()]
|
||
|
|
conn.close()
|
||
|
|
return results
|
||
|
|
|
||
|
|
def add(self, brand: str, model: str, serial_number: str, notes: Optional[str] = None) -> dict:
|
||
|
|
conn = sqlite3.connect(self.db_path)
|
||
|
|
conn.row_factory = sqlite3.Row
|
||
|
|
cursor = conn.cursor()
|
||
|
|
cursor.execute('''
|
||
|
|
INSERT INTO tablets (brand, model, serial_number, status, notes)
|
||
|
|
VALUES (?, ?, ?, 'available', ?)
|
||
|
|
''', (brand, model, serial_number, notes))
|
||
|
|
conn.commit()
|
||
|
|
tablet_id = cursor.lastrowid
|
||
|
|
cursor.execute("SELECT * FROM tablets WHERE id = ?", (tablet_id,))
|
||
|
|
row = cursor.fetchone()
|
||
|
|
conn.close()
|
||
|
|
return dict(row)
|
||
|
|
|
||
|
|
def update_status(self, tablet_id: int, status: str) -> bool:
|
||
|
|
conn = sqlite3.connect(self.db_path)
|
||
|
|
cursor = conn.cursor()
|
||
|
|
cursor.execute("UPDATE tablets SET status = ? WHERE id = ?", (status, tablet_id))
|
||
|
|
conn.commit()
|
||
|
|
changed = cursor.rowcount > 0
|
||
|
|
conn.close()
|
||
|
|
return changed
|
||
|
|
|
||
|
|
def delete(self, tablet_id: int) -> bool:
|
||
|
|
conn = sqlite3.connect(self.db_path)
|
||
|
|
cursor = conn.cursor()
|
||
|
|
cursor.execute("DELETE FROM tablets WHERE id = ?", (tablet_id,))
|
||
|
|
conn.commit()
|
||
|
|
deleted = cursor.rowcount > 0
|
||
|
|
conn.close()
|
||
|
|
return deleted
|
||
|
|
|
||
|
|
|
||
|
|
# Similar implementations for SQLiteUserRepository, SQLiteLoanRepository, etc.
|
||
|
|
```
|
||
|
|
|
||
|
|
### PostgreSQL Implementation
|
||
|
|
|
||
|
|
```python
|
||
|
|
# backend/repositories/postgres_repo.py
|
||
|
|
import psycopg2
|
||
|
|
from psycopg2 import sql
|
||
|
|
from psycopg2.extras import DictCursor
|
||
|
|
from typing import Optional, List
|
||
|
|
from .base_repository import (
|
||
|
|
BaseTabletRepository, BaseUserRepository,
|
||
|
|
BaseLoanRepository, BaseNonLoanableDeviceRepository
|
||
|
|
)
|
||
|
|
|
||
|
|
|
||
|
|
class PostgreSQLTabletRepository(BaseTabletRepository):
|
||
|
|
def __init__(self, connection_string: str):
|
||
|
|
self.connection_string = connection_string
|
||
|
|
self._init_db()
|
||
|
|
|
||
|
|
def _get_connection(self):
|
||
|
|
return psycopg2.connect(self.connection_string)
|
||
|
|
|
||
|
|
def _init_db(self):
|
||
|
|
conn = self._get_connection()
|
||
|
|
cursor = conn.cursor()
|
||
|
|
cursor.execute('''
|
||
|
|
CREATE TABLE IF NOT EXISTS tablets (
|
||
|
|
id SERIAL PRIMARY KEY,
|
||
|
|
brand VARCHAR(100) NOT NULL,
|
||
|
|
model VARCHAR(100) NOT NULL,
|
||
|
|
serial_number VARCHAR(50) UNIQUE NOT NULL,
|
||
|
|
status VARCHAR(20) DEFAULT 'available',
|
||
|
|
notes TEXT,
|
||
|
|
created_at TIMESTAMP DEFAULT NOW(),
|
||
|
|
updated_at TIMESTAMP DEFAULT NOW()
|
||
|
|
)
|
||
|
|
''')
|
||
|
|
cursor.execute('''
|
||
|
|
CREATE INDEX IF NOT EXISTS idx_tablets_serial
|
||
|
|
ON tablets(serial_number)
|
||
|
|
''')
|
||
|
|
cursor.execute('''
|
||
|
|
CREATE INDEX IF NOT EXISTS idx_tablets_status
|
||
|
|
ON tablets(status)
|
||
|
|
''')
|
||
|
|
conn.commit()
|
||
|
|
cursor.close()
|
||
|
|
conn.close()
|
||
|
|
|
||
|
|
def get_by_id(self, tablet_id: int) -> Optional[dict]:
|
||
|
|
conn = self._get_connection()
|
||
|
|
cursor = conn.cursor(cursor_factory=DictCursor)
|
||
|
|
cursor.execute("SELECT * FROM tablets WHERE id = %s", (tablet_id,))
|
||
|
|
row = cursor.fetchone()
|
||
|
|
conn.close()
|
||
|
|
return dict(row) if row else None
|
||
|
|
|
||
|
|
def get_by_serial(self, serial: str) -> Optional[dict]:
|
||
|
|
conn = self._get_connection()
|
||
|
|
cursor = conn.cursor(cursor_factory=DictCursor)
|
||
|
|
cursor.execute("SELECT * FROM tablets WHERE serial_number = %s", (serial,))
|
||
|
|
row = cursor.fetchone()
|
||
|
|
conn.close()
|
||
|
|
return dict(row) if row else None
|
||
|
|
|
||
|
|
def get_all(self, status: Optional[str] = None) -> List[dict]:
|
||
|
|
conn = self._get_connection()
|
||
|
|
cursor = conn.cursor(cursor_factory=DictCursor)
|
||
|
|
query = "SELECT * FROM tablets"
|
||
|
|
params = ()
|
||
|
|
if status:
|
||
|
|
query += " WHERE status = %s"
|
||
|
|
params = (status,)
|
||
|
|
cursor.execute(query, params)
|
||
|
|
results = [dict(row) for row in cursor.fetchall()]
|
||
|
|
conn.close()
|
||
|
|
return results
|
||
|
|
|
||
|
|
def add(self, brand: str, model: str, serial_number: str, notes: Optional[str] = None) -> dict:
|
||
|
|
conn = self._get_connection()
|
||
|
|
cursor = conn.cursor(cursor_factory=DictCursor)
|
||
|
|
cursor.execute('''
|
||
|
|
INSERT INTO tablets (brand, model, serial_number, status, notes)
|
||
|
|
VALUES (%s, %s, %s, 'available', %s)
|
||
|
|
RETURNING *
|
||
|
|
''', (brand, model, serial_number, notes))
|
||
|
|
row = cursor.fetchone()
|
||
|
|
conn.commit()
|
||
|
|
conn.close()
|
||
|
|
return dict(row)
|
||
|
|
|
||
|
|
def update_status(self, tablet_id: int, status: str) -> bool:
|
||
|
|
conn = self._get_connection()
|
||
|
|
cursor = conn.cursor()
|
||
|
|
cursor.execute(
|
||
|
|
"UPDATE tablets SET status = %s, updated_at = NOW() WHERE id = %s",
|
||
|
|
(status, tablet_id)
|
||
|
|
)
|
||
|
|
conn.commit()
|
||
|
|
changed = cursor.rowcount > 0
|
||
|
|
conn.close()
|
||
|
|
return changed
|
||
|
|
|
||
|
|
def delete(self, tablet_id: int) -> bool:
|
||
|
|
conn = self._get_connection()
|
||
|
|
cursor = conn.cursor()
|
||
|
|
cursor.execute("DELETE FROM tablets WHERE id = %s", (tablet_id,))
|
||
|
|
conn.commit()
|
||
|
|
deleted = cursor.rowcount > 0
|
||
|
|
conn.close()
|
||
|
|
return deleted
|
||
|
|
|
||
|
|
|
||
|
|
# Similar implementations for PostgreSQLUserRepository, PostgreSQLLoanRepository, etc.
|
||
|
|
```
|
||
|
|
|
||
|
|
## Step 5: Create Repository Factory
|
||
|
|
|
||
|
|
```python
|
||
|
|
# backend/config/database.py
|
||
|
|
import os
|
||
|
|
from backend.repositories.sqlite_repo import (
|
||
|
|
SQLiteTabletRepository, SQLiteUserRepository,
|
||
|
|
SQLiteLoanRepository, SQLiteNonLoanableDeviceRepository
|
||
|
|
)
|
||
|
|
from backend.repositories.postgres_repo import (
|
||
|
|
PostgreSQLTabletRepository, PostgreSQLUserRepository,
|
||
|
|
PostgreSQLLoanRepository, PostgreSQLNonLoanableDeviceRepository
|
||
|
|
)
|
||
|
|
from backend.repositories.base_repository import (
|
||
|
|
BaseTabletRepository, BaseUserRepository,
|
||
|
|
BaseLoanRepository, BaseNonLoanableDeviceRepository
|
||
|
|
)
|
||
|
|
|
||
|
|
|
||
|
|
class DatabaseConfig:
|
||
|
|
def __init__(self):
|
||
|
|
self.db_type = os.getenv('DB_TYPE', 'sqlite')
|
||
|
|
self.db_url = os.getenv('DB_URL', '')
|
||
|
|
self.db_path = os.getenv('DB_PATH', 'tablets.db')
|
||
|
|
|
||
|
|
@property
|
||
|
|
def is_postgres(self) -> bool:
|
||
|
|
return self.db_type == 'postgres'
|
||
|
|
|
||
|
|
|
||
|
|
def get_tablet_repository() -> BaseTabletRepository:
|
||
|
|
config = DatabaseConfig()
|
||
|
|
if config.is_postgres:
|
||
|
|
return PostgreSQLTabletRepository(config.db_url)
|
||
|
|
else:
|
||
|
|
return SQLiteTabletRepository(config.db_path)
|
||
|
|
|
||
|
|
|
||
|
|
def get_user_repository() -> BaseUserRepository:
|
||
|
|
config = DatabaseConfig()
|
||
|
|
if config.is_postgres:
|
||
|
|
return PostgreSQLUserRepository(config.db_url)
|
||
|
|
else:
|
||
|
|
return SQLiteUserRepository(config.db_path)
|
||
|
|
|
||
|
|
|
||
|
|
def get_loan_repository() -> BaseLoanRepository:
|
||
|
|
config = DatabaseConfig()
|
||
|
|
if config.is_postgres:
|
||
|
|
return PostgreSQLLoanRepository(config.db_url)
|
||
|
|
else:
|
||
|
|
return SQLiteLoanRepository(config.db_path)
|
||
|
|
|
||
|
|
|
||
|
|
def get_non_loanable_device_repository() -> BaseNonLoanableDeviceRepository:
|
||
|
|
config = DatabaseConfig()
|
||
|
|
if config.is_postgres:
|
||
|
|
return PostgreSQLNonLoanableDeviceRepository(config.db_url)
|
||
|
|
else:
|
||
|
|
return SQLiteNonLoanableDeviceRepository(config.db_path)
|
||
|
|
```
|
||
|
|
|
||
|
|
## Step 6: Update Application to Use Repositories
|
||
|
|
|
||
|
|
Modify your application to use the repository pattern:
|
||
|
|
|
||
|
|
```python
|
||
|
|
# In your app.py or service layer
|
||
|
|
from backend.config.database import (
|
||
|
|
get_tablet_repository, get_user_repository,
|
||
|
|
get_loan_repository, get_non_loanable_device_repository
|
||
|
|
)
|
||
|
|
|
||
|
|
# Instead of direct SQLite calls:
|
||
|
|
tablet_repo = get_tablet_repository()
|
||
|
|
user_repo = get_user_repository()
|
||
|
|
loan_repo = get_loan_repository()
|
||
|
|
|
||
|
|
# Example: Loan a tablet
|
||
|
|
def loan_tablet(tablet_id: int, user_id: int):
|
||
|
|
# Get repositories
|
||
|
|
tablet_repo = get_tablet_repository()
|
||
|
|
user_repo = get_user_repository()
|
||
|
|
loan_repo = get_loan_repository()
|
||
|
|
|
||
|
|
# Validate
|
||
|
|
tablet = tablet_repo.get_by_id(tablet_id)
|
||
|
|
if not tablet:
|
||
|
|
raise ValueError("Tablet not found")
|
||
|
|
|
||
|
|
if tablet['status'] != 'available':
|
||
|
|
raise ValueError("Tablet not available")
|
||
|
|
|
||
|
|
user = user_repo.get_by_id(user_id)
|
||
|
|
if not user:
|
||
|
|
raise ValueError("User not found")
|
||
|
|
|
||
|
|
# Check for active loan
|
||
|
|
active_loan = loan_repo.get_active_by_tablet(tablet_id)
|
||
|
|
if active_loan:
|
||
|
|
raise ValueError("Tablet already loaned")
|
||
|
|
|
||
|
|
# Create loan
|
||
|
|
loan = loan_repo.create(tablet_id, user_id)
|
||
|
|
|
||
|
|
# Update tablet status
|
||
|
|
tablet_repo.update_status(tablet_id, 'loaned')
|
||
|
|
|
||
|
|
return loan
|
||
|
|
```
|
||
|
|
|
||
|
|
## Step 7: Create Migration Script
|
||
|
|
|
||
|
|
```python
|
||
|
|
# scripts/migrate_to_postgres.py
|
||
|
|
#!/usr/bin/env python3
|
||
|
|
"""
|
||
|
|
Migration script from SQLite to PostgreSQL
|
||
|
|
"""
|
||
|
|
import sqlite3
|
||
|
|
import psycopg2
|
||
|
|
from psycopg2.extras import DictCursor
|
||
|
|
import argparse
|
||
|
|
from tqdm import tqdm
|
||
|
|
import os
|
||
|
|
|
||
|
|
|
||
|
|
def create_postgres_tables(conn):
|
||
|
|
"""Create all tables in PostgreSQL"""
|
||
|
|
cursor = conn.cursor()
|
||
|
|
|
||
|
|
# Tablets
|
||
|
|
cursor.execute('''
|
||
|
|
CREATE TABLE IF NOT EXISTS tablets (
|
||
|
|
id SERIAL PRIMARY KEY,
|
||
|
|
brand VARCHAR(100) NOT NULL,
|
||
|
|
model VARCHAR(100) NOT NULL,
|
||
|
|
serial_number VARCHAR(50) UNIQUE NOT NULL,
|
||
|
|
status VARCHAR(20) DEFAULT 'available',
|
||
|
|
notes TEXT,
|
||
|
|
created_at TIMESTAMP DEFAULT NOW(),
|
||
|
|
updated_at TIMESTAMP DEFAULT NOW()
|
||
|
|
)
|
||
|
|
''')
|
||
|
|
|
||
|
|
# Users
|
||
|
|
cursor.execute('''
|
||
|
|
CREATE TABLE IF NOT EXISTS users (
|
||
|
|
id SERIAL PRIMARY KEY,
|
||
|
|
name VARCHAR(100) NOT NULL,
|
||
|
|
email VARCHAR(255),
|
||
|
|
phone VARCHAR(20),
|
||
|
|
identification VARCHAR(50) UNIQUE NOT NULL,
|
||
|
|
created_at TIMESTAMP DEFAULT NOW(),
|
||
|
|
updated_at TIMESTAMP DEFAULT NOW()
|
||
|
|
)
|
||
|
|
''')
|
||
|
|
|
||
|
|
# Loans
|
||
|
|
cursor.execute('''
|
||
|
|
CREATE TABLE IF NOT EXISTS loans (
|
||
|
|
id SERIAL PRIMARY KEY,
|
||
|
|
tablet_id INTEGER NOT NULL REFERENCES tablets(id) ON DELETE RESTRICT,
|
||
|
|
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE RESTRICT,
|
||
|
|
loan_date TIMESTAMP NOT NULL DEFAULT NOW(),
|
||
|
|
return_date TIMESTAMP,
|
||
|
|
status VARCHAR(20) DEFAULT 'active',
|
||
|
|
created_at TIMESTAMP DEFAULT NOW()
|
||
|
|
)
|
||
|
|
''')
|
||
|
|
|
||
|
|
# Non-loanable devices
|
||
|
|
cursor.execute('''
|
||
|
|
CREATE TABLE IF NOT EXISTS non_loanable_devices (
|
||
|
|
id SERIAL PRIMARY KEY,
|
||
|
|
brand VARCHAR(100) NOT NULL,
|
||
|
|
model VARCHAR(100) NOT NULL,
|
||
|
|
serial_number VARCHAR(50) UNIQUE NOT NULL,
|
||
|
|
device_type VARCHAR(50) NOT NULL,
|
||
|
|
location VARCHAR(100),
|
||
|
|
status VARCHAR(20) DEFAULT 'available',
|
||
|
|
notes TEXT,
|
||
|
|
purchase_date DATE,
|
||
|
|
purchase_cost DECIMAL(10,2),
|
||
|
|
created_at TIMESTAMP DEFAULT NOW(),
|
||
|
|
updated_at TIMESTAMP DEFAULT NOW()
|
||
|
|
)
|
||
|
|
''')
|
||
|
|
|
||
|
|
# Indexes for performance
|
||
|
|
cursor.execute('CREATE INDEX IF NOT EXISTS idx_tablets_serial ON tablets(serial_number)')
|
||
|
|
cursor.execute('CREATE INDEX IF NOT EXISTS idx_tablets_status ON tablets(status)')
|
||
|
|
cursor.execute('CREATE INDEX IF NOT EXISTS idx_tablets_brand ON tablets(brand)')
|
||
|
|
cursor.execute('CREATE INDEX IF NOT EXISTS idx_users_identification ON users(identification)')
|
||
|
|
cursor.execute('CREATE INDEX IF NOT EXISTS idx_loans_tablet ON loans(tablet_id)')
|
||
|
|
cursor.execute('CREATE INDEX IF NOT EXISTS idx_loans_user ON loans(user_id)')
|
||
|
|
cursor.execute('CREATE INDEX IF NOT EXISTS idx_loans_status ON loans(status)')
|
||
|
|
|
||
|
|
conn.commit()
|
||
|
|
|
||
|
|
|
||
|
|
def migrate_table(conn_sqlite, conn_pg, table_name: str, pg_create_table: str):
|
||
|
|
"""Generic migration for a table"""
|
||
|
|
cursor_sqlite = conn_sqlite.cursor()
|
||
|
|
cursor_pg = conn_pg.cursor()
|
||
|
|
|
||
|
|
# Get all data from SQLite
|
||
|
|
cursor_sqlite.execute(f"SELECT * FROM {table_name}")
|
||
|
|
rows = cursor_sqlite.fetchall()
|
||
|
|
|
||
|
|
if not rows:
|
||
|
|
print(f"No data to migrate for {table_name}")
|
||
|
|
return
|
||
|
|
|
||
|
|
# Get column names
|
||
|
|
column_names = [desc[0] for desc in cursor_sqlite.description]
|
||
|
|
|
||
|
|
# Prepare INSERT statement
|
||
|
|
columns = ', '.join(column_names)
|
||
|
|
placeholders = ', '.join(['%s'] * len(column_names))
|
||
|
|
insert_sql = f"INSERT INTO {table_name} ({columns}) VALUES ({placeholders}) ON CONFLICT DO NOTHING"
|
||
|
|
|
||
|
|
# Migrate data
|
||
|
|
for row in tqdm(rows, desc=f"Migrating {table_name}"):
|
||
|
|
cursor_pg.execute(insert_sql, row)
|
||
|
|
|
||
|
|
conn_pg.commit()
|
||
|
|
print(f"✓ Migrated {len(rows)} rows from {table_name}")
|
||
|
|
|
||
|
|
|
||
|
|
def migrate_all(sqlite_path: str, pg_url: str):
|
||
|
|
"""Migrate all data from SQLite to PostgreSQL"""
|
||
|
|
print("Starting migration from SQLite to PostgreSQL...")
|
||
|
|
|
||
|
|
# Connect to SQLite
|
||
|
|
conn_sqlite = sqlite3.connect(sqlite_path)
|
||
|
|
|
||
|
|
# Connect to PostgreSQL
|
||
|
|
conn_pg = psycopg2.connect(pg_url)
|
||
|
|
|
||
|
|
try:
|
||
|
|
# Create tables
|
||
|
|
print("Creating PostgreSQL tables...")
|
||
|
|
create_postgres_tables(conn_pg)
|
||
|
|
|
||
|
|
# Migrate each table
|
||
|
|
migrate_table(conn_sqlite, conn_pg, 'tablets', '')
|
||
|
|
migrate_table(conn_sqlite, conn_pg, 'users', '')
|
||
|
|
migrate_table(conn_sqlite, conn_pg, 'loans', '')
|
||
|
|
migrate_table(conn_sqlite, conn_pg, 'non_loanable_devices', '')
|
||
|
|
|
||
|
|
print("\n✓ Migration completed successfully!")
|
||
|
|
print(f" SQLite: {sqlite_path}")
|
||
|
|
print(f" PostgreSQL: {pg_url}")
|
||
|
|
|
||
|
|
except Exception as e:
|
||
|
|
conn_pg.rollback()
|
||
|
|
print(f"\n✗ Migration failed: {e}")
|
||
|
|
raise
|
||
|
|
finally:
|
||
|
|
conn_sqlite.close()
|
||
|
|
conn_pg.close()
|
||
|
|
|
||
|
|
|
||
|
|
if __name__ == '__main__':
|
||
|
|
parser = argparse.ArgumentParser(description='Migrate from SQLite to PostgreSQL')
|
||
|
|
parser.add_argument('--sqlite', default='tablets.db', help='SQLite database path')
|
||
|
|
parser.add_argument('--postgres', required=True, help='PostgreSQL connection URL')
|
||
|
|
args = parser.parse_args()
|
||
|
|
|
||
|
|
migrate_all(args.sqlite, args.postgres)
|
||
|
|
```
|
||
|
|
|
||
|
|
## Step 8: Run Migration
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Test the migration first (dry run)
|
||
|
|
python scripts/migrate_to_postgres.py --sqlite tablets.db --postgres postgresql://tablet_user:password@localhost:5432/tablet_management_test
|
||
|
|
|
||
|
|
# Verify data in test database
|
||
|
|
psql -U tablet_user -d tablet_management_test -c "SELECT COUNT(*) FROM tablets;"
|
||
|
|
|
||
|
|
# When ready, migrate to production
|
||
|
|
python scripts/migrate_to_postgres.py --sqlite tablets.db --postgres postgresql://tablet_user:password@localhost:5432/tablet_management
|
||
|
|
```
|
||
|
|
|
||
|
|
## Step 9: Switch to PostgreSQL
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Update environment variables
|
||
|
|
export DB_TYPE=postgres
|
||
|
|
export DB_URL=postgresql://tablet_user:password@localhost:5432/tablet_management
|
||
|
|
|
||
|
|
# Restart application
|
||
|
|
python app.py
|
||
|
|
```
|
||
|
|
|
||
|
|
## Step 10: Verify and Monitor
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Check application logs for errors
|
||
|
|
# Monitor database connections
|
||
|
|
psql -U tablet_user -d tablet_management -c "SELECT COUNT(*) FROM tablets;"
|
||
|
|
|
||
|
|
# Check active connections
|
||
|
|
psql -U postgres -c "SELECT * FROM pg_stat_activity WHERE datname = 'tablet_management';"
|
||
|
|
```
|
||
|
|
|
||
|
|
## Rollback Plan
|
||
|
|
|
||
|
|
If something goes wrong:
|
||
|
|
|
||
|
|
1. **Immediate rollback:**
|
||
|
|
```bash
|
||
|
|
# Switch back to SQLite
|
||
|
|
export DB_TYPE=sqlite
|
||
|
|
export DB_PATH=tablets.db
|
||
|
|
python app.py
|
||
|
|
```
|
||
|
|
|
||
|
|
2. **Data verification:**
|
||
|
|
```bash
|
||
|
|
# Compare counts
|
||
|
|
sqlite3 tablets.db "SELECT COUNT(*) FROM tablets;"
|
||
|
|
psql -U tablet_user -d tablet_management -c "SELECT COUNT(*) FROM tablets;"
|
||
|
|
```
|
||
|
|
|
||
|
|
3. **Backup PostgreSQL data:**
|
||
|
|
```bash
|
||
|
|
pg_dump -U tablet_user -d tablet_management > postgres_backup_$(date +%Y%m%d).sql
|
||
|
|
```
|
||
|
|
|
||
|
|
## Docker Compose (Optional)
|
||
|
|
|
||
|
|
For easy deployment with Docker:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
# docker-compose.yml
|
||
|
|
version: '3.8'
|
||
|
|
|
||
|
|
services:
|
||
|
|
postgres:
|
||
|
|
image: postgres:16-alpine
|
||
|
|
container_name: tablet_db
|
||
|
|
environment:
|
||
|
|
POSTGRES_DB: tablet_management
|
||
|
|
POSTGRES_USER: tablet_user
|
||
|
|
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-changeme}
|
||
|
|
ports:
|
||
|
|
- "5432:5432"
|
||
|
|
volumes:
|
||
|
|
- postgres_data:/var/lib/postgresql/data
|
||
|
|
healthcheck:
|
||
|
|
test: ["CMD-SHELL", "pg_isready -U tablet_user -d tablet_management"]
|
||
|
|
interval: 5s
|
||
|
|
timeout: 5s
|
||
|
|
retries: 5
|
||
|
|
restart: unless-stopped
|
||
|
|
|
||
|
|
app:
|
||
|
|
build: .
|
||
|
|
container_name: tablet_app
|
||
|
|
environment:
|
||
|
|
DB_TYPE: postgres
|
||
|
|
DB_URL: postgresql://tablet_user:${POSTGRES_PASSWORD:-changeme}@postgres:5432/tablet_management
|
||
|
|
ports:
|
||
|
|
- "5000:5000"
|
||
|
|
depends_on:
|
||
|
|
postgres:
|
||
|
|
condition: service_healthy
|
||
|
|
restart: unless-stopped
|
||
|
|
|
||
|
|
volumes:
|
||
|
|
postgres_data:
|
||
|
|
```
|
||
|
|
|
||
|
|
Start with Docker:
|
||
|
|
```bash
|
||
|
|
docker-compose up -d
|
||
|
|
```
|
||
|
|
|
||
|
|
## Benefits of PostgreSQL
|
||
|
|
|
||
|
|
### Performance
|
||
|
|
- **Concurrency:** Multiple writers simultaneously (no lock contention)
|
||
|
|
- **Indexing:** Advanced index types (B-tree, Hash, GiST, GIN, BRIN)
|
||
|
|
- **Query Optimization:** Advanced query planner
|
||
|
|
- **Connection Pooling:** Built-in support
|
||
|
|
|
||
|
|
### Scalability
|
||
|
|
- **Vertical:** Handles large datasets efficiently
|
||
|
|
- **Horizontal:** Read replicas, partitioning, sharding
|
||
|
|
- **Connections:** Supports thousands of concurrent connections
|
||
|
|
|
||
|
|
### Reliability
|
||
|
|
- **ACID Compliance:** Full transaction support
|
||
|
|
- **Point-in-Time Recovery:** Restore to any moment
|
||
|
|
- **Replication:** Master-slave, synchronous, asynchronous
|
||
|
|
- **Backups:** `pg_dump`, `pg_basebackup`, continuous archiving
|
||
|
|
|
||
|
|
### Security
|
||
|
|
- **Authentication:** Multiple methods (password, MD5, SCRAM, LDAP, Kerberos)
|
||
|
|
- **Authorization:** Role-based access control (RBAC)
|
||
|
|
- **Row-Level Security:** Policies for fine-grained access
|
||
|
|
- **Encryption:** SSL, at-rest encryption
|
||
|
|
|
||
|
|
### Features
|
||
|
|
- **JSON Support:** Native JSON/JSONB data type
|
||
|
|
- **Full-Text Search:** Advanced text search capabilities
|
||
|
|
- **Arrays:** Store arrays of values
|
||
|
|
- **Custom Types:** Create your own data types
|
||
|
|
- **Triggers:** Automatic actions on events
|
||
|
|
- **Stored Procedures:** Server-side functions
|
||
|
|
|
||
|
|
## Monitoring PostgreSQL
|
||
|
|
|
||
|
|
### Basic Queries
|
||
|
|
|
||
|
|
```sql
|
||
|
|
-- Active connections
|
||
|
|
SELECT * FROM pg_stat_activity WHERE datname = 'tablet_management';
|
||
|
|
|
||
|
|
-- Table sizes
|
||
|
|
SELECT table_name, pg_size_pretty(pg_total_relation_size(table_name))
|
||
|
|
FROM information_schema.tables WHERE table_schema = 'public';
|
||
|
|
|
||
|
|
-- Index usage
|
||
|
|
SELECT indexrelname, idx_scan, idx_tup_read, idx_tup_fetch
|
||
|
|
FROM pg_stat_user_indexes;
|
||
|
|
|
||
|
|
-- Slow queries (requires pg_stat_statements extension)
|
||
|
|
SELECT query, total_time, calls, mean_time
|
||
|
|
FROM pg_stat_statements ORDER BY mean_time DESC LIMIT 10;
|
||
|
|
```
|
||
|
|
|
||
|
|
### Enable pg_stat_statements
|
||
|
|
|
||
|
|
```sql
|
||
|
|
-- In PostgreSQL
|
||
|
|
CREATE EXTENSION pg_stat_statements;
|
||
|
|
|
||
|
|
-- Then in postgresql.conf
|
||
|
|
shared_preload_libraries = 'pg_stat_statements'
|
||
|
|
pg_stat_statements.track = all
|
||
|
|
```
|
||
|
|
|
||
|
|
## Maintenance Tasks
|
||
|
|
|
||
|
|
### Regular Maintenance
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Vacuum (reclaim space, update statistics)
|
||
|
|
vacuumdb -U tablet_user -d tablet_management --analyze
|
||
|
|
|
||
|
|
# Reindex (rebuild indexes)
|
||
|
|
reindexdb -U tablet_user -d tablet_management
|
||
|
|
```
|
||
|
|
|
||
|
|
### Backup Strategy
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Daily backup
|
||
|
|
pg_dump -U tablet_user -d tablet_management > /backups/tablet_management_$(date +%Y%m%d).sql
|
||
|
|
|
||
|
|
# Compressed backup
|
||
|
|
pg_dump -U tablet_user -d tablet_management | gzip > /backups/tablet_management_$(date +%Y%m%d).sql.gz
|
||
|
|
|
||
|
|
# Continuous archiving (WAL)
|
||
|
|
# In postgresql.conf:
|
||
|
|
wal_level = replica
|
||
|
|
archive_mode = on
|
||
|
|
archive_command = 'test ! -f /backups/wal/%f && cp %p /backups/wal/%f'
|
||
|
|
```
|
||
|
|
|
||
|
|
## Performance Optimization
|
||
|
|
|
||
|
|
### Configuration Tuning
|
||
|
|
|
||
|
|
```conf
|
||
|
|
# postgresql.conf recommendations
|
||
|
|
shared_buffers = 4GB # 25% of total RAM
|
||
|
|
work_mem = 16MB # For complex sorts
|
||
|
|
maintenance_work_mem = 512MB # For VACUUM, index creation
|
||
|
|
effective_cache_size = 12GB # 75% of total RAM
|
||
|
|
random_page_cost = 1.1 # SSD: 1.1, HDD: 4.0
|
||
|
|
max_worker_processes = 8 # Number of CPU cores
|
||
|
|
max_parallel_workers_per_gather = 4 # Parallel query workers
|
||
|
|
max_connections = 200 # Expected max connections
|
||
|
|
```
|
||
|
|
|
||
|
|
### Index Optimization
|
||
|
|
|
||
|
|
```sql
|
||
|
|
-- Add indexes for common queries
|
||
|
|
CREATE INDEX idx_loans_user_status ON loans(user_id, status);
|
||
|
|
CREATE INDEX idx_loans_date_range ON loans(loan_date, return_date);
|
||
|
|
|
||
|
|
-- Partial index for active loans
|
||
|
|
CREATE INDEX idx_loans_active ON loans(tablet_id) WHERE status = 'active';
|
||
|
|
|
||
|
|
-- Composite index for user loans
|
||
|
|
CREATE INDEX idx_loans_user_tablet ON loans(user_id, tablet_id);
|
||
|
|
```
|
||
|
|
|
||
|
|
## Troubleshooting
|
||
|
|
|
||
|
|
### Common Issues
|
||
|
|
|
||
|
|
**Connection refused:**
|
||
|
|
```bash
|
||
|
|
# Check if PostgreSQL is running
|
||
|
|
sudo systemctl status postgresql
|
||
|
|
|
||
|
|
# Check port
|
||
|
|
netstat -tuln | grep 5432
|
||
|
|
```
|
||
|
|
|
||
|
|
**Authentication failed:**
|
||
|
|
```bash
|
||
|
|
# Verify user and password
|
||
|
|
psql -U tablet_user -d tablet_management -h localhost
|
||
|
|
|
||
|
|
# Check pg_hba.conf
|
||
|
|
sudo nano /etc/postgresql/16/main/pg_hba.conf
|
||
|
|
```
|
||
|
|
|
||
|
|
**Database does not exist:**
|
||
|
|
```bash
|
||
|
|
# Create database
|
||
|
|
createdb -U postgres tablet_management
|
||
|
|
```
|
||
|
|
|
||
|
|
**Permission denied:**
|
||
|
|
```sql
|
||
|
|
-- Grant permissions
|
||
|
|
GRANT ALL PRIVILEGES ON DATABASE tablet_management TO tablet_user;
|
||
|
|
GRANT ALL ON SCHEMA public TO tablet_user;
|
||
|
|
```
|
||
|
|
|
||
|
|
### Logs
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# PostgreSQL logs
|
||
|
|
sudo tail -f /var/log/postgresql/postgresql-16-main.log
|
||
|
|
|
||
|
|
# Application logs
|
||
|
|
journalctl -u tablet_management -f
|
||
|
|
```
|
||
|
|
|
||
|
|
## Conclusion
|
||
|
|
|
||
|
|
Migrating from SQLite to PostgreSQL provides:
|
||
|
|
- Better performance at scale
|
||
|
|
- True concurrency
|
||
|
|
- Enhanced reliability
|
||
|
|
- Advanced features
|
||
|
|
- Production-ready infrastructure
|
||
|
|
|
||
|
|
The repository pattern ensures a smooth transition with minimal code changes, and the migration script automates the data transfer process.
|