Student data import implementation: added tables, migration, import script, tests, and documentation.
This commit is contained in:
ijuanes
parent
9662f98169
commit
2959cd5299
19 changed files with 3186 additions and 79 deletions
392
IMPLEMENTATION_SUMMARY.md
Normal file
392
IMPLEMENTATION_SUMMARY.md
Normal file
|
|
@ -0,0 +1,392 @@
|
|||
# Student Data Import Implementation Summary
|
||||
|
||||
## Overview
|
||||
|
||||
This document summarizes the implementation of the student data import functionality for the GestionTablets project. The implementation includes a new database schema for students, CSV import scripts, updated application code, and new frontend templates.
|
||||
|
||||
## Files Created
|
||||
|
||||
### 1. Database Schema Specification
|
||||
**File:** `docs/DATABASE_SCHEMA_SPECIFICATION.md`
|
||||
|
||||
- Complete SQLite database schema specification
|
||||
- Detailed field descriptions and data types
|
||||
- Index definitions for optimal query performance
|
||||
- Data import specifications and validation rules
|
||||
- API endpoint definitions
|
||||
- Migration path from existing database
|
||||
|
||||
### 2. CSV Import Script
|
||||
**File:** `scripts/import_students.py`
|
||||
|
||||
Features:
|
||||
- Reads CSV files encoded in ISO-8859-1 (Latin-1)
|
||||
- Converts encoding to UTF-8 automatically
|
||||
- Splits "Apellidos, Nombre" field into separate "last_name" and "first_name" fields
|
||||
- Converts date format from DD/MM/YYYY to YYYY-MM-DD
|
||||
- Discards the "Estudio" field as per requirements
|
||||
- Validates all required fields (CIAL code, first name, last name)
|
||||
- Handles duplicates gracefully
|
||||
- Provides detailed statistics and error reporting
|
||||
- Supports dry-run mode for testing
|
||||
|
||||
Usage:
|
||||
```bash
|
||||
# Dry run (validate without importing)
|
||||
python3 scripts/import_students.py Datos_programa.csv --dry-run --verbose
|
||||
|
||||
# Actual import
|
||||
python3 scripts/import_students.py Datos_programa.csv --verbose
|
||||
```
|
||||
|
||||
### 3. Database Migration Script
|
||||
**File:** `scripts/migrate_database.py`
|
||||
|
||||
Features:
|
||||
- Creates backup of existing database before migration
|
||||
- Creates new `students` table with all required fields and indexes
|
||||
- Adds `assigned_to_student` column to `tablets` and `non_loanable_devices` tables
|
||||
- Adds `student_id` column to `loans` table
|
||||
- Adds timestamp columns (`created_at`, `updated_at`) to existing tables
|
||||
- Verifies migration success
|
||||
- Supports dry-run mode
|
||||
|
||||
Usage:
|
||||
```bash
|
||||
# Dry run
|
||||
python3 scripts/migrate_database.py --dry-run
|
||||
|
||||
# Actual migration
|
||||
python3 scripts/migrate_database.py
|
||||
```
|
||||
|
||||
## Files Modified
|
||||
|
||||
### 1. Application Code
|
||||
**File:** `app.py`
|
||||
|
||||
Changes:
|
||||
- Added `students` table creation in `init_db()` function
|
||||
- Added indexes for students table
|
||||
- Updated `non_loanable_devices` table to include `assigned_to_student` field
|
||||
- Added new routes:
|
||||
- `/students` - List all students with search, filter, and pagination
|
||||
- `/add_student` - Add new student (GET/POST)
|
||||
- `/student/<id>` - View student details
|
||||
- `/edit_student/<id>` - Edit student (GET/POST)
|
||||
- `/delete_student/<id>` - Delete student
|
||||
- `/import_students` - CSV import interface (GET/POST)
|
||||
- Updated `loan_tablet` route to support optional student association
|
||||
- Added student selection to loan form
|
||||
|
||||
### 2. Base Template
|
||||
**File:** `templates/base.html`
|
||||
|
||||
Changes:
|
||||
- Added "Students" link to navigation menu
|
||||
|
||||
### 3. Loan Tablet Template
|
||||
**File:** `templates/loan_tablet.html`
|
||||
|
||||
Changes:
|
||||
- Updated to use Bootstrap 5 classes (matching other templates)
|
||||
- Added student selection dropdown with search functionality
|
||||
- Improved form layout and styling
|
||||
|
||||
### 4. Requirements
|
||||
**File:** `requirements.txt`
|
||||
|
||||
Updated with all necessary dependencies:
|
||||
- Flask==2.3.3
|
||||
- Flask-Babel==2.0.0
|
||||
- python-dotenv==1.0.0
|
||||
- pytest==7.4.0
|
||||
- pytest-cov==4.1.0
|
||||
- waitress==2.1.2
|
||||
|
||||
## Files Created (Templates)
|
||||
|
||||
### 1. Students List Template
|
||||
**File:** `templates/students.html`
|
||||
|
||||
Features:
|
||||
- Displays all students in a paginated table
|
||||
- Search functionality (by name, CIAL, NIF, etc.)
|
||||
- Filter by gender and study group
|
||||
- Responsive design
|
||||
- Action buttons for view, edit, delete
|
||||
|
||||
### 2. Add Student Template
|
||||
**File:** `templates/add_student.html`
|
||||
|
||||
Features:
|
||||
- Form for adding new students
|
||||
- All required fields with validation
|
||||
- Organized in logical sections (Identification, Personal Info, Study Info)
|
||||
- Responsive design
|
||||
|
||||
### 3. Student Detail Template
|
||||
**File:** `templates/student_detail.html`
|
||||
|
||||
Features:
|
||||
- Displays complete student information
|
||||
- Shows assigned tablets and other devices
|
||||
- Shows loan history
|
||||
- Breadcrumbs for navigation
|
||||
- Edit and delete buttons
|
||||
|
||||
### 4. Edit Student Template
|
||||
**File:** `templates/edit_student.html`
|
||||
|
||||
Features:
|
||||
- Form for editing existing students
|
||||
- Pre-populated with current student data
|
||||
- Same validation as add form
|
||||
- Breadcrumbs for navigation
|
||||
|
||||
### 5. Import Students Template
|
||||
**File:** `templates/import_students.html`
|
||||
|
||||
Features:
|
||||
- File upload form for CSV import
|
||||
- Instructions and requirements
|
||||
- Sample data format preview
|
||||
- Clear visual design
|
||||
|
||||
## Database Schema
|
||||
|
||||
### New Table: students
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS students (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
order_number INTEGER,
|
||||
cial_code TEXT UNIQUE NOT NULL,
|
||||
nif_nie_passport TEXT UNIQUE,
|
||||
registration_number INTEGER,
|
||||
file_number TEXT,
|
||||
first_name TEXT NOT NULL,
|
||||
last_name TEXT NOT NULL,
|
||||
full_name TEXT NOT NULL,
|
||||
birth_date TEXT,
|
||||
gender TEXT CHECK(gender IN ('M', 'F', 'O')),
|
||||
study_group TEXT,
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
CONSTRAINT unique_identification UNIQUE (cial_code, nif_nie_passport)
|
||||
);
|
||||
```
|
||||
|
||||
### Indexes Created
|
||||
|
||||
- `idx_students_cial` - On cial_code
|
||||
- `idx_students_nif` - On nif_nie_passport
|
||||
- `idx_students_name` - On (last_name, first_name)
|
||||
- `idx_students_birth_date` - On birth_date
|
||||
- `idx_students_gender` - On gender
|
||||
- `idx_students_group` - On study_group
|
||||
- `idx_tablets_student` - On tablets.assigned_to_student
|
||||
- `idx_non_loanable_student` - On non_loanable_devices.assigned_to_student
|
||||
- `idx_loans_student` - On loans.student_id
|
||||
|
||||
### Modified Tables
|
||||
|
||||
1. **tablets**: Added `assigned_to_student` column (foreign key to students)
|
||||
2. **non_loanable_devices**: Added `assigned_to_student` column (foreign key to students)
|
||||
3. **loans**: Added `student_id` column (foreign key to students)
|
||||
4. All tables: Added `created_at` and `updated_at` timestamp columns
|
||||
|
||||
## CSV Import Process
|
||||
|
||||
### Field Mapping
|
||||
|
||||
| CSV Field | Database Field | Transformation |
|
||||
|-----------|----------------|----------------|
|
||||
| Nº Or. | order_number | Integer conversion |
|
||||
| Apellidos, Nombre | last_name, first_name | Split on ", " |
|
||||
| Apellidos, Nombre | full_name | Keep as-is |
|
||||
| Fecha Nac. | birth_date | DD/MM/YYYY → YYYY-MM-DD |
|
||||
| C.I.A.L. | cial_code | Direct mapping |
|
||||
| NIF/NIE/Pas. | nif_nie_passport | Direct mapping |
|
||||
| Registro | registration_number | Integer conversion |
|
||||
| Expediente | file_number | Direct mapping (nullable) |
|
||||
| Sexo | gender | Direct mapping (M/F/O) |
|
||||
| Grupo | study_group | Direct mapping |
|
||||
| Estudio | - | **DISCARDED** |
|
||||
|
||||
### Validation Rules
|
||||
|
||||
1. **Required Fields**: cial_code, first_name, last_name
|
||||
2. **Unique Fields**: cial_code, nif_nie_passport (if provided)
|
||||
3. **Date Format**: birth_date must be valid YYYY-MM-DD
|
||||
4. **Gender**: Must be 'M', 'F', or 'O'
|
||||
|
||||
## Testing
|
||||
|
||||
### Import Script Test Results
|
||||
|
||||
```
|
||||
$ python3 scripts/import_students.py Datos_programa.csv --dry-run --verbose
|
||||
|
||||
Processing row 1/1:
|
||||
Raw: {'nº or.': '1', 'apellidos, nombre': 'PERIQUITA DE LOS PALOTES, JUANITA', ...}
|
||||
Processed: {'order_number': 1, 'cial_code': 'B19L64119K', ...}
|
||||
[DRY RUN] Would import: PERIQUITA DE LOS PALOTES, JUANITA (B19L64119K)
|
||||
|
||||
IMPORT STATISTICS
|
||||
Total rows in CSV: 1
|
||||
Rows processed: 1
|
||||
Valid rows: 1
|
||||
Invalid rows: 0
|
||||
```
|
||||
|
||||
### Actual Import Test
|
||||
|
||||
```
|
||||
$ python3 scripts/import_students.py Datos_programa.csv --verbose
|
||||
|
||||
Processing row 1/1:
|
||||
Inserted student ID: 1
|
||||
|
||||
IMPORT STATISTICS
|
||||
Total rows in CSV: 1
|
||||
Rows processed: 1
|
||||
Valid rows: 1
|
||||
Invalid rows: 0
|
||||
Rows imported: 1
|
||||
Duplicates skipped: 0
|
||||
Errors: 0
|
||||
```
|
||||
|
||||
### Database Verification
|
||||
|
||||
```sql
|
||||
SELECT * FROM students;
|
||||
-- Returns: 1 row with all fields correctly populated
|
||||
```
|
||||
|
||||
## New Features
|
||||
|
||||
### 1. Student Management
|
||||
- List all students with pagination
|
||||
- Search and filter students
|
||||
- Add new students
|
||||
- View student details
|
||||
- Edit student information
|
||||
- Delete students (with safety checks)
|
||||
- Import students from CSV
|
||||
|
||||
### 2. Enhanced Loan Management
|
||||
- Loan tablets to students directly
|
||||
- Associate loans with both staff users and students
|
||||
- View student loan history
|
||||
|
||||
### 3. Device Assignment
|
||||
- Assign tablets to students
|
||||
- Assign non-loanable devices to students
|
||||
- View all devices assigned to a student
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Import Students from CSV
|
||||
|
||||
```bash
|
||||
# Import from file
|
||||
python3 scripts/import_students.py path/to/data.csv
|
||||
|
||||
# Import with custom database
|
||||
python3 scripts/import_students.py data.csv --database mydb.db
|
||||
|
||||
# Dry run to validate
|
||||
python3 scripts/import_students.py data.csv --dry-run
|
||||
|
||||
# Skip errors and continue
|
||||
python3 scripts/import_students.py data.csv --skip-errors
|
||||
```
|
||||
|
||||
### Web Interface
|
||||
|
||||
1. Navigate to `/students` to view all students
|
||||
2. Click "Add Student" to add a new student manually
|
||||
3. Click "Import CSV" to upload a CSV file
|
||||
4. Click on a student to view their details, assigned devices, and loan history
|
||||
5. When loaning a tablet, optionally select a student to associate with the loan
|
||||
|
||||
## Migration Instructions
|
||||
|
||||
### For Existing Installations
|
||||
|
||||
1. **Backup your database**:
|
||||
```bash
|
||||
cp tablets.db tablets.db.backup
|
||||
```
|
||||
|
||||
2. **Run the migration script**:
|
||||
```bash
|
||||
python3 scripts/migrate_database.py
|
||||
```
|
||||
|
||||
3. **Import your CSV data**:
|
||||
```bash
|
||||
python3 scripts/import_students.py Datos_programa.csv
|
||||
```
|
||||
|
||||
4. **Start the application**:
|
||||
```bash
|
||||
python3 app.py
|
||||
```
|
||||
|
||||
### For New Installations
|
||||
|
||||
1. **Initialize the database**:
|
||||
```bash
|
||||
python3 app.py # This will create the database with all tables
|
||||
```
|
||||
|
||||
2. **Import your CSV data**:
|
||||
```bash
|
||||
python3 scripts/import_students.py Datos_programa.csv
|
||||
```
|
||||
|
||||
3. **Start the application**:
|
||||
```bash
|
||||
python3 app.py
|
||||
```
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
GestionTablets/
|
||||
├── app.py # Updated with student routes
|
||||
├── scripts/
|
||||
│ ├── import_students.py # CSV import script
|
||||
│ └── migrate_database.py # Database migration script
|
||||
├── templates/
|
||||
│ ├── base.html # Updated with Students link
|
||||
│ ├── loan_tablet.html # Updated with student selection
|
||||
│ ├── students.html # New: Student list
|
||||
│ ├── add_student.html # New: Add student form
|
||||
│ ├── edit_student.html # New: Edit student form
|
||||
│ ├── student_detail.html # New: Student details
|
||||
│ └── import_students.html # New: CSV import interface
|
||||
├── docs/
|
||||
│ └── DATABASE_SCHEMA_SPECIFICATION.md # New: Schema documentation
|
||||
├── requirements.txt # Updated dependencies
|
||||
└── Datos_programa.csv # Sample CSV file
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Test the application**: Run the app and verify all student functionality works
|
||||
2. **Add translations**: Update translation files for new student-related strings
|
||||
3. **Add more CSV files**: Test with additional CSV files to ensure robustness
|
||||
4. **Performance testing**: Test with large CSV files (1000+ rows)
|
||||
5. **Error handling**: Test edge cases (empty files, malformed data, etc.)
|
||||
|
||||
## Notes
|
||||
|
||||
- The implementation preserves all existing functionality
|
||||
- The new student features are fully integrated with the existing tablet and loan management
|
||||
- All templates use consistent styling (Bootstrap 5 classes)
|
||||
- The import script handles encoding conversion automatically
|
||||
- The database migration is safe and creates backups automatically
|
||||
Loading…
Add table
Add a link
Reference in a new issue