tidy up documentation for alpha release
This commit is contained in:
Warp Agent
385
docs/alpha/sds/SDS-02-Database-Design.md
Normal file
385
docs/alpha/sds/SDS-02-Database-Design.md
Normal file
@@ -0,0 +1,385 @@
|
||||
# SDS-02: Database Design
|
||||
|
||||
## 1. Database Overview
|
||||
|
||||
### 1.1 Database System
|
||||
- **Type**: PostgreSQL 14+
|
||||
- **Encoding**: UTF-8
|
||||
- **Connection Pooling**: pgxpool
|
||||
- **Migrations**: Custom migration system
|
||||
|
||||
### 1.2 Database Schema Organization
|
||||
- **Tables**: Organized by domain (users, storage, shares, etc.)
|
||||
- **Indexes**: Performance indexes on foreign keys and frequently queried columns
|
||||
- **Constraints**: Foreign keys, unique constraints, check constraints
|
||||
|
||||
## 2. Core Tables
|
||||
|
||||
### 2.1 Users & Authentication
|
||||
```sql
|
||||
users (
|
||||
id UUID PRIMARY KEY,
|
||||
username VARCHAR(255) UNIQUE NOT NULL,
|
||||
email VARCHAR(255) UNIQUE,
|
||||
password_hash VARCHAR(255) NOT NULL,
|
||||
is_active BOOLEAN DEFAULT true,
|
||||
created_at TIMESTAMP,
|
||||
updated_at TIMESTAMP
|
||||
)
|
||||
|
||||
roles (
|
||||
id UUID PRIMARY KEY,
|
||||
name VARCHAR(255) UNIQUE NOT NULL,
|
||||
description TEXT,
|
||||
created_at TIMESTAMP,
|
||||
updated_at TIMESTAMP
|
||||
)
|
||||
|
||||
permissions (
|
||||
id UUID PRIMARY KEY,
|
||||
resource VARCHAR(255) NOT NULL,
|
||||
action VARCHAR(255) NOT NULL,
|
||||
description TEXT,
|
||||
UNIQUE(resource, action)
|
||||
)
|
||||
|
||||
user_roles (
|
||||
user_id UUID REFERENCES users(id),
|
||||
role_id UUID REFERENCES roles(id),
|
||||
PRIMARY KEY (user_id, role_id)
|
||||
)
|
||||
|
||||
role_permissions (
|
||||
role_id UUID REFERENCES roles(id),
|
||||
permission_id UUID REFERENCES permissions(id),
|
||||
PRIMARY KEY (role_id, permission_id)
|
||||
)
|
||||
|
||||
groups (
|
||||
id UUID PRIMARY KEY,
|
||||
name VARCHAR(255) UNIQUE NOT NULL,
|
||||
description TEXT,
|
||||
created_at TIMESTAMP,
|
||||
updated_at TIMESTAMP
|
||||
)
|
||||
|
||||
user_groups (
|
||||
user_id UUID REFERENCES users(id),
|
||||
group_id UUID REFERENCES groups(id),
|
||||
PRIMARY KEY (user_id, group_id)
|
||||
)
|
||||
```
|
||||
|
||||
### 2.2 Storage Tables
|
||||
```sql
|
||||
zfs_pools (
|
||||
id UUID PRIMARY KEY,
|
||||
name VARCHAR(255) UNIQUE NOT NULL,
|
||||
description TEXT,
|
||||
raid_level VARCHAR(50),
|
||||
status VARCHAR(50),
|
||||
total_capacity_bytes BIGINT,
|
||||
used_capacity_bytes BIGINT,
|
||||
health_status VARCHAR(50),
|
||||
created_at TIMESTAMP,
|
||||
updated_at TIMESTAMP,
|
||||
created_by UUID REFERENCES users(id)
|
||||
)
|
||||
|
||||
zfs_datasets (
|
||||
id UUID PRIMARY KEY,
|
||||
name VARCHAR(255) NOT NULL,
|
||||
pool_name VARCHAR(255) REFERENCES zfs_pools(name),
|
||||
type VARCHAR(50),
|
||||
mount_point VARCHAR(255),
|
||||
used_bytes BIGINT,
|
||||
available_bytes BIGINT,
|
||||
compression VARCHAR(50),
|
||||
quota BIGINT,
|
||||
reservation BIGINT,
|
||||
created_at TIMESTAMP,
|
||||
UNIQUE(pool_name, name)
|
||||
)
|
||||
|
||||
physical_disks (
|
||||
id UUID PRIMARY KEY,
|
||||
device_path VARCHAR(255) UNIQUE NOT NULL,
|
||||
vendor VARCHAR(255),
|
||||
model VARCHAR(255),
|
||||
serial_number VARCHAR(255),
|
||||
size_bytes BIGINT,
|
||||
type VARCHAR(50),
|
||||
status VARCHAR(50),
|
||||
last_synced_at TIMESTAMP
|
||||
)
|
||||
|
||||
storage_repositories (
|
||||
id UUID PRIMARY KEY,
|
||||
name VARCHAR(255) UNIQUE NOT NULL,
|
||||
type VARCHAR(50),
|
||||
path VARCHAR(255),
|
||||
capacity_bytes BIGINT,
|
||||
used_bytes BIGINT,
|
||||
created_at TIMESTAMP,
|
||||
updated_at TIMESTAMP
|
||||
)
|
||||
```
|
||||
|
||||
### 2.3 Shares Tables
|
||||
```sql
|
||||
shares (
|
||||
id UUID PRIMARY KEY,
|
||||
dataset_id UUID REFERENCES zfs_datasets(id),
|
||||
share_type VARCHAR(50),
|
||||
nfs_enabled BOOLEAN DEFAULT false,
|
||||
nfs_options TEXT,
|
||||
nfs_clients TEXT[],
|
||||
smb_enabled BOOLEAN DEFAULT false,
|
||||
smb_share_name VARCHAR(255),
|
||||
smb_path VARCHAR(255),
|
||||
smb_comment TEXT,
|
||||
smb_guest_ok BOOLEAN DEFAULT false,
|
||||
smb_read_only BOOLEAN DEFAULT false,
|
||||
smb_browseable BOOLEAN DEFAULT true,
|
||||
is_active BOOLEAN DEFAULT true,
|
||||
created_at TIMESTAMP,
|
||||
updated_at TIMESTAMP,
|
||||
created_by UUID REFERENCES users(id)
|
||||
)
|
||||
```
|
||||
|
||||
### 2.4 iSCSI Tables
|
||||
```sql
|
||||
iscsi_targets (
|
||||
id UUID PRIMARY KEY,
|
||||
name VARCHAR(255) UNIQUE NOT NULL,
|
||||
alias VARCHAR(255),
|
||||
enabled BOOLEAN DEFAULT true,
|
||||
created_at TIMESTAMP,
|
||||
updated_at TIMESTAMP,
|
||||
created_by UUID REFERENCES users(id)
|
||||
)
|
||||
|
||||
iscsi_luns (
|
||||
id UUID PRIMARY KEY,
|
||||
target_id UUID REFERENCES iscsi_targets(id),
|
||||
lun_number INTEGER,
|
||||
device_path VARCHAR(255),
|
||||
size_bytes BIGINT,
|
||||
created_at TIMESTAMP
|
||||
)
|
||||
|
||||
iscsi_initiators (
|
||||
id UUID PRIMARY KEY,
|
||||
iqn VARCHAR(255) UNIQUE NOT NULL,
|
||||
created_at TIMESTAMP
|
||||
)
|
||||
|
||||
target_initiators (
|
||||
target_id UUID REFERENCES iscsi_targets(id),
|
||||
initiator_id UUID REFERENCES iscsi_initiators(id),
|
||||
PRIMARY KEY (target_id, initiator_id)
|
||||
)
|
||||
```
|
||||
|
||||
### 2.5 Tape Tables
|
||||
```sql
|
||||
vtl_libraries (
|
||||
id UUID PRIMARY KEY,
|
||||
name VARCHAR(255) UNIQUE NOT NULL,
|
||||
vendor VARCHAR(255),
|
||||
model VARCHAR(255),
|
||||
drive_count INTEGER,
|
||||
slot_count INTEGER,
|
||||
status VARCHAR(50),
|
||||
created_at TIMESTAMP,
|
||||
updated_at TIMESTAMP,
|
||||
created_by UUID REFERENCES users(id)
|
||||
)
|
||||
|
||||
physical_libraries (
|
||||
id UUID PRIMARY KEY,
|
||||
vendor VARCHAR(255),
|
||||
model VARCHAR(255),
|
||||
serial_number VARCHAR(255),
|
||||
drive_count INTEGER,
|
||||
slot_count INTEGER,
|
||||
discovered_at TIMESTAMP
|
||||
)
|
||||
```
|
||||
|
||||
### 2.6 Backup Tables
|
||||
```sql
|
||||
backup_jobs (
|
||||
id UUID PRIMARY KEY,
|
||||
name VARCHAR(255) UNIQUE NOT NULL,
|
||||
client_id INTEGER,
|
||||
fileset_id INTEGER,
|
||||
schedule VARCHAR(255),
|
||||
storage_pool_id INTEGER,
|
||||
enabled BOOLEAN DEFAULT true,
|
||||
created_at TIMESTAMP,
|
||||
updated_at TIMESTAMP,
|
||||
created_by UUID REFERENCES users(id)
|
||||
)
|
||||
```
|
||||
|
||||
### 2.7 Monitoring Tables
|
||||
```sql
|
||||
alerts (
|
||||
id UUID PRIMARY KEY,
|
||||
rule_id UUID,
|
||||
severity VARCHAR(50),
|
||||
source VARCHAR(255),
|
||||
message TEXT,
|
||||
status VARCHAR(50),
|
||||
acknowledged_at TIMESTAMP,
|
||||
resolved_at TIMESTAMP,
|
||||
created_at TIMESTAMP
|
||||
)
|
||||
|
||||
alert_rules (
|
||||
id UUID PRIMARY KEY,
|
||||
name VARCHAR(255) UNIQUE NOT NULL,
|
||||
description TEXT,
|
||||
source VARCHAR(255),
|
||||
condition_type VARCHAR(255),
|
||||
condition_config JSONB,
|
||||
severity VARCHAR(50),
|
||||
enabled BOOLEAN DEFAULT true,
|
||||
created_at TIMESTAMP,
|
||||
updated_at TIMESTAMP
|
||||
)
|
||||
```
|
||||
|
||||
### 2.8 Audit Tables
|
||||
```sql
|
||||
audit_logs (
|
||||
id UUID PRIMARY KEY,
|
||||
user_id UUID REFERENCES users(id),
|
||||
action VARCHAR(255),
|
||||
resource_type VARCHAR(255),
|
||||
resource_id VARCHAR(255),
|
||||
method VARCHAR(10),
|
||||
path VARCHAR(255),
|
||||
ip_address VARCHAR(45),
|
||||
user_agent TEXT,
|
||||
request_body JSONB,
|
||||
response_status INTEGER,
|
||||
created_at TIMESTAMP
|
||||
)
|
||||
```
|
||||
|
||||
### 2.9 Task Tables
|
||||
```sql
|
||||
tasks (
|
||||
id UUID PRIMARY KEY,
|
||||
type VARCHAR(255),
|
||||
status VARCHAR(50),
|
||||
progress INTEGER,
|
||||
result JSONB,
|
||||
error_message TEXT,
|
||||
created_at TIMESTAMP,
|
||||
updated_at TIMESTAMP,
|
||||
completed_at TIMESTAMP,
|
||||
created_by UUID REFERENCES users(id)
|
||||
)
|
||||
```
|
||||
|
||||
## 3. Indexes
|
||||
|
||||
### 3.1 Performance Indexes
|
||||
```sql
|
||||
-- Users
|
||||
CREATE INDEX idx_users_username ON users(username);
|
||||
CREATE INDEX idx_users_email ON users(email);
|
||||
|
||||
-- Storage
|
||||
CREATE INDEX idx_zfs_pools_name ON zfs_pools(name);
|
||||
CREATE INDEX idx_zfs_datasets_pool_name ON zfs_datasets(pool_name);
|
||||
|
||||
-- Shares
|
||||
CREATE INDEX idx_shares_dataset_id ON shares(dataset_id);
|
||||
CREATE INDEX idx_shares_created_by ON shares(created_by);
|
||||
|
||||
-- iSCSI
|
||||
CREATE INDEX idx_iscsi_targets_name ON iscsi_targets(name);
|
||||
CREATE INDEX idx_iscsi_luns_target_id ON iscsi_luns(target_id);
|
||||
|
||||
-- Monitoring
|
||||
CREATE INDEX idx_alerts_status ON alerts(status);
|
||||
CREATE INDEX idx_alerts_created_at ON alerts(created_at);
|
||||
CREATE INDEX idx_audit_logs_user_id ON audit_logs(user_id);
|
||||
CREATE INDEX idx_audit_logs_created_at ON audit_logs(created_at);
|
||||
```
|
||||
|
||||
## 4. Migrations
|
||||
|
||||
### 4.1 Migration System
|
||||
- **Location**: `db/migrations/`
|
||||
- **Naming**: `NNN_description.sql`
|
||||
- **Execution**: Sequential execution on startup
|
||||
- **Version Tracking**: `schema_migrations` table
|
||||
|
||||
### 4.2 Migration Files
|
||||
- `001_initial_schema.sql`: Core tables
|
||||
- `002_storage_and_tape_schema.sql`: Storage and tape tables
|
||||
- `003_performance_indexes.sql`: Performance indexes
|
||||
- `004_add_zfs_pools_table.sql`: ZFS pools
|
||||
- `005_add_zfs_datasets_table.sql`: ZFS datasets
|
||||
- `006_add_zfs_shares_and_iscsi.sql`: Shares and iSCSI
|
||||
- `007_add_vendor_to_vtl_libraries.sql`: VTL updates
|
||||
- `008_add_user_groups.sql`: User groups
|
||||
- `009_backup_jobs_schema.sql`: Backup jobs
|
||||
- `010_add_backup_permissions.sql`: Backup permissions
|
||||
- `011_sync_bacula_jobs_function.sql`: Bacula sync function
|
||||
|
||||
## 5. Data Relationships
|
||||
|
||||
### 5.1 Entity Relationships
|
||||
- **Users** → **Roles** (many-to-many)
|
||||
- **Roles** → **Permissions** (many-to-many)
|
||||
- **Users** → **Groups** (many-to-many)
|
||||
- **ZFS Pools** → **ZFS Datasets** (one-to-many)
|
||||
- **ZFS Datasets** → **Shares** (one-to-many)
|
||||
- **iSCSI Targets** → **LUNs** (one-to-many)
|
||||
- **iSCSI Targets** → **Initiators** (many-to-many)
|
||||
|
||||
## 6. Data Integrity
|
||||
|
||||
### 6.1 Constraints
|
||||
- **Primary Keys**: UUID primary keys for all tables
|
||||
- **Foreign Keys**: Referential integrity
|
||||
- **Unique Constraints**: Unique usernames, emails, names
|
||||
- **Check Constraints**: Valid status values, positive numbers
|
||||
|
||||
### 6.2 Cascading Rules
|
||||
- **ON DELETE CASCADE**: Child records deleted with parent
|
||||
- **ON DELETE RESTRICT**: Prevent deletion if referenced
|
||||
- **ON UPDATE CASCADE**: Update foreign keys on parent update
|
||||
|
||||
## 7. Query Optimization
|
||||
|
||||
### 7.1 Query Patterns
|
||||
- **Eager Loading**: Join related data when needed
|
||||
- **Pagination**: Limit and offset for large datasets
|
||||
- **Filtering**: WHERE clauses for filtering
|
||||
- **Sorting**: ORDER BY for sorted results
|
||||
|
||||
### 7.2 Caching Strategy
|
||||
- **Query Result Caching**: Cache frequently accessed queries
|
||||
- **Cache Invalidation**: Invalidate on write operations
|
||||
- **TTL**: Time-to-live for cached data
|
||||
|
||||
## 8. Backup & Recovery
|
||||
|
||||
### 8.1 Backup Strategy
|
||||
- **Regular Backups**: Daily database backups
|
||||
- **Point-in-Time Recovery**: WAL archiving
|
||||
- **Backup Retention**: 30 days retention
|
||||
|
||||
### 8.2 Recovery Procedures
|
||||
- **Full Restore**: Restore from backup
|
||||
- **Point-in-Time**: Restore to specific timestamp
|
||||
- **Selective Restore**: Restore specific tables
|
||||
|
||||
Reference in New Issue
Block a user