calypso/docs/alpha/sds/SDS-02-Database-Design.md

SDS-02: Database Design

1. Database Overview

1.1 Database System

• Type: PostgreSQL 14+
• Encoding: UTF-8
• Connection Pooling: pgxpool
• Migrations: Custom migration system

1.2 Database Schema Organization

• Tables: Organized by domain (users, storage, shares, etc.)
• Indexes: Performance indexes on foreign keys and frequently queried columns
• Constraints: Foreign keys, unique constraints, check constraints

2. Core Tables

2.1 Users & Authentication

users (
  id UUID PRIMARY KEY,
  username VARCHAR(255) UNIQUE NOT NULL,
  email VARCHAR(255) UNIQUE,
  password_hash VARCHAR(255) NOT NULL,
  is_active BOOLEAN DEFAULT true,
  created_at TIMESTAMP,
  updated_at TIMESTAMP
)

roles (
  id UUID PRIMARY KEY,
  name VARCHAR(255) UNIQUE NOT NULL,
  description TEXT,
  created_at TIMESTAMP,
  updated_at TIMESTAMP
)

permissions (
  id UUID PRIMARY KEY,
  resource VARCHAR(255) NOT NULL,
  action VARCHAR(255) NOT NULL,
  description TEXT,
  UNIQUE(resource, action)
)

user_roles (
  user_id UUID REFERENCES users(id),
  role_id UUID REFERENCES roles(id),
  PRIMARY KEY (user_id, role_id)
)

role_permissions (
  role_id UUID REFERENCES roles(id),
  permission_id UUID REFERENCES permissions(id),
  PRIMARY KEY (role_id, permission_id)
)

groups (
  id UUID PRIMARY KEY,
  name VARCHAR(255) UNIQUE NOT NULL,
  description TEXT,
  created_at TIMESTAMP,
  updated_at TIMESTAMP
)

user_groups (
  user_id UUID REFERENCES users(id),
  group_id UUID REFERENCES groups(id),
  PRIMARY KEY (user_id, group_id)
)

2.2 Storage Tables

zfs_pools (
  id UUID PRIMARY KEY,
  name VARCHAR(255) UNIQUE NOT NULL,
  description TEXT,
  raid_level VARCHAR(50),
  status VARCHAR(50),
  total_capacity_bytes BIGINT,
  used_capacity_bytes BIGINT,
  health_status VARCHAR(50),
  created_at TIMESTAMP,
  updated_at TIMESTAMP,
  created_by UUID REFERENCES users(id)
)

zfs_datasets (
  id UUID PRIMARY KEY,
  name VARCHAR(255) NOT NULL,
  pool_name VARCHAR(255) REFERENCES zfs_pools(name),
  type VARCHAR(50),
  mount_point VARCHAR(255),
  used_bytes BIGINT,
  available_bytes BIGINT,
  compression VARCHAR(50),
  quota BIGINT,
  reservation BIGINT,
  created_at TIMESTAMP,
  UNIQUE(pool_name, name)
)

physical_disks (
  id UUID PRIMARY KEY,
  device_path VARCHAR(255) UNIQUE NOT NULL,
  vendor VARCHAR(255),
  model VARCHAR(255),
  serial_number VARCHAR(255),
  size_bytes BIGINT,
  type VARCHAR(50),
  status VARCHAR(50),
  last_synced_at TIMESTAMP
)

storage_repositories (
  id UUID PRIMARY KEY,
  name VARCHAR(255) UNIQUE NOT NULL,
  type VARCHAR(50),
  path VARCHAR(255),
  capacity_bytes BIGINT,
  used_bytes BIGINT,
  created_at TIMESTAMP,
  updated_at TIMESTAMP
)

2.3 Shares Tables

shares (
  id UUID PRIMARY KEY,
  dataset_id UUID REFERENCES zfs_datasets(id),
  share_type VARCHAR(50),
  nfs_enabled BOOLEAN DEFAULT false,
  nfs_options TEXT,
  nfs_clients TEXT[],
  smb_enabled BOOLEAN DEFAULT false,
  smb_share_name VARCHAR(255),
  smb_path VARCHAR(255),
  smb_comment TEXT,
  smb_guest_ok BOOLEAN DEFAULT false,
  smb_read_only BOOLEAN DEFAULT false,
  smb_browseable BOOLEAN DEFAULT true,
  is_active BOOLEAN DEFAULT true,
  created_at TIMESTAMP,
  updated_at TIMESTAMP,
  created_by UUID REFERENCES users(id)
)

2.4 iSCSI Tables

iscsi_targets (
  id UUID PRIMARY KEY,
  name VARCHAR(255) UNIQUE NOT NULL,
  alias VARCHAR(255),
  enabled BOOLEAN DEFAULT true,
  created_at TIMESTAMP,
  updated_at TIMESTAMP,
  created_by UUID REFERENCES users(id)
)

iscsi_luns (
  id UUID PRIMARY KEY,
  target_id UUID REFERENCES iscsi_targets(id),
  lun_number INTEGER,
  device_path VARCHAR(255),
  size_bytes BIGINT,
  created_at TIMESTAMP
)

iscsi_initiators (
  id UUID PRIMARY KEY,
  iqn VARCHAR(255) UNIQUE NOT NULL,
  created_at TIMESTAMP
)

target_initiators (
  target_id UUID REFERENCES iscsi_targets(id),
  initiator_id UUID REFERENCES iscsi_initiators(id),
  PRIMARY KEY (target_id, initiator_id)
)

2.5 Tape Tables

vtl_libraries (
  id UUID PRIMARY KEY,
  name VARCHAR(255) UNIQUE NOT NULL,
  vendor VARCHAR(255),
  model VARCHAR(255),
  drive_count INTEGER,
  slot_count INTEGER,
  status VARCHAR(50),
  created_at TIMESTAMP,
  updated_at TIMESTAMP,
  created_by UUID REFERENCES users(id)
)

physical_libraries (
  id UUID PRIMARY KEY,
  vendor VARCHAR(255),
  model VARCHAR(255),
  serial_number VARCHAR(255),
  drive_count INTEGER,
  slot_count INTEGER,
  discovered_at TIMESTAMP
)

2.6 Backup Tables

backup_jobs (
  id UUID PRIMARY KEY,
  name VARCHAR(255) UNIQUE NOT NULL,
  client_id INTEGER,
  fileset_id INTEGER,
  schedule VARCHAR(255),
  storage_pool_id INTEGER,
  enabled BOOLEAN DEFAULT true,
  created_at TIMESTAMP,
  updated_at TIMESTAMP,
  created_by UUID REFERENCES users(id)
)

2.7 Monitoring Tables

alerts (
  id UUID PRIMARY KEY,
  rule_id UUID,
  severity VARCHAR(50),
  source VARCHAR(255),
  message TEXT,
  status VARCHAR(50),
  acknowledged_at TIMESTAMP,
  resolved_at TIMESTAMP,
  created_at TIMESTAMP
)

alert_rules (
  id UUID PRIMARY KEY,
  name VARCHAR(255) UNIQUE NOT NULL,
  description TEXT,
  source VARCHAR(255),
  condition_type VARCHAR(255),
  condition_config JSONB,
  severity VARCHAR(50),
  enabled BOOLEAN DEFAULT true,
  created_at TIMESTAMP,
  updated_at TIMESTAMP
)

2.8 Audit Tables

audit_logs (
  id UUID PRIMARY KEY,
  user_id UUID REFERENCES users(id),
  action VARCHAR(255),
  resource_type VARCHAR(255),
  resource_id VARCHAR(255),
  method VARCHAR(10),
  path VARCHAR(255),
  ip_address VARCHAR(45),
  user_agent TEXT,
  request_body JSONB,
  response_status INTEGER,
  created_at TIMESTAMP
)

2.9 Task Tables

tasks (
  id UUID PRIMARY KEY,
  type VARCHAR(255),
  status VARCHAR(50),
  progress INTEGER,
  result JSONB,
  error_message TEXT,
  created_at TIMESTAMP,
  updated_at TIMESTAMP,
  completed_at TIMESTAMP,
  created_by UUID REFERENCES users(id)
)

3. Indexes

3.1 Performance Indexes

-- Users
CREATE INDEX idx_users_username ON users(username);
CREATE INDEX idx_users_email ON users(email);

-- Storage
CREATE INDEX idx_zfs_pools_name ON zfs_pools(name);
CREATE INDEX idx_zfs_datasets_pool_name ON zfs_datasets(pool_name);

-- Shares
CREATE INDEX idx_shares_dataset_id ON shares(dataset_id);
CREATE INDEX idx_shares_created_by ON shares(created_by);

-- iSCSI
CREATE INDEX idx_iscsi_targets_name ON iscsi_targets(name);
CREATE INDEX idx_iscsi_luns_target_id ON iscsi_luns(target_id);

-- Monitoring
CREATE INDEX idx_alerts_status ON alerts(status);
CREATE INDEX idx_alerts_created_at ON alerts(created_at);
CREATE INDEX idx_audit_logs_user_id ON audit_logs(user_id);
CREATE INDEX idx_audit_logs_created_at ON audit_logs(created_at);

4. Migrations

4.1 Migration System

• Location: db/migrations/
• Naming: NNN_description.sql
• Execution: Sequential execution on startup
• Version Tracking: schema_migrations table

4.2 Migration Files

• 001_initial_schema.sql: Core tables
• 002_storage_and_tape_schema.sql: Storage and tape tables
• 003_performance_indexes.sql: Performance indexes
• 004_add_zfs_pools_table.sql: ZFS pools
• 005_add_zfs_datasets_table.sql: ZFS datasets
• 006_add_zfs_shares_and_iscsi.sql: Shares and iSCSI
• 007_add_vendor_to_vtl_libraries.sql: VTL updates
• 008_add_user_groups.sql: User groups
• 009_backup_jobs_schema.sql: Backup jobs
• 010_add_backup_permissions.sql: Backup permissions
• 011_sync_bacula_jobs_function.sql: Bacula sync function

5. Data Relationships

5.1 Entity Relationships

• Users → Roles (many-to-many)
• Roles → Permissions (many-to-many)
• Users → Groups (many-to-many)
• ZFS Pools → ZFS Datasets (one-to-many)
• ZFS Datasets → Shares (one-to-many)
• iSCSI Targets → LUNs (one-to-many)
• iSCSI Targets → Initiators (many-to-many)

6. Data Integrity

6.1 Constraints

• Primary Keys: UUID primary keys for all tables
• Foreign Keys: Referential integrity
• Unique Constraints: Unique usernames, emails, names
• Check Constraints: Valid status values, positive numbers

6.2 Cascading Rules

• ON DELETE CASCADE: Child records deleted with parent
• ON DELETE RESTRICT: Prevent deletion if referenced
• ON UPDATE CASCADE: Update foreign keys on parent update

7. Query Optimization

7.1 Query Patterns

• Eager Loading: Join related data when needed
• Pagination: Limit and offset for large datasets
• Filtering: WHERE clauses for filtering
• Sorting: ORDER BY for sorted results

7.2 Caching Strategy

• Query Result Caching: Cache frequently accessed queries
• Cache Invalidation: Invalidate on write operations
• TTL: Time-to-live for cached data

8. Backup & Recovery

8.1 Backup Strategy

• Regular Backups: Daily database backups
• Point-in-Time Recovery: WAL archiving
• Backup Retention: 30 days retention

8.2 Recovery Procedures

• Full Restore: Restore from backup
• Point-in-Time: Restore to specific timestamp
• Selective Restore: Restore specific tables