othman.suseno/calypso
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tidy up documentation for alpha release

Browse Source
This commit is contained in:
Warp Agent
2026-01-04 13:19:40 +07:00
parent 2bb64620d4
commit 70d25e13b8
20 changed files with 4415 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

182
docs/alpha/sds/SDS-00-Overview.md Normal file
View File

@@ -0,0 +1,182 @@
# Software Design Specification (SDS)
## AtlasOS - Calypso Backup Appliance
### Alpha Release
**Version:** 1.0.0-alpha
**Date:** 2025-01-XX
**Status:** In Development
---
## 1. Introduction
### 1.1 Purpose
This document provides a comprehensive Software Design Specification (SDS) for AtlasOS - Calypso, describing the system architecture, component design, database schema, API design, and implementation details.
### 1.2 Scope
This SDS covers:
- System architecture and design patterns
- Component structure and organization
- Database schema and data models
- API design and endpoints
- Security architecture
- Deployment architecture
- Integration patterns
### 1.3 Document Organization
- **SDS-01**: System Architecture
- **SDS-02**: Backend Design
- **SDS-03**: Frontend Design
- **SDS-04**: Database Design
- **SDS-05**: API Design
- **SDS-06**: Security Design
- **SDS-07**: Integration Design
---
## 2. System Architecture Overview
### 2.1 High-Level Architecture
Calypso follows a three-tier architecture:
1. **Presentation Layer**: React-based SPA
2. **Application Layer**: Go-based REST API
3. **Data Layer**: PostgreSQL database
### 2.2 Architecture Patterns
- **Clean Architecture**: Separation of concerns, domain-driven design
- **RESTful API**: Resource-based API design
- **Repository Pattern**: Data access abstraction
- **Service Layer**: Business logic encapsulation
- **Middleware Pattern**: Cross-cutting concerns
### 2.3 Technology Stack
#### Backend
- **Language**: Go 1.21+
- **Framework**: Gin web framework
- **Database**: PostgreSQL 14+
- **Authentication**: JWT tokens
- **Logging**: Zerolog structured logging
#### Frontend
- **Framework**: React 18 with TypeScript
- **Build Tool**: Vite
- **Styling**: TailwindCSS
- **State Management**: Zustand + TanStack Query
- **Routing**: React Router
- **HTTP Client**: Axios
---
## 3. Design Principles
### 3.1 Separation of Concerns
- Clear boundaries between layers
- Single responsibility principle
- Dependency inversion
### 3.2 Scalability
- Stateless API design
- Horizontal scaling capability
- Efficient database queries
### 3.3 Security
- Defense in depth
- Principle of least privilege
- Input validation and sanitization
### 3.4 Maintainability
- Clean code principles
- Comprehensive logging
- Error handling
- Code documentation
### 3.5 Performance
- Response caching
- Database query optimization
- Efficient data structures
- Background job processing
---
## 4. System Components
### 4.1 Backend Components
- **Auth**: Authentication and authorization
- **Storage**: ZFS and storage management
- **Shares**: SMB/NFS share management
- **SCST**: iSCSI target management
- **Tape**: Physical and VTL management
- **Backup**: Bacula/Bareos integration
- **System**: System service management
- **Monitoring**: Metrics and alerting
- **IAM**: User and access management
### 4.2 Frontend Components
- **Pages**: Route-based page components
- **Components**: Reusable UI components
- **API**: API client and queries
- **Store**: Global state management
- **Hooks**: Custom React hooks
- **Utils**: Utility functions
---
## 5. Data Flow
### 5.1 Request Flow
1. User action in frontend
2. API call via Axios
3. Request middleware (auth, logging, rate limiting)
4. Handler processes request
5. Service layer business logic
6. Database operations
7. Response returned to frontend
8. UI update via React Query
### 5.2 Background Jobs
- Disk monitoring (every 5 minutes)
- ZFS pool monitoring (every 2 minutes)
- Metrics collection (every 30 seconds)
- Alert rule evaluation (continuous)
---
## 6. Deployment Architecture
### 6.1 Single-Server Deployment
- Backend API service (systemd)
- Frontend static files (nginx/caddy)
- PostgreSQL database
- External services (ZFS, SCST, Bacula)
### 6.2 Service Management
- Systemd service files
- Auto-restart on failure
- Log rotation
- Health checks
---
## 7. Future Enhancements
### 7.1 Scalability
- Multi-server deployment
- Load balancing
- Database replication
- Distributed caching
### 7.2 Features
- WebSocket real-time updates
- GraphQL API option
- Microservices architecture
- Container orchestration
---
## Document History
| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 1.0.0-alpha | 2025-01-XX | Development Team | Initial SDS document |

302
docs/alpha/sds/SDS-01-System-Architecture.md Normal file
View File

@@ -0,0 +1,302 @@
# SDS-01: System Architecture
## 1. Architecture Overview
### 1.1 Three-Tier Architecture
```
┌─────────────────────────────────────┐
│ Presentation Layer │
│ (React SPA) │
└──────────────┬──────────────────────┘
│ HTTP/REST
┌──────────────▼──────────────────────┐
│ Application Layer │
│ (Go REST API) │
└──────────────┬──────────────────────┘
│ SQL
┌──────────────▼──────────────────────┐
│ Data Layer │
│ (PostgreSQL) │
└─────────────────────────────────────┘
```
### 1.2 Component Layers
#### Backend Layers
1. **Handler Layer**: HTTP request handling, validation
2. **Service Layer**: Business logic, orchestration
3. **Repository Layer**: Data access, database operations
4. **Model Layer**: Data structures, domain models
#### Frontend Layers
1. **Page Layer**: Route-based page components
2. **Component Layer**: Reusable UI components
3. **API Layer**: API client, data fetching
4. **Store Layer**: Global state management
## 2. Backend Architecture
### 2.1 Directory Structure
```
backend/
├── cmd/
│ └── calypso-api/
│ └── main.go
├── internal/
│ ├── auth/
│ ├── storage/
│ ├── shares/
│ ├── scst/
│ ├── tape_physical/
│ ├── tape_vtl/
│ ├── backup/
│ ├── system/
│ ├── monitoring/
│ ├── iam/
│ ├── tasks/
│ └── common/
│ ├── config/
│ ├── database/
│ ├── logger/
│ ├── router/
│ └── cache/
└── db/
└── migrations/
```
### 2.2 Module Organization
Each module follows this structure:
- **handler.go**: HTTP handlers
- **service.go**: Business logic
- **model.go**: Data models (if needed)
- **repository.go**: Database operations (if needed)
### 2.3 Common Components
- **config**: Configuration management
- **database**: Database connection and migrations
- **logger**: Structured logging
- **router**: HTTP router, middleware
- **cache**: Response caching
- **auth**: Authentication middleware
- **audit**: Audit logging middleware
## 3. Frontend Architecture
### 3.1 Directory Structure
```
frontend/
├── src/
│ ├── pages/
│ ├── components/
│ ├── api/
│ ├── store/
│ ├── hooks/
│ ├── lib/
│ └── App.tsx
└── public/
```
### 3.2 Component Organization
- **pages/**: Route-based page components
- **components/**: Reusable UI components
- **ui/**: Base UI components (buttons, inputs, etc.)
- **Layout.tsx**: Main layout component
- **api/**: API client and query definitions
- **store/**: Zustand stores
- **hooks/**: Custom React hooks
- **lib/**: Utility functions
## 4. Request Processing Flow
### 4.1 HTTP Request Flow
```
Client Request
↓
CORS Middleware
↓
Rate Limiting Middleware
↓
Security Headers Middleware
↓
Cache Middleware (if enabled)
↓
Audit Logging Middleware
↓
Authentication Middleware
↓
Permission Middleware
↓
Handler
↓
Service
↓
Database
↓
Response
```
### 4.2 Error Handling Flow
```
Error Occurred
↓
Service Layer Error
↓
Handler Error Handling
↓
Error Response Formatting
↓
HTTP Error Response
↓
Frontend Error Handling
↓
User Notification
```
## 5. Background Services
### 5.1 Monitoring Services
- **Disk Monitor**: Syncs disk information every 5 minutes
- **ZFS Pool Monitor**: Syncs ZFS pool status every 2 minutes
- **Metrics Service**: Collects system metrics every 30 seconds
- **Alert Rule Engine**: Continuously evaluates alert rules
### 5.2 Event System
- **Event Hub**: Broadcasts events to subscribers
- **Metrics Broadcaster**: Broadcasts metrics to WebSocket clients
- **Alert Service**: Processes alerts and notifications
## 6. Data Flow Patterns
### 6.1 Read Operations
```
Frontend Query
↓
API Call
↓
Handler
↓
Service
↓
Database Query
↓
Response
↓
React Query Cache
↓
UI Update
```
### 6.2 Write Operations
```
Frontend Mutation
↓
API Call
↓
Handler (Validation)
↓
Service (Business Logic)
↓
Database Transaction
↓
Cache Invalidation
↓
Response
↓
React Query Invalidation
↓
UI Update
```
## 7. Integration Points
### 7.1 External System Integrations
- **ZFS**: Command-line tools (`zpool`, `zfs`)
- **SCST**: Configuration files and commands
- **Bacula/Bareos**: Database and `bconsole` commands
- **MHVTL**: Configuration and control
- **Systemd**: Service management
### 7.2 Integration Patterns
- **Command Execution**: Execute system commands
- **File Operations**: Read/write configuration files
- **Database Access**: Direct database queries (Bacula)
- **API Calls**: HTTP API calls (future)
## 8. Security Architecture
### 8.1 Authentication Flow
```
Login Request
↓
Credential Validation
↓
JWT Token Generation
↓
Token Response
↓
Token Storage (Frontend)
↓
Token in Request Headers
↓
Token Validation (Middleware)
↓
Request Processing
```
### 8.2 Authorization Flow
```
Authenticated Request
↓
User Role Retrieval
↓
Permission Check
↓
Resource Access Check
↓
Request Processing or Denial
```
## 9. Caching Strategy
### 9.1 Response Caching
- **Cacheable Endpoints**: GET requests only
- **Cache Keys**: Based on URL and query parameters
- **TTL**: Configurable per endpoint
- **Invalidation**: On write operations
### 9.2 Frontend Caching
- **React Query**: Automatic caching and invalidation
- **Stale Time**: 5 minutes default
- **Cache Time**: 30 minutes default
## 10. Logging Architecture
### 10.1 Log Levels
- **DEBUG**: Detailed debugging information
- **INFO**: General informational messages
- **WARN**: Warning messages
- **ERROR**: Error messages
### 10.2 Log Structure
- **Structured Logging**: JSON format
- **Fields**: Timestamp, level, message, context
- **Audit Logs**: Separate audit log table
## 11. Error Handling Architecture
### 11.1 Error Types
- **Validation Errors**: 400 Bad Request
- **Authentication Errors**: 401 Unauthorized
- **Authorization Errors**: 403 Forbidden
- **Not Found Errors**: 404 Not Found
- **Server Errors**: 500 Internal Server Error
### 11.2 Error Response Format
```json
{
"error": "Error message",
"code": "ERROR_CODE",
"details": {}
}
```

385
docs/alpha/sds/SDS-02-Database-Design.md Normal file
View File

@@ -0,0 +1,385 @@
# SDS-02: Database Design
## 1. Database Overview
### 1.1 Database System
- **Type**: PostgreSQL 14+
- **Encoding**: UTF-8
- **Connection Pooling**: pgxpool
- **Migrations**: Custom migration system
### 1.2 Database Schema Organization
- **Tables**: Organized by domain (users, storage, shares, etc.)
- **Indexes**: Performance indexes on foreign keys and frequently queried columns
- **Constraints**: Foreign keys, unique constraints, check constraints
## 2. Core Tables
### 2.1 Users & Authentication
```sql
users (
id UUID PRIMARY KEY,
username VARCHAR(255) UNIQUE NOT NULL,
email VARCHAR(255) UNIQUE,
password_hash VARCHAR(255) NOT NULL,
is_active BOOLEAN DEFAULT true,
created_at TIMESTAMP,
updated_at TIMESTAMP
)
roles (
id UUID PRIMARY KEY,
name VARCHAR(255) UNIQUE NOT NULL,
description TEXT,
created_at TIMESTAMP,
updated_at TIMESTAMP
)
permissions (
id UUID PRIMARY KEY,
resource VARCHAR(255) NOT NULL,
action VARCHAR(255) NOT NULL,
description TEXT,
UNIQUE(resource, action)
)
user_roles (
user_id UUID REFERENCES users(id),
role_id UUID REFERENCES roles(id),
PRIMARY KEY (user_id, role_id)
)
role_permissions (
role_id UUID REFERENCES roles(id),
permission_id UUID REFERENCES permissions(id),
PRIMARY KEY (role_id, permission_id)
)
groups (
id UUID PRIMARY KEY,
name VARCHAR(255) UNIQUE NOT NULL,
description TEXT,
created_at TIMESTAMP,
updated_at TIMESTAMP
)
user_groups (
user_id UUID REFERENCES users(id),
group_id UUID REFERENCES groups(id),
PRIMARY KEY (user_id, group_id)
)
```
### 2.2 Storage Tables
```sql
zfs_pools (
id UUID PRIMARY KEY,
name VARCHAR(255) UNIQUE NOT NULL,
description TEXT,
raid_level VARCHAR(50),
status VARCHAR(50),
total_capacity_bytes BIGINT,
used_capacity_bytes BIGINT,
health_status VARCHAR(50),
created_at TIMESTAMP,
updated_at TIMESTAMP,
created_by UUID REFERENCES users(id)
)
zfs_datasets (
id UUID PRIMARY KEY,
name VARCHAR(255) NOT NULL,
pool_name VARCHAR(255) REFERENCES zfs_pools(name),
type VARCHAR(50),
mount_point VARCHAR(255),
used_bytes BIGINT,
available_bytes BIGINT,
compression VARCHAR(50),
quota BIGINT,
reservation BIGINT,
created_at TIMESTAMP,
UNIQUE(pool_name, name)
)
physical_disks (
id UUID PRIMARY KEY,
device_path VARCHAR(255) UNIQUE NOT NULL,
vendor VARCHAR(255),
model VARCHAR(255),
serial_number VARCHAR(255),
size_bytes BIGINT,
type VARCHAR(50),
status VARCHAR(50),
last_synced_at TIMESTAMP
)
storage_repositories (
id UUID PRIMARY KEY,
name VARCHAR(255) UNIQUE NOT NULL,
type VARCHAR(50),
path VARCHAR(255),
capacity_bytes BIGINT,
used_bytes BIGINT,
created_at TIMESTAMP,
updated_at TIMESTAMP
)
```
### 2.3 Shares Tables
```sql
shares (
id UUID PRIMARY KEY,
dataset_id UUID REFERENCES zfs_datasets(id),
share_type VARCHAR(50),
nfs_enabled BOOLEAN DEFAULT false,
nfs_options TEXT,
nfs_clients TEXT[],
smb_enabled BOOLEAN DEFAULT false,
smb_share_name VARCHAR(255),
smb_path VARCHAR(255),
smb_comment TEXT,
smb_guest_ok BOOLEAN DEFAULT false,
smb_read_only BOOLEAN DEFAULT false,
smb_browseable BOOLEAN DEFAULT true,
is_active BOOLEAN DEFAULT true,
created_at TIMESTAMP,
updated_at TIMESTAMP,
created_by UUID REFERENCES users(id)
)
```
### 2.4 iSCSI Tables
```sql
iscsi_targets (
id UUID PRIMARY KEY,
name VARCHAR(255) UNIQUE NOT NULL,
alias VARCHAR(255),
enabled BOOLEAN DEFAULT true,
created_at TIMESTAMP,
updated_at TIMESTAMP,
created_by UUID REFERENCES users(id)
)
iscsi_luns (
id UUID PRIMARY KEY,
target_id UUID REFERENCES iscsi_targets(id),
lun_number INTEGER,
device_path VARCHAR(255),
size_bytes BIGINT,
created_at TIMESTAMP
)
iscsi_initiators (
id UUID PRIMARY KEY,
iqn VARCHAR(255) UNIQUE NOT NULL,
created_at TIMESTAMP
)
target_initiators (
target_id UUID REFERENCES iscsi_targets(id),
initiator_id UUID REFERENCES iscsi_initiators(id),
PRIMARY KEY (target_id, initiator_id)
)
```
### 2.5 Tape Tables
```sql
vtl_libraries (
id UUID PRIMARY KEY,
name VARCHAR(255) UNIQUE NOT NULL,
vendor VARCHAR(255),
model VARCHAR(255),
drive_count INTEGER,
slot_count INTEGER,
status VARCHAR(50),
created_at TIMESTAMP,
updated_at TIMESTAMP,
created_by UUID REFERENCES users(id)
)
physical_libraries (
id UUID PRIMARY KEY,
vendor VARCHAR(255),
model VARCHAR(255),
serial_number VARCHAR(255),
drive_count INTEGER,
slot_count INTEGER,
discovered_at TIMESTAMP
)
```
### 2.6 Backup Tables
```sql
backup_jobs (
id UUID PRIMARY KEY,
name VARCHAR(255) UNIQUE NOT NULL,
client_id INTEGER,
fileset_id INTEGER,
schedule VARCHAR(255),
storage_pool_id INTEGER,
enabled BOOLEAN DEFAULT true,
created_at TIMESTAMP,
updated_at TIMESTAMP,
created_by UUID REFERENCES users(id)
)
```
### 2.7 Monitoring Tables
```sql
alerts (
id UUID PRIMARY KEY,
rule_id UUID,
severity VARCHAR(50),
source VARCHAR(255),
message TEXT,
status VARCHAR(50),
acknowledged_at TIMESTAMP,
resolved_at TIMESTAMP,
created_at TIMESTAMP
)
alert_rules (
id UUID PRIMARY KEY,
name VARCHAR(255) UNIQUE NOT NULL,
description TEXT,
source VARCHAR(255),
condition_type VARCHAR(255),
condition_config JSONB,
severity VARCHAR(50),
enabled BOOLEAN DEFAULT true,
created_at TIMESTAMP,
updated_at TIMESTAMP
)
```
### 2.8 Audit Tables
```sql
audit_logs (
id UUID PRIMARY KEY,
user_id UUID REFERENCES users(id),
action VARCHAR(255),
resource_type VARCHAR(255),
resource_id VARCHAR(255),
method VARCHAR(10),
path VARCHAR(255),
ip_address VARCHAR(45),
user_agent TEXT,
request_body JSONB,
response_status INTEGER,
created_at TIMESTAMP
)
```
### 2.9 Task Tables
```sql
tasks (
id UUID PRIMARY KEY,
type VARCHAR(255),
status VARCHAR(50),
progress INTEGER,
result JSONB,
error_message TEXT,
created_at TIMESTAMP,
updated_at TIMESTAMP,
completed_at TIMESTAMP,
created_by UUID REFERENCES users(id)
)
```
## 3. Indexes
### 3.1 Performance Indexes
```sql
-- Users
CREATE INDEX idx_users_username ON users(username);
CREATE INDEX idx_users_email ON users(email);
-- Storage
CREATE INDEX idx_zfs_pools_name ON zfs_pools(name);
CREATE INDEX idx_zfs_datasets_pool_name ON zfs_datasets(pool_name);
-- Shares
CREATE INDEX idx_shares_dataset_id ON shares(dataset_id);
CREATE INDEX idx_shares_created_by ON shares(created_by);
-- iSCSI
CREATE INDEX idx_iscsi_targets_name ON iscsi_targets(name);
CREATE INDEX idx_iscsi_luns_target_id ON iscsi_luns(target_id);
-- Monitoring
CREATE INDEX idx_alerts_status ON alerts(status);
CREATE INDEX idx_alerts_created_at ON alerts(created_at);
CREATE INDEX idx_audit_logs_user_id ON audit_logs(user_id);
CREATE INDEX idx_audit_logs_created_at ON audit_logs(created_at);
```
## 4. Migrations
### 4.1 Migration System
- **Location**: `db/migrations/`
- **Naming**: `NNN_description.sql`
- **Execution**: Sequential execution on startup
- **Version Tracking**: `schema_migrations` table
### 4.2 Migration Files
- `001_initial_schema.sql`: Core tables
- `002_storage_and_tape_schema.sql`: Storage and tape tables
- `003_performance_indexes.sql`: Performance indexes
- `004_add_zfs_pools_table.sql`: ZFS pools
- `005_add_zfs_datasets_table.sql`: ZFS datasets
- `006_add_zfs_shares_and_iscsi.sql`: Shares and iSCSI
- `007_add_vendor_to_vtl_libraries.sql`: VTL updates
- `008_add_user_groups.sql`: User groups
- `009_backup_jobs_schema.sql`: Backup jobs
- `010_add_backup_permissions.sql`: Backup permissions
- `011_sync_bacula_jobs_function.sql`: Bacula sync function
## 5. Data Relationships
### 5.1 Entity Relationships
- **Users** → **Roles** (many-to-many)
- **Roles** → **Permissions** (many-to-many)
- **Users** → **Groups** (many-to-many)
- **ZFS Pools** → **ZFS Datasets** (one-to-many)
- **ZFS Datasets** → **Shares** (one-to-many)
- **iSCSI Targets** → **LUNs** (one-to-many)
- **iSCSI Targets** → **Initiators** (many-to-many)
## 6. Data Integrity
### 6.1 Constraints
- **Primary Keys**: UUID primary keys for all tables
- **Foreign Keys**: Referential integrity
- **Unique Constraints**: Unique usernames, emails, names
- **Check Constraints**: Valid status values, positive numbers
### 6.2 Cascading Rules
- **ON DELETE CASCADE**: Child records deleted with parent
- **ON DELETE RESTRICT**: Prevent deletion if referenced
- **ON UPDATE CASCADE**: Update foreign keys on parent update
## 7. Query Optimization
### 7.1 Query Patterns
- **Eager Loading**: Join related data when needed
- **Pagination**: Limit and offset for large datasets
- **Filtering**: WHERE clauses for filtering
- **Sorting**: ORDER BY for sorted results
### 7.2 Caching Strategy
- **Query Result Caching**: Cache frequently accessed queries
- **Cache Invalidation**: Invalidate on write operations
- **TTL**: Time-to-live for cached data
## 8. Backup & Recovery
### 8.1 Backup Strategy
- **Regular Backups**: Daily database backups
- **Point-in-Time Recovery**: WAL archiving
- **Backup Retention**: 30 days retention
### 8.2 Recovery Procedures
- **Full Restore**: Restore from backup
- **Point-in-Time**: Restore to specific timestamp
- **Selective Restore**: Restore specific tables

286
docs/alpha/sds/SDS-03-API-Design.md Normal file
View File

@@ -0,0 +1,286 @@
# SDS-03: API Design
## 1. API Overview
### 1.1 API Style
- **RESTful**: Resource-based API design
- **Versioning**: `/api/v1/` prefix
- **Content-Type**: `application/json`
- **Authentication**: JWT Bearer tokens
### 1.2 API Base URL
```
http://localhost:8080/api/v1
```
## 2. Authentication API
### 2.1 Endpoints
```
POST /auth/login
POST /auth/logout
GET /auth/me
```
### 2.2 Request/Response Examples
#### Login
```http
POST /api/v1/auth/login
Content-Type: application/json
{
"username": "admin",
"password": "password"
}
Response: 200 OK
{
"token": "eyJhbGciOiJIUzI1NiIs...",
"user": {
"id": "uuid",
"username": "admin",
"email": "admin@example.com",
"roles": ["admin"]
}
}
```
#### Get Current User
```http
GET /api/v1/auth/me
Authorization: Bearer <token>
Response: 200 OK
{
"id": "uuid",
"username": "admin",
"email": "admin@example.com",
"roles": ["admin"],
"permissions": ["storage:read", "storage:write", ...]
}
```
## 3. Storage API
### 3.1 ZFS Pools
```
GET /storage/zfs/pools
GET /storage/zfs/pools/:id
POST /storage/zfs/pools
DELETE /storage/zfs/pools/:id
POST /storage/zfs/pools/:id/spare
```
### 3.2 ZFS Datasets
```
GET /storage/zfs/pools/:id/datasets
POST /storage/zfs/pools/:id/datasets
DELETE /storage/zfs/pools/:id/datasets/:dataset
```
### 3.3 Request/Response Examples
#### Create ZFS Pool
```http
POST /api/v1/storage/zfs/pools
Content-Type: application/json
{
"name": "tank",
"raid_level": "mirror",
"disks": ["/dev/sdb", "/dev/sdc"],
"compression": "lz4",
"deduplication": false
}
Response: 201 Created
{
"id": "uuid",
"name": "tank",
"status": "online",
"total_capacity_bytes": 1000000000000,
"created_at": "2025-01-01T00:00:00Z"
}
```
## 4. Shares API
### 4.1 Endpoints
```
GET /shares
GET /shares/:id
POST /shares
PUT /shares/:id
DELETE /shares/:id
```
### 4.2 Request/Response Examples
#### Create Share
```http
POST /api/v1/shares
Content-Type: application/json
{
"dataset_id": "uuid",
"share_type": "both",
"nfs_enabled": true,
"nfs_clients": ["192.168.1.0/24"],
"smb_enabled": true,
"smb_share_name": "shared",
"smb_path": "/mnt/tank/shared",
"smb_guest_ok": false,
"smb_read_only": false
}
Response: 201 Created
{
"id": "uuid",
"dataset_id": "uuid",
"share_type": "both",
"nfs_enabled": true,
"smb_enabled": true,
"created_at": "2025-01-01T00:00:00Z"
}
```
## 5. iSCSI API
### 5.1 Endpoints
```
GET /scst/targets
GET /scst/targets/:id
POST /scst/targets
DELETE /scst/targets/:id
POST /scst/targets/:id/luns
DELETE /scst/targets/:id/luns/:lunId
POST /scst/targets/:id/initiators
GET /scst/initiators
POST /scst/config/apply
```
## 6. System API
### 6.1 Endpoints
```
GET /system/services
GET /system/services/:name
POST /system/services/:name/restart
GET /system/services/:name/logs
GET /system/interfaces
PUT /system/interfaces/:name
GET /system/ntp
POST /system/ntp
GET /system/logs
GET /system/network/throughput
POST /system/execute
POST /system/support-bundle
```
## 7. Monitoring API
### 7.1 Endpoints
```
GET /monitoring/metrics
GET /monitoring/health
GET /monitoring/alerts
GET /monitoring/alerts/:id
POST /monitoring/alerts/:id/acknowledge
POST /monitoring/alerts/:id/resolve
GET /monitoring/rules
POST /monitoring/rules
```
## 8. IAM API
### 8.1 Endpoints
```
GET /iam/users
GET /iam/users/:id
POST /iam/users
PUT /iam/users/:id
DELETE /iam/users/:id
GET /iam/roles
GET /iam/roles/:id
POST /iam/roles
PUT /iam/roles/:id
DELETE /iam/roles/:id
GET /iam/permissions
GET /iam/groups
```
## 9. Error Responses
### 9.1 Error Format
```json
{
"error": "Error message",
"code": "ERROR_CODE",
"details": {
"field": "validation error"
}
}
```
### 9.2 HTTP Status Codes
- **200 OK**: Success
- **201 Created**: Resource created
- **400 Bad Request**: Validation error
- **401 Unauthorized**: Authentication required
- **403 Forbidden**: Permission denied
- **404 Not Found**: Resource not found
- **500 Internal Server Error**: Server error
## 10. Pagination
### 10.1 Pagination Parameters
```
GET /api/v1/resource?page=1&limit=20
```
### 10.2 Pagination Response
```json
{
"data": [...],
"pagination": {
"page": 1,
"limit": 20,
"total": 100,
"total_pages": 5
}
}
```
## 11. Filtering & Sorting
### 11.1 Filtering
```
GET /api/v1/resource?status=active&type=filesystem
```
### 11.2 Sorting
```
GET /api/v1/resource?sort=name&order=asc
```
## 12. Rate Limiting
### 12.1 Rate Limits
- **Default**: 100 requests per minute per IP
- **Authenticated**: 200 requests per minute per user
- **Headers**: `X-RateLimit-Limit`, `X-RateLimit-Remaining`
## 13. Caching
### 13.1 Cache Headers
- **Cache-Control**: `max-age=300` for GET requests
- **ETag**: Entity tags for cache validation
- **Last-Modified**: Last modification time
### 13.2 Cache Invalidation
- **On Write**: Invalidate related cache entries
- **Manual**: Clear cache via admin endpoint

224
docs/alpha/sds/SDS-04-Security-Design.md Normal file
View File

@@ -0,0 +1,224 @@
# SDS-04: Security Design
## 1. Security Overview
### 1.1 Security Principles
- **Defense in Depth**: Multiple layers of security
- **Principle of Least Privilege**: Minimum required permissions
- **Secure by Default**: Secure default configurations
- **Input Validation**: Validate all inputs
- **Output Encoding**: Encode all outputs
## 2. Authentication
### 2.1 Authentication Method
- **JWT Tokens**: JSON Web Tokens for stateless authentication
- **Token Expiration**: Configurable expiration time
- **Token Refresh**: Refresh token mechanism (future)
### 2.2 Password Security
- **Hashing**: bcrypt with cost factor 10
- **Password Requirements**: Minimum length, complexity
- **Password Storage**: Hashed passwords only, never plaintext
### 2.3 Session Management
- **Stateless**: No server-side session storage
- **Token Storage**: Secure storage in frontend (localStorage/sessionStorage)
- **Token Validation**: Validate on every request
## 3. Authorization
### 3.1 Role-Based Access Control (RBAC)
- **Roles**: Admin, Operator, ReadOnly
- **Permissions**: Resource-based permissions (storage:read, storage:write)
- **Role Assignment**: Users assigned to roles
- **Permission Inheritance**: Permissions inherited from roles
### 3.2 Permission Model
```
Resource:Action
Examples:
- storage:read
- storage:write
- iscsi:read
- iscsi:write
- backup:read
- backup:write
- system:read
- system:write
```
### 3.3 Permission Checking
- **Middleware**: Permission middleware checks on protected routes
- **Handler Level**: Additional checks in handlers if needed
- **Service Level**: Business logic permission checks
## 4. Input Validation
### 4.1 Validation Layers
1. **Frontend**: Client-side validation
2. **Handler**: Request validation
3. **Service**: Business logic validation
4. **Database**: Constraint validation
### 4.2 Validation Rules
- **Required Fields**: Check for required fields
- **Type Validation**: Validate data types
- **Format Validation**: Validate formats (email, IP, etc.)
- **Range Validation**: Validate numeric ranges
- **Length Validation**: Validate string lengths
### 4.3 SQL Injection Prevention
- **Parameterized Queries**: Use parameterized queries only
- **No String Concatenation**: Never concatenate SQL strings
- **Input Sanitization**: Sanitize all inputs
## 5. Output Encoding
### 5.1 XSS Prevention
- **HTML Encoding**: Encode HTML in responses
- **JSON Encoding**: Proper JSON encoding
- **Content Security Policy**: CSP headers
### 5.2 Response Headers
```
Content-Security-Policy: default-src 'self'
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
```
## 6. HTTPS & TLS
### 6.1 TLS Configuration
- **TLS Version**: TLS 1.2 minimum
- **Cipher Suites**: Strong cipher suites only
- **Certificate**: Valid SSL certificate
### 6.2 HTTPS Enforcement
- **Redirect HTTP to HTTPS**: Force HTTPS
- **HSTS**: HTTP Strict Transport Security
## 7. Rate Limiting
### 7.1 Rate Limit Strategy
- **IP-Based**: Rate limit by IP address
- **User-Based**: Rate limit by authenticated user
- **Endpoint-Based**: Different limits per endpoint
### 7.2 Rate Limit Configuration
- **Default**: 100 requests/minute
- **Authenticated**: 200 requests/minute
- **Strict Endpoints**: Lower limits for sensitive endpoints
## 8. Audit Logging
### 8.1 Audit Events
- **Authentication**: Login, logout, failed login
- **Authorization**: Permission denied events
- **Data Access**: Read operations (configurable)
- **Data Modification**: Create, update, delete operations
- **System Actions**: System configuration changes
### 8.2 Audit Log Format
```json
{
"id": "uuid",
"user_id": "uuid",
"action": "CREATE_SHARE",
"resource_type": "share",
"resource_id": "uuid",
"method": "POST",
"path": "/api/v1/shares",
"ip_address": "192.168.1.100",
"user_agent": "Mozilla/5.0...",
"request_body": {...},
"response_status": 201,
"created_at": "2025-01-01T00:00:00Z"
}
```
## 9. Error Handling
### 9.1 Error Information
- **Public Errors**: Safe error messages for users
- **Private Errors**: Detailed errors in logs only
- **No Stack Traces**: Never expose stack traces to users
### 9.2 Error Logging
- **Log All Errors**: Log all errors with context
- **Sensitive Data**: Never log passwords, tokens, secrets
- **Error Tracking**: Track error patterns
## 10. File Upload Security
### 10.1 Upload Restrictions
- **File Types**: Whitelist allowed file types
- **File Size**: Maximum file size limits
- **File Validation**: Validate file contents
### 10.2 Storage Security
- **Secure Storage**: Store in secure location
- **Access Control**: Restrict file access
- **Virus Scanning**: Scan uploaded files (future)
## 11. API Security
### 11.1 API Authentication
- **Bearer Tokens**: JWT in Authorization header
- **Token Validation**: Validate on every request
- **Token Expiration**: Enforce token expiration
### 11.2 API Rate Limiting
- **Per IP**: Rate limit by IP address
- **Per User**: Rate limit by authenticated user
- **Per Endpoint**: Different limits per endpoint
## 12. Database Security
### 12.1 Database Access
- **Connection Security**: Encrypted connections
- **Credentials**: Secure credential storage
- **Least Privilege**: Database user with minimum privileges
### 12.2 Data Encryption
- **At Rest**: Database encryption (future)
- **In Transit**: TLS for database connections
- **Sensitive Data**: Encrypt sensitive fields
## 13. System Security
### 13.1 Command Execution
- **Whitelist**: Only allow whitelisted commands
- **Input Validation**: Validate command inputs
- **Output Sanitization**: Sanitize command outputs
### 13.2 File System Access
- **Path Validation**: Validate all file paths
- **Access Control**: Restrict file system access
- **Symlink Protection**: Prevent symlink attacks
## 14. Security Headers
### 14.1 HTTP Security Headers
```
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block
Content-Security-Policy: default-src 'self'
Strict-Transport-Security: max-age=31536000
Referrer-Policy: strict-origin-when-cross-origin
```
## 15. Security Monitoring
### 15.1 Security Events
- **Failed Logins**: Monitor failed login attempts
- **Permission Denials**: Monitor permission denials
- **Suspicious Activity**: Detect suspicious patterns
### 15.2 Alerting
- **Security Alerts**: Alert on security events
- **Thresholds**: Alert thresholds for suspicious activity
- **Notification**: Notify administrators

294
docs/alpha/sds/SDS-05-Integration-Design.md Normal file
View File

@@ -0,0 +1,294 @@
# SDS-05: Integration Design
## 1. Integration Overview
### 1.1 External Systems
Calypso integrates with several external systems:
- **ZFS**: Zettabyte File System for storage management
- **SCST**: SCSI target subsystem for iSCSI
- **Bacula/Bareos**: Backup software
- **MHVTL**: Virtual Tape Library emulation
- **Systemd**: Service management
- **PostgreSQL**: Database system
## 2. ZFS Integration
### 2.1 Integration Method
- **Command Execution**: Execute `zpool` and `zfs` commands
- **Output Parsing**: Parse command output
- **Error Handling**: Handle command errors
### 2.2 ZFS Commands
```bash
# Pool operations
zpool create <pool> <disks>
zpool list
zpool status <pool>
zpool destroy <pool>
# Dataset operations
zfs create <dataset>
zfs list
zfs destroy <dataset>
zfs snapshot <dataset>@<snapshot>
zfs clone <snapshot> <clone>
zfs rollback <snapshot>
```
### 2.3 Data Synchronization
- **Pool Monitor**: Background service syncs pool status every 2 minutes
- **Dataset Monitor**: Real-time dataset information
- **ARC Stats**: Real-time ARC statistics
## 3. SCST Integration
### 3.1 Integration Method
- **Configuration Files**: Read/write SCST configuration files
- **Command Execution**: Execute SCST admin commands
- **Config Apply**: Apply configuration changes
### 3.2 SCST Operations
```bash
# Target management
scstadmin -add_target <target>
scstadmin -enable_target <target>
scstadmin -disable_target <target>
scstadmin -remove_target <target>
# LUN management
scstadmin -add_lun <lun> -driver <driver> -target <target>
scstadmin -remove_lun <lun> -driver <driver> -target <target>
# Initiator management
scstadmin -add_init <initiator> -driver <driver> -target <target>
scstadmin -remove_init <initiator> -driver <driver> -target <target>
# Config apply
scstadmin -write_config /etc/scst.conf
```
### 3.3 Configuration File Format
- **Location**: `/etc/scst.conf`
- **Format**: SCST configuration syntax
- **Backup**: Backup before modifications
## 4. Bacula/Bareos Integration
### 4.1 Integration Methods
- **Database Access**: Direct PostgreSQL access to Bacula database
- **Bconsole Commands**: Execute commands via `bconsole`
- **Job Synchronization**: Sync jobs from Bacula database
### 4.2 Database Schema
- **Tables**: Jobs, Clients, Filesets, Pools, Volumes, Media
- **Queries**: SQL queries to retrieve backup information
- **Updates**: Update job status, volume information
### 4.3 Bconsole Commands
```bash
# Job operations
run job=<job_name>
status job=<job_id>
list jobs
list files jobid=<job_id>
# Client operations
list clients
status client=<client_name>
# Pool operations
list pools
list volumes pool=<pool_name>
# Storage operations
list storage
status storage=<storage_name>
```
### 4.4 Job Synchronization
- **Background Sync**: Periodic sync from Bacula database
- **Real-time Updates**: Update on job completion
- **Status Mapping**: Map Bacula status to Calypso status
## 5. MHVTL Integration
### 5.1 Integration Method
- **Configuration Files**: Read/write MHVTL configuration
- **Command Execution**: Execute MHVTL control commands
- **Status Monitoring**: Monitor VTL status
### 5.2 MHVTL Operations
```bash
# Library operations
vtlcmd -l <library> -s <status>
vtlcmd -l <library> -d <drive> -l <load>
vtlcmd -l <library> -d <drive> -u <unload>
# Media operations
vtlcmd -l <library> -m <media> -l <label>
```
### 5.3 Configuration Management
- **Library Configuration**: Create/update VTL library configs
- **Drive Configuration**: Configure virtual drives
- **Slot Configuration**: Configure virtual slots
## 6. Systemd Integration
### 6.1 Integration Method
- **DBus API**: Use systemd DBus API
- **Command Execution**: Execute `systemctl` commands
- **Service Status**: Query service status
### 6.2 Systemd Operations
```bash
# Service control
systemctl start <service>
systemctl stop <service>
systemctl restart <service>
systemctl status <service>
# Service information
systemctl list-units --type=service
systemctl show <service>
```
### 6.3 Service Management
- **Service Discovery**: Discover available services
- **Status Monitoring**: Monitor service status
- **Log Access**: Access service logs via journalctl
## 7. Network Interface Integration
### 7.1 Integration Method
- **System Commands**: Execute network configuration commands
- **File Operations**: Read/write network configuration files
- **Status Queries**: Query interface status
### 7.2 Network Operations
```bash
# Interface information
ip addr show
ip link show
ethtool <interface>
# Interface configuration
ip addr add <ip>/<mask> dev <interface>
ip link set <interface> up/down
```
### 7.3 Configuration Files
- **Netplan**: Ubuntu network configuration
- **NetworkManager**: NetworkManager configuration
- **ifconfig**: Legacy configuration
## 8. NTP Integration
### 8.1 Integration Method
- **Configuration Files**: Read/write NTP configuration
- **Command Execution**: Execute NTP commands
- **Status Queries**: Query NTP status
### 8.2 NTP Operations
```bash
# NTP status
ntpq -p
timedatectl status
# NTP configuration
timedatectl set-timezone <timezone>
timedatectl set-ntp <true/false>
```
### 8.3 Configuration Files
- **ntp.conf**: NTP daemon configuration
- **chrony.conf**: Chrony configuration (alternative)
## 9. Integration Patterns
### 9.1 Command Execution Pattern
```go
func executeCommand(cmd string, args []string) (string, error) {
ctx := context.Background()
output, err := exec.CommandContext(ctx, cmd, args...).CombinedOutput()
if err != nil {
return "", fmt.Errorf("command failed: %w", err)
}
return string(output), nil
}
```
### 9.2 File Operation Pattern
```go
func readConfigFile(path string) ([]byte, error) {
data, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("failed to read config: %w", err)
}
return data, nil
}
func writeConfigFile(path string, data []byte) error {
if err := os.WriteFile(path, data, 0644); err != nil {
return fmt.Errorf("failed to write config: %w", err)
}
return nil
}
```
### 9.3 Database Integration Pattern
```go
func queryBaculaDB(query string, args ...interface{}) ([]map[string]interface{}, error) {
rows, err := db.Query(query, args...)
if err != nil {
return nil, err
}
defer rows.Close()
// Process rows...
return results, nil
}
```
## 10. Error Handling
### 10.1 Command Execution Errors
- **Timeout**: Command execution timeout
- **Exit Code**: Check command exit codes
- **Output Parsing**: Handle parsing errors
### 10.2 File Operation Errors
- **Permission Errors**: Handle permission denied
- **File Not Found**: Handle missing files
- **Write Errors**: Handle write failures
### 10.3 Database Integration Errors
- **Connection Errors**: Handle database connection failures
- **Query Errors**: Handle SQL query errors
- **Transaction Errors**: Handle transaction failures
## 11. Monitoring & Health Checks
### 11.1 Integration Health
- **ZFS Health**: Monitor ZFS pool health
- **SCST Health**: Monitor SCST service status
- **Bacula Health**: Monitor Bacula service status
- **MHVTL Health**: Monitor MHVTL service status
### 11.2 Health Check Endpoints
```
GET /api/v1/health
GET /api/v1/health/zfs
GET /api/v1/health/scst
GET /api/v1/health/bacula
GET /api/v1/health/vtl
```
## 12. Future Integrations
### 12.1 Planned Integrations
- **LDAP/AD**: Directory service integration
- **Cloud Storage**: Cloud backup integration
- **Monitoring Systems**: Prometheus, Grafana integration
- **Notification Systems**: Email, Slack, PagerDuty integration