tidy up documentation for alpha release
This commit is contained in:
Warp Agent
294
docs/alpha/sds/SDS-05-Integration-Design.md
Normal file
294
docs/alpha/sds/SDS-05-Integration-Design.md
Normal file
@@ -0,0 +1,294 @@
|
||||
# SDS-05: Integration Design
|
||||
|
||||
## 1. Integration Overview
|
||||
|
||||
### 1.1 External Systems
|
||||
Calypso integrates with several external systems:
|
||||
- **ZFS**: Zettabyte File System for storage management
|
||||
- **SCST**: SCSI target subsystem for iSCSI
|
||||
- **Bacula/Bareos**: Backup software
|
||||
- **MHVTL**: Virtual Tape Library emulation
|
||||
- **Systemd**: Service management
|
||||
- **PostgreSQL**: Database system
|
||||
|
||||
## 2. ZFS Integration
|
||||
|
||||
### 2.1 Integration Method
|
||||
- **Command Execution**: Execute `zpool` and `zfs` commands
|
||||
- **Output Parsing**: Parse command output
|
||||
- **Error Handling**: Handle command errors
|
||||
|
||||
### 2.2 ZFS Commands
|
||||
```bash
|
||||
# Pool operations
|
||||
zpool create <pool> <disks>
|
||||
zpool list
|
||||
zpool status <pool>
|
||||
zpool destroy <pool>
|
||||
|
||||
# Dataset operations
|
||||
zfs create <dataset>
|
||||
zfs list
|
||||
zfs destroy <dataset>
|
||||
zfs snapshot <dataset>@<snapshot>
|
||||
zfs clone <snapshot> <clone>
|
||||
zfs rollback <snapshot>
|
||||
```
|
||||
|
||||
### 2.3 Data Synchronization
|
||||
- **Pool Monitor**: Background service syncs pool status every 2 minutes
|
||||
- **Dataset Monitor**: Real-time dataset information
|
||||
- **ARC Stats**: Real-time ARC statistics
|
||||
|
||||
## 3. SCST Integration
|
||||
|
||||
### 3.1 Integration Method
|
||||
- **Configuration Files**: Read/write SCST configuration files
|
||||
- **Command Execution**: Execute SCST admin commands
|
||||
- **Config Apply**: Apply configuration changes
|
||||
|
||||
### 3.2 SCST Operations
|
||||
```bash
|
||||
# Target management
|
||||
scstadmin -add_target <target>
|
||||
scstadmin -enable_target <target>
|
||||
scstadmin -disable_target <target>
|
||||
scstadmin -remove_target <target>
|
||||
|
||||
# LUN management
|
||||
scstadmin -add_lun <lun> -driver <driver> -target <target>
|
||||
scstadmin -remove_lun <lun> -driver <driver> -target <target>
|
||||
|
||||
# Initiator management
|
||||
scstadmin -add_init <initiator> -driver <driver> -target <target>
|
||||
scstadmin -remove_init <initiator> -driver <driver> -target <target>
|
||||
|
||||
# Config apply
|
||||
scstadmin -write_config /etc/scst.conf
|
||||
```
|
||||
|
||||
### 3.3 Configuration File Format
|
||||
- **Location**: `/etc/scst.conf`
|
||||
- **Format**: SCST configuration syntax
|
||||
- **Backup**: Backup before modifications
|
||||
|
||||
## 4. Bacula/Bareos Integration
|
||||
|
||||
### 4.1 Integration Methods
|
||||
- **Database Access**: Direct PostgreSQL access to Bacula database
|
||||
- **Bconsole Commands**: Execute commands via `bconsole`
|
||||
- **Job Synchronization**: Sync jobs from Bacula database
|
||||
|
||||
### 4.2 Database Schema
|
||||
- **Tables**: Jobs, Clients, Filesets, Pools, Volumes, Media
|
||||
- **Queries**: SQL queries to retrieve backup information
|
||||
- **Updates**: Update job status, volume information
|
||||
|
||||
### 4.3 Bconsole Commands
|
||||
```bash
|
||||
# Job operations
|
||||
run job=<job_name>
|
||||
status job=<job_id>
|
||||
list jobs
|
||||
list files jobid=<job_id>
|
||||
|
||||
# Client operations
|
||||
list clients
|
||||
status client=<client_name>
|
||||
|
||||
# Pool operations
|
||||
list pools
|
||||
list volumes pool=<pool_name>
|
||||
|
||||
# Storage operations
|
||||
list storage
|
||||
status storage=<storage_name>
|
||||
```
|
||||
|
||||
### 4.4 Job Synchronization
|
||||
- **Background Sync**: Periodic sync from Bacula database
|
||||
- **Real-time Updates**: Update on job completion
|
||||
- **Status Mapping**: Map Bacula status to Calypso status
|
||||
|
||||
## 5. MHVTL Integration
|
||||
|
||||
### 5.1 Integration Method
|
||||
- **Configuration Files**: Read/write MHVTL configuration
|
||||
- **Command Execution**: Execute MHVTL control commands
|
||||
- **Status Monitoring**: Monitor VTL status
|
||||
|
||||
### 5.2 MHVTL Operations
|
||||
```bash
|
||||
# Library operations
|
||||
vtlcmd -l <library> -s <status>
|
||||
vtlcmd -l <library> -d <drive> -l <load>
|
||||
vtlcmd -l <library> -d <drive> -u <unload>
|
||||
|
||||
# Media operations
|
||||
vtlcmd -l <library> -m <media> -l <label>
|
||||
```
|
||||
|
||||
### 5.3 Configuration Management
|
||||
- **Library Configuration**: Create/update VTL library configs
|
||||
- **Drive Configuration**: Configure virtual drives
|
||||
- **Slot Configuration**: Configure virtual slots
|
||||
|
||||
## 6. Systemd Integration
|
||||
|
||||
### 6.1 Integration Method
|
||||
- **DBus API**: Use systemd DBus API
|
||||
- **Command Execution**: Execute `systemctl` commands
|
||||
- **Service Status**: Query service status
|
||||
|
||||
### 6.2 Systemd Operations
|
||||
```bash
|
||||
# Service control
|
||||
systemctl start <service>
|
||||
systemctl stop <service>
|
||||
systemctl restart <service>
|
||||
systemctl status <service>
|
||||
|
||||
# Service information
|
||||
systemctl list-units --type=service
|
||||
systemctl show <service>
|
||||
```
|
||||
|
||||
### 6.3 Service Management
|
||||
- **Service Discovery**: Discover available services
|
||||
- **Status Monitoring**: Monitor service status
|
||||
- **Log Access**: Access service logs via journalctl
|
||||
|
||||
## 7. Network Interface Integration
|
||||
|
||||
### 7.1 Integration Method
|
||||
- **System Commands**: Execute network configuration commands
|
||||
- **File Operations**: Read/write network configuration files
|
||||
- **Status Queries**: Query interface status
|
||||
|
||||
### 7.2 Network Operations
|
||||
```bash
|
||||
# Interface information
|
||||
ip addr show
|
||||
ip link show
|
||||
ethtool <interface>
|
||||
|
||||
# Interface configuration
|
||||
ip addr add <ip>/<mask> dev <interface>
|
||||
ip link set <interface> up/down
|
||||
```
|
||||
|
||||
### 7.3 Configuration Files
|
||||
- **Netplan**: Ubuntu network configuration
|
||||
- **NetworkManager**: NetworkManager configuration
|
||||
- **ifconfig**: Legacy configuration
|
||||
|
||||
## 8. NTP Integration
|
||||
|
||||
### 8.1 Integration Method
|
||||
- **Configuration Files**: Read/write NTP configuration
|
||||
- **Command Execution**: Execute NTP commands
|
||||
- **Status Queries**: Query NTP status
|
||||
|
||||
### 8.2 NTP Operations
|
||||
```bash
|
||||
# NTP status
|
||||
ntpq -p
|
||||
timedatectl status
|
||||
|
||||
# NTP configuration
|
||||
timedatectl set-timezone <timezone>
|
||||
timedatectl set-ntp <true/false>
|
||||
```
|
||||
|
||||
### 8.3 Configuration Files
|
||||
- **ntp.conf**: NTP daemon configuration
|
||||
- **chrony.conf**: Chrony configuration (alternative)
|
||||
|
||||
## 9. Integration Patterns
|
||||
|
||||
### 9.1 Command Execution Pattern
|
||||
```go
|
||||
func executeCommand(cmd string, args []string) (string, error) {
|
||||
ctx := context.Background()
|
||||
output, err := exec.CommandContext(ctx, cmd, args...).CombinedOutput()
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("command failed: %w", err)
|
||||
}
|
||||
return string(output), nil
|
||||
}
|
||||
```
|
||||
|
||||
### 9.2 File Operation Pattern
|
||||
```go
|
||||
func readConfigFile(path string) ([]byte, error) {
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to read config: %w", err)
|
||||
}
|
||||
return data, nil
|
||||
}
|
||||
|
||||
func writeConfigFile(path string, data []byte) error {
|
||||
if err := os.WriteFile(path, data, 0644); err != nil {
|
||||
return fmt.Errorf("failed to write config: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
### 9.3 Database Integration Pattern
|
||||
```go
|
||||
func queryBaculaDB(query string, args ...interface{}) ([]map[string]interface{}, error) {
|
||||
rows, err := db.Query(query, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
// Process rows...
|
||||
return results, nil
|
||||
}
|
||||
```
|
||||
|
||||
## 10. Error Handling
|
||||
|
||||
### 10.1 Command Execution Errors
|
||||
- **Timeout**: Command execution timeout
|
||||
- **Exit Code**: Check command exit codes
|
||||
- **Output Parsing**: Handle parsing errors
|
||||
|
||||
### 10.2 File Operation Errors
|
||||
- **Permission Errors**: Handle permission denied
|
||||
- **File Not Found**: Handle missing files
|
||||
- **Write Errors**: Handle write failures
|
||||
|
||||
### 10.3 Database Integration Errors
|
||||
- **Connection Errors**: Handle database connection failures
|
||||
- **Query Errors**: Handle SQL query errors
|
||||
- **Transaction Errors**: Handle transaction failures
|
||||
|
||||
## 11. Monitoring & Health Checks
|
||||
|
||||
### 11.1 Integration Health
|
||||
- **ZFS Health**: Monitor ZFS pool health
|
||||
- **SCST Health**: Monitor SCST service status
|
||||
- **Bacula Health**: Monitor Bacula service status
|
||||
- **MHVTL Health**: Monitor MHVTL service status
|
||||
|
||||
### 11.2 Health Check Endpoints
|
||||
```
|
||||
GET /api/v1/health
|
||||
GET /api/v1/health/zfs
|
||||
GET /api/v1/health/scst
|
||||
GET /api/v1/health/bacula
|
||||
GET /api/v1/health/vtl
|
||||
```
|
||||
|
||||
## 12. Future Integrations
|
||||
|
||||
### 12.1 Planned Integrations
|
||||
- **LDAP/AD**: Directory service integration
|
||||
- **Cloud Storage**: Cloud backup integration
|
||||
- **Monitoring Systems**: Prometheus, Grafana integration
|
||||
- **Notification Systems**: Email, Slack, PagerDuty integration
|
||||
|
||||
Reference in New Issue
Block a user