calypso/docs/alpha/sds/SDS-05-Integration-Design.md

# SDS-05: Integration Design

## 1. Integration Overview

### 1.1 External Systems
Calypso integrates with several external systems:
- **ZFS**: Zettabyte File System for storage management
- **SCST**: SCSI target subsystem for iSCSI
- **Bacula/Bareos**: Backup software
- **MHVTL**: Virtual Tape Library emulation
- **Systemd**: Service management
- **PostgreSQL**: Database system

## 2. ZFS Integration

### 2.1 Integration Method
- **Command Execution**: Execute `zpool` and `zfs` commands
- **Output Parsing**: Parse command output
- **Error Handling**: Handle command errors

### 2.2 ZFS Commands
```bash
# Pool operations
zpool create <pool> <disks>
zpool list
zpool status <pool>
zpool destroy <pool>

# Dataset operations
zfs create <dataset>
zfs list
zfs destroy <dataset>
zfs snapshot <dataset>@<snapshot>
zfs clone <snapshot> <clone>
zfs rollback <snapshot>
```

### 2.3 Data Synchronization
- **Pool Monitor**: Background service syncs pool status every 2 minutes
- **Dataset Monitor**: Real-time dataset information
- **ARC Stats**: Real-time ARC statistics

## 3. SCST Integration

### 3.1 Integration Method
- **Configuration Files**: Read/write SCST configuration files
- **Command Execution**: Execute SCST admin commands
- **Config Apply**: Apply configuration changes

### 3.2 SCST Operations
```bash
# Target management
scstadmin -add_target <target>
scstadmin -enable_target <target>
scstadmin -disable_target <target>
scstadmin -remove_target <target>

# LUN management
scstadmin -add_lun <lun> -driver <driver> -target <target>
scstadmin -remove_lun <lun> -driver <driver> -target <target>

# Initiator management
scstadmin -add_init <initiator> -driver <driver> -target <target>
scstadmin -remove_init <initiator> -driver <driver> -target <target>

# Config apply
scstadmin -write_config /etc/scst.conf
```

### 3.3 Configuration File Format
- **Location**: `/etc/scst.conf`
- **Format**: SCST configuration syntax
- **Backup**: Backup before modifications

## 4. Bacula/Bareos Integration

### 4.1 Integration Methods
- **Database Access**: Direct PostgreSQL access to Bacula database
- **Bconsole Commands**: Execute commands via `bconsole`
- **Job Synchronization**: Sync jobs from Bacula database

### 4.2 Database Schema
- **Tables**: Jobs, Clients, Filesets, Pools, Volumes, Media
- **Queries**: SQL queries to retrieve backup information
- **Updates**: Update job status, volume information

### 4.3 Bconsole Commands
```bash
# Job operations
run job=<job_name>
status job=<job_id>
list jobs
list files jobid=<job_id>

# Client operations
list clients
status client=<client_name>

# Pool operations
list pools
list volumes pool=<pool_name>

# Storage operations
list storage
status storage=<storage_name>
```

### 4.4 Job Synchronization
- **Background Sync**: Periodic sync from Bacula database
- **Real-time Updates**: Update on job completion
- **Status Mapping**: Map Bacula status to Calypso status

## 5. MHVTL Integration

### 5.1 Integration Method
- **Configuration Files**: Read/write MHVTL configuration
- **Command Execution**: Execute MHVTL control commands
- **Status Monitoring**: Monitor VTL status

### 5.2 MHVTL Operations
```bash
# Library operations
vtlcmd -l <library> -s <status>
vtlcmd -l <library> -d <drive> -l <load>
vtlcmd -l <library> -d <drive> -u <unload>

# Media operations
vtlcmd -l <library> -m <media> -l <label>
```

### 5.3 Configuration Management
- **Library Configuration**: Create/update VTL library configs
- **Drive Configuration**: Configure virtual drives
- **Slot Configuration**: Configure virtual slots

## 6. Systemd Integration

### 6.1 Integration Method
- **DBus API**: Use systemd DBus API
- **Command Execution**: Execute `systemctl` commands
- **Service Status**: Query service status

### 6.2 Systemd Operations
```bash
# Service control
systemctl start <service>
systemctl stop <service>
systemctl restart <service>
systemctl status <service>

# Service information
systemctl list-units --type=service
systemctl show <service>
```

### 6.3 Service Management
- **Service Discovery**: Discover available services
- **Status Monitoring**: Monitor service status
- **Log Access**: Access service logs via journalctl

## 7. Network Interface Integration

### 7.1 Integration Method
- **System Commands**: Execute network configuration commands
- **File Operations**: Read/write network configuration files
- **Status Queries**: Query interface status

### 7.2 Network Operations
```bash
# Interface information
ip addr show
ip link show
ethtool <interface>

# Interface configuration
ip addr add <ip>/<mask> dev <interface>
ip link set <interface> up/down
```

### 7.3 Configuration Files
- **Netplan**: Ubuntu network configuration
- **NetworkManager**: NetworkManager configuration
- **ifconfig**: Legacy configuration

## 8. NTP Integration

### 8.1 Integration Method
- **Configuration Files**: Read/write NTP configuration
- **Command Execution**: Execute NTP commands
- **Status Queries**: Query NTP status

### 8.2 NTP Operations
```bash
# NTP status
ntpq -p
timedatectl status

# NTP configuration
timedatectl set-timezone <timezone>
timedatectl set-ntp <true/false>
```

### 8.3 Configuration Files
- **ntp.conf**: NTP daemon configuration
- **chrony.conf**: Chrony configuration (alternative)

## 9. Integration Patterns

### 9.1 Command Execution Pattern
```go
func executeCommand(cmd string, args []string) (string, error) {
    ctx := context.Background()
    output, err := exec.CommandContext(ctx, cmd, args...).CombinedOutput()
    if err != nil {
        return "", fmt.Errorf("command failed: %w", err)
    }
    return string(output), nil
}
```

### 9.2 File Operation Pattern
```go
func readConfigFile(path string) ([]byte, error) {
    data, err := os.ReadFile(path)
    if err != nil {
        return nil, fmt.Errorf("failed to read config: %w", err)
    }
    return data, nil
}

func writeConfigFile(path string, data []byte) error {
    if err := os.WriteFile(path, data, 0644); err != nil {
        return fmt.Errorf("failed to write config: %w", err)
    }
    return nil
}
```

### 9.3 Database Integration Pattern
```go
func queryBaculaDB(query string, args ...interface{}) ([]map[string]interface{}, error) {
    rows, err := db.Query(query, args...)
    if err != nil {
        return nil, err
    }
    defer rows.Close()
    
    // Process rows...
    return results, nil
}
```

## 10. Error Handling

### 10.1 Command Execution Errors
- **Timeout**: Command execution timeout
- **Exit Code**: Check command exit codes
- **Output Parsing**: Handle parsing errors

### 10.2 File Operation Errors
- **Permission Errors**: Handle permission denied
- **File Not Found**: Handle missing files
- **Write Errors**: Handle write failures

### 10.3 Database Integration Errors
- **Connection Errors**: Handle database connection failures
- **Query Errors**: Handle SQL query errors
- **Transaction Errors**: Handle transaction failures

## 11. Monitoring & Health Checks

### 11.1 Integration Health
- **ZFS Health**: Monitor ZFS pool health
- **SCST Health**: Monitor SCST service status
- **Bacula Health**: Monitor Bacula service status
- **MHVTL Health**: Monitor MHVTL service status

### 11.2 Health Check Endpoints
```
GET /api/v1/health
GET /api/v1/health/zfs
GET /api/v1/health/scst
GET /api/v1/health/bacula
GET /api/v1/health/vtl
```

## 12. Future Integrations

### 12.1 Planned Integrations
- **LDAP/AD**: Directory service integration
- **Cloud Storage**: Cloud backup integration
- **Monitoring Systems**: Prometheus, Grafana integration
- **Notification Systems**: Email, Slack, PagerDuty integration