calypso/docs/alpha/sds/SDS-05-Integration-Design.md

SDS-05: Integration Design

1. Integration Overview

1.1 External Systems

Calypso integrates with several external systems:

• ZFS: Zettabyte File System for storage management
• SCST: SCSI target subsystem for iSCSI
• Bacula/Bareos: Backup software
• MHVTL: Virtual Tape Library emulation
• Systemd: Service management
• PostgreSQL: Database system

2. ZFS Integration

2.1 Integration Method

• Command Execution: Execute zpool and zfs commands
• Output Parsing: Parse command output
• Error Handling: Handle command errors

2.2 ZFS Commands

# Pool operations
zpool create <pool> <disks>
zpool list
zpool status <pool>
zpool destroy <pool>

# Dataset operations
zfs create <dataset>
zfs list
zfs destroy <dataset>
zfs snapshot <dataset>@<snapshot>
zfs clone <snapshot> <clone>
zfs rollback <snapshot>

2.3 Data Synchronization

• Pool Monitor: Background service syncs pool status every 2 minutes
• Dataset Monitor: Real-time dataset information
• ARC Stats: Real-time ARC statistics

3. SCST Integration

3.1 Integration Method

• Configuration Files: Read/write SCST configuration files
• Command Execution: Execute SCST admin commands
• Config Apply: Apply configuration changes

3.2 SCST Operations

# Target management
scstadmin -add_target <target>
scstadmin -enable_target <target>
scstadmin -disable_target <target>
scstadmin -remove_target <target>

# LUN management
scstadmin -add_lun <lun> -driver <driver> -target <target>
scstadmin -remove_lun <lun> -driver <driver> -target <target>

# Initiator management
scstadmin -add_init <initiator> -driver <driver> -target <target>
scstadmin -remove_init <initiator> -driver <driver> -target <target>

# Config apply
scstadmin -write_config /etc/scst.conf

3.3 Configuration File Format

• Location: /etc/scst.conf
• Format: SCST configuration syntax
• Backup: Backup before modifications

4. Bacula/Bareos Integration

4.1 Integration Methods

• Database Access: Direct PostgreSQL access to Bacula database
• Bconsole Commands: Execute commands via bconsole
• Job Synchronization: Sync jobs from Bacula database

4.2 Database Schema

• Tables: Jobs, Clients, Filesets, Pools, Volumes, Media
• Queries: SQL queries to retrieve backup information
• Updates: Update job status, volume information

4.3 Bconsole Commands

# Job operations
run job=<job_name>
status job=<job_id>
list jobs
list files jobid=<job_id>

# Client operations
list clients
status client=<client_name>

# Pool operations
list pools
list volumes pool=<pool_name>

# Storage operations
list storage
status storage=<storage_name>

4.4 Job Synchronization

• Background Sync: Periodic sync from Bacula database
• Real-time Updates: Update on job completion
• Status Mapping: Map Bacula status to Calypso status

5. MHVTL Integration

5.1 Integration Method

• Configuration Files: Read/write MHVTL configuration
• Command Execution: Execute MHVTL control commands
• Status Monitoring: Monitor VTL status

5.2 MHVTL Operations

# Library operations
vtlcmd -l <library> -s <status>
vtlcmd -l <library> -d <drive> -l <load>
vtlcmd -l <library> -d <drive> -u <unload>

# Media operations
vtlcmd -l <library> -m <media> -l <label>

5.3 Configuration Management

• Library Configuration: Create/update VTL library configs
• Drive Configuration: Configure virtual drives
• Slot Configuration: Configure virtual slots

6. Systemd Integration

6.1 Integration Method

• DBus API: Use systemd DBus API
• Command Execution: Execute systemctl commands
• Service Status: Query service status

6.2 Systemd Operations

# Service control
systemctl start <service>
systemctl stop <service>
systemctl restart <service>
systemctl status <service>

# Service information
systemctl list-units --type=service
systemctl show <service>

6.3 Service Management

• Service Discovery: Discover available services
• Status Monitoring: Monitor service status
• Log Access: Access service logs via journalctl

7. Network Interface Integration

7.1 Integration Method

• System Commands: Execute network configuration commands
• File Operations: Read/write network configuration files
• Status Queries: Query interface status

7.2 Network Operations

# Interface information
ip addr show
ip link show
ethtool <interface>

# Interface configuration
ip addr add <ip>/<mask> dev <interface>
ip link set <interface> up/down

7.3 Configuration Files

• Netplan: Ubuntu network configuration
• NetworkManager: NetworkManager configuration
• ifconfig: Legacy configuration

8. NTP Integration

8.1 Integration Method

• Configuration Files: Read/write NTP configuration
• Command Execution: Execute NTP commands
• Status Queries: Query NTP status

8.2 NTP Operations

# NTP status
ntpq -p
timedatectl status

# NTP configuration
timedatectl set-timezone <timezone>
timedatectl set-ntp <true/false>

8.3 Configuration Files

• ntp.conf: NTP daemon configuration
• chrony.conf: Chrony configuration (alternative)

9. Integration Patterns

9.1 Command Execution Pattern

func executeCommand(cmd string, args []string) (string, error) {
    ctx := context.Background()
    output, err := exec.CommandContext(ctx, cmd, args...).CombinedOutput()
    if err != nil {
        return "", fmt.Errorf("command failed: %w", err)
    }
    return string(output), nil
}

9.2 File Operation Pattern

func readConfigFile(path string) ([]byte, error) {
    data, err := os.ReadFile(path)
    if err != nil {
        return nil, fmt.Errorf("failed to read config: %w", err)
    }
    return data, nil
}

func writeConfigFile(path string, data []byte) error {
    if err := os.WriteFile(path, data, 0644); err != nil {
        return fmt.Errorf("failed to write config: %w", err)
    }
    return nil
}

9.3 Database Integration Pattern

func queryBaculaDB(query string, args ...interface{}) ([]map[string]interface{}, error) {
    rows, err := db.Query(query, args...)
    if err != nil {
        return nil, err
    }
    defer rows.Close()
    
    // Process rows...
    return results, nil
}

10. Error Handling

10.1 Command Execution Errors

• Timeout: Command execution timeout
• Exit Code: Check command exit codes
• Output Parsing: Handle parsing errors

10.2 File Operation Errors

• Permission Errors: Handle permission denied
• File Not Found: Handle missing files
• Write Errors: Handle write failures

10.3 Database Integration Errors

• Connection Errors: Handle database connection failures
• Query Errors: Handle SQL query errors
• Transaction Errors: Handle transaction failures

11. Monitoring & Health Checks

11.1 Integration Health

• ZFS Health: Monitor ZFS pool health
• SCST Health: Monitor SCST service status
• Bacula Health: Monitor Bacula service status
• MHVTL Health: Monitor MHVTL service status

11.2 Health Check Endpoints

GET /api/v1/health
GET /api/v1/health/zfs
GET /api/v1/health/scst
GET /api/v1/health/bacula
GET /api/v1/health/vtl

12. Future Integrations

12.1 Planned Integrations

• LDAP/AD: Directory service integration
• Cloud Storage: Cloud backup integration
• Monitoring Systems: Prometheus, Grafana integration
• Notification Systems: Email, Slack, PagerDuty integration