othman.suseno/calypso
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

alpha-v1.0.0 #3

Merged
othman.suseno merged 24 commits from alpha-v1.0.0 into main 2026-04-19 16:40:01 +00:00
Conversation 0 Commits 24 Files Changed 720 +44530 -1164
720 changed files with 44530 additions and 1164 deletions
Download Patch File Download Diff File Expand all files Collapse all files

0
.scannerwork/.sonar_lock Normal file
View File

6
.scannerwork/report-task.txt Normal file
View File

@@ -0,0 +1,6 @@
projectKey=calypso
serverUrl=https://sonar.avt.data-center.id
serverVersion=25.12.0.117093
dashboardUrl=https://sonar.avt.data-center.id/dashboard?id=calypso
ceTaskId=2bb5340c-a81c-4dee-a5d2-0d498ee5e6c6
ceTaskUrl=https://sonar.avt.data-center.id/api/ce/task?id=2bb5340c-a81c-4dee-a5d2-0d498ee5e6c6

137
AGENTS.md Normal file
View File

@@ -0,0 +1,137 @@
# AGENTS
## Overview
- Languages: Go backend with `github.com/gin-gonic/gin`, `zap`, PostgreSQL migrations and REST handlers under `backend/internal`, plus a Vite+React+TypeScript frontend under `frontend/`.
- Keep the system deterministic: Go 1.22+, PostgreSQL 14+, Node 18+, npm 10+ (use `node --version`/`npm --version` to confirm).
- Secret material lives in `/etc/calypso/config.yaml` or environment variables (never commit private keys, JWT secrets, or database passwords).
- The backend expects `CALYPSO_DB_PASSWORD` and `CALYPSO_JWT_SECRET` (minimum 32 characters) before running locally or in CI.
## Environment Setup
- Run `sudo ./scripts/install-requirements.sh` from the repo root to provision system dependencies on Ubuntu-like hosts before building the backend.
- `config.yaml.example` provides defaults; copy it to `/etc/calypso/config.yaml`, then edit the file or override with the `CALYPSO_*` environment variables before launching.
- For frontend work, `npm install` from `frontend/` once before subsequent builds or `npm run dev` sessions.
## Backend Tooling (Go)
### Build & Run
- `make build` (`go build -o bin/calypso-api ./cmd/calypso-api`) produces the local binary.
- `make run` (`go run ./cmd/calypso-api -config config.yaml.example`) is the quickest local dev server; supply `-config` pointing at your working config.
- Production cross-build: `make build-linux` sets `CGO_ENABLED=0 GOOS=linux GOARCH=amd64` and strips symbols.
- `deploy/systemd/calypso-api.service` is a template—copy it to `/etc/systemd/system/`, `daemon-reload`, `enable`, and `start` to run as service.
### Testing
- `make test` runs `go test ./...` across all packages; it is the authoritative test runner today.
- To run a single test, narrow the package and pass `-run`: `cd backend && go test ./internal/auth -run '^TestHandler_Login$'` (use `.` to target the current package and regular expressions for test names).
- `make test-coverage` generates `coverage.out` and opens a coverage report via `go tool cover -html=coverage.out`.
### Lint & Format
- `make fmt` (`go fmt ./...`) keeps source gofmt-clean; run it after edits and before committing.
- `make lint` requires `golangci-lint`; install it with `make install-deps` (which calls `go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest`).
- `make deps` handles dependency downloads and tidies `go.mod`/`go.sum` when vendor changes are necessary.
## Backend Style Guidelines (Go)
### Imports
- Group imports with standard library packages first, blank line, then third-party packages (gofmt enforces this).
- Do not alias imports unless needed to disambiguate. Keep third-party import paths explicit (e.g., `github.com/gin-gonic/gin`).
- Prefer explicit package names (`config`, `database`, `logger`) rather than dot-importing.
### Formatting & Files
- Always run `gofmt` (or `make fmt`) before committing; gofmt enforces tabs for indentation and consistent formatting.
- Keep files small (100–300 lines) and break responsibilities into `internal/common`, `internal/iam`, etc., mirroring the current layout.
- Maintain doc comments on exported functions, types, and struct fields; these comments start with the identifier they describe.
### Types & Naming
- Use PascalCase (UpperCamel) for exported types/fields/methods and camelCase for internal helpers.
- Keep configuration structs descriptive (`ServerConfig`, `DatabaseConfig`) and align YAML tags with snake_case field names.
- When introducing new structs, mimic the existing tag conventions (`yaml:"field_name"`).
- Use short, meaningful receiver names (`func (h *Handler) ...`), ideally 1–3 characters, matching current files.
- Name boolean helpers `isX`/`shouldY` and prefer `cfg`/`db`/`ctx` as short local variables for configuration, database, and contexts.
### Error Handling
- Wrap errors with `fmt.Errorf("context: %w", err)` and return early. Do not swallow errors unless there is a clear action (e.g., logging + fallback).
- Use `gin.Context` to respond with the appropriate HTTP status codes (`c.JSON(http.StatusBadRequest, gin.H{...})`).
- Log warnings vs. errors via the shared `logger.Logger` (`h.logger.Warn(...)` for expected issues, `Error` for failures, `Info` for happy paths).
- Avoid `panic` in handlers; only panic during initialization (e.g., new logger). Instead, log and return a 5xx status.
- Keep timeouts and cancellation explicit using `context.WithTimeout` when communicating with databases or external systems.
### Logging & Observability
- Use `zap`-backed `logger.NewLogger`, pass service-specific names (e.g., `router`, `auth-handler`), and attach structured pairs (`"user_id", user.ID`).
- Allow log format and level to be configured via `CALYPSO_LOG_FORMAT`/`CALYPSO_LOG_LEVEL` if present.
- Prefer adding caller info and stack traces on `Error`level logs when helpful. Do not sprinkle raw `fmt.Println` or `log.Print`.
### HTTP & Routing
- All HTTP routing lives under `internal/common/router`; register handlers through route groups and middleware (auth/authZ) as seen in `router.NewRouter`.
- Use Gin middleware for sessions, JWT validation, and auditing. Keep handler methods grouped by feature (auth, iam, tasks).
- Keep handler structs simple: inject shared dependencies (`db`, `config`, `logger`) via constructors, then refer to them as fields.
- For JSON requests/responses, mirror struct field names in `json` tags (snake_case) even if Go fields are camelCase.
### Database & Migrations
- Use prepared statements or parameterized queries (see `iam` and `sessions` tables) with `$1` style placeholders.
- Run migrations automatically from `internal/common/database/migrations/` via `database.RunMigrations` at startup.
- Hunters: use `database.DB` wrapper for connection pool handling, close connections via `defer db.Close()`.
- When adding new tables, drop SQL files into `db/migrations/` and re-run migrations via the running service or a migration helper.
### Security Notes
- JWT secrets must be strong (>=32 characters). Use environment variables, never embed secrets in code or checked-in configs.
- Database passwords, API keys, and secrets are populated via env vars (`CALYPSO_DB_PASSWORD`, `CALYPSO_JWT_SECRET`, `CALYPSO_JWT_SECRET`) or `config.yaml` on the host.
- All mutating endpoints should log audit events and guard via IAM roles (see `iam` package for role resolution).
- Bacula client registration and capability pushes require the `bacula-admin` role; enforce it via `requireRole("bacula-admin")` on `/api/v1/bacula/clients` routes.
## Frontend Tooling (React + TypeScript)
### Build & Run
- Work inside `frontend/`. Start the dev server with `npm run dev` (proxies `/api` → `http://localhost:8080`).
- Production builds use `npm run build`, which runs `tsc` before `vite build` to ensure typing correctness.
- Serve production output via `npm run preview` if you need to smoke-test a `dist/` snapshot.
### Testing
- No automated frontend tests run currently. If you add a test harness (Jest/Vitest), ensure `npm test` or equivalent is added to `package.json`.
- For manual verification, interact with the dev server after logging into the backend API from `localhost:3000`.
### Lint & Formatting
- `npm run lint` runs ESLint across `.ts/.tsx` files with the `@typescript-eslint` recommended rules and `react-hooks` enforcement. It also blocks unused directives and sets `--max-warnings 0`.
- The linter requires you to keep hooks exhaustive and avoid unused vars; prefix intentionally unused arguments with `_` (e.g., `_event`).
- `tsconfig.json` sets `strict` mode, `noUnusedLocals`, `noUnusedParameters`, and `noFallthroughCasesInSwitch`. Respect these by keeping types precise and handling every branch.
## Frontend Style Guidelines (TypeScript + React)
### Imports & Aliases
- Use the `@/` alias defined in `tsconfig.json` for slices inside `src/` (e.g., `import Layout from '@/components/Layout'`).
- Import React/third-party packages at the top, followed by `@/` aliased modules, maintaining alphabetical order within each group.
- Prefer named exports for hooks and API utilities, default exports for page components (e.g., `export default function LoginPage()`).
### Formatting & Syntax
- Use single quotes for strings, omit semicolons (the repo follows Prettier/Vite defaults), and keep 2-space indentation inside JSX.
- Keep JSX clean: wrap complex class combinations and conditional fragments in template literals or helper functions, avoid inline object literal props unless necessary.
- Keep React components in kebab-case file names (e.g., `Login.tsx`, `Layout.tsx`) and align component names with file names.
### Types & Naming
- Use `interface` or `type` aliases for props and API shapes (e.g., `interface LoginRequest`), and add optional fields with `?`.
- Names for state setters, handlers, and store selectors follow camelCase (`setAuth`, `handleSubmit`, `loginMutation`).
- Use descriptive names for UI data (`navigation`, `userMenu`, `alerts`). Keep boolean flags prefixed with `is`, `has`, or `should`.
### State, Hooks, & Effects
- Prefer `useState`/`useEffect` for local UI state and `Zustand` stores (`useAuthStore`) for shared state.
- Keep `useEffect` dependency arrays explicit; avoid disabling exhaustive-deps rules unless documented.
- Manage async server interactions with TanStack Query (`useMutation`, `useQuery`) and centralize API clients (`src/api/client.ts`).
### Styling & Layout
- TailwindCSS classes are used for layout (`flex`, `gap`, `bg-...`). Keep className strings readable and group related classes together (spacing, color, typography).
- Use utility classes for responsive behavior (`lg:hidden`, `px-4`). For dynamic class toggling, compute the string in JS before passing it to `className`.
- Keep inline styles minimal; prefer CSS utilities and shared constants (colors, text sizing) defined in tailwind config.
### API & Networking
- Use `src/api/client.ts` as the single Axios instance (`baseURL: '/api/v1'`). Attach request/response interceptors for auth token injection/401 handling.
- Return typed promises from API helpers (e.g., `Promise<LoginResponse>`). Let callers handle success/errors via TanStack Query callbacks.
- Use `authApi.getMe`, `authApi.login`, etc., for all backend interactions; do not re-implement Axios calls directly in UI components.
### Error Handling
- Surface backend errors by reading `error.response?.data?.error` (as seen in `LoginPage`) and displaying friendly messages via inline UI alerts or toast components.
- Clear authentication state on 401 responses using the Axios interceptor (`useAuthStore.getState().clearAuth()`), then redirect to `/login`.
- Display loading or pending states (`loginMutation.isPending`) instead of disabling UI abruptly; provide visual feedback for async actions.
## Repository Practices
- Keep database migration files under `db/migrations/` and refer to them from `internal/common/database/migrations/` for auto-run.
- When adding commands or packages, update the relevant `Makefile` target or `package.json` script, and document the command in this AGENTS file.
- Commit `go.mod`, `go.sum`, and `package-lock.json`/`package.json` updates together with code changes.
- Avoid committing binaries (`bin/`), coverage artifacts (`coverage.out`), or built frontend output (`frontend/dist/`). These are ignored by `.gitignore` but double-check before commits.
## Cursor & Copilot Rules
- There are no `.cursor/rules/`, `.cursorrules`, or `.github/copilot-instructions.md` files in this repository. Follow the guidelines laid out above.

BIN
backend/bin/calypso-api
View File

Binary file not shown.

5
backend/cmd/calypso-api/main.go
View File

@@ -65,12 +65,13 @@ func main() {
r := router.NewRouter(cfg, db, logger)
// Create HTTP server
// Note: WriteTimeout should be 0 for WebSocket connections (they handle their own timeouts)
srv := &http.Server{
Addr: fmt.Sprintf(":%d", cfg.Server.Port),
Handler: r,
ReadTimeout: 15 * time.Second,
WriteTimeout: 15 * time.Second,
IdleTimeout: 60 * time.Second,
WriteTimeout: 0, // 0 means no timeout - needed for WebSocket connections
IdleTimeout: 120 * time.Second, // Increased for WebSocket keep-alive
}
// Setup graceful shutdown

56
backend/go.mod
View File

@@ -5,15 +5,19 @@ go 1.24.0
toolchain go1.24.11
require (
github.com/creack/pty v1.1.24
github.com/gin-gonic/gin v1.10.0
github.com/go-playground/validator/v10 v10.20.0
github.com/golang-jwt/jwt/v5 v5.2.1
github.com/google/uuid v1.6.0
github.com/gorilla/websocket v1.5.3
github.com/lib/pq v1.10.9
github.com/minio/madmin-go/v3 v3.0.110
github.com/minio/minio-go/v7 v7.0.97
github.com/stretchr/testify v1.11.1
go.uber.org/zap v1.27.0
golang.org/x/crypto v0.23.0
golang.org/x/sync v0.7.0
golang.org/x/crypto v0.37.0
golang.org/x/sync v0.15.0
golang.org/x/time v0.14.0
gopkg.in/yaml.v3 v3.0.1
)
@@ -21,29 +25,57 @@ require (
require (
github.com/bytedance/sonic v1.11.6 // indirect
github.com/bytedance/sonic/loader v0.1.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/cloudwego/base64x v0.1.4 // indirect
github.com/cloudwego/iasm v0.2.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/gabriel-vasile/mimetype v1.4.3 // indirect
github.com/gin-contrib/sse v0.1.0 // indirect
github.com/go-ini/ini v1.67.0 // indirect
github.com/go-ole/go-ole v1.3.0 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-playground/validator/v10 v10.20.0 // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/goccy/go-json v0.10.5 // indirect
github.com/golang-jwt/jwt/v4 v4.5.2 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/cpuid/v2 v2.2.7 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/klauspost/cpuid/v2 v2.2.11 // indirect
github.com/klauspost/crc32 v1.3.0 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/lufia/plan9stats v0.0.0-20250317134145-8bc96cf8fc35 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
github.com/minio/crc64nvme v1.1.0 // indirect
github.com/minio/md5-simd v1.1.2 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/pelletier/go-toml/v2 v2.2.2 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/philhofer/fwd v1.2.0 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.63.0 // indirect
github.com/prometheus/procfs v0.16.0 // indirect
github.com/prometheus/prom2json v1.4.2 // indirect
github.com/prometheus/prometheus v0.303.0 // indirect
github.com/rs/xid v1.6.0 // indirect
github.com/safchain/ethtool v0.5.10 // indirect
github.com/secure-io/sio-go v0.3.1 // indirect
github.com/shirou/gopsutil/v3 v3.24.5 // indirect
github.com/shoenig/go-m1cpu v0.1.6 // indirect
github.com/tinylib/msgp v1.3.0 // indirect
github.com/tklauser/go-sysconf v0.3.15 // indirect
github.com/tklauser/numcpus v0.10.0 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/ugorji/go/codec v1.2.12 // indirect
go.uber.org/multierr v1.10.0 // indirect
github.com/yusufpapurcu/wmi v1.2.4 // indirect
go.uber.org/multierr v1.11.0 // indirect
golang.org/x/arch v0.8.0 // indirect
golang.org/x/net v0.25.0 // indirect
golang.org/x/sys v0.20.0 // indirect
golang.org/x/text v0.15.0 // indirect
google.golang.org/protobuf v1.34.1 // indirect
golang.org/x/net v0.39.0 // indirect
golang.org/x/sys v0.34.0 // indirect
golang.org/x/text v0.26.0 // indirect
google.golang.org/protobuf v1.36.6 // indirect
)

137
backend/go.sum
View File

@@ -2,19 +2,31 @@ github.com/bytedance/sonic v1.11.6 h1:oUp34TzMlL+OY1OUWxHqsdkgC/Zfc85zGqw9siXjrc
github.com/bytedance/sonic v1.11.6/go.mod h1:LysEHSvpvDySVdC2f87zGWf6CIKJcAvqab1ZaiQtds4=
github.com/bytedance/sonic/loader v0.1.1 h1:c+e5Pt1k/cy5wMveRDyk2X4B9hF4g7an8N3zCYjJFNM=
github.com/bytedance/sonic/loader v0.1.1/go.mod h1:ncP89zfokxS5LZrJxl5z0UJcsk4M4yY2JpfqGeCtNLU=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/cloudwego/base64x v0.1.4 h1:jwCgWpFanWmN8xoIUHa2rtzmkd5J2plF/dnLS6Xd/0Y=
github.com/cloudwego/base64x v0.1.4/go.mod h1:0zlkT4Wn5C6NdauXdJRhSKRlJvmclQ1hhJgA0rcu/8w=
github.com/cloudwego/iasm v0.2.0 h1:1KNIy1I1H9hNNFEEH3DVnI4UujN+1zjpuk6gwHLTssg=
github.com/cloudwego/iasm v0.2.0/go.mod h1:8rXZaNYT2n95jn+zTI1sDr+IgcD2GVs0nlbbQPiEFhY=
github.com/creack/pty v1.1.24 h1:bJrF4RRfyJnbTJqzRLHzcGaZK1NeM5kTC9jGgovnR1s=
github.com/creack/pty v1.1.24/go.mod h1:08sCNb52WyoAwi2QDyzUCTgcvVFhUzewun7wtTfvcwE=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/gabriel-vasile/mimetype v1.4.3 h1:in2uUcidCuFcDKtdcBxlR0rJ1+fsokWf+uqxgUFjbI0=
github.com/gabriel-vasile/mimetype v1.4.3/go.mod h1:d8uq/6HKRL6CGdk+aubisF/M5GcPfT7nKyLpA0lbSSk=
github.com/gin-contrib/sse v0.1.0 h1:Y/yl/+YNO8GZSjAhjMsSuLt29uWRFHdHYUb5lYOV9qE=
github.com/gin-contrib/sse v0.1.0/go.mod h1:RHrZQHXnP2xjPF+u1gW/2HnVO7nvIa9PG3Gm+fLHvGI=
github.com/gin-gonic/gin v1.10.0 h1:nTuyha1TYqgedzytsKYqna+DfLos46nTv2ygFy86HFU=
github.com/gin-gonic/gin v1.10.0/go.mod h1:4PMNQiOhvDRa013RKVbsiNwoyezlm2rm0uX/T7kzp5Y=
github.com/go-ini/ini v1.67.0 h1:z6ZrTEZqSWOTyH2FlglNbNgARyHG8oLW9gMELqKr06A=
github.com/go-ini/ini v1.67.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
github.com/go-ole/go-ole v1.3.0 h1:Dt6ye7+vXGIKZ7Xtk4s6/xVdGDQynvom7xCFEdWr6uE=
github.com/go-ole/go-ole v1.3.0/go.mod h1:5LS6F96DhAwUc7C+1HLexzMXY1xGRSryjyPPKW6zv78=
github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s=
github.com/go-playground/assert/v2 v2.2.0/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/oXslEjJA=
@@ -23,12 +35,17 @@ github.com/go-playground/universal-translator v0.18.1 h1:Bcnm0ZwsGyWbCzImXv+pAJn
github.com/go-playground/universal-translator v0.18.1/go.mod h1:xekY+UJKNuX9WP91TpwSH2VMlDf28Uj24BCp08ZFTUY=
github.com/go-playground/validator/v10 v10.20.0 h1:K9ISHbSaI0lyB2eWMPJo+kOS/FBExVwjEviJTixqxL8=
github.com/go-playground/validator/v10 v10.20.0/go.mod h1:dbuPbCMFw/DrkbEynArYaCwl3amGuJotoKCe95atGMM=
github.com/goccy/go-json v0.10.2 h1:CrxCmQqYDkv1z7lO7Wbh2HN93uovUHgrECaO5ZrCXAU=
github.com/goccy/go-json v0.10.2/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/golang-jwt/jwt/v4 v4.5.2 h1:YtQM7lnr8iZ+j5q71MGKkNw9Mn7AjHM68uc9g5fXeUI=
github.com/golang-jwt/jwt/v4 v4.5.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
github.com/golang-jwt/jwt/v5 v5.2.1 h1:OuVbFODueb089Lh128TAcimifWaLhJwVflnrgM17wHk=
github.com/golang-jwt/jwt/v5 v5.2.1/go.mod h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk=
github.com/google/go-cmp v0.5.5 h1:Khx7svrCpmxxtHBq5j2mp/xVjsi8hQMfNLvJFAlrGgU=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
@@ -36,25 +53,75 @@ github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aN
github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.0.9/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.2.7 h1:ZWSB3igEs+d0qvnxR/ZBzXVmxkgt8DdzP6m9pfuVLDM=
github.com/klauspost/cpuid/v2 v2.2.7/go.mod h1:Lcz8mBdAVJIBVzewtcLocK12l3Y+JytZYpaMropDUws=
github.com/klauspost/cpuid/v2 v2.2.11 h1:0OwqZRYI2rFrjS4kvkDnqJkKHdHaRnCm68/DY4OxRzU=
github.com/klauspost/cpuid/v2 v2.2.11/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
github.com/klauspost/crc32 v1.3.0 h1:sSmTt3gUt81RP655XGZPElI0PelVTZ6YwCRnPSupoFM=
github.com/klauspost/crc32 v1.3.0/go.mod h1:D7kQaZhnkX/Y0tstFGf8VUzv2UofNGqCjnC3zdHB0Hw=
github.com/knz/go-libedit v1.10.1/go.mod h1:MZTVkCWyz0oBc7JOWP3wNAzd002ZbM/5hgShxwh4x8M=
github.com/kr/pretty v0.2.1 h1:Fmg33tUaq4/8ym9TJN1x7sLJnHVwhP33CNkpYV/7rwI=
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/leodido/go-urn v1.4.0 h1:WT9HwE9SGECu3lg4d/dIA+jxlljEa1/ffXKmRjqdmIQ=
github.com/leodido/go-urn v1.4.0/go.mod h1:bvxc+MVxLKB4z00jd1z+Dvzr47oO32F/QSNjSBOlFxI=
github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw=
github.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
github.com/lufia/plan9stats v0.0.0-20250317134145-8bc96cf8fc35 h1:PpXWgLPs+Fqr325bN2FD2ISlRRztXibcX6e8f5FR5Dc=
github.com/lufia/plan9stats v0.0.0-20250317134145-8bc96cf8fc35/go.mod h1:autxFIvghDt3jPTLoqZ9OZ7s9qTGNAWmYCjVFWPX/zg=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/matttproud/golang_protobuf_extensions v1.0.4 h1:mmDVorXM7PCGKw94cs5zkfA9PSy5pEvNWRP0ET0TIVo=
github.com/matttproud/golang_protobuf_extensions v1.0.4/go.mod h1:BSXmuO+STAnVfrANrmjBb36TMTDstsz7MSK+HVaYKv4=
github.com/minio/crc64nvme v1.1.0 h1:e/tAguZ+4cw32D+IO/8GSf5UVr9y+3eJcxZI2WOO/7Q=
github.com/minio/crc64nvme v1.1.0/go.mod h1:eVfm2fAzLlxMdUGc0EEBGSMmPwmXD5XiNRpnu9J3bvg=
github.com/minio/madmin-go/v3 v3.0.110 h1:FIYekj7YPc430ffpXFWiUtyut3qBt/unIAcDzJn9H5M=
github.com/minio/madmin-go/v3 v3.0.110/go.mod h1:WOe2kYmYl1OIlY2DSRHVQ8j1v4OItARQ6jGyQqcCud8=
github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
github.com/minio/minio-go/v7 v7.0.97 h1:lqhREPyfgHTB/ciX8k2r8k0D93WaFqxbJX36UZq5occ=
github.com/minio/minio-go/v7 v7.0.97/go.mod h1:re5VXuo0pwEtoNLsNuSr0RrLfT/MBtohwdaSmPPSRSk=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/pelletier/go-toml/v2 v2.2.2 h1:aYUidT7k73Pcl9nb2gScu7NSrKCSHIDE89b3+6Wq+LM=
github.com/pelletier/go-toml/v2 v2.2.2/go.mod h1:1t835xjRzz80PqgE6HHgN2JOsmgYu/h4qDAS4n929Rs=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=
github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55 h1:o4JXh1EVt9k/+g42oCprj/FisM4qX9L3sZB3upGN2ZU=
github.com/power-devops/perfstat v0.0.0-20240221224432-82ca36839d55/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
github.com/prometheus/common v0.63.0 h1:YR/EIY1o3mEFP/kZCD7iDMnLPlGyuU2Gb3HIcXnA98k=
github.com/prometheus/common v0.63.0/go.mod h1:VVFF/fBIoToEnWRVkYoXEkq3R3paCoxG9PXP74SnV18=
github.com/prometheus/procfs v0.16.0 h1:xh6oHhKwnOJKMYiYBDWmkHqQPyiY40sny36Cmx2bbsM=
github.com/prometheus/procfs v0.16.0/go.mod h1:8veyXUu3nGP7oaCxhX6yeaM5u4stL2FeMXnCqhDthZg=
github.com/prometheus/prom2json v1.4.2 h1:PxCTM+Whqi/eykO1MKsEL0p/zMpxp9ybpsmdFamw6po=
github.com/prometheus/prom2json v1.4.2/go.mod h1:zuvPm7u3epZSbXPWHny6G+o8ETgu6eAK3oPr6yFkRWE=
github.com/prometheus/prometheus v0.303.0 h1:wsNNsbd4EycMCphYnTmNY9JASBVbp7NWwJna857cGpA=
github.com/prometheus/prometheus v0.303.0/go.mod h1:8PMRi+Fk1WzopMDeb0/6hbNs9nV6zgySkU/zds5Lu3o=
github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
github.com/safchain/ethtool v0.5.10 h1:Im294gZtuf4pSGJRAOGKaASNi3wMeFaGaWuSaomedpc=
github.com/safchain/ethtool v0.5.10/go.mod h1:w9jh2Lx7YBR4UwzLkzCmWl85UY0W2uZdd7/DckVE5+c=
github.com/secure-io/sio-go v0.3.1 h1:dNvY9awjabXTYGsTF1PiCySl9Ltofk9GA3VdWlo7rRc=
github.com/secure-io/sio-go v0.3.1/go.mod h1:+xbkjDzPjwh4Axd07pRKSNriS9SCiYksWnZqdnfpQxs=
github.com/shirou/gopsutil/v3 v3.24.5 h1:i0t8kL+kQTvpAYToeuiVk3TgDeKOFioZO3Ztz/iZ9pI=
github.com/shirou/gopsutil/v3 v3.24.5/go.mod h1:bsoOS1aStSs9ErQ1WWfxllSeS1K5D+U30r2NfcubMVk=
github.com/shoenig/go-m1cpu v0.1.6 h1:nxdKQNcEB6vzgA2E2bvzKIYRuNj7XNJ4S/aRSwKzFtM=
github.com/shoenig/go-m1cpu v0.1.6/go.mod h1:1JJMcUBvfNwpq05QDQVAnx3gUHr9IYF7GNg9SUEw2VQ=
github.com/shoenig/test v0.6.4 h1:kVTaSd7WLz5WZ2IaoM0RSzRsUD+m8wRR+5qvntpn4LU=
github.com/shoenig/test v0.6.4/go.mod h1:byHiCGXqrVaflBLAMq/srcZIHynQPQgeyvkvXnjqq0k=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
@@ -68,39 +135,57 @@ github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXl
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/tinylib/msgp v1.3.0 h1:ULuf7GPooDaIlbyvgAxBV/FI7ynli6LZ1/nVUNu+0ww=
github.com/tinylib/msgp v1.3.0/go.mod h1:ykjzy2wzgrlvpDCRc4LA8UXy6D8bzMSuAF3WD57Gok0=
github.com/tklauser/go-sysconf v0.3.15 h1:VE89k0criAymJ/Os65CSn1IXaol+1wrsFHEB8Ol49K4=
github.com/tklauser/go-sysconf v0.3.15/go.mod h1:Dmjwr6tYFIseJw7a3dRLJfsHAMXZ3nEnL/aZY+0IuI4=
github.com/tklauser/numcpus v0.10.0 h1:18njr6LDBk1zuna922MgdjQuJFjrdppsZG60sHGfjso=
github.com/tklauser/numcpus v0.10.0/go.mod h1:BiTKazU708GQTYF4mB+cmlpT2Is1gLk7XVuEeem8LsQ=
github.com/twitchyliquid64/golang-asm v0.15.1 h1:SU5vSMR7hnwNxj24w34ZyCi/FmDZTkS4MhqMhdFk5YI=
github.com/twitchyliquid64/golang-asm v0.15.1/go.mod h1:a1lVb/DtPvCB8fslRZhAngC2+aY1QWCk3Cedj/Gdt08=
github.com/ugorji/go/codec v1.2.12 h1:9LC83zGrHhuUA9l16C9AHXAqEV/2wBQ4nkvumAE65EE=
github.com/ugorji/go/codec v1.2.12/go.mod h1:UNopzCgEMSXjBc6AOMqYvWC1ktqTAfzJZUZgYf6w6lg=
github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=
github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.uber.org/multierr v1.10.0 h1:S0h4aNzvfcFsC3dRF1jLoaov7oRaKqRGC/pUEJ2yvPQ=
go.uber.org/multierr v1.10.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
go.uber.org/zap v1.27.0 h1:aJMhYGrd5QSmlpLMr2MftRKl7t8J8PTZPA732ud/XR8=
go.uber.org/zap v1.27.0/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E=
golang.org/x/arch v0.0.0-20210923205945-b76863e36670/go.mod h1:5om86z9Hs0C8fWVUuoMHwpExlXzs5Tkyp9hOrfG7pp8=
golang.org/x/arch v0.8.0 h1:3wRIsP3pM4yUptoR96otTUOXI367OS0+c9eeRi9doIc=
golang.org/x/arch v0.8.0/go.mod h1:FEVrYAQjsQXMVJ1nsMoVVXPZg6p2JE2mx8psSWTDQys=
golang.org/x/crypto v0.23.0 h1:dIJU/v2J8Mdglj/8rJ6UUOM3Zc9zLZxVZwwxMooUSAI=
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
golang.org/x/net v0.25.0 h1:d/OCCoBEUq33pjydKrGQhw7IlUPI2Oylr+8qLx49kac=
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
golang.org/x/sync v0.7.0 h1:YsImfSBoP9QPYL0xyKJPq0gcaJdG3rInoqxTWbfQu9M=
golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.37.0 h1:kJNSjF/Xp7kU0iB2Z+9viTPMW4EqqsrywMXLJOOsXSE=
golang.org/x/crypto v0.37.0/go.mod h1:vg+k43peMZ0pUMhYmVAWysMK35e6ioLh3wB8ZCAfbVc=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.39.0 h1:ZCu7HMWDxpXpaiKdhzIfaltL9Lp31x/3fCP11bc6/fY=
golang.org/x/net v0.39.0/go.mod h1:X7NRbYVEA+ewNkCNyJ513WmMdQ3BineSwVtN2zD/d+E=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.15.0 h1:KWH3jNZsfyT6xfAfKiz6MRNmd46ByHDYaZ7KSkCtdW8=
golang.org/x/sync v0.15.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.20.0 h1:Od9JTbYCk261bKm4M/mw7AklTlFYIa0bIp9BgSm1S8Y=
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/text v0.15.0 h1:h1V/4gjBv8v9cjcR6+AR5+/cIYK5N/WAgiv4xlsEtAk=
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/sys v0.29.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.34.0 h1:H5Y5sJ2L2JRdyv7ROF1he/lPdvFsd0mJHFw2ThKHxLA=
golang.org/x/sys v0.34.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.26.0 h1:P42AVeLghgTYr4+xUnTRKDMqpar+PtX7KWuNQL21L8M=
golang.org/x/text v0.26.0/go.mod h1:QK15LZJUUQVJxhz7wXgxSy/CJaTFjd0G+YLonydOVQA=
golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/protobuf v1.34.1 h1:9ddQBjfCyZPOHPUiPxpYESBLc+T8P3E+Vo4IbKZgFWg=
google.golang.org/protobuf v1.34.1/go.mod h1:c6P6GXX6sHbq/GpV6MGZEdwhWPcYBgnhAHhKbcUYpos=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
google.golang.org/protobuf v1.36.6 h1:z1NpPI8ku2WgiWnf+t9wTPsn6eP1L7ksHUlkfLvd9xY=
google.golang.org/protobuf v1.36.6/go.mod h1:jduwjTPXsFjZGTmRluh+L6NjiWu7pchiJ2/5YcXBHnY=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

278
backend/internal/backup/handler.go
View File

@@ -116,3 +116,281 @@ func (h *Handler) CreateJob(c *gin.Context) {
c.JSON(http.StatusCreated, job)
}
// ExecuteBconsoleCommand executes a bconsole command
func (h *Handler) ExecuteBconsoleCommand(c *gin.Context) {
var req struct {
Command string `json:"command" binding:"required"`
}
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "command is required"})
return
}
output, err := h.service.ExecuteBconsoleCommand(c.Request.Context(), req.Command)
if err != nil {
h.logger.Error("Failed to execute bconsole command", "error", err, "command", req.Command)
c.JSON(http.StatusInternalServerError, gin.H{
"error": "failed to execute command",
"output": output,
"details": err.Error(),
})
return
}
c.JSON(http.StatusOK, gin.H{
"output": output,
})
}
// ListClients lists all backup clients with optional filters
func (h *Handler) ListClients(c *gin.Context) {
opts := ListClientsOptions{}
// Parse enabled filter
if enabledStr := c.Query("enabled"); enabledStr != "" {
enabled := enabledStr == "true"
opts.Enabled = &enabled
}
// Parse search query
opts.Search = c.Query("search")
// Parse category filter
if category := c.Query("category"); category != "" {
// Validate category
validCategories := map[string]bool{
"File": true,
"Database": true,
"Virtual": true,
}
if validCategories[category] {
opts.Category = category
}
}
clients, err := h.service.ListClients(c.Request.Context(), opts)
if err != nil {
h.logger.Error("Failed to list clients", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{
"error": "failed to list clients",
"details": err.Error(),
})
return
}
if clients == nil {
clients = []Client{}
}
c.JSON(http.StatusOK, gin.H{
"clients": clients,
"total": len(clients),
})
}
// GetDashboardStats returns dashboard statistics
func (h *Handler) GetDashboardStats(c *gin.Context) {
stats, err := h.service.GetDashboardStats(c.Request.Context())
if err != nil {
h.logger.Error("Failed to get dashboard stats", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to get dashboard stats"})
return
}
c.JSON(http.StatusOK, stats)
}
// ListStoragePools lists all storage pools
func (h *Handler) ListStoragePools(c *gin.Context) {
pools, err := h.service.ListStoragePools(c.Request.Context())
if err != nil {
h.logger.Error("Failed to list storage pools", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to list storage pools"})
return
}
if pools == nil {
pools = []StoragePool{}
}
h.logger.Info("Listed storage pools", "count", len(pools))
c.JSON(http.StatusOK, gin.H{
"pools": pools,
"total": len(pools),
})
}
// ListStorageVolumes lists all storage volumes
func (h *Handler) ListStorageVolumes(c *gin.Context) {
poolName := c.Query("pool_name")
volumes, err := h.service.ListStorageVolumes(c.Request.Context(), poolName)
if err != nil {
h.logger.Error("Failed to list storage volumes", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to list storage volumes"})
return
}
if volumes == nil {
volumes = []StorageVolume{}
}
c.JSON(http.StatusOK, gin.H{
"volumes": volumes,
"total": len(volumes),
})
}
// ListStorageDaemons lists all storage daemons
func (h *Handler) ListStorageDaemons(c *gin.Context) {
daemons, err := h.service.ListStorageDaemons(c.Request.Context())
if err != nil {
h.logger.Error("Failed to list storage daemons", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to list storage daemons"})
return
}
if daemons == nil {
daemons = []StorageDaemon{}
}
c.JSON(http.StatusOK, gin.H{
"daemons": daemons,
"total": len(daemons),
})
}
// CreateStoragePool creates a new storage pool
func (h *Handler) CreateStoragePool(c *gin.Context) {
var req CreatePoolRequest
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
pool, err := h.service.CreateStoragePool(c.Request.Context(), req)
if err != nil {
h.logger.Error("Failed to create storage pool", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusCreated, pool)
}
// DeleteStoragePool deletes a storage pool
func (h *Handler) DeleteStoragePool(c *gin.Context) {
idStr := c.Param("id")
if idStr == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "pool ID is required"})
return
}
var poolID int
if _, err := fmt.Sscanf(idStr, "%d", &poolID); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid pool ID"})
return
}
err := h.service.DeleteStoragePool(c.Request.Context(), poolID)
if err != nil {
h.logger.Error("Failed to delete storage pool", "error", err, "pool_id", poolID)
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"message": "pool deleted successfully"})
}
// CreateStorageVolume creates a new storage volume
func (h *Handler) CreateStorageVolume(c *gin.Context) {
var req CreateVolumeRequest
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
volume, err := h.service.CreateStorageVolume(c.Request.Context(), req)
if err != nil {
h.logger.Error("Failed to create storage volume", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusCreated, volume)
}
// UpdateStorageVolume updates a storage volume
func (h *Handler) UpdateStorageVolume(c *gin.Context) {
idStr := c.Param("id")
if idStr == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "volume ID is required"})
return
}
var volumeID int
if _, err := fmt.Sscanf(idStr, "%d", &volumeID); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid volume ID"})
return
}
var req UpdateVolumeRequest
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
volume, err := h.service.UpdateStorageVolume(c.Request.Context(), volumeID, req)
if err != nil {
h.logger.Error("Failed to update storage volume", "error", err, "volume_id", volumeID)
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, volume)
}
// DeleteStorageVolume deletes a storage volume
func (h *Handler) DeleteStorageVolume(c *gin.Context) {
idStr := c.Param("id")
if idStr == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "volume ID is required"})
return
}
var volumeID int
if _, err := fmt.Sscanf(idStr, "%d", &volumeID); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid volume ID"})
return
}
err := h.service.DeleteStorageVolume(c.Request.Context(), volumeID)
if err != nil {
h.logger.Error("Failed to delete storage volume", "error", err, "volume_id", volumeID)
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"message": "volume deleted successfully"})
}
// ListMedia lists all media from bconsole "list media" command
func (h *Handler) ListMedia(c *gin.Context) {
media, err := h.service.ListMedia(c.Request.Context())
if err != nil {
h.logger.Error("Failed to list media", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
if media == nil {
media = []Media{}
}
h.logger.Info("Listed media", "count", len(media))
c.JSON(http.StatusOK, gin.H{
"media": media,
"total": len(media),
})
}

1717
backend/internal/backup/service.go
View File

File diff suppressed because it is too large Load Diff

676
backend/internal/bacula/handler.go Normal file
View File

@@ -0,0 +1,676 @@
package bacula
import (
"context"
"database/sql"
"encoding/json"
"net/http"
"time"
"github.com/atlasos/calypso/internal/common/database"
"github.com/atlasos/calypso/internal/common/logger"
"github.com/atlasos/calypso/internal/iam"
"github.com/gin-gonic/gin"
"github.com/lib/pq"
"go.uber.org/zap"
)
const (
requestTimeout = 5 * time.Second
maxHistoryEntries = 10
)
type Handler struct {
db *database.DB
logger *logger.Logger
}
func NewHandler(db *database.DB, log *logger.Logger) *Handler {
return &Handler{db: db, logger: log.WithFields(zap.String("component", "bacula-handler"))}
}
type RegisterRequest struct {
Hostname string `json:"hostname" binding:"required"`
IPAddress string `json:"ip_address" binding:"required"`
AgentVersion string `json:"agent_version"`
Status string `json:"status"`
BackupTypes []string `json:"backup_types" binding:"required"`
Metadata map[string]string `json:"metadata"`
}
type UpdateCapabilitiesRequest struct {
BackupTypes []string `json:"backup_types" binding:"required"`
Notes string `json:"notes"`
}
type PingRequest struct {
Status string `json:"status"`
}
type ClientResponse struct {
ID string `json:"id"`
Hostname string `json:"hostname"`
IPAddress string `json:"ip_address"`
AgentVersion string `json:"agent_version"`
Status string `json:"status"`
BackupTypes []string `json:"backup_types"`
PendingBackupTypes []string `json:"pending_backup_types,omitempty"`
PendingRequestedBy string `json:"pending_requested_by,omitempty"`
PendingRequestedAt *time.Time `json:"pending_requested_at,omitempty"`
PendingNotes string `json:"pending_notes,omitempty"`
Metadata map[string]interface{} `json:"metadata,omitempty"`
RegisteredBy string `json:"registered_by"`
LastSeen *time.Time `json:"last_seen,omitempty"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
CapabilityHistory []CapabilityHistoryEntry `json:"capability_history,omitempty"`
}
type CapabilityHistoryEntry struct {
BackupTypes []string `json:"backup_types"`
Source string `json:"source"`
RequestedBy string `json:"requested_by,omitempty"`
RequestedAt time.Time `json:"requested_at"`
Notes string `json:"notes,omitempty"`
}
type PendingUpdateResponse struct {
BackupTypes []string `json:"backup_types"`
RequestedBy string `json:"requested_by,omitempty"`
RequestedAt time.Time `json:"requested_at"`
Notes string `json:"notes,omitempty"`
}
func (h *Handler) Register(c *gin.Context) {
var req RegisterRequest
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request payload"})
return
}
if len(req.BackupTypes) == 0 {
c.JSON(http.StatusBadRequest, gin.H{"error": "backup_types is required"})
return
}
user, err := currentUser(c)
if err != nil {
c.JSON(http.StatusUnauthorized, gin.H{"error": "authentication required"})
return
}
ctx, cancel := context.WithTimeout(c.Request.Context(), requestTimeout)
defer cancel()
backupPayload, err := json.Marshal(req.BackupTypes)
if err != nil {
h.logger.Error("failed to marshal backup types", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to encode backup types"})
return
}
var metadataPayload []byte
if len(req.Metadata) > 0 {
metadataPayload, err = json.Marshal(req.Metadata)
if err != nil {
h.logger.Error("failed to marshal metadata", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to encode metadata"})
return
}
}
status := req.Status
if status == "" {
status = "online"
}
tx, err := h.db.BeginTx(ctx, nil)
if err != nil {
h.logger.Error("failed to begin database transaction", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to register client"})
return
}
defer tx.Rollback()
var row clientRow
err = tx.QueryRowContext(ctx, `
INSERT INTO bacula_clients (
hostname, ip_address, agent_version, status, backup_types, metadata,
registered_by_user_id, last_seen, updated_at
) VALUES ($1, $2, $3, $4, $5, $6, $7, NOW(), NOW())
ON CONFLICT (hostname) DO UPDATE SET
ip_address = EXCLUDED.ip_address,
agent_version = EXCLUDED.agent_version,
status = EXCLUDED.status,
backup_types = EXCLUDED.backup_types,
metadata = COALESCE(EXCLUDED.metadata, bacula_clients.metadata),
registered_by_user_id = EXCLUDED.registered_by_user_id,
last_seen = EXCLUDED.last_seen,
updated_at = NOW()
RETURNING id, hostname, ip_address, agent_version, status, backup_types, metadata,
pending_backup_types, pending_requested_by, pending_requested_at, pending_notes,
registered_by_user_id, last_seen, created_at, updated_at
`, req.Hostname, req.IPAddress, req.AgentVersion, status, backupPayload, metadataPayload, user.ID).Scan(
&row.ID, &row.Hostname, &row.IPAddress, &row.AgentVersion, &row.Status, &row.BackupJSON,
&row.MetadataJSON, &row.PendingBackupJSON, &row.PendingRequestedBy, &row.PendingRequestedAt,
&row.PendingNotes, &row.RegisteredBy, &row.LastSeen, &row.CreatedAt, &row.UpdatedAt,
)
if err != nil {
h.logger.Error("failed to ensure bacula client", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to register client"})
return
}
resp, err := buildClientResponse(&row)
if err != nil {
h.logger.Error("failed to marshal client response", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to build response"})
return
}
if err := insertCapabilityHistory(ctx, tx, row.ID, req.BackupTypes, "agent", user.ID, "agent registration"); err != nil {
h.logger.Error("failed to record capability history", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to record capability history"})
return
}
if len(resp.PendingBackupTypes) > 0 && stringSlicesEqual(resp.PendingBackupTypes, resp.BackupTypes) {
if _, err := tx.ExecContext(ctx, `
UPDATE bacula_clients
SET pending_backup_types = NULL,
pending_requested_by = NULL,
pending_requested_at = NULL,
pending_notes = NULL,
updated_at = NOW()
WHERE id = $1
`, resp.ID); err != nil {
h.logger.Error("failed to clear pending capability update", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to update client"})
return
}
resp.PendingBackupTypes = nil
resp.PendingRequestedBy = ""
resp.PendingRequestedAt = nil
resp.PendingNotes = ""
}
if err := tx.Commit(); err != nil {
h.logger.Error("failed to commit client registration", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to register client"})
return
}
c.JSON(http.StatusOK, resp)
}
func (h *Handler) UpdateCapabilities(c *gin.Context) {
var req UpdateCapabilitiesRequest
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request payload"})
return
}
if len(req.BackupTypes) == 0 {
c.JSON(http.StatusBadRequest, gin.H{"error": "backup_types is required"})
return
}
user, err := currentUser(c)
if err != nil {
c.JSON(http.StatusUnauthorized, gin.H{"error": "authentication required"})
return
}
clientID := c.Param("id")
if clientID == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "client id is required"})
return
}
ctx, cancel := context.WithTimeout(c.Request.Context(), requestTimeout)
defer cancel()
tx, err := h.db.BeginTx(ctx, nil)
if err != nil {
h.logger.Error("failed to begin transaction", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "unable to update capabilities"})
return
}
defer tx.Rollback()
var exists bool
if err := tx.QueryRowContext(ctx, `SELECT EXISTS (SELECT 1 FROM bacula_clients WHERE id = $1)`, clientID).Scan(&exists); err != nil {
h.logger.Error("failed to verify client", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "unable to update client"})
return
}
if !exists {
c.JSON(http.StatusNotFound, gin.H{"error": "client not found"})
return
}
backupPayload, err := json.Marshal(req.BackupTypes)
if err != nil {
h.logger.Error("failed to marshal backup types", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to encode backup types"})
return
}
if _, err := tx.ExecContext(ctx, `
UPDATE bacula_clients
SET pending_backup_types = $1,
pending_requested_by = $2,
pending_requested_at = NOW(),
pending_notes = $3,
updated_at = NOW()
WHERE id = $4
`, backupPayload, user.ID, req.Notes, clientID); err != nil {
h.logger.Error("failed to mark pending update", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to update client"})
return
}
if err := insertCapabilityHistory(ctx, tx, clientID, req.BackupTypes, "ui", user.ID, req.Notes); err != nil {
h.logger.Error("failed to insert capability history", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to record capability change"})
return
}
if err := tx.Commit(); err != nil {
h.logger.Error("failed to commit capability update", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to update client"})
return
}
c.JSON(http.StatusOK, PendingUpdateResponse{
BackupTypes: req.BackupTypes,
RequestedBy: user.ID,
RequestedAt: time.Now(),
Notes: req.Notes,
})
}
func (h *Handler) GetPendingUpdate(c *gin.Context) {
clientID := c.Param("id")
if clientID == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "client id is required"})
return
}
ctx, cancel := context.WithTimeout(c.Request.Context(), requestTimeout)
defer cancel()
var pendingJSON []byte
var requestedBy sql.NullString
var requestedAt sql.NullTime
var notes sql.NullString
err := h.db.QueryRowContext(ctx, `
SELECT pending_backup_types, pending_requested_by, pending_requested_at, pending_notes
FROM bacula_clients
WHERE id = $1
`, clientID).Scan(&pendingJSON, &requestedBy, &requestedAt, &notes)
if err != nil {
if err == sql.ErrNoRows {
c.JSON(http.StatusNotFound, gin.H{"error": "client not found"})
return
}
h.logger.Error("failed to fetch pending update", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to read pending update"})
return
}
if len(pendingJSON) == 0 {
c.Status(http.StatusNoContent)
return
}
var backupTypes []string
if err := json.Unmarshal(pendingJSON, &backupTypes); err != nil {
h.logger.Error("failed to unmarshal pending backup types", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to read pending update"})
return
}
response := PendingUpdateResponse{
BackupTypes: backupTypes,
Notes: notes.String,
}
if requestedBy.Valid {
response.RequestedBy = requestedBy.String
}
if requestedAt.Valid {
response.RequestedAt = requestedAt.Time
}
c.JSON(http.StatusOK, response)
}
func (h *Handler) Ping(c *gin.Context) {
clientID := c.Param("id")
if clientID == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "client id is required"})
return
}
var req PingRequest
if err := c.ShouldBindJSON(&req); err != nil {
// swallow body if absent
}
ctx, cancel := context.WithTimeout(c.Request.Context(), requestTimeout)
defer cancel()
query := `
UPDATE bacula_clients
SET last_seen = NOW(),
status = COALESCE(NULLIF($2, ''), status),
updated_at = NOW()
WHERE id = $1
RETURNING id
`
var id string
err := h.db.QueryRowContext(ctx, query, clientID, req.Status).Scan(&id)
if err != nil {
if err == sql.ErrNoRows {
c.JSON(http.StatusNotFound, gin.H{"error": "client not found"})
return
}
h.logger.Error("failed to update heartbeat", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to update client"})
return
}
c.Status(http.StatusNoContent)
}
func (h *Handler) ListClients(c *gin.Context) {
ctx, cancel := context.WithTimeout(c.Request.Context(), requestTimeout)
defer cancel()
rows, err := h.db.QueryContext(ctx, `
SELECT id, hostname, ip_address, agent_version, status, backup_types, metadata,
pending_backup_types, pending_requested_by, pending_requested_at, pending_notes,
registered_by_user_id, last_seen, created_at, updated_at
FROM bacula_clients
ORDER BY created_at DESC
`)
if err != nil {
h.logger.Error("failed to query clients", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to fetch clients"})
return
}
defer rows.Close()
var clients []*ClientResponse
var ids []string
for rows.Next() {
row := clientRow{}
if err := rows.Scan(&row.ID, &row.Hostname, &row.IPAddress, &row.AgentVersion, &row.Status,
&row.BackupJSON, &row.MetadataJSON, &row.PendingBackupJSON, &row.PendingRequestedBy,
&row.PendingRequestedAt, &row.PendingNotes, &row.RegisteredBy, &row.LastSeen,
&row.CreatedAt, &row.UpdatedAt); err != nil {
h.logger.Error("failed to scan client row", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to fetch clients"})
return
}
resp, err := buildClientResponse(&row)
if err != nil {
h.logger.Error("failed to build client response", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to fetch clients"})
return
}
clients = append(clients, resp)
ids = append(ids, resp.ID)
}
if len(ids) > 0 {
if err := h.attachHistory(ctx, ids, clients); err != nil {
h.logger.Error("failed to attach history", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to fetch client history"})
return
}
}
c.JSON(http.StatusOK, clients)
}
func (h *Handler) GetClient(c *gin.Context) {
clientID := c.Param("id")
if clientID == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "client id is required"})
return
}
ctx, cancel := context.WithTimeout(c.Request.Context(), requestTimeout)
defer cancel()
var row clientRow
err := h.db.QueryRowContext(ctx, `
SELECT id, hostname, ip_address, agent_version, status, backup_types, metadata,
pending_backup_types, pending_requested_by, pending_requested_at, pending_notes,
registered_by_user_id, last_seen, created_at, updated_at
FROM bacula_clients
WHERE id = $1
`, clientID).Scan(&row.ID, &row.Hostname, &row.IPAddress, &row.AgentVersion, &row.Status,
&row.BackupJSON, &row.MetadataJSON, &row.PendingBackupJSON, &row.PendingRequestedBy,
&row.PendingRequestedAt, &row.PendingNotes, &row.RegisteredBy, &row.LastSeen,
&row.CreatedAt, &row.UpdatedAt)
if err != nil {
if err == sql.ErrNoRows {
c.JSON(http.StatusNotFound, gin.H{"error": "client not found"})
return
}
h.logger.Error("failed to read client", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to fetch client"})
return
}
resp, err := buildClientResponse(&row)
if err != nil {
h.logger.Error("failed to build client response", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to fetch client"})
return
}
if err := h.attachHistory(ctx, []string{resp.ID}, []*ClientResponse{resp}); err != nil {
h.logger.Error("failed to attach history", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to fetch client history"})
return
}
c.JSON(http.StatusOK, resp)
}
func (h *Handler) attachHistory(ctx context.Context, ids []string, clients []*ClientResponse) error {
if len(ids) == 0 {
return nil
}
rows, err := h.db.QueryContext(ctx, `
SELECT client_id, backup_types, source, requested_by_user_id, requested_at, notes
FROM bacula_client_capability_history
WHERE client_id = ANY($1)
ORDER BY requested_at DESC
`, pq.Array(ids))
if err != nil {
return err
}
defer rows.Close()
clientMap := make(map[string]*ClientResponse)
for _, client := range clients {
clientMap[client.ID] = client
}
for rows.Next() {
var clientID string
var backupJSON []byte
var source string
var requestedBy sql.NullString
var requestedAt time.Time
var notes sql.NullString
if err := rows.Scan(&clientID, &backupJSON, &source, &requestedBy, &requestedAt, &notes); err != nil {
return err
}
resp, ok := clientMap[clientID]
if !ok {
continue
}
if len(resp.CapabilityHistory) >= maxHistoryEntries {
continue
}
var backupTypes []string
if err := json.Unmarshal(backupJSON, &backupTypes); err != nil {
return err
}
entry := CapabilityHistoryEntry{
BackupTypes: backupTypes,
Source: source,
RequestedAt: requestedAt,
}
if requestedBy.Valid {
entry.RequestedBy = requestedBy.String
}
if notes.Valid {
entry.Notes = notes.String
}
resp.CapabilityHistory = append(resp.CapabilityHistory, entry)
}
return nil
}
type clientRow struct {
ID string
Hostname string
IPAddress string
AgentVersion string
Status string
BackupJSON []byte
MetadataJSON []byte
PendingBackupJSON []byte
PendingRequestedBy sql.NullString
PendingRequestedAt sql.NullTime
PendingNotes sql.NullString
RegisteredBy string
LastSeen sql.NullTime
CreatedAt time.Time
UpdatedAt time.Time
}
func buildClientResponse(row *clientRow) (*ClientResponse, error) {
backupTypes, err := decodeStringSlice(row.BackupJSON)
if err != nil {
return nil, err
}
pendingTypes, err := decodeStringSlice(row.PendingBackupJSON)
if err != nil {
return nil, err
}
metadata, err := decodeMetadata(row.MetadataJSON)
if err != nil {
return nil, err
}
resp := &ClientResponse{
ID: row.ID,
Hostname: row.Hostname,
IPAddress: row.IPAddress,
AgentVersion: row.AgentVersion,
Status: row.Status,
BackupTypes: backupTypes,
Metadata: metadata,
RegisteredBy: row.RegisteredBy,
CreatedAt: row.CreatedAt,
UpdatedAt: row.UpdatedAt,
}
if len(pendingTypes) > 0 {
resp.PendingBackupTypes = pendingTypes
if row.PendingRequestedBy.Valid {
resp.PendingRequestedBy = row.PendingRequestedBy.String
}
if row.PendingRequestedAt.Valid {
resp.PendingRequestedAt = &row.PendingRequestedAt.Time
}
if row.PendingNotes.Valid {
resp.PendingNotes = row.PendingNotes.String
}
}
if row.LastSeen.Valid {
resp.LastSeen = &row.LastSeen.Time
}
return resp, nil
}
func currentUser(c *gin.Context) (*iam.User, error) {
user, exists := c.Get("user")
if !exists {
return nil, sql.ErrNoRows
}
authUser, ok := user.(*iam.User)
if !ok {
return nil, sql.ErrNoRows
}
return authUser, nil
}
func decodeStringSlice(data []byte) ([]string, error) {
if len(data) == 0 {
return []string{}, nil
}
var dst []string
if err := json.Unmarshal(data, &dst); err != nil {
return nil, err
}
return dst, nil
}
func decodeMetadata(data []byte) (map[string]interface{}, error) {
if len(data) == 0 {
return nil, nil
}
var metadata map[string]interface{}
if err := json.Unmarshal(data, &metadata); err != nil {
return nil, err
}
return metadata, nil
}
func insertCapabilityHistory(ctx context.Context, tx *sql.Tx, clientID string, backupTypes []string, source, requestedBy, notes string) error {
payload, err := json.Marshal(backupTypes)
if err != nil {
return err
}
_, err = tx.ExecContext(ctx, `
INSERT INTO bacula_client_capability_history (
client_id, backup_types, source, requested_by_user_id, notes
) VALUES ($1, $2, $3, $4, $5)
`, clientID, payload, source, requestedBy, notes)
return err
}
func stringSlicesEqual(a, b []string) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}

19
backend/internal/common/config/config.go
View File

@@ -10,11 +10,12 @@ import (
// Config represents the application configuration
type Config struct {
Server ServerConfig `yaml:"server"`
Database DatabaseConfig `yaml:"database"`
Auth AuthConfig `yaml:"auth"`
Logging LoggingConfig `yaml:"logging"`
Security SecurityConfig `yaml:"security"`
Server ServerConfig `yaml:"server"`
Database DatabaseConfig `yaml:"database"`
Auth AuthConfig `yaml:"auth"`
Logging LoggingConfig `yaml:"logging"`
Security SecurityConfig `yaml:"security"`
ObjectStorage ObjectStorageConfig `yaml:"object_storage"`
}
// ServerConfig holds HTTP server configuration
@@ -96,6 +97,14 @@ type SecurityHeadersConfig struct {
Enabled bool `yaml:"enabled"`
}
// ObjectStorageConfig holds MinIO configuration
type ObjectStorageConfig struct {
Endpoint string `yaml:"endpoint"`
AccessKey string `yaml:"access_key"`
SecretKey string `yaml:"secret_key"`
UseSSL bool `yaml:"use_ssl"`
}
// Load reads configuration from file and environment variables
func Load(path string) (*Config, error) {
cfg := DefaultConfig()

22
backend/internal/common/database/migrations/003_object_storage_config.sql Normal file
View File

@@ -0,0 +1,22 @@
-- Migration: Object Storage Configuration
-- Description: Creates table for storing MinIO object storage configuration
-- Date: 2026-01-09
CREATE TABLE IF NOT EXISTS object_storage_config (
id SERIAL PRIMARY KEY,
dataset_path VARCHAR(255) NOT NULL UNIQUE,
mount_point VARCHAR(512) NOT NULL,
pool_name VARCHAR(255) NOT NULL,
dataset_name VARCHAR(255) NOT NULL,
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_object_storage_config_pool_name ON object_storage_config(pool_name);
CREATE INDEX IF NOT EXISTS idx_object_storage_config_updated_at ON object_storage_config(updated_at);
COMMENT ON TABLE object_storage_config IS 'Stores MinIO object storage configuration, linking to ZFS datasets';
COMMENT ON COLUMN object_storage_config.dataset_path IS 'Full ZFS dataset path (e.g., pool/dataset)';
COMMENT ON COLUMN object_storage_config.mount_point IS 'Mount point path for the dataset';
COMMENT ON COLUMN object_storage_config.pool_name IS 'ZFS pool name';
COMMENT ON COLUMN object_storage_config.dataset_name IS 'ZFS dataset name';

23
backend/internal/common/database/migrations/012_add_snapshot_schedules_table.sql Normal file
View File

@@ -0,0 +1,23 @@
-- Snapshot schedules table for automated snapshot creation
CREATE TABLE IF NOT EXISTS snapshot_schedules (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL,
dataset VARCHAR(255) NOT NULL,
snapshot_name_template VARCHAR(255) NOT NULL, -- e.g., "auto-%Y-%m-%d-%H%M" or "daily-backup"
schedule_type VARCHAR(50) NOT NULL, -- 'hourly', 'daily', 'weekly', 'monthly', 'cron'
schedule_config JSONB NOT NULL, -- For cron: {"cron": "0 0 * * *"}, for others: {"time": "00:00", "day": 1, etc.}
recursive BOOLEAN NOT NULL DEFAULT false,
enabled BOOLEAN NOT NULL DEFAULT true,
retention_count INTEGER, -- Keep last N snapshots (null = unlimited)
retention_days INTEGER, -- Keep snapshots for N days (null = unlimited)
last_run_at TIMESTAMP,
next_run_at TIMESTAMP,
created_by UUID REFERENCES users(id),
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
UNIQUE(name)
);
CREATE INDEX IF NOT EXISTS idx_snapshot_schedules_enabled ON snapshot_schedules(enabled);
CREATE INDEX IF NOT EXISTS idx_snapshot_schedules_next_run ON snapshot_schedules(next_run_at);
CREATE INDEX IF NOT EXISTS idx_snapshot_schedules_dataset ON snapshot_schedules(dataset);

76
backend/internal/common/database/migrations/013_add_replication_tasks_table.sql Normal file
View File

@@ -0,0 +1,76 @@
-- ZFS Replication Tasks Table
-- Supports both outbound (sender) and inbound (receiver) replication
CREATE TABLE IF NOT EXISTS replication_tasks (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL,
direction VARCHAR(20) NOT NULL, -- 'outbound' (sender) or 'inbound' (receiver)
-- For outbound replication (sender)
source_dataset VARCHAR(512), -- Source dataset on this system (outbound) or remote system (inbound)
target_host VARCHAR(255), -- Target host IP or hostname (for outbound)
target_port INTEGER DEFAULT 22, -- SSH port (default 22, for outbound)
target_user VARCHAR(255) DEFAULT 'root', -- SSH user (for outbound)
target_dataset VARCHAR(512), -- Target dataset on remote system (for outbound)
target_ssh_key_path TEXT, -- Path to SSH private key (for outbound)
-- For inbound replication (receiver)
source_host VARCHAR(255), -- Source host IP or hostname (for inbound)
source_port INTEGER DEFAULT 22, -- SSH port (for inbound)
source_user VARCHAR(255) DEFAULT 'root', -- SSH user (for inbound)
local_dataset VARCHAR(512), -- Local dataset to receive snapshots (for inbound)
-- Common settings
schedule_type VARCHAR(50), -- 'manual', 'hourly', 'daily', 'weekly', 'monthly', 'cron'
schedule_config JSONB, -- Schedule configuration (similar to snapshot schedules)
compression VARCHAR(50) DEFAULT 'lz4', -- 'off', 'lz4', 'gzip', 'zstd'
encryption BOOLEAN DEFAULT false, -- Enable encryption during transfer
recursive BOOLEAN DEFAULT false, -- Replicate recursively
incremental BOOLEAN DEFAULT true, -- Use incremental replication
auto_snapshot BOOLEAN DEFAULT true, -- Auto-create snapshot before replication
-- Status and tracking
enabled BOOLEAN NOT NULL DEFAULT true,
status VARCHAR(50) DEFAULT 'idle', -- 'idle', 'running', 'failed', 'paused'
last_run_at TIMESTAMP,
last_run_status VARCHAR(50), -- 'success', 'failed', 'partial'
last_run_error TEXT,
next_run_at TIMESTAMP,
last_snapshot_sent VARCHAR(512), -- Last snapshot successfully sent (for outbound)
last_snapshot_received VARCHAR(512), -- Last snapshot successfully received (for inbound)
-- Statistics
total_runs INTEGER DEFAULT 0,
successful_runs INTEGER DEFAULT 0,
failed_runs INTEGER DEFAULT 0,
bytes_sent BIGINT DEFAULT 0, -- Total bytes sent (for outbound)
bytes_received BIGINT DEFAULT 0, -- Total bytes received (for inbound)
created_by UUID REFERENCES users(id),
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
-- Validation: ensure required fields based on direction
CONSTRAINT chk_direction CHECK (direction IN ('outbound', 'inbound')),
CONSTRAINT chk_outbound_fields CHECK (
direction != 'outbound' OR (
source_dataset IS NOT NULL AND
target_host IS NOT NULL AND
target_dataset IS NOT NULL
)
),
CONSTRAINT chk_inbound_fields CHECK (
direction != 'inbound' OR (
source_host IS NOT NULL AND
source_dataset IS NOT NULL AND
local_dataset IS NOT NULL
)
)
);
-- Create indexes
CREATE INDEX IF NOT EXISTS idx_replication_tasks_direction ON replication_tasks(direction);
CREATE INDEX IF NOT EXISTS idx_replication_tasks_enabled ON replication_tasks(enabled);
CREATE INDEX IF NOT EXISTS idx_replication_tasks_status ON replication_tasks(status);
CREATE INDEX IF NOT EXISTS idx_replication_tasks_next_run ON replication_tasks(next_run_at);
CREATE INDEX IF NOT EXISTS idx_replication_tasks_source_dataset ON replication_tasks(source_dataset);
CREATE INDEX IF NOT EXISTS idx_replication_tasks_target_host ON replication_tasks(target_host);

30
backend/internal/common/database/migrations/014_add_client_categories_table.sql Normal file
View File

@@ -0,0 +1,30 @@
-- AtlasOS - Calypso
-- Client Categories Schema
-- Version: 14.0
-- Adds category support for backup clients (File, Database, Virtual)
-- Client metadata table to store additional information about clients
-- This extends the Bacula Client table with Calypso-specific metadata
CREATE TABLE IF NOT EXISTS client_metadata (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
client_id INTEGER NOT NULL, -- Bacula Client.ClientId
client_name VARCHAR(255) NOT NULL, -- Bacula Client.Name (for reference)
category VARCHAR(50) NOT NULL DEFAULT 'File', -- 'File', 'Database', 'Virtual'
description TEXT,
tags JSONB, -- Additional tags/metadata as JSON
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
-- Ensure one metadata entry per client
CONSTRAINT unique_client_id UNIQUE (client_id),
CONSTRAINT chk_category CHECK (category IN ('File', 'Database', 'Virtual'))
);
-- Indexes for performance
CREATE INDEX IF NOT EXISTS idx_client_metadata_client_id ON client_metadata(client_id);
CREATE INDEX IF NOT EXISTS idx_client_metadata_client_name ON client_metadata(client_name);
CREATE INDEX IF NOT EXISTS idx_client_metadata_category ON client_metadata(category);
-- Add comment
COMMENT ON TABLE client_metadata IS 'Stores Calypso-specific metadata for backup clients, including category classification';
COMMENT ON COLUMN client_metadata.category IS 'Client category: File (file system backups), Database (database backups), Virtual (virtual machine backups)';

44
backend/internal/common/database/migrations/015_add_bacula_clients_table.sql Normal file
View File

@@ -0,0 +1,44 @@
-- AtlasOS - Calypso
-- Migration 015: Add Bacula clients and capability history tables
--
-- Adds tables for tracking registered Bacula agents, their backup capabilities,
-- and a history log for UI- or agent-triggered capability changes. Pending
-- updates are stored on the client row until the agent pulls them.
CREATE TABLE IF NOT EXISTS bacula_clients (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
hostname TEXT NOT NULL,
ip_address TEXT,
agent_version TEXT,
status TEXT NOT NULL DEFAULT 'online',
backup_types JSONB NOT NULL,
pending_backup_types JSONB,
pending_requested_by UUID,
pending_requested_at TIMESTAMPTZ,
pending_notes TEXT,
metadata JSONB,
registered_by_user_id UUID NOT NULL,
last_seen TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT uniq_bacula_clients_hostname UNIQUE (hostname)
);
CREATE INDEX IF NOT EXISTS idx_bacula_clients_registered_by ON bacula_clients (registered_by_user_id);
CREATE INDEX IF NOT EXISTS idx_bacula_clients_status ON bacula_clients (status);
CREATE TABLE IF NOT EXISTS bacula_client_capability_history (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
client_id UUID NOT NULL REFERENCES bacula_clients (id) ON DELETE CASCADE,
backup_types JSONB NOT NULL,
source TEXT NOT NULL,
requested_by_user_id UUID,
requested_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
notes TEXT
);
CREATE INDEX IF NOT EXISTS idx_bacula_client_capability_history_client ON bacula_client_capability_history (client_id);
CREATE INDEX IF NOT EXISTS idx_bacula_client_capability_history_requested_at ON bacula_client_capability_history (requested_at);
COMMENT ON TABLE bacula_clients IS 'Tracks Bacula clients registered with Calypso, including pending capability pushes.';
COMMENT ON TABLE bacula_client_capability_history IS 'Audit history of backup capability changes per client.';

32
backend/internal/common/router/middleware.go
View File

@@ -13,24 +13,30 @@ import (
// authMiddleware validates JWT tokens and sets user context
func authMiddleware(authHandler *auth.Handler) gin.HandlerFunc {
return func(c *gin.Context) {
// Extract token from Authorization header
var token string
// Try to extract token from Authorization header first
authHeader := c.GetHeader("Authorization")
if authHeader == "" {
c.JSON(http.StatusUnauthorized, gin.H{"error": "missing authorization header"})
if authHeader != "" {
// Parse Bearer token
parts := strings.SplitN(authHeader, " ", 2)
if len(parts) == 2 && parts[0] == "Bearer" {
token = parts[1]
}
}
// If no token from header, try query parameter (for WebSocket)
if token == "" {
token = c.Query("token")
}
// If still no token, return error
if token == "" {
c.JSON(http.StatusUnauthorized, gin.H{"error": "missing authorization token"})
c.Abort()
return
}
// Parse Bearer token
parts := strings.SplitN(authHeader, " ", 2)
if len(parts) != 2 || parts[0] != "Bearer" {
c.JSON(http.StatusUnauthorized, gin.H{"error": "invalid authorization header format"})
c.Abort()
return
}
token := parts[1]
// Validate token and get user
user, err := authHandler.ValidateToken(token)
if err != nil {

138
backend/internal/common/router/router.go
View File

@@ -7,13 +7,17 @@ import (
"github.com/atlasos/calypso/internal/audit"
"github.com/atlasos/calypso/internal/auth"
"github.com/atlasos/calypso/internal/backup"
"github.com/atlasos/calypso/internal/bacula"
"github.com/atlasos/calypso/internal/common/cache"
"github.com/atlasos/calypso/internal/common/config"
"github.com/atlasos/calypso/internal/common/database"
"github.com/atlasos/calypso/internal/common/logger"
"github.com/atlasos/calypso/internal/iam"
"github.com/atlasos/calypso/internal/monitoring"
"github.com/atlasos/calypso/internal/object_storage"
"github.com/atlasos/calypso/internal/scst"
"github.com/atlasos/calypso/internal/shares"
"github.com/atlasos/calypso/internal/storage"
"github.com/atlasos/calypso/internal/system"
"github.com/atlasos/calypso/internal/tape_physical"
@@ -174,6 +178,11 @@ func NewRouter(cfg *config.Config, db *database.DB, log *logger.Logger) *gin.Eng
zfsPoolMonitor := storage.NewZFSPoolMonitor(db, log, 2*time.Minute)
go zfsPoolMonitor.Start(context.Background())
// Start snapshot schedule worker in background (checks schedules every 1 minute)
snapshotService := storage.NewSnapshotService(db, log)
snapshotScheduleWorker := storage.NewSnapshotScheduleWorker(db, log, snapshotService, 1*time.Minute)
go snapshotScheduleWorker.Start(context.Background())
storageGroup := protected.Group("/storage")
storageGroup.Use(requirePermission("storage", "read"))
{
@@ -194,10 +203,83 @@ func NewRouter(cfg *config.Config, db *database.DB, log *logger.Logger) *gin.Eng
storageGroup.GET("/zfs/pools/:id/datasets", storageHandler.ListZFSDatasets)
storageGroup.POST("/zfs/pools/:id/datasets", requirePermission("storage", "write"), storageHandler.CreateZFSDataset)
storageGroup.DELETE("/zfs/pools/:id/datasets/:dataset", requirePermission("storage", "write"), storageHandler.DeleteZFSDataset)
// ZFS Snapshots
storageGroup.GET("/zfs/snapshots", storageHandler.ListSnapshots)
storageGroup.POST("/zfs/snapshots", requirePermission("storage", "write"), storageHandler.CreateSnapshot)
storageGroup.DELETE("/zfs/snapshots/:name", requirePermission("storage", "write"), storageHandler.DeleteSnapshot)
storageGroup.POST("/zfs/snapshots/:name/rollback", requirePermission("storage", "write"), storageHandler.RollbackSnapshot)
storageGroup.POST("/zfs/snapshots/:name/clone", requirePermission("storage", "write"), storageHandler.CloneSnapshot)
// Snapshot Schedules
storageGroup.GET("/zfs/snapshot-schedules", storageHandler.ListSnapshotSchedules)
storageGroup.GET("/zfs/snapshot-schedules/:id", storageHandler.GetSnapshotSchedule)
storageGroup.POST("/zfs/snapshot-schedules", requirePermission("storage", "write"), storageHandler.CreateSnapshotSchedule)
storageGroup.PUT("/zfs/snapshot-schedules/:id", requirePermission("storage", "write"), storageHandler.UpdateSnapshotSchedule)
storageGroup.DELETE("/zfs/snapshot-schedules/:id", requirePermission("storage", "write"), storageHandler.DeleteSnapshotSchedule)
storageGroup.POST("/zfs/snapshot-schedules/:id/toggle", requirePermission("storage", "write"), storageHandler.ToggleSnapshotSchedule)
// Replication Tasks
storageGroup.GET("/zfs/replication-tasks", storageHandler.ListReplicationTasks)
storageGroup.GET("/zfs/replication-tasks/:id", storageHandler.GetReplicationTask)
storageGroup.POST("/zfs/replication-tasks", requirePermission("storage", "write"), storageHandler.CreateReplicationTask)
storageGroup.PUT("/zfs/replication-tasks/:id", requirePermission("storage", "write"), storageHandler.UpdateReplicationTask)
storageGroup.DELETE("/zfs/replication-tasks/:id", requirePermission("storage", "write"), storageHandler.DeleteReplicationTask)
// ZFS ARC Stats
storageGroup.GET("/zfs/arc/stats", storageHandler.GetARCStats)
}
// Shares (CIFS/NFS)
sharesHandler := shares.NewHandler(db, log)
sharesGroup := protected.Group("/shares")
sharesGroup.Use(requirePermission("storage", "read"))
{
sharesGroup.GET("", sharesHandler.ListShares)
sharesGroup.GET("/:id", sharesHandler.GetShare)
sharesGroup.POST("", requirePermission("storage", "write"), sharesHandler.CreateShare)
sharesGroup.PUT("/:id", requirePermission("storage", "write"), sharesHandler.UpdateShare)
sharesGroup.DELETE("/:id", requirePermission("storage", "write"), sharesHandler.DeleteShare)
}
// Object Storage (MinIO)
// Initialize MinIO service if configured
if cfg.ObjectStorage.Endpoint != "" {
objectStorageService, err := object_storage.NewService(
cfg.ObjectStorage.Endpoint,
cfg.ObjectStorage.AccessKey,
cfg.ObjectStorage.SecretKey,
log,
)
if err != nil {
log.Error("Failed to initialize MinIO service", "error", err)
} else {
objectStorageHandler := object_storage.NewHandler(objectStorageService, db, log)
objectStorageGroup := protected.Group("/object-storage")
objectStorageGroup.Use(requirePermission("storage", "read"))
{
// Setup endpoints
objectStorageGroup.GET("/setup/datasets", objectStorageHandler.GetAvailableDatasets)
objectStorageGroup.GET("/setup/current", objectStorageHandler.GetCurrentSetup)
objectStorageGroup.POST("/setup", requirePermission("storage", "write"), objectStorageHandler.SetupObjectStorage)
objectStorageGroup.PUT("/setup", requirePermission("storage", "write"), objectStorageHandler.UpdateObjectStorage)
// Bucket endpoints
// IMPORTANT: More specific routes must come BEFORE less specific ones
objectStorageGroup.GET("/buckets", objectStorageHandler.ListBuckets)
objectStorageGroup.GET("/buckets/:name/objects", objectStorageHandler.ListObjects)
objectStorageGroup.GET("/buckets/:name", objectStorageHandler.GetBucket)
objectStorageGroup.POST("/buckets", requirePermission("storage", "write"), objectStorageHandler.CreateBucket)
objectStorageGroup.DELETE("/buckets/:name", requirePermission("storage", "write"), objectStorageHandler.DeleteBucket)
// User management routes
objectStorageGroup.GET("/users", objectStorageHandler.ListUsers)
objectStorageGroup.POST("/users", requirePermission("storage", "write"), objectStorageHandler.CreateUser)
objectStorageGroup.DELETE("/users/:access_key", requirePermission("storage", "write"), objectStorageHandler.DeleteUser)
// Service account (access key) management routes
objectStorageGroup.GET("/service-accounts", objectStorageHandler.ListServiceAccounts)
objectStorageGroup.POST("/service-accounts", requirePermission("storage", "write"), objectStorageHandler.CreateServiceAccount)
objectStorageGroup.DELETE("/service-accounts/:access_key", requirePermission("storage", "write"), objectStorageHandler.DeleteServiceAccount)
}
}
}
// SCST
scstHandler := scst.NewHandler(db, log)
scstGroup := protected.Group("/scst")
@@ -206,10 +288,12 @@ func NewRouter(cfg *config.Config, db *database.DB, log *logger.Logger) *gin.Eng
scstGroup.GET("/targets", scstHandler.ListTargets)
scstGroup.GET("/targets/:id", scstHandler.GetTarget)
scstGroup.POST("/targets", scstHandler.CreateTarget)
scstGroup.POST("/targets/:id/luns", scstHandler.AddLUN)
scstGroup.POST("/targets/:id/luns", requirePermission("iscsi", "write"), scstHandler.AddLUN)
scstGroup.DELETE("/targets/:id/luns/:lunId", requirePermission("iscsi", "write"), scstHandler.RemoveLUN)
scstGroup.POST("/targets/:id/initiators", scstHandler.AddInitiator)
scstGroup.POST("/targets/:id/enable", scstHandler.EnableTarget)
scstGroup.POST("/targets/:id/disable", scstHandler.DisableTarget)
scstGroup.DELETE("/targets/:id", requirePermission("iscsi", "write"), scstHandler.DeleteTarget)
scstGroup.GET("/initiators", scstHandler.ListAllInitiators)
scstGroup.GET("/initiators/:id", scstHandler.GetInitiator)
scstGroup.DELETE("/initiators/:id", scstHandler.RemoveInitiator)
@@ -223,6 +307,16 @@ func NewRouter(cfg *config.Config, db *database.DB, log *logger.Logger) *gin.Eng
scstGroup.POST("/portals", scstHandler.CreatePortal)
scstGroup.PUT("/portals/:id", scstHandler.UpdatePortal)
scstGroup.DELETE("/portals/:id", scstHandler.DeletePortal)
// Initiator Groups routes
scstGroup.GET("/initiator-groups", scstHandler.ListAllInitiatorGroups)
scstGroup.GET("/initiator-groups/:id", scstHandler.GetInitiatorGroup)
scstGroup.POST("/initiator-groups", requirePermission("iscsi", "write"), scstHandler.CreateInitiatorGroup)
scstGroup.PUT("/initiator-groups/:id", requirePermission("iscsi", "write"), scstHandler.UpdateInitiatorGroup)
scstGroup.DELETE("/initiator-groups/:id", requirePermission("iscsi", "write"), scstHandler.DeleteInitiatorGroup)
scstGroup.POST("/initiator-groups/:id/initiators", requirePermission("iscsi", "write"), scstHandler.AddInitiatorToGroup)
// Config file management
scstGroup.GET("/config/file", requirePermission("iscsi", "read"), scstHandler.GetConfigFile)
scstGroup.PUT("/config/file", requirePermission("iscsi", "write"), scstHandler.UpdateConfigFile)
}
// Physical Tape Libraries
@@ -260,7 +354,18 @@ func NewRouter(cfg *config.Config, db *database.DB, log *logger.Logger) *gin.Eng
}
// System Management
systemService := system.NewService(log)
systemHandler := system.NewHandler(log, tasks.NewEngine(db, log))
// Set service in handler (if handler needs direct access)
// Note: Handler already has service via NewHandler, but we need to ensure it's the same instance
// Start network monitoring with RRD
if err := systemService.StartNetworkMonitoring(context.Background()); err != nil {
log.Warn("Failed to start network monitoring", "error", err)
} else {
log.Info("Network monitoring started with RRD")
}
systemGroup := protected.Group("/system")
systemGroup.Use(requirePermission("system", "read"))
{
@@ -268,8 +373,15 @@ func NewRouter(cfg *config.Config, db *database.DB, log *logger.Logger) *gin.Eng
systemGroup.GET("/services/:name", systemHandler.GetServiceStatus)
systemGroup.POST("/services/:name/restart", systemHandler.RestartService)
systemGroup.GET("/services/:name/logs", systemHandler.GetServiceLogs)
systemGroup.GET("/logs", systemHandler.GetSystemLogs)
systemGroup.GET("/network/throughput", systemHandler.GetNetworkThroughput)
systemGroup.POST("/support-bundle", systemHandler.GenerateSupportBundle)
systemGroup.GET("/interfaces", systemHandler.ListNetworkInterfaces)
systemGroup.GET("/management-ip", systemHandler.GetManagementIPAddress)
systemGroup.PUT("/interfaces/:name", systemHandler.UpdateNetworkInterface)
systemGroup.GET("/ntp", systemHandler.GetNTPSettings)
systemGroup.POST("/ntp", systemHandler.SaveNTPSettings)
systemGroup.POST("/execute", requirePermission("system", "write"), systemHandler.ExecuteCommand)
}
// IAM routes - GetUser can be accessed by user viewing own profile or admin
@@ -330,9 +442,33 @@ func NewRouter(cfg *config.Config, db *database.DB, log *logger.Logger) *gin.Eng
backupGroup := protected.Group("/backup")
backupGroup.Use(requirePermission("backup", "read"))
{
backupGroup.GET("/dashboard/stats", backupHandler.GetDashboardStats)
backupGroup.GET("/jobs", backupHandler.ListJobs)
backupGroup.GET("/jobs/:id", backupHandler.GetJob)
backupGroup.POST("/jobs", requirePermission("backup", "write"), backupHandler.CreateJob)
backupGroup.GET("/clients", backupHandler.ListClients)
backupGroup.GET("/storage/pools", backupHandler.ListStoragePools)
backupGroup.POST("/storage/pools", requirePermission("backup", "write"), backupHandler.CreateStoragePool)
backupGroup.DELETE("/storage/pools/:id", requirePermission("backup", "write"), backupHandler.DeleteStoragePool)
backupGroup.GET("/storage/volumes", backupHandler.ListStorageVolumes)
backupGroup.POST("/storage/volumes", requirePermission("backup", "write"), backupHandler.CreateStorageVolume)
backupGroup.PUT("/storage/volumes/:id", requirePermission("backup", "write"), backupHandler.UpdateStorageVolume)
backupGroup.DELETE("/storage/volumes/:id", requirePermission("backup", "write"), backupHandler.DeleteStorageVolume)
backupGroup.GET("/media", backupHandler.ListMedia)
backupGroup.GET("/storage/daemons", backupHandler.ListStorageDaemons)
backupGroup.POST("/console/execute", requirePermission("backup", "write"), backupHandler.ExecuteBconsoleCommand)
}
baculaHandler := bacula.NewHandler(db, log)
baculaGroup := protected.Group("/bacula/clients")
baculaGroup.Use(requireRole("bacula-admin"))
{
baculaGroup.POST("/register", baculaHandler.Register)
baculaGroup.POST("/:id/capabilities", baculaHandler.UpdateCapabilities)
baculaGroup.GET("/:id/pending-update", baculaHandler.GetPendingUpdate)
baculaGroup.POST("/:id/ping", baculaHandler.Ping)
baculaGroup.GET("", baculaHandler.ListClients)
baculaGroup.GET("/:id", baculaHandler.GetClient)
}
// Monitoring

5
backend/internal/iam/group.go
View File

@@ -88,11 +88,14 @@ func GetUserGroups(db *database.DB, userID string) ([]string, error) {
for rows.Next() {
var groupName string
if err := rows.Scan(&groupName); err != nil {
return nil, err
return []string{}, err
}
groups = append(groups, groupName)
}
if groups == nil {
groups = []string{}
}
return groups, rows.Err()
}

87
backend/internal/iam/handler.go
View File

@@ -69,6 +69,17 @@ func (h *Handler) ListUsers(c *gin.Context) {
permissions, _ := GetUserPermissions(h.db, u.ID)
groups, _ := GetUserGroups(h.db, u.ID)
// Ensure arrays are never nil (use empty slice instead)
if roles == nil {
roles = []string{}
}
if permissions == nil {
permissions = []string{}
}
if groups == nil {
groups = []string{}
}
users = append(users, map[string]interface{}{
"id": u.ID,
"username": u.Username,
@@ -138,6 +149,17 @@ func (h *Handler) GetUser(c *gin.Context) {
permissions, _ := GetUserPermissions(h.db, userID)
groups, _ := GetUserGroups(h.db, userID)
// Ensure arrays are never nil (use empty slice instead)
if roles == nil {
roles = []string{}
}
if permissions == nil {
permissions = []string{}
}
if groups == nil {
groups = []string{}
}
c.JSON(http.StatusOK, gin.H{
"id": user.ID,
"username": user.Username,
@@ -236,6 +258,8 @@ func (h *Handler) UpdateUser(c *gin.Context) {
}
// Allow update if roles or groups are provided, even if no other fields are updated
// Note: req.Roles and req.Groups can be empty arrays ([]), which is different from nil
// Empty array means "remove all roles/groups", nil means "don't change roles/groups"
if len(updates) == 1 && req.Roles == nil && req.Groups == nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "no fields to update"})
return
@@ -259,13 +283,14 @@ func (h *Handler) UpdateUser(c *gin.Context) {
// Update roles if provided
if req.Roles != nil {
h.logger.Info("Updating user roles", "user_id", userID, "roles", *req.Roles)
h.logger.Info("Updating user roles", "user_id", userID, "requested_roles", *req.Roles)
currentRoles, err := GetUserRoles(h.db, userID)
if err != nil {
h.logger.Error("Failed to get current roles for user", "user_id", userID, "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to process user roles"})
return
}
h.logger.Info("Current user roles", "user_id", userID, "current_roles", currentRoles)
rolesToAdd := []string{}
rolesToRemove := []string{}
@@ -298,8 +323,15 @@ func (h *Handler) UpdateUser(c *gin.Context) {
}
}
h.logger.Info("Roles to add", "user_id", userID, "roles_to_add", rolesToAdd, "count", len(rolesToAdd))
h.logger.Info("Roles to remove", "user_id", userID, "roles_to_remove", rolesToRemove, "count", len(rolesToRemove))
// Add new roles
if len(rolesToAdd) == 0 {
h.logger.Info("No roles to add", "user_id", userID)
}
for _, roleName := range rolesToAdd {
h.logger.Info("Processing role to add", "user_id", userID, "role_name", roleName)
roleID, err := GetRoleIDByName(h.db, roleName)
if err != nil {
if err == sql.ErrNoRows {
@@ -311,12 +343,13 @@ func (h *Handler) UpdateUser(c *gin.Context) {
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to process roles"})
return
}
h.logger.Info("Attempting to add role", "user_id", userID, "role_id", roleID, "role_name", roleName, "assigned_by", currentUser.ID)
if err := AddUserRole(h.db, userID, roleID, currentUser.ID); err != nil {
h.logger.Error("Failed to add role to user", "user_id", userID, "role_id", roleID, "error", err)
// Don't return early, continue with other roles
continue
h.logger.Error("Failed to add role to user", "user_id", userID, "role_id", roleID, "role_name", roleName, "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": fmt.Sprintf("failed to add role '%s': %v", roleName, err)})
return
}
h.logger.Info("Role added to user", "user_id", userID, "role_name", roleName)
h.logger.Info("Role successfully added to user", "user_id", userID, "role_id", roleID, "role_name", roleName)
}
// Remove old roles
@@ -415,8 +448,48 @@ func (h *Handler) UpdateUser(c *gin.Context) {
}
}
h.logger.Info("User updated", "user_id", userID)
c.JSON(http.StatusOK, gin.H{"message": "user updated successfully"})
// Fetch updated user data to return
updatedUser, err := GetUserByID(h.db, userID)
if err != nil {
h.logger.Error("Failed to fetch updated user", "user_id", userID, "error", err)
c.JSON(http.StatusOK, gin.H{"message": "user updated successfully"})
return
}
// Get updated roles, permissions, and groups
updatedRoles, _ := GetUserRoles(h.db, userID)
updatedPermissions, _ := GetUserPermissions(h.db, userID)
updatedGroups, _ := GetUserGroups(h.db, userID)
// Ensure arrays are never nil
if updatedRoles == nil {
updatedRoles = []string{}
}
if updatedPermissions == nil {
updatedPermissions = []string{}
}
if updatedGroups == nil {
updatedGroups = []string{}
}
h.logger.Info("User updated", "user_id", userID, "roles", updatedRoles, "groups", updatedGroups)
c.JSON(http.StatusOK, gin.H{
"message": "user updated successfully",
"user": gin.H{
"id": updatedUser.ID,
"username": updatedUser.Username,
"email": updatedUser.Email,
"full_name": updatedUser.FullName,
"is_active": updatedUser.IsActive,
"is_system": updatedUser.IsSystem,
"roles": updatedRoles,
"permissions": updatedPermissions,
"groups": updatedGroups,
"created_at": updatedUser.CreatedAt,
"updated_at": updatedUser.UpdatedAt,
"last_login_at": updatedUser.LastLoginAt,
},
})
}
// DeleteUser deletes a user

30
backend/internal/iam/user.go
View File

@@ -2,6 +2,7 @@ package iam
import (
"database/sql"
"fmt"
"time"
"github.com/atlasos/calypso/internal/common/database"
@@ -90,11 +91,14 @@ func GetUserRoles(db *database.DB, userID string) ([]string, error) {
for rows.Next() {
var role string
if err := rows.Scan(&role); err != nil {
return nil, err
return []string{}, err
}
roles = append(roles, role)
}
if roles == nil {
roles = []string{}
}
return roles, rows.Err()
}
@@ -118,11 +122,14 @@ func GetUserPermissions(db *database.DB, userID string) ([]string, error) {
for rows.Next() {
var perm string
if err := rows.Scan(&perm); err != nil {
return nil, err
return []string{}, err
}
permissions = append(permissions, perm)
}
if permissions == nil {
permissions = []string{}
}
return permissions, rows.Err()
}
@@ -133,8 +140,23 @@ func AddUserRole(db *database.DB, userID, roleID, assignedBy string) error {
VALUES ($1, $2, $3)
ON CONFLICT (user_id, role_id) DO NOTHING
`
_, err := db.Exec(query, userID, roleID, assignedBy)
return err
result, err := db.Exec(query, userID, roleID, assignedBy)
if err != nil {
return fmt.Errorf("failed to insert user role: %w", err)
}
// Check if row was actually inserted (not just skipped due to conflict)
rowsAffected, err := result.RowsAffected()
if err != nil {
return fmt.Errorf("failed to get rows affected: %w", err)
}
if rowsAffected == 0 {
// Row already exists, this is not an error but we should know about it
return nil // ON CONFLICT DO NOTHING means this is expected
}
return nil
}
// RemoveUserRole removes a role from a user

300
backend/internal/object_storage/handler.go Normal file
View File

@@ -0,0 +1,300 @@
package object_storage
import (
"net/http"
"time"
"github.com/atlasos/calypso/internal/common/database"
"github.com/atlasos/calypso/internal/common/logger"
"github.com/gin-gonic/gin"
)
// Handler handles HTTP requests for object storage
type Handler struct {
service *Service
setupService *SetupService
logger *logger.Logger
}
// NewHandler creates a new object storage handler
func NewHandler(service *Service, db *database.DB, log *logger.Logger) *Handler {
return &Handler{
service: service,
setupService: NewSetupService(db, log),
logger: log,
}
}
// ListBuckets lists all buckets
func (h *Handler) ListBuckets(c *gin.Context) {
buckets, err := h.service.ListBuckets(c.Request.Context())
if err != nil {
h.logger.Error("Failed to list buckets", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to list buckets: " + err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"buckets": buckets})
}
// GetBucket gets bucket information
func (h *Handler) GetBucket(c *gin.Context) {
bucketName := c.Param("name")
bucket, err := h.service.GetBucketStats(c.Request.Context(), bucketName)
if err != nil {
h.logger.Error("Failed to get bucket", "bucket", bucketName, "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to get bucket: " + err.Error()})
return
}
c.JSON(http.StatusOK, bucket)
}
// CreateBucketRequest represents a request to create a bucket
type CreateBucketRequest struct {
Name string `json:"name" binding:"required"`
}
// CreateBucket creates a new bucket
func (h *Handler) CreateBucket(c *gin.Context) {
var req CreateBucketRequest
if err := c.ShouldBindJSON(&req); err != nil {
h.logger.Error("Invalid create bucket request", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request: " + err.Error()})
return
}
if err := h.service.CreateBucket(c.Request.Context(), req.Name); err != nil {
h.logger.Error("Failed to create bucket", "bucket", req.Name, "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to create bucket: " + err.Error()})
return
}
c.JSON(http.StatusCreated, gin.H{"message": "bucket created successfully", "name": req.Name})
}
// DeleteBucket deletes a bucket
func (h *Handler) DeleteBucket(c *gin.Context) {
bucketName := c.Param("name")
if err := h.service.DeleteBucket(c.Request.Context(), bucketName); err != nil {
h.logger.Error("Failed to delete bucket", "bucket", bucketName, "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to delete bucket: " + err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"message": "bucket deleted successfully"})
}
// GetAvailableDatasets gets all available pools and datasets for object storage setup
func (h *Handler) GetAvailableDatasets(c *gin.Context) {
datasets, err := h.setupService.GetAvailableDatasets(c.Request.Context())
if err != nil {
h.logger.Error("Failed to get available datasets", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to get available datasets: " + err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"pools": datasets})
}
// SetupObjectStorageRequest represents a request to setup object storage
type SetupObjectStorageRequest struct {
PoolName string `json:"pool_name" binding:"required"`
DatasetName string `json:"dataset_name" binding:"required"`
CreateNew bool `json:"create_new"`
}
// SetupObjectStorage configures object storage with a ZFS dataset
func (h *Handler) SetupObjectStorage(c *gin.Context) {
var req SetupObjectStorageRequest
if err := c.ShouldBindJSON(&req); err != nil {
h.logger.Error("Invalid setup request", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request: " + err.Error()})
return
}
setupReq := SetupRequest{
PoolName: req.PoolName,
DatasetName: req.DatasetName,
CreateNew: req.CreateNew,
}
result, err := h.setupService.SetupObjectStorage(c.Request.Context(), setupReq)
if err != nil {
h.logger.Error("Failed to setup object storage", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to setup object storage: " + err.Error()})
return
}
c.JSON(http.StatusOK, result)
}
// GetCurrentSetup gets the current object storage configuration
func (h *Handler) GetCurrentSetup(c *gin.Context) {
setup, err := h.setupService.GetCurrentSetup(c.Request.Context())
if err != nil {
h.logger.Error("Failed to get current setup", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to get current setup: " + err.Error()})
return
}
if setup == nil {
c.JSON(http.StatusOK, gin.H{"configured": false})
return
}
c.JSON(http.StatusOK, gin.H{"configured": true, "setup": setup})
}
// UpdateObjectStorage updates the object storage configuration
func (h *Handler) UpdateObjectStorage(c *gin.Context) {
var req SetupObjectStorageRequest
if err := c.ShouldBindJSON(&req); err != nil {
h.logger.Error("Invalid update request", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request: " + err.Error()})
return
}
setupReq := SetupRequest{
PoolName: req.PoolName,
DatasetName: req.DatasetName,
CreateNew: req.CreateNew,
}
result, err := h.setupService.UpdateObjectStorage(c.Request.Context(), setupReq)
if err != nil {
h.logger.Error("Failed to update object storage", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to update object storage: " + err.Error()})
return
}
c.JSON(http.StatusOK, result)
}
// ListUsers lists all IAM users
func (h *Handler) ListUsers(c *gin.Context) {
users, err := h.service.ListUsers(c.Request.Context())
if err != nil {
h.logger.Error("Failed to list users", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to list users: " + err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"users": users})
}
// CreateUserRequest represents a request to create a user
type CreateUserRequest struct {
AccessKey string `json:"access_key" binding:"required"`
SecretKey string `json:"secret_key" binding:"required"`
}
// CreateUser creates a new IAM user
func (h *Handler) CreateUser(c *gin.Context) {
var req CreateUserRequest
if err := c.ShouldBindJSON(&req); err != nil {
h.logger.Error("Invalid create user request", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request: " + err.Error()})
return
}
if err := h.service.CreateUser(c.Request.Context(), req.AccessKey, req.SecretKey); err != nil {
h.logger.Error("Failed to create user", "access_key", req.AccessKey, "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to create user: " + err.Error()})
return
}
c.JSON(http.StatusCreated, gin.H{"message": "user created successfully", "access_key": req.AccessKey})
}
// DeleteUser deletes an IAM user
func (h *Handler) DeleteUser(c *gin.Context) {
accessKey := c.Param("access_key")
if err := h.service.DeleteUser(c.Request.Context(), accessKey); err != nil {
h.logger.Error("Failed to delete user", "access_key", accessKey, "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to delete user: " + err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"message": "user deleted successfully"})
}
// ListServiceAccounts lists all service accounts (access keys)
func (h *Handler) ListServiceAccounts(c *gin.Context) {
accounts, err := h.service.ListServiceAccounts(c.Request.Context())
if err != nil {
h.logger.Error("Failed to list service accounts", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to list service accounts: " + err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"service_accounts": accounts})
}
// CreateServiceAccountRequest represents a request to create a service account
type CreateServiceAccountRequest struct {
ParentUser string `json:"parent_user" binding:"required"`
Policy string `json:"policy,omitempty"`
Expiration *string `json:"expiration,omitempty"` // ISO 8601 format
}
// CreateServiceAccount creates a new service account (access key)
func (h *Handler) CreateServiceAccount(c *gin.Context) {
var req CreateServiceAccountRequest
if err := c.ShouldBindJSON(&req); err != nil {
h.logger.Error("Invalid create service account request", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request: " + err.Error()})
return
}
var expiration *time.Time
if req.Expiration != nil {
exp, err := time.Parse(time.RFC3339, *req.Expiration)
if err != nil {
h.logger.Error("Invalid expiration format", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid expiration format, use ISO 8601 (RFC3339)"})
return
}
expiration = &exp
}
account, err := h.service.CreateServiceAccount(c.Request.Context(), req.ParentUser, req.Policy, expiration)
if err != nil {
h.logger.Error("Failed to create service account", "parent_user", req.ParentUser, "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to create service account: " + err.Error()})
return
}
c.JSON(http.StatusCreated, account)
}
// DeleteServiceAccount deletes a service account
func (h *Handler) DeleteServiceAccount(c *gin.Context) {
accessKey := c.Param("access_key")
if err := h.service.DeleteServiceAccount(c.Request.Context(), accessKey); err != nil {
h.logger.Error("Failed to delete service account", "access_key", accessKey, "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to delete service account: " + err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"message": "service account deleted successfully"})
}
// ListObjects lists objects in a bucket
func (h *Handler) ListObjects(c *gin.Context) {
bucketName := c.Param("name") // Changed from "bucket" to "name" to match route
prefix := c.DefaultQuery("prefix", "") // Optional prefix (folder path)
objects, err := h.service.ListObjects(c.Request.Context(), bucketName, prefix)
if err != nil {
h.logger.Error("Failed to list objects", "bucket", bucketName, "prefix", prefix, "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to list objects: " + err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"objects": objects})
}

373
backend/internal/object_storage/service.go Normal file
View File

@@ -0,0 +1,373 @@
package object_storage
import (
"context"
"encoding/json"
"fmt"
"sort"
"strings"
"time"
"github.com/atlasos/calypso/internal/common/logger"
madmin "github.com/minio/madmin-go/v3"
"github.com/minio/minio-go/v7"
"github.com/minio/minio-go/v7/pkg/credentials"
)
// Service handles MinIO object storage operations
type Service struct {
client *minio.Client
adminClient *madmin.AdminClient
logger *logger.Logger
endpoint string
accessKey string
secretKey string
}
// NewService creates a new MinIO service
func NewService(endpoint, accessKey, secretKey string, log *logger.Logger) (*Service, error) {
// Create MinIO client
minioClient, err := minio.New(endpoint, &minio.Options{
Creds: credentials.NewStaticV4(accessKey, secretKey, ""),
Secure: false, // Set to true if using HTTPS
})
if err != nil {
return nil, fmt.Errorf("failed to create MinIO client: %w", err)
}
// Create MinIO Admin client
adminClient, err := madmin.New(endpoint, accessKey, secretKey, false)
if err != nil {
return nil, fmt.Errorf("failed to create MinIO admin client: %w", err)
}
return &Service{
client: minioClient,
adminClient: adminClient,
logger: log,
endpoint: endpoint,
accessKey: accessKey,
secretKey: secretKey,
}, nil
}
// Bucket represents a MinIO bucket
type Bucket struct {
Name string `json:"name"`
CreationDate time.Time `json:"creation_date"`
Size int64 `json:"size"` // Total size in bytes
Objects int64 `json:"objects"` // Number of objects
AccessPolicy string `json:"access_policy"` // private, public-read, public-read-write
}
// ListBuckets lists all buckets in MinIO
func (s *Service) ListBuckets(ctx context.Context) ([]*Bucket, error) {
buckets, err := s.client.ListBuckets(ctx)
if err != nil {
return nil, fmt.Errorf("failed to list buckets: %w", err)
}
result := make([]*Bucket, 0, len(buckets))
for _, bucket := range buckets {
bucketInfo, err := s.getBucketInfo(ctx, bucket.Name)
if err != nil {
s.logger.Warn("Failed to get bucket info", "bucket", bucket.Name, "error", err)
// Continue with basic info
result = append(result, &Bucket{
Name: bucket.Name,
CreationDate: bucket.CreationDate,
Size: 0,
Objects: 0,
AccessPolicy: "private",
})
continue
}
result = append(result, bucketInfo)
}
return result, nil
}
// getBucketInfo gets detailed information about a bucket
func (s *Service) getBucketInfo(ctx context.Context, bucketName string) (*Bucket, error) {
// Get bucket creation date
buckets, err := s.client.ListBuckets(ctx)
if err != nil {
return nil, err
}
var creationDate time.Time
for _, b := range buckets {
if b.Name == bucketName {
creationDate = b.CreationDate
break
}
}
// Get bucket size and object count by listing objects
var size int64
var objects int64
// List objects in bucket to calculate size and count
objectCh := s.client.ListObjects(ctx, bucketName, minio.ListObjectsOptions{
Recursive: true,
})
for object := range objectCh {
if object.Err != nil {
s.logger.Warn("Error listing object", "bucket", bucketName, "error", object.Err)
continue
}
objects++
size += object.Size
}
return &Bucket{
Name: bucketName,
CreationDate: creationDate,
Size: size,
Objects: objects,
AccessPolicy: s.getBucketPolicy(ctx, bucketName),
}, nil
}
// getBucketPolicy gets the access policy for a bucket
func (s *Service) getBucketPolicy(ctx context.Context, bucketName string) string {
policy, err := s.client.GetBucketPolicy(ctx, bucketName)
if err != nil {
return "private"
}
// Parse policy JSON to determine access type
// For simplicity, check if policy allows public read
if policy != "" {
// Check if policy contains public read access
if strings.Contains(policy, "s3:GetObject") && strings.Contains(policy, "Principal") && strings.Contains(policy, "*") {
if strings.Contains(policy, "s3:PutObject") {
return "public-read-write"
}
return "public-read"
}
}
return "private"
}
// CreateBucket creates a new bucket
func (s *Service) CreateBucket(ctx context.Context, bucketName string) error {
err := s.client.MakeBucket(ctx, bucketName, minio.MakeBucketOptions{})
if err != nil {
return fmt.Errorf("failed to create bucket: %w", err)
}
return nil
}
// DeleteBucket deletes a bucket
func (s *Service) DeleteBucket(ctx context.Context, bucketName string) error {
err := s.client.RemoveBucket(ctx, bucketName)
if err != nil {
return fmt.Errorf("failed to delete bucket: %w", err)
}
return nil
}
// GetBucketStats gets statistics for a bucket
func (s *Service) GetBucketStats(ctx context.Context, bucketName string) (*Bucket, error) {
return s.getBucketInfo(ctx, bucketName)
}
// Object represents a MinIO object (file or folder)
type Object struct {
Name string `json:"name"`
Key string `json:"key"` // Full path/key
Size int64 `json:"size"` // Size in bytes (0 for folders)
LastModified time.Time `json:"last_modified"`
IsDir bool `json:"is_dir"` // True if this is a folder/prefix
ETag string `json:"etag,omitempty"`
}
// ListObjects lists objects in a bucket with optional prefix (folder path)
func (s *Service) ListObjects(ctx context.Context, bucketName, prefix string) ([]*Object, error) {
// Ensure prefix ends with / if it's not empty and doesn't already end with /
if prefix != "" && !strings.HasSuffix(prefix, "/") {
prefix += "/"
}
// List objects with prefix
objectCh := s.client.ListObjects(ctx, bucketName, minio.ListObjectsOptions{
Prefix: prefix,
Recursive: false, // Don't recurse, show folders as separate items
})
var objects []*Object
seenFolders := make(map[string]bool)
for object := range objectCh {
if object.Err != nil {
s.logger.Warn("Error listing object", "bucket", bucketName, "prefix", prefix, "error", object.Err)
continue
}
// Extract folder name from object key
relativeKey := strings.TrimPrefix(object.Key, prefix)
// Check if this is a folder (has / in relative path)
if strings.Contains(relativeKey, "/") {
// Extract folder name (first part before /)
folderName := strings.Split(relativeKey, "/")[0]
folderKey := prefix + folderName + "/"
// Only add folder once
if !seenFolders[folderKey] {
seenFolders[folderKey] = true
objects = append(objects, &Object{
Name: folderName,
Key: folderKey,
Size: 0,
LastModified: time.Time{},
IsDir: true,
})
}
} else {
// This is a file in the current directory
objects = append(objects, &Object{
Name: relativeKey,
Key: object.Key,
Size: object.Size,
LastModified: object.LastModified,
IsDir: false,
ETag: object.ETag,
})
}
}
// Sort: folders first, then files, both alphabetically
sort.Slice(objects, func(i, j int) bool {
if objects[i].IsDir != objects[j].IsDir {
return objects[i].IsDir // Folders first
}
return objects[i].Name < objects[j].Name
})
return objects, nil
}
// User represents a MinIO IAM user
type User struct {
AccessKey string `json:"access_key"`
Status string `json:"status"` // "enabled" or "disabled"
CreatedAt time.Time `json:"created_at"`
}
// ListUsers lists all IAM users in MinIO
func (s *Service) ListUsers(ctx context.Context) ([]*User, error) {
users, err := s.adminClient.ListUsers(ctx)
if err != nil {
return nil, fmt.Errorf("failed to list users: %w", err)
}
result := make([]*User, 0, len(users))
for accessKey, userInfo := range users {
status := "enabled"
if userInfo.Status == madmin.AccountDisabled {
status = "disabled"
}
// MinIO doesn't provide creation date, use current time
result = append(result, &User{
AccessKey: accessKey,
Status: status,
CreatedAt: time.Now(),
})
}
return result, nil
}
// CreateUser creates a new IAM user in MinIO
func (s *Service) CreateUser(ctx context.Context, accessKey, secretKey string) error {
err := s.adminClient.AddUser(ctx, accessKey, secretKey)
if err != nil {
return fmt.Errorf("failed to create user: %w", err)
}
return nil
}
// DeleteUser deletes an IAM user from MinIO
func (s *Service) DeleteUser(ctx context.Context, accessKey string) error {
err := s.adminClient.RemoveUser(ctx, accessKey)
if err != nil {
return fmt.Errorf("failed to delete user: %w", err)
}
return nil
}
// ServiceAccount represents a MinIO service account (access key)
type ServiceAccount struct {
AccessKey string `json:"access_key"`
SecretKey string `json:"secret_key,omitempty"` // Only returned on creation
ParentUser string `json:"parent_user"`
Expiration time.Time `json:"expiration,omitempty"`
CreatedAt time.Time `json:"created_at"`
}
// ListServiceAccounts lists all service accounts in MinIO
func (s *Service) ListServiceAccounts(ctx context.Context) ([]*ServiceAccount, error) {
accounts, err := s.adminClient.ListServiceAccounts(ctx, "")
if err != nil {
return nil, fmt.Errorf("failed to list service accounts: %w", err)
}
result := make([]*ServiceAccount, 0, len(accounts.Accounts))
for _, account := range accounts.Accounts {
var expiration time.Time
if account.Expiration != nil {
expiration = *account.Expiration
}
result = append(result, &ServiceAccount{
AccessKey: account.AccessKey,
ParentUser: account.ParentUser,
Expiration: expiration,
CreatedAt: time.Now(), // MinIO doesn't provide creation date
})
}
return result, nil
}
// CreateServiceAccount creates a new service account (access key) in MinIO
func (s *Service) CreateServiceAccount(ctx context.Context, parentUser string, policy string, expiration *time.Time) (*ServiceAccount, error) {
opts := madmin.AddServiceAccountReq{
TargetUser: parentUser,
}
if policy != "" {
opts.Policy = json.RawMessage(policy)
}
if expiration != nil {
opts.Expiration = expiration
}
creds, err := s.adminClient.AddServiceAccount(ctx, opts)
if err != nil {
return nil, fmt.Errorf("failed to create service account: %w", err)
}
return &ServiceAccount{
AccessKey: creds.AccessKey,
SecretKey: creds.SecretKey,
ParentUser: parentUser,
Expiration: creds.Expiration,
CreatedAt: time.Now(),
}, nil
}
// DeleteServiceAccount deletes a service account from MinIO
func (s *Service) DeleteServiceAccount(ctx context.Context, accessKey string) error {
err := s.adminClient.DeleteServiceAccount(ctx, accessKey)
if err != nil {
return fmt.Errorf("failed to delete service account: %w", err)
}
return nil
}

511
backend/internal/object_storage/setup.go Normal file
View File

@@ -0,0 +1,511 @@
package object_storage
import (
"context"
"database/sql"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"time"
"github.com/atlasos/calypso/internal/common/database"
"github.com/atlasos/calypso/internal/common/logger"
)
// SetupService handles object storage setup operations
type SetupService struct {
db *database.DB
logger *logger.Logger
}
// NewSetupService creates a new setup service
func NewSetupService(db *database.DB, log *logger.Logger) *SetupService {
return &SetupService{
db: db,
logger: log,
}
}
// PoolDatasetInfo represents a pool with its datasets
type PoolDatasetInfo struct {
PoolID string `json:"pool_id"`
PoolName string `json:"pool_name"`
Datasets []DatasetInfo `json:"datasets"`
}
// DatasetInfo represents a dataset that can be used for object storage
type DatasetInfo struct {
ID string `json:"id"`
Name string `json:"name"`
FullName string `json:"full_name"` // pool/dataset
MountPoint string `json:"mount_point"`
Type string `json:"type"`
UsedBytes int64 `json:"used_bytes"`
AvailableBytes int64 `json:"available_bytes"`
}
// GetAvailableDatasets returns all pools with their datasets that can be used for object storage
func (s *SetupService) GetAvailableDatasets(ctx context.Context) ([]PoolDatasetInfo, error) {
// Get all pools
poolsQuery := `
SELECT id, name
FROM zfs_pools
WHERE is_active = true
ORDER BY name
`
rows, err := s.db.QueryContext(ctx, poolsQuery)
if err != nil {
return nil, fmt.Errorf("failed to query pools: %w", err)
}
defer rows.Close()
var pools []PoolDatasetInfo
for rows.Next() {
var pool PoolDatasetInfo
if err := rows.Scan(&pool.PoolID, &pool.PoolName); err != nil {
s.logger.Warn("Failed to scan pool", "error", err)
continue
}
// Get datasets for this pool
datasetsQuery := `
SELECT id, name, type, mount_point, used_bytes, available_bytes
FROM zfs_datasets
WHERE pool_name = $1 AND type = 'filesystem'
ORDER BY name
`
datasetRows, err := s.db.QueryContext(ctx, datasetsQuery, pool.PoolName)
if err != nil {
s.logger.Warn("Failed to query datasets", "pool", pool.PoolName, "error", err)
pool.Datasets = []DatasetInfo{}
pools = append(pools, pool)
continue
}
var datasets []DatasetInfo
for datasetRows.Next() {
var ds DatasetInfo
var mountPoint sql.NullString
if err := datasetRows.Scan(&ds.ID, &ds.Name, &ds.Type, &mountPoint, &ds.UsedBytes, &ds.AvailableBytes); err != nil {
s.logger.Warn("Failed to scan dataset", "error", err)
continue
}
ds.FullName = fmt.Sprintf("%s/%s", pool.PoolName, ds.Name)
if mountPoint.Valid {
ds.MountPoint = mountPoint.String
} else {
ds.MountPoint = ""
}
datasets = append(datasets, ds)
}
datasetRows.Close()
pool.Datasets = datasets
pools = append(pools, pool)
}
return pools, nil
}
// SetupRequest represents a request to setup object storage
type SetupRequest struct {
PoolName string `json:"pool_name" binding:"required"`
DatasetName string `json:"dataset_name" binding:"required"`
CreateNew bool `json:"create_new"` // If true, create new dataset instead of using existing
}
// SetupResponse represents the response after setup
type SetupResponse struct {
DatasetPath string `json:"dataset_path"`
MountPoint string `json:"mount_point"`
Message string `json:"message"`
}
// SetupObjectStorage configures MinIO to use a specific ZFS dataset
func (s *SetupService) SetupObjectStorage(ctx context.Context, req SetupRequest) (*SetupResponse, error) {
var datasetPath, mountPoint string
// Normalize dataset name - if it already contains pool name, use it as-is
var fullDatasetName string
if strings.HasPrefix(req.DatasetName, req.PoolName+"/") {
// Dataset name already includes pool name (e.g., "pool/dataset")
fullDatasetName = req.DatasetName
} else {
// Dataset name is just the name (e.g., "dataset"), combine with pool
fullDatasetName = fmt.Sprintf("%s/%s", req.PoolName, req.DatasetName)
}
if req.CreateNew {
// Create new dataset for object storage
// Check if dataset already exists
checkCmd := exec.CommandContext(ctx, "sudo", "zfs", "list", "-H", "-o", "name", fullDatasetName)
if err := checkCmd.Run(); err == nil {
return nil, fmt.Errorf("dataset %s already exists", fullDatasetName)
}
// Create dataset
createCmd := exec.CommandContext(ctx, "sudo", "zfs", "create", fullDatasetName)
if output, err := createCmd.CombinedOutput(); err != nil {
return nil, fmt.Errorf("failed to create dataset: %s - %w", string(output), err)
}
// Get mount point
getMountCmd := exec.CommandContext(ctx, "sudo", "zfs", "get", "-H", "-o", "value", "mountpoint", fullDatasetName)
mountOutput, err := getMountCmd.Output()
if err != nil {
return nil, fmt.Errorf("failed to get mount point: %w", err)
}
mountPoint = strings.TrimSpace(string(mountOutput))
datasetPath = fullDatasetName
s.logger.Info("Created new dataset for object storage", "dataset", fullDatasetName, "mount_point", mountPoint)
} else {
// Use existing dataset
// fullDatasetName already set above
// Verify dataset exists
checkCmd := exec.CommandContext(ctx, "sudo", "zfs", "list", "-H", "-o", "name", fullDatasetName)
if err := checkCmd.Run(); err != nil {
return nil, fmt.Errorf("dataset %s does not exist", fullDatasetName)
}
// Get mount point
getMountCmd := exec.CommandContext(ctx, "sudo", "zfs", "get", "-H", "-o", "value", "mountpoint", fullDatasetName)
mountOutput, err := getMountCmd.Output()
if err != nil {
return nil, fmt.Errorf("failed to get mount point: %w", err)
}
mountPoint = strings.TrimSpace(string(mountOutput))
datasetPath = fullDatasetName
s.logger.Info("Using existing dataset for object storage", "dataset", fullDatasetName, "mount_point", mountPoint)
}
// Ensure mount point directory exists
if mountPoint != "none" && mountPoint != "" {
if err := os.MkdirAll(mountPoint, 0755); err != nil {
return nil, fmt.Errorf("failed to create mount point directory: %w", err)
}
} else {
// If no mount point, use default path
mountPoint = filepath.Join("/opt/calypso/data/pool", req.PoolName, req.DatasetName)
if err := os.MkdirAll(mountPoint, 0755); err != nil {
return nil, fmt.Errorf("failed to create default directory: %w", err)
}
}
// Update MinIO configuration to use the selected dataset
if err := s.updateMinIOConfig(ctx, mountPoint); err != nil {
s.logger.Warn("Failed to update MinIO configuration", "error", err)
// Continue anyway, configuration is saved to database
}
// Save configuration to database
_, err := s.db.ExecContext(ctx, `
INSERT INTO object_storage_config (dataset_path, mount_point, pool_name, dataset_name, created_at, updated_at)
VALUES ($1, $2, $3, $4, NOW(), NOW())
ON CONFLICT (id) DO UPDATE
SET dataset_path = $1, mount_point = $2, pool_name = $3, dataset_name = $4, updated_at = NOW()
`, datasetPath, mountPoint, req.PoolName, req.DatasetName)
if err != nil {
// If table doesn't exist, just log warning
s.logger.Warn("Failed to save configuration to database (table may not exist)", "error", err)
}
return &SetupResponse{
DatasetPath: datasetPath,
MountPoint: mountPoint,
Message: fmt.Sprintf("Object storage configured to use dataset %s at %s. MinIO service needs to be restarted to use the new dataset.", datasetPath, mountPoint),
}, nil
}
// GetCurrentSetup returns the current object storage configuration
func (s *SetupService) GetCurrentSetup(ctx context.Context) (*SetupResponse, error) {
// Check if table exists first
var tableExists bool
checkQuery := `
SELECT EXISTS (
SELECT FROM information_schema.tables
WHERE table_schema = 'public'
AND table_name = 'object_storage_config'
)
`
err := s.db.QueryRowContext(ctx, checkQuery).Scan(&tableExists)
if err != nil {
s.logger.Warn("Failed to check if object_storage_config table exists", "error", err)
return nil, nil // Return nil if can't check
}
if !tableExists {
s.logger.Debug("object_storage_config table does not exist")
return nil, nil // No table, no configuration
}
query := `
SELECT dataset_path, mount_point, pool_name, dataset_name
FROM object_storage_config
ORDER BY updated_at DESC
LIMIT 1
`
var resp SetupResponse
var poolName, datasetName string
err = s.db.QueryRowContext(ctx, query).Scan(&resp.DatasetPath, &resp.MountPoint, &poolName, &datasetName)
if err == sql.ErrNoRows {
s.logger.Debug("No configuration found in database")
return nil, nil // No configuration found
}
if err != nil {
// Check if error is due to table not existing or permission denied
errStr := err.Error()
if strings.Contains(errStr, "does not exist") || strings.Contains(errStr, "permission denied") {
s.logger.Debug("Table does not exist or permission denied, returning nil", "error", errStr)
return nil, nil // Return nil instead of error
}
s.logger.Error("Failed to scan current setup", "error", err)
return nil, fmt.Errorf("failed to get current setup: %w", err)
}
s.logger.Debug("Found current setup", "dataset_path", resp.DatasetPath, "mount_point", resp.MountPoint, "pool", poolName, "dataset", datasetName)
// Use dataset_path directly since it already contains the full path
resp.Message = fmt.Sprintf("Using dataset %s at %s", resp.DatasetPath, resp.MountPoint)
return &resp, nil
}
// UpdateObjectStorage updates the object storage configuration to use a different dataset
// This will update the configuration but won't migrate existing data
func (s *SetupService) UpdateObjectStorage(ctx context.Context, req SetupRequest) (*SetupResponse, error) {
// First check if there's existing configuration
currentSetup, err := s.GetCurrentSetup(ctx)
if err != nil {
return nil, fmt.Errorf("failed to check current setup: %w", err)
}
if currentSetup == nil {
// No existing setup, just do normal setup
return s.SetupObjectStorage(ctx, req)
}
// There's existing setup, proceed with update
var datasetPath, mountPoint string
// Normalize dataset name - if it already contains pool name, use it as-is
var fullDatasetName string
if strings.HasPrefix(req.DatasetName, req.PoolName+"/") {
// Dataset name already includes pool name (e.g., "pool/dataset")
fullDatasetName = req.DatasetName
} else {
// Dataset name is just the name (e.g., "dataset"), combine with pool
fullDatasetName = fmt.Sprintf("%s/%s", req.PoolName, req.DatasetName)
}
if req.CreateNew {
// Create new dataset for object storage
// Check if dataset already exists
checkCmd := exec.CommandContext(ctx, "sudo", "zfs", "list", "-H", "-o", "name", fullDatasetName)
if err := checkCmd.Run(); err == nil {
return nil, fmt.Errorf("dataset %s already exists", fullDatasetName)
}
// Create dataset
createCmd := exec.CommandContext(ctx, "sudo", "zfs", "create", fullDatasetName)
if output, err := createCmd.CombinedOutput(); err != nil {
return nil, fmt.Errorf("failed to create dataset: %s - %w", string(output), err)
}
// Get mount point
getMountCmd := exec.CommandContext(ctx, "sudo", "zfs", "get", "-H", "-o", "value", "mountpoint", fullDatasetName)
mountOutput, err := getMountCmd.Output()
if err != nil {
return nil, fmt.Errorf("failed to get mount point: %w", err)
}
mountPoint = strings.TrimSpace(string(mountOutput))
datasetPath = fullDatasetName
s.logger.Info("Created new dataset for object storage update", "dataset", fullDatasetName, "mount_point", mountPoint)
} else {
// Use existing dataset
// fullDatasetName already set above
// Verify dataset exists
checkCmd := exec.CommandContext(ctx, "sudo", "zfs", "list", "-H", "-o", "name", fullDatasetName)
if err := checkCmd.Run(); err != nil {
return nil, fmt.Errorf("dataset %s does not exist", fullDatasetName)
}
// Get mount point
getMountCmd := exec.CommandContext(ctx, "sudo", "zfs", "get", "-H", "-o", "value", "mountpoint", fullDatasetName)
mountOutput, err := getMountCmd.Output()
if err != nil {
return nil, fmt.Errorf("failed to get mount point: %w", err)
}
mountPoint = strings.TrimSpace(string(mountOutput))
datasetPath = fullDatasetName
s.logger.Info("Using existing dataset for object storage update", "dataset", fullDatasetName, "mount_point", mountPoint)
}
// Ensure mount point directory exists
if mountPoint != "none" && mountPoint != "" {
if err := os.MkdirAll(mountPoint, 0755); err != nil {
return nil, fmt.Errorf("failed to create mount point directory: %w", err)
}
} else {
// If no mount point, use default path
mountPoint = filepath.Join("/opt/calypso/data/pool", req.PoolName, req.DatasetName)
if err := os.MkdirAll(mountPoint, 0755); err != nil {
return nil, fmt.Errorf("failed to create default directory: %w", err)
}
}
// Update configuration in database
_, err = s.db.ExecContext(ctx, `
UPDATE object_storage_config
SET dataset_path = $1, mount_point = $2, pool_name = $3, dataset_name = $4, updated_at = NOW()
WHERE id = (SELECT id FROM object_storage_config ORDER BY updated_at DESC LIMIT 1)
`, datasetPath, mountPoint, req.PoolName, req.DatasetName)
if err != nil {
// If update fails, try insert
_, err = s.db.ExecContext(ctx, `
INSERT INTO object_storage_config (dataset_path, mount_point, pool_name, dataset_name, created_at, updated_at)
VALUES ($1, $2, $3, $4, NOW(), NOW())
ON CONFLICT (dataset_path) DO UPDATE
SET mount_point = $2, pool_name = $3, dataset_name = $4, updated_at = NOW()
`, datasetPath, mountPoint, req.PoolName, req.DatasetName)
if err != nil {
s.logger.Warn("Failed to update configuration in database", "error", err)
}
}
// Update MinIO configuration to use the selected dataset
if err := s.updateMinIOConfig(ctx, mountPoint); err != nil {
s.logger.Warn("Failed to update MinIO configuration", "error", err)
// Continue anyway, configuration is saved to database
} else {
// Restart MinIO service to apply new configuration
if err := s.restartMinIOService(ctx); err != nil {
s.logger.Warn("Failed to restart MinIO service", "error", err)
// Continue anyway, user can restart manually
}
}
return &SetupResponse{
DatasetPath: datasetPath,
MountPoint: mountPoint,
Message: fmt.Sprintf("Object storage updated to use dataset %s at %s. Note: Existing data in previous dataset (%s) is not migrated automatically. MinIO service has been restarted.", datasetPath, mountPoint, currentSetup.DatasetPath),
}, nil
}
// updateMinIOConfig updates MinIO configuration file to use dataset mount point directly
// Note: MinIO erasure coding requires direct directory paths, not symlinks
func (s *SetupService) updateMinIOConfig(ctx context.Context, datasetMountPoint string) error {
configFile := "/opt/calypso/conf/minio/minio.conf"
// Ensure dataset mount point directory exists and has correct ownership
if err := os.MkdirAll(datasetMountPoint, 0755); err != nil {
return fmt.Errorf("failed to create dataset mount point directory: %w", err)
}
// Set ownership to minio-user so MinIO can write to it
if err := exec.CommandContext(ctx, "sudo", "chown", "-R", "minio-user:minio-user", datasetMountPoint).Run(); err != nil {
s.logger.Warn("Failed to set ownership on dataset mount point", "path", datasetMountPoint, "error", err)
// Continue anyway, might already have correct ownership
}
// Set permissions
if err := exec.CommandContext(ctx, "sudo", "chmod", "755", datasetMountPoint).Run(); err != nil {
s.logger.Warn("Failed to set permissions on dataset mount point", "path", datasetMountPoint, "error", err)
}
s.logger.Info("Prepared dataset mount point for MinIO", "path", datasetMountPoint)
// Read current config file
configContent, err := os.ReadFile(configFile)
if err != nil {
// If file doesn't exist, create it
if os.IsNotExist(err) {
configContent = []byte(fmt.Sprintf("MINIO_ROOT_USER=admin\nMINIO_ROOT_PASSWORD=HqBX1IINqFynkWFa\nMINIO_VOLUMES=%s\n", datasetMountPoint))
} else {
return fmt.Errorf("failed to read MinIO config file: %w", err)
}
} else {
// Update MINIO_VOLUMES in config
lines := strings.Split(string(configContent), "\n")
updated := false
for i, line := range lines {
if strings.HasPrefix(strings.TrimSpace(line), "MINIO_VOLUMES=") {
lines[i] = fmt.Sprintf("MINIO_VOLUMES=%s", datasetMountPoint)
updated = true
break
}
}
if !updated {
// Add MINIO_VOLUMES if not found
lines = append(lines, fmt.Sprintf("MINIO_VOLUMES=%s", datasetMountPoint))
}
configContent = []byte(strings.Join(lines, "\n"))
}
// Write updated config using sudo
// Write temp file to a location we can write to
userTempFile := fmt.Sprintf("/tmp/minio.conf.%d.tmp", os.Getpid())
if err := os.WriteFile(userTempFile, configContent, 0644); err != nil {
return fmt.Errorf("failed to write temp config file: %w", err)
}
defer os.Remove(userTempFile) // Cleanup
// Copy temp file to config location with sudo
if err := exec.CommandContext(ctx, "sudo", "cp", userTempFile, configFile).Run(); err != nil {
return fmt.Errorf("failed to update config file: %w", err)
}
// Set proper ownership and permissions
if err := exec.CommandContext(ctx, "sudo", "chown", "minio-user:minio-user", configFile).Run(); err != nil {
s.logger.Warn("Failed to set config file ownership", "error", err)
}
if err := exec.CommandContext(ctx, "sudo", "chmod", "644", configFile).Run(); err != nil {
s.logger.Warn("Failed to set config file permissions", "error", err)
}
s.logger.Info("Updated MinIO configuration", "config_file", configFile, "volumes", datasetMountPoint)
return nil
}
// restartMinIOService restarts the MinIO service to apply new configuration
func (s *SetupService) restartMinIOService(ctx context.Context) error {
// Restart MinIO service using sudo
cmd := exec.CommandContext(ctx, "sudo", "systemctl", "restart", "minio.service")
if err := cmd.Run(); err != nil {
return fmt.Errorf("failed to restart MinIO service: %w", err)
}
// Wait a moment for service to start
time.Sleep(2 * time.Second)
// Verify service is running
checkCmd := exec.CommandContext(ctx, "sudo", "systemctl", "is-active", "minio.service")
output, err := checkCmd.Output()
if err != nil {
return fmt.Errorf("failed to check MinIO service status: %w", err)
}
status := strings.TrimSpace(string(output))
if status != "active" {
return fmt.Errorf("MinIO service is not active after restart, status: %s", status)
}
s.logger.Info("MinIO service restarted successfully")
return nil
}

323
backend/internal/scst/handler.go
View File

@@ -3,11 +3,13 @@ package scst
import (
"fmt"
"net/http"
"strings"
"github.com/atlasos/calypso/internal/common/database"
"github.com/atlasos/calypso/internal/common/logger"
"github.com/atlasos/calypso/internal/tasks"
"github.com/gin-gonic/gin"
"github.com/go-playground/validator/v10"
)
// Handler handles SCST-related API requests
@@ -37,6 +39,11 @@ func (h *Handler) ListTargets(c *gin.Context) {
return
}
// Ensure we return an empty array instead of null
if targets == nil {
targets = []Target{}
}
c.JSON(http.StatusOK, gin.H{"targets": targets})
}
@@ -112,6 +119,11 @@ func (h *Handler) CreateTarget(c *gin.Context) {
return
}
// Set alias to name for frontend compatibility (same as ListTargets)
target.Alias = target.Name
// LUNCount will be 0 for newly created target
target.LUNCount = 0
c.JSON(http.StatusCreated, target)
}
@@ -119,7 +131,7 @@ func (h *Handler) CreateTarget(c *gin.Context) {
type AddLUNRequest struct {
DeviceName string `json:"device_name" binding:"required"`
DevicePath string `json:"device_path" binding:"required"`
LUNNumber int `json:"lun_number" binding:"required"`
LUNNumber int `json:"lun_number"` // Note: cannot use binding:"required" for int as 0 is valid
HandlerType string `json:"handler_type" binding:"required"`
}
@@ -136,17 +148,45 @@ func (h *Handler) AddLUN(c *gin.Context) {
var req AddLUNRequest
if err := c.ShouldBindJSON(&req); err != nil {
h.logger.Error("Failed to bind AddLUN request", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": fmt.Sprintf("invalid request: %v", err)})
// Provide more detailed error message
if validationErr, ok := err.(validator.ValidationErrors); ok {
var errorMessages []string
for _, fieldErr := range validationErr {
errorMessages = append(errorMessages, fmt.Sprintf("%s is required", fieldErr.Field()))
}
c.JSON(http.StatusBadRequest, gin.H{"error": fmt.Sprintf("validation failed: %s", strings.Join(errorMessages, ", "))})
} else {
// Extract error message without full struct name
errMsg := err.Error()
if idx := strings.Index(errMsg, "Key: '"); idx >= 0 {
// Extract field name from error message
fieldStart := idx + 6 // Length of "Key: '"
if fieldEnd := strings.Index(errMsg[fieldStart:], "'"); fieldEnd >= 0 {
fieldName := errMsg[fieldStart : fieldStart+fieldEnd]
c.JSON(http.StatusBadRequest, gin.H{"error": fmt.Sprintf("invalid or missing field: %s", fieldName)})
} else {
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request format"})
}
} else {
c.JSON(http.StatusBadRequest, gin.H{"error": fmt.Sprintf("invalid request: %v", err)})
}
}
return
}
// Validate required fields
// Validate required fields (additional check in case binding doesn't catch it)
if req.DeviceName == "" || req.DevicePath == "" || req.HandlerType == "" {
h.logger.Error("Missing required fields in AddLUN request", "device_name", req.DeviceName, "device_path", req.DevicePath, "handler_type", req.HandlerType)
c.JSON(http.StatusBadRequest, gin.H{"error": "device_name, device_path, and handler_type are required"})
return
}
// Validate LUN number range
if req.LUNNumber < 0 || req.LUNNumber > 255 {
c.JSON(http.StatusBadRequest, gin.H{"error": "lun_number must be between 0 and 255"})
return
}
if err := h.service.AddLUN(c.Request.Context(), target.IQN, req.DeviceName, req.DevicePath, req.LUNNumber, req.HandlerType); err != nil {
h.logger.Error("Failed to add LUN", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
@@ -156,6 +196,48 @@ func (h *Handler) AddLUN(c *gin.Context) {
c.JSON(http.StatusOK, gin.H{"message": "LUN added successfully"})
}
// RemoveLUN removes a LUN from a target
func (h *Handler) RemoveLUN(c *gin.Context) {
targetID := c.Param("id")
lunID := c.Param("lunId")
// Get target
target, err := h.service.GetTarget(c.Request.Context(), targetID)
if err != nil {
c.JSON(http.StatusNotFound, gin.H{"error": "target not found"})
return
}
// Get LUN to get the LUN number
var lunNumber int
err = h.db.QueryRowContext(c.Request.Context(),
"SELECT lun_number FROM scst_luns WHERE id = $1 AND target_id = $2",
lunID, targetID,
).Scan(&lunNumber)
if err != nil {
if strings.Contains(err.Error(), "no rows") {
// LUN already deleted from database - check if it still exists in SCST
// Try to get LUN number from URL or try common LUN numbers
// For now, return success since it's already deleted (idempotent)
h.logger.Info("LUN not found in database, may already be deleted", "lun_id", lunID, "target_id", targetID)
c.JSON(http.StatusOK, gin.H{"message": "LUN already removed or not found"})
return
}
h.logger.Error("Failed to get LUN", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to get LUN"})
return
}
// Remove LUN
if err := h.service.RemoveLUN(c.Request.Context(), target.IQN, lunNumber); err != nil {
h.logger.Error("Failed to remove LUN", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"message": "LUN removed successfully"})
}
// AddInitiatorRequest represents an initiator addition request
type AddInitiatorRequest struct {
InitiatorIQN string `json:"initiator_iqn" binding:"required"`
@@ -186,6 +268,45 @@ func (h *Handler) AddInitiator(c *gin.Context) {
c.JSON(http.StatusOK, gin.H{"message": "Initiator added successfully"})
}
// AddInitiatorToGroupRequest represents a request to add an initiator to a group
type AddInitiatorToGroupRequest struct {
InitiatorIQN string `json:"initiator_iqn" binding:"required"`
}
// AddInitiatorToGroup adds an initiator to a specific group
func (h *Handler) AddInitiatorToGroup(c *gin.Context) {
groupID := c.Param("id")
var req AddInitiatorToGroupRequest
if err := c.ShouldBindJSON(&req); err != nil {
validationErrors := make(map[string]string)
if ve, ok := err.(validator.ValidationErrors); ok {
for _, fe := range ve {
field := strings.ToLower(fe.Field())
validationErrors[field] = fmt.Sprintf("Field '%s' is required", field)
}
}
c.JSON(http.StatusBadRequest, gin.H{
"error": "invalid request",
"validation_errors": validationErrors,
})
return
}
err := h.service.AddInitiatorToGroup(c.Request.Context(), groupID, req.InitiatorIQN)
if err != nil {
if strings.Contains(err.Error(), "not found") || strings.Contains(err.Error(), "already exists") || strings.Contains(err.Error(), "single initiator only") {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
h.logger.Error("Failed to add initiator to group", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to add initiator to group"})
return
}
c.JSON(http.StatusOK, gin.H{"message": "Initiator added to group successfully"})
}
// ListAllInitiators lists all initiators across all targets
func (h *Handler) ListAllInitiators(c *gin.Context) {
initiators, err := h.service.ListAllInitiators(c.Request.Context())
@@ -440,6 +561,23 @@ func (h *Handler) DisableTarget(c *gin.Context) {
c.JSON(http.StatusOK, gin.H{"message": "Target disabled successfully"})
}
// DeleteTarget deletes a target
func (h *Handler) DeleteTarget(c *gin.Context) {
targetID := c.Param("id")
if err := h.service.DeleteTarget(c.Request.Context(), targetID); err != nil {
if err.Error() == "target not found" {
c.JSON(http.StatusNotFound, gin.H{"error": "target not found"})
return
}
h.logger.Error("Failed to delete target", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"message": "Target deleted successfully"})
}
// DeletePortal deletes a portal
func (h *Handler) DeletePortal(c *gin.Context) {
id := c.Param("id")
@@ -474,3 +612,182 @@ func (h *Handler) GetPortal(c *gin.Context) {
c.JSON(http.StatusOK, portal)
}
// CreateInitiatorGroupRequest represents a request to create an initiator group
type CreateInitiatorGroupRequest struct {
TargetID string `json:"target_id" binding:"required"`
GroupName string `json:"group_name" binding:"required"`
}
// CreateInitiatorGroup creates a new initiator group
func (h *Handler) CreateInitiatorGroup(c *gin.Context) {
var req CreateInitiatorGroupRequest
if err := c.ShouldBindJSON(&req); err != nil {
validationErrors := make(map[string]string)
if ve, ok := err.(validator.ValidationErrors); ok {
for _, fe := range ve {
field := strings.ToLower(fe.Field())
validationErrors[field] = fmt.Sprintf("Field '%s' is required", field)
}
}
c.JSON(http.StatusBadRequest, gin.H{
"error": "invalid request",
"validation_errors": validationErrors,
})
return
}
group, err := h.service.CreateInitiatorGroup(c.Request.Context(), req.TargetID, req.GroupName)
if err != nil {
if strings.Contains(err.Error(), "already exists") || strings.Contains(err.Error(), "not found") {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
h.logger.Error("Failed to create initiator group", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to create initiator group"})
return
}
c.JSON(http.StatusOK, group)
}
// UpdateInitiatorGroupRequest represents a request to update an initiator group
type UpdateInitiatorGroupRequest struct {
GroupName string `json:"group_name" binding:"required"`
}
// UpdateInitiatorGroup updates an initiator group
func (h *Handler) UpdateInitiatorGroup(c *gin.Context) {
groupID := c.Param("id")
var req UpdateInitiatorGroupRequest
if err := c.ShouldBindJSON(&req); err != nil {
validationErrors := make(map[string]string)
if ve, ok := err.(validator.ValidationErrors); ok {
for _, fe := range ve {
field := strings.ToLower(fe.Field())
validationErrors[field] = fmt.Sprintf("Field '%s' is required", field)
}
}
c.JSON(http.StatusBadRequest, gin.H{
"error": "invalid request",
"validation_errors": validationErrors,
})
return
}
group, err := h.service.UpdateInitiatorGroup(c.Request.Context(), groupID, req.GroupName)
if err != nil {
if strings.Contains(err.Error(), "not found") || strings.Contains(err.Error(), "already exists") {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
h.logger.Error("Failed to update initiator group", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to update initiator group"})
return
}
c.JSON(http.StatusOK, group)
}
// DeleteInitiatorGroup deletes an initiator group
func (h *Handler) DeleteInitiatorGroup(c *gin.Context) {
groupID := c.Param("id")
err := h.service.DeleteInitiatorGroup(c.Request.Context(), groupID)
if err != nil {
if strings.Contains(err.Error(), "not found") {
c.JSON(http.StatusNotFound, gin.H{"error": err.Error()})
return
}
if strings.Contains(err.Error(), "cannot delete") || strings.Contains(err.Error(), "contains") {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
h.logger.Error("Failed to delete initiator group", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to delete initiator group"})
return
}
c.JSON(http.StatusOK, gin.H{"message": "initiator group deleted successfully"})
}
// GetInitiatorGroup retrieves an initiator group by ID
func (h *Handler) GetInitiatorGroup(c *gin.Context) {
groupID := c.Param("id")
group, err := h.service.GetInitiatorGroup(c.Request.Context(), groupID)
if err != nil {
if strings.Contains(err.Error(), "not found") {
c.JSON(http.StatusNotFound, gin.H{"error": "initiator group not found"})
return
}
h.logger.Error("Failed to get initiator group", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to get initiator group"})
return
}
c.JSON(http.StatusOK, group)
}
// ListAllInitiatorGroups lists all initiator groups
func (h *Handler) ListAllInitiatorGroups(c *gin.Context) {
groups, err := h.service.ListAllInitiatorGroups(c.Request.Context())
if err != nil {
h.logger.Error("Failed to list initiator groups", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to list initiator groups"})
return
}
if groups == nil {
groups = []InitiatorGroup{}
}
c.JSON(http.StatusOK, gin.H{"groups": groups})
}
// GetConfigFile reads the SCST configuration file content
func (h *Handler) GetConfigFile(c *gin.Context) {
configPath := c.DefaultQuery("path", "/etc/scst.conf")
content, err := h.service.ReadConfigFile(c.Request.Context(), configPath)
if err != nil {
h.logger.Error("Failed to read config file", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{
"content": content,
"path": configPath,
})
}
// UpdateConfigFile writes content to SCST configuration file
func (h *Handler) UpdateConfigFile(c *gin.Context) {
var req struct {
Content string `json:"content" binding:"required"`
Path string `json:"path"`
}
if err := c.ShouldBindJSON(&req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request"})
return
}
configPath := req.Path
if configPath == "" {
configPath = "/etc/scst.conf"
}
if err := h.service.WriteConfigFile(c.Request.Context(), configPath, req.Content); err != nil {
h.logger.Error("Failed to write config file", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{
"message": "Configuration file updated successfully",
"path": configPath,
})
}

1122
backend/internal/scst/service.go
View File

File diff suppressed because it is too large Load Diff

147
backend/internal/shares/handler.go Normal file
View File

@@ -0,0 +1,147 @@
package shares
import (
"net/http"
"github.com/atlasos/calypso/internal/common/database"
"github.com/atlasos/calypso/internal/common/logger"
"github.com/gin-gonic/gin"
"github.com/go-playground/validator/v10"
)
// Handler handles Shares-related API requests
type Handler struct {
service *Service
logger *logger.Logger
}
// NewHandler creates a new Shares handler
func NewHandler(db *database.DB, log *logger.Logger) *Handler {
return &Handler{
service: NewService(db, log),
logger: log,
}
}
// ListShares lists all shares
func (h *Handler) ListShares(c *gin.Context) {
shares, err := h.service.ListShares(c.Request.Context())
if err != nil {
h.logger.Error("Failed to list shares", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to list shares"})
return
}
// Ensure we return an empty array instead of null
if shares == nil {
shares = []*Share{}
}
c.JSON(http.StatusOK, gin.H{"shares": shares})
}
// GetShare retrieves a share by ID
func (h *Handler) GetShare(c *gin.Context) {
shareID := c.Param("id")
share, err := h.service.GetShare(c.Request.Context(), shareID)
if err != nil {
if err.Error() == "share not found" {
c.JSON(http.StatusNotFound, gin.H{"error": "share not found"})
return
}
h.logger.Error("Failed to get share", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to get share"})
return
}
c.JSON(http.StatusOK, share)
}
// CreateShare creates a new share
func (h *Handler) CreateShare(c *gin.Context) {
var req CreateShareRequest
if err := c.ShouldBindJSON(&req); err != nil {
h.logger.Error("Invalid create share request", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request: " + err.Error()})
return
}
// Validate request
validate := validator.New()
if err := validate.Struct(req); err != nil {
c.JSON(http.StatusBadRequest, gin.H{"error": "validation failed: " + err.Error()})
return
}
// Get user ID from context (set by auth middleware)
userID, exists := c.Get("user_id")
if !exists {
c.JSON(http.StatusUnauthorized, gin.H{"error": "unauthorized"})
return
}
share, err := h.service.CreateShare(c.Request.Context(), &req, userID.(string))
if err != nil {
if err.Error() == "dataset not found" {
c.JSON(http.StatusNotFound, gin.H{"error": "dataset not found"})
return
}
if err.Error() == "only filesystem datasets can be shared (not volumes)" {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
if err.Error() == "at least one protocol (NFS or SMB) must be enabled" {
c.JSON(http.StatusBadRequest, gin.H{"error": err.Error()})
return
}
h.logger.Error("Failed to create share", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusCreated, share)
}
// UpdateShare updates an existing share
func (h *Handler) UpdateShare(c *gin.Context) {
shareID := c.Param("id")
var req UpdateShareRequest
if err := c.ShouldBindJSON(&req); err != nil {
h.logger.Error("Invalid update share request", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request: " + err.Error()})
return
}
share, err := h.service.UpdateShare(c.Request.Context(), shareID, &req)
if err != nil {
if err.Error() == "share not found" {
c.JSON(http.StatusNotFound, gin.H{"error": "share not found"})
return
}
h.logger.Error("Failed to update share", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, share)
}
// DeleteShare deletes a share
func (h *Handler) DeleteShare(c *gin.Context) {
shareID := c.Param("id")
err := h.service.DeleteShare(c.Request.Context(), shareID)
if err != nil {
if err.Error() == "share not found" {
c.JSON(http.StatusNotFound, gin.H{"error": "share not found"})
return
}
h.logger.Error("Failed to delete share", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"message": "share deleted successfully"})
}

834
backend/internal/shares/service.go Normal file
View File

@@ -0,0 +1,834 @@
package shares
import (
"context"
"database/sql"
"fmt"
"os"
"os/exec"
"strings"
"time"
"github.com/atlasos/calypso/internal/common/database"
"github.com/atlasos/calypso/internal/common/logger"
"github.com/lib/pq"
)
// Service handles Shares (CIFS/NFS) operations
type Service struct {
db *database.DB
logger *logger.Logger
}
// NewService creates a new Shares service
func NewService(db *database.DB, log *logger.Logger) *Service {
return &Service{
db: db,
logger: log,
}
}
// Share represents a filesystem share (NFS/SMB)
type Share struct {
ID string `json:"id"`
DatasetID string `json:"dataset_id"`
DatasetName string `json:"dataset_name"`
MountPoint string `json:"mount_point"`
ShareType string `json:"share_type"` // 'nfs', 'smb', 'both'
NFSEnabled bool `json:"nfs_enabled"`
NFSOptions string `json:"nfs_options,omitempty"`
NFSClients []string `json:"nfs_clients,omitempty"`
SMBEnabled bool `json:"smb_enabled"`
SMBShareName string `json:"smb_share_name,omitempty"`
SMBPath string `json:"smb_path,omitempty"`
SMBComment string `json:"smb_comment,omitempty"`
SMBGuestOK bool `json:"smb_guest_ok"`
SMBReadOnly bool `json:"smb_read_only"`
SMBBrowseable bool `json:"smb_browseable"`
IsActive bool `json:"is_active"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
CreatedBy string `json:"created_by"`
}
// ListShares lists all shares
func (s *Service) ListShares(ctx context.Context) ([]*Share, error) {
query := `
SELECT
zs.id, zs.dataset_id, zd.name as dataset_name, zd.mount_point,
zs.share_type, zs.nfs_enabled, zs.nfs_options, zs.nfs_clients,
zs.smb_enabled, zs.smb_share_name, zs.smb_path, zs.smb_comment,
zs.smb_guest_ok, zs.smb_read_only, zs.smb_browseable,
zs.is_active, zs.created_at, zs.updated_at, zs.created_by
FROM zfs_shares zs
JOIN zfs_datasets zd ON zs.dataset_id = zd.id
ORDER BY zd.name
`
rows, err := s.db.QueryContext(ctx, query)
if err != nil {
if strings.Contains(err.Error(), "does not exist") {
s.logger.Warn("zfs_shares table does not exist, returning empty list")
return []*Share{}, nil
}
return nil, fmt.Errorf("failed to list shares: %w", err)
}
defer rows.Close()
var shares []*Share
for rows.Next() {
var share Share
var mountPoint sql.NullString
var nfsOptions sql.NullString
var smbShareName sql.NullString
var smbPath sql.NullString
var smbComment sql.NullString
var nfsClients []string
err := rows.Scan(
&share.ID, &share.DatasetID, &share.DatasetName, &mountPoint,
&share.ShareType, &share.NFSEnabled, &nfsOptions, pq.Array(&nfsClients),
&share.SMBEnabled, &smbShareName, &smbPath, &smbComment,
&share.SMBGuestOK, &share.SMBReadOnly, &share.SMBBrowseable,
&share.IsActive, &share.CreatedAt, &share.UpdatedAt, &share.CreatedBy,
)
if err != nil {
s.logger.Error("Failed to scan share row", "error", err)
continue
}
share.NFSClients = nfsClients
if mountPoint.Valid {
share.MountPoint = mountPoint.String
}
if nfsOptions.Valid {
share.NFSOptions = nfsOptions.String
}
if smbShareName.Valid {
share.SMBShareName = smbShareName.String
}
if smbPath.Valid {
share.SMBPath = smbPath.String
}
if smbComment.Valid {
share.SMBComment = smbComment.String
}
shares = append(shares, &share)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("error iterating share rows: %w", err)
}
return shares, nil
}
// GetShare retrieves a share by ID
func (s *Service) GetShare(ctx context.Context, shareID string) (*Share, error) {
query := `
SELECT
zs.id, zs.dataset_id, zd.name as dataset_name, zd.mount_point,
zs.share_type, zs.nfs_enabled, zs.nfs_options, zs.nfs_clients,
zs.smb_enabled, zs.smb_share_name, zs.smb_path, zs.smb_comment,
zs.smb_guest_ok, zs.smb_read_only, zs.smb_browseable,
zs.is_active, zs.created_at, zs.updated_at, zs.created_by
FROM zfs_shares zs
JOIN zfs_datasets zd ON zs.dataset_id = zd.id
WHERE zs.id = $1
`
var share Share
var mountPoint sql.NullString
var nfsOptions sql.NullString
var smbShareName sql.NullString
var smbPath sql.NullString
var smbComment sql.NullString
var nfsClients []string
err := s.db.QueryRowContext(ctx, query, shareID).Scan(
&share.ID, &share.DatasetID, &share.DatasetName, &mountPoint,
&share.ShareType, &share.NFSEnabled, &nfsOptions, pq.Array(&nfsClients),
&share.SMBEnabled, &smbShareName, &smbPath, &smbComment,
&share.SMBGuestOK, &share.SMBReadOnly, &share.SMBBrowseable,
&share.IsActive, &share.CreatedAt, &share.UpdatedAt, &share.CreatedBy,
)
if err != nil {
if err == sql.ErrNoRows {
return nil, fmt.Errorf("share not found")
}
return nil, fmt.Errorf("failed to get share: %w", err)
}
share.NFSClients = nfsClients
if mountPoint.Valid {
share.MountPoint = mountPoint.String
}
if nfsOptions.Valid {
share.NFSOptions = nfsOptions.String
}
if smbShareName.Valid {
share.SMBShareName = smbShareName.String
}
if smbPath.Valid {
share.SMBPath = smbPath.String
}
if smbComment.Valid {
share.SMBComment = smbComment.String
}
return &share, nil
}
// CreateShareRequest represents a share creation request
type CreateShareRequest struct {
DatasetID string `json:"dataset_id" binding:"required"`
NFSEnabled bool `json:"nfs_enabled"`
NFSOptions string `json:"nfs_options"`
NFSClients []string `json:"nfs_clients"`
SMBEnabled bool `json:"smb_enabled"`
SMBShareName string `json:"smb_share_name"`
SMBPath string `json:"smb_path"`
SMBComment string `json:"smb_comment"`
SMBGuestOK bool `json:"smb_guest_ok"`
SMBReadOnly bool `json:"smb_read_only"`
SMBBrowseable bool `json:"smb_browseable"`
}
// CreateShare creates a new share
func (s *Service) CreateShare(ctx context.Context, req *CreateShareRequest, userID string) (*Share, error) {
// Validate dataset exists and is a filesystem (not volume)
// req.DatasetID can be either UUID or dataset name
var datasetID, datasetType, datasetName, mountPoint string
var mountPointNull sql.NullString
// Try to find by ID first (UUID)
err := s.db.QueryRowContext(ctx,
"SELECT id, type, name, mount_point FROM zfs_datasets WHERE id = $1",
req.DatasetID,
).Scan(&datasetID, &datasetType, &datasetName, &mountPointNull)
// If not found by ID, try by name
if err == sql.ErrNoRows {
err = s.db.QueryRowContext(ctx,
"SELECT id, type, name, mount_point FROM zfs_datasets WHERE name = $1",
req.DatasetID,
).Scan(&datasetID, &datasetType, &datasetName, &mountPointNull)
}
if err != nil {
if err == sql.ErrNoRows {
return nil, fmt.Errorf("dataset not found")
}
return nil, fmt.Errorf("failed to validate dataset: %w", err)
}
if mountPointNull.Valid {
mountPoint = mountPointNull.String
} else {
mountPoint = "none"
}
if datasetType != "filesystem" {
return nil, fmt.Errorf("only filesystem datasets can be shared (not volumes)")
}
// Determine share type
shareType := "none"
if req.NFSEnabled && req.SMBEnabled {
shareType = "both"
} else if req.NFSEnabled {
shareType = "nfs"
} else if req.SMBEnabled {
shareType = "smb"
} else {
return nil, fmt.Errorf("at least one protocol (NFS or SMB) must be enabled")
}
// Set default NFS options if not provided
nfsOptions := req.NFSOptions
if nfsOptions == "" {
nfsOptions = "rw,sync,no_subtree_check"
}
// Set default SMB share name if not provided
smbShareName := req.SMBShareName
if smbShareName == "" {
// Extract dataset name from full path (e.g., "pool/dataset" -> "dataset")
parts := strings.Split(datasetName, "/")
smbShareName = parts[len(parts)-1]
}
// Set SMB path (use mount_point if available, otherwise use dataset name)
smbPath := req.SMBPath
if smbPath == "" {
if mountPoint != "" && mountPoint != "none" {
smbPath = mountPoint
} else {
smbPath = fmt.Sprintf("/mnt/%s", strings.ReplaceAll(datasetName, "/", "_"))
}
}
// Insert into database
query := `
INSERT INTO zfs_shares (
dataset_id, share_type, nfs_enabled, nfs_options, nfs_clients,
smb_enabled, smb_share_name, smb_path, smb_comment,
smb_guest_ok, smb_read_only, smb_browseable, is_active, created_by
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14)
RETURNING id, created_at, updated_at
`
var shareID string
var createdAt, updatedAt time.Time
// Handle nfs_clients array - use empty array if nil
nfsClients := req.NFSClients
if nfsClients == nil {
nfsClients = []string{}
}
err = s.db.QueryRowContext(ctx, query,
datasetID, shareType, req.NFSEnabled, nfsOptions, pq.Array(nfsClients),
req.SMBEnabled, smbShareName, smbPath, req.SMBComment,
req.SMBGuestOK, req.SMBReadOnly, req.SMBBrowseable, true, userID,
).Scan(&shareID, &createdAt, &updatedAt)
if err != nil {
return nil, fmt.Errorf("failed to create share: %w", err)
}
// Apply NFS export if enabled
if req.NFSEnabled {
if err := s.applyNFSExport(ctx, mountPoint, nfsOptions, req.NFSClients); err != nil {
s.logger.Error("Failed to apply NFS export", "error", err, "share_id", shareID)
// Don't fail the creation, but log the error
}
}
// Apply SMB share if enabled
if req.SMBEnabled {
if err := s.applySMBShare(ctx, smbShareName, smbPath, req.SMBComment, req.SMBGuestOK, req.SMBReadOnly, req.SMBBrowseable); err != nil {
s.logger.Error("Failed to apply SMB share", "error", err, "share_id", shareID)
// Don't fail the creation, but log the error
}
}
// Return the created share
return s.GetShare(ctx, shareID)
}
// UpdateShareRequest represents a share update request
type UpdateShareRequest struct {
NFSEnabled *bool `json:"nfs_enabled"`
NFSOptions *string `json:"nfs_options"`
NFSClients *[]string `json:"nfs_clients"`
SMBEnabled *bool `json:"smb_enabled"`
SMBShareName *string `json:"smb_share_name"`
SMBComment *string `json:"smb_comment"`
SMBGuestOK *bool `json:"smb_guest_ok"`
SMBReadOnly *bool `json:"smb_read_only"`
SMBBrowseable *bool `json:"smb_browseable"`
IsActive *bool `json:"is_active"`
}
// UpdateShare updates an existing share
func (s *Service) UpdateShare(ctx context.Context, shareID string, req *UpdateShareRequest) (*Share, error) {
// Get current share
share, err := s.GetShare(ctx, shareID)
if err != nil {
return nil, err
}
// Build update query dynamically
updates := []string{}
args := []interface{}{}
argIndex := 1
if req.NFSEnabled != nil {
updates = append(updates, fmt.Sprintf("nfs_enabled = $%d", argIndex))
args = append(args, *req.NFSEnabled)
argIndex++
}
if req.NFSOptions != nil {
updates = append(updates, fmt.Sprintf("nfs_options = $%d", argIndex))
args = append(args, *req.NFSOptions)
argIndex++
}
if req.NFSClients != nil {
updates = append(updates, fmt.Sprintf("nfs_clients = $%d", argIndex))
args = append(args, pq.Array(*req.NFSClients))
argIndex++
}
if req.SMBEnabled != nil {
updates = append(updates, fmt.Sprintf("smb_enabled = $%d", argIndex))
args = append(args, *req.SMBEnabled)
argIndex++
}
if req.SMBShareName != nil {
updates = append(updates, fmt.Sprintf("smb_share_name = $%d", argIndex))
args = append(args, *req.SMBShareName)
argIndex++
}
if req.SMBComment != nil {
updates = append(updates, fmt.Sprintf("smb_comment = $%d", argIndex))
args = append(args, *req.SMBComment)
argIndex++
}
if req.SMBGuestOK != nil {
updates = append(updates, fmt.Sprintf("smb_guest_ok = $%d", argIndex))
args = append(args, *req.SMBGuestOK)
argIndex++
}
if req.SMBReadOnly != nil {
updates = append(updates, fmt.Sprintf("smb_read_only = $%d", argIndex))
args = append(args, *req.SMBReadOnly)
argIndex++
}
if req.SMBBrowseable != nil {
updates = append(updates, fmt.Sprintf("smb_browseable = $%d", argIndex))
args = append(args, *req.SMBBrowseable)
argIndex++
}
if req.IsActive != nil {
updates = append(updates, fmt.Sprintf("is_active = $%d", argIndex))
args = append(args, *req.IsActive)
argIndex++
}
if len(updates) == 0 {
return share, nil // No changes
}
// Update share_type based on enabled protocols
nfsEnabled := share.NFSEnabled
smbEnabled := share.SMBEnabled
if req.NFSEnabled != nil {
nfsEnabled = *req.NFSEnabled
}
if req.SMBEnabled != nil {
smbEnabled = *req.SMBEnabled
}
shareType := "none"
if nfsEnabled && smbEnabled {
shareType = "both"
} else if nfsEnabled {
shareType = "nfs"
} else if smbEnabled {
shareType = "smb"
}
updates = append(updates, fmt.Sprintf("share_type = $%d", argIndex))
args = append(args, shareType)
argIndex++
updates = append(updates, fmt.Sprintf("updated_at = NOW()"))
args = append(args, shareID)
query := fmt.Sprintf(`
UPDATE zfs_shares
SET %s
WHERE id = $%d
`, strings.Join(updates, ", "), argIndex)
_, err = s.db.ExecContext(ctx, query, args...)
if err != nil {
return nil, fmt.Errorf("failed to update share: %w", err)
}
// Re-apply NFS export if NFS is enabled
if nfsEnabled {
nfsOptions := share.NFSOptions
if req.NFSOptions != nil {
nfsOptions = *req.NFSOptions
}
nfsClients := share.NFSClients
if req.NFSClients != nil {
nfsClients = *req.NFSClients
}
if err := s.applyNFSExport(ctx, share.MountPoint, nfsOptions, nfsClients); err != nil {
s.logger.Error("Failed to apply NFS export", "error", err, "share_id", shareID)
}
} else {
// Remove NFS export if disabled
if err := s.removeNFSExport(ctx, share.MountPoint); err != nil {
s.logger.Error("Failed to remove NFS export", "error", err, "share_id", shareID)
}
}
// Re-apply SMB share if SMB is enabled
if smbEnabled {
smbShareName := share.SMBShareName
if req.SMBShareName != nil {
smbShareName = *req.SMBShareName
}
smbPath := share.SMBPath
smbComment := share.SMBComment
if req.SMBComment != nil {
smbComment = *req.SMBComment
}
smbGuestOK := share.SMBGuestOK
if req.SMBGuestOK != nil {
smbGuestOK = *req.SMBGuestOK
}
smbReadOnly := share.SMBReadOnly
if req.SMBReadOnly != nil {
smbReadOnly = *req.SMBReadOnly
}
smbBrowseable := share.SMBBrowseable
if req.SMBBrowseable != nil {
smbBrowseable = *req.SMBBrowseable
}
if err := s.applySMBShare(ctx, smbShareName, smbPath, smbComment, smbGuestOK, smbReadOnly, smbBrowseable); err != nil {
s.logger.Error("Failed to apply SMB share", "error", err, "share_id", shareID)
}
} else {
// Remove SMB share if disabled
if err := s.removeSMBShare(ctx, share.SMBShareName); err != nil {
s.logger.Error("Failed to remove SMB share", "error", err, "share_id", shareID)
}
}
return s.GetShare(ctx, shareID)
}
// DeleteShare deletes a share
func (s *Service) DeleteShare(ctx context.Context, shareID string) error {
// Get share to get mount point and share name
share, err := s.GetShare(ctx, shareID)
if err != nil {
return err
}
// Remove NFS export
if share.NFSEnabled {
if err := s.removeNFSExport(ctx, share.MountPoint); err != nil {
s.logger.Error("Failed to remove NFS export", "error", err, "share_id", shareID)
}
}
// Remove SMB share
if share.SMBEnabled {
if err := s.removeSMBShare(ctx, share.SMBShareName); err != nil {
s.logger.Error("Failed to remove SMB share", "error", err, "share_id", shareID)
}
}
// Delete from database
_, err = s.db.ExecContext(ctx, "DELETE FROM zfs_shares WHERE id = $1", shareID)
if err != nil {
return fmt.Errorf("failed to delete share: %w", err)
}
return nil
}
// applyNFSExport adds or updates an NFS export in /etc/exports
func (s *Service) applyNFSExport(ctx context.Context, mountPoint, options string, clients []string) error {
if mountPoint == "" || mountPoint == "none" {
return fmt.Errorf("mount point is required for NFS export")
}
// Build client list (default to * if empty)
clientList := "*"
if len(clients) > 0 {
clientList = strings.Join(clients, " ")
}
// Build export line (format: /path client(options))
exportLine := fmt.Sprintf("%s %s(%s)", mountPoint, clientList, options)
// Read current exports file (use actual config file, not symlink)
exportsPath := "/opt/calypso/conf/nfs/exports"
exportsContent, err := os.ReadFile(exportsPath)
if err != nil && !os.IsNotExist(err) {
return fmt.Errorf("failed to read exports file: %w", err)
}
lines := strings.Split(string(exportsContent), "\n")
var newLines []string
found := false
// Check if this mount point already exists
for _, line := range lines {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
newLines = append(newLines, line)
continue
}
// Check if this line is for our mount point
if strings.HasPrefix(line, mountPoint+" ") {
newLines = append(newLines, exportLine)
found = true
} else {
newLines = append(newLines, line)
}
}
// Add if not found
if !found {
newLines = append(newLines, exportLine)
}
// Write back to file (use sudo to ensure proper permissions)
newContent := strings.Join(newLines, "\n") + "\n"
tempFile := exportsPath + ".tmp"
if err := os.WriteFile(tempFile, []byte(newContent), 0644); err != nil {
return fmt.Errorf("failed to write temporary exports file: %w", err)
}
// Move temp file to actual location using sudo
cmd := exec.CommandContext(ctx, "sudo", "mv", tempFile, exportsPath)
if output, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("failed to move exports file: %s: %w", string(output), err)
}
// Apply exports (reload NFS exports)
cmd = exec.CommandContext(ctx, "sudo", "exportfs", "-ra")
if output, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("failed to apply exports: %s: %w", string(output), err)
}
// Restart NFS service to ensure changes are picked up
cmd = exec.CommandContext(ctx, "sudo", "systemctl", "restart", "nfs-kernel-server")
if output, err := cmd.CombinedOutput(); err != nil {
s.logger.Warn("Failed to restart NFS service", "error", string(output))
// Don't fail, as exportfs -ra should be sufficient
}
s.logger.Info("NFS export applied", "mount_point", mountPoint, "clients", clientList)
return nil
}
// removeNFSExport removes an NFS export from exports file
func (s *Service) removeNFSExport(ctx context.Context, mountPoint string) error {
if mountPoint == "" || mountPoint == "none" {
return nil // Nothing to remove
}
exportsPath := "/opt/calypso/conf/nfs/exports"
exportsContent, err := os.ReadFile(exportsPath)
if err != nil {
if os.IsNotExist(err) {
return nil // File doesn't exist, nothing to remove
}
return fmt.Errorf("failed to read exports file: %w", err)
}
lines := strings.Split(string(exportsContent), "\n")
var newLines []string
for _, line := range lines {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
newLines = append(newLines, line)
continue
}
// Skip lines for this mount point
if strings.HasPrefix(line, mountPoint+" ") {
continue
}
newLines = append(newLines, line)
}
// Write back to file (use sudo to ensure proper permissions)
newContent := strings.Join(newLines, "\n")
if newContent != "" && !strings.HasSuffix(newContent, "\n") {
newContent += "\n"
}
tempFile := exportsPath + ".tmp"
if err := os.WriteFile(tempFile, []byte(newContent), 0644); err != nil {
return fmt.Errorf("failed to write temporary exports file: %w", err)
}
// Move temp file to actual location using sudo
cmd := exec.CommandContext(ctx, "sudo", "mv", tempFile, exportsPath)
if output, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("failed to move exports file: %s: %w", string(output), err)
}
// Apply exports (reload NFS exports)
cmd = exec.CommandContext(ctx, "sudo", "exportfs", "-ra")
if output, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("failed to apply exports: %s: %w", string(output), err)
}
// Restart NFS service to ensure changes are picked up
cmd = exec.CommandContext(ctx, "sudo", "systemctl", "restart", "nfs-kernel-server")
if output, err := cmd.CombinedOutput(); err != nil {
s.logger.Warn("Failed to restart NFS service", "error", string(output))
// Don't fail, as exportfs -ra should be sufficient
}
s.logger.Info("NFS export removed", "mount_point", mountPoint)
return nil
}
// applySMBShare adds or updates an SMB share in /etc/samba/smb.conf
func (s *Service) applySMBShare(ctx context.Context, shareName, path, comment string, guestOK, readOnly, browseable bool) error {
if shareName == "" {
return fmt.Errorf("SMB share name is required")
}
if path == "" {
return fmt.Errorf("SMB path is required")
}
smbConfPath := "/etc/samba/smb.conf"
smbContent, err := os.ReadFile(smbConfPath)
if err != nil {
return fmt.Errorf("failed to read smb.conf: %w", err)
}
// Parse and update smb.conf
lines := strings.Split(string(smbContent), "\n")
var newLines []string
inShare := false
shareStart := -1
for i, line := range lines {
trimmed := strings.TrimSpace(line)
// Check if we're entering our share section
if strings.HasPrefix(trimmed, "[") && strings.HasSuffix(trimmed, "]") {
sectionName := trimmed[1 : len(trimmed)-1]
if sectionName == shareName {
inShare = true
shareStart = i
continue
} else if inShare {
// We've left our share section, insert the share config here
newLines = append(newLines, s.buildSMBShareConfig(shareName, path, comment, guestOK, readOnly, browseable))
inShare = false
}
}
if inShare {
// Skip lines until we find the next section or end of file
continue
}
newLines = append(newLines, line)
}
// If we were still in the share at the end, add it
if inShare {
newLines = append(newLines, s.buildSMBShareConfig(shareName, path, comment, guestOK, readOnly, browseable))
} else if shareStart == -1 {
// Share doesn't exist, add it at the end
newLines = append(newLines, "")
newLines = append(newLines, s.buildSMBShareConfig(shareName, path, comment, guestOK, readOnly, browseable))
}
// Write back to file
newContent := strings.Join(newLines, "\n")
if err := os.WriteFile(smbConfPath, []byte(newContent), 0644); err != nil {
return fmt.Errorf("failed to write smb.conf: %w", err)
}
// Reload Samba
cmd := exec.CommandContext(ctx, "sudo", "systemctl", "reload", "smbd")
if output, err := cmd.CombinedOutput(); err != nil {
// Try restart if reload fails
cmd = exec.CommandContext(ctx, "sudo", "systemctl", "restart", "smbd")
if output2, err2 := cmd.CombinedOutput(); err2 != nil {
return fmt.Errorf("failed to reload/restart smbd: %s / %s: %w", string(output), string(output2), err2)
}
}
s.logger.Info("SMB share applied", "share_name", shareName, "path", path)
return nil
}
// buildSMBShareConfig builds the SMB share configuration block
func (s *Service) buildSMBShareConfig(shareName, path, comment string, guestOK, readOnly, browseable bool) string {
var config []string
config = append(config, fmt.Sprintf("[%s]", shareName))
if comment != "" {
config = append(config, fmt.Sprintf(" comment = %s", comment))
}
config = append(config, fmt.Sprintf(" path = %s", path))
if guestOK {
config = append(config, " guest ok = yes")
} else {
config = append(config, " guest ok = no")
}
if readOnly {
config = append(config, " read only = yes")
} else {
config = append(config, " read only = no")
}
if browseable {
config = append(config, " browseable = yes")
} else {
config = append(config, " browseable = no")
}
return strings.Join(config, "\n")
}
// removeSMBShare removes an SMB share from /etc/samba/smb.conf
func (s *Service) removeSMBShare(ctx context.Context, shareName string) error {
if shareName == "" {
return nil // Nothing to remove
}
smbConfPath := "/etc/samba/smb.conf"
smbContent, err := os.ReadFile(smbConfPath)
if err != nil {
if os.IsNotExist(err) {
return nil // File doesn't exist, nothing to remove
}
return fmt.Errorf("failed to read smb.conf: %w", err)
}
lines := strings.Split(string(smbContent), "\n")
var newLines []string
inShare := false
for _, line := range lines {
trimmed := strings.TrimSpace(line)
// Check if we're entering our share section
if strings.HasPrefix(trimmed, "[") && strings.HasSuffix(trimmed, "]") {
sectionName := trimmed[1 : len(trimmed)-1]
if sectionName == shareName {
inShare = true
continue
} else if inShare {
// We've left our share section
inShare = false
}
}
if inShare {
// Skip lines in this share section
continue
}
newLines = append(newLines, line)
}
// Write back to file
newContent := strings.Join(newLines, "\n")
if err := os.WriteFile(smbConfPath, []byte(newContent), 0644); err != nil {
return fmt.Errorf("failed to write smb.conf: %w", err)
}
// Reload Samba
cmd := exec.CommandContext(ctx, "sudo", "systemctl", "reload", "smbd")
if output, err := cmd.CombinedOutput(); err != nil {
// Try restart if reload fails
cmd = exec.CommandContext(ctx, "sudo", "systemctl", "restart", "smbd")
if output2, err2 := cmd.CombinedOutput(); err2 != nil {
return fmt.Errorf("failed to reload/restart smbd: %s / %s: %w", string(output), string(output2), err2)
}
}
s.logger.Info("SMB share removed", "share_name", shareName)
return nil
}

4
backend/internal/storage/disk.go
View File

@@ -195,7 +195,7 @@ func (s *DiskService) getZFSPoolForDisk(ctx context.Context, devicePath string)
deviceName := strings.TrimPrefix(devicePath, "/dev/")
// Get all ZFS pools
cmd := exec.CommandContext(ctx, "zpool", "list", "-H", "-o", "name")
cmd := exec.CommandContext(ctx, "sudo", "zpool", "list", "-H", "-o", "name")
output, err := cmd.Output()
if err != nil {
return ""
@@ -208,7 +208,7 @@ func (s *DiskService) getZFSPoolForDisk(ctx context.Context, devicePath string)
}
// Check pool status for this device
statusCmd := exec.CommandContext(ctx, "zpool", "status", poolName)
statusCmd := exec.CommandContext(ctx, "sudo", "zpool", "status", poolName)
statusOutput, err := statusCmd.Output()
if err != nil {
continue

451
backend/internal/storage/handler.go
View File

@@ -15,14 +15,17 @@ import (
// Handler handles storage-related API requests
type Handler struct {
diskService *DiskService
lvmService *LVMService
zfsService *ZFSService
arcService *ARCService
taskEngine *tasks.Engine
db *database.DB
logger *logger.Logger
cache *cache.Cache // Cache for invalidation
diskService *DiskService
lvmService *LVMService
zfsService *ZFSService
snapshotService *SnapshotService
snapshotScheduleService *SnapshotScheduleService
replicationService *ReplicationService
arcService *ARCService
taskEngine *tasks.Engine
db *database.DB
logger *logger.Logger
cache *cache.Cache // Cache for invalidation
}
// SetCache sets the cache instance for cache invalidation
@@ -32,14 +35,18 @@ func (h *Handler) SetCache(c *cache.Cache) {
// NewHandler creates a new storage handler
func NewHandler(db *database.DB, log *logger.Logger) *Handler {
snapshotService := NewSnapshotService(db, log)
return &Handler{
diskService: NewDiskService(db, log),
lvmService: NewLVMService(db, log),
zfsService: NewZFSService(db, log),
arcService: NewARCService(log),
taskEngine: tasks.NewEngine(db, log),
db: db,
logger: log,
diskService: NewDiskService(db, log),
lvmService: NewLVMService(db, log),
zfsService: NewZFSService(db, log),
snapshotService: snapshotService,
snapshotScheduleService: NewSnapshotScheduleService(db, log, snapshotService),
replicationService: NewReplicationService(db, log),
arcService: NewARCService(log),
taskEngine: tasks.NewEngine(db, log),
db: db,
logger: log,
}
}
@@ -509,3 +516,417 @@ func (h *Handler) GetARCStats(c *gin.Context) {
c.JSON(http.StatusOK, stats)
}
// ListSnapshots lists all snapshots, optionally filtered by dataset
func (h *Handler) ListSnapshots(c *gin.Context) {
datasetFilter := c.DefaultQuery("dataset", "")
snapshots, err := h.snapshotService.ListSnapshots(c.Request.Context(), datasetFilter)
if err != nil {
h.logger.Error("Failed to list snapshots", "error", err, "dataset", datasetFilter)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to list snapshots: " + err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"snapshots": snapshots})
}
// CreateSnapshotRequest represents a request to create a snapshot
type CreateSnapshotRequest struct {
Dataset string `json:"dataset" binding:"required"`
Name string `json:"name" binding:"required"`
Recursive bool `json:"recursive"`
}
// CreateSnapshot creates a new snapshot
func (h *Handler) CreateSnapshot(c *gin.Context) {
var req CreateSnapshotRequest
if err := c.ShouldBindJSON(&req); err != nil {
h.logger.Error("Invalid create snapshot request", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request: " + err.Error()})
return
}
if err := h.snapshotService.CreateSnapshot(c.Request.Context(), req.Dataset, req.Name, req.Recursive); err != nil {
h.logger.Error("Failed to create snapshot", "error", err, "dataset", req.Dataset, "name", req.Name)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to create snapshot: " + err.Error()})
return
}
c.JSON(http.StatusCreated, gin.H{"message": "snapshot created successfully"})
}
// DeleteSnapshot deletes a snapshot
func (h *Handler) DeleteSnapshot(c *gin.Context) {
snapshotName := c.Param("name")
recursive := c.DefaultQuery("recursive", "false") == "true"
if err := h.snapshotService.DeleteSnapshot(c.Request.Context(), snapshotName, recursive); err != nil {
h.logger.Error("Failed to delete snapshot", "error", err, "snapshot", snapshotName)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to delete snapshot: " + err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"message": "snapshot deleted successfully"})
}
// RollbackSnapshotRequest represents a request to rollback to a snapshot
type RollbackSnapshotRequest struct {
Force bool `json:"force"`
}
// RollbackSnapshot rolls back a dataset to a snapshot
func (h *Handler) RollbackSnapshot(c *gin.Context) {
snapshotName := c.Param("name")
var req RollbackSnapshotRequest
if err := c.ShouldBindJSON(&req); err != nil {
// Default to false if not provided
req.Force = false
}
if err := h.snapshotService.RollbackSnapshot(c.Request.Context(), snapshotName, req.Force); err != nil {
h.logger.Error("Failed to rollback snapshot", "error", err, "snapshot", snapshotName)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to rollback snapshot: " + err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"message": "snapshot rollback completed successfully"})
}
// CloneSnapshotRequest represents a request to clone a snapshot
type CloneSnapshotRequest struct {
CloneName string `json:"clone_name" binding:"required"`
}
// CloneSnapshot clones a snapshot to a new dataset
func (h *Handler) CloneSnapshot(c *gin.Context) {
snapshotName := c.Param("name")
var req CloneSnapshotRequest
if err := c.ShouldBindJSON(&req); err != nil {
h.logger.Error("Invalid clone snapshot request", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request: " + err.Error()})
return
}
if err := h.snapshotService.CloneSnapshot(c.Request.Context(), snapshotName, req.CloneName); err != nil {
h.logger.Error("Failed to clone snapshot", "error", err, "snapshot", snapshotName, "clone", req.CloneName)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to clone snapshot: " + err.Error()})
return
}
c.JSON(http.StatusCreated, gin.H{"message": "snapshot cloned successfully", "clone_name": req.CloneName})
}
// ListSnapshotSchedules lists all snapshot schedules
func (h *Handler) ListSnapshotSchedules(c *gin.Context) {
schedules, err := h.snapshotScheduleService.ListSchedules(c.Request.Context())
if err != nil {
h.logger.Error("Failed to list snapshot schedules", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to list snapshot schedules: " + err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"schedules": schedules})
}
// GetSnapshotSchedule retrieves a snapshot schedule by ID
func (h *Handler) GetSnapshotSchedule(c *gin.Context) {
id := c.Param("id")
schedule, err := h.snapshotScheduleService.GetSchedule(c.Request.Context(), id)
if err != nil {
if err.Error() == "schedule not found" {
c.JSON(http.StatusNotFound, gin.H{"error": "schedule not found"})
return
}
h.logger.Error("Failed to get snapshot schedule", "error", err, "id", id)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to get snapshot schedule: " + err.Error()})
return
}
c.JSON(http.StatusOK, schedule)
}
// CreateSnapshotScheduleRequest represents a request to create a snapshot schedule
type CreateSnapshotScheduleRequest struct {
Name string `json:"name" binding:"required"`
Dataset string `json:"dataset" binding:"required"`
SnapshotNameTemplate string `json:"snapshot_name_template" binding:"required"`
ScheduleType string `json:"schedule_type" binding:"required"`
ScheduleConfig map[string]interface{} `json:"schedule_config" binding:"required"`
Recursive bool `json:"recursive"`
RetentionCount *int `json:"retention_count"`
RetentionDays *int `json:"retention_days"`
}
// CreateSnapshotSchedule creates a new snapshot schedule
func (h *Handler) CreateSnapshotSchedule(c *gin.Context) {
var req CreateSnapshotScheduleRequest
if err := c.ShouldBindJSON(&req); err != nil {
h.logger.Error("Invalid create snapshot schedule request", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request: " + err.Error()})
return
}
userID, _ := c.Get("user_id")
userIDStr := ""
if userID != nil {
userIDStr = userID.(string)
}
schedule, err := h.snapshotScheduleService.CreateSchedule(c.Request.Context(), &CreateScheduleRequest{
Name: req.Name,
Dataset: req.Dataset,
SnapshotNameTemplate: req.SnapshotNameTemplate,
ScheduleType: req.ScheduleType,
ScheduleConfig: req.ScheduleConfig,
Recursive: req.Recursive,
RetentionCount: req.RetentionCount,
RetentionDays: req.RetentionDays,
}, userIDStr)
if err != nil {
h.logger.Error("Failed to create snapshot schedule", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to create snapshot schedule: " + err.Error()})
return
}
c.JSON(http.StatusCreated, schedule)
}
// UpdateSnapshotSchedule updates an existing snapshot schedule
func (h *Handler) UpdateSnapshotSchedule(c *gin.Context) {
id := c.Param("id")
var req CreateSnapshotScheduleRequest
if err := c.ShouldBindJSON(&req); err != nil {
h.logger.Error("Invalid update snapshot schedule request", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request: " + err.Error()})
return
}
schedule, err := h.snapshotScheduleService.UpdateSchedule(c.Request.Context(), id, &CreateScheduleRequest{
Name: req.Name,
Dataset: req.Dataset,
SnapshotNameTemplate: req.SnapshotNameTemplate,
ScheduleType: req.ScheduleType,
ScheduleConfig: req.ScheduleConfig,
Recursive: req.Recursive,
RetentionCount: req.RetentionCount,
RetentionDays: req.RetentionDays,
})
if err != nil {
if err.Error() == "schedule not found" {
c.JSON(http.StatusNotFound, gin.H{"error": "schedule not found"})
return
}
h.logger.Error("Failed to update snapshot schedule", "error", err, "id", id)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to update snapshot schedule: " + err.Error()})
return
}
c.JSON(http.StatusOK, schedule)
}
// DeleteSnapshotSchedule deletes a snapshot schedule
func (h *Handler) DeleteSnapshotSchedule(c *gin.Context) {
id := c.Param("id")
if err := h.snapshotScheduleService.DeleteSchedule(c.Request.Context(), id); err != nil {
if err.Error() == "schedule not found" {
c.JSON(http.StatusNotFound, gin.H{"error": "schedule not found"})
return
}
h.logger.Error("Failed to delete snapshot schedule", "error", err, "id", id)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to delete snapshot schedule: " + err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"message": "snapshot schedule deleted successfully"})
}
// ToggleSnapshotSchedule enables or disables a snapshot schedule
func (h *Handler) ToggleSnapshotSchedule(c *gin.Context) {
id := c.Param("id")
var req struct {
Enabled bool `json:"enabled" binding:"required"`
}
if err := c.ShouldBindJSON(&req); err != nil {
h.logger.Error("Invalid toggle snapshot schedule request", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request: " + err.Error()})
return
}
if err := h.snapshotScheduleService.ToggleSchedule(c.Request.Context(), id, req.Enabled); err != nil {
if err.Error() == "schedule not found" {
c.JSON(http.StatusNotFound, gin.H{"error": "schedule not found"})
return
}
h.logger.Error("Failed to toggle snapshot schedule", "error", err, "id", id)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to toggle snapshot schedule: " + err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"message": "snapshot schedule toggled successfully"})
}
// ListReplicationTasks lists all replication tasks
func (h *Handler) ListReplicationTasks(c *gin.Context) {
direction := c.Query("direction") // Optional filter: "outbound" or "inbound"
tasks, err := h.replicationService.ListReplicationTasks(c.Request.Context(), direction)
if err != nil {
h.logger.Error("Failed to list replication tasks", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to list replication tasks: " + err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"tasks": tasks})
}
// GetReplicationTask retrieves a replication task by ID
func (h *Handler) GetReplicationTask(c *gin.Context) {
id := c.Param("id")
task, err := h.replicationService.GetReplicationTask(c.Request.Context(), id)
if err != nil {
if err.Error() == "replication task not found" {
c.JSON(http.StatusNotFound, gin.H{"error": "replication task not found"})
return
}
h.logger.Error("Failed to get replication task", "error", err, "id", id)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to get replication task: " + err.Error()})
return
}
c.JSON(http.StatusOK, task)
}
// CreateReplicationTaskRequest represents a request to create a replication task
type CreateReplicationTaskRequest struct {
Name string `json:"name" binding:"required"`
Direction string `json:"direction" binding:"required"`
SourceDataset *string `json:"source_dataset"`
TargetHost *string `json:"target_host"`
TargetPort *int `json:"target_port"`
TargetUser *string `json:"target_user"`
TargetDataset *string `json:"target_dataset"`
TargetSSHKeyPath *string `json:"target_ssh_key_path"`
SourceHost *string `json:"source_host"`
SourcePort *int `json:"source_port"`
SourceUser *string `json:"source_user"`
LocalDataset *string `json:"local_dataset"`
ScheduleType *string `json:"schedule_type"`
ScheduleConfig map[string]interface{} `json:"schedule_config"`
Compression string `json:"compression"`
Encryption bool `json:"encryption"`
Recursive bool `json:"recursive"`
Incremental bool `json:"incremental"`
AutoSnapshot bool `json:"auto_snapshot"`
Enabled bool `json:"enabled"`
}
// CreateReplicationTask creates a new replication task
func (h *Handler) CreateReplicationTask(c *gin.Context) {
var req CreateReplicationTaskRequest
if err := c.ShouldBindJSON(&req); err != nil {
h.logger.Error("Invalid create replication task request", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request: " + err.Error()})
return
}
userID, _ := c.Get("user_id")
userIDStr := ""
if userID != nil {
userIDStr = userID.(string)
}
task, err := h.replicationService.CreateReplicationTask(c.Request.Context(), &CreateReplicationRequest{
Name: req.Name,
Direction: req.Direction,
SourceDataset: req.SourceDataset,
TargetHost: req.TargetHost,
TargetPort: req.TargetPort,
TargetUser: req.TargetUser,
TargetDataset: req.TargetDataset,
TargetSSHKeyPath: req.TargetSSHKeyPath,
SourceHost: req.SourceHost,
SourcePort: req.SourcePort,
SourceUser: req.SourceUser,
LocalDataset: req.LocalDataset,
ScheduleType: req.ScheduleType,
ScheduleConfig: req.ScheduleConfig,
Compression: req.Compression,
Encryption: req.Encryption,
Recursive: req.Recursive,
Incremental: req.Incremental,
AutoSnapshot: req.AutoSnapshot,
Enabled: req.Enabled,
}, userIDStr)
if err != nil {
h.logger.Error("Failed to create replication task", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to create replication task: " + err.Error()})
return
}
c.JSON(http.StatusCreated, task)
}
// UpdateReplicationTask updates an existing replication task
func (h *Handler) UpdateReplicationTask(c *gin.Context) {
id := c.Param("id")
var req CreateReplicationTaskRequest
if err := c.ShouldBindJSON(&req); err != nil {
h.logger.Error("Invalid update replication task request", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request: " + err.Error()})
return
}
task, err := h.replicationService.UpdateReplicationTask(c.Request.Context(), id, &CreateReplicationRequest{
Name: req.Name,
Direction: req.Direction,
SourceDataset: req.SourceDataset,
TargetHost: req.TargetHost,
TargetPort: req.TargetPort,
TargetUser: req.TargetUser,
TargetDataset: req.TargetDataset,
TargetSSHKeyPath: req.TargetSSHKeyPath,
SourceHost: req.SourceHost,
SourcePort: req.SourcePort,
SourceUser: req.SourceUser,
LocalDataset: req.LocalDataset,
ScheduleType: req.ScheduleType,
ScheduleConfig: req.ScheduleConfig,
Compression: req.Compression,
Encryption: req.Encryption,
Recursive: req.Recursive,
Incremental: req.Incremental,
AutoSnapshot: req.AutoSnapshot,
Enabled: req.Enabled,
})
if err != nil {
if err.Error() == "replication task not found" {
c.JSON(http.StatusNotFound, gin.H{"error": "replication task not found"})
return
}
h.logger.Error("Failed to update replication task", "error", err, "id", id)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to update replication task: " + err.Error()})
return
}
c.JSON(http.StatusOK, task)
}
// DeleteReplicationTask deletes a replication task
func (h *Handler) DeleteReplicationTask(c *gin.Context) {
id := c.Param("id")
if err := h.replicationService.DeleteReplicationTask(c.Request.Context(), id); err != nil {
if err.Error() == "replication task not found" {
c.JSON(http.StatusNotFound, gin.H{"error": "replication task not found"})
return
}
h.logger.Error("Failed to delete replication task", "error", err, "id", id)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to delete replication task: " + err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"message": "replication task deleted successfully"})
}

553
backend/internal/storage/replication.go Normal file
View File

@@ -0,0 +1,553 @@
package storage
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"time"
"github.com/atlasos/calypso/internal/common/database"
"github.com/atlasos/calypso/internal/common/logger"
"github.com/google/uuid"
)
// ReplicationService handles ZFS replication task management
type ReplicationService struct {
db *database.DB
logger *logger.Logger
}
// NewReplicationService creates a new replication service
func NewReplicationService(db *database.DB, log *logger.Logger) *ReplicationService {
return &ReplicationService{
db: db,
logger: log,
}
}
// ReplicationTask represents a ZFS replication task
type ReplicationTask struct {
ID string `json:"id"`
Name string `json:"name"`
Direction string `json:"direction"` // "outbound" or "inbound"
SourceDataset *string `json:"source_dataset,omitempty"`
TargetHost *string `json:"target_host,omitempty"`
TargetPort *int `json:"target_port,omitempty"`
TargetUser *string `json:"target_user,omitempty"`
TargetDataset *string `json:"target_dataset,omitempty"`
TargetSSHKeyPath *string `json:"target_ssh_key_path,omitempty"`
SourceHost *string `json:"source_host,omitempty"`
SourcePort *int `json:"source_port,omitempty"`
SourceUser *string `json:"source_user,omitempty"`
LocalDataset *string `json:"local_dataset,omitempty"`
ScheduleType *string `json:"schedule_type,omitempty"`
ScheduleConfig map[string]interface{} `json:"schedule_config,omitempty"`
Compression string `json:"compression"`
Encryption bool `json:"encryption"`
Recursive bool `json:"recursive"`
Incremental bool `json:"incremental"`
AutoSnapshot bool `json:"auto_snapshot"`
Enabled bool `json:"enabled"`
Status string `json:"status"`
LastRunAt *time.Time `json:"last_run_at,omitempty"`
LastRunStatus *string `json:"last_run_status,omitempty"`
LastRunError *string `json:"last_run_error,omitempty"`
NextRunAt *time.Time `json:"next_run_at,omitempty"`
LastSnapshotSent *string `json:"last_snapshot_sent,omitempty"`
LastSnapshotReceived *string `json:"last_snapshot_received,omitempty"`
TotalRuns int `json:"total_runs"`
SuccessfulRuns int `json:"successful_runs"`
FailedRuns int `json:"failed_runs"`
BytesSent int64 `json:"bytes_sent"`
BytesReceived int64 `json:"bytes_received"`
CreatedBy string `json:"created_by,omitempty"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
// CreateReplicationRequest represents a request to create a replication task
type CreateReplicationRequest struct {
Name string `json:"name" binding:"required"`
Direction string `json:"direction" binding:"required"` // "outbound" or "inbound"
SourceDataset *string `json:"source_dataset"`
TargetHost *string `json:"target_host"`
TargetPort *int `json:"target_port"`
TargetUser *string `json:"target_user"`
TargetDataset *string `json:"target_dataset"`
TargetSSHKeyPath *string `json:"target_ssh_key_path"`
SourceHost *string `json:"source_host"`
SourcePort *int `json:"source_port"`
SourceUser *string `json:"source_user"`
LocalDataset *string `json:"local_dataset"`
ScheduleType *string `json:"schedule_type"`
ScheduleConfig map[string]interface{} `json:"schedule_config"`
Compression string `json:"compression"`
Encryption bool `json:"encryption"`
Recursive bool `json:"recursive"`
Incremental bool `json:"incremental"`
AutoSnapshot bool `json:"auto_snapshot"`
Enabled bool `json:"enabled"`
}
// ListReplicationTasks lists all replication tasks, optionally filtered by direction
func (s *ReplicationService) ListReplicationTasks(ctx context.Context, directionFilter string) ([]*ReplicationTask, error) {
var query string
var args []interface{}
if directionFilter != "" {
query = `
SELECT id, name, direction, source_dataset, target_host, target_port, target_user, target_dataset,
target_ssh_key_path, source_host, source_port, source_user, local_dataset,
schedule_type, schedule_config, compression, encryption, recursive, incremental,
auto_snapshot, enabled, status, last_run_at, last_run_status, last_run_error,
next_run_at, last_snapshot_sent, last_snapshot_received,
total_runs, successful_runs, failed_runs, bytes_sent, bytes_received,
created_by, created_at, updated_at
FROM replication_tasks
WHERE direction = $1
ORDER BY created_at DESC
`
args = []interface{}{directionFilter}
} else {
query = `
SELECT id, name, direction, source_dataset, target_host, target_port, target_user, target_dataset,
target_ssh_key_path, source_host, source_port, source_user, local_dataset,
schedule_type, schedule_config, compression, encryption, recursive, incremental,
auto_snapshot, enabled, status, last_run_at, last_run_status, last_run_error,
next_run_at, last_snapshot_sent, last_snapshot_received,
total_runs, successful_runs, failed_runs, bytes_sent, bytes_received,
created_by, created_at, updated_at
FROM replication_tasks
ORDER BY direction, created_at DESC
`
args = []interface{}{}
}
rows, err := s.db.QueryContext(ctx, query, args...)
if err != nil {
return nil, fmt.Errorf("failed to query replication tasks: %w", err)
}
defer rows.Close()
var tasks []*ReplicationTask
for rows.Next() {
task, err := s.scanReplicationTask(rows)
if err != nil {
s.logger.Error("Failed to scan replication task", "error", err)
continue
}
tasks = append(tasks, task)
}
return tasks, rows.Err()
}
// GetReplicationTask retrieves a replication task by ID
func (s *ReplicationService) GetReplicationTask(ctx context.Context, id string) (*ReplicationTask, error) {
query := `
SELECT id, name, direction, source_dataset, target_host, target_port, target_user, target_dataset,
target_ssh_key_path, source_host, source_port, source_user, local_dataset,
schedule_type, schedule_config, compression, encryption, recursive, incremental,
auto_snapshot, enabled, status, last_run_at, last_run_status, last_run_error,
next_run_at, last_snapshot_sent, last_snapshot_received,
total_runs, successful_runs, failed_runs, bytes_sent, bytes_received,
created_by, created_at, updated_at
FROM replication_tasks
WHERE id = $1
`
row := s.db.QueryRowContext(ctx, query, id)
task, err := s.scanReplicationTaskRow(row)
if err != nil {
if err == sql.ErrNoRows {
return nil, fmt.Errorf("replication task not found")
}
return nil, fmt.Errorf("failed to get replication task: %w", err)
}
return task, nil
}
// CreateReplicationTask creates a new replication task
func (s *ReplicationService) CreateReplicationTask(ctx context.Context, req *CreateReplicationRequest, createdBy string) (*ReplicationTask, error) {
id := uuid.New().String()
// Validate direction-specific fields
if req.Direction == "outbound" {
if req.SourceDataset == nil || req.TargetHost == nil || req.TargetDataset == nil {
return nil, fmt.Errorf("outbound replication requires source_dataset, target_host, and target_dataset")
}
} else if req.Direction == "inbound" {
if req.SourceHost == nil || req.SourceDataset == nil || req.LocalDataset == nil {
return nil, fmt.Errorf("inbound replication requires source_host, source_dataset, and local_dataset")
}
} else {
return nil, fmt.Errorf("invalid direction: must be 'outbound' or 'inbound'")
}
// Set defaults
if req.Compression == "" {
req.Compression = "lz4"
}
if req.TargetPort == nil {
defaultPort := 22
req.TargetPort = &defaultPort
}
if req.SourcePort == nil {
defaultPort := 22
req.SourcePort = &defaultPort
}
if req.TargetUser == nil {
defaultUser := "root"
req.TargetUser = &defaultUser
}
if req.SourceUser == nil {
defaultUser := "root"
req.SourceUser = &defaultUser
}
// Marshal schedule config to JSON
var scheduleConfigJSON sql.NullString
if req.ScheduleConfig != nil {
configJSON, err := json.Marshal(req.ScheduleConfig)
if err != nil {
return nil, fmt.Errorf("failed to marshal schedule config: %w", err)
}
scheduleConfigJSON = sql.NullString{String: string(configJSON), Valid: true}
}
query := `
INSERT INTO replication_tasks (
id, name, direction, source_dataset, target_host, target_port, target_user, target_dataset,
target_ssh_key_path, source_host, source_port, source_user, local_dataset,
schedule_type, schedule_config, compression, encryption, recursive, incremental,
auto_snapshot, enabled, status, created_by, created_at, updated_at
) VALUES (
$1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, NOW(), NOW()
)
RETURNING id, name, direction, source_dataset, target_host, target_port, target_user, target_dataset,
target_ssh_key_path, source_host, source_port, source_user, local_dataset,
schedule_type, schedule_config, compression, encryption, recursive, incremental,
auto_snapshot, enabled, status, last_run_at, last_run_status, last_run_error,
next_run_at, last_snapshot_sent, last_snapshot_received,
total_runs, successful_runs, failed_runs, bytes_sent, bytes_received,
created_by, created_at, updated_at
`
var scheduleTypeStr sql.NullString
if req.ScheduleType != nil {
scheduleTypeStr = sql.NullString{String: *req.ScheduleType, Valid: true}
}
row := s.db.QueryRowContext(ctx, query,
id, req.Name, req.Direction,
req.SourceDataset, req.TargetHost, req.TargetPort, req.TargetUser, req.TargetDataset,
req.TargetSSHKeyPath, req.SourceHost, req.SourcePort, req.SourceUser, req.LocalDataset,
scheduleTypeStr, scheduleConfigJSON, req.Compression, req.Encryption, req.Recursive, req.Incremental,
req.AutoSnapshot, req.Enabled, "idle", createdBy,
)
task, err := s.scanReplicationTaskRow(row)
if err != nil {
return nil, fmt.Errorf("failed to create replication task: %w", err)
}
s.logger.Info("Replication task created", "id", id, "name", req.Name, "direction", req.Direction)
return task, nil
}
// UpdateReplicationTask updates an existing replication task
func (s *ReplicationService) UpdateReplicationTask(ctx context.Context, id string, req *CreateReplicationRequest) (*ReplicationTask, error) {
// Validate direction-specific fields
if req.Direction == "outbound" {
if req.SourceDataset == nil || req.TargetHost == nil || req.TargetDataset == nil {
return nil, fmt.Errorf("outbound replication requires source_dataset, target_host, and target_dataset")
}
} else if req.Direction == "inbound" {
if req.SourceHost == nil || req.SourceDataset == nil || req.LocalDataset == nil {
return nil, fmt.Errorf("inbound replication requires source_host, source_dataset, and local_dataset")
}
} else {
return nil, fmt.Errorf("invalid direction: must be 'outbound' or 'inbound'")
}
// Set defaults
if req.Compression == "" {
req.Compression = "lz4"
}
if req.TargetPort == nil {
defaultPort := 22
req.TargetPort = &defaultPort
}
if req.SourcePort == nil {
defaultPort := 22
req.SourcePort = &defaultPort
}
if req.TargetUser == nil {
defaultUser := "root"
req.TargetUser = &defaultUser
}
if req.SourceUser == nil {
defaultUser := "root"
req.SourceUser = &defaultUser
}
// Marshal schedule config to JSON
var scheduleConfigJSON sql.NullString
if req.ScheduleConfig != nil {
configJSON, err := json.Marshal(req.ScheduleConfig)
if err != nil {
return nil, fmt.Errorf("failed to marshal schedule config: %w", err)
}
scheduleConfigJSON = sql.NullString{String: string(configJSON), Valid: true}
}
var scheduleTypeStr sql.NullString
if req.ScheduleType != nil {
scheduleTypeStr = sql.NullString{String: *req.ScheduleType, Valid: true}
}
query := `
UPDATE replication_tasks
SET name = $1, direction = $2, source_dataset = $3, target_host = $4, target_port = $5,
target_user = $6, target_dataset = $7, target_ssh_key_path = $8, source_host = $9,
source_port = $10, source_user = $11, local_dataset = $12, schedule_type = $13,
schedule_config = $14, compression = $15, encryption = $16, recursive = $17,
incremental = $18, auto_snapshot = $19, enabled = $20, updated_at = NOW()
WHERE id = $21
RETURNING id, name, direction, source_dataset, target_host, target_port, target_user, target_dataset,
target_ssh_key_path, source_host, source_port, source_user, local_dataset,
schedule_type, schedule_config, compression, encryption, recursive, incremental,
auto_snapshot, enabled, status, last_run_at, last_run_status, last_run_error,
next_run_at, last_snapshot_sent, last_snapshot_received,
total_runs, successful_runs, failed_runs, bytes_sent, bytes_received,
created_by, created_at, updated_at
`
row := s.db.QueryRowContext(ctx, query,
req.Name, req.Direction,
req.SourceDataset, req.TargetHost, req.TargetPort, req.TargetUser, req.TargetDataset,
req.TargetSSHKeyPath, req.SourceHost, req.SourcePort, req.SourceUser, req.LocalDataset,
scheduleTypeStr, scheduleConfigJSON, req.Compression, req.Encryption, req.Recursive, req.Incremental,
req.AutoSnapshot, req.Enabled, id,
)
task, err := s.scanReplicationTaskRow(row)
if err != nil {
if err == sql.ErrNoRows {
return nil, fmt.Errorf("replication task not found")
}
return nil, fmt.Errorf("failed to update replication task: %w", err)
}
s.logger.Info("Replication task updated", "id", id)
return task, nil
}
// DeleteReplicationTask deletes a replication task
func (s *ReplicationService) DeleteReplicationTask(ctx context.Context, id string) error {
query := `DELETE FROM replication_tasks WHERE id = $1`
result, err := s.db.ExecContext(ctx, query, id)
if err != nil {
return fmt.Errorf("failed to delete replication task: %w", err)
}
rowsAffected, err := result.RowsAffected()
if err != nil {
return fmt.Errorf("failed to get rows affected: %w", err)
}
if rowsAffected == 0 {
return fmt.Errorf("replication task not found")
}
s.logger.Info("Replication task deleted", "id", id)
return nil
}
// scanReplicationTaskRow scans a single replication task row
func (s *ReplicationService) scanReplicationTaskRow(row *sql.Row) (*ReplicationTask, error) {
var task ReplicationTask
var sourceDataset, targetHost, targetUser, targetDataset, targetSSHKeyPath sql.NullString
var sourceHost, sourceUser, localDataset sql.NullString
var targetPort, sourcePort sql.NullInt64
var scheduleType, scheduleConfigJSON sql.NullString
var lastRunAt, nextRunAt sql.NullTime
var lastRunStatus, lastRunError, lastSnapshotSent, lastSnapshotReceived sql.NullString
var createdBy sql.NullString
err := row.Scan(
&task.ID, &task.Name, &task.Direction,
&sourceDataset, &targetHost, &targetPort, &targetUser, &targetDataset,
&targetSSHKeyPath, &sourceHost, &sourcePort, &sourceUser, &localDataset,
&scheduleType, &scheduleConfigJSON, &task.Compression, &task.Encryption,
&task.Recursive, &task.Incremental, &task.AutoSnapshot, &task.Enabled, &task.Status,
&lastRunAt, &lastRunStatus, &lastRunError, &nextRunAt,
&lastSnapshotSent, &lastSnapshotReceived,
&task.TotalRuns, &task.SuccessfulRuns, &task.FailedRuns,
&task.BytesSent, &task.BytesReceived,
&createdBy, &task.CreatedAt, &task.UpdatedAt,
)
if err != nil {
return nil, err
}
// Handle nullable fields
if sourceDataset.Valid {
task.SourceDataset = &sourceDataset.String
}
if targetHost.Valid {
task.TargetHost = &targetHost.String
}
if targetPort.Valid {
port := int(targetPort.Int64)
task.TargetPort = &port
}
if targetUser.Valid {
task.TargetUser = &targetUser.String
}
if targetDataset.Valid {
task.TargetDataset = &targetDataset.String
}
if targetSSHKeyPath.Valid {
task.TargetSSHKeyPath = &targetSSHKeyPath.String
}
if sourceHost.Valid {
task.SourceHost = &sourceHost.String
}
if sourcePort.Valid {
port := int(sourcePort.Int64)
task.SourcePort = &port
}
if sourceUser.Valid {
task.SourceUser = &sourceUser.String
}
if localDataset.Valid {
task.LocalDataset = &localDataset.String
}
if scheduleType.Valid {
task.ScheduleType = &scheduleType.String
}
if scheduleConfigJSON.Valid {
if err := json.Unmarshal([]byte(scheduleConfigJSON.String), &task.ScheduleConfig); err != nil {
return nil, fmt.Errorf("failed to unmarshal schedule config: %w", err)
}
}
if lastRunAt.Valid {
task.LastRunAt = &lastRunAt.Time
}
if lastRunStatus.Valid {
task.LastRunStatus = &lastRunStatus.String
}
if lastRunError.Valid {
task.LastRunError = &lastRunError.String
}
if nextRunAt.Valid {
task.NextRunAt = &nextRunAt.Time
}
if lastSnapshotSent.Valid {
task.LastSnapshotSent = &lastSnapshotSent.String
}
if lastSnapshotReceived.Valid {
task.LastSnapshotReceived = &lastSnapshotReceived.String
}
if createdBy.Valid {
task.CreatedBy = createdBy.String
}
return &task, nil
}
// scanReplicationTask scans a replication task from rows
func (s *ReplicationService) scanReplicationTask(rows *sql.Rows) (*ReplicationTask, error) {
var task ReplicationTask
var sourceDataset, targetHost, targetUser, targetDataset, targetSSHKeyPath sql.NullString
var sourceHost, sourceUser, localDataset sql.NullString
var targetPort, sourcePort sql.NullInt64
var scheduleType, scheduleConfigJSON sql.NullString
var lastRunAt, nextRunAt sql.NullTime
var lastRunStatus, lastRunError, lastSnapshotSent, lastSnapshotReceived sql.NullString
var createdBy sql.NullString
err := rows.Scan(
&task.ID, &task.Name, &task.Direction,
&sourceDataset, &targetHost, &targetPort, &targetUser, &targetDataset,
&targetSSHKeyPath, &sourceHost, &sourcePort, &sourceUser, &localDataset,
&scheduleType, &scheduleConfigJSON, &task.Compression, &task.Encryption,
&task.Recursive, &task.Incremental, &task.AutoSnapshot, &task.Enabled, &task.Status,
&lastRunAt, &lastRunStatus, &lastRunError, &nextRunAt,
&lastSnapshotSent, &lastSnapshotReceived,
&task.TotalRuns, &task.SuccessfulRuns, &task.FailedRuns,
&task.BytesSent, &task.BytesReceived,
&createdBy, &task.CreatedAt, &task.UpdatedAt,
)
if err != nil {
return nil, err
}
// Handle nullable fields (same as scanReplicationTaskRow)
if sourceDataset.Valid {
task.SourceDataset = &sourceDataset.String
}
if targetHost.Valid {
task.TargetHost = &targetHost.String
}
if targetPort.Valid {
port := int(targetPort.Int64)
task.TargetPort = &port
}
if targetUser.Valid {
task.TargetUser = &targetUser.String
}
if targetDataset.Valid {
task.TargetDataset = &targetDataset.String
}
if targetSSHKeyPath.Valid {
task.TargetSSHKeyPath = &targetSSHKeyPath.String
}
if sourceHost.Valid {
task.SourceHost = &sourceHost.String
}
if sourcePort.Valid {
port := int(sourcePort.Int64)
task.SourcePort = &port
}
if sourceUser.Valid {
task.SourceUser = &sourceUser.String
}
if localDataset.Valid {
task.LocalDataset = &localDataset.String
}
if scheduleType.Valid {
task.ScheduleType = &scheduleType.String
}
if scheduleConfigJSON.Valid {
if err := json.Unmarshal([]byte(scheduleConfigJSON.String), &task.ScheduleConfig); err != nil {
return nil, fmt.Errorf("failed to unmarshal schedule config: %w", err)
}
}
if lastRunAt.Valid {
task.LastRunAt = &lastRunAt.Time
}
if lastRunStatus.Valid {
task.LastRunStatus = &lastRunStatus.String
}
if lastRunError.Valid {
task.LastRunError = &lastRunError.String
}
if nextRunAt.Valid {
task.NextRunAt = &nextRunAt.Time
}
if lastSnapshotSent.Valid {
task.LastSnapshotSent = &lastSnapshotSent.String
}
if lastSnapshotReceived.Valid {
task.LastSnapshotReceived = &lastSnapshotReceived.String
}
if createdBy.Valid {
task.CreatedBy = createdBy.String
}
return &task, nil
}

259
backend/internal/storage/snapshot.go Normal file
View File

@@ -0,0 +1,259 @@
package storage
import (
"context"
"fmt"
"strings"
"time"
"github.com/atlasos/calypso/internal/common/database"
"github.com/atlasos/calypso/internal/common/logger"
)
// SnapshotService handles ZFS snapshot operations
type SnapshotService struct {
db *database.DB
logger *logger.Logger
}
// NewSnapshotService creates a new snapshot service
func NewSnapshotService(db *database.DB, log *logger.Logger) *SnapshotService {
return &SnapshotService{
db: db,
logger: log,
}
}
// Snapshot represents a ZFS snapshot
type Snapshot struct {
ID string `json:"id"`
Name string `json:"name"` // Full snapshot name (e.g., "pool/dataset@snapshot-name")
Dataset string `json:"dataset"` // Dataset name (e.g., "pool/dataset")
SnapshotName string `json:"snapshot_name"` // Snapshot name only (e.g., "snapshot-name")
Created time.Time `json:"created"`
Referenced int64 `json:"referenced"` // Size in bytes
Used int64 `json:"used"` // Used space in bytes
IsLatest bool `json:"is_latest"` // Whether this is the latest snapshot for the dataset
}
// ListSnapshots lists all snapshots, optionally filtered by dataset
func (s *SnapshotService) ListSnapshots(ctx context.Context, datasetFilter string) ([]*Snapshot, error) {
// Build zfs list command
args := []string{"list", "-t", "snapshot", "-H", "-o", "name,creation,referenced,used"}
if datasetFilter != "" {
// List snapshots for specific dataset
args = append(args, datasetFilter)
} else {
// List all snapshots
args = append(args, "-r")
}
cmd := zfsCommand(ctx, args...)
output, err := cmd.Output()
if err != nil {
return nil, fmt.Errorf("failed to list snapshots: %w", err)
}
lines := strings.Split(strings.TrimSpace(string(output)), "\n")
var snapshots []*Snapshot
// Track latest snapshot per dataset
latestSnapshots := make(map[string]*Snapshot)
for _, line := range lines {
if line == "" {
continue
}
fields := strings.Fields(line)
if len(fields) < 4 {
continue
}
fullName := fields[0]
creationStr := fields[1]
referencedStr := fields[2]
usedStr := fields[3]
// Parse snapshot name (format: pool/dataset@snapshot-name)
parts := strings.Split(fullName, "@")
if len(parts) != 2 {
continue
}
dataset := parts[0]
snapshotName := parts[1]
// Parse creation time (Unix timestamp)
var creationTime time.Time
if timestamp, err := parseZFSUnixTimestamp(creationStr); err == nil {
creationTime = timestamp
} else {
// Fallback to current time if parsing fails
creationTime = time.Now()
}
// Parse sizes
referenced := parseSnapshotSize(referencedStr)
used := parseSnapshotSize(usedStr)
snapshot := &Snapshot{
ID: fullName,
Name: fullName,
Dataset: dataset,
SnapshotName: snapshotName,
Created: creationTime,
Referenced: referenced,
Used: used,
IsLatest: false,
}
snapshots = append(snapshots, snapshot)
// Track latest snapshot per dataset
if latest, exists := latestSnapshots[dataset]; !exists || snapshot.Created.After(latest.Created) {
latestSnapshots[dataset] = snapshot
}
}
// Mark latest snapshots
for _, snapshot := range snapshots {
if latest, exists := latestSnapshots[snapshot.Dataset]; exists && latest.ID == snapshot.ID {
snapshot.IsLatest = true
}
}
return snapshots, nil
}
// CreateSnapshot creates a new snapshot
func (s *SnapshotService) CreateSnapshot(ctx context.Context, dataset, snapshotName string, recursive bool) error {
// Validate dataset exists
cmd := zfsCommand(ctx, "list", "-H", "-o", "name", dataset)
if err := cmd.Run(); err != nil {
return fmt.Errorf("dataset %s does not exist: %w", dataset, err)
}
// Build snapshot name
fullSnapshotName := fmt.Sprintf("%s@%s", dataset, snapshotName)
// Build command
args := []string{"snapshot"}
if recursive {
args = append(args, "-r")
}
args = append(args, fullSnapshotName)
cmd = zfsCommand(ctx, args...)
output, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("failed to create snapshot: %s: %w", string(output), err)
}
s.logger.Info("Snapshot created", "snapshot", fullSnapshotName, "dataset", dataset, "recursive", recursive)
return nil
}
// DeleteSnapshot deletes a snapshot
func (s *SnapshotService) DeleteSnapshot(ctx context.Context, snapshotName string, recursive bool) error {
// Build command
args := []string{"destroy"}
if recursive {
args = append(args, "-r")
}
args = append(args, snapshotName)
cmd := zfsCommand(ctx, args...)
output, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("failed to delete snapshot: %s: %w", string(output), err)
}
s.logger.Info("Snapshot deleted", "snapshot", snapshotName, "recursive", recursive)
return nil
}
// RollbackSnapshot rolls back a dataset to a snapshot
func (s *SnapshotService) RollbackSnapshot(ctx context.Context, snapshotName string, force bool) error {
// Build command
args := []string{"rollback"}
if force {
args = append(args, "-r", "-f")
} else {
args = append(args, "-r")
}
args = append(args, snapshotName)
cmd := zfsCommand(ctx, args...)
output, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("failed to rollback snapshot: %s: %w", string(output), err)
}
s.logger.Info("Snapshot rollback completed", "snapshot", snapshotName, "force", force)
return nil
}
// CloneSnapshot clones a snapshot to a new dataset
func (s *SnapshotService) CloneSnapshot(ctx context.Context, snapshotName, cloneName string) error {
// Build command
args := []string{"clone", snapshotName, cloneName}
cmd := zfsCommand(ctx, args...)
output, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("failed to clone snapshot: %s: %w", string(output), err)
}
s.logger.Info("Snapshot cloned", "snapshot", snapshotName, "clone", cloneName)
return nil
}
// parseZFSUnixTimestamp parses ZFS Unix timestamp string
func parseZFSUnixTimestamp(timestampStr string) (time.Time, error) {
// ZFS returns Unix timestamp as string
timestamp, err := time.Parse("20060102150405", timestampStr)
if err != nil {
// Try parsing as Unix timestamp integer
var unixTime int64
if _, err := fmt.Sscanf(timestampStr, "%d", &unixTime); err == nil {
return time.Unix(unixTime, 0), nil
}
return time.Time{}, err
}
return timestamp, nil
}
// parseZFSSize parses ZFS size string (e.g., "1.2M", "4.5G", "850K")
// Note: This function is also defined in zfs.go, but we need it here for snapshot parsing
func parseSnapshotSize(sizeStr string) int64 {
if sizeStr == "-" || sizeStr == "" {
return 0
}
// Remove any whitespace
sizeStr = strings.TrimSpace(sizeStr)
// Parse size with unit
var size float64
var unit string
if _, err := fmt.Sscanf(sizeStr, "%f%s", &size, &unit); err != nil {
return 0
}
// Convert to bytes
unit = strings.ToUpper(unit)
switch unit {
case "K", "KB":
return int64(size * 1024)
case "M", "MB":
return int64(size * 1024 * 1024)
case "G", "GB":
return int64(size * 1024 * 1024 * 1024)
case "T", "TB":
return int64(size * 1024 * 1024 * 1024 * 1024)
default:
// Assume bytes if no unit
return int64(size)
}
}

606
backend/internal/storage/snapshot_schedule.go Normal file
View File

@@ -0,0 +1,606 @@
package storage
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"sort"
"strings"
"time"
"github.com/atlasos/calypso/internal/common/database"
"github.com/atlasos/calypso/internal/common/logger"
"github.com/google/uuid"
)
// SnapshotScheduleService handles snapshot schedule management
type SnapshotScheduleService struct {
db *database.DB
logger *logger.Logger
snapshotService *SnapshotService
}
// NewSnapshotScheduleService creates a new snapshot schedule service
func NewSnapshotScheduleService(db *database.DB, log *logger.Logger, snapshotService *SnapshotService) *SnapshotScheduleService {
return &SnapshotScheduleService{
db: db,
logger: log,
snapshotService: snapshotService,
}
}
// SnapshotSchedule represents a scheduled snapshot task
type SnapshotSchedule struct {
ID string `json:"id"`
Name string `json:"name"`
Dataset string `json:"dataset"`
SnapshotNameTemplate string `json:"snapshot_name_template"`
ScheduleType string `json:"schedule_type"` // hourly, daily, weekly, monthly, cron
ScheduleConfig map[string]interface{} `json:"schedule_config"`
Recursive bool `json:"recursive"`
Enabled bool `json:"enabled"`
RetentionCount *int `json:"retention_count,omitempty"`
RetentionDays *int `json:"retention_days,omitempty"`
LastRunAt *time.Time `json:"last_run_at,omitempty"`
NextRunAt *time.Time `json:"next_run_at,omitempty"`
CreatedBy string `json:"created_by,omitempty"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
// CreateScheduleRequest represents a request to create a snapshot schedule
type CreateScheduleRequest struct {
Name string `json:"name" binding:"required"`
Dataset string `json:"dataset" binding:"required"`
SnapshotNameTemplate string `json:"snapshot_name_template" binding:"required"`
ScheduleType string `json:"schedule_type" binding:"required"`
ScheduleConfig map[string]interface{} `json:"schedule_config" binding:"required"`
Recursive bool `json:"recursive"`
RetentionCount *int `json:"retention_count"`
RetentionDays *int `json:"retention_days"`
}
// ListSchedules lists all snapshot schedules
func (s *SnapshotScheduleService) ListSchedules(ctx context.Context) ([]*SnapshotSchedule, error) {
query := `
SELECT id, name, dataset, snapshot_name_template, schedule_type, schedule_config,
recursive, enabled, retention_count, retention_days,
last_run_at, next_run_at, created_by, created_at, updated_at
FROM snapshot_schedules
ORDER BY created_at DESC
`
rows, err := s.db.QueryContext(ctx, query)
if err != nil {
return nil, fmt.Errorf("failed to query schedules: %w", err)
}
defer rows.Close()
var schedules []*SnapshotSchedule
for rows.Next() {
schedule, err := s.scanSchedule(rows)
if err != nil {
s.logger.Error("Failed to scan schedule", "error", err)
continue
}
schedules = append(schedules, schedule)
}
return schedules, rows.Err()
}
// GetSchedule retrieves a schedule by ID
func (s *SnapshotScheduleService) GetSchedule(ctx context.Context, id string) (*SnapshotSchedule, error) {
query := `
SELECT id, name, dataset, snapshot_name_template, schedule_type, schedule_config,
recursive, enabled, retention_count, retention_days,
last_run_at, next_run_at, created_by, created_at, updated_at
FROM snapshot_schedules
WHERE id = $1
`
row := s.db.QueryRowContext(ctx, query, id)
schedule, err := s.scanScheduleRow(row)
if err != nil {
if err == sql.ErrNoRows {
return nil, fmt.Errorf("schedule not found")
}
return nil, fmt.Errorf("failed to get schedule: %w", err)
}
return schedule, nil
}
// CreateSchedule creates a new snapshot schedule
func (s *SnapshotScheduleService) CreateSchedule(ctx context.Context, req *CreateScheduleRequest, createdBy string) (*SnapshotSchedule, error) {
id := uuid.New().String()
// Calculate next run time
nextRunAt, err := s.calculateNextRun(req.ScheduleType, req.ScheduleConfig)
if err != nil {
return nil, fmt.Errorf("failed to calculate next run time: %w", err)
}
// Marshal schedule config to JSON
configJSON, err := json.Marshal(req.ScheduleConfig)
if err != nil {
return nil, fmt.Errorf("failed to marshal schedule config: %w", err)
}
query := `
INSERT INTO snapshot_schedules (
id, name, dataset, snapshot_name_template, schedule_type, schedule_config,
recursive, enabled, retention_count, retention_days,
next_run_at, created_by, created_at, updated_at
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, NOW(), NOW())
RETURNING id, name, dataset, snapshot_name_template, schedule_type, schedule_config,
recursive, enabled, retention_count, retention_days,
last_run_at, next_run_at, created_by, created_at, updated_at
`
row := s.db.QueryRowContext(ctx, query,
id, req.Name, req.Dataset, req.SnapshotNameTemplate, req.ScheduleType,
string(configJSON), req.Recursive, true, req.RetentionCount, req.RetentionDays,
nextRunAt, createdBy,
)
schedule, err := s.scanScheduleRow(row)
if err != nil {
return nil, fmt.Errorf("failed to create schedule: %w", err)
}
s.logger.Info("Snapshot schedule created", "id", id, "name", req.Name, "dataset", req.Dataset)
return schedule, nil
}
// UpdateSchedule updates an existing schedule
func (s *SnapshotScheduleService) UpdateSchedule(ctx context.Context, id string, req *CreateScheduleRequest) (*SnapshotSchedule, error) {
// Calculate next run time
nextRunAt, err := s.calculateNextRun(req.ScheduleType, req.ScheduleConfig)
if err != nil {
return nil, fmt.Errorf("failed to calculate next run time: %w", err)
}
// Marshal schedule config to JSON
configJSON, err := json.Marshal(req.ScheduleConfig)
if err != nil {
return nil, fmt.Errorf("failed to marshal schedule config: %w", err)
}
query := `
UPDATE snapshot_schedules
SET name = $1, dataset = $2, snapshot_name_template = $3, schedule_type = $4,
schedule_config = $5, recursive = $6, retention_count = $7, retention_days = $8,
next_run_at = $9, updated_at = NOW()
WHERE id = $10
RETURNING id, name, dataset, snapshot_name_template, schedule_type, schedule_config,
recursive, enabled, retention_count, retention_days,
last_run_at, next_run_at, created_by, created_at, updated_at
`
row := s.db.QueryRowContext(ctx, query,
req.Name, req.Dataset, req.SnapshotNameTemplate, req.ScheduleType,
string(configJSON), req.Recursive, req.RetentionCount, req.RetentionDays,
nextRunAt, id,
)
schedule, err := s.scanScheduleRow(row)
if err != nil {
if err == sql.ErrNoRows {
return nil, fmt.Errorf("schedule not found")
}
return nil, fmt.Errorf("failed to update schedule: %w", err)
}
s.logger.Info("Snapshot schedule updated", "id", id)
return schedule, nil
}
// DeleteSchedule deletes a schedule
func (s *SnapshotScheduleService) DeleteSchedule(ctx context.Context, id string) error {
query := `DELETE FROM snapshot_schedules WHERE id = $1`
result, err := s.db.ExecContext(ctx, query, id)
if err != nil {
return fmt.Errorf("failed to delete schedule: %w", err)
}
rowsAffected, err := result.RowsAffected()
if err != nil {
return fmt.Errorf("failed to get rows affected: %w", err)
}
if rowsAffected == 0 {
return fmt.Errorf("schedule not found")
}
s.logger.Info("Snapshot schedule deleted", "id", id)
return nil
}
// ToggleSchedule enables or disables a schedule
func (s *SnapshotScheduleService) ToggleSchedule(ctx context.Context, id string, enabled bool) error {
query := `UPDATE snapshot_schedules SET enabled = $1, updated_at = NOW() WHERE id = $2`
result, err := s.db.ExecContext(ctx, query, enabled, id)
if err != nil {
return fmt.Errorf("failed to toggle schedule: %w", err)
}
rowsAffected, err := result.RowsAffected()
if err != nil {
return fmt.Errorf("failed to get rows affected: %w", err)
}
if rowsAffected == 0 {
return fmt.Errorf("schedule not found")
}
s.logger.Info("Snapshot schedule toggled", "id", id, "enabled", enabled)
return nil
}
// scanScheduleRow scans a single schedule row
func (s *SnapshotScheduleService) scanScheduleRow(row *sql.Row) (*SnapshotSchedule, error) {
var schedule SnapshotSchedule
var configJSON string
var lastRunAt, nextRunAt sql.NullTime
var retentionCount, retentionDays sql.NullInt64
var createdBy sql.NullString
err := row.Scan(
&schedule.ID, &schedule.Name, &schedule.Dataset, &schedule.SnapshotNameTemplate,
&schedule.ScheduleType, &configJSON, &schedule.Recursive, &schedule.Enabled,
&retentionCount, &retentionDays, &lastRunAt, &nextRunAt,
&createdBy, &schedule.CreatedAt, &schedule.UpdatedAt,
)
if err != nil {
return nil, err
}
// Parse schedule config JSON
if err := json.Unmarshal([]byte(configJSON), &schedule.ScheduleConfig); err != nil {
return nil, fmt.Errorf("failed to unmarshal schedule config: %w", err)
}
// Handle nullable fields
if lastRunAt.Valid {
schedule.LastRunAt = &lastRunAt.Time
}
if nextRunAt.Valid {
schedule.NextRunAt = &nextRunAt.Time
}
if retentionCount.Valid {
count := int(retentionCount.Int64)
schedule.RetentionCount = &count
}
if retentionDays.Valid {
days := int(retentionDays.Int64)
schedule.RetentionDays = &days
}
if createdBy.Valid {
schedule.CreatedBy = createdBy.String
}
return &schedule, nil
}
// scanSchedule scans a schedule from rows
func (s *SnapshotScheduleService) scanSchedule(rows *sql.Rows) (*SnapshotSchedule, error) {
var schedule SnapshotSchedule
var configJSON string
var lastRunAt, nextRunAt sql.NullTime
var retentionCount, retentionDays sql.NullInt64
var createdBy sql.NullString
err := rows.Scan(
&schedule.ID, &schedule.Name, &schedule.Dataset, &schedule.SnapshotNameTemplate,
&schedule.ScheduleType, &configJSON, &schedule.Recursive, &schedule.Enabled,
&retentionCount, &retentionDays, &lastRunAt, &nextRunAt,
&createdBy, &schedule.CreatedAt, &schedule.UpdatedAt,
)
if err != nil {
return nil, err
}
// Parse schedule config JSON
if err := json.Unmarshal([]byte(configJSON), &schedule.ScheduleConfig); err != nil {
return nil, fmt.Errorf("failed to unmarshal schedule config: %w", err)
}
// Handle nullable fields
if lastRunAt.Valid {
schedule.LastRunAt = &lastRunAt.Time
}
if nextRunAt.Valid {
schedule.NextRunAt = &nextRunAt.Time
}
if retentionCount.Valid {
count := int(retentionCount.Int64)
schedule.RetentionCount = &count
}
if retentionDays.Valid {
days := int(retentionDays.Int64)
schedule.RetentionDays = &days
}
if createdBy.Valid {
schedule.CreatedBy = createdBy.String
}
return &schedule, nil
}
// calculateNextRun calculates the next run time based on schedule type and config
func (s *SnapshotScheduleService) calculateNextRun(scheduleType string, config map[string]interface{}) (*time.Time, error) {
now := time.Now()
switch scheduleType {
case "hourly":
nextRun := now.Add(1 * time.Hour)
// Round to next hour
nextRun = nextRun.Truncate(time.Hour).Add(1 * time.Hour)
return &nextRun, nil
case "daily":
timeStr, ok := config["time"].(string)
if !ok {
timeStr = "00:00"
}
// Parse time (HH:MM format)
hour, min := 0, 0
fmt.Sscanf(timeStr, "%d:%d", &hour, &min)
nextRun := time.Date(now.Year(), now.Month(), now.Day(), hour, min, 0, 0, now.Location())
if nextRun.Before(now) {
nextRun = nextRun.Add(24 * time.Hour)
}
return &nextRun, nil
case "weekly":
dayOfWeek, ok := config["day"].(float64) // JSON numbers are float64
if !ok {
dayOfWeek = 0 // Sunday
}
timeStr, ok := config["time"].(string)
if !ok {
timeStr = "00:00"
}
hour, min := 0, 0
fmt.Sscanf(timeStr, "%d:%d", &hour, &min)
// Calculate next occurrence
currentDay := int(now.Weekday())
targetDay := int(dayOfWeek)
daysUntil := (targetDay - currentDay + 7) % 7
if daysUntil == 0 {
// If same day, check if time has passed
today := time.Date(now.Year(), now.Month(), now.Day(), hour, min, 0, 0, now.Location())
if today.Before(now) {
daysUntil = 7
}
}
nextRun := time.Date(now.Year(), now.Month(), now.Day(), hour, min, 0, 0, now.Location()).AddDate(0, 0, daysUntil)
return &nextRun, nil
case "monthly":
day, ok := config["day"].(float64)
if !ok {
day = 1
}
timeStr, ok := config["time"].(string)
if !ok {
timeStr = "00:00"
}
hour, min := 0, 0
fmt.Sscanf(timeStr, "%d:%d", &hour, &min)
// Calculate next occurrence
nextRun := time.Date(now.Year(), now.Month(), int(day), hour, min, 0, 0, now.Location())
if nextRun.Before(now) {
nextRun = nextRun.AddDate(0, 1, 0)
}
return &nextRun, nil
case "cron":
// For cron, we'll use a simple implementation
// In production, you'd want to use a proper cron parser library
_, ok := config["cron"].(string)
if !ok {
return nil, fmt.Errorf("cron expression not found in config")
}
// Simple implementation: assume next run is in 1 hour
// TODO: Implement proper cron parsing
nextRun := now.Add(1 * time.Hour)
return &nextRun, nil
default:
return nil, fmt.Errorf("unknown schedule type: %s", scheduleType)
}
}
// GetDueSchedules retrieves all enabled schedules that are due to run
func (s *SnapshotScheduleService) GetDueSchedules(ctx context.Context) ([]*SnapshotSchedule, error) {
now := time.Now()
query := `
SELECT id, name, dataset, snapshot_name_template, schedule_type, schedule_config,
recursive, enabled, retention_count, retention_days,
last_run_at, next_run_at, created_by, created_at, updated_at
FROM snapshot_schedules
WHERE enabled = true AND next_run_at <= $1
ORDER BY next_run_at ASC
`
rows, err := s.db.QueryContext(ctx, query, now)
if err != nil {
return nil, fmt.Errorf("failed to query due schedules: %w", err)
}
defer rows.Close()
var schedules []*SnapshotSchedule
for rows.Next() {
schedule, err := s.scanSchedule(rows)
if err != nil {
s.logger.Error("Failed to scan schedule", "error", err)
continue
}
schedules = append(schedules, schedule)
}
return schedules, rows.Err()
}
// ExecuteSchedule executes a snapshot schedule
func (s *SnapshotScheduleService) ExecuteSchedule(ctx context.Context, schedule *SnapshotSchedule) error {
now := time.Now()
// Generate snapshot name from template
snapshotName := s.generateSnapshotName(schedule.SnapshotNameTemplate, now)
// Create snapshot
if err := s.snapshotService.CreateSnapshot(ctx, schedule.Dataset, snapshotName, schedule.Recursive); err != nil {
return fmt.Errorf("failed to create snapshot: %w", err)
}
s.logger.Info("Snapshot created from schedule", "schedule", schedule.Name, "snapshot", snapshotName, "dataset", schedule.Dataset)
// Calculate next run time
nextRunAt, err := s.calculateNextRun(schedule.ScheduleType, schedule.ScheduleConfig)
if err != nil {
s.logger.Error("Failed to calculate next run time", "error", err, "schedule", schedule.Name)
// Set a default next run time (1 hour from now) if calculation fails
defaultNextRun := now.Add(1 * time.Hour)
nextRunAt = &defaultNextRun
}
// Update schedule with last run time and next run time
query := `
UPDATE snapshot_schedules
SET last_run_at = $1, next_run_at = $2, updated_at = NOW()
WHERE id = $3
`
if _, err := s.db.ExecContext(ctx, query, now, nextRunAt, schedule.ID); err != nil {
s.logger.Error("Failed to update schedule after execution", "error", err, "schedule", schedule.Name)
// Don't return error here - snapshot was created successfully
}
// Handle retention
if err := s.handleRetention(ctx, schedule); err != nil {
s.logger.Error("Failed to handle retention", "error", err, "schedule", schedule.Name)
// Don't return error - retention is best effort
}
return nil
}
// generateSnapshotName generates a snapshot name from a template
func (s *SnapshotScheduleService) generateSnapshotName(template string, t time.Time) string {
// Replace common time format placeholders
name := template
name = strings.ReplaceAll(name, "%Y", t.Format("2006"))
name = strings.ReplaceAll(name, "%m", t.Format("01"))
name = strings.ReplaceAll(name, "%d", t.Format("02"))
name = strings.ReplaceAll(name, "%H", t.Format("15"))
name = strings.ReplaceAll(name, "%M", t.Format("04"))
name = strings.ReplaceAll(name, "%S", t.Format("05"))
name = strings.ReplaceAll(name, "%Y-%m-%d", t.Format("2006-01-02"))
name = strings.ReplaceAll(name, "%Y-%m-%d-%H%M", t.Format("2006-01-02-1504"))
name = strings.ReplaceAll(name, "%Y-%m-%d-%H%M%S", t.Format("2006-01-02-150405"))
// If no placeholders were replaced, append timestamp
if name == template {
name = fmt.Sprintf("%s-%d", template, t.Unix())
}
return name
}
// handleRetention handles snapshot retention based on schedule settings
func (s *SnapshotScheduleService) handleRetention(ctx context.Context, schedule *SnapshotSchedule) error {
// Get all snapshots for this dataset
snapshots, err := s.snapshotService.ListSnapshots(ctx, schedule.Dataset)
if err != nil {
return fmt.Errorf("failed to list snapshots: %w", err)
}
// Filter snapshots that match this schedule's naming pattern
matchingSnapshots := make([]*Snapshot, 0)
for _, snapshot := range snapshots {
// Check if snapshot name matches the template pattern
if s.snapshotMatchesTemplate(snapshot.SnapshotName, schedule.SnapshotNameTemplate) {
matchingSnapshots = append(matchingSnapshots, snapshot)
}
}
now := time.Now()
snapshotsToDelete := make([]*Snapshot, 0)
// Apply retention_count if set
if schedule.RetentionCount != nil && *schedule.RetentionCount > 0 {
if len(matchingSnapshots) > *schedule.RetentionCount {
// Sort by creation time (oldest first)
sort.Slice(matchingSnapshots, func(i, j int) bool {
return matchingSnapshots[i].Created.Before(matchingSnapshots[j].Created)
})
// Mark excess snapshots for deletion
excessCount := len(matchingSnapshots) - *schedule.RetentionCount
for i := 0; i < excessCount; i++ {
snapshotsToDelete = append(snapshotsToDelete, matchingSnapshots[i])
}
}
}
// Apply retention_days if set
if schedule.RetentionDays != nil && *schedule.RetentionDays > 0 {
cutoffTime := now.AddDate(0, 0, -*schedule.RetentionDays)
for _, snapshot := range matchingSnapshots {
if snapshot.Created.Before(cutoffTime) {
// Check if not already marked for deletion
alreadyMarked := false
for _, marked := range snapshotsToDelete {
if marked.ID == snapshot.ID {
alreadyMarked = true
break
}
}
if !alreadyMarked {
snapshotsToDelete = append(snapshotsToDelete, snapshot)
}
}
}
}
// Delete snapshots
for _, snapshot := range snapshotsToDelete {
if err := s.snapshotService.DeleteSnapshot(ctx, snapshot.Name, schedule.Recursive); err != nil {
s.logger.Error("Failed to delete snapshot for retention", "error", err, "snapshot", snapshot.Name)
continue
}
s.logger.Info("Deleted snapshot for retention", "snapshot", snapshot.Name, "schedule", schedule.Name)
}
return nil
}
// snapshotMatchesTemplate checks if a snapshot name matches a template pattern
func (s *SnapshotScheduleService) snapshotMatchesTemplate(snapshotName, template string) bool {
if template == "" {
return false
}
// Extract the base name from template (everything before the first %)
// This handles templates like "auto-%Y-%m-%d" or "daily-backup-%Y%m%d"
baseName := template
if idx := strings.Index(template, "%"); idx != -1 {
baseName = template[:idx]
}
// If template starts with a placeholder, check if snapshot name matches the pattern
// by checking if it starts with any reasonable prefix
if baseName == "" {
// Template starts with placeholder - use a more lenient check
// Check if snapshot name matches common patterns
return true // Match all if template is just placeholders
}
// Check if snapshot name starts with the base name
// This handles cases like template "auto-%Y-%m-%d" matching snapshot "auto-2024-01-15"
return strings.HasPrefix(snapshotName, baseName)
}

87
backend/internal/storage/snapshot_schedule_worker.go Normal file
View File

@@ -0,0 +1,87 @@
package storage
import (
"context"
"time"
"github.com/atlasos/calypso/internal/common/database"
"github.com/atlasos/calypso/internal/common/logger"
)
// SnapshotScheduleWorker handles periodic execution of snapshot schedules
type SnapshotScheduleWorker struct {
scheduleService *SnapshotScheduleService
logger *logger.Logger
interval time.Duration
stopCh chan struct{}
}
// NewSnapshotScheduleWorker creates a new snapshot schedule worker
func NewSnapshotScheduleWorker(db *database.DB, log *logger.Logger, snapshotService *SnapshotService, interval time.Duration) *SnapshotScheduleWorker {
scheduleService := NewSnapshotScheduleService(db, log, snapshotService)
return &SnapshotScheduleWorker{
scheduleService: scheduleService,
logger: log,
interval: interval,
stopCh: make(chan struct{}),
}
}
// Start starts the snapshot schedule worker background service
func (w *SnapshotScheduleWorker) Start(ctx context.Context) {
w.logger.Info("Starting snapshot schedule worker", "interval", w.interval)
ticker := time.NewTicker(w.interval)
defer ticker.Stop()
// Run initial check immediately
w.processSchedules(ctx)
for {
select {
case <-ctx.Done():
w.logger.Info("Snapshot schedule worker stopped")
return
case <-w.stopCh:
w.logger.Info("Snapshot schedule worker stopped")
return
case <-ticker.C:
w.processSchedules(ctx)
}
}
}
// Stop stops the snapshot schedule worker service
func (w *SnapshotScheduleWorker) Stop() {
close(w.stopCh)
}
// processSchedules processes all due snapshot schedules
func (w *SnapshotScheduleWorker) processSchedules(ctx context.Context) {
w.logger.Debug("Checking for due snapshot schedules")
// Get all schedules that are due to run
schedules, err := w.scheduleService.GetDueSchedules(ctx)
if err != nil {
w.logger.Error("Failed to get due schedules", "error", err)
return
}
if len(schedules) == 0 {
w.logger.Debug("No snapshot schedules due to run")
return
}
w.logger.Info("Found due snapshot schedules", "count", len(schedules))
// Execute each schedule
for _, schedule := range schedules {
w.logger.Info("Executing snapshot schedule", "schedule", schedule.Name, "dataset", schedule.Dataset)
if err := w.scheduleService.ExecuteSchedule(ctx, schedule); err != nil {
w.logger.Error("Failed to execute snapshot schedule", "error", err, "schedule", schedule.Name)
continue
}
w.logger.Info("Successfully executed snapshot schedule", "schedule", schedule.Name)
}
}

136
backend/internal/storage/zfs.go
View File

@@ -16,6 +16,16 @@ import (
"github.com/lib/pq"
)
// zfsCommand executes a ZFS command with sudo
func zfsCommand(ctx context.Context, args ...string) *exec.Cmd {
return exec.CommandContext(ctx, "sudo", append([]string{"zfs"}, args...)...)
}
// zpoolCommand executes a ZPOOL command with sudo
func zpoolCommand(ctx context.Context, args ...string) *exec.Cmd {
return exec.CommandContext(ctx, "sudo", append([]string{"zpool"}, args...)...)
}
// ZFSService handles ZFS pool management
type ZFSService struct {
db *database.DB
@@ -115,6 +125,10 @@ func (s *ZFSService) CreatePool(ctx context.Context, name string, raidLevel stri
var args []string
args = append(args, "create", "-f") // -f to force creation
// Set default mountpoint to /opt/calypso/data/pool/<pool-name>
mountPoint := fmt.Sprintf("/opt/calypso/data/pool/%s", name)
args = append(args, "-m", mountPoint)
// Note: compression is a filesystem property, not a pool property
// We'll set it after pool creation using zfs set
@@ -155,9 +169,15 @@ func (s *ZFSService) CreatePool(ctx context.Context, name string, raidLevel stri
args = append(args, disks...)
}
// Execute zpool create
s.logger.Info("Creating ZFS pool", "name", name, "raid_level", raidLevel, "disks", disks, "args", args)
cmd := exec.CommandContext(ctx, "zpool", args...)
// Create mountpoint directory if it doesn't exist
if err := os.MkdirAll(mountPoint, 0755); err != nil {
return nil, fmt.Errorf("failed to create mountpoint directory %s: %w", mountPoint, err)
}
s.logger.Info("Created mountpoint directory", "path", mountPoint)
// Execute zpool create (with sudo for permissions)
s.logger.Info("Creating ZFS pool", "name", name, "raid_level", raidLevel, "disks", disks, "mountpoint", mountPoint, "args", args)
cmd := zpoolCommand(ctx, args...)
output, err := cmd.CombinedOutput()
if err != nil {
errorMsg := string(output)
@@ -170,7 +190,7 @@ func (s *ZFSService) CreatePool(ctx context.Context, name string, raidLevel stri
// Set filesystem properties (compression, etc.) after pool creation
// ZFS creates a root filesystem with the same name as the pool
if compression != "" && compression != "off" {
cmd = exec.CommandContext(ctx, "zfs", "set", fmt.Sprintf("compression=%s", compression), name)
cmd = zfsCommand(ctx, "set", fmt.Sprintf("compression=%s", compression), name)
output, err = cmd.CombinedOutput()
if err != nil {
s.logger.Warn("Failed to set compression property", "pool", name, "compression", compression, "error", string(output))
@@ -185,7 +205,7 @@ func (s *ZFSService) CreatePool(ctx context.Context, name string, raidLevel stri
if err != nil {
// Try to destroy the pool if we can't get info
s.logger.Warn("Failed to get pool info, attempting to destroy pool", "name", name, "error", err)
exec.CommandContext(ctx, "zpool", "destroy", "-f", name).Run()
zpoolCommand(ctx, "destroy", "-f", name).Run()
return nil, fmt.Errorf("failed to get pool info after creation: %w", err)
}
@@ -219,7 +239,7 @@ func (s *ZFSService) CreatePool(ctx context.Context, name string, raidLevel stri
if err != nil {
// Cleanup: destroy pool if database insert fails
s.logger.Error("Failed to save pool to database, destroying pool", "name", name, "error", err)
exec.CommandContext(ctx, "zpool", "destroy", "-f", name).Run()
zpoolCommand(ctx, "destroy", "-f", name).Run()
return nil, fmt.Errorf("failed to save pool to database: %w", err)
}
@@ -243,7 +263,7 @@ func (s *ZFSService) CreatePool(ctx context.Context, name string, raidLevel stri
// getPoolInfo retrieves information about a ZFS pool
func (s *ZFSService) getPoolInfo(ctx context.Context, poolName string) (*ZFSPool, error) {
// Get pool size and used space
cmd := exec.CommandContext(ctx, "zpool", "list", "-H", "-o", "name,size,allocated", poolName)
cmd := zpoolCommand(ctx, "list", "-H", "-o", "name,size,allocated", poolName)
output, err := cmd.CombinedOutput()
if err != nil {
errorMsg := string(output)
@@ -322,7 +342,7 @@ func parseZFSSize(sizeStr string) (int64, error) {
// getSpareDisks retrieves spare disks from zpool status
func (s *ZFSService) getSpareDisks(ctx context.Context, poolName string) ([]string, error) {
cmd := exec.CommandContext(ctx, "zpool", "status", poolName)
cmd := zpoolCommand(ctx, "status", poolName)
output, err := cmd.CombinedOutput()
if err != nil {
return nil, fmt.Errorf("failed to get pool status: %w", err)
@@ -363,7 +383,7 @@ func (s *ZFSService) getSpareDisks(ctx context.Context, poolName string) ([]stri
// getCompressRatio gets the compression ratio from ZFS
func (s *ZFSService) getCompressRatio(ctx context.Context, poolName string) (float64, error) {
cmd := exec.CommandContext(ctx, "zfs", "get", "-H", "-o", "value", "compressratio", poolName)
cmd := zfsCommand(ctx, "get", "-H", "-o", "value", "compressratio", poolName)
output, err := cmd.Output()
if err != nil {
return 1.0, err
@@ -406,16 +426,20 @@ func (s *ZFSService) ListPools(ctx context.Context) ([]*ZFSPool, error) {
for rows.Next() {
var pool ZFSPool
var description sql.NullString
var createdBy sql.NullString
err := rows.Scan(
&pool.ID, &pool.Name, &description, &pool.RaidLevel, pq.Array(&pool.Disks),
&pool.SizeBytes, &pool.UsedBytes, &pool.Compression, &pool.Deduplication,
&pool.AutoExpand, &pool.ScrubInterval, &pool.IsActive, &pool.HealthStatus,
&pool.CreatedAt, &pool.UpdatedAt, &pool.CreatedBy,
&pool.CreatedAt, &pool.UpdatedAt, &createdBy,
)
if err != nil {
s.logger.Error("Failed to scan pool row", "error", err)
s.logger.Error("Failed to scan pool row", "error", err, "error_type", fmt.Sprintf("%T", err))
continue // Skip this pool instead of failing entire query
}
if createdBy.Valid {
pool.CreatedBy = createdBy.String
}
if description.Valid {
pool.Description = description.String
}
@@ -501,7 +525,7 @@ func (s *ZFSService) DeletePool(ctx context.Context, poolID string) error {
// Destroy ZFS pool with -f flag to force destroy (works for both empty and non-empty pools)
// The -f flag is needed to destroy pools even if they have datasets or are in use
s.logger.Info("Destroying ZFS pool", "pool", pool.Name)
cmd := exec.CommandContext(ctx, "zpool", "destroy", "-f", pool.Name)
cmd := zpoolCommand(ctx, "destroy", "-f", pool.Name)
output, err := cmd.CombinedOutput()
if err != nil {
errorMsg := string(output)
@@ -516,6 +540,15 @@ func (s *ZFSService) DeletePool(ctx context.Context, poolID string) error {
s.logger.Info("ZFS pool destroyed successfully", "pool", pool.Name)
}
// Remove mount point directory (default: /opt/calypso/data/pool/<pool-name>)
mountPoint := fmt.Sprintf("/opt/calypso/data/pool/%s", pool.Name)
if err := os.RemoveAll(mountPoint); err != nil {
s.logger.Warn("Failed to remove mount point directory", "mountpoint", mountPoint, "error", err)
// Don't fail pool deletion if mount point removal fails
} else {
s.logger.Info("Removed mount point directory", "mountpoint", mountPoint)
}
// Mark disks as unused
for _, diskPath := range pool.Disks {
_, err = s.db.ExecContext(ctx,
@@ -550,7 +583,7 @@ func (s *ZFSService) AddSpareDisk(ctx context.Context, poolID string, diskPaths
}
// Verify pool exists in ZFS and check if disks are already spare
cmd := exec.CommandContext(ctx, "zpool", "status", pool.Name)
cmd := zpoolCommand(ctx, "status", pool.Name)
output, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("pool %s does not exist in ZFS: %w", pool.Name, err)
@@ -575,7 +608,7 @@ func (s *ZFSService) AddSpareDisk(ctx context.Context, poolID string, diskPaths
// Execute zpool add
s.logger.Info("Adding spare disks to ZFS pool", "pool", pool.Name, "disks", diskPaths)
cmd = exec.CommandContext(ctx, "zpool", args...)
cmd = zpoolCommand(ctx, args...)
output, err = cmd.CombinedOutput()
if err != nil {
errorMsg := string(output)
@@ -610,6 +643,7 @@ func (s *ZFSService) AddSpareDisk(ctx context.Context, poolID string, diskPaths
// ZFSDataset represents a ZFS dataset
type ZFSDataset struct {
ID string `json:"id"`
Name string `json:"name"`
Pool string `json:"pool"`
Type string `json:"type"` // filesystem, volume, snapshot
@@ -628,7 +662,7 @@ type ZFSDataset struct {
func (s *ZFSService) ListDatasets(ctx context.Context, poolName string) ([]*ZFSDataset, error) {
// Get datasets from database
query := `
SELECT name, pool_name, type, mount_point,
SELECT id, name, pool_name, type, mount_point,
used_bytes, available_bytes, referenced_bytes,
compression, deduplication, quota, reservation,
created_at
@@ -654,7 +688,7 @@ func (s *ZFSService) ListDatasets(ctx context.Context, poolName string) ([]*ZFSD
var mountPoint sql.NullString
err := rows.Scan(
&ds.Name, &ds.Pool, &ds.Type, &mountPoint,
&ds.ID, &ds.Name, &ds.Pool, &ds.Type, &mountPoint,
&ds.UsedBytes, &ds.AvailableBytes, &ds.ReferencedBytes,
&ds.Compression, &ds.Deduplication, &ds.Quota, &ds.Reservation,
&ds.CreatedAt,
@@ -696,10 +730,36 @@ func (s *ZFSService) CreateDataset(ctx context.Context, poolName string, req Cre
// Construct full dataset name
fullName := poolName + "/" + req.Name
// For filesystem datasets, create mount directory if mount point is provided
if req.Type == "filesystem" && req.MountPoint != "" {
// Clean and validate mount point path
mountPath := filepath.Clean(req.MountPoint)
// Get pool mount point to validate dataset mount point is within pool directory
poolMountPoint := fmt.Sprintf("/opt/calypso/data/pool/%s", poolName)
var mountPath string
// For filesystem datasets, validate and set mount point
if req.Type == "filesystem" {
if req.MountPoint != "" {
// User provided mount point - validate it's within pool directory
mountPath = filepath.Clean(req.MountPoint)
// Check if mount point is within pool mount point directory
poolMountAbs, err := filepath.Abs(poolMountPoint)
if err != nil {
return nil, fmt.Errorf("failed to resolve pool mount point: %w", err)
}
mountPathAbs, err := filepath.Abs(mountPath)
if err != nil {
return nil, fmt.Errorf("failed to resolve mount point: %w", err)
}
// Check if mount path is within pool mount point directory
relPath, err := filepath.Rel(poolMountAbs, mountPathAbs)
if err != nil || strings.HasPrefix(relPath, "..") {
return nil, fmt.Errorf("mount point must be within pool directory: %s (pool mount: %s)", mountPath, poolMountPoint)
}
} else {
// No mount point provided - use default: /opt/calypso/data/pool/<pool-name>/<dataset-name>/
mountPath = filepath.Join(poolMountPoint, req.Name)
}
// Check if directory already exists
if info, err := os.Stat(mountPath); err == nil {
@@ -748,14 +808,14 @@ func (s *ZFSService) CreateDataset(ctx context.Context, poolName string, req Cre
args = append(args, "-o", fmt.Sprintf("compression=%s", req.Compression))
}
// Set mount point if provided (only for filesystems, not volumes)
if req.Type == "filesystem" && req.MountPoint != "" {
args = append(args, "-o", fmt.Sprintf("mountpoint=%s", req.MountPoint))
// Set mount point for filesystems (always set, either user-provided or default)
if req.Type == "filesystem" {
args = append(args, "-o", fmt.Sprintf("mountpoint=%s", mountPath))
}
// Execute zfs create
s.logger.Info("Creating ZFS dataset", "name", fullName, "type", req.Type)
cmd := exec.CommandContext(ctx, "zfs", args...)
cmd := zfsCommand(ctx, args...)
output, err := cmd.CombinedOutput()
if err != nil {
errorMsg := string(output)
@@ -765,7 +825,7 @@ func (s *ZFSService) CreateDataset(ctx context.Context, poolName string, req Cre
// Set quota if specified (for filesystems)
if req.Type == "filesystem" && req.Quota > 0 {
quotaCmd := exec.CommandContext(ctx, "zfs", "set", fmt.Sprintf("quota=%d", req.Quota), fullName)
quotaCmd := zfsCommand(ctx, "set", fmt.Sprintf("quota=%d", req.Quota), fullName)
if quotaOutput, err := quotaCmd.CombinedOutput(); err != nil {
s.logger.Warn("Failed to set quota", "dataset", fullName, "error", err, "output", string(quotaOutput))
}
@@ -773,7 +833,7 @@ func (s *ZFSService) CreateDataset(ctx context.Context, poolName string, req Cre
// Set reservation if specified
if req.Reservation > 0 {
resvCmd := exec.CommandContext(ctx, "zfs", "set", fmt.Sprintf("reservation=%d", req.Reservation), fullName)
resvCmd := zfsCommand(ctx, "set", fmt.Sprintf("reservation=%d", req.Reservation), fullName)
if resvOutput, err := resvCmd.CombinedOutput(); err != nil {
s.logger.Warn("Failed to set reservation", "dataset", fullName, "error", err, "output", string(resvOutput))
}
@@ -785,30 +845,30 @@ func (s *ZFSService) CreateDataset(ctx context.Context, poolName string, req Cre
if err != nil {
s.logger.Error("Failed to get pool ID", "pool", poolName, "error", err)
// Try to destroy the dataset if we can't save to database
exec.CommandContext(ctx, "zfs", "destroy", "-r", fullName).Run()
zfsCommand(ctx, "destroy", "-r", fullName).Run()
return nil, fmt.Errorf("failed to get pool ID: %w", err)
}
// Get dataset info from ZFS to save to database
cmd = exec.CommandContext(ctx, "zfs", "list", "-H", "-o", "name,used,avail,refer,compress,dedup,quota,reservation,mountpoint", fullName)
cmd = zfsCommand(ctx, "list", "-H", "-o", "name,used,avail,refer,compress,dedup,quota,reservation,mountpoint", fullName)
output, err = cmd.CombinedOutput()
if err != nil {
s.logger.Error("Failed to get dataset info", "name", fullName, "error", err)
// Try to destroy the dataset if we can't get info
exec.CommandContext(ctx, "zfs", "destroy", "-r", fullName).Run()
zfsCommand(ctx, "destroy", "-r", fullName).Run()
return nil, fmt.Errorf("failed to get dataset info: %w", err)
}
// Parse dataset info
lines := strings.TrimSpace(string(output))
if lines == "" {
exec.CommandContext(ctx, "zfs", "destroy", "-r", fullName).Run()
zfsCommand(ctx, "destroy", "-r", fullName).Run()
return nil, fmt.Errorf("dataset not found after creation")
}
fields := strings.Fields(lines)
if len(fields) < 9 {
exec.CommandContext(ctx, "zfs", "destroy", "-r", fullName).Run()
zfsCommand(ctx, "destroy", "-r", fullName).Run()
return nil, fmt.Errorf("invalid dataset info format")
}
@@ -823,7 +883,7 @@ func (s *ZFSService) CreateDataset(ctx context.Context, poolName string, req Cre
// Determine dataset type
datasetType := req.Type
typeCmd := exec.CommandContext(ctx, "zfs", "get", "-H", "-o", "value", "type", fullName)
typeCmd := zfsCommand(ctx, "get", "-H", "-o", "value", "type", fullName)
if typeOutput, err := typeCmd.Output(); err == nil {
volType := strings.TrimSpace(string(typeOutput))
if volType == "volume" {
@@ -837,7 +897,7 @@ func (s *ZFSService) CreateDataset(ctx context.Context, poolName string, req Cre
quota := int64(-1)
if datasetType == "volume" {
// For volumes, get volsize
volsizeCmd := exec.CommandContext(ctx, "zfs", "get", "-H", "-o", "value", "volsize", fullName)
volsizeCmd := zfsCommand(ctx, "get", "-H", "-o", "value", "volsize", fullName)
if volsizeOutput, err := volsizeCmd.Output(); err == nil {
volsizeStr := strings.TrimSpace(string(volsizeOutput))
if volsizeStr != "-" && volsizeStr != "none" {
@@ -867,7 +927,7 @@ func (s *ZFSService) CreateDataset(ctx context.Context, poolName string, req Cre
// Get creation time
createdAt := time.Now()
creationCmd := exec.CommandContext(ctx, "zfs", "get", "-H", "-o", "value", "creation", fullName)
creationCmd := zfsCommand(ctx, "get", "-H", "-o", "value", "creation", fullName)
if creationOutput, err := creationCmd.Output(); err == nil {
creationStr := strings.TrimSpace(string(creationOutput))
if t, err := time.Parse("Mon Jan 2 15:04:05 2006", creationStr); err == nil {
@@ -899,7 +959,7 @@ func (s *ZFSService) CreateDataset(ctx context.Context, poolName string, req Cre
if err != nil {
s.logger.Error("Failed to save dataset to database", "name", fullName, "error", err)
// Try to destroy the dataset if we can't save to database
exec.CommandContext(ctx, "zfs", "destroy", "-r", fullName).Run()
zfsCommand(ctx, "destroy", "-r", fullName).Run()
return nil, fmt.Errorf("failed to save dataset to database: %w", err)
}
@@ -927,7 +987,7 @@ func (s *ZFSService) CreateDataset(ctx context.Context, poolName string, req Cre
func (s *ZFSService) DeleteDataset(ctx context.Context, datasetName string) error {
// Check if dataset exists and get its mount point before deletion
var mountPoint string
cmd := exec.CommandContext(ctx, "zfs", "list", "-H", "-o", "name,mountpoint", datasetName)
cmd := zfsCommand(ctx, "list", "-H", "-o", "name,mountpoint", datasetName)
output, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("dataset %s does not exist: %w", datasetName, err)
@@ -946,7 +1006,7 @@ func (s *ZFSService) DeleteDataset(ctx context.Context, datasetName string) erro
// Get dataset type to determine if we should clean up mount directory
var datasetType string
typeCmd := exec.CommandContext(ctx, "zfs", "get", "-H", "-o", "value", "type", datasetName)
typeCmd := zfsCommand(ctx, "get", "-H", "-o", "value", "type", datasetName)
typeOutput, err := typeCmd.Output()
if err == nil {
datasetType = strings.TrimSpace(string(typeOutput))
@@ -969,7 +1029,7 @@ func (s *ZFSService) DeleteDataset(ctx context.Context, datasetName string) erro
// Delete the dataset from ZFS (use -r for recursive to delete children)
s.logger.Info("Deleting ZFS dataset", "name", datasetName, "mountpoint", mountPoint)
cmd = exec.CommandContext(ctx, "zfs", "destroy", "-r", datasetName)
cmd = zfsCommand(ctx, "destroy", "-r", datasetName)
output, err = cmd.CombinedOutput()
if err != nil {
errorMsg := string(output)

15
backend/internal/storage/zfs_pool_monitor.go
View File

@@ -2,6 +2,7 @@ package storage
import (
"context"
"fmt"
"os/exec"
"regexp"
"strconv"
@@ -98,11 +99,17 @@ type PoolInfo struct {
func (m *ZFSPoolMonitor) getSystemPools(ctx context.Context) (map[string]PoolInfo, error) {
pools := make(map[string]PoolInfo)
// Get pool list
cmd := exec.CommandContext(ctx, "zpool", "list", "-H", "-o", "name,size,alloc,free,health")
output, err := cmd.Output()
// Get pool list (with sudo for permissions)
cmd := exec.CommandContext(ctx, "sudo", "zpool", "list", "-H", "-o", "name,size,alloc,free,health")
output, err := cmd.CombinedOutput()
if err != nil {
return nil, err
// If no pools exist, zpool list returns exit code 1 but that's OK
// Check if output is empty (no pools) vs actual error
outputStr := strings.TrimSpace(string(output))
if outputStr == "" || strings.Contains(outputStr, "no pools available") {
return pools, nil // No pools, return empty map (not an error)
}
return nil, fmt.Errorf("zpool list failed: %w, output: %s", err, outputStr)
}
lines := strings.Split(strings.TrimSpace(string(output)), "\n")

161
backend/internal/system/handler.go
View File

@@ -3,6 +3,7 @@ package system
import (
"net/http"
"strconv"
"time"
"github.com/atlasos/calypso/internal/common/logger"
"github.com/atlasos/calypso/internal/tasks"
@@ -131,3 +132,163 @@ func (h *Handler) ListNetworkInterfaces(c *gin.Context) {
c.JSON(http.StatusOK, gin.H{"interfaces": interfaces})
}
// GetManagementIPAddress returns the management IP address
func (h *Handler) GetManagementIPAddress(c *gin.Context) {
ip, err := h.service.GetManagementIPAddress(c.Request.Context())
if err != nil {
h.logger.Error("Failed to get management IP address", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to get management IP address"})
return
}
c.JSON(http.StatusOK, gin.H{"ip_address": ip})
}
// SaveNTPSettings saves NTP configuration to the OS
func (h *Handler) SaveNTPSettings(c *gin.Context) {
var settings NTPSettings
if err := c.ShouldBindJSON(&settings); err != nil {
h.logger.Error("Invalid request body", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request body"})
return
}
// Validate timezone
if settings.Timezone == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "timezone is required"})
return
}
// Validate NTP servers
if len(settings.NTPServers) == 0 {
c.JSON(http.StatusBadRequest, gin.H{"error": "at least one NTP server is required"})
return
}
if err := h.service.SaveNTPSettings(c.Request.Context(), settings); err != nil {
h.logger.Error("Failed to save NTP settings", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"message": "NTP settings saved successfully"})
}
// GetNTPSettings retrieves current NTP configuration
func (h *Handler) GetNTPSettings(c *gin.Context) {
settings, err := h.service.GetNTPSettings(c.Request.Context())
if err != nil {
h.logger.Error("Failed to get NTP settings", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to get NTP settings"})
return
}
c.JSON(http.StatusOK, gin.H{"settings": settings})
}
// UpdateNetworkInterface updates a network interface configuration
func (h *Handler) UpdateNetworkInterface(c *gin.Context) {
ifaceName := c.Param("name")
if ifaceName == "" {
c.JSON(http.StatusBadRequest, gin.H{"error": "interface name is required"})
return
}
var req struct {
IPAddress string `json:"ip_address" binding:"required"`
Subnet string `json:"subnet" binding:"required"`
Gateway string `json:"gateway,omitempty"`
DNS1 string `json:"dns1,omitempty"`
DNS2 string `json:"dns2,omitempty"`
Role string `json:"role,omitempty"`
}
if err := c.ShouldBindJSON(&req); err != nil {
h.logger.Error("Invalid request body", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "invalid request body"})
return
}
// Convert to service request
serviceReq := UpdateNetworkInterfaceRequest{
IPAddress: req.IPAddress,
Subnet: req.Subnet,
Gateway: req.Gateway,
DNS1: req.DNS1,
DNS2: req.DNS2,
Role: req.Role,
}
updatedIface, err := h.service.UpdateNetworkInterface(c.Request.Context(), ifaceName, serviceReq)
if err != nil {
h.logger.Error("Failed to update network interface", "interface", ifaceName, "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
return
}
c.JSON(http.StatusOK, gin.H{"interface": updatedIface})
}
// GetSystemLogs retrieves recent system logs
func (h *Handler) GetSystemLogs(c *gin.Context) {
limitStr := c.DefaultQuery("limit", "30")
limit, err := strconv.Atoi(limitStr)
if err != nil || limit <= 0 || limit > 100 {
limit = 30
}
logs, err := h.service.GetSystemLogs(c.Request.Context(), limit)
if err != nil {
h.logger.Error("Failed to get system logs", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to get system logs"})
return
}
c.JSON(http.StatusOK, gin.H{"logs": logs})
}
// GetNetworkThroughput retrieves network throughput data from RRD
func (h *Handler) GetNetworkThroughput(c *gin.Context) {
// Default to last 5 minutes
durationStr := c.DefaultQuery("duration", "5m")
duration, err := time.ParseDuration(durationStr)
if err != nil {
duration = 5 * time.Minute
}
data, err := h.service.GetNetworkThroughput(c.Request.Context(), duration)
if err != nil {
h.logger.Error("Failed to get network throughput", "error", err)
c.JSON(http.StatusInternalServerError, gin.H{"error": "failed to get network throughput"})
return
}
c.JSON(http.StatusOK, gin.H{"data": data})
}
// ExecuteCommand executes a shell command
func (h *Handler) ExecuteCommand(c *gin.Context) {
var req struct {
Command string `json:"command" binding:"required"`
Service string `json:"service,omitempty"` // Optional: system, scst, storage, backup, tape
}
if err := c.ShouldBindJSON(&req); err != nil {
h.logger.Error("Invalid request body", "error", err)
c.JSON(http.StatusBadRequest, gin.H{"error": "command is required"})
return
}
// Execute command based on service context
output, err := h.service.ExecuteCommand(c.Request.Context(), req.Command, req.Service)
if err != nil {
h.logger.Error("Failed to execute command", "error", err, "command", req.Command, "service", req.Service)
c.JSON(http.StatusInternalServerError, gin.H{
"error": err.Error(),
"output": output, // Include output even on error
})
return
}
c.JSON(http.StatusOK, gin.H{"output": output})
}

292
backend/internal/system/rrd.go Normal file
View File

@@ -0,0 +1,292 @@
package system
import (
"context"
"fmt"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"time"
"github.com/atlasos/calypso/internal/common/logger"
)
// RRDService handles RRD database operations for network monitoring
type RRDService struct {
logger *logger.Logger
rrdDir string
interfaceName string
}
// NewRRDService creates a new RRD service
func NewRRDService(log *logger.Logger, rrdDir string, interfaceName string) *RRDService {
return &RRDService{
logger: log,
rrdDir: rrdDir,
interfaceName: interfaceName,
}
}
// NetworkStats represents network interface statistics
type NetworkStats struct {
Interface string `json:"interface"`
RxBytes uint64 `json:"rx_bytes"`
TxBytes uint64 `json:"tx_bytes"`
RxPackets uint64 `json:"rx_packets"`
TxPackets uint64 `json:"tx_packets"`
Timestamp time.Time `json:"timestamp"`
}
// GetNetworkStats reads network statistics from /proc/net/dev
func (r *RRDService) GetNetworkStats(ctx context.Context, interfaceName string) (*NetworkStats, error) {
data, err := os.ReadFile("/proc/net/dev")
if err != nil {
return nil, fmt.Errorf("failed to read /proc/net/dev: %w", err)
}
lines := strings.Split(string(data), "\n")
for _, line := range lines {
line = strings.TrimSpace(line)
if !strings.HasPrefix(line, interfaceName+":") {
continue
}
// Parse line: interface: rx_bytes rx_packets ... tx_bytes tx_packets ...
parts := strings.Fields(line)
if len(parts) < 17 {
continue
}
// Extract statistics
// Format: interface: rx_bytes rx_packets rx_errs rx_drop ... tx_bytes tx_packets ...
rxBytes, err := strconv.ParseUint(parts[1], 10, 64)
if err != nil {
continue
}
rxPackets, err := strconv.ParseUint(parts[2], 10, 64)
if err != nil {
continue
}
txBytes, err := strconv.ParseUint(parts[9], 10, 64)
if err != nil {
continue
}
txPackets, err := strconv.ParseUint(parts[10], 10, 64)
if err != nil {
continue
}
return &NetworkStats{
Interface: interfaceName,
RxBytes: rxBytes,
TxBytes: txBytes,
RxPackets: rxPackets,
TxPackets: txPackets,
Timestamp: time.Now(),
}, nil
}
return nil, fmt.Errorf("interface %s not found in /proc/net/dev", interfaceName)
}
// InitializeRRD creates RRD database if it doesn't exist
func (r *RRDService) InitializeRRD(ctx context.Context) error {
// Ensure RRD directory exists
if err := os.MkdirAll(r.rrdDir, 0755); err != nil {
return fmt.Errorf("failed to create RRD directory: %w", err)
}
rrdFile := filepath.Join(r.rrdDir, fmt.Sprintf("network-%s.rrd", r.interfaceName))
// Check if RRD file already exists
if _, err := os.Stat(rrdFile); err == nil {
r.logger.Info("RRD file already exists", "file", rrdFile)
return nil
}
// Create RRD database
// Use COUNTER type to track cumulative bytes, RRD will calculate rate automatically
// DS:inbound:COUNTER:20:0:U - inbound cumulative bytes, 20s heartbeat
// DS:outbound:COUNTER:20:0:U - outbound cumulative bytes, 20s heartbeat
// RRA:AVERAGE:0.5:1:600 - 1 sample per step, 600 steps (100 minutes at 10s interval)
// RRA:AVERAGE:0.5:6:700 - 6 samples per step, 700 steps (11.6 hours at 1min interval)
// RRA:AVERAGE:0.5:60:730 - 60 samples per step, 730 steps (5 days at 1hour interval)
// RRA:MAX:0.5:1:600 - Max values for same intervals
// RRA:MAX:0.5:6:700
// RRA:MAX:0.5:60:730
cmd := exec.CommandContext(ctx, "rrdtool", "create", rrdFile,
"--step", "10", // 10 second step
"DS:inbound:COUNTER:20:0:U", // Inbound cumulative bytes, 20s heartbeat
"DS:outbound:COUNTER:20:0:U", // Outbound cumulative bytes, 20s heartbeat
"RRA:AVERAGE:0.5:1:600", // 10s resolution, 100 minutes
"RRA:AVERAGE:0.5:6:700", // 1min resolution, 11.6 hours
"RRA:AVERAGE:0.5:60:730", // 1hour resolution, 5 days
"RRA:MAX:0.5:1:600", // Max values
"RRA:MAX:0.5:6:700",
"RRA:MAX:0.5:60:730",
)
output, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("failed to create RRD: %s: %w", string(output), err)
}
r.logger.Info("RRD database created", "file", rrdFile)
return nil
}
// UpdateRRD updates RRD database with new network statistics
func (r *RRDService) UpdateRRD(ctx context.Context, stats *NetworkStats) error {
rrdFile := filepath.Join(r.rrdDir, fmt.Sprintf("network-%s.rrd", stats.Interface))
// Update with cumulative byte counts (COUNTER type)
// RRD will automatically calculate the rate (bytes per second)
cmd := exec.CommandContext(ctx, "rrdtool", "update", rrdFile,
fmt.Sprintf("%d:%d:%d", stats.Timestamp.Unix(), stats.RxBytes, stats.TxBytes),
)
output, err := cmd.CombinedOutput()
if err != nil {
return fmt.Errorf("failed to update RRD: %s: %w", string(output), err)
}
return nil
}
// FetchRRDData fetches data from RRD database for graphing
func (r *RRDService) FetchRRDData(ctx context.Context, startTime time.Time, endTime time.Time, resolution string) ([]NetworkDataPoint, error) {
rrdFile := filepath.Join(r.rrdDir, fmt.Sprintf("network-%s.rrd", r.interfaceName))
// Check if RRD file exists
if _, err := os.Stat(rrdFile); os.IsNotExist(err) {
return []NetworkDataPoint{}, nil
}
// Fetch data using rrdtool fetch
// Use AVERAGE consolidation with appropriate resolution
cmd := exec.CommandContext(ctx, "rrdtool", "fetch", rrdFile,
"AVERAGE",
"--start", fmt.Sprintf("%d", startTime.Unix()),
"--end", fmt.Sprintf("%d", endTime.Unix()),
"--resolution", resolution,
)
output, err := cmd.CombinedOutput()
if err != nil {
return nil, fmt.Errorf("failed to fetch RRD data: %s: %w", string(output), err)
}
// Parse rrdtool fetch output
// Format:
// inbound outbound
// 1234567890: 1.2345678901e+06 2.3456789012e+06
points := []NetworkDataPoint{}
lines := strings.Split(string(output), "\n")
// Skip header lines
dataStart := false
for _, line := range lines {
line = strings.TrimSpace(line)
if line == "" {
continue
}
// Check if this is the data section
if strings.Contains(line, "inbound") && strings.Contains(line, "outbound") {
dataStart = true
continue
}
if !dataStart {
continue
}
// Parse data line: timestamp: inbound_value outbound_value
parts := strings.Fields(line)
if len(parts) < 3 {
continue
}
// Parse timestamp
timestampStr := strings.TrimSuffix(parts[0], ":")
timestamp, err := strconv.ParseInt(timestampStr, 10, 64)
if err != nil {
continue
}
// Parse inbound (bytes per second from COUNTER, convert to Mbps)
inboundStr := parts[1]
inbound, err := strconv.ParseFloat(inboundStr, 64)
if err != nil || inbound < 0 {
// Skip NaN or negative values
continue
}
// Convert bytes per second to Mbps (bytes/s * 8 / 1000000)
inboundMbps := inbound * 8 / 1000000
// Parse outbound
outboundStr := parts[2]
outbound, err := strconv.ParseFloat(outboundStr, 64)
if err != nil || outbound < 0 {
// Skip NaN or negative values
continue
}
outboundMbps := outbound * 8 / 1000000
// Format time as MM:SS
t := time.Unix(timestamp, 0)
timeStr := fmt.Sprintf("%02d:%02d", t.Minute(), t.Second())
points = append(points, NetworkDataPoint{
Time: timeStr,
Inbound: inboundMbps,
Outbound: outboundMbps,
})
}
return points, nil
}
// NetworkDataPoint represents a single data point for graphing
type NetworkDataPoint struct {
Time string `json:"time"`
Inbound float64 `json:"inbound"` // Mbps
Outbound float64 `json:"outbound"` // Mbps
}
// StartCollector starts a background goroutine to periodically collect and update RRD
func (r *RRDService) StartCollector(ctx context.Context, interval time.Duration) error {
// Initialize RRD if needed
if err := r.InitializeRRD(ctx); err != nil {
return fmt.Errorf("failed to initialize RRD: %w", err)
}
go func() {
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
// Get current stats
stats, err := r.GetNetworkStats(ctx, r.interfaceName)
if err != nil {
r.logger.Warn("Failed to get network stats", "error", err)
continue
}
// Update RRD with cumulative byte counts
// RRD COUNTER type will automatically calculate rate
if err := r.UpdateRRD(ctx, stats); err != nil {
r.logger.Warn("Failed to update RRD", "error", err)
}
}
}
}()
return nil
}

752
backend/internal/system/service.go
View File

@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"fmt"
"os"
"os/exec"
"strings"
"time"
@@ -11,18 +12,98 @@ import (
"github.com/atlasos/calypso/internal/common/logger"
)
// NTPSettings represents NTP configuration
type NTPSettings struct {
Timezone string `json:"timezone"`
NTPServers []string `json:"ntp_servers"`
}
// Service handles system management operations
type Service struct {
logger *logger.Logger
logger *logger.Logger
rrdService *RRDService
}
// detectPrimaryInterface detects the primary network interface (first non-loopback with IP)
func detectPrimaryInterface(ctx context.Context) string {
// Try to get default route interface
cmd := exec.CommandContext(ctx, "ip", "route", "show", "default")
output, err := cmd.Output()
if err == nil {
lines := strings.Split(string(output), "\n")
for _, line := range lines {
if strings.Contains(line, "dev ") {
parts := strings.Fields(line)
for i, part := range parts {
if part == "dev" && i+1 < len(parts) {
iface := parts[i+1]
if iface != "lo" {
return iface
}
}
}
}
}
}
// Fallback: get first non-loopback interface with IP
cmd = exec.CommandContext(ctx, "ip", "-4", "addr", "show")
output, err = cmd.Output()
if err == nil {
lines := strings.Split(string(output), "\n")
for _, line := range lines {
line = strings.TrimSpace(line)
// Look for interface name line (e.g., "2: ens18: <BROADCAST...")
if len(line) > 0 && line[0] >= '0' && line[0] <= '9' && strings.Contains(line, ":") {
parts := strings.Fields(line)
if len(parts) >= 2 {
iface := strings.TrimSuffix(parts[1], ":")
if iface != "" && iface != "lo" {
// Check if this interface has an IP (next lines will have "inet")
// For simplicity, return first non-loopback interface
return iface
}
}
}
}
}
// Final fallback
return "eth0"
}
// NewService creates a new system service
func NewService(log *logger.Logger) *Service {
// Initialize RRD service for network monitoring
rrdDir := "/var/lib/calypso/rrd"
// Auto-detect primary interface
ctx := context.Background()
interfaceName := detectPrimaryInterface(ctx)
log.Info("Detected primary network interface", "interface", interfaceName)
rrdService := NewRRDService(log, rrdDir, interfaceName)
return &Service{
logger: log,
logger: log,
rrdService: rrdService,
}
}
// StartNetworkMonitoring starts the RRD collector for network monitoring
func (s *Service) StartNetworkMonitoring(ctx context.Context) error {
return s.rrdService.StartCollector(ctx, 10*time.Second)
}
// GetNetworkThroughput fetches network throughput data from RRD
func (s *Service) GetNetworkThroughput(ctx context.Context, duration time.Duration) ([]NetworkDataPoint, error) {
endTime := time.Now()
startTime := endTime.Add(-duration)
// Use 10 second resolution for recent data
return s.rrdService.FetchRRDData(ctx, startTime, endTime, "10")
}
// ServiceStatus represents a systemd service status
type ServiceStatus struct {
Name string `json:"name"`
@@ -35,31 +116,37 @@ type ServiceStatus struct {
// GetServiceStatus retrieves the status of a systemd service
func (s *Service) GetServiceStatus(ctx context.Context, serviceName string) (*ServiceStatus, error) {
cmd := exec.CommandContext(ctx, "systemctl", "show", serviceName,
"--property=ActiveState,SubState,LoadState,Description,ActiveEnterTimestamp",
"--value", "--no-pager")
output, err := cmd.Output()
if err != nil {
return nil, fmt.Errorf("failed to get service status: %w", err)
}
lines := strings.Split(strings.TrimSpace(string(output)), "\n")
if len(lines) < 4 {
return nil, fmt.Errorf("invalid service status output")
}
status := &ServiceStatus{
Name: serviceName,
ActiveState: strings.TrimSpace(lines[0]),
SubState: strings.TrimSpace(lines[1]),
LoadState: strings.TrimSpace(lines[2]),
Description: strings.TrimSpace(lines[3]),
Name: serviceName,
}
// Parse timestamp if available
if len(lines) > 4 && lines[4] != "" {
if t, err := time.Parse("Mon 2006-01-02 15:04:05 MST", strings.TrimSpace(lines[4])); err == nil {
status.Since = t
// Get each property individually to ensure correct parsing
properties := map[string]*string{
"ActiveState": &status.ActiveState,
"SubState": &status.SubState,
"LoadState": &status.LoadState,
"Description": &status.Description,
}
for prop, target := range properties {
cmd := exec.CommandContext(ctx, "systemctl", "show", serviceName, "--property", prop, "--value", "--no-pager")
output, err := cmd.Output()
if err != nil {
s.logger.Warn("Failed to get property", "service", serviceName, "property", prop, "error", err)
continue
}
*target = strings.TrimSpace(string(output))
}
// Get timestamp if available
cmd := exec.CommandContext(ctx, "systemctl", "show", serviceName, "--property", "ActiveEnterTimestamp", "--value", "--no-pager")
output, err := cmd.Output()
if err == nil {
timestamp := strings.TrimSpace(string(output))
if timestamp != "" {
if t, err := time.Parse("Mon 2006-01-02 15:04:05 MST", timestamp); err == nil {
status.Since = t
}
}
}
@@ -69,10 +156,15 @@ func (s *Service) GetServiceStatus(ctx context.Context, serviceName string) (*Se
// ListServices lists all Calypso-related services
func (s *Service) ListServices(ctx context.Context) ([]ServiceStatus, error) {
services := []string{
"ssh",
"sshd",
"smbd",
"iscsi-scst",
"nfs-server",
"nfs",
"mhvtl",
"calypso-api",
"scst",
"iscsi-scst",
"mhvtl",
"postgresql",
}
@@ -128,6 +220,108 @@ func (s *Service) GetJournalLogs(ctx context.Context, serviceName string, lines
return logs, nil
}
// SystemLogEntry represents a parsed system log entry
type SystemLogEntry struct {
Time string `json:"time"`
Level string `json:"level"`
Source string `json:"source"`
Message string `json:"message"`
}
// GetSystemLogs retrieves recent system logs from journalctl
func (s *Service) GetSystemLogs(ctx context.Context, limit int) ([]SystemLogEntry, error) {
if limit <= 0 || limit > 100 {
limit = 30 // Default to 30 logs
}
cmd := exec.CommandContext(ctx, "journalctl",
"-n", fmt.Sprintf("%d", limit),
"-o", "json",
"--no-pager",
"--since", "1 hour ago") // Only get logs from last hour
output, err := cmd.Output()
if err != nil {
return nil, fmt.Errorf("failed to get system logs: %w", err)
}
var logs []SystemLogEntry
linesOutput := strings.Split(strings.TrimSpace(string(output)), "\n")
for _, line := range linesOutput {
if line == "" {
continue
}
var logEntry map[string]interface{}
if err := json.Unmarshal([]byte(line), &logEntry); err != nil {
continue
}
// Parse timestamp (__REALTIME_TIMESTAMP is in microseconds)
var timeStr string
if timestamp, ok := logEntry["__REALTIME_TIMESTAMP"].(float64); ok {
// Convert microseconds to nanoseconds for time.Unix (1 microsecond = 1000 nanoseconds)
t := time.Unix(0, int64(timestamp)*1000)
timeStr = t.Format("15:04:05")
} else if timestamp, ok := logEntry["_SOURCE_REALTIME_TIMESTAMP"].(float64); ok {
t := time.Unix(0, int64(timestamp)*1000)
timeStr = t.Format("15:04:05")
} else {
timeStr = time.Now().Format("15:04:05")
}
// Parse log level (priority)
level := "INFO"
if priority, ok := logEntry["PRIORITY"].(float64); ok {
switch int(priority) {
case 0: // emerg
level = "EMERG"
case 1, 2, 3: // alert, crit, err
level = "ERROR"
case 4: // warning
level = "WARN"
case 5: // notice
level = "NOTICE"
case 6: // info
level = "INFO"
case 7: // debug
level = "DEBUG"
}
}
// Parse source (systemd unit or syslog identifier)
source := "system"
if unit, ok := logEntry["_SYSTEMD_UNIT"].(string); ok && unit != "" {
// Remove .service suffix if present
source = strings.TrimSuffix(unit, ".service")
} else if ident, ok := logEntry["SYSLOG_IDENTIFIER"].(string); ok && ident != "" {
source = ident
} else if comm, ok := logEntry["_COMM"].(string); ok && comm != "" {
source = comm
}
// Parse message
message := ""
if msg, ok := logEntry["MESSAGE"].(string); ok {
message = msg
}
if message != "" {
logs = append(logs, SystemLogEntry{
Time: timeStr,
Level: level,
Source: source,
Message: message,
})
}
}
// Reverse to get newest first
for i, j := 0, len(logs)-1; i < j; i, j = i+1, j-1 {
logs[i], logs[j] = logs[j], logs[i]
}
return logs, nil
}
// GenerateSupportBundle generates a diagnostic support bundle
func (s *Service) GenerateSupportBundle(ctx context.Context, outputPath string) error {
// Create bundle directory
@@ -183,6 +377,9 @@ type NetworkInterface struct {
Status string `json:"status"` // "Connected" or "Down"
Speed string `json:"speed"` // e.g., "10 Gbps", "1 Gbps"
Role string `json:"role"` // "Management", "ISCSI", or empty
Gateway string `json:"gateway,omitempty"`
DNS1 string `json:"dns1,omitempty"`
DNS2 string `json:"dns2,omitempty"`
}
// ListNetworkInterfaces lists all network interfaces
@@ -297,6 +494,103 @@ func (s *Service) ListNetworkInterfaces(ctx context.Context) ([]NetworkInterface
}
}
// Get default gateway for each interface
cmd = exec.CommandContext(ctx, "ip", "route", "show")
output, err = cmd.Output()
if err == nil {
lines = strings.Split(string(output), "\n")
for _, line := range lines {
line = strings.TrimSpace(line)
if line == "" {
continue
}
// Parse default route: "default via 10.10.14.1 dev ens18"
if strings.HasPrefix(line, "default via ") {
parts := strings.Fields(line)
// Find "via" and "dev" in the parts
var gateway string
var ifaceName string
for i, part := range parts {
if part == "via" && i+1 < len(parts) {
gateway = parts[i+1]
}
if part == "dev" && i+1 < len(parts) {
ifaceName = parts[i+1]
}
}
if gateway != "" && ifaceName != "" {
if iface, exists := interfaceMap[ifaceName]; exists {
iface.Gateway = gateway
s.logger.Info("Set default gateway for interface", "name", ifaceName, "gateway", gateway)
}
}
} else if strings.Contains(line, " via ") && strings.Contains(line, " dev ") {
// Parse network route: "10.10.14.0/24 via 10.10.14.1 dev ens18"
// Or: "192.168.1.0/24 via 192.168.1.1 dev eth0"
parts := strings.Fields(line)
var gateway string
var ifaceName string
for i, part := range parts {
if part == "via" && i+1 < len(parts) {
gateway = parts[i+1]
}
if part == "dev" && i+1 < len(parts) {
ifaceName = parts[i+1]
}
}
// Only set gateway if it's not already set (prefer default route)
if gateway != "" && ifaceName != "" {
if iface, exists := interfaceMap[ifaceName]; exists {
if iface.Gateway == "" {
iface.Gateway = gateway
s.logger.Info("Set gateway from network route for interface", "name", ifaceName, "gateway", gateway)
}
}
}
}
}
} else {
s.logger.Warn("Failed to get routes", "error", err)
}
// Get DNS servers from systemd-resolved or /etc/resolv.conf
// Try systemd-resolved first
cmd = exec.CommandContext(ctx, "systemd-resolve", "--status")
output, err = cmd.Output()
dnsServers := []string{}
if err == nil {
// Parse DNS from systemd-resolve output
lines = strings.Split(string(output), "\n")
for _, line := range lines {
line = strings.TrimSpace(line)
if strings.HasPrefix(line, "DNS Servers:") {
// Format: "DNS Servers: 8.8.8.8 8.8.4.4"
parts := strings.Fields(line)
if len(parts) >= 3 {
dnsServers = parts[2:]
}
break
}
}
} else {
// Fallback to /etc/resolv.conf
data, err := os.ReadFile("/etc/resolv.conf")
if err == nil {
lines = strings.Split(string(data), "\n")
for _, line := range lines {
line = strings.TrimSpace(line)
if strings.HasPrefix(line, "nameserver ") {
dns := strings.TrimPrefix(line, "nameserver ")
dns = strings.TrimSpace(dns)
if dns != "" {
dnsServers = append(dnsServers, dns)
}
}
}
}
}
// Convert map to slice
var interfaces []NetworkInterface
s.logger.Debug("Converting interface map to slice", "map_size", len(interfaceMap))
@@ -319,6 +613,14 @@ func (s *Service) ListNetworkInterfaces(ctx context.Context) ([]NetworkInterface
}
}
// Set DNS servers (use first two if available)
if len(dnsServers) > 0 {
iface.DNS1 = dnsServers[0]
}
if len(dnsServers) > 1 {
iface.DNS2 = dnsServers[1]
}
// Determine role based on interface name or IP (simple heuristic)
// You can enhance this with configuration file or database lookup
if strings.Contains(iface.Name, "eth") || strings.Contains(iface.Name, "ens") {
@@ -345,3 +647,401 @@ func (s *Service) ListNetworkInterfaces(ctx context.Context) ([]NetworkInterface
s.logger.Info("Listed network interfaces", "count", len(interfaces))
return interfaces, nil
}
// GetManagementIPAddress returns the IP address of the management interface
func (s *Service) GetManagementIPAddress(ctx context.Context) (string, error) {
interfaces, err := s.ListNetworkInterfaces(ctx)
if err != nil {
return "", fmt.Errorf("failed to list network interfaces: %w", err)
}
// First, try to find interface with Role "Management"
for _, iface := range interfaces {
if iface.Role == "Management" && iface.IPAddress != "" && iface.Status == "Connected" {
s.logger.Info("Found management interface", "interface", iface.Name, "ip", iface.IPAddress)
return iface.IPAddress, nil
}
}
// Fallback: use interface with default route (primary interface)
for _, iface := range interfaces {
if iface.Gateway != "" && iface.IPAddress != "" && iface.Status == "Connected" {
s.logger.Info("Using primary interface as management", "interface", iface.Name, "ip", iface.IPAddress)
return iface.IPAddress, nil
}
}
// Final fallback: use first connected interface with IP
for _, iface := range interfaces {
if iface.IPAddress != "" && iface.Status == "Connected" && iface.Name != "lo" {
s.logger.Info("Using first connected interface as management", "interface", iface.Name, "ip", iface.IPAddress)
return iface.IPAddress, nil
}
}
return "", fmt.Errorf("no management interface found")
}
// UpdateNetworkInterfaceRequest represents the request to update a network interface
type UpdateNetworkInterfaceRequest struct {
IPAddress string `json:"ip_address"`
Subnet string `json:"subnet"`
Gateway string `json:"gateway,omitempty"`
DNS1 string `json:"dns1,omitempty"`
DNS2 string `json:"dns2,omitempty"`
Role string `json:"role,omitempty"`
}
// UpdateNetworkInterface updates network interface configuration
func (s *Service) UpdateNetworkInterface(ctx context.Context, ifaceName string, req UpdateNetworkInterfaceRequest) (*NetworkInterface, error) {
// Validate interface exists
cmd := exec.CommandContext(ctx, "ip", "link", "show", ifaceName)
if err := cmd.Run(); err != nil {
return nil, fmt.Errorf("interface %s not found: %w", ifaceName, err)
}
// Remove existing IP address if any
cmd = exec.CommandContext(ctx, "ip", "addr", "flush", "dev", ifaceName)
cmd.Run() // Ignore error, interface might not have IP
// Set new IP address and subnet
ipWithSubnet := fmt.Sprintf("%s/%s", req.IPAddress, req.Subnet)
cmd = exec.CommandContext(ctx, "ip", "addr", "add", ipWithSubnet, "dev", ifaceName)
output, err := cmd.CombinedOutput()
if err != nil {
s.logger.Error("Failed to set IP address", "interface", ifaceName, "error", err, "output", string(output))
return nil, fmt.Errorf("failed to set IP address: %w", err)
}
// Remove existing default route if any
cmd = exec.CommandContext(ctx, "ip", "route", "del", "default")
cmd.Run() // Ignore error, might not exist
// Set gateway if provided
if req.Gateway != "" {
cmd = exec.CommandContext(ctx, "ip", "route", "add", "default", "via", req.Gateway, "dev", ifaceName)
output, err = cmd.CombinedOutput()
if err != nil {
s.logger.Error("Failed to set gateway", "interface", ifaceName, "error", err, "output", string(output))
return nil, fmt.Errorf("failed to set gateway: %w", err)
}
}
// Update DNS in systemd-resolved or /etc/resolv.conf
if req.DNS1 != "" || req.DNS2 != "" {
// Try using systemd-resolve first
cmd = exec.CommandContext(ctx, "systemd-resolve", "--status")
if cmd.Run() == nil {
// systemd-resolve is available, use it
dnsServers := []string{}
if req.DNS1 != "" {
dnsServers = append(dnsServers, req.DNS1)
}
if req.DNS2 != "" {
dnsServers = append(dnsServers, req.DNS2)
}
if len(dnsServers) > 0 {
// Use resolvectl to set DNS (newer systemd)
cmd = exec.CommandContext(ctx, "resolvectl", "dns", ifaceName, strings.Join(dnsServers, " "))
if cmd.Run() != nil {
// Fallback to systemd-resolve
cmd = exec.CommandContext(ctx, "systemd-resolve", "--interface", ifaceName, "--set-dns", strings.Join(dnsServers, " "))
output, err = cmd.CombinedOutput()
if err != nil {
s.logger.Warn("Failed to set DNS via systemd-resolve", "error", err, "output", string(output))
}
}
}
} else {
// Fallback: update /etc/resolv.conf
resolvContent := "# Generated by Calypso\n"
if req.DNS1 != "" {
resolvContent += fmt.Sprintf("nameserver %s\n", req.DNS1)
}
if req.DNS2 != "" {
resolvContent += fmt.Sprintf("nameserver %s\n", req.DNS2)
}
tmpPath := "/tmp/resolv.conf." + fmt.Sprintf("%d", time.Now().Unix())
if err := os.WriteFile(tmpPath, []byte(resolvContent), 0644); err != nil {
s.logger.Warn("Failed to write temporary resolv.conf", "error", err)
} else {
cmd = exec.CommandContext(ctx, "sh", "-c", fmt.Sprintf("mv %s /etc/resolv.conf", tmpPath))
output, err = cmd.CombinedOutput()
if err != nil {
s.logger.Warn("Failed to update /etc/resolv.conf", "error", err, "output", string(output))
}
}
}
}
// Bring interface up
cmd = exec.CommandContext(ctx, "ip", "link", "set", ifaceName, "up")
output, err = cmd.CombinedOutput()
if err != nil {
s.logger.Warn("Failed to bring interface up", "interface", ifaceName, "error", err, "output", string(output))
}
// Return updated interface
updatedIface := &NetworkInterface{
Name: ifaceName,
IPAddress: req.IPAddress,
Subnet: req.Subnet,
Gateway: req.Gateway,
DNS1: req.DNS1,
DNS2: req.DNS2,
Role: req.Role,
Status: "Connected",
Speed: "Unknown", // Will be updated on next list
}
s.logger.Info("Updated network interface", "interface", ifaceName, "ip", req.IPAddress, "subnet", req.Subnet)
return updatedIface, nil
}
// SaveNTPSettings saves NTP configuration to the OS
func (s *Service) SaveNTPSettings(ctx context.Context, settings NTPSettings) error {
// Set timezone using timedatectl
if settings.Timezone != "" {
cmd := exec.CommandContext(ctx, "timedatectl", "set-timezone", settings.Timezone)
output, err := cmd.CombinedOutput()
if err != nil {
s.logger.Error("Failed to set timezone", "timezone", settings.Timezone, "error", err, "output", string(output))
return fmt.Errorf("failed to set timezone: %w", err)
}
s.logger.Info("Timezone set", "timezone", settings.Timezone)
}
// Configure NTP servers in systemd-timesyncd
if len(settings.NTPServers) > 0 {
configPath := "/etc/systemd/timesyncd.conf"
// Build config content
configContent := "[Time]\n"
configContent += "NTP="
for i, server := range settings.NTPServers {
if i > 0 {
configContent += " "
}
configContent += server
}
configContent += "\n"
// Write to temporary file first, then move to final location (requires root)
tmpPath := "/tmp/timesyncd.conf." + fmt.Sprintf("%d", time.Now().Unix())
if err := os.WriteFile(tmpPath, []byte(configContent), 0644); err != nil {
s.logger.Error("Failed to write temporary NTP config", "error", err)
return fmt.Errorf("failed to write temporary NTP configuration: %w", err)
}
// Move to final location using sudo (requires root privileges)
cmd := exec.CommandContext(ctx, "sh", "-c", fmt.Sprintf("mv %s %s", tmpPath, configPath))
output, err := cmd.CombinedOutput()
if err != nil {
s.logger.Error("Failed to move NTP config", "error", err, "output", string(output))
os.Remove(tmpPath) // Clean up temp file
return fmt.Errorf("failed to move NTP configuration: %w", err)
}
// Restart systemd-timesyncd to apply changes
cmd = exec.CommandContext(ctx, "systemctl", "restart", "systemd-timesyncd")
output, err = cmd.CombinedOutput()
if err != nil {
s.logger.Error("Failed to restart systemd-timesyncd", "error", err, "output", string(output))
return fmt.Errorf("failed to restart systemd-timesyncd: %w", err)
}
s.logger.Info("NTP servers configured", "servers", settings.NTPServers)
}
return nil
}
// GetNTPSettings retrieves current NTP configuration from the OS
func (s *Service) GetNTPSettings(ctx context.Context) (*NTPSettings, error) {
settings := &NTPSettings{
NTPServers: []string{},
}
// Get current timezone using timedatectl
cmd := exec.CommandContext(ctx, "timedatectl", "show", "--property=Timezone", "--value")
output, err := cmd.Output()
if err != nil {
s.logger.Warn("Failed to get timezone", "error", err)
settings.Timezone = "Etc/UTC" // Default fallback
} else {
settings.Timezone = strings.TrimSpace(string(output))
if settings.Timezone == "" {
settings.Timezone = "Etc/UTC"
}
}
// Read NTP servers from systemd-timesyncd config
configPath := "/etc/systemd/timesyncd.conf"
data, err := os.ReadFile(configPath)
if err != nil {
s.logger.Warn("Failed to read NTP config", "error", err)
// Default NTP servers if config file doesn't exist
settings.NTPServers = []string{"pool.ntp.org", "time.google.com"}
} else {
// Parse NTP servers from config file
lines := strings.Split(string(data), "\n")
for _, line := range lines {
line = strings.TrimSpace(line)
if strings.HasPrefix(line, "NTP=") {
ntpLine := strings.TrimPrefix(line, "NTP=")
if ntpLine != "" {
servers := strings.Fields(ntpLine)
settings.NTPServers = servers
break
}
}
}
// If no NTP servers found in config, use defaults
if len(settings.NTPServers) == 0 {
settings.NTPServers = []string{"pool.ntp.org", "time.google.com"}
}
}
return settings, nil
}
// ExecuteCommand executes a shell command and returns the output
// service parameter is optional and can be: system, scst, storage, backup, tape
func (s *Service) ExecuteCommand(ctx context.Context, command string, service string) (string, error) {
// Sanitize command - basic security check
command = strings.TrimSpace(command)
if command == "" {
return "", fmt.Errorf("command cannot be empty")
}
// Block dangerous commands that could harm the system
dangerousCommands := []string{
"rm -rf /",
"dd if=",
":(){ :|:& };:",
"mkfs",
"fdisk",
"parted",
"format",
"> /dev/sd",
"mkfs.ext",
"mkfs.xfs",
"mkfs.btrfs",
"wipefs",
}
commandLower := strings.ToLower(command)
for _, dangerous := range dangerousCommands {
if strings.Contains(commandLower, dangerous) {
return "", fmt.Errorf("command blocked for security reasons")
}
}
// Service-specific command handling
switch service {
case "scst":
// Allow SCST admin commands
if strings.HasPrefix(command, "scstadmin") {
// SCST commands are safe
break
}
case "backup":
// Allow bconsole commands
if strings.HasPrefix(command, "bconsole") {
// Backup console commands are safe
break
}
case "storage":
// Allow ZFS and storage commands
if strings.HasPrefix(command, "zfs") || strings.HasPrefix(command, "zpool") || strings.HasPrefix(command, "lsblk") {
// Storage commands are safe
break
}
case "tape":
// Allow tape library commands
if strings.HasPrefix(command, "mtx") || strings.HasPrefix(command, "lsscsi") || strings.HasPrefix(command, "sg_") {
// Tape commands are safe
break
}
}
// Execute command with timeout (30 seconds)
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
// Check if command already has sudo (reuse commandLower from above)
hasSudo := strings.HasPrefix(commandLower, "sudo ")
// Determine if command needs sudo based on service and command type
needsSudo := false
if !hasSudo {
// Commands that typically need sudo
sudoCommands := []string{
"scstadmin",
"systemctl",
"zfs",
"zpool",
"mount",
"umount",
"ip link",
"ip addr",
"iptables",
"journalctl",
}
for _, sudoCmd := range sudoCommands {
if strings.HasPrefix(commandLower, sudoCmd) {
needsSudo = true
break
}
}
// Service-specific sudo requirements
switch service {
case "scst":
// All SCST admin commands need sudo
if strings.HasPrefix(commandLower, "scstadmin") {
needsSudo = true
}
case "storage":
// ZFS commands typically need sudo
if strings.HasPrefix(commandLower, "zfs") || strings.HasPrefix(commandLower, "zpool") {
needsSudo = true
}
case "system":
// System commands like systemctl need sudo
if strings.HasPrefix(commandLower, "systemctl") || strings.HasPrefix(commandLower, "journalctl") {
needsSudo = true
}
}
}
// Build command with or without sudo
var cmd *exec.Cmd
if needsSudo && !hasSudo {
// Use sudo for privileged commands (if not already present)
cmd = exec.CommandContext(ctx, "sudo", "sh", "-c", command)
} else {
// Regular command (or already has sudo)
cmd = exec.CommandContext(ctx, "sh", "-c", command)
}
cmd.Env = append(os.Environ(), "TERM=xterm-256color")
cmd.Env = append(os.Environ(), "TERM=xterm-256color")
output, err := cmd.CombinedOutput()
if err != nil {
// Return output even if there's an error (some commands return non-zero exit codes)
outputStr := string(output)
if len(outputStr) > 0 {
return outputStr, nil
}
return "", fmt.Errorf("command execution failed: %w", err)
}
return string(output), nil
}

328
backend/internal/system/terminal.go Normal file
View File

@@ -0,0 +1,328 @@
package system
import (
"encoding/json"
"io"
"net/http"
"os"
"os/exec"
"os/user"
"sync"
"syscall"
"time"
"github.com/atlasos/calypso/internal/common/logger"
"github.com/creack/pty"
"github.com/gin-gonic/gin"
"github.com/gorilla/websocket"
)
const (
// WebSocket timeouts
writeWait = 10 * time.Second
pongWait = 60 * time.Second
pingPeriod = (pongWait * 9) / 10
)
var upgrader = websocket.Upgrader{
ReadBufferSize: 4096,
WriteBufferSize: 4096,
CheckOrigin: func(r *http.Request) bool {
// Allow all origins - in production, validate against allowed domains
return true
},
}
// TerminalSession manages a single terminal session
type TerminalSession struct {
conn *websocket.Conn
pty *os.File
cmd *exec.Cmd
logger *logger.Logger
mu sync.RWMutex
closed bool
username string
done chan struct{}
}
// HandleTerminalWebSocket handles WebSocket connection for terminal
func HandleTerminalWebSocket(c *gin.Context, log *logger.Logger) {
// Verify authentication
userID, exists := c.Get("user_id")
if !exists {
log.Warn("Terminal WebSocket: unauthorized access", "ip", c.ClientIP())
c.JSON(http.StatusUnauthorized, gin.H{"error": "unauthorized"})
return
}
username, _ := c.Get("username")
if username == nil {
username = userID
}
log.Info("Terminal WebSocket: connection attempt", "username", username, "ip", c.ClientIP())
// Upgrade connection
conn, err := upgrader.Upgrade(c.Writer, c.Request, nil)
if err != nil {
log.Error("Terminal WebSocket: upgrade failed", "error", err)
return
}
log.Info("Terminal WebSocket: connection upgraded", "username", username)
// Create session
session := &TerminalSession{
conn: conn,
logger: log,
username: username.(string),
done: make(chan struct{}),
}
// Start terminal
if err := session.startPTY(); err != nil {
log.Error("Terminal WebSocket: failed to start PTY", "error", err, "username", username)
session.sendError(err.Error())
session.close()
return
}
// Handle messages and PTY output
go session.handleRead()
go session.handleWrite()
}
// startPTY starts the PTY session
func (s *TerminalSession) startPTY() error {
// Get user info
currentUser, err := user.Lookup(s.username)
if err != nil {
// Fallback to current user
currentUser, err = user.Current()
if err != nil {
return err
}
}
// Determine shell
shell := os.Getenv("SHELL")
if shell == "" {
shell = "/bin/bash"
}
// Create command
s.cmd = exec.Command(shell)
s.cmd.Env = append(os.Environ(),
"TERM=xterm-256color",
"HOME="+currentUser.HomeDir,
"USER="+currentUser.Username,
"USERNAME="+currentUser.Username,
)
s.cmd.Dir = currentUser.HomeDir
// Start PTY
ptyFile, err := pty.Start(s.cmd)
if err != nil {
return err
}
s.pty = ptyFile
// Set initial size
pty.Setsize(ptyFile, &pty.Winsize{
Rows: 24,
Cols: 80,
})
return nil
}
// handleRead handles incoming WebSocket messages
func (s *TerminalSession) handleRead() {
defer s.close()
// Set read deadline and pong handler
s.conn.SetReadDeadline(time.Now().Add(pongWait))
s.conn.SetPongHandler(func(string) error {
s.conn.SetReadDeadline(time.Now().Add(pongWait))
return nil
})
for {
select {
case <-s.done:
return
default:
messageType, data, err := s.conn.ReadMessage()
if err != nil {
if websocket.IsUnexpectedCloseError(err, websocket.CloseGoingAway, websocket.CloseAbnormalClosure) {
s.logger.Error("Terminal WebSocket: read error", "error", err)
}
return
}
// Handle binary messages (raw input)
if messageType == websocket.BinaryMessage {
s.writeToPTY(data)
continue
}
// Handle text messages (JSON commands)
if messageType == websocket.TextMessage {
var msg map[string]interface{}
if err := json.Unmarshal(data, &msg); err != nil {
continue
}
switch msg["type"] {
case "input":
if data, ok := msg["data"].(string); ok {
s.writeToPTY([]byte(data))
}
case "resize":
if cols, ok1 := msg["cols"].(float64); ok1 {
if rows, ok2 := msg["rows"].(float64); ok2 {
s.resizePTY(uint16(cols), uint16(rows))
}
}
case "ping":
s.writeWS(websocket.TextMessage, []byte(`{"type":"pong"}`))
}
}
}
}
}
// handleWrite handles PTY output to WebSocket
func (s *TerminalSession) handleWrite() {
defer s.close()
ticker := time.NewTicker(pingPeriod)
defer ticker.Stop()
// Read from PTY and write to WebSocket
buffer := make([]byte, 4096)
for {
select {
case <-s.done:
return
case <-ticker.C:
// Send ping
if err := s.writeWS(websocket.PingMessage, nil); err != nil {
return
}
default:
// Read from PTY
if s.pty != nil {
n, err := s.pty.Read(buffer)
if err != nil {
if err != io.EOF {
s.logger.Error("Terminal WebSocket: PTY read error", "error", err)
}
return
}
if n > 0 {
// Write binary data to WebSocket
if err := s.writeWS(websocket.BinaryMessage, buffer[:n]); err != nil {
return
}
}
}
}
}
}
// writeToPTY writes data to PTY
func (s *TerminalSession) writeToPTY(data []byte) {
s.mu.RLock()
closed := s.closed
pty := s.pty
s.mu.RUnlock()
if closed || pty == nil {
return
}
if _, err := pty.Write(data); err != nil {
s.logger.Error("Terminal WebSocket: PTY write error", "error", err)
}
}
// resizePTY resizes the PTY
func (s *TerminalSession) resizePTY(cols, rows uint16) {
s.mu.RLock()
closed := s.closed
ptyFile := s.pty
s.mu.RUnlock()
if closed || ptyFile == nil {
return
}
// Use pty.Setsize from package, not method from variable
pty.Setsize(ptyFile, &pty.Winsize{
Cols: cols,
Rows: rows,
})
}
// writeWS writes message to WebSocket
func (s *TerminalSession) writeWS(messageType int, data []byte) error {
s.mu.RLock()
closed := s.closed
conn := s.conn
s.mu.RUnlock()
if closed || conn == nil {
return io.ErrClosedPipe
}
conn.SetWriteDeadline(time.Now().Add(writeWait))
return conn.WriteMessage(messageType, data)
}
// sendError sends error message
func (s *TerminalSession) sendError(errMsg string) {
msg := map[string]interface{}{
"type": "error",
"error": errMsg,
}
data, _ := json.Marshal(msg)
s.writeWS(websocket.TextMessage, data)
}
// close closes the terminal session
func (s *TerminalSession) close() {
s.mu.Lock()
defer s.mu.Unlock()
if s.closed {
return
}
s.closed = true
close(s.done)
// Close PTY
if s.pty != nil {
s.pty.Close()
}
// Kill process
if s.cmd != nil && s.cmd.Process != nil {
s.cmd.Process.Signal(syscall.SIGTERM)
time.Sleep(100 * time.Millisecond)
if s.cmd.ProcessState == nil || !s.cmd.ProcessState.Exited() {
s.cmd.Process.Kill()
}
}
// Close WebSocket
if s.conn != nil {
s.conn.Close()
}
s.logger.Info("Terminal WebSocket: session closed", "username", s.username)
}

1
bacula-config Symbolic link
View File

@@ -0,0 +1 @@
/opt/calypso/conf/bacula

BIN
dist/airgap/calypso-appliance-1.0.0-airgap.tar.gz vendored Normal file
View File

Binary file not shown.

1
dist/airgap/calypso-appliance-1.0.0-airgap.tar.gz.sha256 vendored Normal file
View File

@@ -0,0 +1 @@
a4040ea4ab486d4671772e4b832b4be5741bf58764fc731873b585df1d805e66 calypso-appliance-1.0.0-airgap.tar.gz

59
dist/airgap/calypso-appliance-1.0.0-airgap/README.md vendored Normal file
View File

@@ -0,0 +1,59 @@
# Calypso Appliance - Airgap Installation Bundle
**Version:** 1.0.0
**Target OS:** Ubuntu Server 24.04 LTS
**Installation Type:** Offline/Airgap
## Bundle Contents
- **binaries/**: Pre-built Calypso binaries
- **frontend/**: Built frontend assets
- **packages/debs/**: All required DEB packages with dependencies (59 packages + ~200-300 dependencies)
- **scripts/**: Installation scripts
- **services/**: Systemd service files
- **migrations/**: Database migration files
- **configs/**: Configuration templates
- **third_party/**: Optional third-party binaries (Go, Node.js)
## Installation
### Prerequisites
- Ubuntu Server 24.04 LTS
- Root or sudo access
- Minimum 10GB free disk space
- Kernel headers installed (for kernel modules)
### Quick Install
```bash
cd /path/to/bundle
sudo ./install-airgap.sh
```
The installer will:
1. Install all packages from bundled DEB files
2. Install Calypso binaries and frontend
3. Setup database and configuration
4. Install systemd services
### Post-Installation
1. Review configuration: `/etc/calypso/calypso.yaml`
2. Install kernel modules (ZFS, SCST, mhVTL) if needed
3. Start services: `systemctl start calypso-api`
4. Enable services: `systemctl enable calypso-api`
## Verification
Check service status:
```bash
systemctl status calypso-api
curl http://localhost:8080/api/v1/health
```
## Support
For issues, check:
- Installation logs: `/var/log/calypso/install.log`
- Service logs: `journalctl -u calypso-api`

BIN
dist/airgap/calypso-appliance-1.0.0-airgap/binaries/calypso-api vendored Executable file
View File

Binary file not shown.

64
dist/airgap/calypso-appliance-1.0.0-airgap/configs/config.yaml.template vendored Normal file
View File

@@ -0,0 +1,64 @@
# AtlasOS - Calypso API Configuration Template
# This file will be copied to /etc/calypso/config.yaml during installation
# Environment variables from /etc/calypso/secrets.env will be used for sensitive values
server:
port: 8080
host: "0.0.0.0"
read_timeout: 15s
write_timeout: 15s
idle_timeout: 60s
# Response caching configuration
cache:
enabled: true # Enable response caching
default_ttl: 5m # Default cache TTL (5 minutes)
max_age: 300 # Cache-Control max-age in seconds (5 minutes)
database:
host: "localhost"
port: 5432
user: "calypso"
password: "" # Set via CALYPSO_DB_PASSWORD environment variable
database: "calypso"
ssl_mode: "disable"
# Connection pool optimization
max_connections: 25
max_idle_conns: 5
conn_max_lifetime: 5m
auth:
jwt_secret: "" # Set via CALYPSO_JWT_SECRET environment variable
token_lifetime: 24h
argon2:
memory: 65536 # 64 MB
iterations: 3
parallelism: 4
salt_length: 16
key_length: 32
logging:
level: "info" # debug, info, warn, error
format: "json" # json or text
# CORS configuration
cors:
allowed_origins:
- "http://localhost:3000"
- "http://localhost:5173"
allowed_methods:
- "GET"
- "POST"
- "PUT"
- "DELETE"
- "PATCH"
allowed_headers:
- "Content-Type"
- "Authorization"
allow_credentials: true
# Rate limiting
rate_limit:
enabled: true
requests_per_minute: 100
authenticated_requests_per_minute: 200

66
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/assets/chart-vendor-CnBPFalK.js vendored Normal file
View File

File diff suppressed because one or more lines are too long

1
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/assets/chart-vendor-CnBPFalK.js.map vendored Normal file
View File

File diff suppressed because one or more lines are too long

1
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/assets/index-BY0QR2-n.css vendored Normal file
View File

File diff suppressed because one or more lines are too long

207
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/assets/index-ChZMKDzx.js vendored Normal file
View File

File diff suppressed because one or more lines are too long

1
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/assets/index-ChZMKDzx.js.map vendored Normal file
View File

File diff suppressed because one or more lines are too long

6
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/assets/index.es-BqkUSV0N.js vendored Normal file
View File

File diff suppressed because one or more lines are too long

1
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/assets/index.es-BqkUSV0N.js.map vendored Normal file
View File

File diff suppressed because one or more lines are too long

3
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/assets/purify.es-Bzr520pe.js vendored Normal file
View File

File diff suppressed because one or more lines are too long

1
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/assets/purify.es-Bzr520pe.js.map vendored Normal file
View File

File diff suppressed because one or more lines are too long

2
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/assets/query-vendor-BeRQtQod.js vendored Normal file
View File

File diff suppressed because one or more lines are too long

1
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/assets/query-vendor-BeRQtQod.js.map vendored Normal file
View File

File diff suppressed because one or more lines are too long

2
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/assets/react-vendor-Dvs2KPqW.js vendored Normal file
View File

File diff suppressed because one or more lines are too long

1
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/assets/react-vendor-Dvs2KPqW.js.map vendored Normal file
View File

File diff suppressed because one or more lines are too long

2
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/assets/terminal-vendor-l0sNRNKZ.js vendored Normal file
View File

@@ -0,0 +1,2 @@
//# sourceMappingURL=terminal-vendor-l0sNRNKZ.js.map

1
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/assets/terminal-vendor-l0sNRNKZ.js.map vendored Normal file
View File

@@ -0,0 +1 @@
{"version":3,"file":"terminal-vendor-l0sNRNKZ.js","sources":[],"sourcesContent":[],"names":[],"mappings":""}

2
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/assets/ui-vendor-C4xvlrdo.js vendored Normal file
View File

File diff suppressed because one or more lines are too long

1
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/assets/ui-vendor-C4xvlrdo.js.map vendored Normal file
View File

File diff suppressed because one or more lines are too long

7
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/assets/utils-vendor-CJpcXyE0.js vendored Normal file
View File

File diff suppressed because one or more lines are too long

1
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/assets/utils-vendor-CJpcXyE0.js.map vendored Normal file
View File

File diff suppressed because one or more lines are too long

29
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/index.html vendored Normal file
View File

@@ -0,0 +1,29 @@
<!doctype html>
<html class="dark" lang="en">
<head>
<meta charset="UTF-8" />
<link rel="icon" type="image/png" href="/logo.png" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>AtlasOS - Calypso</title>
<!-- Material Symbols -->
<link
href="https://fonts.googleapis.com/css2?family=Material+Symbols+Outlined:wght,FILL@100..700,0..1&display=swap"
rel="stylesheet"
/>
<link
href="https://fonts.googleapis.com/css2?family=Material+Symbols+Rounded:opsz,wght,FILL,GRAD@20..48,100..700,0..1,-50..200&display=swap"
rel="stylesheet"
/>
<script type="module" crossorigin src="/assets/index-ChZMKDzx.js"></script>
<link rel="modulepreload" crossorigin href="/assets/react-vendor-Dvs2KPqW.js">
<link rel="modulepreload" crossorigin href="/assets/query-vendor-BeRQtQod.js">
<link rel="modulepreload" crossorigin href="/assets/utils-vendor-CJpcXyE0.js">
<link rel="modulepreload" crossorigin href="/assets/ui-vendor-C4xvlrdo.js">
<link rel="modulepreload" crossorigin href="/assets/chart-vendor-CnBPFalK.js">
<link rel="stylesheet" crossorigin href="/assets/index-BY0QR2-n.css">
</head>
<body>
<div id="root"></div>
</body>
</html>

BIN
dist/airgap/calypso-appliance-1.0.0-airgap/frontend/logo.png vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 22 KiB

215
dist/airgap/calypso-appliance-1.0.0-airgap/install-airgap.sh vendored Executable file
View File

@@ -0,0 +1,215 @@
#!/bin/bash
#
# Calypso Appliance - Airgap Installer
# Installs Calypso from bundled files without internet connection
#
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
BUNDLE_DIR="$SCRIPT_DIR"
VERSION="${VERSION:-1.0.0}"
# Colors
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m'
log_info() { echo -e "${GREEN}[INFO]${NC} $1"; }
log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; }
log_error() { echo -e "${RED}[ERROR]${NC} $1"; }
log_step() { echo -e "\n${BLUE}==>${NC} $1"; }
# Check root
if [[ $EUID -ne 0 ]]; then
log_error "This script must be run as root (use sudo)"
exit 1
fi
# Check OS
if [ ! -f /etc/os-release ]; then
log_error "Cannot detect OS"
exit 1
fi
. /etc/os-release
if [[ "$ID" != "ubuntu" ]] || [[ "$VERSION_ID" != "24.04" ]]; then
log_warn "This installer is designed for Ubuntu 24.04 LTS"
log_warn "Detected: $ID $VERSION_ID"
read -p "Continue anyway? (y/N) " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
exit 1
fi
fi
log_step "Calypso Appliance - Airgap Installation"
log_info "Bundle directory: $BUNDLE_DIR"
# 1. Install packages from bundle
log_step "Installing system packages from bundle..."
if [ -d "$BUNDLE_DIR/packages/debs" ] && [ "$(ls -A $BUNDLE_DIR/packages/debs/*.deb 2>/dev/null)" ]; then
log_info "Installing packages from bundle..."
# Create local apt repository
DEBS_DIR="$BUNDLE_DIR/packages/debs"
REPO_DIR="/tmp/calypso-local-repo"
mkdir -p "$REPO_DIR"
cp "$DEBS_DIR"/*.deb "$REPO_DIR/" 2>/dev/null || true
# Create Packages.gz if not exists
cd "$REPO_DIR"
if [ ! -f Packages.gz ]; then
dpkg-scanpackages . /dev/null 2>/dev/null | gzip -9c > Packages.gz
fi
# Add to sources.list temporarily
echo "deb [trusted=yes] file://$REPO_DIR ./" > /tmp/calypso-local.list
mv /tmp/calypso-local.list /etc/apt/sources.list.d/calypso-local.list
# Update apt cache
apt-get update -qq
# Install packages from list
PACKAGE_LIST="$BUNDLE_DIR/packages/package-list.txt"
PACKAGES=$(grep -v '^#' "$PACKAGE_LIST" | grep -v '^$' | tr '\n' ' ')
log_info "Installing packages..."
apt-get install -y --allow-unauthenticated $PACKAGES || {
log_warn "Some packages failed to install, trying individual installation..."
while IFS= read -r package || [ -n "$package" ]; do
[[ "$package" =~ ^#.*$ ]] && continue
[[ -z "$package" ]] && continue
apt-get install -y --allow-unauthenticated "$package" 2>/dev/null || log_warn "Failed: $package"
done < "$PACKAGE_LIST"
}
# Cleanup
rm -f /etc/apt/sources.list.d/calypso-local.list
apt-get update -qq
log_info "✓ Packages installed from bundle"
else
log_warn "Package DEB files not found in bundle"
log_warn "You need to install packages manually"
log_info "Package list: $BUNDLE_DIR/packages/package-list.txt"
read -p "Install packages from system repositories? (y/N) " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
apt-get update
xargs -a "$BUNDLE_DIR/packages/package-list.txt" -r apt-get install -y
fi
fi
# 2. Install Go from bundle (if available)
if [ -f "$BUNDLE_DIR/third_party/go.tar.gz" ]; then
log_step "Installing Go from bundle..."
rm -rf /usr/local/go
tar -C /usr/local -xzf "$BUNDLE_DIR/third_party/go.tar.gz"
export PATH=$PATH:/usr/local/go/bin
if ! grep -q "/usr/local/go/bin" /etc/profile; then
echo 'export PATH=$PATH:/usr/local/go/bin' >> /etc/profile
fi
log_info "✓ Go installed"
fi
# 3. Install Node.js from bundle (if available)
if [ -f "$BUNDLE_DIR/third_party/nodejs.tar.xz" ]; then
log_step "Installing Node.js from bundle..."
rm -rf /usr/local/node
mkdir -p /usr/local/node
tar -C /usr/local/node -xJf "$BUNDLE_DIR/third_party/nodejs.tar.xz" --strip-components=1
export PATH=$PATH:/usr/local/node/bin
if ! grep -q "/usr/local/node/bin" /etc/profile; then
echo 'export PATH=$PATH:/usr/local/node/bin' >> /etc/profile
fi
log_info "✓ Node.js installed"
fi
# 4. Create filesystem structure
log_step "Creating filesystem structure..."
INSTALL_PREFIX="/opt/adastra/calypso"
CONFIG_DIR="/etc/calypso"
DATA_DIR="/srv/calypso"
LOG_DIR="/var/log/calypso"
LIB_DIR="/var/lib/calypso"
mkdir -p "$INSTALL_PREFIX/releases/${VERSION}"/{bin,web,migrations,scripts}
mkdir -p "$INSTALL_PREFIX/third_party"
mkdir -p "$CONFIG_DIR"/{tls,integrations,system,scst,nfs,samba,clamav}
mkdir -p "$DATA_DIR"/{db,backups,object,shares,vtl,iscsi,uploads,cache,_system,quarantine}
mkdir -p "$LOG_DIR" "$LIB_DIR" "/run/calypso"
# Create calypso user
if ! id "calypso" &>/dev/null; then
useradd -r -s /bin/false -d "$LIB_DIR" -c "Calypso Appliance" calypso
fi
# 5. Install binaries
log_step "Installing Calypso binaries..."
cp "$BUNDLE_DIR/binaries/calypso-api" "$INSTALL_PREFIX/releases/${VERSION}/bin/"
chmod +x "$INSTALL_PREFIX/releases/${VERSION}/bin/calypso-api"
# Install frontend
cp -r "$BUNDLE_DIR/frontend"/* "$INSTALL_PREFIX/releases/${VERSION}/web/"
# Install migrations
cp -r "$BUNDLE_DIR/migrations"/* "$INSTALL_PREFIX/releases/${VERSION}/migrations/" 2>/dev/null || true
# Install scripts
cp -r "$BUNDLE_DIR/scripts"/* "$INSTALL_PREFIX/releases/${VERSION}/scripts/" 2>/dev/null || true
chmod +x "$INSTALL_PREFIX/releases/${VERSION}/scripts"/*.sh 2>/dev/null || true
# Create symlink
ln -sfn "releases/${VERSION}" "$INSTALL_PREFIX/current"
# Set ownership
chown -R calypso:calypso "$INSTALL_PREFIX" "$LIB_DIR" "$LOG_DIR" "$DATA_DIR" 2>/dev/null || chown -R root:root "$INSTALL_PREFIX" "$LIB_DIR" "$LOG_DIR" "$DATA_DIR"
log_info "✓ Binaries installed"
# 6. Install systemd services
log_step "Installing systemd services..."
if [ -d "$BUNDLE_DIR/services" ]; then
cp "$BUNDLE_DIR/services"/*.service /etc/systemd/system/ 2>/dev/null || true
systemctl daemon-reload
log_info "✓ Systemd services installed"
fi
# 7. Setup database
log_step "Setting up database..."
if systemctl is-active --quiet postgresql 2>/dev/null || systemctl is-enabled --quiet postgresql 2>/dev/null; then
sudo -u postgres createdb calypso 2>/dev/null || true
sudo -u postgres createuser calypso 2>/dev/null || true
sudo -u postgres psql -c "ALTER USER calypso WITH PASSWORD 'calypso_change_me';" 2>/dev/null || true
sudo -u postgres psql -c "GRANT ALL PRIVILEGES ON DATABASE calypso TO calypso;" 2>/dev/null || true
log_info "✓ Database setup complete"
else
log_warn "PostgreSQL not running. Please start it and run database setup manually."
fi
# 8. Generate configuration
log_step "Generating configuration..."
if [ -f "$BUNDLE_DIR/configs/config.yaml.template" ]; then
cp "$BUNDLE_DIR/configs/config.yaml.template" "$CONFIG_DIR/calypso.yaml"
# Generate secrets
if command -v openssl &> /dev/null; then
JWT_SECRET=$(openssl rand -hex 32)
sed -i "s/CHANGE_ME_JWT_SECRET/$JWT_SECRET/g" "$CONFIG_DIR/calypso.yaml"
fi
log_info "✓ Configuration generated"
fi
log_step "Installation Complete!"
log_info ""
log_info "Next steps:"
log_info "1. Review configuration: $CONFIG_DIR/calypso.yaml"
log_info "2. Install kernel modules (ZFS, SCST, mhVTL) if needed"
log_info "3. Start services: systemctl start calypso-api"
log_info "4. Enable services: systemctl enable calypso-api"
log_info ""
log_info "Calypso API will be available at: http://localhost:8080"

229
dist/airgap/calypso-appliance-1.0.0-airgap/install.sh vendored Executable file
View File

@@ -0,0 +1,229 @@
#!/bin/bash
#
# AtlasOS - Calypso Appliance Installer
# Complete installation script for Calypso backup appliance
# Target OS: Ubuntu Server 24.04 LTS
#
# Usage: sudo ./installer/alpha/install.sh [options]
#
# Options:
# --version VERSION Install specific version (default: auto-detect)
# --skip-deps Skip system dependencies installation
# --skip-zfs Skip ZFS installation
# --skip-scst Skip SCST installation
# --skip-mhvtl Skip MHVTL installation
# --skip-bacula Skip Bacula installation
# --config-only Only setup configuration, don't install binaries
#
set -euo pipefail
# Script directory
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
INSTALLER_DIR="$SCRIPT_DIR"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Logging functions
log_info() {
echo -e "${GREEN}[INFO]${NC} $1"
}
log_warn() {
echo -e "${YELLOW}[WARN]${NC} $1"
}
log_error() {
echo -e "${RED}[ERROR]${NC} $1"
}
log_step() {
echo -e "\n${BLUE}==>${NC} $1"
}
# Configuration
CALYPSO_VERSION="${CALYPSO_VERSION:-1.0.0-alpha}"
INSTALL_PREFIX="/opt/adastra/calypso"
CONFIG_DIR="/etc/calypso"
DATA_DIR="/srv/calypso"
LOG_DIR="/var/log/calypso"
LIB_DIR="/var/lib/calypso"
RUN_DIR="/run/calypso"
# Flags
SKIP_DEPS=false
SKIP_ZFS=false
SKIP_SCST=false
SKIP_MHVTL=false
SKIP_BACULA=false
CONFIG_ONLY=false
# Parse arguments
while [[ $# -gt 0 ]]; do
case $1 in
--version)
CALYPSO_VERSION="$2"
shift 2
;;
--skip-deps)
SKIP_DEPS=true
shift
;;
--skip-zfs)
SKIP_ZFS=true
shift
;;
--skip-scst)
SKIP_SCST=true
shift
;;
--skip-mhvtl)
SKIP_MHVTL=true
shift
;;
--skip-bacula)
SKIP_BACULA=true
shift
;;
--config-only)
CONFIG_ONLY=true
shift
;;
*)
log_error "Unknown option: $1"
exit 1
;;
esac
done
# Check if running as root
if [[ $EUID -ne 0 ]]; then
log_error "This script must be run as root (use sudo)"
exit 1
fi
# Detect OS
if ! grep -q "Ubuntu" /etc/os-release 2>/dev/null; then
log_warn "This installer is designed for Ubuntu. Proceeding anyway..."
fi
log_info "=========================================="
log_info "AtlasOS - Calypso Appliance Installer"
log_info "Version: ${CALYPSO_VERSION}"
log_info "=========================================="
log_info ""
# Source helper scripts
source "$INSTALLER_DIR/scripts/helpers.sh"
source "$INSTALLER_DIR/scripts/filesystem.sh"
source "$INSTALLER_DIR/scripts/dependencies.sh"
source "$INSTALLER_DIR/scripts/components.sh"
source "$INSTALLER_DIR/scripts/application.sh"
source "$INSTALLER_DIR/scripts/database.sh"
source "$INSTALLER_DIR/scripts/services.sh"
source "$INSTALLER_DIR/scripts/configuration.sh"
source "$INSTALLER_DIR/scripts/configure-services.sh"
source "$INSTALLER_DIR/scripts/post-install.sh"
# Main installation function
main() {
log_step "Starting Calypso Appliance Installation"
# Pre-flight checks
log_step "Pre-flight Checks"
check_prerequisites
# Create filesystem structure
log_step "Creating Filesystem Structure"
create_filesystem_structure
# Install system dependencies
if [[ "$SKIP_DEPS" == "false" ]]; then
log_step "Installing System Dependencies"
install_system_dependencies
else
log_info "Skipping system dependencies installation"
fi
# Install components
if [[ "$SKIP_ZFS" == "false" ]]; then
log_step "Installing ZFS"
install_zfs || log_warn "ZFS installation failed or already installed"
fi
if [[ "$SKIP_SCST" == "false" ]]; then
log_step "Installing SCST"
install_scst || log_warn "SCST installation failed or already installed"
fi
if [[ "$SKIP_MHVTL" == "false" ]]; then
log_step "Installing MHVTL"
install_mhvtl || log_warn "MHVTL installation failed or already installed"
fi
if [[ "$SKIP_BACULA" == "false" ]]; then
log_step "Installing Bacula (Optional)"
install_bacula || log_warn "Bacula installation skipped or failed"
fi
# File sharing services are installed in dependencies step
# ClamAV is installed in dependencies step
# Build and install application
if [[ "$CONFIG_ONLY" == "false" ]]; then
log_step "Building and Installing Application"
build_and_install_application
else
log_info "Skipping application build (config-only mode)"
fi
# Setup database
log_step "Setting Up Database"
setup_database
# Setup configuration
log_step "Setting Up Configuration"
setup_configuration
# Install systemd services
log_step "Installing Systemd Services"
install_systemd_services
# Configure file sharing and antivirus services
log_step "Configuring File Sharing and Antivirus Services"
configure_all_services
# Final verification
log_step "Verifying Installation"
verify_installation
# Post-installation setup
log_step "Post-Installation Setup"
post_install_setup
# Print summary
print_installation_summary
}
# Run main installation
main
log_info ""
log_info "=========================================="
log_info "Installation Complete!"
log_info "=========================================="
log_info ""
log_info "Next steps:"
log_info "1. Configure /etc/calypso/config.yaml"
log_info "2. Set environment variables in /etc/calypso/secrets.env"
log_info "3. Start services: sudo systemctl start calypso-api"
log_info "4. Access web UI: http://<server-ip>:3000"
log_info "5. Login with default admin credentials (check above)"
log_info ""

213
dist/airgap/calypso-appliance-1.0.0-airgap/migrations/001_initial_schema.sql vendored Normal file
View File

@@ -0,0 +1,213 @@
-- AtlasOS - Calypso
-- Initial Database Schema
-- Version: 1.0
-- Users table
CREATE TABLE IF NOT EXISTS users (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
username VARCHAR(255) NOT NULL UNIQUE,
email VARCHAR(255) NOT NULL UNIQUE,
password_hash VARCHAR(255) NOT NULL,
full_name VARCHAR(255),
is_active BOOLEAN NOT NULL DEFAULT true,
is_system BOOLEAN NOT NULL DEFAULT false,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
last_login_at TIMESTAMP
);
-- Roles table
CREATE TABLE IF NOT EXISTS roles (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(100) NOT NULL UNIQUE,
description TEXT,
is_system BOOLEAN NOT NULL DEFAULT false,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- Permissions table
CREATE TABLE IF NOT EXISTS permissions (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL UNIQUE,
resource VARCHAR(100) NOT NULL,
action VARCHAR(100) NOT NULL,
description TEXT,
created_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- User roles junction table
CREATE TABLE IF NOT EXISTS user_roles (
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
role_id UUID NOT NULL REFERENCES roles(id) ON DELETE CASCADE,
assigned_at TIMESTAMP NOT NULL DEFAULT NOW(),
assigned_by UUID REFERENCES users(id),
PRIMARY KEY (user_id, role_id)
);
-- Role permissions junction table
CREATE TABLE IF NOT EXISTS role_permissions (
role_id UUID NOT NULL REFERENCES roles(id) ON DELETE CASCADE,
permission_id UUID NOT NULL REFERENCES permissions(id) ON DELETE CASCADE,
granted_at TIMESTAMP NOT NULL DEFAULT NOW(),
PRIMARY KEY (role_id, permission_id)
);
-- Sessions table
CREATE TABLE IF NOT EXISTS sessions (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
token_hash VARCHAR(255) NOT NULL UNIQUE,
ip_address INET,
user_agent TEXT,
expires_at TIMESTAMP NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
last_activity_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- Audit log table
CREATE TABLE IF NOT EXISTS audit_log (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id UUID REFERENCES users(id),
username VARCHAR(255),
action VARCHAR(100) NOT NULL,
resource_type VARCHAR(100) NOT NULL,
resource_id VARCHAR(255),
method VARCHAR(10),
path TEXT,
ip_address INET,
user_agent TEXT,
request_body JSONB,
response_status INTEGER,
error_message TEXT,
created_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- Tasks table (for async operations)
CREATE TABLE IF NOT EXISTS tasks (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
type VARCHAR(100) NOT NULL,
status VARCHAR(50) NOT NULL DEFAULT 'pending',
progress INTEGER NOT NULL DEFAULT 0,
message TEXT,
error_message TEXT,
created_by UUID REFERENCES users(id),
started_at TIMESTAMP,
completed_at TIMESTAMP,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
metadata JSONB
);
-- Alerts table
CREATE TABLE IF NOT EXISTS alerts (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
severity VARCHAR(20) NOT NULL,
source VARCHAR(100) NOT NULL,
title VARCHAR(255) NOT NULL,
message TEXT NOT NULL,
resource_type VARCHAR(100),
resource_id VARCHAR(255),
is_acknowledged BOOLEAN NOT NULL DEFAULT false,
acknowledged_by UUID REFERENCES users(id),
acknowledged_at TIMESTAMP,
resolved_at TIMESTAMP,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
metadata JSONB
);
-- System configuration table
CREATE TABLE IF NOT EXISTS system_config (
key VARCHAR(255) PRIMARY KEY,
value TEXT NOT NULL,
description TEXT,
is_encrypted BOOLEAN NOT NULL DEFAULT false,
updated_by UUID REFERENCES users(id),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
created_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- Indexes for performance
CREATE INDEX IF NOT EXISTS idx_users_username ON users(username);
CREATE INDEX IF NOT EXISTS idx_users_email ON users(email);
CREATE INDEX IF NOT EXISTS idx_users_active ON users(is_active);
CREATE INDEX IF NOT EXISTS idx_sessions_user_id ON sessions(user_id);
CREATE INDEX IF NOT EXISTS idx_sessions_token_hash ON sessions(token_hash);
CREATE INDEX IF NOT EXISTS idx_sessions_expires_at ON sessions(expires_at);
CREATE INDEX IF NOT EXISTS idx_audit_log_user_id ON audit_log(user_id);
CREATE INDEX IF NOT EXISTS idx_audit_log_created_at ON audit_log(created_at);
CREATE INDEX IF NOT EXISTS idx_audit_log_resource ON audit_log(resource_type, resource_id);
CREATE INDEX IF NOT EXISTS idx_tasks_status ON tasks(status);
CREATE INDEX IF NOT EXISTS idx_tasks_type ON tasks(type);
CREATE INDEX IF NOT EXISTS idx_tasks_created_by ON tasks(created_by);
CREATE INDEX IF NOT EXISTS idx_alerts_severity ON alerts(severity);
CREATE INDEX IF NOT EXISTS idx_alerts_acknowledged ON alerts(is_acknowledged);
CREATE INDEX IF NOT EXISTS idx_alerts_created_at ON alerts(created_at);
-- Insert default system roles
INSERT INTO roles (name, description, is_system) VALUES
('admin', 'Full system access and configuration', true),
('operator', 'Day-to-day operations and monitoring', true),
('readonly', 'Read-only access for monitoring and reporting', true)
ON CONFLICT (name) DO NOTHING;
-- Insert default permissions
INSERT INTO permissions (name, resource, action, description) VALUES
-- System permissions
('system:read', 'system', 'read', 'View system information'),
('system:write', 'system', 'write', 'Modify system configuration'),
('system:manage', 'system', 'manage', 'Full system management'),
-- Storage permissions
('storage:read', 'storage', 'read', 'View storage information'),
('storage:write', 'storage', 'write', 'Modify storage configuration'),
('storage:manage', 'storage', 'manage', 'Full storage management'),
-- Tape permissions
('tape:read', 'tape', 'read', 'View tape library information'),
('tape:write', 'tape', 'write', 'Perform tape operations'),
('tape:manage', 'tape', 'manage', 'Full tape management'),
-- iSCSI permissions
('iscsi:read', 'iscsi', 'read', 'View iSCSI configuration'),
('iscsi:write', 'iscsi', 'write', 'Modify iSCSI configuration'),
('iscsi:manage', 'iscsi', 'manage', 'Full iSCSI management'),
-- IAM permissions
('iam:read', 'iam', 'read', 'View users and roles'),
('iam:write', 'iam', 'write', 'Modify users and roles'),
('iam:manage', 'iam', 'manage', 'Full IAM management'),
-- Audit permissions
('audit:read', 'audit', 'read', 'View audit logs'),
-- Monitoring permissions
('monitoring:read', 'monitoring', 'read', 'View monitoring data'),
('monitoring:write', 'monitoring', 'write', 'Acknowledge alerts')
ON CONFLICT (name) DO NOTHING;
-- Assign permissions to roles
-- Admin gets all permissions
INSERT INTO role_permissions (role_id, permission_id)
SELECT r.id, p.id
FROM roles r, permissions p
WHERE r.name = 'admin'
ON CONFLICT DO NOTHING;
-- Operator gets read and write (but not manage) for most resources
INSERT INTO role_permissions (role_id, permission_id)
SELECT r.id, p.id
FROM roles r, permissions p
WHERE r.name = 'operator'
AND p.action IN ('read', 'write')
AND p.resource IN ('storage', 'tape', 'iscsi', 'monitoring')
ON CONFLICT DO NOTHING;
-- ReadOnly gets only read permissions
INSERT INTO role_permissions (role_id, permission_id)
SELECT r.id, p.id
FROM roles r, permissions p
WHERE r.name = 'readonly'
AND p.action = 'read'
ON CONFLICT DO NOTHING;

227
dist/airgap/calypso-appliance-1.0.0-airgap/migrations/002_storage_and_tape_schema.sql vendored Normal file
View File

@@ -0,0 +1,227 @@
-- AtlasOS - Calypso
-- Storage and Tape Component Schema
-- Version: 2.0
-- ZFS pools table
CREATE TABLE IF NOT EXISTS zfs_pools (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL UNIQUE,
description TEXT,
raid_level VARCHAR(50) NOT NULL, -- stripe, mirror, raidz, raidz2, raidz3
disks TEXT[] NOT NULL, -- array of device paths
size_bytes BIGINT NOT NULL,
used_bytes BIGINT NOT NULL DEFAULT 0,
compression VARCHAR(50) NOT NULL DEFAULT 'lz4', -- off, lz4, zstd, gzip
deduplication BOOLEAN NOT NULL DEFAULT false,
auto_expand BOOLEAN NOT NULL DEFAULT false,
scrub_interval INTEGER NOT NULL DEFAULT 30, -- days
is_active BOOLEAN NOT NULL DEFAULT true,
health_status VARCHAR(50) NOT NULL DEFAULT 'online', -- online, degraded, faulted, offline
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
created_by UUID REFERENCES users(id)
);
-- Disk repositories table
CREATE TABLE IF NOT EXISTS disk_repositories (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL UNIQUE,
description TEXT,
volume_group VARCHAR(255) NOT NULL,
logical_volume VARCHAR(255) NOT NULL,
size_bytes BIGINT NOT NULL,
used_bytes BIGINT NOT NULL DEFAULT 0,
filesystem_type VARCHAR(50),
mount_point TEXT,
is_active BOOLEAN NOT NULL DEFAULT true,
warning_threshold_percent INTEGER NOT NULL DEFAULT 80,
critical_threshold_percent INTEGER NOT NULL DEFAULT 90,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
created_by UUID REFERENCES users(id)
);
-- Physical disks table
CREATE TABLE IF NOT EXISTS physical_disks (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
device_path VARCHAR(255) NOT NULL UNIQUE,
vendor VARCHAR(255),
model VARCHAR(255),
serial_number VARCHAR(255),
size_bytes BIGINT NOT NULL,
sector_size INTEGER,
is_ssd BOOLEAN NOT NULL DEFAULT false,
health_status VARCHAR(50) NOT NULL DEFAULT 'unknown',
health_details JSONB,
is_used BOOLEAN NOT NULL DEFAULT false,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- Volume groups table
CREATE TABLE IF NOT EXISTS volume_groups (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL UNIQUE,
size_bytes BIGINT NOT NULL,
free_bytes BIGINT NOT NULL,
physical_volumes TEXT[],
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- SCST iSCSI targets table
CREATE TABLE IF NOT EXISTS scst_targets (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
iqn VARCHAR(512) NOT NULL UNIQUE,
target_type VARCHAR(50) NOT NULL, -- 'disk', 'vtl', 'physical_tape'
name VARCHAR(255) NOT NULL,
description TEXT,
is_active BOOLEAN NOT NULL DEFAULT true,
single_initiator_only BOOLEAN NOT NULL DEFAULT false,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
created_by UUID REFERENCES users(id)
);
-- SCST LUN mappings table
CREATE TABLE IF NOT EXISTS scst_luns (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
target_id UUID NOT NULL REFERENCES scst_targets(id) ON DELETE CASCADE,
lun_number INTEGER NOT NULL,
device_name VARCHAR(255) NOT NULL,
device_path VARCHAR(512) NOT NULL,
handler_type VARCHAR(50) NOT NULL, -- 'vdisk', 'sg', 'tape'
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
UNIQUE(target_id, lun_number)
);
-- SCST initiator groups table
CREATE TABLE IF NOT EXISTS scst_initiator_groups (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
target_id UUID NOT NULL REFERENCES scst_targets(id) ON DELETE CASCADE,
group_name VARCHAR(255) NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
UNIQUE(target_id, group_name)
);
-- SCST initiators table
CREATE TABLE IF NOT EXISTS scst_initiators (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
group_id UUID NOT NULL REFERENCES scst_initiator_groups(id) ON DELETE CASCADE,
iqn VARCHAR(512) NOT NULL,
is_active BOOLEAN NOT NULL DEFAULT true,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
UNIQUE(group_id, iqn)
);
-- Physical tape libraries table
CREATE TABLE IF NOT EXISTS physical_tape_libraries (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL UNIQUE,
serial_number VARCHAR(255),
vendor VARCHAR(255),
model VARCHAR(255),
changer_device_path VARCHAR(512),
changer_stable_path VARCHAR(512),
slot_count INTEGER,
drive_count INTEGER,
is_active BOOLEAN NOT NULL DEFAULT true,
discovered_at TIMESTAMP NOT NULL DEFAULT NOW(),
last_inventory_at TIMESTAMP,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- Physical tape drives table
CREATE TABLE IF NOT EXISTS physical_tape_drives (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
library_id UUID NOT NULL REFERENCES physical_tape_libraries(id) ON DELETE CASCADE,
drive_number INTEGER NOT NULL,
device_path VARCHAR(512),
stable_path VARCHAR(512),
vendor VARCHAR(255),
model VARCHAR(255),
serial_number VARCHAR(255),
drive_type VARCHAR(50), -- 'LTO-8', 'LTO-9', etc.
status VARCHAR(50) NOT NULL DEFAULT 'unknown', -- 'idle', 'loading', 'ready', 'error'
current_tape_barcode VARCHAR(255),
is_active BOOLEAN NOT NULL DEFAULT true,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
UNIQUE(library_id, drive_number)
);
-- Physical tape slots table
CREATE TABLE IF NOT EXISTS physical_tape_slots (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
library_id UUID NOT NULL REFERENCES physical_tape_libraries(id) ON DELETE CASCADE,
slot_number INTEGER NOT NULL,
barcode VARCHAR(255),
tape_present BOOLEAN NOT NULL DEFAULT false,
tape_type VARCHAR(50),
last_updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
UNIQUE(library_id, slot_number)
);
-- Virtual tape libraries table
CREATE TABLE IF NOT EXISTS virtual_tape_libraries (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL UNIQUE,
description TEXT,
mhvtl_library_id INTEGER,
backing_store_path TEXT NOT NULL,
slot_count INTEGER NOT NULL DEFAULT 10,
drive_count INTEGER NOT NULL DEFAULT 2,
is_active BOOLEAN NOT NULL DEFAULT true,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
created_by UUID REFERENCES users(id)
);
-- Virtual tape drives table
CREATE TABLE IF NOT EXISTS virtual_tape_drives (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
library_id UUID NOT NULL REFERENCES virtual_tape_libraries(id) ON DELETE CASCADE,
drive_number INTEGER NOT NULL,
device_path VARCHAR(512),
stable_path VARCHAR(512),
status VARCHAR(50) NOT NULL DEFAULT 'idle',
current_tape_id UUID,
is_active BOOLEAN NOT NULL DEFAULT true,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
UNIQUE(library_id, drive_number)
);
-- Virtual tapes table
CREATE TABLE IF NOT EXISTS virtual_tapes (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
library_id UUID NOT NULL REFERENCES virtual_tape_libraries(id) ON DELETE CASCADE,
barcode VARCHAR(255) NOT NULL,
slot_number INTEGER,
image_file_path TEXT NOT NULL,
size_bytes BIGINT NOT NULL DEFAULT 0,
used_bytes BIGINT NOT NULL DEFAULT 0,
tape_type VARCHAR(50) NOT NULL DEFAULT 'LTO-8',
status VARCHAR(50) NOT NULL DEFAULT 'idle', -- 'idle', 'in_drive', 'exported'
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
UNIQUE(library_id, barcode)
);
-- Indexes for performance
CREATE INDEX IF NOT EXISTS idx_disk_repositories_name ON disk_repositories(name);
CREATE INDEX IF NOT EXISTS idx_disk_repositories_active ON disk_repositories(is_active);
CREATE INDEX IF NOT EXISTS idx_physical_disks_device_path ON physical_disks(device_path);
CREATE INDEX IF NOT EXISTS idx_scst_targets_iqn ON scst_targets(iqn);
CREATE INDEX IF NOT EXISTS idx_scst_targets_type ON scst_targets(target_type);
CREATE INDEX IF NOT EXISTS idx_scst_luns_target_id ON scst_luns(target_id);
CREATE INDEX IF NOT EXISTS idx_scst_initiators_group_id ON scst_initiators(group_id);
CREATE INDEX IF NOT EXISTS idx_physical_tape_libraries_name ON physical_tape_libraries(name);
CREATE INDEX IF NOT EXISTS idx_physical_tape_drives_library_id ON physical_tape_drives(library_id);
CREATE INDEX IF NOT EXISTS idx_physical_tape_slots_library_id ON physical_tape_slots(library_id);
CREATE INDEX IF NOT EXISTS idx_virtual_tape_libraries_name ON virtual_tape_libraries(name);
CREATE INDEX IF NOT EXISTS idx_virtual_tape_drives_library_id ON virtual_tape_drives(library_id);
CREATE INDEX IF NOT EXISTS idx_virtual_tapes_library_id ON virtual_tapes(library_id);
CREATE INDEX IF NOT EXISTS idx_virtual_tapes_barcode ON virtual_tapes(barcode);

22
dist/airgap/calypso-appliance-1.0.0-airgap/migrations/003_object_storage_config.sql vendored Normal file
View File

@@ -0,0 +1,22 @@
-- Migration: Object Storage Configuration
-- Description: Creates table for storing MinIO object storage configuration
-- Date: 2026-01-09
CREATE TABLE IF NOT EXISTS object_storage_config (
id SERIAL PRIMARY KEY,
dataset_path VARCHAR(255) NOT NULL UNIQUE,
mount_point VARCHAR(512) NOT NULL,
pool_name VARCHAR(255) NOT NULL,
dataset_name VARCHAR(255) NOT NULL,
created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_object_storage_config_pool_name ON object_storage_config(pool_name);
CREATE INDEX IF NOT EXISTS idx_object_storage_config_updated_at ON object_storage_config(updated_at);
COMMENT ON TABLE object_storage_config IS 'Stores MinIO object storage configuration, linking to ZFS datasets';
COMMENT ON COLUMN object_storage_config.dataset_path IS 'Full ZFS dataset path (e.g., pool/dataset)';
COMMENT ON COLUMN object_storage_config.mount_point IS 'Mount point path for the dataset';
COMMENT ON COLUMN object_storage_config.pool_name IS 'ZFS pool name';
COMMENT ON COLUMN object_storage_config.dataset_name IS 'ZFS dataset name';

174
dist/airgap/calypso-appliance-1.0.0-airgap/migrations/003_performance_indexes.sql vendored Normal file
View File

@@ -0,0 +1,174 @@
-- AtlasOS - Calypso
-- Performance Optimization: Database Indexes
-- Version: 3.0
-- Description: Adds indexes for frequently queried columns to improve query performance
-- ============================================================================
-- Authentication & Authorization Indexes
-- ============================================================================
-- Users table indexes
-- Username is frequently queried during login
CREATE INDEX IF NOT EXISTS idx_users_username ON users(username);
-- Email lookups
CREATE INDEX IF NOT EXISTS idx_users_email ON users(email);
-- Active user lookups
CREATE INDEX IF NOT EXISTS idx_users_is_active ON users(is_active) WHERE is_active = true;
-- Sessions table indexes
-- Token hash lookups are very frequent (every authenticated request)
CREATE INDEX IF NOT EXISTS idx_sessions_token_hash ON sessions(token_hash);
-- User session lookups
CREATE INDEX IF NOT EXISTS idx_sessions_user_id ON sessions(user_id);
-- Expired session cleanup (index on expires_at for efficient cleanup queries)
CREATE INDEX IF NOT EXISTS idx_sessions_expires_at ON sessions(expires_at);
-- User roles junction table
-- Lookup roles for a user (frequent during permission checks)
CREATE INDEX IF NOT EXISTS idx_user_roles_user_id ON user_roles(user_id);
-- Lookup users with a role
CREATE INDEX IF NOT EXISTS idx_user_roles_role_id ON user_roles(role_id);
-- Role permissions junction table
-- Lookup permissions for a role
CREATE INDEX IF NOT EXISTS idx_role_permissions_role_id ON role_permissions(role_id);
-- Lookup roles with a permission
CREATE INDEX IF NOT EXISTS idx_role_permissions_permission_id ON role_permissions(permission_id);
-- ============================================================================
-- Audit & Monitoring Indexes
-- ============================================================================
-- Audit log indexes
-- Time-based queries (most common audit log access pattern)
CREATE INDEX IF NOT EXISTS idx_audit_log_created_at ON audit_log(created_at DESC);
-- User activity queries
CREATE INDEX IF NOT EXISTS idx_audit_log_user_id ON audit_log(user_id);
-- Resource-based queries
CREATE INDEX IF NOT EXISTS idx_audit_log_resource ON audit_log(resource_type, resource_id);
-- Alerts table indexes
-- Time-based ordering (default ordering in ListAlerts)
CREATE INDEX IF NOT EXISTS idx_alerts_created_at ON alerts(created_at DESC);
-- Severity filtering
CREATE INDEX IF NOT EXISTS idx_alerts_severity ON alerts(severity);
-- Source filtering
CREATE INDEX IF NOT EXISTS idx_alerts_source ON alerts(source);
-- Acknowledgment status
CREATE INDEX IF NOT EXISTS idx_alerts_acknowledged ON alerts(is_acknowledged) WHERE is_acknowledged = false;
-- Resource-based queries
CREATE INDEX IF NOT EXISTS idx_alerts_resource ON alerts(resource_type, resource_id);
-- Composite index for common filter combinations
CREATE INDEX IF NOT EXISTS idx_alerts_severity_acknowledged ON alerts(severity, is_acknowledged, created_at DESC);
-- ============================================================================
-- Task Management Indexes
-- ============================================================================
-- Tasks table indexes
-- Status filtering (very common in task queries)
CREATE INDEX IF NOT EXISTS idx_tasks_status ON tasks(status);
-- Created by user
CREATE INDEX IF NOT EXISTS idx_tasks_created_by ON tasks(created_by);
-- Time-based queries
CREATE INDEX IF NOT EXISTS idx_tasks_created_at ON tasks(created_at DESC);
-- Status + time composite (common query pattern)
CREATE INDEX IF NOT EXISTS idx_tasks_status_created_at ON tasks(status, created_at DESC);
-- Failed tasks lookup (for alerting)
CREATE INDEX IF NOT EXISTS idx_tasks_failed_recent ON tasks(status, created_at DESC) WHERE status = 'failed';
-- ============================================================================
-- Storage Indexes
-- ============================================================================
-- Disk repositories indexes
-- Active repository lookups
CREATE INDEX IF NOT EXISTS idx_disk_repositories_is_active ON disk_repositories(is_active) WHERE is_active = true;
-- Name lookups
CREATE INDEX IF NOT EXISTS idx_disk_repositories_name ON disk_repositories(name);
-- Volume group lookups
CREATE INDEX IF NOT EXISTS idx_disk_repositories_vg ON disk_repositories(volume_group);
-- Physical disks indexes
-- Device path lookups (for sync operations)
CREATE INDEX IF NOT EXISTS idx_physical_disks_device_path ON physical_disks(device_path);
-- ============================================================================
-- SCST Indexes
-- ============================================================================
-- SCST targets indexes
-- IQN lookups (frequent during target operations)
CREATE INDEX IF NOT EXISTS idx_scst_targets_iqn ON scst_targets(iqn);
-- Active target lookups
CREATE INDEX IF NOT EXISTS idx_scst_targets_is_active ON scst_targets(is_active) WHERE is_active = true;
-- SCST LUNs indexes
-- Target + LUN lookups (very frequent)
CREATE INDEX IF NOT EXISTS idx_scst_luns_target_lun ON scst_luns(target_id, lun_number);
-- Device path lookups
CREATE INDEX IF NOT EXISTS idx_scst_luns_device_path ON scst_luns(device_path);
-- SCST initiator groups indexes
-- Target + group name lookups
CREATE INDEX IF NOT EXISTS idx_scst_initiator_groups_target ON scst_initiator_groups(target_id);
-- Group name lookups
CREATE INDEX IF NOT EXISTS idx_scst_initiator_groups_name ON scst_initiator_groups(group_name);
-- SCST initiators indexes
-- Group + IQN lookups
CREATE INDEX IF NOT EXISTS idx_scst_initiators_group_iqn ON scst_initiators(group_id, iqn);
-- Active initiator lookups
CREATE INDEX IF NOT EXISTS idx_scst_initiators_is_active ON scst_initiators(is_active) WHERE is_active = true;
-- ============================================================================
-- Tape Library Indexes
-- ============================================================================
-- Physical tape libraries indexes
-- Serial number lookups (for discovery)
CREATE INDEX IF NOT EXISTS idx_physical_tape_libraries_serial ON physical_tape_libraries(serial_number);
-- Active library lookups
CREATE INDEX IF NOT EXISTS idx_physical_tape_libraries_is_active ON physical_tape_libraries(is_active) WHERE is_active = true;
-- Physical tape drives indexes
-- Library + drive number lookups (very frequent)
CREATE INDEX IF NOT EXISTS idx_physical_tape_drives_library_drive ON physical_tape_drives(library_id, drive_number);
-- Status filtering
CREATE INDEX IF NOT EXISTS idx_physical_tape_drives_status ON physical_tape_drives(status);
-- Physical tape slots indexes
-- Library + slot number lookups
CREATE INDEX IF NOT EXISTS idx_physical_tape_slots_library_slot ON physical_tape_slots(library_id, slot_number);
-- Barcode lookups
CREATE INDEX IF NOT EXISTS idx_physical_tape_slots_barcode ON physical_tape_slots(barcode) WHERE barcode IS NOT NULL;
-- Virtual tape libraries indexes
-- MHVTL library ID lookups
CREATE INDEX IF NOT EXISTS idx_virtual_tape_libraries_mhvtl_id ON virtual_tape_libraries(mhvtl_library_id);
-- Active library lookups
CREATE INDEX IF NOT EXISTS idx_virtual_tape_libraries_is_active ON virtual_tape_libraries(is_active) WHERE is_active = true;
-- Virtual tape drives indexes
-- Library + drive number lookups (very frequent)
CREATE INDEX IF NOT EXISTS idx_virtual_tape_drives_library_drive ON virtual_tape_drives(library_id, drive_number);
-- Status filtering
CREATE INDEX IF NOT EXISTS idx_virtual_tape_drives_status ON virtual_tape_drives(status);
-- Current tape lookups
CREATE INDEX IF NOT EXISTS idx_virtual_tape_drives_current_tape ON virtual_tape_drives(current_tape_id) WHERE current_tape_id IS NOT NULL;
-- Virtual tapes indexes
-- Library + slot number lookups
CREATE INDEX IF NOT EXISTS idx_virtual_tapes_library_slot ON virtual_tapes(library_id, slot_number);
-- Barcode lookups
CREATE INDEX IF NOT EXISTS idx_virtual_tapes_barcode ON virtual_tapes(barcode) WHERE barcode IS NOT NULL;
-- Status filtering
CREATE INDEX IF NOT EXISTS idx_virtual_tapes_status ON virtual_tapes(status);
-- ============================================================================
-- Statistics Update
-- ============================================================================
-- Update table statistics for query planner
ANALYZE;

31
dist/airgap/calypso-appliance-1.0.0-airgap/migrations/004_add_zfs_pools_table.sql vendored Normal file
View File

@@ -0,0 +1,31 @@
-- AtlasOS - Calypso
-- Add ZFS Pools Table
-- Version: 4.0
-- Note: This migration adds the zfs_pools table that was added to migration 002
-- but may not have been applied if migration 002 was run before the table was added
-- ZFS pools table
CREATE TABLE IF NOT EXISTS zfs_pools (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL UNIQUE,
description TEXT,
raid_level VARCHAR(50) NOT NULL, -- stripe, mirror, raidz, raidz2, raidz3
disks TEXT[] NOT NULL, -- array of device paths
size_bytes BIGINT NOT NULL,
used_bytes BIGINT NOT NULL DEFAULT 0,
compression VARCHAR(50) NOT NULL DEFAULT 'lz4', -- off, lz4, zstd, gzip
deduplication BOOLEAN NOT NULL DEFAULT false,
auto_expand BOOLEAN NOT NULL DEFAULT false,
scrub_interval INTEGER NOT NULL DEFAULT 30, -- days
is_active BOOLEAN NOT NULL DEFAULT true,
health_status VARCHAR(50) NOT NULL DEFAULT 'online', -- online, degraded, faulted, offline
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
created_by UUID REFERENCES users(id)
);
-- Create index on name for faster lookups
CREATE INDEX IF NOT EXISTS idx_zfs_pools_name ON zfs_pools(name);
CREATE INDEX IF NOT EXISTS idx_zfs_pools_created_by ON zfs_pools(created_by);
CREATE INDEX IF NOT EXISTS idx_zfs_pools_health_status ON zfs_pools(health_status);

35
dist/airgap/calypso-appliance-1.0.0-airgap/migrations/005_add_zfs_datasets_table.sql vendored Normal file
View File

@@ -0,0 +1,35 @@
-- AtlasOS - Calypso
-- Add ZFS Datasets Table
-- Version: 5.0
-- Description: Stores ZFS dataset metadata in database for faster queries and consistency
-- ZFS datasets table
CREATE TABLE IF NOT EXISTS zfs_datasets (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(512) NOT NULL UNIQUE, -- Full dataset name (e.g., pool/dataset)
pool_id UUID NOT NULL REFERENCES zfs_pools(id) ON DELETE CASCADE,
pool_name VARCHAR(255) NOT NULL, -- Denormalized for faster queries
type VARCHAR(50) NOT NULL, -- filesystem, volume, snapshot
mount_point TEXT, -- Mount point path (null for volumes)
used_bytes BIGINT NOT NULL DEFAULT 0,
available_bytes BIGINT NOT NULL DEFAULT 0,
referenced_bytes BIGINT NOT NULL DEFAULT 0,
compression VARCHAR(50) NOT NULL DEFAULT 'lz4', -- off, lz4, zstd, gzip
deduplication VARCHAR(50) NOT NULL DEFAULT 'off', -- off, on, verify
quota BIGINT DEFAULT -1, -- -1 for unlimited, bytes otherwise
reservation BIGINT NOT NULL DEFAULT 0, -- Reserved space in bytes
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
created_by UUID REFERENCES users(id)
);
-- Create indexes for faster lookups
CREATE INDEX IF NOT EXISTS idx_zfs_datasets_pool_id ON zfs_datasets(pool_id);
CREATE INDEX IF NOT EXISTS idx_zfs_datasets_pool_name ON zfs_datasets(pool_name);
CREATE INDEX IF NOT EXISTS idx_zfs_datasets_name ON zfs_datasets(name);
CREATE INDEX IF NOT EXISTS idx_zfs_datasets_type ON zfs_datasets(type);
CREATE INDEX IF NOT EXISTS idx_zfs_datasets_created_by ON zfs_datasets(created_by);
-- Composite index for common queries (list datasets by pool)
CREATE INDEX IF NOT EXISTS idx_zfs_datasets_pool_type ON zfs_datasets(pool_id, type);

50
dist/airgap/calypso-appliance-1.0.0-airgap/migrations/006_add_zfs_shares_and_iscsi.sql vendored Normal file
View File

@@ -0,0 +1,50 @@
-- AtlasOS - Calypso
-- Add ZFS Shares and iSCSI Export Tables
-- Version: 6.0
-- Description: Separate tables for filesystem shares (NFS/SMB) and volume iSCSI exports
-- ZFS Filesystem Shares Table (for NFS/SMB)
CREATE TABLE IF NOT EXISTS zfs_shares (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
dataset_id UUID NOT NULL REFERENCES zfs_datasets(id) ON DELETE CASCADE,
share_type VARCHAR(50) NOT NULL, -- 'nfs', 'smb', 'both'
nfs_enabled BOOLEAN NOT NULL DEFAULT false,
nfs_options TEXT, -- e.g., "rw,sync,no_subtree_check"
nfs_clients TEXT[], -- Allowed client IPs/networks
smb_enabled BOOLEAN NOT NULL DEFAULT false,
smb_share_name VARCHAR(255), -- SMB share name
smb_path TEXT, -- SMB share path (usually same as mount_point)
smb_comment TEXT,
smb_guest_ok BOOLEAN NOT NULL DEFAULT false,
smb_read_only BOOLEAN NOT NULL DEFAULT false,
smb_browseable BOOLEAN NOT NULL DEFAULT true,
is_active BOOLEAN NOT NULL DEFAULT true,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
created_by UUID REFERENCES users(id),
UNIQUE(dataset_id) -- One share config per dataset
);
-- ZFS Volume iSCSI Exports Table
CREATE TABLE IF NOT EXISTS zfs_iscsi_exports (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
dataset_id UUID NOT NULL REFERENCES zfs_datasets(id) ON DELETE CASCADE,
target_id UUID REFERENCES scst_targets(id) ON DELETE SET NULL, -- Link to SCST target
lun_number INTEGER, -- LUN number in the target
device_path TEXT, -- /dev/zvol/pool/volume path
is_active BOOLEAN NOT NULL DEFAULT true,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
created_by UUID REFERENCES users(id),
UNIQUE(dataset_id) -- One iSCSI export per volume
);
-- Create indexes
CREATE INDEX IF NOT EXISTS idx_zfs_shares_dataset_id ON zfs_shares(dataset_id);
CREATE INDEX IF NOT EXISTS idx_zfs_shares_type ON zfs_shares(share_type);
CREATE INDEX IF NOT EXISTS idx_zfs_shares_active ON zfs_shares(is_active);
CREATE INDEX IF NOT EXISTS idx_zfs_iscsi_exports_dataset_id ON zfs_iscsi_exports(dataset_id);
CREATE INDEX IF NOT EXISTS idx_zfs_iscsi_exports_target_id ON zfs_iscsi_exports(target_id);
CREATE INDEX IF NOT EXISTS idx_zfs_iscsi_exports_active ON zfs_iscsi_exports(is_active);

3
dist/airgap/calypso-appliance-1.0.0-airgap/migrations/007_add_vendor_to_vtl_libraries.sql vendored Normal file
View File

@@ -0,0 +1,3 @@
-- Add vendor column to virtual_tape_libraries table
ALTER TABLE virtual_tape_libraries ADD COLUMN IF NOT EXISTS vendor VARCHAR(255);

45
dist/airgap/calypso-appliance-1.0.0-airgap/migrations/008_add_user_groups.sql vendored Normal file
View File

@@ -0,0 +1,45 @@
-- Add user groups feature
-- Groups table
CREATE TABLE IF NOT EXISTS groups (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL UNIQUE,
description TEXT,
is_system BOOLEAN NOT NULL DEFAULT false,
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- User groups junction table
CREATE TABLE IF NOT EXISTS user_groups (
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
group_id UUID NOT NULL REFERENCES groups(id) ON DELETE CASCADE,
assigned_at TIMESTAMP NOT NULL DEFAULT NOW(),
assigned_by UUID REFERENCES users(id),
PRIMARY KEY (user_id, group_id)
);
-- Group roles junction table (groups can have roles)
CREATE TABLE IF NOT EXISTS group_roles (
group_id UUID NOT NULL REFERENCES groups(id) ON DELETE CASCADE,
role_id UUID NOT NULL REFERENCES roles(id) ON DELETE CASCADE,
granted_at TIMESTAMP NOT NULL DEFAULT NOW(),
PRIMARY KEY (group_id, role_id)
);
-- Indexes
CREATE INDEX IF NOT EXISTS idx_groups_name ON groups(name);
CREATE INDEX IF NOT EXISTS idx_user_groups_user_id ON user_groups(user_id);
CREATE INDEX IF NOT EXISTS idx_user_groups_group_id ON user_groups(group_id);
CREATE INDEX IF NOT EXISTS idx_group_roles_group_id ON group_roles(group_id);
CREATE INDEX IF NOT EXISTS idx_group_roles_role_id ON group_roles(role_id);
-- Insert default system groups
INSERT INTO groups (name, description, is_system) VALUES
('wheel', 'System administrators group', true),
('operators', 'System operators group', true),
('backup', 'Backup operators group', true),
('auditors', 'Auditors group', true),
('storage_admins', 'Storage administrators group', true),
('services', 'Service accounts group', true)
ON CONFLICT (name) DO NOTHING;

34
dist/airgap/calypso-appliance-1.0.0-airgap/migrations/009_backup_jobs_schema.sql vendored Normal file
View File

@@ -0,0 +1,34 @@
-- AtlasOS - Calypso
-- Backup Jobs Schema
-- Version: 9.0
-- Backup jobs table
CREATE TABLE IF NOT EXISTS backup_jobs (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
job_id INTEGER NOT NULL UNIQUE, -- Bareos job ID
job_name VARCHAR(255) NOT NULL,
client_name VARCHAR(255) NOT NULL,
job_type VARCHAR(50) NOT NULL, -- 'Backup', 'Restore', 'Verify', 'Copy', 'Migrate'
job_level VARCHAR(50) NOT NULL, -- 'Full', 'Incremental', 'Differential', 'Since'
status VARCHAR(50) NOT NULL, -- 'Running', 'Completed', 'Failed', 'Canceled', 'Waiting'
bytes_written BIGINT NOT NULL DEFAULT 0,
files_written INTEGER NOT NULL DEFAULT 0,
duration_seconds INTEGER,
started_at TIMESTAMP,
ended_at TIMESTAMP,
error_message TEXT,
storage_name VARCHAR(255),
pool_name VARCHAR(255),
volume_name VARCHAR(255),
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW()
);
-- Indexes for performance
CREATE INDEX IF NOT EXISTS idx_backup_jobs_job_id ON backup_jobs(job_id);
CREATE INDEX IF NOT EXISTS idx_backup_jobs_job_name ON backup_jobs(job_name);
CREATE INDEX IF NOT EXISTS idx_backup_jobs_client_name ON backup_jobs(client_name);
CREATE INDEX IF NOT EXISTS idx_backup_jobs_status ON backup_jobs(status);
CREATE INDEX IF NOT EXISTS idx_backup_jobs_started_at ON backup_jobs(started_at DESC);
CREATE INDEX IF NOT EXISTS idx_backup_jobs_job_type ON backup_jobs(job_type);

39
dist/airgap/calypso-appliance-1.0.0-airgap/migrations/010_add_backup_permissions.sql vendored Normal file
View File

@@ -0,0 +1,39 @@
-- AtlasOS - Calypso
-- Add Backup Permissions
-- Version: 10.0
-- Insert backup permissions
INSERT INTO permissions (name, resource, action, description) VALUES
('backup:read', 'backup', 'read', 'View backup jobs and history'),
('backup:write', 'backup', 'write', 'Create and manage backup jobs'),
('backup:manage', 'backup', 'manage', 'Full backup management')
ON CONFLICT (name) DO NOTHING;
-- Assign backup permissions to roles
-- Admin gets all backup permissions (explicitly assign since admin query in 001 only runs once)
INSERT INTO role_permissions (role_id, permission_id)
SELECT r.id, p.id
FROM roles r, permissions p
WHERE r.name = 'admin'
AND p.resource = 'backup'
ON CONFLICT DO NOTHING;
-- Operator gets read and write permissions for backup
INSERT INTO role_permissions (role_id, permission_id)
SELECT r.id, p.id
FROM roles r, permissions p
WHERE r.name = 'operator'
AND p.resource = 'backup'
AND p.action IN ('read', 'write')
ON CONFLICT DO NOTHING;
-- ReadOnly gets only read permission for backup
INSERT INTO role_permissions (role_id, permission_id)
SELECT r.id, p.id
FROM roles r, permissions p
WHERE r.name = 'readonly'
AND p.resource = 'backup'
AND p.action = 'read'
ON CONFLICT DO NOTHING;

209
dist/airgap/calypso-appliance-1.0.0-airgap/migrations/011_sync_bacula_jobs_function.sql vendored Normal file
View File

@@ -0,0 +1,209 @@
-- AtlasOS - Calypso
-- PostgreSQL Function to Sync Jobs from Bacula to Calypso
-- Version: 11.0
--
-- This function syncs jobs from Bacula database (Job table) to Calypso database (backup_jobs table)
-- Uses dblink extension to query Bacula database from Calypso database
--
-- Prerequisites:
-- 1. dblink extension must be installed: CREATE EXTENSION IF NOT EXISTS dblink;
-- 2. User must have access to both databases
-- 3. Connection parameters must be configured in the function
-- Create function to sync jobs from Bacula to Calypso
CREATE OR REPLACE FUNCTION sync_bacula_jobs(
bacula_db_name TEXT DEFAULT 'bacula',
bacula_host TEXT DEFAULT 'localhost',
bacula_port INTEGER DEFAULT 5432,
bacula_user TEXT DEFAULT 'calypso',
bacula_password TEXT DEFAULT ''
)
RETURNS TABLE(
jobs_synced INTEGER,
jobs_inserted INTEGER,
jobs_updated INTEGER,
errors INTEGER
) AS $$
DECLARE
conn_str TEXT;
jobs_count INTEGER := 0;
inserted_count INTEGER := 0;
updated_count INTEGER := 0;
error_count INTEGER := 0;
job_record RECORD;
BEGIN
-- Build dblink connection string
conn_str := format(
'dbname=%s host=%s port=%s user=%s password=%s',
bacula_db_name,
bacula_host,
bacula_port,
bacula_user,
bacula_password
);
-- Query jobs from Bacula database using dblink
FOR job_record IN
SELECT * FROM dblink(
conn_str,
$QUERY$
SELECT
j.JobId,
j.Name as job_name,
COALESCE(c.Name, 'unknown') as client_name,
CASE
WHEN j.Type = 'B' THEN 'Backup'
WHEN j.Type = 'R' THEN 'Restore'
WHEN j.Type = 'V' THEN 'Verify'
WHEN j.Type = 'C' THEN 'Copy'
WHEN j.Type = 'M' THEN 'Migrate'
ELSE 'Backup'
END as job_type,
CASE
WHEN j.Level = 'F' THEN 'Full'
WHEN j.Level = 'I' THEN 'Incremental'
WHEN j.Level = 'D' THEN 'Differential'
WHEN j.Level = 'S' THEN 'Since'
ELSE 'Full'
END as job_level,
CASE
WHEN j.JobStatus = 'T' THEN 'Running'
WHEN j.JobStatus = 'C' THEN 'Completed'
WHEN j.JobStatus = 'f' OR j.JobStatus = 'F' THEN 'Failed'
WHEN j.JobStatus = 'A' THEN 'Canceled'
WHEN j.JobStatus = 'W' THEN 'Waiting'
ELSE 'Waiting'
END as status,
COALESCE(j.JobBytes, 0) as bytes_written,
COALESCE(j.JobFiles, 0) as files_written,
j.StartTime as started_at,
j.EndTime as ended_at,
CASE
WHEN j.EndTime IS NOT NULL AND j.StartTime IS NOT NULL
THEN EXTRACT(EPOCH FROM (j.EndTime - j.StartTime))::INTEGER
ELSE NULL
END as duration_seconds
FROM Job j
LEFT JOIN Client c ON j.ClientId = c.ClientId
ORDER BY j.StartTime DESC
LIMIT 1000
$QUERY$
) AS t(
job_id INTEGER,
job_name TEXT,
client_name TEXT,
job_type TEXT,
job_level TEXT,
status TEXT,
bytes_written BIGINT,
files_written INTEGER,
started_at TIMESTAMP,
ended_at TIMESTAMP,
duration_seconds INTEGER
)
LOOP
BEGIN
-- Check if job already exists (before insert/update)
IF EXISTS (SELECT 1 FROM backup_jobs WHERE job_id = job_record.job_id) THEN
updated_count := updated_count + 1;
ELSE
inserted_count := inserted_count + 1;
END IF;
-- Upsert job to backup_jobs table
INSERT INTO backup_jobs (
job_id, job_name, client_name, job_type, job_level, status,
bytes_written, files_written, started_at, ended_at, duration_seconds,
updated_at
) VALUES (
job_record.job_id,
job_record.job_name,
job_record.client_name,
job_record.job_type,
job_record.job_level,
job_record.status,
job_record.bytes_written,
job_record.files_written,
job_record.started_at,
job_record.ended_at,
job_record.duration_seconds,
NOW()
)
ON CONFLICT (job_id) DO UPDATE SET
job_name = EXCLUDED.job_name,
client_name = EXCLUDED.client_name,
job_type = EXCLUDED.job_type,
job_level = EXCLUDED.job_level,
status = EXCLUDED.status,
bytes_written = EXCLUDED.bytes_written,
files_written = EXCLUDED.files_written,
started_at = EXCLUDED.started_at,
ended_at = EXCLUDED.ended_at,
duration_seconds = EXCLUDED.duration_seconds,
updated_at = NOW();
jobs_count := jobs_count + 1;
EXCEPTION
WHEN OTHERS THEN
error_count := error_count + 1;
-- Log error but continue with next job
RAISE WARNING 'Error syncing job %: %', job_record.job_id, SQLERRM;
END;
END LOOP;
-- Return summary
RETURN QUERY SELECT jobs_count, inserted_count, updated_count, error_count;
END;
$$ LANGUAGE plpgsql;
-- Create a simpler version that uses current database connection settings
-- This version assumes Bacula is on same host/port with same user
CREATE OR REPLACE FUNCTION sync_bacula_jobs_simple()
RETURNS TABLE(
jobs_synced INTEGER,
jobs_inserted INTEGER,
jobs_updated INTEGER,
errors INTEGER
) AS $$
DECLARE
current_user_name TEXT;
current_host TEXT;
current_port INTEGER;
current_db TEXT;
BEGIN
-- Get current connection info
SELECT
current_user,
COALESCE(inet_server_addr()::TEXT, 'localhost'),
COALESCE(inet_server_port(), 5432),
current_database()
INTO
current_user_name,
current_host,
current_port,
current_db;
-- Call main function with current connection settings
-- Note: password needs to be passed or configured in .pgpass
RETURN QUERY
SELECT * FROM sync_bacula_jobs(
'bacula', -- Try 'bacula' first
current_host,
current_port,
current_user_name,
'' -- Empty password - will use .pgpass or peer authentication
);
END;
$$ LANGUAGE plpgsql;
-- Grant execute permission to calypso user
GRANT EXECUTE ON FUNCTION sync_bacula_jobs(TEXT, TEXT, INTEGER, TEXT, TEXT) TO calypso;
GRANT EXECUTE ON FUNCTION sync_bacula_jobs_simple() TO calypso;
-- Create index if not exists (should already exist from migration 009)
CREATE INDEX IF NOT EXISTS idx_backup_jobs_job_id ON backup_jobs(job_id);
CREATE INDEX IF NOT EXISTS idx_backup_jobs_updated_at ON backup_jobs(updated_at);
COMMENT ON FUNCTION sync_bacula_jobs IS 'Syncs jobs from Bacula database to Calypso backup_jobs table using dblink';
COMMENT ON FUNCTION sync_bacula_jobs_simple IS 'Simplified version that uses current connection settings (requires .pgpass for password)';

209
dist/airgap/calypso-appliance-1.0.0-airgap/migrations/011_sync_bacula_jobs_function.sql.bak vendored Normal file
View File

@@ -0,0 +1,209 @@
-- AtlasOS - Calypso
-- PostgreSQL Function to Sync Jobs from Bacula to Calypso
-- Version: 11.0
--
-- This function syncs jobs from Bacula database (Job table) to Calypso database (backup_jobs table)
-- Uses dblink extension to query Bacula database from Calypso database
--
-- Prerequisites:
-- 1. dblink extension must be installed: CREATE EXTENSION IF NOT EXISTS dblink;
-- 2. User must have access to both databases
-- 3. Connection parameters must be configured in the function
-- Create function to sync jobs from Bacula to Calypso
CREATE OR REPLACE FUNCTION sync_bacula_jobs(
bacula_db_name TEXT DEFAULT 'bacula',
bacula_host TEXT DEFAULT 'localhost',
bacula_port INTEGER DEFAULT 5432,
bacula_user TEXT DEFAULT 'calypso',
bacula_password TEXT DEFAULT ''
)
RETURNS TABLE(
jobs_synced INTEGER,
jobs_inserted INTEGER,
jobs_updated INTEGER,
errors INTEGER
) AS $$
DECLARE
conn_str TEXT;
jobs_count INTEGER := 0;
inserted_count INTEGER := 0;
updated_count INTEGER := 0;
error_count INTEGER := 0;
job_record RECORD;
BEGIN
-- Build dblink connection string
conn_str := format(
'dbname=%s host=%s port=%s user=%s password=%s',
bacula_db_name,
bacula_host,
bacula_port,
bacula_user,
bacula_password
);
-- Query jobs from Bacula database using dblink
FOR job_record IN
SELECT * FROM dblink(
conn_str,
$$
SELECT
j.JobId,
j.Name as job_name,
COALESCE(c.Name, 'unknown') as client_name,
CASE
WHEN j.Type = 'B' THEN 'Backup'
WHEN j.Type = 'R' THEN 'Restore'
WHEN j.Type = 'V' THEN 'Verify'
WHEN j.Type = 'C' THEN 'Copy'
WHEN j.Type = 'M' THEN 'Migrate'
ELSE 'Backup'
END as job_type,
CASE
WHEN j.Level = 'F' THEN 'Full'
WHEN j.Level = 'I' THEN 'Incremental'
WHEN j.Level = 'D' THEN 'Differential'
WHEN j.Level = 'S' THEN 'Since'
ELSE 'Full'
END as job_level,
CASE
WHEN j.JobStatus = 'T' THEN 'Running'
WHEN j.JobStatus = 'C' THEN 'Completed'
WHEN j.JobStatus = 'f' OR j.JobStatus = 'F' THEN 'Failed'
WHEN j.JobStatus = 'A' THEN 'Canceled'
WHEN j.JobStatus = 'W' THEN 'Waiting'
ELSE 'Waiting'
END as status,
COALESCE(j.JobBytes, 0) as bytes_written,
COALESCE(j.JobFiles, 0) as files_written,
j.StartTime as started_at,
j.EndTime as ended_at,
CASE
WHEN j.EndTime IS NOT NULL AND j.StartTime IS NOT NULL
THEN EXTRACT(EPOCH FROM (j.EndTime - j.StartTime))::INTEGER
ELSE NULL
END as duration_seconds
FROM Job j
LEFT JOIN Client c ON j.ClientId = c.ClientId
ORDER BY j.StartTime DESC
LIMIT 1000
$$
) AS t(
job_id INTEGER,
job_name TEXT,
client_name TEXT,
job_type TEXT,
job_level TEXT,
status TEXT,
bytes_written BIGINT,
files_written INTEGER,
started_at TIMESTAMP,
ended_at TIMESTAMP,
duration_seconds INTEGER
)
LOOP
BEGIN
-- Check if job already exists (before insert/update)
IF EXISTS (SELECT 1 FROM backup_jobs WHERE job_id = job_record.job_id) THEN
updated_count := updated_count + 1;
ELSE
inserted_count := inserted_count + 1;
END IF;
-- Upsert job to backup_jobs table
INSERT INTO backup_jobs (
job_id, job_name, client_name, job_type, job_level, status,
bytes_written, files_written, started_at, ended_at, duration_seconds,
updated_at
) VALUES (
job_record.job_id,
job_record.job_name,
job_record.client_name,
job_record.job_type,
job_record.job_level,
job_record.status,
job_record.bytes_written,
job_record.files_written,
job_record.started_at,
job_record.ended_at,
job_record.duration_seconds,
NOW()
)
ON CONFLICT (job_id) DO UPDATE SET
job_name = EXCLUDED.job_name,
client_name = EXCLUDED.client_name,
job_type = EXCLUDED.job_type,
job_level = EXCLUDED.job_level,
status = EXCLUDED.status,
bytes_written = EXCLUDED.bytes_written,
files_written = EXCLUDED.files_written,
started_at = EXCLUDED.started_at,
ended_at = EXCLUDED.ended_at,
duration_seconds = EXCLUDED.duration_seconds,
updated_at = NOW();
jobs_count := jobs_count + 1;
EXCEPTION
WHEN OTHERS THEN
error_count := error_count + 1;
-- Log error but continue with next job
RAISE WARNING 'Error syncing job %: %', job_record.job_id, SQLERRM;
END;
END LOOP;
-- Return summary
RETURN QUERY SELECT jobs_count, inserted_count, updated_count, error_count;
END;
$$ LANGUAGE plpgsql;
-- Create a simpler version that uses current database connection settings
-- This version assumes Bacula is on same host/port with same user
CREATE OR REPLACE FUNCTION sync_bacula_jobs_simple()
RETURNS TABLE(
jobs_synced INTEGER,
jobs_inserted INTEGER,
jobs_updated INTEGER,
errors INTEGER
) AS $$
DECLARE
current_user_name TEXT;
current_host TEXT;
current_port INTEGER;
current_db TEXT;
BEGIN
-- Get current connection info
SELECT
current_user,
COALESCE(inet_server_addr()::TEXT, 'localhost'),
COALESCE(inet_server_port(), 5432),
current_database()
INTO
current_user_name,
current_host,
current_port,
current_db;
-- Call main function with current connection settings
-- Note: password needs to be passed or configured in .pgpass
RETURN QUERY
SELECT * FROM sync_bacula_jobs(
'bacula', -- Try 'bacula' first
current_host,
current_port,
current_user_name,
'' -- Empty password - will use .pgpass or peer authentication
);
END;
$$ LANGUAGE plpgsql;
-- Grant execute permission to calypso user
GRANT EXECUTE ON FUNCTION sync_bacula_jobs(TEXT, TEXT, INTEGER, TEXT, TEXT) TO calypso;
GRANT EXECUTE ON FUNCTION sync_bacula_jobs_simple() TO calypso;
-- Create index if not exists (should already exist from migration 009)
CREATE INDEX IF NOT EXISTS idx_backup_jobs_job_id ON backup_jobs(job_id);
CREATE INDEX IF NOT EXISTS idx_backup_jobs_updated_at ON backup_jobs(updated_at);
COMMENT ON FUNCTION sync_bacula_jobs IS 'Syncs jobs from Bacula database to Calypso backup_jobs table using dblink';
COMMENT ON FUNCTION sync_bacula_jobs_simple IS 'Simplified version that uses current connection settings (requires .pgpass for password)';

23
dist/airgap/calypso-appliance-1.0.0-airgap/migrations/012_add_snapshot_schedules_table.sql vendored Normal file
View File

@@ -0,0 +1,23 @@
-- Snapshot schedules table for automated snapshot creation
CREATE TABLE IF NOT EXISTS snapshot_schedules (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL,
dataset VARCHAR(255) NOT NULL,
snapshot_name_template VARCHAR(255) NOT NULL, -- e.g., "auto-%Y-%m-%d-%H%M" or "daily-backup"
schedule_type VARCHAR(50) NOT NULL, -- 'hourly', 'daily', 'weekly', 'monthly', 'cron'
schedule_config JSONB NOT NULL, -- For cron: {"cron": "0 0 * * *"}, for others: {"time": "00:00", "day": 1, etc.}
recursive BOOLEAN NOT NULL DEFAULT false,
enabled BOOLEAN NOT NULL DEFAULT true,
retention_count INTEGER, -- Keep last N snapshots (null = unlimited)
retention_days INTEGER, -- Keep snapshots for N days (null = unlimited)
last_run_at TIMESTAMP,
next_run_at TIMESTAMP,
created_by UUID REFERENCES users(id),
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
UNIQUE(name)
);
CREATE INDEX IF NOT EXISTS idx_snapshot_schedules_enabled ON snapshot_schedules(enabled);
CREATE INDEX IF NOT EXISTS idx_snapshot_schedules_next_run ON snapshot_schedules(next_run_at);
CREATE INDEX IF NOT EXISTS idx_snapshot_schedules_dataset ON snapshot_schedules(dataset);

76
dist/airgap/calypso-appliance-1.0.0-airgap/migrations/013_add_replication_tasks_table.sql vendored Normal file
View File

@@ -0,0 +1,76 @@
-- ZFS Replication Tasks Table
-- Supports both outbound (sender) and inbound (receiver) replication
CREATE TABLE IF NOT EXISTS replication_tasks (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255) NOT NULL,
direction VARCHAR(20) NOT NULL, -- 'outbound' (sender) or 'inbound' (receiver)
-- For outbound replication (sender)
source_dataset VARCHAR(512), -- Source dataset on this system (outbound) or remote system (inbound)
target_host VARCHAR(255), -- Target host IP or hostname (for outbound)
target_port INTEGER DEFAULT 22, -- SSH port (default 22, for outbound)
target_user VARCHAR(255) DEFAULT 'root', -- SSH user (for outbound)
target_dataset VARCHAR(512), -- Target dataset on remote system (for outbound)
target_ssh_key_path TEXT, -- Path to SSH private key (for outbound)
-- For inbound replication (receiver)
source_host VARCHAR(255), -- Source host IP or hostname (for inbound)
source_port INTEGER DEFAULT 22, -- SSH port (for inbound)
source_user VARCHAR(255) DEFAULT 'root', -- SSH user (for inbound)
local_dataset VARCHAR(512), -- Local dataset to receive snapshots (for inbound)
-- Common settings
schedule_type VARCHAR(50), -- 'manual', 'hourly', 'daily', 'weekly', 'monthly', 'cron'
schedule_config JSONB, -- Schedule configuration (similar to snapshot schedules)
compression VARCHAR(50) DEFAULT 'lz4', -- 'off', 'lz4', 'gzip', 'zstd'
encryption BOOLEAN DEFAULT false, -- Enable encryption during transfer
recursive BOOLEAN DEFAULT false, -- Replicate recursively
incremental BOOLEAN DEFAULT true, -- Use incremental replication
auto_snapshot BOOLEAN DEFAULT true, -- Auto-create snapshot before replication
-- Status and tracking
enabled BOOLEAN NOT NULL DEFAULT true,
status VARCHAR(50) DEFAULT 'idle', -- 'idle', 'running', 'failed', 'paused'
last_run_at TIMESTAMP,
last_run_status VARCHAR(50), -- 'success', 'failed', 'partial'
last_run_error TEXT,
next_run_at TIMESTAMP,
last_snapshot_sent VARCHAR(512), -- Last snapshot successfully sent (for outbound)
last_snapshot_received VARCHAR(512), -- Last snapshot successfully received (for inbound)
-- Statistics
total_runs INTEGER DEFAULT 0,
successful_runs INTEGER DEFAULT 0,
failed_runs INTEGER DEFAULT 0,
bytes_sent BIGINT DEFAULT 0, -- Total bytes sent (for outbound)
bytes_received BIGINT DEFAULT 0, -- Total bytes received (for inbound)
created_by UUID REFERENCES users(id),
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
-- Validation: ensure required fields based on direction
CONSTRAINT chk_direction CHECK (direction IN ('outbound', 'inbound')),
CONSTRAINT chk_outbound_fields CHECK (
direction != 'outbound' OR (
source_dataset IS NOT NULL AND
target_host IS NOT NULL AND
target_dataset IS NOT NULL
)
),
CONSTRAINT chk_inbound_fields CHECK (
direction != 'inbound' OR (
source_host IS NOT NULL AND
source_dataset IS NOT NULL AND
local_dataset IS NOT NULL
)
)
);
-- Create indexes
CREATE INDEX IF NOT EXISTS idx_replication_tasks_direction ON replication_tasks(direction);
CREATE INDEX IF NOT EXISTS idx_replication_tasks_enabled ON replication_tasks(enabled);
CREATE INDEX IF NOT EXISTS idx_replication_tasks_status ON replication_tasks(status);
CREATE INDEX IF NOT EXISTS idx_replication_tasks_next_run ON replication_tasks(next_run_at);
CREATE INDEX IF NOT EXISTS idx_replication_tasks_source_dataset ON replication_tasks(source_dataset);
CREATE INDEX IF NOT EXISTS idx_replication_tasks_target_host ON replication_tasks(target_host);

30
dist/airgap/calypso-appliance-1.0.0-airgap/migrations/014_add_client_categories_table.sql vendored Normal file
View File

@@ -0,0 +1,30 @@
-- AtlasOS - Calypso
-- Client Categories Schema
-- Version: 14.0
-- Adds category support for backup clients (File, Database, Virtual)
-- Client metadata table to store additional information about clients
-- This extends the Bacula Client table with Calypso-specific metadata
CREATE TABLE IF NOT EXISTS client_metadata (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
client_id INTEGER NOT NULL, -- Bacula Client.ClientId
client_name VARCHAR(255) NOT NULL, -- Bacula Client.Name (for reference)
category VARCHAR(50) NOT NULL DEFAULT 'File', -- 'File', 'Database', 'Virtual'
description TEXT,
tags JSONB, -- Additional tags/metadata as JSON
created_at TIMESTAMP NOT NULL DEFAULT NOW(),
updated_at TIMESTAMP NOT NULL DEFAULT NOW(),
-- Ensure one metadata entry per client
CONSTRAINT unique_client_id UNIQUE (client_id),
CONSTRAINT chk_category CHECK (category IN ('File', 'Database', 'Virtual'))
);
-- Indexes for performance
CREATE INDEX IF NOT EXISTS idx_client_metadata_client_id ON client_metadata(client_id);
CREATE INDEX IF NOT EXISTS idx_client_metadata_client_name ON client_metadata(client_name);
CREATE INDEX IF NOT EXISTS idx_client_metadata_category ON client_metadata(category);
-- Add comment
COMMENT ON TABLE client_metadata IS 'Stores Calypso-specific metadata for backup clients, including category classification';
COMMENT ON COLUMN client_metadata.category IS 'Client category: File (file system backups), Database (database backups), Virtual (virtual machine backups)';

44
dist/airgap/calypso-appliance-1.0.0-airgap/migrations/015_add_bacula_clients_table.sql vendored Normal file
View File

@@ -0,0 +1,44 @@
-- AtlasOS - Calypso
-- Migration 015: Add Bacula clients and capability history tables
--
-- Adds tables for tracking registered Bacula agents, their backup capabilities,
-- and a history log for UI- or agent-triggered capability changes. Pending
-- updates are stored on the client row until the agent pulls them.
CREATE TABLE IF NOT EXISTS bacula_clients (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
hostname TEXT NOT NULL,
ip_address TEXT,
agent_version TEXT,
status TEXT NOT NULL DEFAULT 'online',
backup_types JSONB NOT NULL,
pending_backup_types JSONB,
pending_requested_by UUID,
pending_requested_at TIMESTAMPTZ,
pending_notes TEXT,
metadata JSONB,
registered_by_user_id UUID NOT NULL,
last_seen TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT uniq_bacula_clients_hostname UNIQUE (hostname)
);
CREATE INDEX IF NOT EXISTS idx_bacula_clients_registered_by ON bacula_clients (registered_by_user_id);
CREATE INDEX IF NOT EXISTS idx_bacula_clients_status ON bacula_clients (status);
CREATE TABLE IF NOT EXISTS bacula_client_capability_history (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
client_id UUID NOT NULL REFERENCES bacula_clients (id) ON DELETE CASCADE,
backup_types JSONB NOT NULL,
source TEXT NOT NULL,
requested_by_user_id UUID,
requested_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
notes TEXT
);
CREATE INDEX IF NOT EXISTS idx_bacula_client_capability_history_client ON bacula_client_capability_history (client_id);
CREATE INDEX IF NOT EXISTS idx_bacula_client_capability_history_requested_at ON bacula_client_capability_history (requested_at);
COMMENT ON TABLE bacula_clients IS 'Tracks Bacula clients registered with Calypso, including pending capability pushes.';
COMMENT ON TABLE bacula_client_capability_history IS 'Audit history of backup capability changes per client.';

BIN
dist/airgap/calypso-appliance-1.0.0-airgap/packages/debs/Packages.gz vendored Normal file
View File

Binary file not shown.

BIN
dist/airgap/calypso-appliance-1.0.0-airgap/packages/debs/adduser_3.137ubuntu1_all.deb vendored Normal file
View File

Binary file not shown.

BIN
dist/airgap/calypso-appliance-1.0.0-airgap/packages/debs/apt-utils_2.8.3_amd64.deb vendored Normal file
View File

Binary file not shown.

BIN
dist/airgap/calypso-appliance-1.0.0-airgap/packages/debs/apt_2.8.3_amd64.deb vendored Normal file
View File

Binary file not shown.

BIN
dist/airgap/calypso-appliance-1.0.0-airgap/packages/debs/attr_1%3a2.5.2-1build1.1_amd64.deb vendored Normal file
View File

Binary file not shown.

BIN
dist/airgap/calypso-appliance-1.0.0-airgap/packages/debs/bacula-bscan_13.0.4-1build3_amd64.deb vendored Normal file
View File

Binary file not shown.

BIN
dist/airgap/calypso-appliance-1.0.0-airgap/packages/debs/bacula-client_13.0.4-1build3_all.deb vendored Normal file
View File

Binary file not shown.

BIN
dist/airgap/calypso-appliance-1.0.0-airgap/packages/debs/bacula-common-sqlite3_13.0.4-1build3_amd64.deb vendored Normal file
View File

Binary file not shown.

BIN
dist/airgap/calypso-appliance-1.0.0-airgap/packages/debs/bacula-common_13.0.4-1build3_amd64.deb vendored Normal file
View File

Binary file not shown.

BIN
dist/airgap/calypso-appliance-1.0.0-airgap/packages/debs/bacula-console_13.0.4-1build3_amd64.deb vendored Normal file
View File

Binary file not shown.

BIN
dist/airgap/calypso-appliance-1.0.0-airgap/packages/debs/bacula-fd_13.0.4-1build3_amd64.deb vendored Normal file
View File

Binary file not shown.

BIN
dist/airgap/calypso-appliance-1.0.0-airgap/packages/debs/bacula-sd_13.0.4-1build3_amd64.deb vendored Normal file
View File

Binary file not shown.

BIN
dist/airgap/calypso-appliance-1.0.0-airgap/packages/debs/binutils-common_2.42-4ubuntu2.8_amd64.deb vendored Normal file
View File

Binary file not shown.

BIN
dist/airgap/calypso-appliance-1.0.0-airgap/packages/debs/binutils-x86-64-linux-gnu_2.42-4ubuntu2.8_amd64.deb vendored Normal file
View File

Binary file not shown.

Some files were not shown because too many files have changed in this diff Show More