Operations and recovery

Supervise the worker

The terminal and dashboard are clients. Live/shadow safety requires the dedicated jorli-worker host under an OS supervisor after setup and broker validation. jorli worker run remains the foreground diagnostic form.

# Linux systemd user service or macOS LaunchAgent
JORLI_WORKER_BIN="$(command -v jorli-worker)" ./install/install-worker.sh
./install/install-worker.sh --uninstall

# Elevated PowerShell: native automatic Windows Service
./install/install-worker.ps1 -WorkerPath (Get-Command jorli-worker.exe).Source
./install/install-worker.ps1 -Uninstall

Use --check / -Check to validate a definition and the worker version without registration. Linux attempts user lingering and warns when administrator action is needed. Windows installation prompts for the current user’s service credential because that identity owns ~/.jorli and DPAPI secrets; never install it as LocalSystem. The service has bounded restart recovery. macOS worker logs live under ~/.jorli/logs; the native host emits structured JSON on systemd/Windows.

Pair Jorli monitoring

On the trusted execution host, pair once using the Convex HTTP-actions site URL (the .site URL, not the .cloud client URL):

jorli remote pair --endpoint https://your-deployment.convex.site --name trading-desk
jorli remote status
jorli worker run

The Jorli session token is entered in a secret prompt and exchanged only during pairing. The agent then uses a random 256-bit device credential from the OS secret store. Every poll advances a persisted monotonic sequence, so a captured request cannot be replayed. Web and mobile expose worker/deployment health, positions, recent orders/fills, alerts, completed-trade review, pause, and emergency kill. A command is not complete while it is pending; completed includes state observed on the execution host. Live creation/activation, strategy/risk edits, credentials, risk increases, and kill clearing remain local-only.

Production preflight

1. jorli doctor must show compatible Kite/Dhan binaries, usable SQLite, and a fresh heartbeat.
2. Authenticate the selected account. Dhan live use also needs Data API and static-IP readiness.
3. Run a broker-source backtest, approve that immutable version, then run shadow through restart.
4. Verify dashboard risk usage and compare broker positions/orders/trades.
5. Complete minimum-quantity entry, stop, coordinated exit, and restart smoke for that account.

Backup and restore

Pause deployments and stop the worker before copying ~/.jorli. For online backup use SQLite’s .backup command; never copy only one WAL/database file. Keep owner-only permissions and never export broker keyrings.

jorli backup [--output /secure/path/jorli.jbak]
jorli kill --yes
# Pause deployments, stop the supervised worker, and wait for its heartbeat to become stale.
jorli restore /secure/path/jorli.jbak --yes
jorli export [--output /secure/path/jorli.json]

backup first uses SQLite’s online backup API and integrity check. It then writes an authenticated JORLIBAK2 archive using AES-256-CBC plus HMAC-SHA256 with independent keys derived by PBKDF2-SHA256 (600,000 iterations). Enter the passphrase through the secret prompt; automation can use JORLI_BACKUP_PASSPHRASE, but command arguments are rejected. Losing the passphrase makes the archive unrecoverable.

Restore verifies HMAC before decrypting, checks SQLite integrity, creates an encrypted pre-restore safety backup, atomically swaps the database with rollback, and reruns migrations. It refuses to run without kill, paused deployments, a stale worker heartbeat, explicit --yes, and exclusive execution host ownership. Kill stays active. Compare local intents with broker orders/trades/positions, start the worker so reconciliation runs, and clear kill only after no live entry is unresolved. export remains unencrypted and excludes configuration, conversations, AI keys, and broker credentials.

Database integrity and migrations

jorli database check
jorli database check --full
jorli database migrations

Doctor runs quick_check and verifies the schema version. The worker repeats quick-check once per IST day, audits the result, and activates global kill if it is unhealthy. Schema creation, additive column changes, migration-history insertion, and version advance commit in one transaction. A database from a newer agent fails closed; Jorli never attempts an automatic downgrade.

Before upgrading, create and verify an encrypted .jbak. Migration history records the from/to versions and rollback instruction. Rollback means stopping the worker, activating kill, and using jorli restore with the pre-upgrade archive and the prior compatible binary. Do not edit schema_version or manufacture migration rows.

Incident-support bundle

jorli support bundle [--output /secure/path/jorli-support.zip]

The owner-only bundle contains health, integrity/migration state, safe configuration, deployments, recent intents/audit events, and at most the last 1 MB from each of 20 logs. It excludes the raw database, strategy YAML, conversations, and every secret store. Secret fields are removed; account, broker-order, device, email, and phone identifiers are hashed or redacted; free text strips home paths, bearer/JWT tokens, credential assignments, and emails. manifest.json records every entry’s byte length and SHA-256. Redaction is defense in depth—always inspect the archive before sharing.

Scheduled recovery drills

jorli drill configure /secure/path/jorli.jbak --days 7
jorli drill status
jorli drill run

Configuration asks for the archive passphrase through a secret prompt, proves the archive immediately, and stores the passphrase only in the OS credential store. Every 1–90 days, the worker authenticates and decrypts that archive to an ephemeral file, runs full SQLite integrity and schema compatibility checks, records the archive SHA-256, then deletes the isolated database. It never replaces, opens for writing, or migrates live state. Doctor fails on the latest drill error and warns when a drill is unconfigured or overdue. Replace the configured archive after backup rotation.

Clock synchronization

jorli clock status
jorli clock check

The worker compares the host with three NTP v4 references every 15 minutes and persists the median signed drift. Responses are bounded and validated for server mode and stratum. Doctor warns when the check is missing or stale and fails when no source responds or drift exceeds two seconds. An unsafe clock blocks new live entries without disabling reconciliation, protective orders, or exits. Fix the host time service and run jorli clock check; do not edit the stored health state.

Exchange holidays and broker maintenance

jorli calendar status --broker kite
jorli calendar list --year 2026
jorli calendar maintenance add --broker dhan --from 2026-07-13T18:00:00+05:30 --to 2026-07-13T20:00:00+05:30 --reason "Broker notice"
jorli calendar maintenance remove maint-abc123

The built-in 2026 equity/derivatives holiday set is source/version stamped and includes later official closure updates. Calendar coverage expires explicitly after 31 December 2026 so an old binary cannot assume future dates are tradable. Maintenance windows are bounded, persisted, audited, and scoped to all brokers, Kite, or Dhan. Weekends, holidays, expired coverage, and applicable maintenance block new live entries while reconciliation, protection, and exits remain available.

Retention and audit export

jorli retention preview
jorli retention apply --yes
jorli audit export --from 2026-01-01T00:00:00+05:30 --output /secure/path/audit.json

Retention defaults remove reproducible candle cache after 730 days, logs after 30 days, and generated exports after 90 days. Apply requires explicit confirmation and records an audit event. Audit events, strategies, deployments, orders/fills, attribution, approvals, and encrypted backups are never automatically deleted. Audit export writes the complete selected date range to an owner-only document; each canonical event is SHA-256 chained and the final root hash supports independent verification.

Observability and alerts

jorli observe status
jorli observe events --take 100
jorli observe alerts
jorli observe ack <alert-id>

The worker publishes .NET meter counters/histograms and trace spans while persisting tick totals, failures, duration, structured events, and trace/span correlation. Alert payloads redact common credential and email patterns, deduplicate repeated incidents, and retain recurrence counts. Alerts always enter the immutable local audit and also route through authenticated remote snapshots when the host is paired. Acknowledgement is audited; acknowledge only after remediation. Doctor fails on open critical alerts and warns on open errors.

Trusted automatic upgrades

jorli update configure --manifest https://releases.example/manifest.json \
  --public-key /secure/release-public-key.pem --install-dir /home/user/.local/bin --automatic
jorli update status
jorli update check
jorli update apply --yes

The trusted ECDSA key verifies a canonical manifest covering the version, schema compatibility, and SHA-256 digest of both CLI and worker artifacts. Downloads require HTTPS. The rollback passphrase is secret-prompted into the OS credential store. Before replacement, Jorli preserves both old binaries and an authenticated encrypted database archive; it then atomically swaps the pair, probes their versions, and runs the new CLI’s full migration/integrity check. Any failure restores binaries and database and raises a critical alert. A successful supervised-startup update exits for native-service restart. Active deployments require global kill plus pause. Enable automatic mode only for artifacts signed by the configured production release key.

Primary and standby workers

Pair each host independently, restore the same approved state, and configure the same lease name:

jorli worker configure --role primary --lease trading-desk
jorli worker configure --role standby --lease trading-desk
jorli worker status

The authenticated remote control plane atomically grants one holder and issues a higher fencing token after every release/expiry promotion. Lease requests use their own monotonic anti-replay sequence. The default 120-second lease blocks new risk 45 seconds before expiry if renewal is unavailable. A standby performs no broker mutation or strategy evaluation while waiting; after acquisition it must reconcile authoritative broker state before its token becomes execution-ready. Every live entry checks holder, reconciliation, and remaining lease duration. Graceful shutdown releases immediately; crashes rely on expiry. Agent Monitor shows holder/token state. Run a controlled operator-owned failover drill before relying on standby execution.

Incident essentials

• Ambiguous mutation: never retry; activate kill, inspect correlation id in orders/trades, and reconcile positions.
• Protective failure: treat as critical; verify filled quantity and add broker-side protection or flatten deliberately before clearing kill.
• Changed account/auth: never rebind an approval; reauthenticate the bound account or create a new deployment after flatness.
• Database/host loss: manage exposure at the broker, restore a consistent backup, and never manufacture approval/fill/protection states in SQLite.
• Credential exposure: revoke at the AI/broker owner, stop the worker, preserve sanitized evidence, and inspect logs/transcripts.