Added Git-Sync Mirror Agent to fold-stack with support for GitHub, Forgejo, Radicle, Internet Archive, and Web3.storage
This commit is contained in:
@@ -17,6 +17,7 @@ Fold-Stack provides a modular, self-contained environment for:
|
||||
- **Data Replication**: Rclone for syncing data to Google Drive, Internet Archive, and Web3.storage.
|
||||
- **Document Compilation**: Typst for fast, modern document creation.
|
||||
- **LaTeX Collaboration**: Overleaf CE for collaborative LaTeX editing.
|
||||
- **Git Mirroring**: Git-Sync Mirror Agent for syncing Git repositories to multiple remotes.
|
||||
|
||||
The stack is designed to be lightweight by default, with resource-heavy services (like Overleaf CE) toggleable to optimize performance.
|
||||
|
||||
@@ -37,6 +38,7 @@ Before setting up Fold-Stack, ensure you have the following:
|
||||
- Google Drive (for `gdrive` remote).
|
||||
- Internet Archive (for `ia` remote).
|
||||
- Web3.storage (for `web3` remote, requires an API token).
|
||||
- SSH keys for GitHub and Forgejo (for Git-Sync).
|
||||
|
||||
---
|
||||
|
||||
@@ -121,7 +123,48 @@ Fold-Stack uses Rclone to replicate data to Google Drive, Internet Archive, and
|
||||
\`\`\`
|
||||
You should see: `gdrive:`, `ia:`, `nextcloud:`, `web3:`.
|
||||
|
||||
### 4. Start the Stack
|
||||
### 4. Configure Git-Sync Mirror Agent
|
||||
|
||||
The Git-Sync Mirror Agent syncs a local Git repository to multiple remotes (GitHub, Forgejo, Radicle, Internet Archive, and optionally Web3.storage).
|
||||
|
||||
1. **Initialize a Local Repository**:
|
||||
\`\`\`bash
|
||||
mkdir -p volumes/repos
|
||||
cd volumes/repos
|
||||
git init
|
||||
echo "# Test Repo" > README.md
|
||||
git add .
|
||||
git commit -m "Initial commit"
|
||||
git branch -M main
|
||||
\`\`\`
|
||||
|
||||
2. **Set Up SSH Keys for GitHub and Forgejo**:
|
||||
- Generate SSH keys if you don’t already have them:
|
||||
\`\`\`bash
|
||||
ssh-keygen -t ed25519 -C "your_email@example.com" -f ~/.ssh/github_key
|
||||
ssh-keygen -t ed25519 -C "your_email@example.com" -f ~/.ssh/forgejo_key
|
||||
\`\`\`
|
||||
- Add the public keys to GitHub and Forgejo:
|
||||
- GitHub: Add `~/.ssh/github_key.pub` to your GitHub account (Settings > SSH and GPG keys).
|
||||
- Forgejo: Add `~/.ssh/forgejo_key.pub` to your Forgejo account (http://localhost:3000/user/settings/keys).
|
||||
- Copy the private keys to the `git-sync` secrets directory:
|
||||
\`\`\`bash
|
||||
cp ~/.ssh/github_key config/git-sync/secrets/github.key
|
||||
cp ~/.ssh/forgejo_key config/git-sync/secrets/forgejo.key
|
||||
chmod 600 config/git-sync/secrets/github.key config/git-sync/secrets/forgejo.key
|
||||
\`\`\`
|
||||
|
||||
3. **Configure Remotes**:
|
||||
Edit `config/git-sync/remotes.conf` to match your repository URLs:
|
||||
\`\`\`
|
||||
github|git|git@github.com:mrhavens/mirror-repo.git|1
|
||||
forgejo|git|git@localhost:2222/mrhavens/mirror-repo.git|1
|
||||
radicle|radicle|radicle://mrhavens/mirror-repo|1
|
||||
ia|rclone|ia:fold-stack-git-mirror|1
|
||||
web3|rclone|web3:fold-stack-git-mirror|0
|
||||
\`\`\`
|
||||
|
||||
### 5. Start the Stack
|
||||
|
||||
Fold-Stack uses Docker Compose to manage services. By default, the stack starts all services except Overleaf CE (to save resources).
|
||||
|
||||
@@ -140,7 +183,7 @@ Fold-Stack uses Docker Compose to manage services. By default, the stack starts
|
||||
./scripts/enable-typst.sh
|
||||
\`\`\`
|
||||
|
||||
### 5. Verify Services are Running
|
||||
### 6. Verify Services are Running
|
||||
|
||||
Check the status of all containers:
|
||||
\`\`\`bash
|
||||
@@ -158,7 +201,7 @@ docker logs <container_name>
|
||||
\`\`\`
|
||||
For example: `docker logs overleaf_dev`.
|
||||
|
||||
### 6. Stop the Stack
|
||||
### 7. Stop the Stack
|
||||
|
||||
To stop all services:
|
||||
\`\`\`bash
|
||||
@@ -183,6 +226,7 @@ Below are the URLs to access each service running in Fold-Stack. All services ar
|
||||
| **Nextcloud** | [http://localhost:8081](http://localhost:8081) | File storage and sharing platform. | Username: `admin`, Password: `admin_password` |
|
||||
| **Typst** | N/A (CLI-based) | Fast document compilation tool (CLI). | N/A |
|
||||
| **Overleaf CE** | [http://localhost:8090](http://localhost:8090) | Collaborative LaTeX editor (run `./scripts/enable-overleaf.sh` to start). | First user registration is admin (email: `admin@example.com`). |
|
||||
| **Git-Sync** | N/A (CLI-based) | Git repository mirroring agent. | N/A |
|
||||
|
||||
---
|
||||
|
||||
@@ -378,6 +422,46 @@ docker compose -f docker-compose.dev.yml stop overleaf overleaf-mongo overleaf-r
|
||||
|
||||
---
|
||||
|
||||
### 12. **Git-Sync: Mirror a Git Repository**
|
||||
|
||||
The Git-Sync Mirror Agent watches the local repository at `./volumes/repos` and syncs changes to GitHub, Forgejo, Radicle, Internet Archive, and optionally Web3.storage.
|
||||
|
||||
1. **Add a Commit to the Local Repository**:
|
||||
\`\`\`bash
|
||||
cd volumes/repos
|
||||
echo "Change 1" >> README.md
|
||||
git add .
|
||||
git commit -m "Change 1"
|
||||
\`\`\`
|
||||
|
||||
2. **Monitor Git-Sync Logs**:
|
||||
\`\`\`bash
|
||||
docker logs git_sync_dev --follow
|
||||
\`\`\`
|
||||
You should see the sync process for each configured remote.
|
||||
|
||||
3. **Verify Sync**:
|
||||
- **GitHub**: Check your GitHub repository (`mrhavens/mirror-repo`).
|
||||
- **Forgejo**: Check `http://localhost:3000/mrhavens/mirror-repo`.
|
||||
- **Internet Archive**: Check `fold-stack-git-mirror` for Git bundles.
|
||||
- **Web3.storage**: Enable in `remotes.conf` and check `fold-stack-git-mirror`.
|
||||
|
||||
**Configuration**:
|
||||
- Edit `config/git-sync/.env` to adjust settings:
|
||||
\`\`\`
|
||||
SYNC_INTERVAL=300 # Sync check interval in seconds
|
||||
PUSH_MODE=push # "push" for git push, "bundle" for git bundle
|
||||
SIGN_COMMITS=false # Set to true to enable commit signing (requires GPG)
|
||||
LOG_LEVEL=INFO # Log verbosity (INFO, ERROR)
|
||||
RETRY_MAX=3 # Max retry attempts for failed syncs
|
||||
RETRY_BACKOFF=5 # Base backoff time in seconds for retries
|
||||
\`\`\`
|
||||
|
||||
**Logs**:
|
||||
- Logs are stored in `./volumes/logs` with filenames like `sync-<timestamp>.log`.
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Troubleshooting
|
||||
|
||||
### General Issues
|
||||
@@ -419,6 +503,14 @@ docker compose -f docker-compose.dev.yml stop overleaf overleaf-mongo overleaf-r
|
||||
\`\`\`
|
||||
Ensure MongoDB and Redis are healthy before Overleaf starts (handled by `depends_on` in `docker-compose.dev.yml`).
|
||||
|
||||
### Git-Sync Issues
|
||||
- **Sync Fails**: Check logs:
|
||||
\`\`\`bash
|
||||
docker logs git_sync_dev
|
||||
\`\`\`
|
||||
Ensure SSH keys are correctly set up and remotes are accessible.
|
||||
- **Radicle Not Syncing**: Radicle sync is a placeholder. Implement the `rad` CLI in `entrypoint.sh` if needed.
|
||||
|
||||
---
|
||||
|
||||
## 📚 Additional Resources
|
||||
@@ -451,6 +543,6 @@ Contributions are welcome! To contribute:
|
||||
|
||||
## 📅 Last Updated
|
||||
|
||||
This README was last updated on **May 26, 2025, at 08:49 PM CDT**.
|
||||
This README was last updated on **May 26, 2025, at 09:21 PM CDT**.
|
||||
|
||||
---
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
[nextcloud]
|
||||
type = webdav
|
||||
url = http://localhost:8081/remote.php/dav/files/admin/
|
||||
vendor = admin
|
||||
user = admin
|
||||
pass = 700-kf6PNutLqpTd5heFH7_qV4Be4qqsIi1duRb4
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
# Format: remote_name|type|url|enabled (1 for enabled, 0 for disabled)
|
||||
github|git|git@github.com:mrhavens/mirror-repo.git|1
|
||||
forgejo|git|git@localhost:2222/mrhavens/mirror-repo.git|1
|
||||
radicle|radicle|radicle://mrhavens/mirror-repo|1
|
||||
ia|rclone|ia:fold-stack-git-mirror|1
|
||||
web3|rclone|web3:fold-stack-git-mirror|0
|
||||
@@ -0,0 +1,4 @@
|
||||
{
|
||||
"branches": ["*"],
|
||||
"exclude_tags": []
|
||||
}
|
||||
@@ -189,6 +189,16 @@ services:
|
||||
networks:
|
||||
- fold-network
|
||||
|
||||
git-sync:
|
||||
build: ./git-sync
|
||||
container_name: git_sync_dev
|
||||
volumes:
|
||||
- ./config/git-sync:/config/git-sync:ro
|
||||
- ./volumes/repos:/repos/local
|
||||
- ./volumes/logs:/logs
|
||||
networks:
|
||||
- fold-network
|
||||
|
||||
networks:
|
||||
fold-network:
|
||||
driver: bridge
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
# Stage 1: Build stage
|
||||
FROM alpine:3.19 AS builder
|
||||
|
||||
# Install build dependencies
|
||||
RUN apk add --no-cache build-base git openssh-client curl bash
|
||||
|
||||
# Install Rclone (statically compiled binary for Alpine)
|
||||
RUN curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip && unzip rclone-current-linux-amd64.zip && mv rclone-*-linux-amd64/rclone /usr/bin/ && rm -rf rclone-*-linux-amd64 rclone-current-linux-amd64.zip
|
||||
|
||||
# Stage 2: Runtime stage
|
||||
FROM alpine:3.19
|
||||
|
||||
# Install runtime dependencies
|
||||
RUN apk add --no-cache git openssh-client bash inotify-tools ca-certificates
|
||||
|
||||
# Copy Rclone from the builder stage
|
||||
COPY --from=builder /usr/bin/rclone /usr/bin/rclone
|
||||
|
||||
# Set up SSH directory and permissions
|
||||
RUN mkdir -p /root/.ssh && chmod 700 /root/.ssh
|
||||
|
||||
# Copy entrypoint script
|
||||
COPY entrypoint.sh /entrypoint.sh
|
||||
RUN chmod +x /entrypoint.sh
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /repos
|
||||
|
||||
# Define volumes for configuration, secrets, and logs
|
||||
VOLUME /config/git-sync
|
||||
VOLUME /repos/local
|
||||
VOLUME /logs
|
||||
|
||||
# Entrypoint
|
||||
ENTRYPOINT ["/entrypoint.sh"]
|
||||
CMD []
|
||||
@@ -0,0 +1,188 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
# Load environment variables
|
||||
if [ -f "/config/git-sync/.env" ]; then
|
||||
set -a
|
||||
source /config/git-sync/.env
|
||||
set +a
|
||||
else
|
||||
echo "ERROR: /config/git-sync/.env not found. Exiting."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Ensure required directories exist
|
||||
mkdir -p /logs /repos/local /root/.ssh
|
||||
|
||||
# Set up SSH keys
|
||||
if [ -f "/config/git-sync/secrets/github.key" ]; then
|
||||
cp /config/git-sync/secrets/github.key /root/.ssh/github.key
|
||||
chmod 600 /root/.ssh/github.key
|
||||
echo "Host github.com" >> /root/.ssh/config
|
||||
echo " HostName github.com" >> /root/.ssh/config
|
||||
echo " User git" >> /root/.ssh/config
|
||||
echo " IdentityFile /root/.ssh/github.key" >> /root/.ssh/config
|
||||
echo " StrictHostKeyChecking no" >> /root/.ssh/config
|
||||
else
|
||||
echo "WARNING: GitHub SSH key not found. GitHub sync will fail."
|
||||
fi
|
||||
|
||||
if [ -f "/config/git-sync/secrets/forgejo.key" ]; then
|
||||
cp /config/git-sync/secrets/forgejo.key /root/.ssh/forgejo.key
|
||||
chmod 600 /root/.ssh/forgejo.key
|
||||
echo "Host localhost" >> /root/.ssh/config
|
||||
echo " HostName localhost" >> /root/.ssh/config
|
||||
echo " Port 2222" >> /root/.ssh/config
|
||||
echo " User git" >> /root/.ssh/config
|
||||
echo " IdentityFile /root/.ssh/forgejo.key" >> /root/.ssh/config
|
||||
echo " StrictHostKeyChecking no" >> /root/.ssh/config
|
||||
else
|
||||
echo "WARNING: Forgejo SSH key not found. Forgejo sync will fail."
|
||||
fi
|
||||
|
||||
# Initialize log file
|
||||
LOG_FILE="/logs/sync-1748312796.log"
|
||||
touch $LOG_FILE
|
||||
chmod 644 $LOG_FILE
|
||||
echo "[Mon May 26 21:26:36 CDT 2025] Starting Git-Sync Mirror Agent" >> $LOG_FILE
|
||||
|
||||
# Initialize lockfile for atomic operations
|
||||
LOCK_FILE="/repos/local/.git-sync.lock"
|
||||
touch $LOCK_FILE
|
||||
|
||||
# Function to log messages
|
||||
log_message() {
|
||||
local level=$1
|
||||
local message=$2
|
||||
echo "[Mon May 26 21:26:36 CDT 2025] [$level] $message" >> $LOG_FILE
|
||||
if [ "$level" = "INFO" ] && [ "$LOG_LEVEL" = "INFO" ]; then
|
||||
echo "[Mon May 26 21:26:36 CDT 2025] [$level] $message"
|
||||
elif [ "$level" = "ERROR" ]; then
|
||||
echo "[Mon May 26 21:26:36 CDT 2025] [$level] $message" >&2
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to execute with lock
|
||||
execute_with_lock() {
|
||||
exec 100>$LOCK_FILE
|
||||
flock 100
|
||||
$@
|
||||
exec 100>&-
|
||||
}
|
||||
|
||||
# Function to detect changes in the local repository
|
||||
detect_changes() {
|
||||
log_message "INFO" "Checking for changes in local repository..."
|
||||
cd /repos/local
|
||||
if [ ! -d ".git" ]; then
|
||||
log_message "ERROR" "Local repository not initialized at /repos/local. Exiting."
|
||||
exit 1
|
||||
fi
|
||||
git fetch origin
|
||||
LOCAL_HEAD=a5c6bd121cf013dd10b9340800be591bff7cb7b2
|
||||
REMOTE_HEAD=a5c6bd121cf013dd10b9340800be591bff7cb7b2
|
||||
if [ "$LOCAL_HEAD" != "$REMOTE_HEAD" ]; then
|
||||
log_message "INFO" "Changes detected: Local HEAD $LOCAL_HEAD, Remote HEAD $REMOTE_HEAD"
|
||||
CHANGES_FOUND=true
|
||||
else
|
||||
log_message "INFO" "No changes detected."
|
||||
CHANGES_FOUND=false
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to sign commits (placeholder, requires GPG setup)
|
||||
sign_commits_if_enabled() {
|
||||
if [ "$SIGN_COMMITS" = "true" ]; then
|
||||
log_message "INFO" "Commit signing enabled but not implemented. Skipping."
|
||||
# TODO: Implement GPG signing
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to sync to a Git remote (GitHub/Forgejo)
|
||||
sync_to_git_remote() {
|
||||
local remote_name=$1
|
||||
local url=$2
|
||||
log_message "INFO" "Syncing to $remote_name at $url..."
|
||||
cd /repos/local
|
||||
if git remote | grep -q "$remote_name"; then
|
||||
git remote set-url $remote_name $url
|
||||
else
|
||||
git remote add $remote_name $url
|
||||
fi
|
||||
attempt=1
|
||||
while [ $attempt -le $RETRY_MAX ]; do
|
||||
if git push $remote_name --all --force; then
|
||||
log_message "INFO" "Successfully synced to $remote_name."
|
||||
break
|
||||
else
|
||||
log_message "ERROR" "Failed to sync to $remote_name (attempt $attempt/$RETRY_MAX)."
|
||||
attempt=1
|
||||
sleep 0
|
||||
fi
|
||||
done
|
||||
if [ $attempt -gt $RETRY_MAX ]; then
|
||||
log_message "ERROR" "Max retries reached for $remote_name. Giving up."
|
||||
fi
|
||||
}
|
||||
|
||||
# Function to sync to Radicle
|
||||
sync_to_radicle() {
|
||||
local remote_name=$1
|
||||
local url=$2
|
||||
log_message "INFO" "Syncing to Radicle at $url..."
|
||||
# Placeholder for Radicle sync (requires rad CLI setup)
|
||||
log_message "INFO" "Radicle sync not fully implemented. Skipping."
|
||||
# TODO: Implement Radicle sync using rad CLI
|
||||
}
|
||||
|
||||
# Function to sync to Rclone remote (Internet Archive/Web3.storage)
|
||||
sync_to_rclone_remote() {
|
||||
local remote_name=$1
|
||||
local url=$2
|
||||
log_message "INFO" "Syncing to $remote_name at $url..."
|
||||
# Create a Git bundle
|
||||
cd /repos/local
|
||||
BUNDLE_FILE="/tmp/repo-1748312796.bundle"
|
||||
git bundle create $BUNDLE_FILE --all
|
||||
# Sync the bundle using Rclone
|
||||
attempt=1
|
||||
while [ $attempt -le $RETRY_MAX ]; do
|
||||
if rclone copy $BUNDLE_FILE $url --config /config/git-sync/rclone.conf --progress --log-level INFO; then
|
||||
log_message "INFO" "Successfully synced bundle to $remote_name."
|
||||
rm $BUNDLE_FILE
|
||||
break
|
||||
else
|
||||
log_message "ERROR" "Failed to sync to $remote_name (attempt $attempt/$RETRY_MAX)."
|
||||
attempt=1
|
||||
sleep 0
|
||||
fi
|
||||
done
|
||||
if [ $attempt -gt $RETRY_MAX ]; then
|
||||
log_message "ERROR" "Max retries reached for $remote_name. Giving up."
|
||||
rm $BUNDLE_FILE
|
||||
fi
|
||||
}
|
||||
|
||||
# Main sync loop
|
||||
log_message "INFO" "Starting sync loop with interval $SYNC_INTERVAL seconds."
|
||||
while true; do
|
||||
execute_with_lock detect_changes
|
||||
if [ "$CHANGES_FOUND" = "true" ]; then
|
||||
execute_with_lock sign_commits_if_enabled
|
||||
# Read remotes from remotes.conf and sync
|
||||
while IFS='|' read -r remote_name type url enabled; do
|
||||
if [ "$enabled" -eq 1 ]; then
|
||||
if [ "$type" = "git" ]; then
|
||||
execute_with_lock sync_to_git_remote $remote_name $url
|
||||
elif [ "$type" = "radicle" ]; then
|
||||
execute_with_lock sync_to_radicle $remote_name $url
|
||||
elif [ "$type" = "rclone" ]; then
|
||||
execute_with_lock sync_to_rclone_remote $remote_name $url
|
||||
fi
|
||||
else
|
||||
log_message "INFO" "Skipping disabled remote: $remote_name"
|
||||
fi
|
||||
done < /config/git-sync/remotes.conf
|
||||
fi
|
||||
sleep $SYNC_INTERVAL
|
||||
done
|
||||
@@ -112,6 +112,9 @@ docker logs overleaf_mongo_dev --tail=20 2>&1 || print_warning "Overleaf Mongo c
|
||||
print_section "Overleaf Redis Logs (last 20 lines)"
|
||||
docker logs overleaf_redis_dev --tail=20 2>&1 || print_warning "Overleaf Redis container not found (run ./scripts/enable-overleaf.sh to start)."
|
||||
|
||||
print_section "Git-Sync Logs (last 20 lines)"
|
||||
docker logs git_sync_dev --tail=20 2>&1 || print_warning "Git-Sync container not found."
|
||||
|
||||
# 7. Check Volume Permissions and Contents
|
||||
print_section "Forgejo Volume Permissions"
|
||||
ls -ld ./volumes/forgejo || print_error "Missing volumes/forgejo"
|
||||
@@ -143,6 +146,14 @@ print_section "Overleaf Volume Permissions"
|
||||
ls -ld ./volumes/overleaf || print_warning "Missing volumes/overleaf (needed for Overleaf CE persistence)"
|
||||
ls -la ./volumes/overleaf || print_warning "Overleaf volume contents not accessible"
|
||||
|
||||
print_section "Git Repositories Volume Permissions"
|
||||
ls -ld ./volumes/repos || print_error "Missing volumes/repos (needed for Git-Sync)"
|
||||
ls -la ./volumes/repos || print_warning "Git repositories volume contents not accessible"
|
||||
|
||||
print_section "Logs Volume Permissions"
|
||||
ls -ld ./volumes/logs || print_error "Missing volumes/logs (needed for logging)"
|
||||
ls -la ./volumes/logs || print_warning "Logs volume contents not accessible"
|
||||
|
||||
# 8. Check Entrypoint Script for Forgejo
|
||||
print_section "Forgejo Entrypoint Script Check (forgejo-entrypoint.sh)"
|
||||
head -n 10 scripts/forgejo-entrypoint.sh 2>/dev/null || print_warning "Missing forgejo-entrypoint.sh script"
|
||||
|
||||
+1
-1
@@ -2,5 +2,5 @@
|
||||
set -e
|
||||
|
||||
echo "Starting fold-stack development environment (excluding Overleaf CE by default)..."
|
||||
docker compose -f docker-compose.dev.yml up -d --build ghost forgejo radicle pandoc mailhog trilium hedgedoc nextcloud rclone typst
|
||||
docker compose -f docker-compose.dev.yml up -d --build ghost forgejo radicle pandoc mailhog trilium hedgedoc nextcloud rclone typst git-sync
|
||||
echo "Core services started. To enable Overleaf CE, run: ./scripts/enable-overleaf.sh"
|
||||
|
||||
Reference in New Issue
Block a user