Stratégie de Backups¶
Introduction¶
Une stratégie de backup robuste est essentielle pour protéger les données et assurer la continuité des services OpenStack. Cette section couvre les sauvegardes des bases de données, configurations, et volumes Ceph.
Prérequis¶
- OpenStack en production
- Ceph Storage configuré
- Stockage externe pour les backups
- Monitoring en place
Points à apprendre¶
Architecture Backup¶
graph LR
subgraph Production["OpenStack Production"]
mariadb[(MariaDB Galera)]
config["Configurations<br/>/etc/kolla/*"]
ceph[(Ceph Cluster)]
secrets["Secrets<br/>passwords.yml"]
end
subgraph Backup["Backup Infrastructure"]
backup_server["Backup Server<br/>Orchestration"]
local[(Local Storage<br/>7 jours)]
s3[(S3 Object Storage<br/>90 jours)]
offsite[(Offsite/Tape<br/>1 an)]
end
mariadb -->|mysqldump Daily| backup_server
config -->|rsync Daily| backup_server
ceph -->|rbd export Weekly| backup_server
secrets -->|encrypted Daily| backup_server
backup_server --> local
local --> s3
s3 -->|Monthly| offsite
Politique de rétention (3-2-1)¶
graph TB
subgraph Copies["3 Copies"]
prod["Production"]
local["Backup Local"]
offsite["Backup Offsite"]
end
subgraph Media["2 Médias différents"]
disk["SSD/HDD"]
s3["Object Storage"]
end
subgraph Remote["1 Copie hors site"]
remote["Autre datacenter<br/>ou Cloud"]
end
prod -->|Daily| local
local -->|Sync| s3
s3 -->|Weekly| remote
Protection contre: Sinistre local, Ransomware, Erreur humaine
Backup MariaDB Galera¶
#!/bin/bash
# backup-mariadb.sh
set -e
BACKUP_DIR="/backup/mariadb"
DATE=$(date +%Y%m%d_%H%M%S)
RETENTION_DAYS=7
mkdir -p $BACKUP_DIR
# === Full backup avec mariabackup ===
docker exec mariadb mariabackup \
--backup \
--target-dir=/backup/full_${DATE} \
--user=root \
--password="${MARIADB_ROOT_PASSWORD}"
# Copier hors du container
docker cp mariadb:/backup/full_${DATE} $BACKUP_DIR/
# Compression
tar -czvf $BACKUP_DIR/mariadb_full_${DATE}.tar.gz \
-C $BACKUP_DIR full_${DATE}
rm -rf $BACKUP_DIR/full_${DATE}
# === Ou mysqldump pour backup logique ===
docker exec mariadb mysqldump \
--all-databases \
--single-transaction \
--routines \
--triggers \
--events \
-u root -p"${MARIADB_ROOT_PASSWORD}" \
> $BACKUP_DIR/mariadb_dump_${DATE}.sql
gzip $BACKUP_DIR/mariadb_dump_${DATE}.sql
# === Nettoyage anciens backups ===
find $BACKUP_DIR -name "*.tar.gz" -mtime +${RETENTION_DAYS} -delete
find $BACKUP_DIR -name "*.sql.gz" -mtime +${RETENTION_DAYS} -delete
echo "Backup completed: $BACKUP_DIR/mariadb_full_${DATE}.tar.gz"
# Vérifier l'intégrité
docker exec mariadb mariabackup \
--prepare \
--target-dir=/backup/full_${DATE} 2>&1 | tail -5
Backup configurations Kolla¶
#!/bin/bash
# backup-kolla-config.sh
set -e
BACKUP_DIR="/backup/kolla"
DATE=$(date +%Y%m%d_%H%M%S)
KOLLA_DIR="/etc/kolla"
mkdir -p $BACKUP_DIR
# === Backup configurations ===
tar -czvf $BACKUP_DIR/kolla_config_${DATE}.tar.gz \
--exclude='*.log' \
--exclude='*.pid' \
$KOLLA_DIR
# === Backup passwords (chiffré) ===
gpg --symmetric --cipher-algo AES256 \
--output $BACKUP_DIR/passwords_${DATE}.yml.gpg \
$KOLLA_DIR/passwords.yml
# === Backup certificats ===
tar -czvf $BACKUP_DIR/certificates_${DATE}.tar.gz \
$KOLLA_DIR/certificates/
# === Backup inventory ===
cp /etc/kolla/inventory $BACKUP_DIR/inventory_${DATE}
cp /etc/kolla/globals.yml $BACKUP_DIR/globals_${DATE}.yml
# === Checksums ===
cd $BACKUP_DIR
sha256sum *_${DATE}* > checksums_${DATE}.txt
echo "Kolla backup completed: $BACKUP_DIR"
ls -lh $BACKUP_DIR/*_${DATE}*
Backup Ceph¶
#!/bin/bash
# backup-ceph.sh
set -e
BACKUP_DIR="/backup/ceph"
DATE=$(date +%Y%m%d_%H%M%S)
POOL="volumes"
mkdir -p $BACKUP_DIR
# === Export configuration Ceph ===
ceph config dump > $BACKUP_DIR/ceph_config_${DATE}.txt
ceph osd dump > $BACKUP_DIR/ceph_osd_${DATE}.txt
ceph mon dump > $BACKUP_DIR/ceph_mon_${DATE}.txt
ceph auth list > $BACKUP_DIR/ceph_auth_${DATE}.txt
# === Backup CRUSH map ===
ceph osd getcrushmap -o $BACKUP_DIR/crushmap_${DATE}.bin
crushtool -d $BACKUP_DIR/crushmap_${DATE}.bin \
-o $BACKUP_DIR/crushmap_${DATE}.txt
# === Snapshot des volumes critiques ===
for vol in $(rbd ls $POOL | head -20); do
echo "Creating snapshot for $vol..."
rbd snap create ${POOL}/${vol}@backup_${DATE}
done
# === Export d'un volume spécifique ===
# rbd export volumes/volume-xxx $BACKUP_DIR/volume-xxx_${DATE}.raw
# === Backup images Glance ===
IMAGES_POOL="images"
for img in $(rbd ls $IMAGES_POOL); do
echo "Backing up image: $img"
rbd export ${IMAGES_POOL}/${img} $BACKUP_DIR/image_${img}_${DATE}.raw
done
# === Cleanup old snapshots ===
OLD_DATE=$(date -d "7 days ago" +%Y%m%d)
for vol in $(rbd ls $POOL); do
for snap in $(rbd snap ls ${POOL}/${vol} | grep backup_ | awk '{print $2}'); do
snap_date=$(echo $snap | cut -d'_' -f2)
if [[ "$snap_date" < "$OLD_DATE" ]]; then
echo "Removing old snapshot: ${POOL}/${vol}@${snap}"
rbd snap rm ${POOL}/${vol}@${snap}
fi
done
done
echo "Ceph backup completed"
Backup automatisé avec script master¶
#!/bin/bash
# backup-openstack.sh - Script de backup complet
set -e
LOG_FILE="/var/log/openstack-backup.log"
BACKUP_ROOT="/backup"
DATE=$(date +%Y%m%d_%H%M%S)
ALERT_EMAIL="admin@example.com"
exec > >(tee -a $LOG_FILE) 2>&1
echo "=========================================="
echo "OpenStack Backup Started: $(date)"
echo "=========================================="
# Fonction de notification
notify() {
local status=$1
local message=$2
echo "[$status] $message"
# Envoyer alerte si échec
if [ "$status" == "ERROR" ]; then
echo "$message" | mail -s "Backup FAILED: $(hostname)" $ALERT_EMAIL
fi
}
# 1. Backup MariaDB
echo -e "\n=== [1/5] MariaDB Backup ==="
if /opt/scripts/backup-mariadb.sh; then
notify "OK" "MariaDB backup completed"
else
notify "ERROR" "MariaDB backup failed"
exit 1
fi
# 2. Backup Configurations
echo -e "\n=== [2/5] Configuration Backup ==="
if /opt/scripts/backup-kolla-config.sh; then
notify "OK" "Configuration backup completed"
else
notify "ERROR" "Configuration backup failed"
exit 1
fi
# 3. Backup Ceph
echo -e "\n=== [3/5] Ceph Backup ==="
if /opt/scripts/backup-ceph.sh; then
notify "OK" "Ceph backup completed"
else
notify "ERROR" "Ceph backup failed"
exit 1
fi
# 4. Sync to S3
echo -e "\n=== [4/5] S3 Sync ==="
if aws s3 sync $BACKUP_ROOT s3://openstack-backups/$(hostname)/ \
--storage-class STANDARD_IA; then
notify "OK" "S3 sync completed"
else
notify "ERROR" "S3 sync failed"
fi
# 5. Cleanup local backups
echo -e "\n=== [5/5] Cleanup ==="
find $BACKUP_ROOT -type f -mtime +7 -delete
notify "OK" "Cleanup completed"
# Résumé
echo -e "\n=========================================="
echo "Backup Completed: $(date)"
echo "=========================================="
du -sh $BACKUP_ROOT/*
# Envoyer rapport
cat << EOF | mail -s "Backup SUCCESS: $(hostname)" $ALERT_EMAIL
OpenStack Backup Report
=======================
Date: $(date)
Host: $(hostname)
Backup Sizes:
$(du -sh $BACKUP_ROOT/*)
Status: SUCCESS
EOF
Crontab backup¶
# /etc/cron.d/openstack-backup
# Daily database backup at 2 AM
0 2 * * * root /opt/scripts/backup-mariadb.sh >> /var/log/backup-mariadb.log 2>&1
# Daily config backup at 3 AM
0 3 * * * root /opt/scripts/backup-kolla-config.sh >> /var/log/backup-config.log 2>&1
# Weekly Ceph backup on Sunday at 4 AM
0 4 * * 0 root /opt/scripts/backup-ceph.sh >> /var/log/backup-ceph.log 2>&1
# Full backup weekly
0 1 * * 0 root /opt/scripts/backup-openstack.sh >> /var/log/backup-full.log 2>&1
Restauration MariaDB¶
#!/bin/bash
# restore-mariadb.sh
BACKUP_FILE=$1
if [ -z "$BACKUP_FILE" ]; then
echo "Usage: $0 <backup_file>"
echo "Available backups:"
ls -la /backup/mariadb/
exit 1
fi
# === Restauration avec mariabackup ===
if [[ "$BACKUP_FILE" == *.tar.gz ]]; then
echo "Restoring from mariabackup..."
# Arrêter le service
docker stop mariadb
# Extraire le backup
tar -xzvf $BACKUP_FILE -C /tmp/
BACKUP_DIR=$(basename $BACKUP_FILE .tar.gz)
# Préparer le backup
mariabackup --prepare --target-dir=/tmp/$BACKUP_DIR
# Supprimer les anciennes données
rm -rf /var/lib/docker/volumes/mariadb/_data/*
# Restaurer
mariabackup --copy-back --target-dir=/tmp/$BACKUP_DIR
# Permissions
chown -R 27:27 /var/lib/docker/volumes/mariadb/_data/
# Redémarrer
docker start mariadb
echo "Restoration completed. Checking status..."
sleep 10
docker exec mariadb mysql -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
fi
# === Restauration avec mysqldump ===
if [[ "$BACKUP_FILE" == *.sql.gz ]]; then
echo "Restoring from mysqldump..."
gunzip -c $BACKUP_FILE | docker exec -i mariadb mysql -u root -p"${MARIADB_ROOT_PASSWORD}"
echo "Restoration completed"
fi
Diagramme processus backup¶
flowchart TB
start([Start])
subgraph Daily["Daily Backup"]
d1["MariaDB mysqldump"]
d2["Kolla configurations"]
d3["Certificats et secrets"]
d4["Compress et chiffrer"]
d1 --> d2 --> d3 --> d4
end
subgraph Weekly["Weekly Backup"]
w1["Ceph pool snapshots"]
w2["Export images Glance"]
w3["Full system backup"]
w1 --> w2 --> w3
end
s3["Upload to S3"]
s3_check{S3 upload success?}
clean["Clean local old backups"]
alert1["Alert admin"]
retry["Retry in 1 hour"]
verify["Verify backup integrity"]
verify_check{Verification OK?}
success["Send success report"]
alert2["Alert admin"]
failed["Mark backup as failed"]
subgraph Monthly["Monthly"]
m1["Archive to offsite"]
m2["Test restore procedure"]
m3["Update documentation"]
m1 --> m2 --> m3
end
finish([Stop])
start --> Daily
Daily --> Weekly
Weekly --> s3
s3 --> s3_check
s3_check -->|yes| clean
s3_check -->|no| alert1 --> retry
clean --> verify
verify --> verify_check
verify_check -->|yes| success
verify_check -->|no| alert2 --> failed
success --> Monthly
Monthly --> finish
Exemples pratiques¶
Test de restauration¶
#!/bin/bash
# test-restore.sh
# 1. Créer un environnement de test
docker run -d --name mariadb-test \
-e MYSQL_ROOT_PASSWORD=test \
mariadb:10.6
# 2. Restaurer le backup
LATEST_BACKUP=$(ls -t /backup/mariadb/*.sql.gz | head -1)
gunzip -c $LATEST_BACKUP | docker exec -i mariadb-test mysql -u root -ptest
# 3. Vérifier les données
docker exec mariadb-test mysql -u root -ptest -e "
SELECT table_schema, COUNT(*) as tables
FROM information_schema.tables
GROUP BY table_schema;
"
# 4. Nettoyer
docker rm -f mariadb-test
echo "Restore test completed successfully"
Monitoring des backups¶
# /etc/prometheus/rules/backup-alerts.yml
groups:
- name: backup-monitoring
rules:
- alert: BackupTooOld
expr: time() - backup_last_success_timestamp > 86400 * 2
for: 1h
labels:
severity: critical
annotations:
summary: "Backup older than 2 days"
- alert: BackupFailed
expr: backup_last_status == 0
for: 0m
labels:
severity: critical
annotations:
summary: "Last backup failed"
- alert: BackupStorageLow
expr: backup_storage_free_bytes < 50 * 1024 * 1024 * 1024
for: 1h
labels:
severity: warning
annotations:
summary: "Backup storage below 50GB"
Ressources¶
Checkpoint¶
- Script backup MariaDB fonctionnel
- Backup configurations Kolla automatisé
- Snapshots Ceph configurés
- Sync S3 en place
- Crontab backup configuré
- Procédure de restauration documentée
- Test de restauration effectué
- Monitoring des backups actif