From 0000000000000000000000000000000000000000 Mon Sep 17 00:00:00 2001 From: Dietmar Maurer Date: Mon, 6 Apr 2020 12:16:59 +0200 Subject: [PATCH] PVE-Backup: Proxmox backup patches for QEMU MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit For PBS, using dirty bitmaps is supported via QEMU's MIRROR_SYNC_MODE_BITMAP. When the feature is used, the data-write callback is only executed for any changed chunks, the PBS rust code will reuse chunks from the previous index for everything it doesn't receive if reuse_index is true. On error or cancellation, all dirty bitmaps are removed to ensure consistency. By using a JobTxn, we can sync dirty bitmaps only when *all* jobs were successful - meaning we don't need to remove them when the backup fails, since QEMU's BITMAP_SYNC_MODE_ON_SUCCESS will now handle that for us. A sequential transaction is used, so drives will be backed up one after the other. The backup and backup-cancel QMP calls are coroutines. This has the benefit that calls are asynchronous to the main loop, i.e. long running operations like connecting to a PBS server will no longer hang the VM. backup_job_create() and job_cancel_sync() cannot be run from a coroutine and requires an acuqired AioContext, so the job creation and canceling are extracted as bottom halves and called from the respective QMP coroutines. To communicate the finishing state, a dedicated property is used for query-backup: 'finishing'. A dedicated state is explicitly not used, since that would break compatibility with older qemu-server versions. The first call to job_cancel_sync() will cancel and free all jobs in the transaction, but it is necessary to pick a job that is: 1. still referenced. For this, there is a job_ref directly after job creation paired with a job_unref in cleanup paths. 2. not yet finalized. In job_cancel_bh(), the first job that's not completed yet is used. This is not necessarily the first job in the list, because pvebackup_co_complete_stream() might not yet have removed a completed job when job_cancel_bh() runs. Why even bother with the bottom half at all and not use job_cancel() in qmp_backup_cancel() directly? The reason is that qmp_backup_cancel() is a coroutine, so it will hang when reaching AIO_WAIT_WHILE() and job_cancel() might end up calling that. Regarding BackupPerf performance settings. For now, only the max-workers setting is exposed, because: 1. use-copy-range would need to be implemented in backup-dump and the feature was actually turned off by default in QEMU itself, because it didn't provide the expected benefit, see commit 6a30f663d4 ("qapi: backup: disable copy_range by default"). 2. max-chunk: enforced to be at least the backup cluster size (4 MiB for PBS) and otherwise maximum of source and target cluster size. And block-copy has a maximum buffer size of 1 MiB, so setting a larger max-chunk doesn't even have an effect. To make the setting sensibly usable the check would need to be removed and optionally the block-copy max buffer size would need to be bumped. I tried doing just that, and tested different source/target combinations with different max-chunk settings, but there were no noticable improvements over the default "unlimited" (resulting in 1 MiB for block-copy). Signed-off-by: Dietmar Maurer [SR: Add dirty-bitmap tracking for incremental backups Add query_proxmox_support and query-pbs-bitmap-info QMP calls Use a transaction to synchronize job states Co-routine and async-related improvements Improve finishing backups/cleanups Various other improvements] Signed-off-by: Stefan Reiter [FG: add master key support] Signed-off-by: Fabian Grünbichler Signed-off-by: Thomas Lamprecht [WB: add PBS namespace support] Signed-off-by: Wolfgang Bumiller [FE: add new force parameter to job_cancel_sync calls adapt for new job lock mechanism replacing AioContext locks adapt to QAPI changes improve canceling allow passing max-workers setting] Signed-off-by: Fiona Ebner --- block/meson.build | 5 + block/monitor/block-hmp-cmds.c | 40 ++ blockdev.c | 1 + hmp-commands-info.hx | 14 + hmp-commands.hx | 31 + include/monitor/hmp.h | 3 + meson.build | 1 + monitor/hmp-cmds.c | 72 +++ proxmox-backup-client.c | 146 +++++ proxmox-backup-client.h | 60 ++ pve-backup.c | 1097 ++++++++++++++++++++++++++++++++ qapi/block-core.json | 226 +++++++ qapi/common.json | 13 + qapi/machine.json | 15 +- 14 files changed, 1711 insertions(+), 13 deletions(-) create mode 100644 proxmox-backup-client.c create mode 100644 proxmox-backup-client.h create mode 100644 pve-backup.c diff --git a/block/meson.build b/block/meson.build index f580f95395..5bcebb934b 100644 --- a/block/meson.build +++ b/block/meson.build @@ -49,6 +49,11 @@ block_ss.add(files( ), zstd, zlib, gnutls) block_ss.add(files('../vma-writer.c'), libuuid) +block_ss.add(files( + '../proxmox-backup-client.c', + '../pve-backup.c', +), libproxmox_backup_qemu) + softmmu_ss.add(when: 'CONFIG_TCG', if_true: files('blkreplay.c')) softmmu_ss.add(files('block-ram-registrar.c')) diff --git a/block/monitor/block-hmp-cmds.c b/block/monitor/block-hmp-cmds.c index ca2599de44..636509b83e 100644 --- a/block/monitor/block-hmp-cmds.c +++ b/block/monitor/block-hmp-cmds.c @@ -1029,3 +1029,43 @@ void hmp_change_medium(Monitor *mon, const char *device, const char *target, qmp_blockdev_change_medium(device, NULL, target, arg, true, force, !!read_only, read_only_mode, errp); } + +void coroutine_fn hmp_backup_cancel(Monitor *mon, const QDict *qdict) +{ + Error *error = NULL; + + qmp_backup_cancel(&error); + + hmp_handle_error(mon, error); +} + +void coroutine_fn hmp_backup(Monitor *mon, const QDict *qdict) +{ + Error *error = NULL; + + int dir = qdict_get_try_bool(qdict, "directory", 0); + const char *backup_file = qdict_get_str(qdict, "backupfile"); + const char *devlist = qdict_get_try_str(qdict, "devlist"); + int64_t speed = qdict_get_try_int(qdict, "speed", 0); + + qmp_backup( + backup_file, + NULL, // PBS password + NULL, // PBS keyfile + NULL, // PBS key_password + NULL, // PBS master_keyfile + NULL, // PBS fingerprint + NULL, // PBS backup-ns + NULL, // PBS backup-id + false, 0, // PBS backup-time + false, false, // PBS use-dirty-bitmap + false, false, // PBS compress + false, false, // PBS encrypt + true, dir ? BACKUP_FORMAT_DIR : BACKUP_FORMAT_VMA, + NULL, NULL, + devlist, qdict_haskey(qdict, "speed"), speed, + false, 0, // BackupPerf max-workers + &error); + + hmp_handle_error(mon, error); +} diff --git a/blockdev.c b/blockdev.c index 9a010f3a86..b9505c95d3 100644 --- a/blockdev.c +++ b/blockdev.c @@ -37,6 +37,7 @@ #include "block/blockjob.h" #include "block/dirty-bitmap.h" #include "block/qdict.h" +#include "block/blockjob_int.h" #include "block/throttle-groups.h" #include "monitor/monitor.h" #include "qemu/error-report.h" diff --git a/hmp-commands-info.hx b/hmp-commands-info.hx index a166bff3d5..4b75966c2e 100644 --- a/hmp-commands-info.hx +++ b/hmp-commands-info.hx @@ -486,6 +486,20 @@ SRST Show the current VM UUID. ERST + + { + .name = "backup", + .args_type = "", + .params = "", + .help = "show backup status", + .cmd = hmp_info_backup, + }, + +SRST + ``info backup`` + Show backup status. +ERST + #if defined(CONFIG_SLIRP) { .name = "usernet", diff --git a/hmp-commands.hx b/hmp-commands.hx index b66d7fc4ab..896430dae8 100644 --- a/hmp-commands.hx +++ b/hmp-commands.hx @@ -101,6 +101,37 @@ ERST SRST ``block_stream`` Copy data from a backing file into a block device. +ERST + + { + .name = "backup", + .args_type = "directory:-d,backupfile:s,speed:o?,devlist:s?", + .params = "[-d] backupfile [speed [devlist]]", + .help = "create a VM Backup." + "\n\t\t\t Use -d to dump data into a directory instead" + "\n\t\t\t of using VMA format.", + .cmd = hmp_backup, + .coroutine = true, + }, + +SRST +``backup`` + Create a VM backup. +ERST + + { + .name = "backup_cancel", + .args_type = "", + .params = "", + .help = "cancel the current VM backup", + .cmd = hmp_backup_cancel, + .coroutine = true, + }, + +SRST +``backup_cancel`` + Cancel the current VM backup. + ERST { diff --git a/include/monitor/hmp.h b/include/monitor/hmp.h index c012bad741..2e504db706 100644 --- a/include/monitor/hmp.h +++ b/include/monitor/hmp.h @@ -32,6 +32,7 @@ void hmp_info_savevm(Monitor *mon, const QDict *qdict); void hmp_info_migrate(Monitor *mon, const QDict *qdict); void hmp_info_migrate_capabilities(Monitor *mon, const QDict *qdict); void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict); +void hmp_info_backup(Monitor *mon, const QDict *qdict); void hmp_info_cpus(Monitor *mon, const QDict *qdict); void hmp_info_vnc(Monitor *mon, const QDict *qdict); void hmp_info_spice(Monitor *mon, const QDict *qdict); @@ -84,6 +85,8 @@ void hmp_change_vnc(Monitor *mon, const char *device, const char *target, void hmp_change_medium(Monitor *mon, const char *device, const char *target, const char *arg, const char *read_only, bool force, Error **errp); +void hmp_backup(Monitor *mon, const QDict *qdict); +void hmp_backup_cancel(Monitor *mon, const QDict *qdict); void hmp_migrate(Monitor *mon, const QDict *qdict); void hmp_device_add(Monitor *mon, const QDict *qdict); void hmp_device_del(Monitor *mon, const QDict *qdict); diff --git a/meson.build b/meson.build index 603cdb97bb..d307d8eabf 100644 --- a/meson.build +++ b/meson.build @@ -1528,6 +1528,7 @@ keyutils = dependency('libkeyutils', required: false, has_gettid = cc.has_function('gettid') libuuid = cc.find_library('uuid', required: true) +libproxmox_backup_qemu = cc.find_library('proxmox_backup_qemu', required: true) # libselinux selinux = dependency('libselinux', diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c index 435f9334f9..9a67e544ce 100644 --- a/monitor/hmp-cmds.c +++ b/monitor/hmp-cmds.c @@ -21,6 +21,7 @@ #include "qemu/help_option.h" #include "monitor/monitor-internal.h" #include "qapi/error.h" +#include "qapi/qapi-commands-block-core.h" #include "qapi/qapi-commands-control.h" #include "qapi/qapi-commands-migration.h" #include "qapi/qapi-commands-misc.h" @@ -144,6 +145,77 @@ void hmp_sync_profile(Monitor *mon, const QDict *qdict) } } +void hmp_info_backup(Monitor *mon, const QDict *qdict) +{ + BackupStatus *info; + PBSBitmapInfoList *bitmap_info; + + info = qmp_query_backup(NULL); + + if (!info) { + monitor_printf(mon, "Backup status: not initialized\n"); + return; + } + + if (info->status) { + if (info->errmsg) { + monitor_printf(mon, "Backup status: %s - %s\n", + info->status, info->errmsg); + } else { + monitor_printf(mon, "Backup status: %s\n", info->status); + } + } + + if (info->backup_file) { + monitor_printf(mon, "Start time: %s", ctime(&info->start_time)); + if (info->end_time) { + monitor_printf(mon, "End time: %s", ctime(&info->end_time)); + } + + monitor_printf(mon, "Backup file: %s\n", info->backup_file); + monitor_printf(mon, "Backup uuid: %s\n", info->uuid); + + if (!(info->has_total && info->total)) { + // this should not happen normally + monitor_printf(mon, "Total size: %d\n", 0); + } else { + size_t total_or_dirty = info->total; + bitmap_info = qmp_query_pbs_bitmap_info(NULL); + + while (bitmap_info) { + monitor_printf(mon, "Drive %s:\n", + bitmap_info->value->drive); + monitor_printf(mon, " bitmap action: %s\n", + PBSBitmapAction_str(bitmap_info->value->action)); + monitor_printf(mon, " size: %zd\n", + bitmap_info->value->size); + monitor_printf(mon, " dirty: %zd\n", + bitmap_info->value->dirty); + bitmap_info = bitmap_info->next; + } + + qapi_free_PBSBitmapInfoList(bitmap_info); + + int zero_per = (info->has_zero_bytes && info->zero_bytes) ? + (info->zero_bytes * 100)/info->total : 0; + monitor_printf(mon, "Total size: %zd\n", info->total); + int trans_per = (info->transferred * 100)/total_or_dirty; + monitor_printf(mon, "Transferred bytes: %zd (%d%%)\n", + info->transferred, trans_per); + monitor_printf(mon, "Zero bytes: %zd (%d%%)\n", + info->zero_bytes, zero_per); + + if (info->has_reused) { + int reused_per = (info->reused * 100)/total_or_dirty; + monitor_printf(mon, "Reused bytes: %zd (%d%%)\n", + info->reused, reused_per); + } + } + } + + qapi_free_BackupStatus(info); +} + void hmp_exit_preconfig(Monitor *mon, const QDict *qdict) { Error *err = NULL; diff --git a/proxmox-backup-client.c b/proxmox-backup-client.c new file mode 100644 index 0000000000..0923037dec --- /dev/null +++ b/proxmox-backup-client.c @@ -0,0 +1,146 @@ +#include "proxmox-backup-client.h" +#include "qemu/main-loop.h" +#include "block/aio-wait.h" +#include "qapi/error.h" + +/* Proxmox Backup Server client bindings using coroutines */ + +// This is called from another thread, so we use aio_co_schedule() +static void proxmox_backup_schedule_wake(void *data) { + CoCtxData *waker = (CoCtxData *)data; + aio_co_schedule(waker->ctx, waker->co); +} + +int coroutine_fn +proxmox_backup_co_connect(ProxmoxBackupHandle *pbs, Error **errp) +{ + Coroutine *co = qemu_coroutine_self(); + AioContext *ctx = qemu_get_current_aio_context(); + CoCtxData waker = { .co = co, .ctx = ctx }; + char *pbs_err = NULL; + int pbs_res = -1; + + proxmox_backup_connect_async(pbs, proxmox_backup_schedule_wake, &waker, &pbs_res, &pbs_err); + qemu_coroutine_yield(); + if (pbs_res < 0) { + if (errp) error_setg(errp, "backup connect failed: %s", pbs_err ? pbs_err : "unknown error"); + if (pbs_err) proxmox_backup_free_error(pbs_err); + } + return pbs_res; +} + +int coroutine_fn +proxmox_backup_co_add_config( + ProxmoxBackupHandle *pbs, + const char *name, + const uint8_t *data, + uint64_t size, + Error **errp) +{ + Coroutine *co = qemu_coroutine_self(); + AioContext *ctx = qemu_get_current_aio_context(); + CoCtxData waker = { .co = co, .ctx = ctx }; + char *pbs_err = NULL; + int pbs_res = -1; + + proxmox_backup_add_config_async( + pbs, name, data, size ,proxmox_backup_schedule_wake, &waker, &pbs_res, &pbs_err); + qemu_coroutine_yield(); + if (pbs_res < 0) { + if (errp) error_setg(errp, "backup add_config %s failed: %s", name, pbs_err ? pbs_err : "unknown error"); + if (pbs_err) proxmox_backup_free_error(pbs_err); + } + return pbs_res; +} + +int coroutine_fn +proxmox_backup_co_register_image( + ProxmoxBackupHandle *pbs, + const char *device_name, + uint64_t size, + bool incremental, + Error **errp) +{ + Coroutine *co = qemu_coroutine_self(); + AioContext *ctx = qemu_get_current_aio_context(); + CoCtxData waker = { .co = co, .ctx = ctx }; + char *pbs_err = NULL; + int pbs_res = -1; + + proxmox_backup_register_image_async( + pbs, device_name, size, incremental, proxmox_backup_schedule_wake, &waker, &pbs_res, &pbs_err); + qemu_coroutine_yield(); + if (pbs_res < 0) { + if (errp) error_setg(errp, "backup register image failed: %s", pbs_err ? pbs_err : "unknown error"); + if (pbs_err) proxmox_backup_free_error(pbs_err); + } + return pbs_res; +} + +int coroutine_fn +proxmox_backup_co_finish( + ProxmoxBackupHandle *pbs, + Error **errp) +{ + Coroutine *co = qemu_coroutine_self(); + AioContext *ctx = qemu_get_current_aio_context(); + CoCtxData waker = { .co = co, .ctx = ctx }; + char *pbs_err = NULL; + int pbs_res = -1; + + proxmox_backup_finish_async( + pbs, proxmox_backup_schedule_wake, &waker, &pbs_res, &pbs_err); + qemu_coroutine_yield(); + if (pbs_res < 0) { + if (errp) error_setg(errp, "backup finish failed: %s", pbs_err ? pbs_err : "unknown error"); + if (pbs_err) proxmox_backup_free_error(pbs_err); + } + return pbs_res; +} + +int coroutine_fn +proxmox_backup_co_close_image( + ProxmoxBackupHandle *pbs, + uint8_t dev_id, + Error **errp) +{ + Coroutine *co = qemu_coroutine_self(); + AioContext *ctx = qemu_get_current_aio_context(); + CoCtxData waker = { .co = co, .ctx = ctx }; + char *pbs_err = NULL; + int pbs_res = -1; + + proxmox_backup_close_image_async( + pbs, dev_id, proxmox_backup_schedule_wake, &waker, &pbs_res, &pbs_err); + qemu_coroutine_yield(); + if (pbs_res < 0) { + if (errp) error_setg(errp, "backup close image failed: %s", pbs_err ? pbs_err : "unknown error"); + if (pbs_err) proxmox_backup_free_error(pbs_err); + } + return pbs_res; +} + +int coroutine_fn +proxmox_backup_co_write_data( + ProxmoxBackupHandle *pbs, + uint8_t dev_id, + const uint8_t *data, + uint64_t offset, + uint64_t size, + Error **errp) +{ + Coroutine *co = qemu_coroutine_self(); + AioContext *ctx = qemu_get_current_aio_context(); + CoCtxData waker = { .co = co, .ctx = ctx }; + char *pbs_err = NULL; + int pbs_res = -1; + + proxmox_backup_write_data_async( + pbs, dev_id, data, offset, size, proxmox_backup_schedule_wake, &waker, &pbs_res, &pbs_err); + qemu_coroutine_yield(); + if (pbs_res < 0) { + if (errp) error_setg(errp, "backup write data failed: %s", pbs_err ? pbs_err : "unknown error"); + if (pbs_err) proxmox_backup_free_error(pbs_err); + } + return pbs_res; +} diff --git a/proxmox-backup-client.h b/proxmox-backup-client.h new file mode 100644 index 0000000000..8cbf645b2c --- /dev/null +++ b/proxmox-backup-client.h @@ -0,0 +1,60 @@ +#ifndef PROXMOX_BACKUP_CLIENT_H +#define PROXMOX_BACKUP_CLIENT_H + +#include "qemu/osdep.h" +#include "qemu/coroutine.h" +#include "proxmox-backup-qemu.h" + +typedef struct CoCtxData { + Coroutine *co; + AioContext *ctx; + void *data; +} CoCtxData; + +// FIXME: Remove once coroutines are supported for QMP +void block_on_coroutine_fn(CoroutineEntry *entry, void *entry_arg); + +int coroutine_fn +proxmox_backup_co_connect( + ProxmoxBackupHandle *pbs, + Error **errp); + +int coroutine_fn +proxmox_backup_co_add_config( + ProxmoxBackupHandle *pbs, + const char *name, + const uint8_t *data, + uint64_t size, + Error **errp); + +int coroutine_fn +proxmox_backup_co_register_image( + ProxmoxBackupHandle *pbs, + const char *device_name, + uint64_t size, + bool incremental, + Error **errp); + + +int coroutine_fn +proxmox_backup_co_finish( + ProxmoxBackupHandle *pbs, + Error **errp); + +int coroutine_fn +proxmox_backup_co_close_image( + ProxmoxBackupHandle *pbs, + uint8_t dev_id, + Error **errp); + +int coroutine_fn +proxmox_backup_co_write_data( + ProxmoxBackupHandle *pbs, + uint8_t dev_id, + const uint8_t *data, + uint64_t offset, + uint64_t size, + Error **errp); + + +#endif /* PROXMOX_BACKUP_CLIENT_H */ diff --git a/pve-backup.c b/pve-backup.c new file mode 100644 index 0000000000..dd72ee0ed6 --- /dev/null +++ b/pve-backup.c @@ -0,0 +1,1097 @@ +#include "proxmox-backup-client.h" +#include "vma.h" + +#include "qemu/osdep.h" +#include "qemu/module.h" +#include "sysemu/block-backend.h" +#include "sysemu/blockdev.h" +#include "block/block_int-global-state.h" +#include "block/blockjob.h" +#include "block/dirty-bitmap.h" +#include "qapi/qapi-commands-block.h" +#include "qapi/qmp/qerror.h" +#include "qemu/cutils.h" + +#include + +/* PVE backup state and related function */ + +/* + * Note: A resume from a qemu_coroutine_yield can happen in a different thread, + * so you may not use normal mutexes within coroutines: + * + * ---bad-example--- + * qemu_rec_mutex_lock(lock) + * ... + * qemu_coroutine_yield() // wait for something + * // we are now inside a different thread + * qemu_rec_mutex_unlock(lock) // Crash - wrong thread!! + * ---end-bad-example-- + * + * ==> Always use CoMutext inside coroutines. + * ==> Never acquire/release AioContext withing coroutines (because that use QemuRecMutex) + * + */ + +const char *PBS_BITMAP_NAME = "pbs-incremental-dirty-bitmap"; + +static struct PVEBackupState { + struct { + // Everything accessed from qmp_backup_query command is protected using + // this lock. Do NOT hold this lock for long times, as it is sometimes + // acquired from coroutines, and thus any wait time may block the guest. + QemuMutex lock; + Error *error; + time_t start_time; + time_t end_time; + char *backup_file; + uuid_t uuid; + char uuid_str[37]; + size_t total; + size_t dirty; + size_t transferred; + size_t reused; + size_t zero_bytes; + GList *bitmap_list; + bool finishing; + bool starting; + } stat; + int64_t speed; + BackupPerf perf; + VmaWriter *vmaw; + ProxmoxBackupHandle *pbs; + GList *di_list; + JobTxn *txn; + CoMutex backup_mutex; + CoMutex dump_callback_mutex; +} backup_state; + +static void pvebackup_init(void) +{ + qemu_mutex_init(&backup_state.stat.lock); + qemu_co_mutex_init(&backup_state.backup_mutex); + qemu_co_mutex_init(&backup_state.dump_callback_mutex); +} + +// initialize PVEBackupState at startup +opts_init(pvebackup_init); + +typedef struct PVEBackupDevInfo { + BlockDriverState *bs; + size_t size; + uint64_t block_size; + uint8_t dev_id; + int completed_ret; // INT_MAX if not completed + char targetfile[PATH_MAX]; + BdrvDirtyBitmap *bitmap; + BlockDriverState *target; + BlockJob *job; +} PVEBackupDevInfo; + +static void pvebackup_propagate_error(Error *err) +{ + qemu_mutex_lock(&backup_state.stat.lock); + error_propagate(&backup_state.stat.error, err); + qemu_mutex_unlock(&backup_state.stat.lock); +} + +static bool pvebackup_error_or_canceled(void) +{ + qemu_mutex_lock(&backup_state.stat.lock); + bool error_or_canceled = !!backup_state.stat.error; + qemu_mutex_unlock(&backup_state.stat.lock); + + return error_or_canceled; +} + +static void pvebackup_add_transfered_bytes(size_t transferred, size_t zero_bytes, size_t reused) +{ + qemu_mutex_lock(&backup_state.stat.lock); + backup_state.stat.zero_bytes += zero_bytes; + backup_state.stat.transferred += transferred; + backup_state.stat.reused += reused; + qemu_mutex_unlock(&backup_state.stat.lock); +} + +// This may get called from multiple coroutines in multiple io-threads +// Note1: this may get called after job_cancel() +static int coroutine_fn +pvebackup_co_dump_pbs_cb( + void *opaque, + uint64_t start, + uint64_t bytes, + const void *pbuf) +{ + assert(qemu_in_coroutine()); + + const uint64_t size = bytes; + const unsigned char *buf = pbuf; + PVEBackupDevInfo *di = opaque; + + assert(backup_state.pbs); + assert(buf); + + Error *local_err = NULL; + int pbs_res = -1; + + bool is_zero_block = size == di->block_size && buffer_is_zero(buf, size); + + qemu_co_mutex_lock(&backup_state.dump_callback_mutex); + + // avoid deadlock if job is cancelled + if (pvebackup_error_or_canceled()) { + qemu_co_mutex_unlock(&backup_state.dump_callback_mutex); + return -1; + } + + uint64_t transferred = 0; + uint64_t reused = 0; + while (transferred < size) { + uint64_t left = size - transferred; + uint64_t to_transfer = left < di->block_size ? left : di->block_size; + + pbs_res = proxmox_backup_co_write_data(backup_state.pbs, di->dev_id, + is_zero_block ? NULL : buf + transferred, start + transferred, + to_transfer, &local_err); + transferred += to_transfer; + + if (pbs_res < 0) { + pvebackup_propagate_error(local_err); + qemu_co_mutex_unlock(&backup_state.dump_callback_mutex); + return pbs_res; + } + + reused += pbs_res == 0 ? to_transfer : 0; + } + + qemu_co_mutex_unlock(&backup_state.dump_callback_mutex); + pvebackup_add_transfered_bytes(size, is_zero_block ? size : 0, reused); + + return size; +} + +// This may get called from multiple coroutines in multiple io-threads +static int coroutine_fn +pvebackup_co_dump_vma_cb( + void *opaque, + uint64_t start, + uint64_t bytes, + const void *pbuf) +{ + assert(qemu_in_coroutine()); + + const uint64_t size = bytes; + const unsigned char *buf = pbuf; + PVEBackupDevInfo *di = opaque; + + int ret = -1; + + assert(backup_state.vmaw); + assert(buf); + + uint64_t remaining = size; + + uint64_t cluster_num = start / VMA_CLUSTER_SIZE; + if ((cluster_num * VMA_CLUSTER_SIZE) != start) { + Error *local_err = NULL; + error_setg(&local_err, + "got unaligned write inside backup dump " + "callback (sector %ld)", start); + pvebackup_propagate_error(local_err); + return -1; // not aligned to cluster size + } + + while (remaining > 0) { + qemu_co_mutex_lock(&backup_state.dump_callback_mutex); + // avoid deadlock if job is cancelled + if (pvebackup_error_or_canceled()) { + qemu_co_mutex_unlock(&backup_state.dump_callback_mutex); + return -1; + } + + size_t zero_bytes = 0; + ret = vma_writer_write(backup_state.vmaw, di->dev_id, cluster_num, buf, &zero_bytes); + qemu_co_mutex_unlock(&backup_state.dump_callback_mutex); + + ++cluster_num; + buf += VMA_CLUSTER_SIZE; + if (ret < 0) { + Error *local_err = NULL; + vma_writer_error_propagate(backup_state.vmaw, &local_err); + pvebackup_propagate_error(local_err); + return ret; + } else { + if (remaining >= VMA_CLUSTER_SIZE) { + assert(ret == VMA_CLUSTER_SIZE); + pvebackup_add_transfered_bytes(VMA_CLUSTER_SIZE, zero_bytes, 0); + remaining -= VMA_CLUSTER_SIZE; + } else { + assert(ret == remaining); + pvebackup_add_transfered_bytes(remaining, zero_bytes, 0); + remaining = 0; + } + } + } + + return size; +} + +// assumes the caller holds backup_mutex +static void coroutine_fn pvebackup_co_cleanup(void) +{ + assert(qemu_in_coroutine()); + + qemu_mutex_lock(&backup_state.stat.lock); + backup_state.stat.finishing = true; + qemu_mutex_unlock(&backup_state.stat.lock); + + if (backup_state.vmaw) { + Error *local_err = NULL; + vma_writer_close(backup_state.vmaw, &local_err); + + if (local_err != NULL) { + pvebackup_propagate_error(local_err); + } + + backup_state.vmaw = NULL; + } + + if (backup_state.pbs) { + if (!pvebackup_error_or_canceled()) { + Error *local_err = NULL; + proxmox_backup_co_finish(backup_state.pbs, &local_err); + if (local_err != NULL) { + pvebackup_propagate_error(local_err); + } + } + + proxmox_backup_disconnect(backup_state.pbs); + backup_state.pbs = NULL; + } + + g_list_free(backup_state.di_list); + backup_state.di_list = NULL; + + qemu_mutex_lock(&backup_state.stat.lock); + backup_state.stat.end_time = time(NULL); + backup_state.stat.finishing = false; + qemu_mutex_unlock(&backup_state.stat.lock); +} + +static void coroutine_fn pvebackup_co_complete_stream(void *opaque) +{ + PVEBackupDevInfo *di = opaque; + int ret = di->completed_ret; + + qemu_mutex_lock(&backup_state.stat.lock); + bool starting = backup_state.stat.starting; + qemu_mutex_unlock(&backup_state.stat.lock); + if (starting) { + /* in 'starting' state, no tasks have been run yet, meaning we can (and + * must) skip all cleanup, as we don't know what has and hasn't been + * initialized yet. */ + return; + } + + qemu_co_mutex_lock(&backup_state.backup_mutex); + + if (ret < 0) { + Error *local_err = NULL; + error_setg(&local_err, "job failed with err %d - %s", ret, strerror(-ret)); + pvebackup_propagate_error(local_err); + } + + di->bs = NULL; + + assert(di->target == NULL); + + bool error_or_canceled = pvebackup_error_or_canceled(); + + if (backup_state.vmaw) { + vma_writer_close_stream(backup_state.vmaw, di->dev_id); + } + + if (backup_state.pbs && !error_or_canceled) { + Error *local_err = NULL; + proxmox_backup_co_close_image(backup_state.pbs, di->dev_id, &local_err); + if (local_err != NULL) { + pvebackup_propagate_error(local_err); + } + } + + if (di->job) { + WITH_JOB_LOCK_GUARD() { + job_unref_locked(&di->job->job); + di->job = NULL; + } + } + + // remove self from job list + backup_state.di_list = g_list_remove(backup_state.di_list, di); + + g_free(di); + + /* call cleanup if we're the last job */ + if (!g_list_first(backup_state.di_list)) { + pvebackup_co_cleanup(); + } + + qemu_co_mutex_unlock(&backup_state.backup_mutex); +} + +static void pvebackup_complete_cb(void *opaque, int ret) +{ + PVEBackupDevInfo *di = opaque; + di->completed_ret = ret; + + /* + * Schedule stream cleanup in async coroutine. close_image and finish might + * take a while, so we can't block on them here. This way it also doesn't + * matter if we're already running in a coroutine or not. + * Note: di is a pointer to an entry in the global backup_state struct, so + * it stays valid. + */ + Coroutine *co = qemu_coroutine_create(pvebackup_co_complete_stream, di); + aio_co_enter(qemu_get_aio_context(), co); +} + +/* + * job_cancel(_sync) does not like to be called from coroutines, so defer to + * main loop processing via a bottom half. Assumes that caller holds + * backup_mutex. + */ +static void job_cancel_bh(void *opaque) { + CoCtxData *data = (CoCtxData*)opaque; + + /* + * Be careful to pick a valid job to cancel: + * 1. job_cancel_sync() does not expect the job to be finalized already. + * 2. job_exit() might run between scheduling and running job_cancel_bh() + * and pvebackup_co_complete_stream() might not have removed the job from + * the list yet (in fact, cannot, because it waits for the backup_mutex). + * Requiring !job_is_completed() ensures that no finalized job is picked. + */ + GList *bdi = g_list_first(backup_state.di_list); + while (bdi) { + if (bdi->data) { + BlockJob *bj = ((PVEBackupDevInfo *)bdi->data)->job; + if (bj) { + Job *job = &bj->job; + WITH_JOB_LOCK_GUARD() { + if (!job_is_completed_locked(job)) { + job_cancel_sync_locked(job, true); + /* + * It's enough to cancel one job in the transaction, the + * rest will follow automatically. + */ + break; + } + } + } + } + bdi = g_list_next(bdi); + } + + aio_co_enter(data->ctx, data->co); +} + +void coroutine_fn qmp_backup_cancel(Error **errp) +{ + Error *cancel_err = NULL; + error_setg(&cancel_err, "backup canceled"); + pvebackup_propagate_error(cancel_err); + + qemu_co_mutex_lock(&backup_state.backup_mutex); + + if (backup_state.vmaw) { + /* make sure vma writer does not block anymore */ + vma_writer_set_error(backup_state.vmaw, "backup canceled"); + } + + if (backup_state.pbs) { + proxmox_backup_abort(backup_state.pbs, "backup canceled"); + } + + CoCtxData data = { + .ctx = qemu_get_current_aio_context(), + .co = qemu_coroutine_self(), + }; + aio_bh_schedule_oneshot(data.ctx, job_cancel_bh, &data); + qemu_coroutine_yield(); + + qemu_co_mutex_unlock(&backup_state.backup_mutex); +} + +// assumes the caller holds backup_mutex +static int coroutine_fn pvebackup_co_add_config( + const char *file, + const char *name, + BackupFormat format, + const char *backup_dir, + VmaWriter *vmaw, + ProxmoxBackupHandle *pbs, + Error **errp) +{ + int res = 0; + + char *cdata = NULL; + gsize clen = 0; + GError *err = NULL; + if (!g_file_get_contents(file, &cdata, &clen, &err)) { + error_setg(errp, "unable to read file '%s'", file); + return 1; + } + + char *basename = g_path_get_basename(file); + if (name == NULL) name = basename; + + if (format == BACKUP_FORMAT_VMA) { + if (vma_writer_add_config(vmaw, name, cdata, clen) != 0) { + error_setg(errp, "unable to add %s config data to vma archive", file); + goto err; + } + } else if (format == BACKUP_FORMAT_PBS) { + if (proxmox_backup_co_add_config(pbs, name, (unsigned char *)cdata, clen, errp) < 0) + goto err; + } else if (format == BACKUP_FORMAT_DIR) { + char config_path[PATH_MAX]; + snprintf(config_path, PATH_MAX, "%s/%s", backup_dir, name); + if (!g_file_set_contents(config_path, cdata, clen, &err)) { + error_setg(errp, "unable to write config file '%s'", config_path); + goto err; + } + } + + out: + g_free(basename); + g_free(cdata); + return res; + + err: + res = -1; + goto out; +} + +/* + * backup_job_create can *not* be run from a coroutine (and requires an + * acquired AioContext), so this can't either. + * The caller is responsible that backup_mutex is held nonetheless. + */ +static void create_backup_jobs_bh(void *opaque) { + + assert(!qemu_in_coroutine()); + + CoCtxData *data = (CoCtxData*)opaque; + Error **errp = (Error**)data->data; + + Error *local_err = NULL; + + /* create job transaction to synchronize bitmap commit and cancel all + * jobs in case one errors */ + if (backup_state.txn) { + job_txn_unref(backup_state.txn); + } + backup_state.txn = job_txn_new_seq(); + + /* create and start all jobs (paused state) */ + GList *l = backup_state.di_list; + while (l) { + PVEBackupDevInfo *di = (PVEBackupDevInfo *)l->data; + l = g_list_next(l); + + assert(di->target != NULL); + + MirrorSyncMode sync_mode = MIRROR_SYNC_MODE_FULL; + BitmapSyncMode bitmap_mode = BITMAP_SYNC_MODE_NEVER; + if (di->bitmap) { + sync_mode = MIRROR_SYNC_MODE_BITMAP; + bitmap_mode = BITMAP_SYNC_MODE_ON_SUCCESS; + } + AioContext *aio_context = bdrv_get_aio_context(di->bs); + aio_context_acquire(aio_context); + + BlockJob *job = backup_job_create( + NULL, di->bs, di->target, backup_state.speed, sync_mode, di->bitmap, + bitmap_mode, false, NULL, &backup_state.perf, BLOCKDEV_ON_ERROR_REPORT, + BLOCKDEV_ON_ERROR_REPORT, JOB_DEFAULT, pvebackup_complete_cb, di, backup_state.txn, + &local_err); + + aio_context_release(aio_context); + + di->job = job; + if (job) { + WITH_JOB_LOCK_GUARD() { + job_ref_locked(&job->job); + } + } + + if (!job || local_err) { + error_setg(errp, "backup_job_create failed: %s", + local_err ? error_get_pretty(local_err) : "null"); + break; + } + + bdrv_unref(di->target); + di->target = NULL; + } + + if (*errp) { + /* + * It's enough to cancel one job in the transaction, the rest will + * follow automatically. + */ + bool canceled = false; + l = backup_state.di_list; + while (l) { + PVEBackupDevInfo *di = (PVEBackupDevInfo *)l->data; + l = g_list_next(l); + + if (di->target) { + bdrv_unref(di->target); + di->target = NULL; + } + + if (di->job) { + WITH_JOB_LOCK_GUARD() { + if (!canceled) { + job_cancel_sync_locked(&di->job->job, true); + canceled = true; + } + job_unref_locked(&di->job->job); + di->job = NULL; + } + } + } + } + + /* return */ + aio_co_enter(data->ctx, data->co); +} + +UuidInfo coroutine_fn *qmp_backup( + const char *backup_file, + const char *password, + const char *keyfile, + const char *key_password, + const char *master_keyfile, + const char *fingerprint, + const char *backup_ns, + const char *backup_id, + bool has_backup_time, int64_t backup_time, + bool has_use_dirty_bitmap, bool use_dirty_bitmap, + bool has_compress, bool compress, + bool has_encrypt, bool encrypt, + bool has_format, BackupFormat format, + const char *config_file, + const char *firewall_file, + const char *devlist, + bool has_speed, int64_t speed, + bool has_max_workers, int64_t max_workers, + Error **errp) +{ + assert(qemu_in_coroutine()); + + qemu_co_mutex_lock(&backup_state.backup_mutex); + + BlockBackend *blk; + BlockDriverState *bs = NULL; + const char *backup_dir = NULL; + Error *local_err = NULL; + uuid_t uuid; + VmaWriter *vmaw = NULL; + ProxmoxBackupHandle *pbs = NULL; + gchar **devs = NULL; + GList *di_list = NULL; + GList *l; + UuidInfo *uuid_info; + + const char *config_name = "qemu-server.conf"; + const char *firewall_name = "qemu-server.fw"; + + if (backup_state.di_list) { + error_set(errp, ERROR_CLASS_GENERIC_ERROR, + "previous backup not finished"); + qemu_co_mutex_unlock(&backup_state.backup_mutex); + return NULL; + } + + /* Todo: try to auto-detect format based on file name */ + format = has_format ? format : BACKUP_FORMAT_VMA; + + if (devlist) { + devs = g_strsplit_set(devlist, ",;:", -1); + + gchar **d = devs; + while (d && *d) { + blk = blk_by_name(*d); + if (blk) { + bs = blk_bs(blk); + if (!bdrv_co_is_inserted(bs)) { + error_setg(errp, QERR_DEVICE_HAS_NO_MEDIUM, *d); + goto err; + } + PVEBackupDevInfo *di = g_new0(PVEBackupDevInfo, 1); + di->bs = bs; + di_list = g_list_append(di_list, di); + } else { + error_set(errp, ERROR_CLASS_DEVICE_NOT_FOUND, + "Device '%s' not found", *d); + goto err; + } + d++; + } + + } else { + BdrvNextIterator it; + + bs = NULL; + for (bs = bdrv_first(&it); bs; bs = bdrv_next(&it)) { + if (!bdrv_co_is_inserted(bs) || bdrv_is_read_only(bs)) { + continue; + } + + PVEBackupDevInfo *di = g_new0(PVEBackupDevInfo, 1); + di->bs = bs; + di_list = g_list_append(di_list, di); + } + } + + if (!di_list) { + error_set(errp, ERROR_CLASS_GENERIC_ERROR, "empty device list"); + goto err; + } + + size_t total = 0; + + l = di_list; + while (l) { + PVEBackupDevInfo *di = (PVEBackupDevInfo *)l->data; + l = g_list_next(l); + if (bdrv_op_is_blocked(di->bs, BLOCK_OP_TYPE_BACKUP_SOURCE, errp)) { + goto err; + } + + ssize_t size = bdrv_getlength(di->bs); + if (size < 0) { + error_setg_errno(errp, -size, "bdrv_getlength failed"); + goto err; + } + di->size = size; + total += size; + + di->completed_ret = INT_MAX; + } + + uuid_generate(uuid); + + qemu_mutex_lock(&backup_state.stat.lock); + backup_state.stat.reused = 0; + + /* clear previous backup's bitmap_list */ + if (backup_state.stat.bitmap_list) { + GList *bl = backup_state.stat.bitmap_list; + while (bl) { + g_free(((PBSBitmapInfo *)bl->data)->drive); + g_free(bl->data); + bl = g_list_next(bl); + } + g_list_free(backup_state.stat.bitmap_list); + backup_state.stat.bitmap_list = NULL; + } + + if (format == BACKUP_FORMAT_PBS) { + if (!password) { + error_set(errp, ERROR_CLASS_GENERIC_ERROR, "missing parameter 'password'"); + goto err_mutex; + } + if (!backup_id) { + error_set(errp, ERROR_CLASS_GENERIC_ERROR, "missing parameter 'backup-id'"); + goto err_mutex; + } + if (!has_backup_time) { + error_set(errp, ERROR_CLASS_GENERIC_ERROR, "missing parameter 'backup-time'"); + goto err_mutex; + } + + int dump_cb_block_size = PROXMOX_BACKUP_DEFAULT_CHUNK_SIZE; // Hardcoded (4M) + firewall_name = "fw.conf"; + + char *pbs_err = NULL; + pbs = proxmox_backup_new_ns( + backup_file, + backup_ns, + backup_id, + backup_time, + dump_cb_block_size, + password, + keyfile, + key_password, + master_keyfile, + has_compress ? compress : true, + has_encrypt ? encrypt : !!keyfile, + fingerprint, + &pbs_err); + + if (!pbs) { + error_set(errp, ERROR_CLASS_GENERIC_ERROR, + "proxmox_backup_new failed: %s", pbs_err); + proxmox_backup_free_error(pbs_err); + goto err_mutex; + } + + int connect_result = proxmox_backup_co_connect(pbs, errp); + if (connect_result < 0) + goto err_mutex; + + /* register all devices */ + l = di_list; + while (l) { + PVEBackupDevInfo *di = (PVEBackupDevInfo *)l->data; + l = g_list_next(l); + + di->block_size = dump_cb_block_size; + + const char *devname = bdrv_get_device_name(di->bs); + PBSBitmapAction action = PBS_BITMAP_ACTION_NOT_USED; + size_t dirty = di->size; + + BdrvDirtyBitmap *bitmap = bdrv_find_dirty_bitmap(di->bs, PBS_BITMAP_NAME); + bool expect_only_dirty = false; + + if (has_use_dirty_bitmap && use_dirty_bitmap) { + if (bitmap == NULL) { + bitmap = bdrv_create_dirty_bitmap(di->bs, dump_cb_block_size, PBS_BITMAP_NAME, errp); + if (!bitmap) { + goto err_mutex; + } + action = PBS_BITMAP_ACTION_NEW; + } else { + expect_only_dirty = proxmox_backup_check_incremental(pbs, devname, di->size) != 0; + } + + if (expect_only_dirty) { + /* track clean chunks as reused */ + dirty = MIN(bdrv_get_dirty_count(bitmap), di->size); + backup_state.stat.reused += di->size - dirty; + action = PBS_BITMAP_ACTION_USED; + } else { + /* mark entire bitmap as dirty to make full backup */ + bdrv_set_dirty_bitmap(bitmap, 0, di->size); + if (action != PBS_BITMAP_ACTION_NEW) { + action = PBS_BITMAP_ACTION_INVALID; + } + } + di->bitmap = bitmap; + } else { + /* after a full backup the old dirty bitmap is invalid anyway */ + if (bitmap != NULL) { + bdrv_release_dirty_bitmap(bitmap); + action = PBS_BITMAP_ACTION_NOT_USED_REMOVED; + } + } + + int dev_id = proxmox_backup_co_register_image(pbs, devname, di->size, expect_only_dirty, errp); + if (dev_id < 0) { + goto err_mutex; + } + + if (!(di->target = bdrv_backup_dump_create(dump_cb_block_size, di->size, pvebackup_co_dump_pbs_cb, di, errp))) { + goto err_mutex; + } + + di->dev_id = dev_id; + + PBSBitmapInfo *info = g_malloc(sizeof(*info)); + info->drive = g_strdup(devname); + info->action = action; + info->size = di->size; + info->dirty = dirty; + backup_state.stat.bitmap_list = g_list_append(backup_state.stat.bitmap_list, info); + } + } else if (format == BACKUP_FORMAT_VMA) { + vmaw = vma_writer_create(backup_file, uuid, &local_err); + if (!vmaw) { + if (local_err) { + error_propagate(errp, local_err); + } + goto err_mutex; + } + + /* register all devices for vma writer */ + l = di_list; + while (l) { + PVEBackupDevInfo *di = (PVEBackupDevInfo *)l->data; + l = g_list_next(l); + + if (!(di->target = bdrv_backup_dump_create(VMA_CLUSTER_SIZE, di->size, pvebackup_co_dump_vma_cb, di, errp))) { + goto err_mutex; + } + + const char *devname = bdrv_get_device_name(di->bs); + di->dev_id = vma_writer_register_stream(vmaw, devname, di->size); + if (di->dev_id <= 0) { + error_set(errp, ERROR_CLASS_GENERIC_ERROR, + "register_stream failed"); + goto err_mutex; + } + } + } else if (format == BACKUP_FORMAT_DIR) { + if (mkdir(backup_file, 0640) != 0) { + error_setg_errno(errp, errno, "can't create directory '%s'\n", + backup_file); + goto err_mutex; + } + backup_dir = backup_file; + + l = di_list; + while (l) { + PVEBackupDevInfo *di = (PVEBackupDevInfo *)l->data; + l = g_list_next(l); + + const char *devname = bdrv_get_device_name(di->bs); + snprintf(di->targetfile, PATH_MAX, "%s/%s.raw", backup_dir, devname); + + int flags = BDRV_O_RDWR; + bdrv_img_create(di->targetfile, "raw", NULL, NULL, NULL, + di->size, flags, false, &local_err); + if (local_err) { + error_propagate(errp, local_err); + goto err_mutex; + } + + di->target = bdrv_co_open(di->targetfile, NULL, NULL, flags, &local_err); + if (!di->target) { + error_propagate(errp, local_err); + goto err_mutex; + } + } + } else { + error_set(errp, ERROR_CLASS_GENERIC_ERROR, "unknown backup format"); + goto err_mutex; + } + + + /* add configuration file to archive */ + if (config_file) { + if (pvebackup_co_add_config(config_file, config_name, format, backup_dir, + vmaw, pbs, errp) != 0) { + goto err_mutex; + } + } + + /* add firewall file to archive */ + if (firewall_file) { + if (pvebackup_co_add_config(firewall_file, firewall_name, format, backup_dir, + vmaw, pbs, errp) != 0) { + goto err_mutex; + } + } + /* initialize global backup_state now */ + /* note: 'reused' and 'bitmap_list' are initialized earlier */ + + if (backup_state.stat.error) { + error_free(backup_state.stat.error); + backup_state.stat.error = NULL; + } + + backup_state.stat.start_time = time(NULL); + backup_state.stat.end_time = 0; + + if (backup_state.stat.backup_file) { + g_free(backup_state.stat.backup_file); + } + backup_state.stat.backup_file = g_strdup(backup_file); + + uuid_copy(backup_state.stat.uuid, uuid); + uuid_unparse_lower(uuid, backup_state.stat.uuid_str); + char *uuid_str = g_strdup(backup_state.stat.uuid_str); + + backup_state.stat.total = total; + backup_state.stat.dirty = total - backup_state.stat.reused; + backup_state.stat.transferred = 0; + backup_state.stat.zero_bytes = 0; + backup_state.stat.finishing = false; + backup_state.stat.starting = true; + + qemu_mutex_unlock(&backup_state.stat.lock); + + backup_state.speed = (has_speed && speed > 0) ? speed : 0; + + backup_state.perf = (BackupPerf){ .max_workers = 16 }; + if (has_max_workers) { + backup_state.perf.max_workers = max_workers; + } + + backup_state.vmaw = vmaw; + backup_state.pbs = pbs; + + backup_state.di_list = di_list; + + uuid_info = g_malloc0(sizeof(*uuid_info)); + uuid_info->UUID = uuid_str; + + /* Run create_backup_jobs_bh outside of coroutine (in BH) but keep + * backup_mutex locked. This is fine, a CoMutex can be held across yield + * points, and we'll release it as soon as the BH reschedules us. + */ + CoCtxData waker = { + .co = qemu_coroutine_self(), + .ctx = qemu_get_current_aio_context(), + .data = &local_err, + }; + aio_bh_schedule_oneshot(waker.ctx, create_backup_jobs_bh, &waker); + qemu_coroutine_yield(); + + if (local_err) { + error_propagate(errp, local_err); + goto err; + } + + qemu_co_mutex_unlock(&backup_state.backup_mutex); + + qemu_mutex_lock(&backup_state.stat.lock); + backup_state.stat.starting = false; + qemu_mutex_unlock(&backup_state.stat.lock); + + /* start the first job in the transaction */ + job_txn_start_seq(backup_state.txn); + + return uuid_info; + +err_mutex: + qemu_mutex_unlock(&backup_state.stat.lock); + +err: + + l = di_list; + while (l) { + PVEBackupDevInfo *di = (PVEBackupDevInfo *)l->data; + l = g_list_next(l); + + if (di->target) { + bdrv_co_unref(di->target); + } + + if (di->targetfile[0]) { + unlink(di->targetfile); + } + g_free(di); + } + g_list_free(di_list); + backup_state.di_list = NULL; + + if (devs) { + g_strfreev(devs); + } + + if (vmaw) { + Error *err = NULL; + vma_writer_close(vmaw, &err); + unlink(backup_file); + } + + if (pbs) { + proxmox_backup_disconnect(pbs); + backup_state.pbs = NULL; + } + + if (backup_dir) { + rmdir(backup_dir); + } + + qemu_co_mutex_unlock(&backup_state.backup_mutex); + return NULL; +} + +BackupStatus *qmp_query_backup(Error **errp) +{ + BackupStatus *info = g_malloc0(sizeof(*info)); + + qemu_mutex_lock(&backup_state.stat.lock); + + if (!backup_state.stat.start_time) { + /* not started, return {} */ + qemu_mutex_unlock(&backup_state.stat.lock); + return info; + } + + info->has_start_time = true; + info->start_time = backup_state.stat.start_time; + + if (backup_state.stat.backup_file) { + info->backup_file = g_strdup(backup_state.stat.backup_file); + } + + info->uuid = g_strdup(backup_state.stat.uuid_str); + + if (backup_state.stat.end_time) { + if (backup_state.stat.error) { + info->status = g_strdup("error"); + info->errmsg = g_strdup(error_get_pretty(backup_state.stat.error)); + } else { + info->status = g_strdup("done"); + } + info->has_end_time = true; + info->end_time = backup_state.stat.end_time; + } else { + info->status = g_strdup("active"); + } + + info->has_total = true; + info->total = backup_state.stat.total; + info->has_dirty = true; + info->dirty = backup_state.stat.dirty; + info->has_zero_bytes = true; + info->zero_bytes = backup_state.stat.zero_bytes; + info->has_transferred = true; + info->transferred = backup_state.stat.transferred; + info->has_reused = true; + info->reused = backup_state.stat.reused; + info->finishing = backup_state.stat.finishing; + + qemu_mutex_unlock(&backup_state.stat.lock); + + return info; +} + +PBSBitmapInfoList *qmp_query_pbs_bitmap_info(Error **errp) +{ + PBSBitmapInfoList *head = NULL, **p_next = &head; + + qemu_mutex_lock(&backup_state.stat.lock); + + GList *l = backup_state.stat.bitmap_list; + while (l) { + PBSBitmapInfo *info = (PBSBitmapInfo *)l->data; + l = g_list_next(l); + + /* clone bitmap info to avoid auto free after QMP marshalling */ + PBSBitmapInfo *info_ret = g_malloc0(sizeof(*info_ret)); + info_ret->drive = g_strdup(info->drive); + info_ret->action = info->action; + info_ret->size = info->size; + info_ret->dirty = info->dirty; + + PBSBitmapInfoList *info_list = g_malloc0(sizeof(*info_list)); + info_list->value = info_ret; + + *p_next = info_list; + p_next = &info_list->next; + } + + qemu_mutex_unlock(&backup_state.stat.lock); + + return head; +} + +ProxmoxSupportStatus *qmp_query_proxmox_support(Error **errp) +{ + ProxmoxSupportStatus *ret = g_malloc0(sizeof(*ret)); + ret->pbs_library_version = g_strdup(proxmox_backup_qemu_version()); + ret->pbs_dirty_bitmap = true; + ret->pbs_dirty_bitmap_savevm = true; + ret->query_bitmap_info = true; + ret->pbs_masterkey = true; + ret->backup_max_workers = true; + return ret; +} diff --git a/qapi/block-core.json b/qapi/block-core.json index 542add004b..4ec70acf95 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -835,6 +835,232 @@ { 'command': 'query-block', 'returns': ['BlockInfo'], 'allow-preconfig': true } +## +# @BackupStatus: +# +# Detailed backup status. +# +# @status: string describing the current backup status. +# This can be 'active', 'done', 'error'. If this field is not +# returned, no backup process has been initiated +# +# @errmsg: error message (only returned if status is 'error') +# +# @total: total amount of bytes involved in the backup process +# +# @dirty: with incremental mode (PBS) this is the amount of bytes involved +# in the backup process which are marked dirty. +# +# @transferred: amount of bytes already backed up. +# +# @reused: amount of bytes reused due to deduplication. +# +# @zero-bytes: amount of 'zero' bytes detected. +# +# @start-time: time (epoch) when backup job started. +# +# @end-time: time (epoch) when backup job finished. +# +# @backup-file: backup file name +# +# @uuid: uuid for this backup job +# +# @finishing: if status='active' and finishing=true, then the backup process is +# waiting for the target to finish. +# +## +{ 'struct': 'BackupStatus', + 'data': {'*status': 'str', '*errmsg': 'str', '*total': 'int', '*dirty': 'int', + '*transferred': 'int', '*zero-bytes': 'int', '*reused': 'int', + '*start-time': 'int', '*end-time': 'int', + '*backup-file': 'str', '*uuid': 'str', 'finishing': 'bool' } } + +## +# @BackupFormat: +# +# An enumeration of supported backup formats. +# +# @vma: Proxmox vma backup format +## +{ 'enum': 'BackupFormat', + 'data': [ 'vma', 'dir', 'pbs' ] } + +## +# @backup: +# +# Starts a VM backup. +# +# @backup-file: the backup file name +# +# @format: format of the backup file +# +# @config-file: a configuration file to include into +# the backup archive. +# +# @speed: the maximum speed, in bytes per second +# +# @devlist: list of block device names (separated by ',', ';' +# or ':'). By default the backup includes all writable block devices. +# +# @password: backup server passsword (required for format 'pbs') +# +# @keyfile: keyfile used for encryption (optional for format 'pbs') +# +# @key-password: password for keyfile (optional for format 'pbs') +# +# @master-keyfile: PEM-formatted master public keyfile (optional for format 'pbs') +# +# @fingerprint: server cert fingerprint (optional for format 'pbs') +# +# @backup-ns: backup namespace (required for format 'pbs') +# +# @backup-id: backup ID (required for format 'pbs') +# +# @backup-time: backup timestamp (Unix epoch, required for format 'pbs') +# +# @use-dirty-bitmap: use dirty bitmap to detect incremental changes since last job (optional for format 'pbs') +# +# @compress: use compression (optional for format 'pbs', defaults to true) +# +# @encrypt: use encryption ((optional for format 'pbs', defaults to true if there is a keyfile) +# +# @max-workers: see @BackupPerf for details. Default 16. +# +# Returns: the uuid of the backup job +# +## +{ 'command': 'backup', 'data': { 'backup-file': 'str', + '*password': 'str', + '*keyfile': 'str', + '*key-password': 'str', + '*master-keyfile': 'str', + '*fingerprint': 'str', + '*backup-ns': 'str', + '*backup-id': 'str', + '*backup-time': 'int', + '*use-dirty-bitmap': 'bool', + '*compress': 'bool', + '*encrypt': 'bool', + '*format': 'BackupFormat', + '*config-file': 'str', + '*firewall-file': 'str', + '*devlist': 'str', + '*speed': 'int', + '*max-workers': 'int' }, + 'returns': 'UuidInfo', 'coroutine': true } + +## +# @query-backup: +# +# Returns information about current/last backup task. +# +# Returns: @BackupStatus +# +## +{ 'command': 'query-backup', 'returns': 'BackupStatus' } + +## +# @backup-cancel: +# +# Cancel the current executing backup process. +# +# Returns: nothing on success +# +# Notes: This command succeeds even if there is no backup process running. +# +## +{ 'command': 'backup-cancel', 'coroutine': true } + +## +# @ProxmoxSupportStatus: +# +# Contains info about supported features added by Proxmox. +# +# @pbs-dirty-bitmap: True if dirty-bitmap-incremental backups to PBS are +# supported. +# +# @query-bitmap-info: True if the 'query-pbs-bitmap-info' QMP call is supported. +# +# @pbs-dirty-bitmap-savevm: True if 'dirty-bitmaps' migration capability can +# safely be set for savevm-async. +# +# @pbs-masterkey: True if the QMP backup call supports the 'master_keyfile' +# parameter. +# +# @pbs-library-version: Running version of libproxmox-backup-qemu0 library. +# +## +{ 'struct': 'ProxmoxSupportStatus', + 'data': { 'pbs-dirty-bitmap': 'bool', + 'query-bitmap-info': 'bool', + 'pbs-dirty-bitmap-savevm': 'bool', + 'pbs-masterkey': 'bool', + 'pbs-library-version': 'str', + 'backup-max-workers': 'bool' } } + +## +# @query-proxmox-support: +# +# Returns information about supported features added by Proxmox. +# +# Returns: @ProxmoxSupportStatus +# +## +{ 'command': 'query-proxmox-support', 'returns': 'ProxmoxSupportStatus' } + +## +# @PBSBitmapAction: +# +# An action taken on a dirty-bitmap when a backup job was started. +# +# @not-used: Bitmap mode was not enabled. +# +# @not-used-removed: Bitmap mode was not enabled, but a bitmap from a +# previous backup still existed and was removed. +# +# @new: A new bitmap was attached to the drive for this backup. +# +# @used: An existing bitmap will be used to only backup changed data. +# +# @invalid: A bitmap existed, but had to be cleared since it's associated +# base snapshot did not match the base given for the current job or +# the crypt mode has changed. +# +## +{ 'enum': 'PBSBitmapAction', + 'data': ['not-used', 'not-used-removed', 'new', 'used', 'invalid'] } + +## +# @PBSBitmapInfo: +# +# Contains information about dirty bitmaps used for each drive in a PBS backup. +# +# @drive: The underlying drive. +# +# @action: The action that was taken when the backup started. +# +# @size: The total size of the drive. +# +# @dirty: How much of the drive is considered dirty and will be backed up, +# or 'size' if everything will be. +# +## +{ 'struct': 'PBSBitmapInfo', + 'data': { 'drive': 'str', 'action': 'PBSBitmapAction', 'size': 'int', + 'dirty': 'int' } } + +## +# @query-pbs-bitmap-info: +# +# Returns information about dirty bitmaps used on the most recently started +# backup. Returns nothing when the last backup was not using PBS or if no +# backup occured in this session. +# +# Returns: @PBSBitmapInfo +# +## +{ 'command': 'query-pbs-bitmap-info', 'returns': ['PBSBitmapInfo'] } + ## # @BlockDeviceTimedStats: # diff --git a/qapi/common.json b/qapi/common.json index 356db3f670..aae8a3b682 100644 --- a/qapi/common.json +++ b/qapi/common.json @@ -206,3 +206,16 @@ ## { 'struct': 'HumanReadableText', 'data': { 'human-readable-text': 'str' } } + +## +# @UuidInfo: +# +# Guest UUID information (Universally Unique Identifier). +# +# @UUID: the UUID of the guest +# +# Since: 0.14.0 +# +# Notes: If no UUID was specified for the guest, a null UUID is returned. +## +{ 'struct': 'UuidInfo', 'data': {'UUID': 'str'} } diff --git a/qapi/machine.json b/qapi/machine.json index 47f3facdb2..46760978ae 100644 --- a/qapi/machine.json +++ b/qapi/machine.json @@ -4,6 +4,8 @@ # This work is licensed under the terms of the GNU GPL, version 2 or later. # See the COPYING file in the top-level directory. +{ 'include': 'common.json' } + ## # = Machines ## @@ -228,19 +230,6 @@ ## { 'command': 'query-target', 'returns': 'TargetInfo' } -## -# @UuidInfo: -# -# Guest UUID information (Universally Unique Identifier). -# -# @UUID: the UUID of the guest -# -# Since: 0.14 -# -# Notes: If no UUID was specified for the guest, a null UUID is returned. -## -{ 'struct': 'UuidInfo', 'data': {'UUID': 'str'} } - ## # @query-uuid: #