summaryrefslogtreecommitdiff
path: root/migration.c
AgeCommit message (Collapse)Author
2013-06-27rdma: introduce capability x-rdma-pin-allMichael R. Hines
This capability allows you to disable dynamic chunk registration for better throughput on high-performance links. For example, using an 8GB RAM virtual machine with all 8GB of memory in active use and the VM itself is completely idle using a 40 gbps infiniband link: 1. x-rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps 2. x-rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps These numbers would of course scale up to whatever size virtual machine you have to migrate using RDMA. Enabling this feature does *not* have any measurable affect on migration *downtime*. This is because, without this feature, all of the memory will have already been registered already in advance during the bulk round and does not need to be re-registered during the successive iteration rounds. Reviewed-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Reviewed-by: Chegu Vinod <chegu_vinod@hp.com> Reviewed-by: Eric Blake <eblake@redhat.com> Tested-by: Chegu Vinod <chegu_vinod@hp.com> Tested-by: Michael R. Hines <mrhines@us.ibm.com> Signed-off-by: Michael R. Hines <mrhines@us.ibm.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-06-27rdma: export throughput w/ MigrationStats QMPMichael R. Hines
This exposes throughput (in megabits/sec) through QMP. Reviewed-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Reviewed-by: Chegu Vinod <chegu_vinod@hp.com> Tested-by: Chegu Vinod <chegu_vinod@hp.com> Tested-by: Michael R. Hines <mrhines@us.ibm.com> Signed-off-by: Michael R. Hines <mrhines@us.ibm.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-06-01migration: Remove duplicate bandwidth_limit setLei Li
bandwidth_limit is double set in migrate_init(), remove one. Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com> Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
2013-05-17Revert "migration: don't account sleep time for calculating bandwidth"Michael Roth
This reverts commit 7161082c8d8cf167c508976887a0a63f4db92b51. Reverting this patch fixes a divide-by-zero error in qemu that can be fairly reliably triggered by doing block migration. In this case, the configuration/error was: source: temp/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -L temp-bios -M pc-i440fx-1.4 -m 512M -kernel boot/vmlinuz-x86_64 -initrd boot/test-initramfs-x86_64.img.gz -vga std -append seed=1234 -drive file=disk1.img,if=virtio -drive file=disk2.img,if=virtio -device virtio-net-pci,netdev=net0 -netdev user,id=net0 -monitor unix:/tmp/vm-hmp.sock,server,nowait -qmp unix:/tmp/vm-qmp.sock,server,nowait -vnc :100 16837 Floating point exception(core dumped) target: temp/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -L temp-bios -M pc-i440fx-1.4 -m 512M -kernel boot/vmlinuz-x86_64 -initrd boot/test-initramfs-x86_64.img.gz -vga std -append seed=1234 -drive file=target_disk1.img,if=virtio -drive file=target_disk2.img,if=virtio -device virtio-net-pci,netdev=net0 -netdev user,id=net0 -incoming unix:/tmp/migrate.sock -monitor unix:/tmp/vm-hmp-incoming.sock,server,nowait -qmp unix:/tmp/vm-qmp-incoming.sock,server,nowait -vnc :101 Receiving block device images 20 % 21 % load of migration failed This revert potentially re-introduces a bug that was present in 1.4, but fixes a prevalent issue with block migration so we should revert it for now and take an updated patch later. Conflicts: migration.c * fixed up to remove logic introduced in 7161082c while leaving changes in HEAD intact Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com> Message-id: 1368739544-31021-1-git-send-email-mdroth@linux.vnet.ibm.com Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2013-04-17migration: reflect incoming failure to shellEric Blake
Management apps like libvirt don't know to pay attention to stderr unless there is a non-zero exit status. * migration.c (process_incoming_migration_co): Exit with non-zero status on failure. Signed-off-by: Eric Blake <eblake@redhat.com> Message-id: 1366149041-626-1-git-send-email-eblake@redhat.com Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2013-04-02oslib-posix: rename socket_set_nonblock() to qemu_set_nonblock()Stefan Hajnoczi
The fcntl(fd, F_SETFL, O_NONBLOCK) flag is not specific to sockets. Rename to qemu_set_nonblock() just like qemu_set_cloexec(). Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
2013-03-26migration: do not sent zero pages in bulk stagePeter Lieven
during bulk stage of ram migration if a page is a zero page do not send it at all. the memory at the destination reads as zero anyway. even if there is an madvise with QEMU_MADV_DONTNEED at the target upon receipt of a zero page I have observed that the target starts swapping if the memory is overcommitted. it seems that the pages are dropped asynchronously. this patch also updates QMP to return the number of skipped pages in MigrationStats. Signed-off-by: Peter Lieven <pl@kamp.de> Reviewed-by: Eric Blake <eblake@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: inline migrate_fd_closePaolo Bonzini
Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: eliminate s->migration_filePaolo Bonzini
The indirection is useless now. Backends can open s->file directly. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: move contents of migration_close to migrate_fd_cleanupPaolo Bonzini
With this patch, the migration_file is not needed anymore. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: move rate limiting to QEMUFilePaolo Bonzini
Rate limiting is now simply a byte counter; client call qemu_file_rate_limit() manually to determine if they have to exit. So it is possible and simple to move the functionality to QEMUFile. This makes the remaining functionality of s->file redundant; in the next patch we can remove it and write directly to s->migration_file. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: small changes around rate-limitingPaolo Bonzini
This patch extracts a few small changes from the next patch, which are unrelated to adding generic rate-limiting functionality to QEMUFile. Make migration_set_rate_limit a simple accessor, and use qemu_file_set_rate_limit consistently. Also fix a typo where INT_MAX should have been SIZE_MAX. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: use qemu_ftell to compute bandwidthPaolo Bonzini
Prepare for when s->bytes_xfer will be removed. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: use QEMUFile for writing outgoing migration dataPaolo Bonzini
Second, drop the file descriptor indirection, and write directly to the QEMUFile. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: use QEMUFile for migration channel lifetimePaolo Bonzini
As a start, use QEMUFile to store the destination and close it. qemu_get_fd gets a file descriptor that will be used by the write callbacks. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: use qemu_file_rate_limit consistentlyPaolo Bonzini
Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: remove useless qemu_file_get_error checkPaolo Bonzini
migration_put_buffer is never called if there has been an error. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: detect error before sleepingPaolo Bonzini
Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: eliminate last_roundPaolo Bonzini
We will go around the loop exactly once after setting last_round. Eliminate the variable altogether. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11Rename buffered_ to migration_Juan Quintela
This is consistent once that we have moved everything to migration.c Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: yay, buffering is gonePaolo Bonzini
Buffering was needed because blocking writes could take a long time and starve other threads seeking to grab the big QEMU mutex. Now that all writes (except within _complete callbacks) are done outside the big QEMU mutex, we do not need buffering at all. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: run setup callbacks out of big lockPaolo Bonzini
Only the migration_bitmap_sync() call needs the iothread lock. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: run pending/iterate callbacks out of big lockPaolo Bonzini
This makes it possible to do blocking writes directly to the socket, with no buffer in the middle. For RAM, only the migration_bitmap_sync() call needs the iothread lock. For block migration, it is needed by the block layer (including bdrv_drain_all and dirty bitmap access), but because some code is shared between iterate and complete, all of mig_save_device_dirty is run with the lock taken. In the savevm case, the iterate callback runs within the big lock. This is annoying because it complicates the rules. Luckily we do not need to do anything about it: the RAM iterate callback does not need the iothread lock, and block migration never runs during savevm. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: cleanup migration (including thread) in the iothreadPaolo Bonzini
Perform final cleanup in a bottom half, and add joining the thread to the series of cleanup actions. migrate_fd_error remains for connection error, but it doesn't need to cleanup anything anymore. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: prepare to access s->state outside critical sectionsPaolo Bonzini
Accessing s->state outside the big QEMU lock will simplify a bit the locking/unlocking of the iothread lock. We need to keep the lock in migrate_fd_error and migrate_fd_completed, however, because they call migrate_fd_cleanup. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: add migrate_set_state tracepointKazuya Saito
Signed-off-by: Kazuya Saito <saito.kazuya@jp.fujitsu.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: do not nest flushing of device dataPaolo Bonzini
Completion of migration is currently done with a "nested" loop that invokes buffered_flush: migrate_fd_completed is called by buffered_file_thread, which calls migrate_fd_cleanup, which calls buffered_close (via qemu_fclose), which flushes the buffer. Simplify this, by reusing the buffered_flush call of buffered_file_thread. Then if qemu_savevm_state_complete was called, and the buffer is empty (including the QEMUFile buffer, for which we need the previous patch), we are done. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: simplify error handlingPaolo Bonzini
Always use qemu_file_get_error to detect errors, since that is how QEMUFile itself drops I/O after an error occurs. There is no need to propagate and check return values all the time. Also remove the "complete" member, since we know that it is set (via migrate_fd_cleanup) only when the state changes. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: use qemu_file_set_errorPaolo Bonzini
Remove the return value of buffered_flush, pass it via the error code of s->file. Once this is done, the error can be retrieved simply via migrate_fd_close's call to qemu_fclose. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: flush all data to fd when buffered_flush is calledPaolo Bonzini
Including data that resided in the QEMUFile's own buffer. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: move more error handling to migrate_fd_cleanupPaolo Bonzini
The next patch will add more cases where qemu_savevm_state_cancel needs to be called; prepare for that already, the function can be called twice with no ill effect. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: always use vm_stop_force_statePaolo Bonzini
vm_stop_force_state does: if (runstate_is_running()) { vm_stop(state); } else { runstate_set(state); } migration.c does: if (runstate_is_running()) { vm_stop(state); } else { vm_stop_force_state(state); } The code run is the same even if we always use vm_stop_force_state in migration.c. Reviewed-by: Orit Wasserman <owasserm@redhat.com> Reviewed-by: Juan Quintela <quintela@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-03-11migration: simplify while loopPaolo Bonzini
Unify the goto around the loop, with the exit condition at the end of it. Both can be expressed as "while (ret >= 0)". Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
2013-02-22migration: calculate expected_downtimeJuan Quintela
We removed the calculation in commit e4ed1541ac9413eac494a03532e34beaf8a7d1c5 Now we add it back. We need to create dirty_bytes_rate because we can't include cpu-all.h from migration.c, and there is no other way to include TARGET_PAGE_SIZE. Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2013-02-22migration: don't account sleep time for calculating bandwidthJuan Quintela
While we are sleeping we are not sending, so we should not use that time to estimate our bandwidth. Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2013-02-22migration: calculate end time after we have sent the dataJuan Quintela
Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2013-02-22migration: change initial value of expected_downtimeJuan Quintela
0 is a very bad initial value, what we are trying to get is max_downtime, so that is a much better estimation. Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Orit Wasserman <owasserm@redhat.com>
2013-02-11migration: restrict scope of incoming fd read handlerStefan Hajnoczi
The incoming migration is processed in a coroutine and uses an fd read handler to enter the yielded coroutine when data becomes available. The read handler was set too broadly, so that spurious coroutine entries were be triggered if other coroutine users yielded (like the block layer's bdrv_write() function). Install the fd read only only when yielding for more data to become available. This prevents spurious coroutine entries which break code that assumes only a specific set of places can re-enter the coroutine. This patch fixes crashes in block/raw-posix.c that are triggered with "migrate -b" when qiov becomes a dangling pointer due to a spurious coroutine entry that frees qiov early. Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> Message-id: 1360598505-5512-1-git-send-email-stefanha@redhat.com Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2013-02-11error: Strip trailing '\n' from error string arguments (again)Markus Armbruster
Commit 6daf194d and be62a2eb got rid of a bunch, but they keep coming back. Tracked down with this Coccinelle semantic patch: @r@ expression err, eno, cls, fmt; position p; @@ ( error_report(fmt, ...)@p | error_set(err, cls, fmt, ...)@p | error_set_errno(err, eno, cls, fmt, ...)@p | error_setg(err, fmt, ...)@p | error_setg_errno(err, eno, fmt, ...)@p ) @script:python@ fmt << r.fmt; p << r.p; @@ if "\\n" in str(fmt): print "%s:%s:%s:%s" % (p[0].file, p[0].line, p[0].column, fmt) Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-id: 1360354939-10994-4-git-send-email-armbru@redhat.com Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2013-01-17migration: remove argument to qemu_savevm_state_cancelJuan Quintela
Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
2013-01-17migration: Only go to the iterate stage if there is anything to sendJuan Quintela
Signed-off-by: Orit Wasserman <owasserm@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
2013-01-17migration: unfold rest of migrate_fd_put_ready() into threadJuan Quintela
This will allow us finer control in next patches. Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
2013-01-17migration: move exit condition to migration threadJuan Quintela
Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
2013-01-17migration: Add buffered_flush error handlingJuan Quintela
Now that we have error handling we can do proper handling of buffered_flush(). Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
2013-01-17migration: move beginning stage to the migration threadJuan Quintela
Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Reviewed-by: Eric Blake <eblake@redhat.com> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
2013-01-17migration: fix off-by-one in buffered_rate_limitPaolo Bonzini
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Reviewed-by: Eric Blake <eblake@redhat.com>
2013-01-17migration: remove double call to migrate_fd_closePaolo Bonzini
The call in buffered_close is enough, because buffered_close is called already by migrate_fd_cleanup. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Reviewed-by: Eric Blake <eblake@redhat.com>
2013-01-17migration: make function staticPaolo Bonzini
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Reviewed-by: Eric Blake <eblake@redhat.com>
2013-01-17use XFER_LIMIT_RATIO consistentlyPaolo Bonzini
commit 5b4e1eb769eee892b44d3f6b2369b05196442f59 missed this use. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com> Reviewed-by: Reviewed-by: Eric Blake <eblake@redhat.com>
2012-12-20migration: merge QEMUFileBuffered into MigrationStateJuan Quintela
Avoid splitting the state of outgoing migration, more or less arbitrarily, between two data structures. QEMUFileBuffered anyway is used only during migration. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Juan Quintela <quintela@redhat.com>
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-16 08:39:42 +0000