summaryrefslogtreecommitdiff
path: root/ui/vnc-enc-tight.c
AgeCommit message (Collapse)Author
2019-09-17vnc: fix memory leak when vnc disconnectLi Qiang
Currently when qemu receives a vnc connect, it creates a 'VncState' to represent this connection. In 'vnc_worker_thread_loop' it creates a local 'VncState'. The connection 'VcnState' and local 'VncState' exchange data in 'vnc_async_encoding_start' and 'vnc_async_encoding_end'. In 'zrle_compress_data' it calls 'deflateInit2' to allocate the libz library opaque data. The 'VncState' used in 'zrle_compress_data' is the local 'VncState'. In 'vnc_zrle_clear' it calls 'deflateEnd' to free the libz library opaque data. The 'VncState' used in 'vnc_zrle_clear' is the connection 'VncState'. In currently implementation there will be a memory leak when the vnc disconnect. Following is the asan output backtrack: Direct leak of 29760 byte(s) in 5 object(s) allocated from: 0 0xffffa67ef3c3 in __interceptor_calloc (/lib64/libasan.so.4+0xd33c3) 1 0xffffa65071cb in g_malloc0 (/lib64/libglib-2.0.so.0+0x571cb) 2 0xffffa5e968f7 in deflateInit2_ (/lib64/libz.so.1+0x78f7) 3 0xaaaacec58613 in zrle_compress_data ui/vnc-enc-zrle.c:87 4 0xaaaacec58613 in zrle_send_framebuffer_update ui/vnc-enc-zrle.c:344 5 0xaaaacec34e77 in vnc_send_framebuffer_update ui/vnc.c:919 6 0xaaaacec5e023 in vnc_worker_thread_loop ui/vnc-jobs.c:271 7 0xaaaacec5e5e7 in vnc_worker_thread ui/vnc-jobs.c:340 8 0xaaaacee4d3c3 in qemu_thread_start util/qemu-thread-posix.c:502 9 0xffffa544e8bb in start_thread (/lib64/libpthread.so.0+0x78bb) 10 0xffffa53965cb in thread_start (/lib64/libc.so.6+0xd55cb) This is because the opaque allocated in 'deflateInit2' is not freed in 'deflateEnd'. The reason is that the 'deflateEnd' calls 'deflateStateCheck' and in the latter will check whether 's->strm != strm'(libz's data structure). This check will be true so in 'deflateEnd' it just return 'Z_STREAM_ERROR' and not free the data allocated in 'deflateInit2'. The reason this happens is that the 'VncState' contains the whole 'VncZrle', so when calling 'deflateInit2', the 's->strm' will be the local address. So 's->strm != strm' will be true. To fix this issue, we need to make 'zrle' of 'VncState' to be a pointer. Then the connection 'VncState' and local 'VncState' exchange mechanism will work as expection. The 'tight' of 'VncState' has the same issue, let's also turn it to a pointer. Reported-by: Ying Fang <fangying1@huawei.com> Signed-off-by: Li Qiang <liq3ea@163.com> Message-id: 20190831153922.121308-1-liq3ea@163.com Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2019-06-12Include qemu-common.h exactly where neededMarkus Armbruster
No header includes qemu-common.h after this commit, as prescribed by qemu-common.h's file comment. Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-Id: <20190523143508.25387-5-armbru@redhat.com> [Rebased with conflicts resolved automatically, except for include/hw/arm/xlnx-zynqmp.h hw/arm/nrf51_soc.c hw/arm/msf2-soc.c block/qcow2-refcount.c block/qcow2-cluster.c block/qcow2-cache.c target/arm/cpu.h target/lm32/cpu.h target/m68k/cpu.h target/mips/cpu.h target/moxie/cpu.h target/nios2/cpu.h target/openrisc/cpu.h target/riscv/cpu.h target/tilegx/cpu.h target/tricore/cpu.h target/unicore32/cpu.h target/xtensa/cpu.h; bsd-user/main.c and net/tap-bsd.c fixed up]
2019-03-18vnc: fix unalignment access in tight_pack24Li Qiang
When adding '-fsanitize=undefined' in compiling configuration and connect VM with vnc, it reports following error: ui/vnc-enc-tight.c:910:13: runtime error: load of misaligned address 0x621000466513 for type 'uint32_t', which requires 4 byte alignment This patch fix this issue. Signed-off-by: Li Qiang <liq3ea@163.com> Message-id: 20190318010442.14897-1-liq3ea@163.com Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2018-08-24ui/vnc: Remove useless parenthesis around DIV_ROUND_UP macroPhilippe Mathieu-Daudé
Patch created mechanically by rerunning: $ spatch --sp-file scripts/coccinelle/round.cocci \ --macro-file scripts/cocci-macro-file.h \ --dir . --in-place Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> Message-Id: <20180704153919.12432-7-f4bug@amsat.org> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2017-08-31vnc: use DIV_ROUND_UPMarc-André Lureau
I used the clang-tidy qemu-round check to generate the fix: https://github.com/elmarco/clang-tools-extra Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com> Reviewed-by: Richard Henderson <rth@twiddle.net>
2017-06-20qapi: merge QInt and QFloat in QNumMarc-André Lureau
We would like to use a same QObject type to represent numbers, whether they are int, uint, or floats. Getters will allow some compatibility between the various types if the number fits other representations. Add a few more tests while at it. Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com> Message-Id: <20170607163635.17635-7-marcandre.lureau@redhat.com> Reviewed-by: Markus Armbruster <armbru@redhat.com> [parse_stats_intervals() simplified a bit, comment in test_visitor_in_int_overflow() tidied up, suppress bogus warnings] Signed-off-by: Markus Armbruster <armbru@redhat.com>
2016-09-28ui/vnc-enc-tight: remove switch and have single returnAlex Bennée
When enabling the sanitizer build it will complain about control reaching a non-void function. Normally the compiler should detect that there is only one possible exit given a static VNC_SERVER_FB_BYTES. As we always expect a static VNC_SERVER_FB_BYTES I've added a compile time assert and just called the sub-function directly. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2016-07-15vnc-tight: fix regression with libxenstorePeter Lieven
commit 095497ff added thread local storage for the color counting palette. Unfortunately, a VncPalette is about 7kB on a x86_64 system. This memory is reserved from the stack of every thread and it exhausted the stack space of a libxenstore thread. Fix this by allocating memory only for the VNC encoding thread. Fixes: 095497ffc66b7f031ff2a17f1e50f5cb105ce588 Reported-by: Juergen Gross <jgross@suse.com> Tested-by: Juergen Gross <jgross@suse.com> Signed-off-by: Peter Lieven <pl@kamp.de> Message-id: 1468575911-20656-1-git-send-email-pl@kamp.de Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2016-07-15vnc-enc-tight: fix off-by-one bugHerongguang (Stephen)
In tight_encode_indexed_rect32, buf(or src)’s size is count. In for loop, the logic is supposed to be that i is an index into src, i should be incremented when incrementing src. This is broken when src is incremented but i is not before while loop, resulting in off-by-one bug in while loop. Signed-off-by: He Rongguang <herongguang.he@huawei.com> Message-id: 5784B8EB.7010008@huawei.com Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2016-07-12vnc-enc-tight: use thread local storage for palettePeter Lieven
currently the color counting palette is allocated from heap, used and destroyed for each single subrect. Use a static palette per thread for this purpose and avoid the malloc and free for each update. Signed-off-by: Peter Lieven <pl@kamp.de> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com> Message-id: 1467280846-9674-1-git-send-email-pl@kamp.de Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2016-02-04ui: Clean up includesPeter Maydell
Clean up includes so that osdep.h is included first and headers which it implies are not included manually. This commit was created with scripts/clean-includes. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Message-id: 1454089805-5470-2-git-send-email-peter.maydell@linaro.org
2014-12-10vnc-enc-tight: fix Arguments in wrong orderGonglei
Arguments in wrong order (SWAPPED_ARGUMENTS) The positions of arguments in the call to tight_fill_palette do not match the ordering of the parameters: &fg is passed to bg &bg is passed to fg Cc: Gerd Hoffmann <kraxel@redhat.com> Signed-off-by: Gonglei <arei.gonglei@huawei.com> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2014-09-05console: Remove unused QEMU_BIG_ENDIAN_FLAGBenjamin Herrenschmidt
If we need to, we should use the pixman formats instead but for now this is unused except in commented out code so take it out to avoid further confusion about surface endianness. Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2014-06-02vnc-enc-tight: Fix divide-by-zero in tight_detect_smooth_image{16,24,32}Gonglei
Spotted by Coverity: (1) Event assignment: Assigning: "pixels" = "0". (2) Event cond_true: Condition "y < h", taking true branch (3) Event cond_false: Condition "x < w", taking false branch (4) Event loop_end: Reached end of loop (5) Event divide_by_zero: In expression "(stats[0] + stats[1]) * 100U / pixels", division by expression "pixels" which may be zero has undefined behavior. 290 DEFINE_DETECT_FUNCTION(16) 291 DEFINE_DETECT_FUNCTION(32) Signed-off-by: Gonglei <arei.gonglei@huawei.com> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2014-03-10vnc: Fix tight_detect_smooth_image() for lossless caseMarkus Armbruster
VncTight member uint8_t quality is either (uint8_t)-1 for lossless or less than 10 for lossy. tight_detect_smooth_image() first promotes it to int, then compares with -1. Always unequal, so we always execute the lossy code. Reads beyond tight_conf[] and returns crap when quality is actually lossless. Compare to (uint8_t)-1 instead, like we do elsewhere. Spotted by Coverity. Signed-off-by: Markus Armbruster <armbru@redhat.com> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2013-03-18vnc: stop using DisplayStateGerd Hoffmann
Rework DisplayStateListener callbacks to not use the DisplayState any more. Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2012-12-19misc: move include files to include/qemu/Paolo Bonzini
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2012-12-19qapi: move include files to include/qobject/Paolo Bonzini
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2012-12-14pixman: fix vnc tight png/jpeg supportGerd Hoffmann
This patch adds an x argument to qemu_pixman_linebuf_fill so it can also be used to convert a partial scanline. Then fix tight + png/jpeg encoding by passing in the x+y offset, so the data is read from the correct screen location instead of the upper left corner. Cc: 1087974@bugs.launchpad.net Cc: qemu-stable@nongnu.org Reported-by: Tim Hardeneck <thardeck@suse.de> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com> Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
2012-11-01pixman/vnc: remove dead code.Gerd Hoffmann
Switching the vnc server framebuffer to use 32bpp unconditionally turns the code bits which handle 8 and 16 bpp into dead code. Remove them. Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2012-11-01pixman/vnc: remove rgb_prepare_row* functionsGerd Hoffmann
Let pixman do it instead. Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2012-11-01pixman/vnc: use pixman images in vnc.Gerd Hoffmann
The vnc code uses *three* DisplaySurfaces: First is the surface of the actual QemuConsole, usually the guest screen, but could also be a text console (monitor/serial reachable via Ctrl-Alt-<nr> keys). This is left as-is. Second is the current server's view of the screen content. The vnc code uses this to figure which parts of the guest screen did _really_ change to reduce the amount of updates sent to the vnc clients. It is also used as data source when sending out the updates to the clients. This surface gets replaced by a pixman image. The format changes too, instead of using the guest screen format we'll use fixed 32bit rgb framebuffer and convert the pixels on the fly when comparing and updating the server framebuffer. Third surface carries the format expected by the vnc client. That isn't used to store image data. This surface is switched to PixelFormat and a boolean for bigendian byte order. Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
2011-08-20Use glib memory allocation and free functionsAnthony Liguori
qemu_malloc/qemu_free no longer exist after this commit. Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2011-06-27vnc: Fix compilation with --enable-vnc-pngStefan Weil
Commit f26e428da505709ec03b2ed2c9eb3db82b30bd7b fixed compilation with --enable-vnc-png, but broke it with --enable-vnc-png. The breakage is caused by pngconfig.h which checks whether setjmp.h was already included and fails because qemu-common.h includes setjmp.h. The check is disabled by defining PNG_SKIP_SETJMP_CHECK. Cc: Blue Swirl <blauwirbel@gmail.com> Signed-off-by: Stefan Weil <weil@mail.berlios.de> Reviewed-by: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2011-06-26Fix MinGW compilation when --enable-vnc-jpeg is specifiedRoy Tam
Fix conflicting types for 'INT32' in basetsd.h and jmorecfg.h by including qemu-common.h first. Signed-off-by: Roy Tam <roytam@gmail.com> Acked-by: Stefan Weil <weil@mail.berlios.de> Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
2011-04-10vnc: tight: Fix crash after 2GB of outputMichael Tokarev
fix 2Gb integer overflow in in VNC tight and zlib encodings As found by Roland Dreier <roland@purestorage.com> (excellent catch!), when amount of VNC compressed data produced by zlib and sent to client exceeds 2Gb, integer overflow occurs because currently, we calculate amount of data produced at each step by comparing saved total_out with new total_out, and total_out is something which grows without bounds. Compare it with previous avail_out instead of total_out, and leave total_out alone. The same code is used in vnc-enc-tight.c and vnc-enc-zlib.c, so fix both cases. There, there's no actual need to save previous_out value, since capacity-offset (which is how that value is calculated) stays the same so it can be recalculated again after call to deflate(), but whole thing becomes less readable this way. Reported-by: Roland Dreier <roland@purestorage.com> Signed-off-by: Michael Tokarev <mjt@tls.msk.ru> Signed-off-by: Corentin Chary <corentin.chary@gmail.com> Signed-off-by: Aurelien Jarno <aurelien@aurel32.net>
2011-02-24ui/vnc-enc-tight.c: Fix compile failure if CONFIG_VNC_JPEG not definedPeter Maydell
Add some missing #ifdefs to fix compilation failures in the !CONFIG_VNC_JPEG case introduced by commit ce702e93. Signed-off-by: Peter Maydell <peter.maydell@linaro.org> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2011-02-23vnc: add a non-adaptive optionCorentin Chary
This option allow to disable adaptive behaviors in some encodings. Signed-off-by: Corentin Chary <corentincj@iksaif.net> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2011-02-23vnc: tight: tweak adaptive tight settingsCorentin Chary
The force_jpeg threshold was too low. Signed-off-by: Corentin Chary <corentincj@iksaif.net> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2011-02-23vnc: fix uint8_t comparisons with negative valuesCorentin Chary
Signed-off-by: Corentin Chary <corentincj@iksaif.net> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2011-02-23vnc: tight: use the update frequency to choose between lossy and losslessCorentin Chary
Use the new update frequency infrastructure to use jpeg for regions with high update frequency. Signed-off-by: Corentin Chary <corentincj@iksaif.net> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2010-10-05Fix spelling in commentsStefan Weil
multifuction -> multifunction successfull -> successful. Signed-off-by: Stefan Weil <weil@mail.berlios.de>
2010-09-04vnc: use bswapNN() rather than bswap_NN()Izumi Tsutsui
bswap_NN() variants are not always available in CONFIG_MACHINE_BSWAP_H case and bswapNN() are public APIs in "bswap.h". Signed-off-by: Izumi Tsutsui <tsutsui@ceres.dti.ne.jp> Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
2010-09-04vnc: tight: remove unused variableSerge Ziryukin
Signed-off-by: Serge Ziryukin <ftrvxmtrx@gmail.com> Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
2010-09-04size_t is unsigned, change to ssize_t to handle errors from ↵Jes Sorensen
tight_compress_data() Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com> Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
2010-07-31Initialize a variable in all casesBlue Swirl
Commit d167f9bc06a577d6c85b8ed6991c1efe175aae7d missed this one: /src/qemu/ui/vnc-enc-tight.c:1483: warning: 'ret' may be used uninitialized in this function Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
2010-07-31Fix uint8_t comparison with negative valueBlue Swirl
Commit 7bccf57383cca60a778d5c543ac80c9f62d89ef2 missed this one: /src/qemu/ui/vnc-enc-tight.c: In function 'send_sub_rect': /src/qemu/ui/vnc-enc-tight.c:1527: warning: comparison is always true due to limited range of data type Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
2010-07-27Fix mingw32 buildBlue Swirl
Fix mingw32 build errors like /src/qemu/ui/vnc-enc-tight.c: In function 'tight_detect_smooth_image24': /src/qemu/ui/vnc-enc-tight.c:119: error: 'uint' undeclared (first use in this function) Replace 'uint' with proper 'unsigned int'. Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
2010-07-27Initialize a variable in all casesBlue Swirl
Fix a warning with some GCCs: /src/qemu/ui/vnc-enc-tight.c: In function `send_sub_rect_nojpeg': /src/qemu/ui/vnc-enc-tight.c:1458: warning: `ret' might be used uninitialized in this function Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
2010-07-27Fix uint8_t comparisons with negative valuesBlue Swirl
Fix the following warnings: /src/qemu/hw/ide/core.c: In function `ide_drive_pio_post_load': /src/qemu/hw/ide/core.c:2767: warning: comparison is always false due to limited range of data type /src/qemu/ui/vnc-enc-tight.c: In function `tight_detect_smooth_image': /src/qemu/ui/vnc-enc-tight.c:284: warning: comparison is always true due to limited range of data type /src/qemu/ui/vnc-enc-tight.c:297: warning: comparison is always true due to limited range of data type /src/qemu/ui/vnc-enc-tight.c: In function `tight_encode_indexed_rect16': /src/qemu/ui/vnc-enc-tight.c:456: warning: comparison is always false due to limited range of data type /src/qemu/ui/vnc-enc-tight.c: In function `tight_encode_indexed_rect32': /src/qemu/ui/vnc-enc-tight.c:457: warning: comparison is always false due to limited range of data type Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
2010-07-26vnc: tight: split send_sub_rectCorentin Chary
Split send_sub_rect in send_sub_rect_jpeg and send_sub_rect_nojpeg to remove all these #ifdef CONFIG_JPEG. Signed-off-by: Corentin Chary <corentincj@iksaif.net> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2010-07-26vnc: tight: fix rgb_prepare_rowCorentin Chary
rgb_prepare_row bpp depends on the server display surface, not the client. Signed-off-by: Corentin Chary <corentincj@iksaif.net> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2010-07-26vnc: fix tight png memory leakCorentin Chary
The tight.png buffer was never released. Signed-off-by: Corentin Chary <corentincj@iksaif.net> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2010-07-26vnc: encapsulate encoding membersCorentin Chary
This will allow to implement the threaded VNC server in a more cleaner way. Signed-off-by: Corentin Chary <corentincj@iksaif.net> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2010-07-26vnc: tight: stop using qdict for palette stuffCorentin Chary
Profiling with callgrind seems to show that a lot of time is spent in the palette code (mostly due to memory allocation and qdict to int conversion). This patch adds a VncPalette implementation. The palette is stored in a hash table, like qdict, but which does way less memory allocations, and doesn't suffer from the QObject overhead. Signed-off-by: Corentin Chary <corentincj@iksaif.net> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2010-07-26vnc: tight: specific zlib level and filters for each compression levelCorentin Chary
Disable png filters for lower compression levels. This should lower the CPU consumption and reduce encoding time. This isn't in tight_conf because: * tight_conf structure must not change, because it's shared with other tight implementations (libvncserver, etc..). * it'd exceed the 80 col limit. * PNG_ macros are only defined if CONFIG_VNC_PNG is defined Signed-off-by: Corentin Chary <corentincj@iksaif.net> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2010-07-26vnc: tight add PNG encodingCorentin Chary
Introduce a new encoding: VNC_ENCODING_TIGHT_PNG [1] (-269) with a new tight filter VNC_TIGHT_PNG (0x0A). When the client tells it supports the Tight PNG encoding, the server will use tight, but will always send encoding pixels using PNG instead of zlib. If the client also told it support JPEG, then the server can send JPEG, because PNG will only be used in the cases zlib was used in normal tight. This encoding was introduced to speed up HTML5 based VNC clients like noVNC [2], but can also be used on devices like iPhone where PNG can be rendered in hardware. [1] http://wiki.qemu.org/VNC_Tight_PNG [2] http://github.com/kanaka/noVNC/ Signed-off-by: Corentin Chary <corentincj@iksaif.net> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2010-07-26vnc: tight: remove a memleak in send_jpeg_rect()Corentin Chary
buf was never freed. Signed-off-by: Corentin Chary <corentincj@iksaif.net> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2010-07-26vnc: tight: don't forget do at the last colorCorentin Chary
While using indexed colors, the last color was never added to the palette. Triggered with ubuntu livecd. Signed-off-by: Corentin Chary <corentincj@iksaif.net> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
2010-07-26vnc: rename vnc-encoding-* vnc-enc-*Corentin Chary
For the same reason that we don't use vnc-authentication-sasl.c but vnc-auth-sals.c. Because it's tooooo long. Signed-off-by: Corentin Chary <corentincj@iksaif.net> Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-15 18:31:25 +0000