PixelPaint: Fix BucketTool out of memory crashes

The BFS implementation for BucketTool's flood-fill had sitations
which could result in infinite loop, causing OOM crashes due to
the queue growing unbounded. The way to fix this is to keep track
of the pixels we have already visited in the flood-fill algorithm
and ignore those if we ever encounter them again.

This also fixes the crashing issue from #9003. We still need a
better way to account for transparency, but that is beyond the scope
of this commit, and this issue still exists without any transparent
pixels.

1 files changed, 5 insertions, 0 deletions

diff --git a/Userland/Applications/PixelPaint/BucketTool.cpp b/Userland/Applications/PixelPaint/BucketTool.cpp
index 4161d61204..cf255603fb 100644
--- a/Userland/Applications/PixelPaint/BucketTool.cpp
+++ b/Userland/Applications/PixelPaint/BucketTool.cpp
@@ -7,6 +7,7 @@
 #include "BucketTool.h"
 #include "ImageEditor.h"
 #include "Layer.h"
+#include <AK/HashTable.h>
 #include <AK/Queue.h>
 #include <LibGUI/BoxLayout.h>
 #include <LibGUI/Label.h>
@@ -47,8 +48,12 @@ static void flood_fill(Gfx::Bitmap& bitmap, Gfx::IntPoint const& start_position,
 
     Queue<Gfx::IntPoint> queue;
     queue.enqueue(start_position);
+    HashTable<Gfx::IntPoint> visited;
     while (!queue.is_empty()) {
         auto position = queue.dequeue();
+        if (visited.contains(position))
+            continue;
+        visited.set(position);
 
         auto pixel_color = bitmap.get_pixel<Gfx::StorageFormat::BGRA8888>(position.x(), position.y());
         if (color_distance_squared(pixel_color, target_color) > threshold_normalized_squared)