LibGfx: Fix a slight mistake in AA ellipse error calculation

The initial signs were wrong for the deltas of f(x), the ellipse
equation. This seemed to be fine for larger circles/ellipses but
broke things at a small scale, this was previously fixed with a
horrible "error = error / 4" hack. With this change, all ellipses
look a little better :^)

This also fixed a signed integer overflow Andreas found with UBSAN,
which happened for circles with a 1px radius.

1 files changed, 2 insertions, 7 deletions

diff --git a/Userland/Libraries/LibGfx/AntiAliasingPainter.cpp b/Userland/Libraries/LibGfx/AntiAliasingPainter.cpp
index 9f3d6f65b8..a4b922f50d 100644
--- a/Userland/Libraries/LibGfx/AntiAliasingPainter.cpp
+++ b/Userland/Libraries/LibGfx/AntiAliasingPainter.cpp
@@ -443,8 +443,8 @@ FLATTEN AntiAliasingPainter::Range AntiAliasingPainter::draw_ellipse_part(
     int f_squared = y * y;
 
     // 1st and 2nd order differences of f(i)*f(i)
-    int delta_f_squared = -(static_cast<int64_t>(b_squared) * subpixel_resolution * subpixel_resolution) / a_squared;
-    int delta2_f_squared = 2 * delta_f_squared;
+    int delta_f_squared = (static_cast<int64_t>(b_squared) * subpixel_resolution * subpixel_resolution) / a_squared;
+    int delta2_f_squared = -delta_f_squared - delta_f_squared;
 
     // edge_intersection_area/subpixel_resolution = percentage of pixel intersected by circle
     // (aka the alpha for the pixel)
@@ -487,11 +487,6 @@ FLATTEN AntiAliasingPainter::Range AntiAliasingPainter::draw_ellipse_part(
     auto correct = [&] {
         int error = y - y_hat;
 
-        // FIXME: The alpha values seem too low, which makes things look
-        // overly pointy. This fixes that, though there's probably a better
-        // solution to be found. (This issue seems to exist in the base algorithm)
-        error /= 4;
-
         delta2_y += error;
         delta_y += error;
     };