Building Resize Handles That Actually Work: Lessons from a Game Engine Editor

Adding resize capabilities to the ColumbaEngine.

by PigeonCodeur
devc++columba engineUI/UX

Building Resize Handles That Actually Work: Lessons from a Game Engine Editor

You know that moment when you’re using a design tool and you grab the corner of an element, expecting to resize it, but instead you accidentally move the whole thing? Or worse—nothing happens at all.

I just spent the afternoon implementing proper resize handles for a game engine editor, and it reminded me why good editor UX is so much harder than it looks. Here’s what I learned building a system that prioritizes handle clicks over dragging, integrates with undo/redo, and doesn’t break when you have complex UI hierarchies.

The Problem with “Simple” Resize Systems

Most resize implementations I’ve seen follow this pattern:

  1. Draw some handles on a selection outline
  2. Check if mouse clicks hit the handles
  3. Resize on drag

Sounds straightforward. But the devil lives in the interaction priority. When you click near a resize handle, what should happen? In many editors, there’s an annoying ambiguity—sometimes you resize, sometimes you select a different element, sometimes you start dragging.

The key insight: Handle clicks need absolute priority in the hit detection hierarchy.

Building Priority-Based Click Detection

Here’s how I structured the click detection in the EntityFinder system:

virtual void onEvent(const OnMouseClick& event) override
{
    bool hit = false;

    // Priority 1: Check resize handles first
    for (const auto& elem : viewGroup<PositionComponent, ResizeHandleComponent>())
    {
        if (inClipBound(elem->entity, event.pos.x, event.pos.y))
        {
            hit = true;
            auto selectedId = selectionOutline.get<SelectedEntity>()->id;
            if (selectedId != 0)
            {
                ecsRef->sendEvent(StartResize{ selectedId, 
                    elem->get<ResizeHandleComponent>()->handle, 
                    event.pos.x, event.pos.y });
            }
            break;
        }
    }

    // Priority 2: Only check entities if no handle was clicked
    if (not hit)
    {
        // ... existing entity selection logic
    }
}

This priority system eliminates the frustrating “did I click the handle or the element?” problem. Handles always win.

The Eight-Handle Layout

I went with the classic eight-handle approach: four corners plus four edge midpoints. Each handle type maps to specific resize behavior:

enum class ResizeHandle : uint8_t
{
    None = 0,
    TopLeft, Top, TopRight,
    Left, Right,
    BottomLeft, Bottom, BottomRight
};

The tricky part is positioning these handles correctly. They need to:

  • Anchor to the selection outline’s boundaries
  • Stay visible at different zoom levels
  • Have a consistent hit target size (8px)

I solved this with a helper function in the outline creation:

auto createResizeHandle = [&](ResizeHandle handleType, AnchorType anchorX, AnchorType anchorY) {
    auto handle = makeUiSimple2DShape(ecs, Shape2D::Square, handleSize, handleSize, handleColor);
    
    auto handleAnchor = handle.get<UiAnchor>();
    auto handleResizeComp = ecs->attach<ResizeHandleComponent>(handle.entity);
    handleResizeComp->handle = handleType;
    
    // Position based on anchor types
    if (anchorX == AnchorType::Left)
        handleAnchor->setLeftAnchor({outline.entity.id, AnchorType::Left, -handleSize/2});
    else if (anchorX == AnchorType::Right)
        handleAnchor->setRightAnchor({outline.entity.id, AnchorType::Right, -handleSize/2});
    // ... etc for all positions
};

The offset of -handleSize/2 centers the handle on the outline edge—a small detail that makes the interaction feel precise.

Resize Logic That Doesn’t Fight You

The actual resize math varies by handle type. Corner handles modify both position and size, while edge handles only change one dimension:

switch (activeHandle)
{
    case ResizeHandle::TopLeft:
        newX = resizeStartX + deltaX;
        newY = resizeStartY + deltaY;
        newWidth = resizeStartWidth - deltaX;
        newHeight = resizeStartHeight - deltaY;
        break;
    case ResizeHandle::Right:
        newWidth = resizeStartWidth + deltaX;
        break;
    // ... other cases
}

Notice that when resizing from the top-left, the position changes inverse to the delta. This keeps the bottom-right corner fixed while the top-left moves—exactly what users expect.

I added a minimum size constraint to prevent elements from disappearing:

constexpr float minSize = 10.0f;
if (newWidth < minSize || newHeight < minSize)
    return;

Undo/Redo Integration

The resize system needed to play nicely with the existing command pattern. I created a ResizeCommand that captures the complete before/after state:

struct ResizeCommand : public InspectorCommands
{
    _unique_id entityId;
    ResizeHandle handle;
    float startWidth, startHeight, startX, startY;
    float endWidth, endHeight, endX, endY;
    
    virtual void execute() override {
        auto pos = ecsRef->getComponent<PositionComponent>(entityId);
        pos->setX(endX);
        pos->setY(endY);
        pos->setWidth(endWidth);
        pos->setHeight(endHeight);
    }
    
    virtual void undo() override {
        auto pos = ecsRef->getComponent<PositionComponent>(entityId);
        pos->setX(startX);
        pos->setY(startY);
        pos->setWidth(startWidth);
        pos->setHeight(startHeight);
    }
};

The command gets created when the resize operation ends, capturing the full transformation in one atomic operation.

Real-Time Visual Feedback

During resize, both the element and its selection outline update in real-time. This required careful coordination between the resize logic and the outline positioning:

// Update the element
pos->setX(newX);
pos->setY(newY);
pos->setWidth(newWidth);
pos->setHeight(newHeight);

// Update selection outline to match
auto ent = ecsRef->getEntity("SelectionOutline");
if (ent && ent->get<SelectedEntity>()->id == resizingEntity)
{
    auto outlinePos = ent->get<PositionComponent>();
    outlinePos->setX(newX - 2.f);
    outlinePos->setY(newY - 2.f);
    outlinePos->setWidth(newWidth + 4.f);
    outlinePos->setHeight(newHeight + 4.f);
}

The outline stays 2 pixels larger than the element on all sides, maintaining the visual relationship throughout the resize operation.

What I’d Do Differently

If I were starting fresh, I’d consider:

Constraint system integration

The current implementation checks for anchor constraints but could be smarter about respecting layout relationships during resize.

Aspect ratio locking

Adding Shift+drag to maintain proportions would make the tool more flexible.

Visual cursor feedback

Changing the cursor to resize arrows when hovering over handles would improve discoverability.

The Bigger Picture

Building resize handles taught me that good editor UX comes down to eliminating ambiguity. Users should never wonder “what will happen if I click here?” The system should handle the complexity so they can focus on their creative work.

The technical implementation—events, commands, hit detection—is just scaffolding. The real challenge is making interactions feel natural and predictable. When someone grabs a corner handle, they expect to resize from that corner. When they press Ctrl+Z, they expect the resize to undo completely.

Meeting those expectations requires thinking beyond the happy path to all the edge cases and failure modes. But when it works, the editor just disappears and lets users create.

Want to see the full implementation? The complete code is available in the ColumbaEngine repository with all the resize handle logic, priority-based detection, and undo/redo integration.