狸村 Mystic Web · 狸村
目录

GUI: Introduction

Introduction

Welcome to the "GUI" chapter of our "Building a Simple Engine" series! After implementing a camera system in the previous chapter, we’ll now focus on adding a graphical user interface (GUI) to our Vulkan application. A well-designed GUI is essential for creating interactive applications that allow users to control settings, display information, and interact with the 3D scene.

In this chapter, we’ll integrate a popular immediate-mode GUI library called Dear ImGui with our Vulkan engine. Dear ImGui is widely used in the game and graphics industry due to its simplicity, performance, and flexibility. It allows developers to quickly create debug interfaces, tools, and in-game menus without the complexity of traditional retained-mode GUI systems.

This chapter will guide you through integrating a professional GUI system into your Vulkan engine. We’ll start by setting up Dear ImGui with Vulkan, establishing the foundation for all GUI functionality. The integration requires careful management of Vulkan resources—we’ll create dedicated buffers, textures, and pipelines that work alongside your existing rendering systems without interference.

User input handling becomes more complex when you need to support both 3D scene navigation and GUI interaction. We’ll implement a system that can distinguish between input intended for the 3D world and input meant for interface elements, ensuring smooth interaction with both.

Rather than overwhelming you with exhaustive widget examples, we’ll focus on the key integration concepts that enable GUI functionality. Understanding these principles will let you implement any interface elements your project needs.

The rendering integration presents interesting challenges—your GUI needs to render on top of your 3D scene without disrupting the existing pipeline. We’ll solve this by carefully managing render passes and ensuring proper depth testing and blending.

Finally, we’ll implement object picking, which bridges the gap between your GUI and 3D scene. This feature allows users to click on 3D objects and see their properties in the interface, creating a cohesive development environment.

By the end of this chapter, you’ll have a functional GUI system that you can use to control your camera, adjust rendering settings, and interact with your 3D scene. This will serve as a foundation for more advanced features in later chapters, such as material editors, scene hierarchies, and debugging tools.

Prerequisites

This chapter builds directly on the Camera & Transformations chapter, as we’ll extend the camera system we developed there to work seamlessly with GUI interaction. The camera controls need to be aware of when the user is interacting with interface elements versus navigating the 3D scene.

You’ll also need a solid understanding of several core Vulkan concepts. The rendering pipeline and command buffer knowledge is crucial because GUI rendering requires careful coordination with your existing 3D rendering—we’ll be recording GUI draw calls into the same command buffers while managing different pipeline states.

Buffer and image creation skills are essential since Dear ImGui requires dedicated vertex and index buffers for its geometry, plus texture resources for fonts and any custom UI textures. Understanding descriptor sets and layouts becomes important as we’ll need to create descriptors specifically for GUI rendering that don’t interfere with your 3D scene descriptors.

Pipeline creation knowledge ties everything together, as we’ll build a specialized graphics pipeline for GUI rendering with different vertex input, shaders, and render state than your 3D pipeline.

A basic understanding of input handling concepts will help you follow along as we implement the dual-mode input system that can distinguish between 3D navigation and GUI interaction.

You should also be familiar with the following chapters from the main tutorial:

Let’s begin by exploring how to implement a professional GUI system with Dear ImGui and Vulkan.

GUI: Setting Up Dear ImGui

Setting Up Dear ImGui

In this section, we’ll set up Dear ImGui in our Vulkan application. Dear ImGui (also known simply as ImGui) is a bloat-free graphical user interface library for C++. It outputs optimized vertex buffers that you can render with your 3D-pipeline-enabled application. It’s particularly well-suited for integration with graphics APIs like Vulkan.

Adding ImGui to Your Project

First, we need to add ImGui to our project. There are several ways to do this:

  1. Git Submodule: Add ImGui as a Git submodule to your project

  2. Package Manager: Use a package manager like vcpkg or Conan

  3. Manual Integration: Download and include the ImGui source files directly

For this tutorial, we’ll use the manual integration approach for simplicity:

# Clone ImGui repository
git clone https://github.com/ocornut/imgui.git external/imgui

# Copy necessary files to your project
cp external/imgui/imgui.h include/
cp external/imgui/imgui.cpp src/
cp external/imgui/imgui_draw.cpp src/
cp external/imgui/imgui_widgets.cpp src/
cp external/imgui/imgui_tables.cpp src/
cp external/imgui/imgui_demo.cpp src/

Next, update your CMakeLists.txt to include these files:

# ImGui files
set(IMGUI_SOURCES
    src/imgui.cpp
    src/imgui_draw.cpp
    src/imgui_widgets.cpp
    src/imgui_tables.cpp
    src/imgui_demo.cpp
)

# Our custom ImGui Vulkan integration
set(IMGUI_VULKAN_SOURCES
    src/imgui_vulkan_util.cpp
)

add_executable(VulkanApp
    src/main.cpp
    ${IMGUI_SOURCES}
    ${IMGUI_VULKAN_SOURCES}
)

target_include_directories(VulkanApp PRIVATE include)

Creating an ImGui Integration

Let’s implement the ImGuiVulkanUtil class to handle the integration between ImGui and Vulkan.

The ImGuiVulkanUtil class serves as the bridge between ImGui’s immediate-mode GUI system and Vulkan’s explicit graphics API. This integration requires careful management of GPU resources, synchronization, and rendering state to efficiently display user interface elements alongside our 3D graphics. Let’s break down the class architecture into logical components to understand how each part contributes to the overall integration.

ImGuiVulkanUtil Architecture: GPU Resource Management Foundation

First, we establish the core Vulkan resources needed to render ImGui’s dynamically generated UI geometry on the GPU.

// ImGuiVulkanUtil.h
#pragma once

#include <vulkan/vulkan_raii.hpp>
#include <imgui.h>

class ImGuiVulkanUtil {
private:
    // Core GPU rendering resources for UI display
    // These objects form the foundation of our ImGui-to-Vulkan rendering pipeline
    vk::raii::Sampler sampler{nullptr};                    // Texture sampling configuration for font rendering
    Buffer vertexBuffer;                                    // Dynamic vertex buffer for UI geometry
    Buffer indexBuffer;                                     // Dynamic index buffer for UI triangle connectivity
    uint32_t vertexCount = 0;                              // Current vertex count for draw commands
    uint32_t indexCount = 0;                               // Current index count for draw commands
    Image fontImage;                                        // GPU texture containing ImGui font atlas
    ImageView fontImageView;                                // Shader-accessible view of font texture

The GPU resource foundation reflects ImGui’s dynamic rendering model, where UI geometry is generated fresh each frame based on the current interface layout. The vertex and index buffers use host-visible memory to enable efficient CPU updates, while the font texture remains static once loaded. This hybrid approach balances the need for dynamic UI updates with the performance benefits of GPU-resident font data.

The buffer sizing strategy must accommodate ImGui’s variable geometry output, which can change dramatically based on UI complexity. Unlike static 3D models, ImGui generates different amounts of geometry each frame, requiring our buffers to resize dynamically or be pre-allocated with sufficient capacity for worst-case scenarios.

ImGuiVulkanUtil Architecture: Vulkan Pipeline Infrastructure

Next, we set up the Vulkan pipeline objects that define how UI geometry is processed and rendered by the GPU.

    // Vulkan pipeline infrastructure for UI rendering
    // These objects define the complete GPU processing pipeline for ImGui elements
    vk::raii::PipelineCache pipelineCache{nullptr};        // Pipeline compilation cache for faster startup
    vk::raii::PipelineLayout pipelineLayout{nullptr};      // Resource binding layout (textures, uniforms)
    vk::raii::Pipeline pipeline{nullptr};                  // Complete graphics pipeline for UI rendering
    vk::raii::DescriptorPool descriptorPool{nullptr};      // Pool for allocating descriptor sets
    vk::raii::DescriptorSetLayout descriptorSetLayout{nullptr}; // Layout defining shader resource bindings
    vk::raii::DescriptorSet descriptorSet{nullptr};        // Actual resource bindings for font texture

The pipeline infrastructure creates a specialized graphics pipeline optimized for UI rendering, which differs significantly from typical 3D rendering pipelines. UI rendering typically requires alpha blending for transparency effects, operates in 2D screen space rather than 3D world space, and uses simpler shading models focused on texture sampling rather than complex lighting calculations.

Note

Frames-in-flight safety: If your renderer uses more than one frame in flight and you do not stall the GPU between frames, you must duplicate the dynamic ImGui buffers (vertex/index) per frame-in-flight. Using a single shared vertex/index buffer risks the CPU overwriting data still in use by the GPU from a previous frame. The simple single-buffer members shown above are for conceptual clarity; in production, store vectors of buffers/memories sized to the max frames in flight and update/bind the buffers for the current frame index.

The descriptor system manages the connection between our CPU-side resources and the GPU shaders. For UI rendering, this primarily involves binding the font atlas texture to the fragment shader, though more complex UI systems might include additional textures for icons, backgrounds, or other visual elements.

ImGuiVulkanUtil Architecture: Device Context and System Integration

Then, we maintain references to the Vulkan device context and manage integration with the broader graphics system.

    // Vulkan device context and system integration
    // These references connect our UI system to the broader Vulkan application context
    vk::raii::Device* device = nullptr;                    // Primary Vulkan device for resource creation
    vk::raii::PhysicalDevice* physicalDevice = nullptr;    // GPU hardware info for capability queries
    vk::raii::Queue* graphicsQueue = nullptr;              // Command submission queue for UI rendering
    uint32_t graphicsQueueFamily = 0;                      // Queue family index for validation

The device context integration demonstrates the explicit nature of Vulkan’s resource management, where every operation requires specific device and queue references. Unlike higher-level graphics APIs that maintain global state, Vulkan requires explicit specification of which GPU device and command queue should handle each operation.

The queue family index enables validation and optimization by ensuring that UI rendering operations use compatible queue types. While UI rendering typically uses the same graphics queue as 3D rendering, some applications might benefit from dedicated queues for different rendering responsibilities.

ImGuiVulkanUtil Architecture: UI State and Rendering Configuration

After that, we manage UI-specific state including styling, rendering parameters, and dynamic update tracking.

    // UI state management and rendering configuration
    // These members control the visual appearance and dynamic behavior of the UI system
    ImGuiStyle vulkanStyle;                                 // Custom visual styling for Vulkan applications

    // Push constants for efficient per-frame parameter updates
    // This structure enables fast updates of transformation and styling data
    struct PushConstBlock {
        glm::vec2 scale;                                    // UI scaling factors for different screen sizes
        glm::vec2 translate;                                // Translation offset for UI positioning
    } pushConstBlock;

    // Dynamic state tracking for performance optimization
    bool needsUpdateBuffers = false;                        // Flag indicating buffer resize requirements

    // Modern Vulkan rendering configuration
    vk::PipelineRenderingCreateInfo renderingInfo{};        // Dynamic rendering setup parameters
    vk::Format colorFormat = vk::Format::eB8G8R8A8Unorm;   // Target framebuffer format

The styling and configuration management reflects ImGui’s flexibility in visual presentation while maintaining compatibility with Vulkan’s explicit rendering model. The push constants provide an efficient mechanism for updating per-frame parameters like screen resolution changes or UI scaling factors without requiring descriptor set updates.

The dynamic state tracking optimizes performance by avoiding unnecessary GPU resource updates when the UI layout remains stable between frames. This optimization becomes particularly important in applications with complex UIs where buffer updates could otherwise impact frame rates.

ImGuiVulkanUtil Architecture: Public Interface and Lifecycle Management

Finally, we define the external interface that applications use to integrate ImGui rendering into their Vulkan rendering pipeline.

public:
    // Lifecycle management for proper resource initialization and cleanup
    ImGuiVulkanUtil(vk::raii::Device& device, vk::raii::PhysicalDevice& physicalDevice,
                   vk::raii::Queue& graphicsQueue, uint32_t graphicsQueueFamily);
    ~ImGuiVulkanUtil();

    // Core functionality methods for ImGui integration
    void init(float width, float height);                   // Initialize ImGui context and configure display
    void initResources();                                    // Create all Vulkan resources for rendering
    void setStyle(uint32_t index);                          // Apply visual styling themes
    void updateTexture(ImTextureData* tex);                 // Dynamically update/create textures (v1.92+)

    // Frame-by-frame rendering operations
    bool newFrame();                                         // Begin new ImGui frame and generate geometry
    void updateBuffers();                                    // Upload updated geometry to GPU buffers
    void drawFrame(vk::raii::CommandBuffer& commandBuffer); // Record rendering commands to command buffer

    // Input event handling for interactive UI elements
    void handleKey(int key, int scancode, int action, int mods); // Process keyboard input events
    void handleMousePos(float x, float y);                  // Process mouse movement events (v1.87+)
    void handleMouseButton(int button, bool pressed);       // Process mouse button events (v1.87+)
    bool getWantKeyCapture();                               // Query if ImGui wants keyboard focus
    void charPressed(uint32_t key);                         // Handle character input for text widgets
};

The public interface design balances ease of integration with performance considerations, separating one-time setup operations from per-frame rendering tasks. The initialization methods handle the expensive resource creation that should happen once during application startup, while the frame-by-frame methods focus on efficient updates and rendering.

The input handling interface enables proper integration with existing input systems, allowing ImGui to capture relevant events while passing through others to the main application. This cooperative approach ensures that UI elements can respond to user interaction without interfering with 3D scene controls or other input handling.

Implementing the ImGuiVulkanUtil Class

Now let’s implement the methods of our ImGuiVulkanUtil class for the Vulkan implementation.

Constructor and Destructor

First, let’s implement the constructor and destructor:

ImGuiVulkanUtil::ImGuiVulkanUtil(vk::raii::Device& device, vk::raii::PhysicalDevice& physicalDevice,
                               vk::raii::Queue& graphicsQueue, uint32_t graphicsQueueFamily)
    : device(&device), physicalDevice(&physicalDevice),
      graphicsQueue(&graphicsQueue), graphicsQueueFamily(graphicsQueueFamily),
      // Initialize buffers directly
      vertexBuffer(*device, 1,
                 vk::BufferUsageFlagBits::eVertexBuffer,
                 vk::MemoryPropertyFlagBits::eHostVisible | vk::MemoryPropertyFlagBits::eHostCoherent),
      indexBuffer(*device, 1,
                vk::BufferUsageFlagBits::eIndexBuffer,
                vk::MemoryPropertyFlagBits::eHostVisible | vk::MemoryPropertyFlagBits::eHostCoherent) {

    // Set up dynamic rendering info
    renderingInfo.colorAttachmentCount = 1;
    vk::Format formats[] = { colorFormat };
    renderingInfo.pColorAttachmentFormats = &colorFormat;
}

ImGuiVulkanUtil::~ImGuiVulkanUtil() {
    // Wait for device to finish operations before destroying resources
    // NOTE: waitIdle() is acceptable in destructors/cleanup code but should NEVER be used
    // in the main rendering loop as it causes severe performance issues. For frame
    // synchronization, use fences and semaphores instead.
    if (device) {
        device->waitIdle();
    }

    // All resources are automatically cleaned up by their destructors
    // No manual cleanup needed

    // ImGui context is destroyed separately
}

Initialization

Next, let’s implement the initialization methods:

void ImGuiVulkanUtil::init(float width, float height) {
    // Initialize ImGui context
    IMGUI_CHECKVERSION();
    ImGui::CreateContext();

    // Configure ImGui
    ImGuiIO& io = ImGui::GetIO();
    io.ConfigFlags |= ImGuiConfigFlags_NavEnableKeyboard;  // Enable keyboard controls
    io.ConfigFlags |= ImGuiConfigFlags_DockingEnable;      // Enable docking

    // Inform ImGui that we support the new texture update protocol (v1.92+)
    // This enables support for dynamic font textures and multiple texture atlases
    io.BackendFlags |= ImGuiBackendFlags_RendererHasTextures;

    // Set display size
    io.DisplaySize = ImVec2(width, height);
    io.DisplayFramebufferScale = ImVec2(1.0f, 1.0f);

    // Set up style
    vulkanStyle = ImGui::GetStyle();
    vulkanStyle.Colors[ImGuiCol_TitleBg] = ImVec4(1.0f, 0.0f, 0.0f, 0.6f);
    vulkanStyle.Colors[ImGuiCol_TitleBgActive] = ImVec4(1.0f, 0.0f, 0.0f, 0.8f);
    vulkanStyle.Colors[ImGuiCol_MenuBarBg] = ImVec4(1.0f, 0.0f, 0.0f, 0.4f);
    vulkanStyle.Colors[ImGuiCol_Header] = ImVec4(1.0f, 0.0f, 0.0f, 0.4f);
    vulkanStyle.Colors[ImGuiCol_CheckMark] = ImVec4(0.0f, 1.0f, 0.0f, 1.0f);

    // Apply default style
    setStyle(0);
}

void ImGuiVulkanUtil::setStyle(uint32_t index) {
    ImGuiStyle& style = ImGui::GetStyle();

    switch (index) {
        case 0:
            // Custom Vulkan style
            style = vulkanStyle;
            break;
        case 1:
            // Classic style
            ImGui::StyleColorsClassic();
            break;
        case 2:
            // Dark style
            ImGui::StyleColorsDark();
            break;
        case 3:
            // Light style
            ImGui::StyleColorsLight();
            break;
    }
}

Resource Initialization

Now let’s implement the method to initialize all Vulkan resources needed for ImGui rendering. This complex process involves several distinct steps that work together to create the GPU resources required for text and UI rendering.

Resource Initialization: Core Vulkan Resources

Now let’s implement the initResources method to initialize the common Vulkan resources needed for ImGui rendering. We’ll start by creating the sampler, descriptor pool, and descriptor set layout.

void ImGuiVulkanUtil::initResources() {
    // Configure texture sampling parameters for optimal text rendering
    // These settings directly impact text quality and performance
    vk::SamplerCreateInfo samplerInfo{};
    samplerInfo.magFilter = vk::Filter::eLinear;                    // Smooth scaling when magnified
    samplerInfo.minFilter = vk::Filter::eLinear;                    // Smooth scaling when minified
    samplerInfo.mipmapMode = vk::SamplerMipmapMode::eLinear;        // Smooth transitions between mip levels
    samplerInfo.addressModeU = vk::SamplerAddressMode::eClampToEdge;  // Prevent texture wrapping
    samplerInfo.addressModeV = vk::SamplerAddressMode::eClampToEdge;  // Clean edge handling
    samplerInfo.addressModeW = vk::SamplerAddressMode::eClampToEdge;  // 3D consistency
    samplerInfo.borderColor = vk::BorderColor::eFloatOpaqueWhite;   // White border for clamped areas

    sampler = device->createSampler(samplerInfo);                   // Create the GPU sampler object

    // Create descriptor pool for shader resource binding
    // Descriptors provide the interface between shaders and GPU resources
    vk::DescriptorPoolSize poolSize{vk::DescriptorType::eCombinedImageSampler, 1};

    vk::DescriptorPoolCreateInfo poolInfo{};
    poolInfo.flags = vk::DescriptorPoolCreateFlagBits::eFreeDescriptorSet;     // Allow individual descriptor set freeing
    poolInfo.maxSets = 2;                                                      // Maximum number of descriptor sets
    poolInfo.poolSizeCount = 1;                                                // Number of pool size specifications
    poolInfo.pPoolSizes = &poolSize;                                           // Pool size configuration

    descriptorPool = device->createDescriptorPool(poolInfo);                   // Create descriptor pool

    // Create descriptor set layout defining shader resource interface
    // This layout must match the binding declarations in the ImGui shaders
    vk::DescriptorSetLayoutBinding binding{};
    binding.descriptorType = vk::DescriptorType::eCombinedImageSampler;        // Combined texture and sampler
    binding.descriptorCount = 1;                                               // Single texture binding
    binding.stageFlags = vk::ShaderStageFlagBits::eFragment;                   // Used in fragment shader
    binding.binding = 0;                                                       // Shader binding point 0

    vk::DescriptorSetLayoutCreateInfo layoutInfo{};
    layoutInfo.bindingCount = 1;                                               // Number of bindings in layout
    layoutInfo.pBindings = &binding;                                           // Binding configuration array

    descriptorSetLayout = device->createDescriptorSetLayout(layoutInfo);       // Create layout object

    // Allocate descriptor set from pool using the defined layout
    // This creates the actual binding that connects GPU resources to shaders
    vk::DescriptorSetAllocateInfo allocInfo{};
    allocInfo.descriptorPool = *descriptorPool;                                // Source pool for allocation
    allocInfo.descriptorSetCount = 1;                                          // Number of sets to allocate
    vk::DescriptorSetLayout layouts[] = {*descriptorSetLayout};                // Layout template array
    allocInfo.pSetLayouts = layouts;                                           // Layout configuration

    descriptorSet = std::move(device->allocateDescriptorSets(allocInfo).front()); // Allocate and store set

    // Update descriptor set with actual font texture and sampler resources
    // This final step connects the physical GPU resources to the shader binding points
    vk::DescriptorImageInfo imageInfo{};
    imageInfo.imageLayout = vk::ImageLayout::eShaderReadOnlyOptimal;           // Expected image layout
    imageInfo.imageView = fontImageView.getHandle();                           // Font texture view
    imageInfo.sampler = *sampler;                                              // Texture sampler

    vk::WriteDescriptorSet writeSet{};
    writeSet.dstSet = *descriptorSet;                                          // Target descriptor set
    writeSet.descriptorCount = 1;                                              // Number of resources to bind
    writeSet.descriptorType = vk::DescriptorType::eCombinedImageSampler;       // Resource type
    writeSet.pImageInfo = &imageInfo;                                          // Image resource information
    writeSet.dstBinding = 0;                                                   // Binding point in shader

    device->updateDescriptorSets(1, &writeSet, 0, nullptr);                   // Execute the binding update

    // Create pipeline cache
    vk::PipelineCacheCreateInfo pipelineCacheInfo{};
    pipelineCache = device->createPipelineCache(pipelineCacheInfo);

    // Create pipeline layout
    vk::PushConstantRange pushConstantRange{};
    pushConstantRange.stageFlags = vk::ShaderStageFlagBits::eVertex;
    pushConstantRange.offset = 0;
    pushConstantRange.size = sizeof(PushConstBlock);

    vk::PipelineLayoutCreateInfo pipelineLayoutInfo{};
    pipelineLayoutInfo.setLayoutCount = 1;
    vk::DescriptorSetLayout setLayouts[] = {*descriptorSetLayout};
    pipelineLayoutInfo.pSetLayouts = setLayouts;
    pipelineLayoutInfo.pushConstantRangeCount = 1;
    pipelineLayoutInfo.pPushConstantRanges = &pushConstantRange;

    pipelineLayout = device->createPipelineLayout(pipelineLayoutInfo);

    // Create the graphics pipeline with dynamic rendering
    // ... (shader loading, pipeline state setup, etc.)

    // For brevity, we're omitting the full pipeline creation code here
    // In a real implementation, you would:
    // 1. Load the vertex and fragment shaders
    // 2. Set up all the pipeline state (vertex input, input assembly, rasterization, etc.)
    // 3. Include the renderingInfo in the pipeline creation to enable dynamic rendering
}

Dynamic Texture Handling

Modern Dear ImGui (v1.92+) supports dynamic texture updates. Instead of a one-time static upload, we implement a protocol where the renderer backend updates textures as requested by ImGui. We implement the updateTexture method which handles the creation and upload of texture data to the GPU. This method will be called during the rendering phase if ImGui indicates a texture needs an update.

void ImGuiVulkanUtil::updateTexture(ImTextureData* tex) {
    if (tex->Status == ImTextureStatus_WantCreate || tex->Status == ImTextureStatus_WantUpdates) {
        int texWidth = tex->Width;
        int texHeight = tex->Height;
        unsigned char* fontData = (unsigned char*)tex->Pixels;

        if (!fontData) return;

        vk::DeviceSize uploadSize = texWidth * texHeight * tex->BytesPerPixel;
        vk::Format format = (tex->BytesPerPixel == 4) ? vk::Format::eR8G8B8A8Unorm : vk::Format::eR8Unorm;

        if (tex->Status == ImTextureStatus_WantCreate) {
            // Create optimized GPU image for texture storage
            vk::Extent3D extent{static_cast<uint32_t>(texWidth), static_cast<uint32_t>(texHeight), 1};
            fontImage = Image(*device, extent, format,
                            vk::ImageUsageFlagBits::eSampled | vk::ImageUsageFlagBits::eTransferDst,
                            vk::MemoryPropertyFlagBits::eDeviceLocal);

            fontImageView = ImageView(*device, fontImage.getHandle(), format,
                                   vk::ImageAspectFlagBits::eColor);
        }

        // Create staging buffer for efficient CPU-to-GPU data transfer
        Buffer stagingBuffer(*device, uploadSize, vk::BufferUsageFlagBits::eTransferSrc,
                           vk::MemoryPropertyFlagBits::eHostVisible | vk::MemoryPropertyFlagBits::eHostCoherent);

        // Copy data to staging buffer
        void* data = stagingBuffer.map();
        memcpy(data, fontData, uploadSize);
        stagingBuffer.unmap();

        // Transition image layout and copy data
        transitionImageLayout(fontImage.getHandle(), format,
                             vk::ImageLayout::eUndefined, vk::ImageLayout::eTransferDstOptimal);
        copyBufferToImage(stagingBuffer.getHandle(), fontImage.getHandle(),
                         static_cast<uint32_t>(texWidth), static_cast<uint32_t>(texHeight));
        transitionImageLayout(fontImage.getHandle(), format,
                             vk::ImageLayout::eTransferDstOptimal, vk::ImageLayout::eShaderReadOnlyOptimal);

        // Store descriptor set handle as the ImTextureID
        // In this implementation, we use a single descriptor set for the font atlas
        tex->SetTexID((ImTextureID)(intptr_t)(VkDescriptorSet)*descriptorSet);
        tex->SetStatus(ImTextureStatus_OK);
    }
}

Frame Management and Rendering

Finally, let’s implement the methods for frame management and rendering:

bool ImGuiVulkanUtil::newFrame() {
    // Start a new ImGui frame
    ImGui::NewFrame();

    // Create your UI elements here
    // For example:
    ImGui::Begin("Vulkan ImGui Demo");
    ImGui::Text("Hello, Vulkan!");
    if (ImGui::Button("Click me!")) {
        // Handle button click
    }
    ImGui::End();

    // End the frame
    ImGui::EndFrame();

    // Render to generate draw data
    ImGui::Render();

    // Check if buffers need updating
    ImDrawData* drawData = ImGui::GetDrawData();
    if (drawData && drawData->CmdListsCount > 0) {
        if (drawData->TotalVtxCount > vertexCount || drawData->TotalIdxCount > indexCount) {
            needsUpdateBuffers = true;
            return true;
        }
    }

    return false;
}

void ImGuiVulkanUtil::updateBuffers() {
    ImDrawData* drawData = ImGui::GetDrawData();
    if (!drawData || drawData->CmdListsCount == 0) {
        return;
    }

    // Calculate required buffer sizes
    vk::DeviceSize vertexBufferSize = drawData->TotalVtxCount * sizeof(ImDrawVert);
    vk::DeviceSize indexBufferSize = drawData->TotalIdxCount * sizeof(ImDrawIdx);

    // Resize buffers if needed
    if (drawData->TotalVtxCount > vertexCount) {
        // Recreate vertex buffer with new size
        vertexBuffer = Buffer(*device, vertexBufferSize,
                            vk::BufferUsageFlagBits::eVertexBuffer,
                            vk::MemoryPropertyFlagBits::eHostVisible | vk::MemoryPropertyFlagBits::eHostCoherent);
        vertexCount = drawData->TotalVtxCount;
    }

    if (drawData->TotalIdxCount > indexCount) {
        // Recreate index buffer with new size
        indexBuffer = Buffer(*device, indexBufferSize,
                           vk::BufferUsageFlagBits::eIndexBuffer,
                           vk::MemoryPropertyFlagBits::eHostVisible | vk::MemoryPropertyFlagBits::eHostCoherent);
        indexCount = drawData->TotalIdxCount;
    }

    // Upload data to buffers
    ImDrawVert* vtxDst = static_cast<ImDrawVert*>(vertexBuffer.map());
    ImDrawIdx* idxDst = static_cast<ImDrawIdx*>(indexBuffer.map());

    for (int n = 0; n < drawData->CmdListsCount; n++) {
        const ImDrawList* cmdList = drawData->CmdLists[n];
        memcpy(vtxDst, cmdList->VtxBuffer.Data, cmdList->VtxBuffer.Size * sizeof(ImDrawVert));
        memcpy(idxDst, cmdList->IdxBuffer.Data, cmdList->IdxBuffer.Size * sizeof(ImDrawIdx));
        vtxDst += cmdList->VtxBuffer.Size;
        idxDst += cmdList->IdxBuffer.Size;
    }

    vertexBuffer.unmap();
    indexBuffer.unmap();
}

Begin a rendering scope

Before issuing any UI draw commands, we open a dynamic rendering scope that targets the current framebuffer. This replaces vkCmdBeginRenderPass/EndRenderPass and keeps the UI pass lightweight.

void ImGuiVulkanUtil::drawFrame(vk::raii::CommandBuffer& commandBuffer) {
    ImDrawData* drawData = ImGui::GetDrawData();
    if (!drawData || drawData->CmdListsCount == 0) {
        return;
    }

    // Process dynamic texture updates (v1.92+ RendererHasTextures protocol)
    // This handles font atlas regeneration and any user-provided textures
    if (drawData->Textures) {
        for (int n = 0; n < drawData->Textures->Size; n++) {
            ImTextureData* tex = (*drawData->Textures)[n];
            if (tex->Status != ImTextureStatus_OK) {
                updateTexture(tex);
            }
        }
    }

    // Begin dynamic rendering
    vk::RenderingAttachmentInfo colorAttachment{};
    // Note: In a real implementation, you would set imageView, imageLayout,
    // loadOp, storeOp, and clearValue based on your swapchain image

    vk::RenderingInfo renderingInfo{};
    renderingInfo.renderArea = vk::Rect2D{{0, 0}, {static_cast<uint32_t>(drawData->DisplaySize.x),
                                                   static_cast<uint32_t>(drawData->DisplaySize.y)}};
    renderingInfo.layerCount = 1;
    renderingInfo.colorAttachmentCount = 1;
    renderingInfo.pColorAttachments = &colorAttachment;

    commandBuffer.beginRendering(renderingInfo);

At this point, commands affect the UI overlay only. Next we bind state that doesn’t change per draw.

Bind pipeline and set viewport

    // Bind the pipeline used for ImGui
    commandBuffer.bindPipeline(vk::PipelineBindPoint::eGraphics, *pipeline);

    // Configure viewport for UI pixel coordinates
    vk::Viewport viewport{};
    viewport.width = drawData->DisplaySize.x;
    viewport.height = drawData->DisplaySize.y;
    viewport.minDepth = 0.0f;
    viewport.maxDepth = 1.0f;
    commandBuffer.setViewport(0, viewport);

The pipeline has blending and raster states tailored for UI. The viewport maps ImGui’s coordinate system to the framebuffer.

Push per-frame constants

    // Convert from ImGui coordinates into NDC via a simple scale/translate
    pushConstBlock.scale = glm::vec2(2.0f / drawData->DisplaySize.x, 2.0f / drawData->DisplaySize.y);
    pushConstBlock.translate = glm::vec2(-1.0f);
    commandBuffer.pushConstants(*pipelineLayout, vk::ShaderStageFlagBits::eVertex,
                              0, sizeof(PushConstBlock), &pushConstBlock);

This keeps the shader simple and avoids per-vertex work for coordinate transforms.

Bind geometry buffers

    // We already filled these buffers this frame
    vk::Buffer vertexBuffers[] = { vertexBuffer.getHandle() };
    vk::DeviceSize offsets[] = { 0 };
    commandBuffer.bindVertexBuffers(0, 1, vertexBuffers, offsets);
    commandBuffer.bindIndexBuffer(indexBuffer.getHandle(), 0, vk::IndexType::eUint16);

Iterate command lists, set scissor, draw

    int vertexOffset = 0;
    int indexOffset = 0;

    for (int i = 0; i < drawData->CmdListsCount; i++) {
        const ImDrawList* cmdList = drawData->CmdLists[i];

        for (int j = 0; j < cmdList->CmdBuffer.Size; j++) {
            const ImDrawCmd* pcmd = &cmdList->CmdBuffer[j];

            // Clip per draw call
            vk::Rect2D scissor{};
            scissor.offset.x = std::max(static_cast<int32_t>(pcmd->ClipRect.x), 0);
            scissor.offset.y = std::max(static_cast<int32_t>(pcmd->ClipRect.y), 0);
            scissor.extent.width = static_cast<uint32_t>(pcmd->ClipRect.z - pcmd->ClipRect.x);
            scissor.extent.height = static_cast<uint32_t>(pcmd->ClipRect.w - pcmd->ClipRect.y);
            commandBuffer.setScissor(0, scissor);

            // Bind font (and any UI) textures for this draw
            // The TexID now stores the actual descriptor set handle (VkDescriptorSet)
            VkDescriptorSet texHandle = (VkDescriptorSet)pcmd->GetTexID();
            if (texHandle) {
                commandBuffer.bindDescriptorSets(vk::PipelineBindPoint::eGraphics,
                                               *pipelineLayout, 0, {vk::DescriptorSet(texHandle)}, {});
            } else {
                // Fallback to default font if no specific texture is bound
                commandBuffer.bindDescriptorSets(vk::PipelineBindPoint::eGraphics,
                                               *pipelineLayout, 0, {*descriptorSet}, {});
            }

            // Issue indexed draw for this UI batch
            commandBuffer.drawIndexed(pcmd->ElemCount, 1, indexOffset, vertexOffset, 0);
            indexOffset += pcmd->ElemCount;
        }

        vertexOffset += cmdList->VtxBuffer.Size;
    }

Each ImDrawCmd provides a scissor rect that clips widgets efficiently without extra passes.

End the rendering scope

    // Close the rendering scope for the UI overlay
    commandBuffer.endRendering();
}

Input Handling

Let’s implement the input handling methods using the modern event-based API introduced in ImGui v1.87. This replaces legacy direct state modification (like io.KeysDown[] or io.MousePos) with a robust event queue that properly handles input timing and multiple devices.

void ImGuiVulkanUtil::handleKey(int key, int scancode, int action, int mods) {
    ImGuiIO& io = ImGui::GetIO();

    // Map the platform-specific key action to a boolean state
    // In GLFW: GLFW_RELEASE = 0, GLFW_PRESS = 1, GLFW_REPEAT = 2
    bool pressed = (action != 0);

    // Modern ImGui (v1.87+) uses AddKeyEvent to queue input events.
    // This handles key states, modifiers, and repeat logic internally.
    // Most backends can cast native key codes directly to ImGuiKey.
    io.AddKeyEvent((ImGuiKey)key, pressed);
}

void ImGuiVulkanUtil::handleMousePos(float x, float y) {
    ImGuiIO& io = ImGui::GetIO();
    // Modern event API for mouse position
    io.AddMousePosEvent(x, y);
}

void ImGuiVulkanUtil::handleMouseButton(int button, bool pressed) {
    ImGuiIO& io = ImGui::GetIO();
    // Modern event API for mouse buttons (0: Left, 1: Right, 2: Middle)
    io.AddMouseButtonEvent(button, pressed);
}

bool ImGuiVulkanUtil::getWantKeyCapture() {
    return ImGui::GetIO().WantCaptureKeyboard;
}

void ImGuiVulkanUtil::charPressed(uint32_t key) {
    ImGuiIO& io = ImGui::GetIO();
    io.AddInputCharacter(key);
}

Using the ImGuiVulkanUtil Class

Now that we’ve implemented our ImGuiVulkanUtil class, let’s see how to use it in a Vulkan application:

// In your application class
ImGuiVulkanUtil imGui;

// During initialization
void initImGui() {
    // Initialize ImGui directly
    imGui = ImGuiVulkanUtil(
        device,
        physicalDevice,
        graphicsQueue,
        graphicsQueueFamily
    );

    imGui.init(swapChainExtent.width, swapChainExtent.height);
    imGui.initResources(); // No renderPass needed with dynamic rendering
}

// In your render loop
void drawFrame() {
    // ... existing frame preparation code ...

    // Update ImGui
    if (imGui.newFrame()) {
        imGui.updateBuffers();
    }

    // Begin command buffer recording
    // Note: With dynamic rendering, we don't need to begin a render pass
    // The ImGui drawFrame method will handle dynamic rendering internally

    // Render scene using dynamic rendering
    // ...

    // Render ImGui (in multi-frame renderers, pass the current frame index to bind per-frame buffers)
    imGui.drawFrame(commandBuffer);

    // ... submit command buffer ...
}

// Input handling
// This example shows how to handle input with GLFW, but you can adapt this
// to work with any windowing library's input system

// Example key callback function for GLFW
void keyCallback(GLFWwindow* window, int key, int scancode, int action, int mods) {
    // First check if ImGui wants to capture this input
    imGui.handleKey(key, scancode, action, mods);

    // If ImGui doesn't want to capture the keyboard, process for your application
    if (!imGui.getWantKeyCapture()) {
        // Process key for your application
    }
}

// Example character input callback for GLFW
void charCallback(GLFWwindow* window, unsigned int codepoint) {
    imGui.charPressed(codepoint);
}

// With other windowing libraries, you would implement similar callback functions
// using their equivalent APIs and event systems

// Cleanup
void cleanup() {
    // ... existing cleanup code ...

    // ImGui will be automatically cleaned up when the application exits
    // No manual cleanup needed
}

Testing the Integration

To verify that our ImGui integration is working correctly, we can use the ImGui demo window, which showcases all of ImGui’s features:

// In your ImGuiVulkanUtil::newFrame method
bool ImGuiVulkanUtil::newFrame() {
    ImGui::NewFrame();

    // Show the demo window
    ImGui::ShowDemoWindow();

    ImGui::EndFrame();
    ImGui::Render();

    // Check if buffers need updating
    // ...
}

With this implementation, you have a Vulkan implementation for ImGui that allows you to customize the rendering process to fit your specific needs.

In the next section, we’ll explore how to handle input for both the GUI and the 3D scene.

GUI: Input Handling

Input Handling

One of the challenges when integrating a GUI into a 3D application is managing input events. We need to ensure that input events are correctly routed to either the GUI or the 3D scene. For example, if the user is interacting with a UI element, we don’t want their mouse movements to also rotate the camera.

In this section, we’ll explore how to handle input for both the GUI and the 3D scene, ensuring a smooth user experience regardless of the windowing library you choose to use.

Note

A windowing library is a software framework that provides functionality for creating and managing application windows, handling user input events (keyboard, mouse, touch), and interfacing with the operating system’s display and input systems. Examples include GLFW, SDL, Qt, and SFML. These libraries abstract the platform-specific details of window management and input handling, allowing developers to write code that works across different operating systems without dealing with platform-specific APIs directly.

Creating a Platform-Agnostic Input System

To create an effective input system that works with any windowing library, we need to abstract the input mechanisms and provide a clean interface. Let’s define a simple input system that can be adapted to different platforms:

// InputSystem.h
#pragma once

#include <functional>
#include <unordered_map>
#include <vector>
#include <glm/glm.hpp>

// Input actions that our application can respond to
enum class InputAction {
    MOVE_FORWARD,
    MOVE_BACKWARD,
    MOVE_LEFT,
    MOVE_RIGHT,
    MOVE_UP,
    MOVE_DOWN,
    LOOK_UP,
    LOOK_DOWN,
    LOOK_LEFT,
    LOOK_RIGHT,
    ZOOM_IN,
    ZOOM_OUT,
    TOGGLE_UI_MODE,
    // Add more actions as needed
};

// Input state that tracks the current state of inputs
struct InputState {
    glm::vec2 cursorPosition = {0.0f, 0.0f};
    glm::vec2 cursorDelta = {0.0f, 0.0f};
    bool mouseButtons[3] = {false, false, false};
    float scrollDelta = 0.0f;

    // For touch input
    struct TouchPoint {
        int id;
        glm::vec2 position;
        glm::vec2 delta;
    };
    std::vector<TouchPoint> touchPoints;

    // Reset delta values after each frame
    void resetDeltas() {
        cursorDelta = {0.0f, 0.0f};
        scrollDelta = 0.0f;
        for (auto& touch : touchPoints) {
            touch.delta = {0.0f, 0.0f};
        }
    }
};

class InputSystem {
public:
    static void Initialize();
    static void Shutdown();

    // Update input state (called once per frame)
    static void Update(float deltaTime);

    // Register a callback for an input action
    static void RegisterActionCallback(InputAction action, std::function<void(float)> callback);

    // Process a platform-specific input event
    static bool ProcessInputEvent(void* event);

    // Get the current input state
    static const InputState& GetInputState();

    // Check if ImGui is capturing input
    static bool IsImGuiCapturingKeyboard();
    static bool IsImGuiCapturingMouse();

private:
    static InputState inputState;
    static std::unordered_map<InputAction, std::function<void(float)>> actionCallbacks;
};

Input Prioritization

The general approach for input handling in applications with both 3D navigation and GUI is:

  1. First, check if the GUI is capturing input (e.g., mouse is over a UI element)

  2. If the GUI is not capturing input, then process the input for 3D navigation

Let’s implement this approach using our cross-platform input system:

void processInput(float deltaTime) {
    // Check if ImGui is capturing keyboard input
    bool imguiCapturingKeyboard = InputSystem::IsImGuiCapturingKeyboard();

    // Check if ImGui is capturing mouse input
    bool imguiCapturingMouse = InputSystem::IsImGuiCapturingMouse();

    // Get the current input state
    const InputState& inputState = InputSystem::GetInputState();

    // Process keyboard input for camera movement if ImGui is not capturing keyboard
    if (!imguiCapturingKeyboard) {
        // Forward these to the camera system
        // This could be done through the action callback system
        if (InputSystem::IsActionActive(InputAction::MOVE_FORWARD))
            camera.processKeyboard(CameraMovement::FORWARD, deltaTime);
        if (InputSystem::IsActionActive(InputAction::MOVE_BACKWARD))
            camera.processKeyboard(CameraMovement::BACKWARD, deltaTime);
        if (InputSystem::IsActionActive(InputAction::MOVE_LEFT))
            camera.processKeyboard(CameraMovement::LEFT, deltaTime);
        if (InputSystem::IsActionActive(InputAction::MOVE_RIGHT))
            camera.processKeyboard(CameraMovement::RIGHT, deltaTime);
        if (InputSystem::IsActionActive(InputAction::MOVE_UP))
            camera.processKeyboard(CameraMovement::UP, deltaTime);
        if (InputSystem::IsActionActive(InputAction::MOVE_DOWN))
            camera.processKeyboard(CameraMovement::DOWN, deltaTime);
    }

    // Process mouse/touch input for camera rotation if ImGui is not capturing mouse
    if (!imguiCapturingMouse) {
        if (inputState.cursorDelta.x != 0.0f || inputState.cursorDelta.y != 0.0f) {
            camera.processMouseMovement(inputState.cursorDelta.x, -inputState.cursorDelta.y);
        }

        if (inputState.scrollDelta != 0.0f) {
            camera.processMouseScroll(inputState.scrollDelta);
        }
    }
}

Implementing Platform Adapters for Input

While our input system design is platform-agnostic, we still need platform-specific adapters to bridge between our unified interface and each windowing library’s native input events. Here’s an example implementation using GLFW, a popular windowing library:

Example: GLFW Implementation

// InputSystem_GLFW.cpp

#include "InputSystem.h"
#include <GLFW/glfw3.h>
#include <imgui.h>

// Store the GLFW window pointer
static GLFWwindow* gWindow = nullptr;
static bool mouseCaptureMode = false;

// GLFW callback functions
static void glfwMouseButtonCallback(GLFWwindow* window, int button, int action, int mods) {
    if (button >= 0 && button < 3) {
        InputState& state = InputSystem::GetInputState();
        state.mouseButtons[button] = action == GLFW_PRESS;
    }
}

static void glfwCursorPosCallback(GLFWwindow* window, double xpos, double ypos) {
    InputState& state = InputSystem::GetInputState();

    // Calculate delta from last position
    glm::vec2 newPos(static_cast<float>(xpos), static_cast<float>(ypos));
    state.cursorDelta = newPos - state.cursorPosition;
    state.cursorPosition = newPos;
}

static void glfwScrollCallback(GLFWwindow* window, double xoffset, double yoffset) {
    InputState& state = InputSystem::GetInputState();
    state.scrollDelta = static_cast<float>(yoffset);
}

static void glfwKeyCallback(GLFWwindow* window, int key, int scancode, int action, int mods) {
    // Map GLFW keys to our input actions
    if (action == GLFW_PRESS || action == GLFW_RELEASE) {
        bool pressed = (action == GLFW_PRESS);

        // Toggle mouse capture mode with Escape key
        if (key == GLFW_KEY_ESCAPE && pressed) {
            mouseCaptureMode = !mouseCaptureMode;

            if (mouseCaptureMode) {
                glfwSetInputMode(window, GLFW_CURSOR, GLFW_CURSOR_DISABLED);
            } else {
                glfwSetInputMode(window, GLFW_CURSOR, GLFW_CURSOR_NORMAL);
            }
        }

        // Map other keys to actions
        // ...
    }
}

void InputSystem::Initialize(GLFWwindow* window) {
    gWindow = window;

    // Set up GLFW callbacks
    glfwSetMouseButtonCallback(window, glfwMouseButtonCallback);
    glfwSetCursorPosCallback(window, glfwCursorPosCallback);
    glfwSetScrollCallback(window, glfwScrollCallback);
    glfwSetKeyCallback(window, glfwKeyCallback);

    // Initially capture the cursor for camera control
    mouseCaptureMode = true;
    glfwSetInputMode(window, GLFW_CURSOR, GLFW_CURSOR_DISABLED);
}

void InputSystem::Update(float deltaTime) {
    // Poll for input events
    glfwPollEvents();

    // Update key states for continuous actions (like movement)
    if (glfwGetKey(gWindow, GLFW_KEY_W) == GLFW_PRESS) {
        if (auto it = actionCallbacks.find(InputAction::MOVE_FORWARD); it != actionCallbacks.end()) {
            it->second(deltaTime);
        }
    }

    // ... other keys ...

    // Reset delta values after processing
    inputState.resetDeltas();
}

bool InputSystem::IsImGuiCapturingKeyboard() {
    return ImGui::GetIO().WantCaptureKeyboard;
}

bool InputSystem::IsImGuiCapturingMouse() {
    return ImGui::GetIO().WantCaptureMouse;
}

Input Modes

For applications that need different input modes (e.g., camera control vs. UI interaction), we can implement a mode system:

// Define input modes
enum class InputMode {
    CAMERA_CONTROL,
    UI_INTERACTION,
    OBJECT_MANIPULATION
};

// Current input mode
static InputMode currentInputMode = InputMode::CAMERA_CONTROL;

// Set the input mode
void setInputMode(InputMode mode) {
    currentInputMode = mode;

    // Update platform-specific settings based on the mode
    // This example shows how to implement this with GLFW
    if (mode == InputMode::CAMERA_CONTROL) {
        // In GLFW, we can disable the cursor for camera control
        glfwSetInputMode(gWindow, GLFW_CURSOR, GLFW_CURSOR_DISABLED);
    } else {
        // For UI interaction, we want the cursor to be visible
        glfwSetInputMode(gWindow, GLFW_CURSOR, GLFW_CURSOR_NORMAL);
    }

    // With other windowing libraries, you would use their equivalent APIs
}

// Toggle between camera control and UI interaction modes
void toggleInputMode() {
    if (currentInputMode == InputMode::CAMERA_CONTROL) {
        setInputMode(InputMode::UI_INTERACTION);
    } else {
        setInputMode(InputMode::CAMERA_CONTROL);
    }
}

Handling GUI-Specific Input

Some GUI interactions might require special handling. For example, you might want to implement drag-and-drop functionality or custom keyboard shortcuts for UI elements:

void drawGUI() {
    // Start a new ImGui frame
    ImGui::NewFrame();

    // Create a window for camera controls
    ImGui::Begin("Camera Controls");

    // Add a button to reset camera position
    if (ImGui::Button("Reset Camera")) {
        camera.setPosition(glm::vec3(0.0f, 0.0f, 3.0f));
        camera.setYaw(-90.0f);
        camera.setPitch(0.0f);
    }

    // Add sliders for camera settings
    float movementSpeed = camera.getMovementSpeed();
    if (ImGui::SliderFloat("Movement Speed", &movementSpeed, 1.0f, 10.0f)) {
        camera.setMovementSpeed(movementSpeed);
    }

    float sensitivity = camera.getMouseSensitivity();
    if (ImGui::SliderFloat("Mouse Sensitivity", &sensitivity, 0.1f, 1.0f)) {
        camera.setMouseSensitivity(sensitivity);
    }

    float zoom = camera.getZoom();
    if (ImGui::SliderFloat("Zoom", &zoom, 1.0f, 45.0f)) {
        camera.setZoom(zoom);
    }

    ImGui::End();

    // Render ImGui
    ImGui::Render();
}

Integrating Input Handling with the Main Loop

Finally, let’s integrate our input handling system with the main loop:

void mainLoop() {
    // Main application loop
    while (isRunning) {
        // Calculate delta time
        float deltaTime = calculateDeltaTime();

        // Update input system
        InputSystem::Update(deltaTime);

        // Process input for camera and other systems
        processInput(deltaTime);

        // Draw GUI
        drawGUI();

        // Update uniform buffer with latest camera data
        updateUniformBuffer(currentFrame);

        // Draw frame
        drawFrame();
    }
}

Main Loop Integration

The input system needs to be integrated with your application’s main loop. Here’s an example of how to do this with GLFW, but similar principles apply to other windowing libraries:

// Example main loop with GLFW
void runMainLoop() {
    // Initialize input system with your window
    // With GLFW, this would look like:
    InputSystem::Initialize(window);

    // Main loop - with GLFW, we check if the window should close
    // Other libraries would have their own condition
    while (!glfwWindowShouldClose(window)) {
        float deltaTime = calculateDeltaTime();

        // Update input and process events
        // This would be platform-specific
        InputSystem::Update(deltaTime);

        // Rest of the main loop is platform-independent
        processInput(deltaTime);
        drawGUI();
        updateUniformBuffer(currentFrame);
        drawFrame();
    }
}

Advanced Input Handling Techniques

For more complex applications, you might want to consider these advanced input handling techniques:

Gesture Recognition

Gesture recognition can enhance the user experience regardless of which windowing library you use:

// GestureRecognizer.h
#pragma once

#include <glm/glm.hpp>
#include <vector>
#include <functional>

enum class GestureType {
    TAP,
    DOUBLE_TAP,
    LONG_PRESS,
    SWIPE,
    PINCH,
    ROTATE,
    PAN
};

struct GestureEvent {
    GestureType type;
    glm::vec2 position;
    glm::vec2 delta;
    float scale;  // For pinch
    float rotation;  // For rotate
    int pointerCount;
};

class GestureRecognizer {
public:
    static void Initialize();
    static void Update(const InputState& inputState, float deltaTime);

    // Register callbacks for different gesture types
    static void RegisterGestureCallback(GestureType type, std::function<void(const GestureEvent&)> callback);

private:
    static void detectTap(const InputState& inputState);
    static void detectSwipe(const InputState& inputState);
    static void detectPinch(const InputState& inputState);
    static void detectRotate(const InputState& inputState);
    static void detectPan(const InputState& inputState);

    static std::unordered_map<GestureType, std::function<void(const GestureEvent&)>> gestureCallbacks;
};

Input Context System

For more complex applications with different input requirements in different states:

// InputContext.h
#pragma once

#include <string>
#include <unordered_map>
#include <functional>
#include <stack>

class InputContext {
public:
    // Create a new input context
    static void CreateContext(const std::string& name);

    // Push a context onto the stack (making it active)
    static void PushContext(const std::string& name);

    // Pop the top context from the stack
    static void PopContext();

    // Get the current active context
    static std::string GetActiveContext();

    // Register an action handler for a specific context
    static void RegisterActionHandler(const std::string& contextName, InputAction action, std::function<void(float)> handler);

    // Process an action in the current context
    static void ProcessAction(InputAction action, float deltaTime);

private:
    static std::unordered_map<std::string, std::unordered_map<InputAction, std::function<void(float)>>> contextHandlers;
    static std::stack<std::string> contextStack;
};

With these advanced input handling techniques, your application can provide a consistent and intuitive user experience. In the next section, we’ll explore how to create various UI elements to control your application.

GUI: UI Elements and Integration Concepts

UI Elements and Integration Concepts

Now that we have set up ImGui and implemented input handling, let’s explore the key concepts of integrating a GUI with your Vulkan application. We’ll focus on the integration aspects rather than exhaustive ImGui widget examples, as those are well-documented in the ImGui documentation.

GUI Integration Concepts

When integrating a GUI into a 3D application, there are several important concepts to consider:

  1. Separation of Concerns: Keep your GUI code separate from your rendering code to maintain clean architecture.

  2. Performance Impact: GUIs can impact performance, especially with complex layouts or frequent updates.

  3. Input Management: Properly handle input to ensure it’s routed to either the GUI or the 3D scene.

  4. Rendering Order: The GUI is typically rendered after the 3D scene, as an overlay.

  5. State Management: Use the GUI to modify application state in a controlled manner.

Basic ImGui Usage

ImGui follows an immediate-mode paradigm, where the UI is recreated every frame. Here’s a simple example:

void drawGUI() {
    // Start a new ImGui frame
    ImGui::NewFrame();

    // Create a window
    ImGui::Begin("Settings");

    // Add UI elements here
    static bool enableFeature = false;
    if (ImGui::Checkbox("Enable Feature", &enableFeature)) {
        // This code runs when the checkbox value changes
        updateFeatureState(enableFeature);
    }

    static float value = 0.5f;
    if (ImGui::SliderFloat("Parameter", &value, 0.0f, 1.0f)) {
        // This code runs when the slider value changes
        updateParameter(value);
    }

    ImGui::End();

    // Render ImGui
    ImGui::Render();
}

For a comprehensive guide to all available ImGui widgets and their options, please refer to the official ImGui documentation and demo: https://github.com/ocornut/imgui/blob/master/imgui_demo.cpp

GUI Design Considerations for Vulkan Applications

When designing a GUI for your Vulkan application, consider these aspects:

Memory Management

ImGui generates vertex and index buffers that need to be uploaded to the GPU. Ensure these resources are properly managed:

  1. Buffer Sizing: Allocate buffers with sufficient size or implement resizing logic

  2. Memory Types: Use host-visible memory for frequent updates

  3. Synchronization: Ensure buffer updates are synchronized with rendering

Command Buffer Integration

Integrate ImGui rendering commands with your Vulkan command buffers:

// Record commands for scene rendering
// ...

// Record ImGui rendering commands
imGuiUtil.drawFrame(commandBuffer);

// Submit command buffer
// ...

Descriptor Resources

ImGui requires descriptors for its font texture. Ensure your descriptor pool has sufficient capacity:

// Create descriptor pool with enough capacity for ImGui
vk::DescriptorPoolSize poolSizes[] = {
    { vk::DescriptorType::eCombinedImageSampler, 50 },
    // Other descriptor types...
};

vk::DescriptorPoolCreateInfo poolInfo{};
poolInfo.flags = vk::DescriptorPoolCreateFlagBits::eFreeDescriptorSet;
poolInfo.maxSets = 50;
poolInfo.poolSizeCount = static_cast<uint32_t>(std::size(poolSizes));
poolInfo.pPoolSizes = poolSizes;

descriptorPool = device.createDescriptorPool(poolInfo);

Performance Considerations

When integrating ImGui with Vulkan, consider these performance aspects:

  1. Command Buffer Recording: Record ImGui commands efficiently, ideally once per frame

  2. Descriptor Management: Minimize descriptor set allocations and updates

  3. Buffer Updates: Optimize vertex and index buffer updates

  4. Pipeline State: Use a dedicated pipeline for ImGui to minimize state changes

  5. Render Pass Integration: Consider whether to use a separate render pass or subpass for the GUI

Frames-in-Flight: Duplicate Dynamic Buffers Per Frame

If your renderer uses multiple frames in flight (e.g., double/triple buffering) without a device wait-idle between frames, ImGui’s dynamic vertex and index buffers must not be shared across frames. Otherwise, the CPU can overwrite data that the GPU from a previous frame is still reading.

  • Allocate one vertex buffer and one index buffer per frame-in-flight.

  • Update/bind the buffers for the current frame index only.

  • Size each buffer to the frame’s ImDrawData TotalVtxCount/TotalIdxCount, growing as needed.

Example sketch:

class ImGuiSystem {
  // ...
  std::vector<vk::raii::Buffer> vertexBuffers;
  std::vector<vk::raii::DeviceMemory> vertexMemories;
  std::vector<vk::raii::Buffer> indexBuffers;
  std::vector<vk::raii::DeviceMemory> indexMemories;
  std::vector<uint32_t> vertexCounts;
  std::vector<uint32_t> indexCounts;

  bool Initialize(Renderer* renderer, uint32_t w, uint32_t h) {
    // ... create pipelines, font, descriptors ...
    const uint32_t frames = renderer->GetMaxFramesInFlight();
    vertexBuffers.resize(frames);
    vertexMemories.resize(frames);
    indexBuffers.resize(frames);
    indexMemories.resize(frames);
    vertexCounts.assign(frames, 0);
    indexCounts.assign(frames, 0);
    return true;
  }

  void Render(vk::raii::CommandBuffer& cmd, uint32_t frameIndex) {
    ImGui::Render();
    updateBuffers(frameIndex);
    // bind per-frame buffers
    std::array vb = {*vertexBuffers[frameIndex]};
    std::array<vk::DeviceSize,1> offs{};
    cmd.bindVertexBuffers(0, vb, offs);
    cmd.bindIndexBuffer(*indexBuffers[frameIndex], 0, vk::IndexType::eUint16);
    // draw lists...
  }

  void updateBuffers(uint32_t frameIndex) {
    ImDrawData* dd = ImGui::GetDrawData();
    if (!dd || dd->CmdListsCount == 0) return;
    vk::DeviceSize vbytes = dd->TotalVtxCount * sizeof(ImDrawVert);
    vk::DeviceSize ibytes = dd->TotalIdxCount * sizeof(ImDrawIdx);
    // grow-per-frame if needed, then map/copy for this frame only
    // ...
  }
};

When integrating with your main renderer, pass the current frame index to the ImGui render call:

// inside your frame loop after scene rendering
imguiSystem->Render(commandBuffers[currentFrame], currentFrame);

Organizing Your GUI Code

For maintainable GUI code, consider these organizational patterns:

  1. Component-Based Approach: Split your GUI into logical components

  2. State Management: Use a centralized state store that the GUI can modify

  3. Event System: Implement an event system for GUI-triggered actions

  4. Lazy Updates: Only update Vulkan resources when GUI settings actually change

// Component-based approach example
class VulkanGUI {
private:
    // GUI state
    struct {
        bool showRenderSettings = true;
        bool showPerformance = true;
        bool showSceneControls = true;
    } state;

    // Components
    void drawRenderSettingsPanel();
    void drawPerformancePanel();
    void drawSceneControlsPanel();

public:
    void draw() {
        // Start a new ImGui frame
        ImGui::NewFrame();

        // Draw components based on state
        if (state.showRenderSettings) drawRenderSettingsPanel();
        if (state.showPerformance) drawPerformancePanel();
        if (state.showSceneControls) drawSceneControlsPanel();

        // Main menu for toggling panels
        if (ImGui::BeginMainMenuBar()) {
            if (ImGui::BeginMenu("View")) {
                ImGui::MenuItem("Render Settings", nullptr, &state.showRenderSettings);
                ImGui::MenuItem("Performance", nullptr, &state.showPerformance);
                ImGui::MenuItem("Scene Controls", nullptr, &state.showSceneControls);
                ImGui::EndMenu();
            }
            ImGui::EndMainMenuBar();
        }

        // Render ImGui
        ImGui::Render();
    }
};

Displaying Textures in ImGui

A common requirement in GUI systems is displaying textures, such as rendered scenes, material previews, or icons. ImGui provides the ability to display textures through its ImGui::Image and ImGui::ImageButton functions. To use these with Vulkan, you need to properly set up descriptor sets for your textures.

Setting Up Texture Descriptors

To display a Vulkan texture in ImGui, you need to:

  1. Create a descriptor set layout for the texture

  2. Allocate a descriptor set

  3. Update the descriptor set with your texture’s image view and sampler

  4. Pass the descriptor set handle to ImGui

Create the descriptor set layout

This layout declares a single combined image sampler the shader can sample from when ImGui draws the quad.

// Create a descriptor set layout for textures
vk::DescriptorSetLayoutBinding binding{};
binding.descriptorType = vk::DescriptorType::eCombinedImageSampler;
binding.descriptorCount = 1;
binding.stageFlags = vk::ShaderStageFlagBits::eFragment;
binding.binding = 0;

vk::DescriptorSetLayoutCreateInfo layoutInfo{};
layoutInfo.bindingCount = 1;
layoutInfo.pBindings = &binding;

vk::raii::DescriptorSetLayout textureSetLayout = device.createDescriptorSetLayout(layoutInfo);
Allocate a descriptor set

Allocate one set per texture you want to show in ImGui.

// Allocate a descriptor set for each texture
vk::DescriptorSetAllocateInfo allocInfo{};
allocInfo.descriptorPool = *descriptorPool;
allocInfo.descriptorSetCount = 1;
vk::DescriptorSetLayout layouts[] = {*textureSetLayout};
allocInfo.pSetLayouts = layouts;

vk::raii::DescriptorSet textureDescriptorSet = std::move(device.allocateDescriptorSets(allocInfo).front());
Update the descriptor set

Point the descriptor at your image view and sampler in shader‑read layout.

// Update the descriptor set with your texture
vk::DescriptorImageInfo imageInfo{};
imageInfo.imageLayout = vk::ImageLayout::eShaderReadOnlyOptimal;
imageInfo.imageView = textureImageView.getHandle();
imageInfo.sampler = *textureSampler;

vk::WriteDescriptorSet writeSet{};
writeSet.dstSet = *textureDescriptorSet;
writeSet.descriptorCount = 1;
writeSet.descriptorType = vk::DescriptorType::eCombinedImageSampler;
writeSet.pImageInfo = &imageInfo;
writeSet.dstBinding = 0;

device.updateDescriptorSets(1, &writeSet, 0, nullptr);

Use it in ImGui

Once you have set up the descriptor set, you can use it with ImGui’s image functions:

// Store the descriptor set as ImTextureID (which is just a void*)
ImTextureID textureId = (ImTextureID)(VkDescriptorSet)*textureDescriptorSet;

// Display the texture in ImGui
ImGui::Begin("Texture Viewer");

// Display as a simple image
ImGui::Image(textureId, ImVec2(width, height));

// Or as an image button
if (ImGui::ImageButton(textureId, ImVec2(width, height))) {
    // Handle button click
}

// You can also apply tinting and modify UV coordinates
ImGui::Image(textureId, ImVec2(width, height),
             ImVec2(0, 0), ImVec2(1, 1),  // UV coordinates (0,0) to (1,1) for the full texture
             ImVec4(1, 1, 1, 1),          // Tint color (white = no tint)
             ImVec4(1, 1, 1, 0.5));       // Border color

ImGui::End();

Complete Example: Texture Manager for ImGui

Here’s a more complete example of a texture manager class that handles multiple textures for ImGui:

class ImGuiTextureManager {
private:
    vk::raii::Device* device = nullptr;
    vk::raii::DescriptorPool* descriptorPool = nullptr;
    vk::raii::DescriptorSetLayout descriptorSetLayout{nullptr};

    struct TextureInfo {
        vk::raii::DescriptorSet descriptorSet{nullptr};
        uint32_t width;
        uint32_t height;
    };

    std::unordered_map<std::string, TextureInfo> textures;

public:
    ImGuiTextureManager(vk::raii::Device& device, vk::raii::DescriptorPool& descriptorPool)
        : device(&device), descriptorPool(&descriptorPool) {

        // Create descriptor set layout for textures
        vk::DescriptorSetLayoutBinding binding{};
        binding.descriptorType = vk::DescriptorType::eCombinedImageSampler;
        binding.descriptorCount = 1;
        binding.stageFlags = vk::ShaderStageFlagBits::eFragment;
        binding.binding = 0;

        vk::DescriptorSetLayoutCreateInfo layoutInfo{};
        layoutInfo.bindingCount = 1;
        layoutInfo.pBindings = &binding;

        descriptorSetLayout = device.createDescriptorSetLayout(layoutInfo);
    }

    // Register a texture for use with ImGui
    ImTextureID registerTexture(const std::string& name, vk::ImageView imageView,
                               vk::Sampler sampler, uint32_t width, uint32_t height) {

        // Allocate descriptor set
        vk::DescriptorSetAllocateInfo allocInfo{};
        allocInfo.descriptorPool = **descriptorPool;
        allocInfo.descriptorSetCount = 1;
        vk::DescriptorSetLayout layouts[] = {*descriptorSetLayout};
        allocInfo.pSetLayouts = layouts;

        vk::raii::DescriptorSet descriptorSet = std::move(device->allocateDescriptorSets(allocInfo).front());

        // Update descriptor set
        vk::DescriptorImageInfo imageInfo{};
        imageInfo.imageLayout = vk::ImageLayout::eShaderReadOnlyOptimal;
        imageInfo.imageView = imageView;
        imageInfo.sampler = sampler;

        vk::WriteDescriptorSet writeSet{};
        writeSet.dstSet = *descriptorSet;
        writeSet.descriptorCount = 1;
        writeSet.descriptorType = vk::DescriptorType::eCombinedImageSampler;
        writeSet.pImageInfo = &imageInfo;
        writeSet.dstBinding = 0;

        device->updateDescriptorSets(1, &writeSet, 0, nullptr);

        // Store texture info
        textures[name] = {std::move(descriptorSet), width, height};

        // Return the descriptor set as ImTextureID
        return (ImTextureID)(VkDescriptorSet)*textures[name].descriptorSet;
    }

    // Get a previously registered texture
    ImTextureID getTexture(const std::string& name) {
        if (textures.find(name) == textures.end()) {
            throw std::runtime_error("Texture not found: " + name);
        }

        return (ImTextureID)(VkDescriptorSet)*textures[name].descriptorSet;
    }

    // Get texture dimensions
    ImVec2 getTextureDimensions(const std::string& name) {
        if (textures.find(name) == textures.end()) {
            throw std::runtime_error("Texture not found: " + name);
        }

        return ImVec2(static_cast<float>(textures[name].width),
                     static_cast<float>(textures[name].height));
    }
};

Usage Example

Here’s how you might use the texture manager in your application:

// During initialization
ImGuiTextureManager textureManager(device, descriptorPool);

// Register textures (e.g., after loading a model or rendering to a texture)
ImTextureID albedoTexId = textureManager.registerTexture(
    "albedo",
    albedoImageView,
    textureSampler,
    albedoWidth,
    albedoHeight
);

ImTextureID normalMapId = textureManager.registerTexture(
    "normalMap",
    normalMapImageView,
    textureSampler,
    normalMapWidth,
    normalMapHeight
);

// In your GUI rendering code
void drawMaterialEditor() {
    ImGui::Begin("Material Editor");

    // Display textures
    ImGui::Text("Albedo Texture:");
    ImGui::Image(textureManager.getTexture("albedo"),
                ImVec2(200, 200));

    ImGui::Text("Normal Map:");
    ImGui::Image(textureManager.getTexture("normalMap"),
                ImVec2(200, 200));

    // Material properties
    static float roughness = 0.5f;
    if (ImGui::SliderFloat("Roughness", &roughness, 0.0f, 1.0f)) {
        updateMaterialProperty("roughness", roughness);
    }

    static float metallic = 0.0f;
    if (ImGui::SliderFloat("Metallic", &metallic, 0.0f, 1.0f)) {
        updateMaterialProperty("metallic", metallic);
    }

    ImGui::End();
}

Performance Considerations

When working with textures in ImGui, keep these performance considerations in mind:

  1. Descriptor Management: Create descriptor sets for textures only when needed and reuse them

  2. Texture Size: Consider using smaller preview versions of textures for the UI

  3. Mipmap Selection: For large textures, ensure proper mipmap selection to avoid aliasing

  4. Texture Updates: If a texture changes frequently, use a staging buffer for updates

  5. Texture Atlas: For many small textures (like icons), consider using a texture atlas

By properly managing textures in your ImGui integration, you can create rich interfaces that display rendered content, material previews, and other visual elements directly in your GUI.

Object Picking: Interacting with the 3D Scene

An important aspect of GUI integration is handling object picking - selecting 3D objects with the mouse. This requires coordination between ImGui and your 3D scene:

void handleMouseInput(float mouseX, float mouseY) {
    // First, check if ImGui is using this input
    ImGuiIO& io = ImGui::GetIO();
    if (io.WantCaptureMouse) {
        // ImGui is using the mouse, don't use it for 3D picking
        return;
    }

    // ImGui isn't using the mouse, so we can use it for 3D picking
    pickObject(mouseX, mouseY);
}

void pickObject(float mouseX, float mouseY) {
    // Convert screen coordinates to normalized device coordinates
    float ndcX = (2.0f * mouseX) / windowWidth - 1.0f;
    float ndcY = 1.0f - (2.0f * mouseY) / windowHeight;

    // Create a ray from the camera through the mouse position
    glm::vec4 clipCoords(ndcX, ndcY, -1.0f, 1.0f);
    glm::vec4 eyeCoords = glm::inverse(projectionMatrix) * clipCoords;
    eyeCoords = glm::vec4(eyeCoords.x, eyeCoords.y, -1.0f, 0.0f);

    glm::vec3 rayDirection = glm::normalize(glm::vec3(
        glm::inverse(viewMatrix) * eyeCoords
    ));

    glm::vec3 rayOrigin = camera.getPosition();

    // Test for intersections with scene objects
    float closestHit = std::numeric_limits<float>::max();
    int hitObjectId = -1;

    for (size_t i = 0; i < sceneObjects.size(); i++) {
        float hitDistance;
        if (rayIntersectsObject(rayOrigin, rayDirection, sceneObjects[i], hitDistance)) {
            if (hitDistance < closestHit) {
                closestHit = hitDistance;
                hitObjectId = static_cast<int>(i);
            }
        }
    }

    // If we hit an object, select it
    if (hitObjectId >= 0) {
        selectObject(hitObjectId);
    }
}

Implementing Ray-Object Intersection

For object picking to work, you need to implement ray-object intersection tests. Here’s a simple example for sphere intersection:

bool rayIntersectsSphere(
    const glm::vec3& rayOrigin,
    const glm::vec3& rayDirection,
    const glm::vec3& sphereCenter,
    float sphereRadius,
    float& outDistance
) {
    glm::vec3 oc = rayOrigin - sphereCenter;
    float a = glm::dot(rayDirection, rayDirection);
    float b = 2.0f * glm::dot(oc, rayDirection);
    float c = glm::dot(oc, oc) - sphereRadius * sphereRadius;
    float discriminant = b * b - 4 * a * c;

    if (discriminant < 0) {
        return false; // No intersection
    }

    // Calculate the closest intersection point
    float t = (-b - sqrt(discriminant)) / (2.0f * a);
    if (t < 0) {
        // Try the other intersection point
        t = (-b + sqrt(discriminant)) / (2.0f * a);
        if (t < 0) {
            return false; // Both intersection points are behind the ray
        }
    }

    outDistance = t;
    return true;
}

Visualizing Selected Objects

Once an object is selected, you can visualize the selection:

void drawScene(vk::raii::CommandBuffer& commandBuffer) {
    // Draw all objects
    for (size_t i = 0; i < sceneObjects.size(); i++) {
        // If this object is selected, use a different pipeline
        if (static_cast<int>(i) == selectedObjectId) {
            commandBuffer.bindPipeline(vk::PipelineBindPoint::eGraphics, *highlightPipeline);
        } else {
            commandBuffer.bindPipeline(vk::PipelineBindPoint::eGraphics, *standardPipeline);
        }

        // Draw the object
        drawObject(commandBuffer, sceneObjects[i]);
    }
}

Integrating Picking with ImGui

You can also display information about the selected object in the GUI:

void drawObjectPropertiesPanel() {
    if (selectedObjectId < 0) {
        return; // No object selected
    }

    ImGui::Begin("Object Properties");

    SceneObject& obj = sceneObjects[selectedObjectId];

    // Display object properties
    ImGui::Text("Object ID: %d", selectedObjectId);
    ImGui::Text("Name: %s", obj.name.c_str());

    // Edit object properties
    glm::vec3 position = obj.position;
    if (ImGui::DragFloat3("Position", &position[0], 0.1f)) {
        obj.position = position;
        updateObjectTransform(selectedObjectId);
    }

    glm::vec3 rotation = obj.rotation;
    if (ImGui::DragFloat3("Rotation", &rotation[0], 1.0f, -180.0f, 180.0f)) {
        obj.rotation = rotation;
        updateObjectTransform(selectedObjectId);
    }

    glm::vec3 scale = obj.scale;
    if (ImGui::DragFloat3("Scale", &scale[0], 0.1f, 0.1f, 10.0f)) {
        obj.scale = scale;
        updateObjectTransform(selectedObjectId);
    }

    ImGui::End();
}

Object picking creates a powerful interaction model where users can select and manipulate 3D objects directly, while using the GUI to fine-tune properties. This combination of direct manipulation and precise control provides an intuitive user experience.

Balancing GUI and 3D Interaction

When designing your application, consider how to balance GUI-based controls with direct 3D interaction:

  1. Use GUI for:

    • Precise numerical inputs

    • Complex settings with many options

    • Hierarchical data visualization

    • Application-wide controls

  2. Use 3D Interaction for:

    • Object placement and movement

    • Camera navigation

    • Direct manipulation of scene elements

    • Intuitive spatial operations

  3. Hybrid Approaches:

    • Gizmos for 3D transformation with precise control

    • Context menus that appear near selected objects

    • Property panels that update based on selection

By thoughtfully integrating ImGui with your Vulkan application and implementing object picking, you can create a powerful and intuitive user interface that combines the strengths of both 2D GUI controls and direct 3D interaction.

In the next section, we’ll explore more details about integrating the GUI rendering with the Vulkan rendering pipeline.

GUI: Vulkan Integration

Vulkan Integration

In this section, we’ll explore how to properly integrate ImGui rendering with the Vulkan rendering pipeline. While we’ve already covered the basic setup in the "Setting Up Dear ImGui" section, here we’ll dive deeper into the technical details of how ImGui works with Vulkan and how to optimize the integration.

Understanding the Rendering Flow

Before we dive into the implementation details, let’s understand how ImGui rendering fits into the Vulkan rendering pipeline:

  1. Prepare Frame: Begin a new frame in ImGui and create UI elements

  2. Generate Draw Data: ImGui generates vertex and index buffers for the UI

  3. Record Commands: Record Vulkan commands to render the ImGui draw data

  4. Submit Commands: Submit the commands to the Vulkan queue

  5. Present: Present the rendered frame to the screen

This flow needs to be integrated with your existing Vulkan rendering pipeline, which typically involves:

  1. Acquiring the next swap chain image

  2. Recording command buffers for scene rendering

  3. Submitting command buffers

  4. Presenting the rendered image

Dynamic Rendering Configuration

ImGui can be integrated with Vulkan’s dynamic rendering feature, which simplifies the rendering process by eliminating the need for explicit render passes and framebuffers:

// When initializing ImGui, we set up our custom Vulkan renderer with dynamic rendering
ImGuiVulkanRenderer renderer;
// ... configure the renderer ...
renderer.initialize(*device, *physicalDevice);

// Set up dynamic rendering info
vk::PipelineRenderingCreateInfo renderingInfo{};
renderingInfo.colorAttachmentCount = 1;
vk::Format formats[] = { vk::Format::eB8G8R8A8Unorm };
renderingInfo.pColorAttachmentFormats = formats;
renderer.setDynamicRenderingInfo(renderingInfo);

Dynamic rendering simplifies the integration by removing the dependency on render passes and framebuffers, making the code more flexible and easier to maintain.

Command Buffer Integration

There are two main approaches to integrating ImGui commands with your Vulkan command buffers:

  1. Single Command Buffer: Record both scene and ImGui rendering commands in the same command buffer

  2. Multiple Command Buffers: Use separate command buffers for scene and ImGui rendering

Let’s look at both approaches:

Single Command Buffer Approach

This is the simplest approach and works well for most applications. With dynamic rendering, the code becomes even cleaner:

Command Buffer Initialization

The frame rendering process begins with command buffer preparation, where we set up the recording state and prepare for GPU command submission.

void drawFrame() {
    // ... existing frame preparation code ...

    // Initialize command buffer recording
    // This tells Vulkan we're about to record a sequence of GPU commands
    vk::CommandBufferBeginInfo beginInfo{};
    commandBuffer.begin(beginInfo);

Command buffer recording represents the heart of Vulkan’s explicit GPU control model. Unlike older APIs where rendering commands are immediately submitted to the GPU, Vulkan allows us to build up a complete sequence of operations before submission. This approach enables powerful optimizations like command reordering, parallel command buffer construction, and efficient GPU scheduling.

The 'begin' operation transitions the command buffer from an initial state into a recording state, where subsequent API calls will be captured as GPU instructions rather than executed immediately. This explicit state management gives us precise control over when and how GPU work is submitted, enabling the fine-grained performance control that makes Vulkan so powerful for demanding applications.

Dynamic Rendering Attachment Setup

Dynamic rendering requires us to explicitly describe our render targets and their properties, replacing the traditional render pass system with a more flexible approach.

    // Configure color attachment for the main render target
    // This describes how the GPU should handle the color output
    vk::RenderingAttachmentInfo colorAttachment{};
    colorAttachment.imageView = *swapChainImageViews[imageIndex];       // Target swapchain image
    colorAttachment.imageLayout = vk::ImageLayout::eColorAttachmentOptimal;  // Optimal layout for color output
    colorAttachment.loadOp = vk::AttachmentLoadOp::eClear;              // Clear the image before rendering
    colorAttachment.storeOp = vk::AttachmentStoreOp::eStore;            // Preserve results after rendering
    colorAttachment.clearValue.color = std::array<float, 4>{0.0f, 0.0f, 0.0f, 1.0f};  // Clear to black

    // Configure depth attachment for 3D depth testing
    // This enables proper occlusion and depth sorting for 3D objects
    vk::RenderingAttachmentInfo depthAttachment{};
    depthAttachment.imageView = *depthImageView;                        // Depth buffer image
    depthAttachment.imageLayout = vk::ImageLayout::eDepthStencilAttachmentOptimal;  // Optimal for depth operations
    depthAttachment.loadOp = vk::AttachmentLoadOp::eClear;              // Clear depth buffer to far plane
    depthAttachment.storeOp = vk::AttachmentStoreOp::eDontCare;         // Don't preserve depth after rendering
    depthAttachment.clearValue.depthStencil = vk::ClearDepthStencilValue{1.0f, 0};  // Clear to maximum depth

The attachment configuration system provides explicit control over how the GPU handles our render targets throughout the rendering process. By specifying load and store operations, we can optimize memory bandwidth by only preserving data that needs to carry forward to subsequent passes. The clear operations ensure we start with a known state, preventing visual artifacts from previous frame data.

Image layout transitions happen automatically based on our specifications, with the GPU driver handling the necessary memory barriers and cache flushes to ensure data coherency. The optimal layouts we specify here tell the driver to arrange the image data in whatever format provides the best performance for the intended usage, rather than forcing a specific memory organization.

Dynamic Rendering Pass Setup

With our attachments configured, we now assemble them into a complete rendering pass that describes the full rendering operation to the GPU.

    // Assemble the complete rendering operation description
    // This ties together all our attachments and rendering parameters
    vk::RenderingInfo renderingInfo{};
    renderingInfo.renderArea = vk::Rect2D{{0, 0}, swapChainExtent};     // Render to entire swapchain area
    renderingInfo.layerCount = 1;                                       // Single layer (not array rendering)
    renderingInfo.colorAttachmentCount = 1;                             // One color output
    renderingInfo.pColorAttachments = &colorAttachment;                 // Our configured color attachment
    renderingInfo.pDepthAttachment = &depthAttachment;                  // Our configured depth attachment

    // Begin the dynamic rendering pass
    // This establishes the rendering context for subsequent draw commands
    commandBuffer.beginRendering(renderingInfo);

Dynamic rendering represents a significant evolution from traditional Vulkan render passes, providing greater flexibility while maintaining the performance benefits of explicit GPU control. Instead of pre-defining render pass objects at initialization time, we can specify render targets and their properties at command recording time, enabling more dynamic and flexible rendering architectures.

The render area specification allows for partial-screen rendering, which can provide significant performance benefits when only portions of the screen need updating. For full-screen rendering like our case, we specify the entire swapchain extent to ensure complete coverage.

3D Scene Rendering

The main scene rendering phase handles all 3D geometry, lighting, and material rendering within the established rendering context.

    // Execute 3D scene rendering
    // All your existing 3D geometry, lighting, and material rendering happens here
    // ... your existing scene rendering code ...

    // Complete the 3D rendering pass
    // This finalizes all 3D rendering operations and prepares for UI overlay
    commandBuffer.endRendering();

The scene rendering phase operates within the rendering context we established, with the GPU automatically handling depth testing, color blending, and other rasterization operations according to our pipeline configurations. All draw commands issued between beginRendering and endRendering will target our configured attachments with the specified clear and store behaviors.

The explicit endRendering call ensures that all scene rendering operations are properly completed and that render targets are transitioned to appropriate states for subsequent operations. This explicit control allows the GPU driver to perform optimal scheduling and memory management for the rendering workload.

UI Overlay Integration

The final rendering phase integrates ImGui UI elements as an overlay on top of the 3D scene, requiring careful coordination between the two rendering systems.

    // Render ImGui UI overlay on top of the 3D scene
    // The custom renderer handles ImGui's own dynamic rendering setup internally
    // This includes vertex buffer uploads, pipeline binding, and draw command generation
    renderer.render(ImGui::GetDrawData(), commandBuffer);

    // Finalize command buffer recording
    // This transitions the command buffer to executable state for GPU submission
    commandBuffer.end();

    // Submit command buffer
    // ... your existing submission code ...
}

Multiple Command Buffers Approach

This approach gives you more flexibility and can be useful for more complex rendering pipelines. With dynamic rendering, it becomes even more straightforward:

Multi-Buffer: Scene Command Buffer Recording

The multiple command buffer approach begins by isolating 3D scene rendering into its own dedicated command buffer, providing greater flexibility for complex rendering pipelines.

void drawFrame() {
    // ... existing frame preparation code ...

    // Initialize scene-specific command buffer recording
    // This dedicated buffer will contain only 3D geometry and lighting operations
    vk::CommandBufferBeginInfo beginInfo{};
    sceneCommandBuffer.begin(beginInfo);

Separating scene rendering into its own command buffer provides several architectural advantages. First, it enables parallel command buffer recording where different threads can simultaneously build scene and UI command sequences, improving CPU utilization on multi-core systems. Second, it allows for independent optimization of each rendering phase, where scene rendering can use different GPU queues or submission timing than UI rendering.

This separation also facilitates advanced rendering techniques like multi-frame latency optimization, where scene rendering can be decoupled from UI updates to maintain consistent frame timing even when one system experiences performance variations.

Multi-Buffer: Scene Attachment Configuration

The scene rendering setup mirrors the single-buffer approach but with explicit ownership of the attachment configuration within the scene command buffer.

    // Configure scene rendering attachments with explicit ownership
    // These configurations belong specifically to the scene rendering pass
    vk::RenderingAttachmentInfo colorAttachment{};
    colorAttachment.imageView = *swapChainImageViews[imageIndex];        // Target swapchain image
    colorAttachment.imageLayout = vk::ImageLayout::eColorAttachmentOptimal;  // Optimal for color rendering
    colorAttachment.loadOp = vk::AttachmentLoadOp::eClear;               // Clear for fresh scene start
    colorAttachment.storeOp = vk::AttachmentStoreOp::eStore;             // Preserve for UI overlay
    colorAttachment.clearValue.color = std::array<float, 4>{0.0f, 0.0f, 0.0f, 1.0f};  // Clear to black

    // Configure depth attachment for 3D scene depth testing
    // UI rendering won't need depth testing, so this is scene-specific
    vk::RenderingAttachmentInfo depthAttachment{};
    depthAttachment.imageView = *depthImageView;                         // Scene depth buffer
    depthAttachment.imageLayout = vk::ImageLayout::eDepthStencilAttachmentOptimal;  // Optimal for depth ops
    depthAttachment.loadOp = vk::AttachmentLoadOp::eClear;               // Clear depth for new frame
    depthAttachment.storeOp = vk::AttachmentStoreOp::eDontCare;          // UI doesn't need depth data
    depthAttachment.clearValue.depthStencil = vk::ClearDepthStencilValue{1.0f, 0};  // Clear to far plane

The attachment configuration for scene rendering emphasizes the separation of concerns between 3D and UI rendering. The store operation for the color attachment ensures that scene rendering results are preserved for the subsequent UI overlay, while the depth attachment uses "don’t care" storage since UI elements typically render without depth testing.

This explicit configuration makes the rendering dependencies clear and helps optimize memory bandwidth by only preserving the data that subsequent passes actually need.

Multi-Buffer: Scene Rendering Execution

The scene rendering execution occurs within its dedicated command buffer, providing isolated control over 3D rendering operations.

    // Assemble scene rendering configuration
    // This defines the complete 3D rendering context
    vk::RenderingInfo renderingInfo{};
    renderingInfo.renderArea = vk::Rect2D{{0, 0}, swapChainExtent};      // Full screen rendering
    renderingInfo.layerCount = 1;                                        // Single rendering layer
    renderingInfo.colorAttachmentCount = 1;                              // One color output
    renderingInfo.pColorAttachments = &colorAttachment;                  // Scene color configuration
    renderingInfo.pDepthAttachment = &depthAttachment;                   // Scene depth configuration

    // Execute complete 3D scene rendering pass
    sceneCommandBuffer.beginRendering(renderingInfo);
    // All 3D geometry, lighting, materials, and effects render here
    // ... your existing scene rendering code ...
    sceneCommandBuffer.endRendering();

    // Finalize scene command buffer for submission
    sceneCommandBuffer.end();

The scene rendering execution benefits from having its own isolated command buffer context, where all GPU state changes and draw calls are contained within a clearly defined scope. This isolation makes debugging easier, as scene-specific rendering issues can be analyzed independently of UI rendering complexity.

Command buffer finalization with end() transitions the buffer to an executable state, ready for GPU submission, while maintaining clear boundaries between different rendering responsibilities.

Multi-Buffer: UI Command Buffer Setup

The UI rendering phase begins with its own command buffer recording, configured specifically for overlay rendering requirements.

    // Initialize UI-specific command buffer recording
    // This dedicated buffer handles only UI overlay operations
    imguiCommandBuffer.begin(beginInfo);

    // Configure UI attachment to preserve scene rendering results
    // This is the key difference from scene rendering - we load existing content
    colorAttachment.loadOp = vk::AttachmentLoadOp::eLoad;                // Preserve scene rendering

    // Ensure proper ordering/visibility between scene and UI when using multiple command buffers.
    // If you submit scene and UI command buffers separately, synchronize them either by:
    // - Submitting both on the same queue with a semaphore (scene signals, UI waits with stage = COLOR_ATTACHMENT_OUTPUT), or
    // - Recording a pipeline barrier in the UI command buffer before beginRendering() to make scene color writes visible.
    // Example barrier inserted in the UI command buffer:
    {
        vk::ImageMemoryBarrier2 barrier{
            .srcStageMask = vk::PipelineStageFlagBits2::eColorAttachmentOutput,
            .srcAccessMask = vk::AccessFlagBits2::eColorAttachmentWrite,
            .dstStageMask = vk::PipelineStageFlagBits2::eColorAttachmentOutput,
            .dstAccessMask = vk::AccessFlagBits2::eColorAttachmentRead | vk::AccessFlagBits2::eColorAttachmentWrite,
            .oldLayout = vk::ImageLayout::eColorAttachmentOptimal,
            .newLayout = vk::ImageLayout::eColorAttachmentOptimal,
            .image = *swapChainImages[imageIndex],
            .subresourceRange = { vk::ImageAspectFlagBits::eColor, 0, 1, 0, 1 }
        };
        vk::DependencyInfo depInfo{ .imageMemoryBarrierCount = 1, .pImageMemoryBarriers = &barrier };
        imguiCommandBuffer.pipelineBarrier2(depInfo);
    }

    // UI rendering typically doesn't need depth testing
    // Remove depth attachment to optimize UI rendering performance
    renderingInfo.pDepthAttachment = nullptr;

The UI command buffer setup demonstrates the power of the multi-buffer approach through its different attachment configuration. By changing the load operation to eLoad, we preserve the scene rendering results as the foundation for UI overlay rendering. This approach is more explicit and controllable than relying on automatic render pass dependencies.

Removing the depth attachment for UI rendering eliminates unnecessary depth testing overhead, since UI elements typically render in screen space without complex occlusion relationships. This optimization can provide measurable performance improvements, especially on mobile GPUs where bandwidth is at a premium.

Multi-Buffer: UI Rendering and Submission Coordination

The final phase handles UI rendering execution and coordinates the submission of both command buffers in the correct order.

    // Execute UI overlay rendering
    // The custom renderer handles ImGui's dynamic rendering internally
    renderer.render(ImGui::GetDrawData(), imguiCommandBuffer);

    // Finalize UI command buffer
    imguiCommandBuffer.end();

    // Coordinate submission of both command buffers in dependency order
    // Scene must complete before UI to ensure proper overlay rendering
    std::array<vk::CommandBuffer, 2> submitCommandBuffers = {
        *sceneCommandBuffer,     // Execute scene rendering first
        *imguiCommandBuffer      // Then execute UI overlay
    };

    // Configure batch submission for optimal GPU utilization
    vk::SubmitInfo submitInfo{};
    submitInfo.commandBufferCount = static_cast<uint32_t>(submitCommandBuffers.size());
    submitInfo.pCommandBuffers = submitCommandBuffers.data();

    // Submit both command buffers as a cohesive frame
    // ... rest of your submission code ...
}

Handling Multiple Viewports

ImGui supports multiple viewports, which allows UI windows to be detached from the main window. To support this feature, we need to handle additional steps:

// In your main loop, after rendering ImGui
if (ImGui::GetIO().ConfigFlags & ImGuiConfigFlags_ViewportsEnable) {
    ImGui::UpdatePlatformWindows();
    ImGui::RenderPlatformWindowsDefault();
}

This will render any detached ImGui windows. Note that this feature requires additional platform-specific code and may not be necessary for all applications.

Handling Window Resize

When the window is resized, you need to recreate the swap chain and update ImGui:

void recreateSwapChain() {
    // ... existing swap chain recreation code ...

    // Update ImGui display size
    ImGuiIO& io = ImGui::GetIO();
    io.DisplaySize = ImVec2(static_cast<float>(swapChainExtent.width),
                           static_cast<float>(swapChainExtent.height));
}

Performance Considerations

Here are some tips to optimize ImGui rendering performance in Vulkan:

  1. Minimize State Changes: Try to render all ImGui elements in a single pass to minimize state changes.

  2. Use Appropriate Descriptor Pool Sizes: Allocate enough descriptors for ImGui to avoid running out of descriptors.

  3. Consider Secondary Command Buffers: For complex UIs, consider using secondary command buffers to record ImGui commands in parallel.

  4. Optimize UI Updates: Only update UI elements that change, and consider using ImGui’s Begin() function with the ImGuiWindowFlags_NoDecoration flag for static UI elements.

  5. Use ImGui’s Memory Allocators: ImGui allows you to provide custom memory allocators, which can be useful for controlling memory usage.

Complete Integration Example

Let’s put everything together in a complete example that integrates ImGui with a Vulkan application:

class VulkanApplication {
private:
    // ... existing Vulkan members ...

    // ImGui-specific members
    vk::raii::DescriptorPool imguiPool = nullptr;
    bool showDemoWindow = true;
    bool showMetricsWindow = false;

public:
    void initVulkan() {
        // ... existing Vulkan initialization ...

        // Initialize ImGui
        createImGuiDescriptorPool();
        initImGui();
    }

    void createImGuiDescriptorPool() {
        // ImGui typically needs a handful of descriptors (font texture + user UI textures).
        // Adjust these values to your app's needs (e.g., expected number of UI textures, buffers).
        // As a starting point:
        vk::DescriptorPoolSize poolSizes[] =
        {
            { vk::DescriptorType::eSampler, 8 },
            { vk::DescriptorType::eCombinedImageSampler, 128 },   // font + user-provided textures
            { vk::DescriptorType::eSampledImage, 128 },
            { vk::DescriptorType::eStorageImage, 8 },
            { vk::DescriptorType::eUniformTexelBuffer, 8 },
            { vk::DescriptorType::eStorageTexelBuffer, 8 },
            { vk::DescriptorType::eUniformBuffer, 32 },
            { vk::DescriptorType::eStorageBuffer, 32 },
            { vk::DescriptorType::eUniformBufferDynamic, 16 },
            { vk::DescriptorType::eStorageBufferDynamic, 16 },
            { vk::DescriptorType::eInputAttachment, 8 }
        };

        // A conservative maxSets equals the sum of descriptor counts.
        uint32_t maxSets = 0;
        for (const auto& ps : poolSizes) maxSets += ps.descriptorCount;

        vk::DescriptorPoolCreateInfo poolInfo{
            .flags = vk::DescriptorPoolCreateFlagBits::eFreeDescriptorSet,
            .maxSets = maxSets,
            .poolSizeCount = static_cast<uint32_t>(std::size(poolSizes)),
            .pPoolSizes = poolSizes
        };

        imguiPool = vk::raii::DescriptorPool(device, poolInfo);
    }

    void initImGui() {
        // Initialize ImGui context
        IMGUI_CHECKVERSION();
        ImGui::CreateContext();
        ImGuiIO& io = ImGui::GetIO();
        io.ConfigFlags |= ImGuiConfigFlags_NavEnableKeyboard;
        io.ConfigFlags |= ImGuiConfigFlags_DockingEnable;

        // Set up ImGui style
        ImGui::StyleColorsDark();

        // Initialize our custom backend
        int width = static_cast<int>(swapChainExtent.width);
        int height = static_cast<int>(swapChainExtent.height);
        ImGuiPlatform::Init(width, height);

        // Initialize our custom ImGui Vulkan renderer with dynamic rendering
        ImGuiVulkanRenderer renderer;
        renderer.initialize(
            *instance,
            *physicalDevice,
            *device,
            graphicsFamily,
            *graphicsQueue,
            *imguiPool,
            static_cast<uint32_t>(swapChainImages.size()),
            vk::SampleCountFlagBits::e1
        );

        // Set up dynamic rendering info
        vk::PipelineRenderingCreateInfo renderingInfo{};
        renderingInfo.colorAttachmentCount = 1;
        vk::Format formats[] = { swapChainImageFormat };
        renderingInfo.pColorAttachmentFormats = formats;
        renderer.setDynamicRenderingInfo(renderingInfo);

        // Note: With modern ImGui (v1.92+), we no longer need to manually upload fonts here.
        // The renderer will handle font atlas updates automatically during the draw phase
        // using the RendererHasTextures protocol.
    }

    void drawFrame() {
        // ... existing frame preparation code ...

        // Start the ImGui frame
        ImGui::NewFrame();

        // Create ImGui UI
        createImGuiUI();

        // Render ImGui
        ImGui::Render();

        // ... existing command buffer recording code ...

        // Begin dynamic rendering for scene
        vk::RenderingAttachmentInfo colorAttachment{};
        colorAttachment.imageView = *swapChainImageViews[imageIndex];
        colorAttachment.imageLayout = vk::ImageLayout::eColorAttachmentOptimal;
        colorAttachment.loadOp = vk::AttachmentLoadOp::eClear;
        colorAttachment.storeOp = vk::AttachmentStoreOp::eStore;
        colorAttachment.clearValue.color = std::array<float, 4>{0.0f, 0.0f, 0.0f, 1.0f};

        vk::RenderingAttachmentInfo depthAttachment{};
        depthAttachment.imageView = *depthImageView;
        depthAttachment.imageLayout = vk::ImageLayout::eDepthStencilAttachmentOptimal;
        depthAttachment.loadOp = vk::AttachmentLoadOp::eClear;
        depthAttachment.storeOp = vk::AttachmentStoreOp::eDontCare;
        depthAttachment.clearValue.depthStencil = vk::ClearDepthStencilValue{1.0f, 0};

        vk::RenderingInfo renderingInfo{};
        renderingInfo.renderArea = vk::Rect2D{{0, 0}, swapChainExtent};
        renderingInfo.layerCount = 1;
        renderingInfo.colorAttachmentCount = 1;
        renderingInfo.pColorAttachments = &colorAttachment;
        renderingInfo.pDepthAttachment = &depthAttachment;

        commandBuffer.beginRendering(renderingInfo);

        // Render 3D scene
        // ... your existing scene rendering code ...

        commandBuffer.endRendering();

        // Render ImGui using our custom renderer
        // ImGui will handle its own dynamic rendering internally
        renderer.render(ImGui::GetDrawData(), commandBuffer);

        // ... existing command buffer submission code ...
    }

    void createImGuiUI() {
        // Menu bar
        if (ImGui::BeginMainMenuBar()) {
            if (ImGui::BeginMenu("File")) {
                if (ImGui::MenuItem("Exit", "Alt+F4")) {
                    // Generic way to request application exit
                    requestApplicationExit();
                }
                ImGui::EndMenu();
            }

            if (ImGui::BeginMenu("View")) {
                ImGui::MenuItem("Demo Window", nullptr, &showDemoWindow);
                ImGui::MenuItem("Metrics", nullptr, &showMetricsWindow);
                ImGui::EndMenu();
            }

            ImGui::EndMainMenuBar();
        }

        // Demo window
        if (showDemoWindow) {
            ImGui::ShowDemoWindow(&showDemoWindow);
        }

        // Metrics window
        if (showMetricsWindow) {
            ImGui::ShowMetricsWindow(&showMetricsWindow);
        }

        // Custom windows
        ImGui::Begin("Settings");

        static float color[3] = { 0.5f, 0.5f, 0.5f };
        if (ImGui::ColorEdit3("Clear Color", color)) {
            // Update clear color
            clearColor = { color[0], color[1], color[2], 1.0f };
        }

        static int selectedModel = 0;
        const char* models[] = { "Cube", "Sphere", "Teapot", "Custom Model" };
        if (ImGui::Combo("Model", &selectedModel, models, IM_ARRAYSIZE(models))) {
            // Change model
            loadModel(models[selectedModel]);
        }

        ImGui::End();
    }

    void cleanup() {
        // ... existing cleanup code ...

        // Cleanup ImGui
        renderer.cleanup();
        ImGuiPlatform::Shutdown();  // Our custom platform backend
        ImGui::DestroyContext();
    }
};

Advanced Topics

Custom Shaders for ImGui

ImGui uses its own shaders for rendering, but you can customize them if needed:

// Create custom shader modules
vk::raii::ShaderModule customVertShaderModule = createShaderModule("custom_imgui_vert.spv");
vk::raii::ShaderModule customFragShaderModule = createShaderModule("custom_imgui_frag.spv");

// Initialize our custom renderer with custom shaders and dynamic rendering
ImGuiVulkanRenderer renderer;
renderer.initialize(
    *instance,
    *physicalDevice,
    *device,
    queueFamily,
    *queue,
    *descriptorPool,
    minImageCount,
    imageCount,
    vk::SampleCountFlagBits::e1
);

// Set up dynamic rendering info
vk::PipelineRenderingCreateInfo renderingInfo{};
renderingInfo.colorAttachmentCount = 1;
vk::Format formats[] = { swapChainImageFormat };
renderingInfo.pColorAttachmentFormats = formats;
renderer.setDynamicRenderingInfo(renderingInfo);

// Set custom shaders
renderer.setCustomShaders(
    customVertShaderModule,
    customFragShaderModule
);

Rendering ImGui to a Texture

You can render ImGui to a texture instead of directly to the screen, which can be useful for creating in-game UI elements:

// Create a texture to render ImGui to
vk::raii::Image imguiTargetImage = createImage(
    width, height,
    vk::Format::eR8G8B8A8Unorm,
    vk::ImageTiling::eOptimal,
    vk::ImageUsageFlagBits::eColorAttachment | vk::ImageUsageFlagBits::eSampled
);

// Create image view
vk::raii::ImageView imguiTargetImageView = createImageView(
    imguiTargetImage,
    vk::Format::eR8G8B8A8Unorm,
    vk::ImageAspectFlagBits::eColor
);

// Render ImGui to the texture using dynamic rendering
vk::RenderingAttachmentInfo colorAttachment{};
colorAttachment.imageView = *imguiTargetImageView;
colorAttachment.imageLayout = vk::ImageLayout::eColorAttachmentOptimal;
colorAttachment.loadOp = vk::AttachmentLoadOp::eClear;
colorAttachment.storeOp = vk::AttachmentStoreOp::eStore;
colorAttachment.clearValue.color = std::array<float, 4>{0.0f, 0.0f, 0.0f, 0.0f};

vk::RenderingInfo renderingInfo{};
renderingInfo.renderArea = vk::Rect2D{{0, 0}, {width, height}};
renderingInfo.layerCount = 1;
renderingInfo.colorAttachmentCount = 1;
renderingInfo.pColorAttachments = &colorAttachment;

commandBuffer.beginRendering(renderingInfo);
renderer.render(ImGui::GetDrawData(), commandBuffer);
commandBuffer.endRendering();

// Later, use the texture in your 3D scene
// ...

Handling High DPI Displays

For high DPI displays, you need to handle scaling correctly across different platforms:

// Cross-platform display scaling
void updateDisplayScale(int width, int height, float scaleX, float scaleY) {
    ImGuiIO& io = ImGui::GetIO();
    io.DisplaySize = ImVec2(static_cast<float>(width), static_cast<float>(height));
    io.DisplayFramebufferScale = ImVec2(scaleX, scaleY);

    // Update our platform backend
    ImGuiPlatform::SetDisplaySize(width, height);
}

// Platform-specific implementations
// Here's an example using GLFW, but you can implement similar functions
// for any windowing library you choose to use

void updateDisplayScaleWithGLFW(GLFWwindow* window) {
    // Get the framebuffer size (which may differ from window size on high DPI displays)
    int width, height;
    glfwGetFramebufferSize(window, &width, &height);

    // Get the content scale (DPI scaling factor)
    float xscale, yscale;
    glfwGetWindowContentScale(window, &xscale, &yscale);

    // Update ImGui with the correct display size and scale
    updateDisplayScale(width, height, xscale, yscale);
}

// With other windowing libraries, you would use their equivalent APIs
// to get the framebuffer size and DPI scaling factor

ImGui Utility Class

To encapsulate all the ImGui functionality in a way that works across different platforms, let’s create a utility class similar to the one mentioned in the Vulkan-Samples repository:

// ImGuiUtil.h
#pragma once

import vulkan_hpp;
#include <imgui.h>
#include <functional>
#include <memory>

class ImGuiUtil {
public:
    // Initialize ImGui with Vulkan using dynamic rendering
    static void Init(
        vk::raii::Instance& instance,
        vk::raii::PhysicalDevice& physicalDevice,
        vk::raii::Device& device,
        uint32_t queueFamily,
        vk::raii::Queue& queue,
        uint32_t minImageCount,
        uint32_t imageCount,
        vk::Format swapChainImageFormat,
        vk::SampleCountFlagBits msaaSamples = vk::SampleCountFlagBits::e1
    );

    // Shutdown ImGui
    static void Shutdown();

    // Start a new frame
    static void NewFrame();

    // Render ImGui draw data to a command buffer
    static void Render(vk::raii::CommandBuffer& commandBuffer);

    // Update display size
    static void UpdateDisplaySize(int width, int height, float scaleX = 1.0f, float scaleY = 1.0f);

    // Process platform-specific input event
    static bool ProcessInputEvent(void* event);

    // Set input callback
    static void SetInputCallback(std::function<void(ImGuiIO&)> callback);

private:
    // Create descriptor pool for ImGui
    static void createDescriptorPool();

    // Upload fonts
    static void uploadFonts();

    // Begin single-time commands
    static vk::raii::CommandBuffer beginSingleTimeCommands();

    // End single-time commands
    static void endSingleTimeCommands(vk::raii::CommandBuffer& commandBuffer);

    // Vulkan objects - using inline static initialization (C++17)
    inline static vk::raii::Instance* instance = nullptr;
    inline static vk::raii::PhysicalDevice* physicalDevice = nullptr;
    inline static vk::raii::Device* device = nullptr;
    inline static uint32_t queueFamily = 0;
    inline static vk::raii::Queue* queue = nullptr;
    inline static vk::raii::DescriptorPool descriptorPool = nullptr;
    inline static vk::raii::CommandPool commandPool = nullptr;
    inline static vk::PipelineRenderingCreateInfo renderingInfo{};

    // Input callback
    inline static std::function<void(ImGuiIO&)> inputCallback = nullptr;

    // Initialization state
    inline static bool initialized = false;
};

// ImGuiUtil.cpp
#include "ImGuiUtil.h"

void ImGuiUtil::Init(
    vk::raii::Instance& instance,
    vk::raii::PhysicalDevice& physicalDevice,
    vk::raii::Device& device,
    uint32_t queueFamily,
    vk::raii::Queue& queue,
    uint32_t minImageCount,
    uint32_t imageCount,
    vk::Format swapChainImageFormat,
    vk::SampleCountFlagBits msaaSamples
) {
    ImGuiUtil::instance = &instance;
    ImGuiUtil::physicalDevice = &physicalDevice;
    ImGuiUtil::device = &device;
    ImGuiUtil::queueFamily = queueFamily;
    ImGuiUtil::queue = &queue;

    // Set up dynamic rendering info
    renderingInfo.colorAttachmentCount = 1;
    vk::Format formats[] = { swapChainImageFormat };
    renderingInfo.pColorAttachmentFormats = formats;

    // Create command pool for font upload
    vk::CommandPoolCreateInfo poolInfo{
        .flags = vk::CommandPoolCreateFlagBits::eTransient,
        .queueFamilyIndex = queueFamily
    };
    commandPool = vk::raii::CommandPool(device, poolInfo);

    // Create descriptor pool
    createDescriptorPool();

    // Initialize ImGui context
    IMGUI_CHECKVERSION();
    ImGui::CreateContext();
    ImGuiIO& io = ImGui::GetIO();
    io.ConfigFlags |= ImGuiConfigFlags_NavEnableKeyboard;
    io.ConfigFlags |= ImGuiConfigFlags_DockingEnable;

    // Set up ImGui style
    ImGui::StyleColorsDark();

    // Initialize our custom Vulkan renderer with dynamic rendering
    renderer = ImGuiVulkanRenderer();
    renderer.initialize(
        *instance,
        *physicalDevice,
        *device,
        queueFamily,
        *queue,
        *descriptorPool,
        minImageCount,
        imageCount,
        msaaSamples
    );

    // Set dynamic rendering info
    renderer.setDynamicRenderingInfo(renderingInfo);

    // Upload fonts
    uploadFonts();

    initialized = true;
}

void ImGuiUtil::Shutdown() {
    if (!initialized) return;

    // Wait for device to finish operations
    device->waitIdle();

    // Cleanup ImGui
    renderer.cleanup();
    ImGui::DestroyContext();

    // Cleanup Vulkan resources
    commandPool = nullptr;
    descriptorPool = nullptr;

    // Reset pointers
    instance = nullptr;
    physicalDevice = nullptr;
    device = nullptr;
    queue = nullptr;

    initialized = false;
}

void ImGuiUtil::NewFrame() {
    if (!initialized) return;

    // Update ImGui IO with platform-specific input
    ImGuiIO& io = ImGui::GetIO();

    // Call input callback if registered
    if (inputCallback) {
        inputCallback(io);
    }

    ImGui::NewFrame();
}

void ImGuiUtil::Render(vk::raii::CommandBuffer& commandBuffer) {
    if (!initialized) return;

    ImGui::Render();
    renderer.render(ImGui::GetDrawData(), commandBuffer);
}

void ImGuiUtil::UpdateDisplaySize(int width, int height, float scaleX, float scaleY) {
    if (!initialized) return;

    ImGuiIO& io = ImGui::GetIO();
    io.DisplaySize = ImVec2(static_cast<float>(width), static_cast<float>(height));
    io.DisplayFramebufferScale = ImVec2(scaleX, scaleY);
}

bool ImGuiUtil::ProcessInputEvent(void* event) {
    // Platform-specific event processing would go here
    // This is a placeholder for the actual implementation
    return false;
}

void ImGuiUtil::SetInputCallback(std::function<void(ImGuiIO&)> callback) {
    inputCallback = callback;
}

void ImGuiUtil::createDescriptorPool() {
    // Tune these to match your expected number of UI textures and buffers.
    vk::DescriptorPoolSize poolSizes[] =
    {
        { vk::DescriptorType::eSampler, 8 },
        { vk::DescriptorType::eCombinedImageSampler, 128 },
        { vk::DescriptorType::eSampledImage, 128 },
        { vk::DescriptorType::eStorageImage, 8 },
        { vk::DescriptorType::eUniformTexelBuffer, 8 },
        { vk::DescriptorType::eStorageTexelBuffer, 8 },
        { vk::DescriptorType::eUniformBuffer, 32 },
        { vk::DescriptorType::eStorageBuffer, 32 },
        { vk::DescriptorType::eUniformBufferDynamic, 16 },
        { vk::DescriptorType::eStorageBufferDynamic, 16 },
        { vk::DescriptorType::eInputAttachment, 8 }
    };

    uint32_t maxSets = 0;
    for (const auto& ps : poolSizes) maxSets += ps.descriptorCount;

    vk::DescriptorPoolCreateInfo poolInfo{
        .flags = vk::DescriptorPoolCreateFlagBits::eFreeDescriptorSet,
        .maxSets = maxSets,
        .poolSizeCount = static_cast<uint32_t>(std::size(poolSizes)),
        .pPoolSizes = poolSizes
    };

    descriptorPool = vk::raii::DescriptorPool(*device, poolInfo);
}

void ImGuiUtil::uploadFonts() {
    vk::raii::CommandBuffer commandBuffer = beginSingleTimeCommands();
    renderer.uploadFonts(commandBuffer);
    endSingleTimeCommands(commandBuffer);
}

vk::raii::CommandBuffer ImGuiUtil::beginSingleTimeCommands() {
    vk::CommandBufferAllocateInfo allocInfo{
        .commandPool = *commandPool,
        .level = vk::CommandBufferLevel::ePrimary,
        .commandBufferCount = 1
    };

    vk::raii::CommandBuffer commandBuffer = vk::raii::CommandBuffers(*device, allocInfo).front();

    vk::CommandBufferBeginInfo beginInfo{
        .flags = vk::CommandBufferUsageFlagBits::eOneTimeSubmit
    };

    commandBuffer.begin(beginInfo);

    return commandBuffer;
}

void ImGuiUtil::endSingleTimeCommands(vk::raii::CommandBuffer& commandBuffer) {
    commandBuffer.end();

    vk::SubmitInfo submitInfo{
        .commandBufferCount = 1,
        .pCommandBuffers = &*commandBuffer
    };

    queue->submit(submitInfo);
    queue->waitIdle();
}

Conclusion

In this section, we’ve explored how to integrate ImGui with Vulkan, including command buffer integration, render pass configuration, and performance considerations. By creating a flexible implementation, we’ve ensured that our GUI system works well with any windowing system you choose.

The key improvements we’ve made include:

  1. Creating a platform-agnostic integration approach

  2. Implementing a flexible input system that works with various windowing libraries

  3. Developing a versatile ImGui utility class

  4. Designing a window-system-independent integration

With this knowledge, you can create a robust GUI system for your Vulkan application that provides a smooth user experience regardless of which windowing system you use.

In the next section, we’ll wrap up with a conclusion and discuss potential improvements to our GUI system.

GUI: Conclusion

Conclusion

In this chapter, we’ve built a comprehensive GUI system for our Vulkan application using Dear ImGui. Let’s summarize what we’ve learned and discuss potential improvements.

What We’ve Learned

  • Flexible ImGui Setup: We explored how to integrate Dear ImGui with Vulkan in a way that works across different platforms, including desktop and mobile. We created an implementation that doesn’t rely on specific windowing systems like GLFW.

  • Versatile Input Handling: We implemented a robust input handling system that correctly routes input events to either the GUI or the 3D scene, ensuring a smooth user experience on any device.

  • UI Elements: We learned how to create various UI elements, from basic components like buttons and sliders to more complex elements like tables and plots, and how to organize them into a cohesive interface that works well on both desktop and mobile platforms.

  • Vulkan Integration: We dove deep into the technical details of integrating ImGui with the Vulkan rendering pipeline, including command buffer integration, render pass configuration, and performance considerations.

With these components in place, we now have a solid foundation for creating interactive applications with Vulkan that can run on multiple platforms. Our GUI system allows users to control settings, display information, and interact with the 3D scene through an intuitive interface, whether they’re using a desktop computer, a mobile phone, or a tablet.

Potential Improvements

While our GUI system is functional, there are several ways it could be enhanced:

  • Targeted Optimizations: Implement specific optimizations for better performance on each target platform.

  • Touch-Friendly UI: Enhance the UI elements to be more touch-friendly for mobile platforms, with larger hit areas and gesture support.

  • Adaptive Layouts: Create layouts that automatically adapt to different screen sizes and orientations, from desktop monitors to mobile phones.

  • Custom Styling: Create a custom theme that matches your application’s visual style, rather than using the default ImGui style.

  • Localization: Add support for multiple languages by implementing a localization system for UI text.

  • Accessibility: Improve accessibility by adding features like keyboard navigation, screen reader support, and high-contrast modes.

  • Persistent Settings: Implement a system to save and load UI settings between application sessions.

  • Advanced Layout: Use ImGui’s docking features to create more complex UI layouts, such as dockable panels.

  • Custom Widgets: Develop custom widgets for specific needs in your application, such as a color wheel, a curve editor, or a node graph editor.

  • Performance Optimization: Profile and optimize the GUI rendering to minimize its impact on overall application performance, especially on mobile devices with limited resources.

  • Battery Efficiency: For mobile platforms, optimize the GUI rendering to minimize battery usage.

Integration with Other Systems

As you continue building your Vulkan engine, consider how the GUI system integrates with other components:

  • Scene Graph: How can the GUI be used to visualize and edit the scene graph hierarchy across different platforms?

  • Material System: Can you create a material editor using the GUI to adjust material properties in real-time, with interfaces that work well on both desktop and mobile?

  • Animation System: How might the GUI be used to control and visualize animations, with controls that are appropriate for each platform?

  • Physics System: Could the GUI provide tools for setting up and debugging physics simulations, with different interaction models for desktop and mobile?

  • Device-Specific Features: How can you leverage specific features (like haptic feedback on mobile) while maintaining a consistent core experience?

By addressing these questions, you can create a more cohesive and powerful engine that leverages the GUI for both development and runtime functionality across multiple platforms.

Cross-Platform Considerations

When developing a GUI system that works across platforms, keep these considerations in mind:

  • Input Methods: Different platforms have different primary input methods (mouse/keyboard vs. touch).

  • Screen Sizes: Interfaces need to work on screens ranging from small phones to large monitors.

  • Performance Constraints: Mobile devices typically have less processing power and memory than desktops.

  • Battery Life: On mobile devices, efficient rendering is crucial for battery life.

  • Platform Conventions: Users expect applications to follow platform-specific UI conventions.

  • Testing: Cross-platform applications require testing on all target platforms.

Alternative GUI Libraries for Vulkan

While we’ve focused on Dear ImGui in this chapter, there are several other GUI libraries that work well with Vulkan. Understanding the options can help you choose the right tool for your specific needs:

  • Nuklear: A minimalist immediate-mode GUI library with a small footprint. It’s designed to be embedded directly into applications and supports Vulkan among other rendering backends. Nuklear is used in smaller indie games and tools due to its simplicity and low overhead.

  • Qt: A comprehensive UI framework that added Vulkan support in Qt 5.10. Qt provides a more traditional retained-mode GUI approach with a rich set of widgets and tools. It’s used in applications like the Autodesk Maya viewport and various CAD software.

  • CEGUI: The Crazy Eddie’s GUI system is a free library providing windowing and widgets for games and simulation applications. It has Vulkan renderer support and is used in some indie game engines.

  • Ultralight: A lightweight, high-performance HTML renderer designed for game and application UIs. It can be integrated with Vulkan and is used by developers who want to leverage web technologies for their interfaces.

  • Noesis GUI: A commercial UI middleware that supports XAML and can render through Vulkan. It’s used in games like Dauntless and provides a designer-friendly workflow.

When choosing a GUI library for your Vulkan application, consider factors like:

  • Development paradigm (immediate-mode vs. retained-mode)

  • Performance requirements

  • Designer-friendliness

  • Learning curve

  • Licensing and cost

  • Platform support

  • Integration complexity

Dear ImGui, which we’ve used in this chapter, strikes a good balance for many developers due to its simplicity, performance, and ease of integration with Vulkan.

Final Thoughts

A well-designed GUI is essential for creating user-friendly applications that can reach a wide audience. It serves as the primary way users interact with your application and can significantly impact the user experience. By understanding how to integrate Dear ImGui with Vulkan and implementing a robust input handling system that works with basic inputs for mouse and keyboard, you’ve taken a major step toward creating professional-quality applications.

Remember that the code provided in this chapter is a starting point. Feel free to modify and extend it to suit your specific needs and application requirements. The flexibility of our approach allows for a wide range of customization and extension while maintaining compatibility with multiple platforms.

In the next chapter, we’ll explore how to load and render 3D models, which will allow us to create more complex and visually interesting scenes.

Vulkan 是 Khronos Group Inc. 的注册商标

教程内容版权归原作者,遵循 CC BY-SA 4.0;本站独立代码及设计除外。