Project 2 | Congyu Wang

Project Overview

This project explores fundamental concepts in image processing through filters and frequency domain analysis. Part 1 focuses on implementing edge detection algorithms from scratch, comparing different convolution methods, and understanding how filters reveal image structure through derivatives and frequency analysis.

Part 1: Filters and Edges

Part 1.1: Convolution Implementation and Comparison

In this section, we implement convolution operations using pure NumPy and compare them with scipy.signal.convolve2d. We explore different implementation approaches and analyze their performance characteristics.

Implementation Approaches

1. Four-Loop Convolution (Naive Implementation)

def convolve_4loops(image, kernel):

    img = np.array(img, dtype=float)
    kernel = np.array(kernel, dtype=float)
    
    pad_h = kernel.shape[0] // 2
    pad_w = kernel.shape[1] // 2

    padded_img = np.pad(img, ((pad_h, pad_h), (pad_w, pad_w)), 'constant')
    
    result = np.zeros_like(img)
    
    for i in range(img.shape[0]):
        for j in range(img.shape[1]):
            for k in range(kernel.shape[0]):
                for l in range(kernel.shape[1]):
                    result[i, j] += padded_img[i+k, j+l] * kernel[k, l]
    
    return result

2. Two-Loop Convolution (Optimized)

def conv_two_loops(img, kernel):

    img = np.array(img, dtype=float)
    kernel = np.array(kernel, dtype=float)
    
    pad_h = kernel.shape[0] // 2
    pad_w = kernel.shape[1] // 2
    
    padded_img = np.pad(img, ((pad_h, pad_h), (pad_w, pad_w)), 'constant')
    
    result = np.zeros_like(img)
    
    for i in range(img.shape[0]):
        for j in range(img.shape[1]):
            region = padded_img[i:i+kernel.shape[0], j:j+kernel.shape[1]]
            result[i, j] = np.sum(region * kernel)
    
    return result

3. SciPy Reference Implementation

from scipy.signal import convolve2d
result = convolve2d(image, kernel, mode='same')

Results and Analysis

Original Image
Test image for convolution comparison

Four-Loop Convolution
Naive implementation result

Two-Loop Convolution
Optimized NumPy implementation

SciPy Convolution
Reference implementation result

Performance and Boundary Handling Analysis

Runtime Comparison:

Four-Loop Implementation: 21.09 seconds
Two-Loop Implementation: 8.71 seconds
SciPy Implementation: 0.14 seconds

Key Insights:

SciPy is ~152× faster than four-loop implementation
Vectorization provides ~2.4× speedup
Custom implementations: zero-padding; SciPy: multiple boundary conditions

Part 1.2: Partial Derivatives and Edge Detection

Edge detection through gradient computation reveals image structure by highlighting regions of rapid intensity change. We implement finite difference operators to compute partial derivatives and construct gradient-based edge maps.

Mathematical Foundation

Finite Difference Operators:

X-direction (vertical edges): $D_x = \begin{bmatrix} 1 & -1 \end{bmatrix}$

Y-direction (horizontal edges): $D_y = \begin{bmatrix} 1 \ -1 \end{bmatrix}$

Gradient Magnitude: $|\nabla I| = \sqrt{(\frac{\partial I}{\partial x})^2 + (\frac{\partial I}{\partial y})^2}$

Results

Original Image
Cameraman test image

∂I/∂x
Vertical edge detection

∂I/∂y
Horizontal edge detection

Gradient Magnitude
Combined edge strength

Binary Edge Map
Thresholded at 0.15

Edge Detection Analysis

Threshold Selection:

Threshold = 0.15: Balances edge completeness with noise suppression
Lower values capture noise; higher values miss weak edges

Trade-offs:

Finite difference method is noise-sensitive but preserves structural edges
Simple thresholding may fragment thin edges

Part 1.3: Gaussian and DoG Filters

Gaussian smoothing reduces noise before edge detection, while Difference of Gaussians (DoG) filters provide a more robust approach to edge detection by combining smoothing and differentiation in a single operation.

Filter Construction

Gaussian Kernel Generation:

import cv2
# Create separable Gaussian kernels
kernel_1d = cv2.getGaussianKernel(ksize=5, sigma=1.0)
gaussian_2d = kernel_1d @ kernel_1d.T

DoG Filter Construction: $\text{DoG}_x = G_\sigma * D_x$ $\text{DoG}_y = G_\sigma * D_y$

Where $G_\sigma$ is the Gaussian kernel and $D_x, D_y$ are finite difference operators.

Filter Visualizations

Gaussian Kernel
σ = 1.0, size = 5×5

DoG X Filter
Gaussian derivative in X

DoG Y Filter
Gaussian derivative in Y

Application Results

Gaussian Smoothed
Noise reduction preprocessing

DoG Gradient Magnitude
Single-step edge detection

Method Comparison
Finite difference vs DoG

Comparison: Finite Difference vs. DoG Method

Finite Difference: Simple but noise-sensitive, requires separate smoothing DoG Method: Built-in noise suppression, single operation, parameter-dependent

Key Results:

DoG produces cleaner edge maps with better noise robustness
DoG is computationally more efficient (single vs. two-step process)
Both methods provide similar edge localization accuracy

Part 2: Fun with Frequencies

Part 2.1: Image “Sharpening” with Unsharp Mask Filter

The unsharp mask filter is a classic image sharpening technique that enhances edges and fine details by amplifying high-frequency components. Despite its name, the process involves creating a “mask” from a blurred (unsharp) version of the image.

Mathematical Foundation

The unsharp mask operation can be expressed as:

\[\text{Sharpened} = \text{Original} + \alpha \times (\text{Original} - \text{Blurred})\]

Where:

$\alpha$ is the sharpening strength parameter
$(\text{Original} - \text{Blurred})$ represents the high-frequency details

This can be rewritten as: $\text{Sharpened} = (1 + \alpha) \times \text{Original} - \alpha \times \text{Blurred}$

How Unsharp Mask Works

Blur the original image using a Gaussian filter to remove high frequencies
Extract high frequencies by subtracting the blurred version from the original
Add amplified high frequencies back to the original image

The relationship to frequency domain:

Low frequencies: Preserved from the original image
High frequencies: Amplified by factor $\alpha$
Result: Enhanced edge contrast and detail sharpness

Results

Taj Mahal Sharpening:

Original Image
Taj Mahal photograph

Blurred Version
Gaussian blur (σ = 2.0)

High Frequency Details
Original - Blurred

Sharpened Result
α = 1.0, enhanced architectural details

Building Architecture Sharpening:

Original Building
Modern architecture

Pre-blurred Version
Gaussian blur (σ = 1.0)

Further Blurred
For unsharp mask (σ = 2.0)

High Frequency Details
Pre-blurred - Further Blurred

Sharpened Result
α = 1.0, applied to pre-blurred image

Process Flow:
Original → Pre-blur (σ=1) → Unsharp mask → Final result

Analysis: Varying Sharpening Amount

α Parameter Effects:

α = 0.5: Subtle enhancement
α = 1.0: Moderate sharpening (optimal balance)
α = 1.5: Strong sharpening
α > 2.0: Over-sharpening artifacts

Key Points: Unsharp mask enhances edges and fine textures; excessive values cause halos and noise amplification.

Part 2.2: Hybrid Images

Hybrid images are static images that change in interpretation based on viewing distance. They combine the high-frequency components of one image with the low-frequency components of another, creating a fascinating perceptual illusion.

Technical Approach

Process Overview:

Image Alignment: Register two images to the same coordinate system
Frequency Separation: Apply complementary filters (low-pass and high-pass)
Combination: Add filtered components to create hybrid result
Analysis: Examine frequency domain properties

Mathematical Foundation: $\text{Hybrid} = \text{LowPass}(\text{Image}_1) + \text{HighPass}(\text{Image}_2)$

Featured Example: Derek + Nutmeg

Complete Process Visualization:

Derek (Original)
Source for low frequencies

Nutmeg (Original)
Source for high frequencies

Frequency Domain Analysis:

Derek FFT
Frequency spectrum

Nutmeg FFT
Frequency spectrum

Hybrid FFT
Combined spectrum

Filtered Components:

Derek Low-pass Filtered
Cutoff frequency: 15 cycles/image

Nutmeg High-pass Filtered
Cutoff frequency: 15 cycles/image

Final Hybrid Result:

Derek + Nutmeg Hybrid
View from distance to see Derek, up close to see Nutmeg

Additional Hybrid Examples

Custom Hybrid 1
Up close: Mona Lisa, From distance: Einstein

Custom Hybrid 2
Up close: David sculpture, From distance: Statue of Liberty

Cutoff Frequency Selection

Parameter Guidelines:

< 10 cycles/image: Insufficient detail
10-20 cycles/image: Optimal range
> 25 cycles/image: Competing details

Success Factors: Proper alignment, complementary content, contrast matching

Part 2.3: Gaussian and Laplacian Stacks

Multi-resolution image analysis using Gaussian and Laplacian pyramids enables sophisticated blending operations that preserve natural transitions across different frequency bands.

Mathematical Framework

Gaussian Stack: $G_0 = \text{Original Image}$ $G_k = G_{k-1} * \text{Gaussian}(\sigma_k)$

Laplacian Stack: $L_k = G_k - G_{k+1}$ $L_{\text{final}} = G_{\text{final}}$

Orange and Apple Blending Process

Multi-resolution Blending Process:

The multi-resolution blending technique operates by decomposing images into frequency bands using Laplacian stacks, blending each band separately with the mask at corresponding resolution levels, and then reconstructing the final result. This approach ensures smooth transitions across all spatial frequencies.

Process Steps:

Decomposition: Both source images are decomposed into Laplacian stacks (frequency bands)
Mask Processing: The blending mask is also decomposed into a Gaussian stack
Band-wise Blending: Each frequency band is blended using the corresponding mask level
Reconstruction: All blended bands are summed to create the final seamless result

Multi-resolution Blending Visualization
Shows the complete process: original images, Laplacian decomposition, mask-based blending at each level, and final reconstruction. Each row represents a different frequency band, demonstrating how details at different scales are seamlessly combined.

Final Reconstructed Result:

"Oraple" - Blended Result
Seamless multi-frequency blending

Multi-resolution Blending Results

The multi-resolution blending technique demonstrates superior performance, showing:

Frequency Band Preservation: Each level maintains specific spatial frequency information
Natural Transitions: Smooth blending without visible seams
Detail Preservation: Fine textures maintained across the blend boundary
Scalability: Technique works across different image sizes and content types

Part 2.4: Multiresolution Blending with Irregular Masks

Advanced blending using irregular masks enables more sophisticated image compositions that follow natural object boundaries rather than simple geometric shapes.

Custom Mask Creation Method

This project implements an Interactive Drawing-Based Mask Creation system that provides real-time, user-controlled mask generation:

Key Features:

Interactive Brush Tool: Circular brush with adjustable size for precise painting
Real-time Visualization: Semi-transparent overlay shows mask boundaries during creation
Dual Editing Modes: Paint (left-click) and erase (right-click) functionality
Automatic Smoothing: Gaussian blur post-processing ensures smooth transitions

The system seamlessly integrates with multi-resolution pyramid blending, automatically generating mask pyramids for each frequency band while preserving user-defined boundaries.