How It Works
============

This section provides a conceptual overview of how LLMize uses Large Language Models for numerical optimization.

The LLM Optimization Paradigm
------------------------------

Traditional optimization algorithms rely on mathematical operations to generate new solutions. LLMize takes a different approach: it uses Large Language Models to understand the optimization problem and generate improved solutions based on previous examples.

The key insight is that LLMs can:
1. **Learn patterns** from solution-score pairs
2. **Generate new solutions** that follow learned patterns
3. **Adapt strategies** based on optimization progress

The Optimization Loop
---------------------

All LLMize optimizers follow a similar loop:

.. image:: https://via.placeholder.com/600x300?text=LLM+Optimization+Loop
   :alt: LLM Optimization Loop Diagram

1. **Initialization**
   - Provide initial solutions and their scores
   - LLM learns the relationship between solutions and scores

2. **Prompt Generation**
   - Format solution-score pairs as examples
   - Add problem description and instructions
   - Include optimization-specific strategies

3. **LLM Inference**
   - Send prompt to the LLM
   - LLM generates new candidate solutions
   - Solutions follow learned patterns and constraints

4. **Evaluation**
   - Apply objective function to each new solution
   - Calculate scores for all candidates
   - Track best solutions and convergence

5. **Selection**
   - Keep the best solutions for the next iteration
   - Maintain diversity (depending on optimizer)
   - Update optimization history

6. **Repeat**
   - Loop continues for specified steps or until convergence

Optimizer Strategies
--------------------

OPRO: Direct Prompting
~~~~~~~~~~~~~~~~~~~~~~

OPRO (Optimization by PROmpting) is the simplest approach:

- Shows the LLM examples of solutions and scores
- Asks the LLM to generate better solutions
- Relies on the LLM's pattern recognition capabilities

**Prompt Structure:**
::

    Problem: [problem description]
    
    Examples:
    solution1 -> score1
    solution2 -> score2
    ...
    
    Generate N new solutions with better scores.

ADOPRO: Adaptive Prompting
~~~~~~~~~~~~~~~~~~~~~~~~~~

ADOPRO enhances OPRO with adaptive strategies:

- Monitors optimization progress
- Adjusts prompts based on performance
- Adds specific instructions when stuck

**Adaptations include:**
- "Try more diverse solutions" (if converging too fast)
- "Focus on improving the best solution" (if progress is slow)
- "Explore different solution regions" (if in local optimum)

HLMEA: Evolutionary Approach
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

HLMEA combines LLM generation with evolutionary algorithms:

- Maintains a population of solutions
- Uses evolutionary operators (selection, crossover, mutation)
- LLM guides the evolutionary process

**Evolutionary Steps:**
1. Select parents from best solutions
2. Apply crossover (combine parent solutions)
3. Apply mutation (modify solutions)
4. LLM generates new population using these principles

HLMSA: Simulated Annealing
~~~~~~~~~~~~~~~~~~~~~~~~~~

HLMSA incorporates simulated annealing concepts:

- Uses temperature to control exploration
- Accepts worse solutions probabilistically
- Gradually focuses on exploitation

**Annealing Process:**
- High temperature: Accept many diverse solutions
- Low temperature: Focus on local improvements
- LLM adjusts perturbation based on temperature

Why LLMs Work for Optimization
------------------------------

1. **Pattern Recognition**: LLMs excel at finding patterns in solution-score relationships
2. **Constraint Understanding**: Natural language constraints are easily encoded
3. **Creative Generation**: LLMs can generate novel solutions beyond mathematical operators
4. **Adaptability**: The same LLM can solve vastly different problems
5. **Interpretability**: Solutions can be explained in natural language

Limitations and Considerations
------------------------------

API Costs
~~~~~~~~~~

Each optimization step requires API calls:
- Cost scales with: steps × batch_size
- Mitigation: Use parallel evaluation, early stopping

Convergence Guarantees
~~~~~~~~~~~~~~~~~~~~~

Unlike traditional optimizers, LLM-based methods:
- Have no theoretical convergence guarantees
- May produce inconsistent results
- Depend on LLM capabilities and training

Solution Quality
~~~~~~~~~~~~~~~~

Quality depends on:
- Clarity of problem description
- Quality of initial examples
- LLM model capabilities
- Prompt engineering

Best Practices
--------------

1. **Clear Problem Description**
   - Be specific about objectives and constraints
   - Include domain knowledge when helpful
   - Provide context about solution format

2. **Good Initial Examples**
   - Use diverse initial solutions
   - Ensure examples follow desired format
   - Include both good and poor examples

3. **Appropriate Parameters**
   - Start with default settings
   - Adjust temperature based on exploration needs
   - Use callbacks for monitoring and control

4. **Monitor Progress**
   - Track best_score_history
   - Watch for premature convergence
   - Adjust strategy based on progress

Future Directions
-----------------

The field of LLM-based optimization is rapidly evolving. Future improvements may include:

- Multi-modal LLMs for visual problems
- Reinforcement learning for prompt optimization
- Hybrid approaches combining traditional and LLM methods
- Specialized optimization models
- Better theoretical foundations

References
----------

- OPRO Paper: https://arxiv.org/abs/2309.03409
- LLMs for Engineering Optimization: https://arxiv.org/abs/2503.19620