Multi-objective RL: two value tables, one weighted decision

What it shows

Delay and energy are different currencies, so learning them together in one table blurs them. MultiObjectiveOrchestrator keeps two tables, delay_values and energy_values, indexed by the same state (the application type and a task-length bucket, state_key = type|bucket with buckets at 1000, 5000 and 20000 MI) and by one action per tier the architecture allows. Each table is updated with the textbook rule Q += 0.2 * (reward + 0.9 * max Q(next) - Q) (LEARNING_RATE = 0.2, DISCOUNT_RATE = 0.9, EPSILON = 0.1). The delay reward is -total_delay / max_latency and the energy reward is the negative estimated energy divided by REFERENCE_ENERGY_WH = 0.01, so 0.01 Wh maps to a reward of -1; a failed task applies -FAILURE_PENALTY = -200 to both tables. Keeping the objectives in separate tables means the trade-off is a decision-time parameter, not a retraining: find_computing_node scores each action as DELAY_WEIGHT * delay_value + ENERGY_WEIGHT * energy_value, both set to 1.0 here.

flowchart TD
    task["Task"] --> state["state_key: type|length bucket"]
    state --> eps{"random < 0.1?"}
    eps -->|yes| explore["Random feasible tier"]
    eps -->|no| score["score = 1.0*D + 1.0*E"]
    score --> dtab["delay_values"]
    score --> etab["energy_values"]
    explore --> run["Run on chosen tier"]
    score --> run
    run --> res{"task SUCCESS?"}
    res -->|yes| dr["d_r = -delay/latency<br/>e_r = -energy/0.01"]
    res -->|no| pen["both rewards = -200"]
    dr --> upd["Q += 0.2 (r + 0.9 maxQ - Q)"]
    pen --> upd
    upd --> state
    classDef tables fill:#cfe8ff,stroke:#333;
    class dtab,etab tables;

The scenario

settings/ runs one scenario, once, on a fixed seed, on the same topology as the ant colony and priority examples:

Run it

From the repository root, after pip install -e .:

python examples/multi_objective/multi_objective.py

No-install variant:

PYTHONPATH=src python examples/multi_objective/multi_objective.py

What to look for

SCALARIZED_Q             devices= 300 tasks= 16870 ok= 12367 avg_delay=0.185    cloud= 1870 edge=10994 mist= 4006 energy=73.2994

This is a policy to tune rather than trust at the default weights: set ENERGY_WEIGHT to 0.1 (or drop REFERENCE_ENERGY_WH) to let the delay table dominate and watch the cloud placements collapse.

Files

Where to go next