Context-aware DRL: a small neural network values each tier from the device context

What it shows

This answers what a table cannot: what to do in a context the agent has never seen. ContextAwareOrchestrator replaces the Q-table with a two-layer network (HIDDEN_UNITS = 12 over FEATURE_COUNT = 10 inputs) whose inputs describe the device and the task: an application one-hot (3), the task length scaled by LENGTH_SCALE = 100000, the battery level, a cellular flag, a mobility flag and the load of each of the three tiers. Because nearby contexts share the same weights, a low-battery cellular phone produces a sensible value even if that exact combination never appeared before. Training is online semi-gradient temporal-difference in _train: the target is reward + 0.9 * max Q_target(next) (DISCOUNT_RATE = 0.9), the weights move by LEARNING_RATE = 0.02, and the target network is refreshed every TARGET_SYNC = 200 updates. The reward is -total_delay / max_latency for a success and FAILURE_PENALTY = -1.0 for a failure. Exploration starts at EPSILON_START = 0.3 and decays by EPSILON_DECAY = 0.999 per decision down to EPSILON_END = 0.02. The network is built from plain Python lists, so the example keeps the dependency-free promise of the package.

flowchart TD
    task["Task"] --> feats["features: 10 inputs"]
    feats --> hidden["12 tanh units"]
    hidden --> out["3 tier values"]
    out --> eps{"random < epsilon?"}
    eps -->|yes| explore["Random feasible tier"]
    eps -->|no| greedy["Tier with max value"]
    explore --> node["Least loaded node of the tier"]
    greedy --> node
    node --> exec["Run on Mist, Edge or Cloud"]
    exec --> reward["r = -delay/latency, or -1.0 if failed"]
    reward --> train["_train: semi-gradient TD"]
    train --> sync["target = online every 200"]
    sync --> feats
    classDef net fill:#cfe8ff,stroke:#333;
    class train,sync net;

The scenario

settings/ runs one scenario, once, on a fixed seed, for twice as long as the other adaptive examples:

Run it

From the repository root, after pip install -e .:

python examples/contextaware_drl/contextaware_drl.py

No-install variant:

PYTHONPATH=src python examples/contextaware_drl/contextaware_drl.py

This run is the slowest of the four (a 10-minute scenario with a forward pass per task), so give it a few minutes.

What to look for

CONTEXT_AWARE            devices= 300 tasks= 43500 ok= 37637 avg_delay=0.1531   cloud= 7164 edge=33629 mist= 2707 energy=197.7162

Raise EPSILON_DECAY toward 1.0 (slower decay) to keep exploring, or lower it to let the network commit sooner.

Files

Where to go next