Model analysis report for

▶
root: {} 10 keys
- report_name: 'rldriving_driving.analyse_lat_rl'
- route_list: '/home/batman/xx/routes/default_test_2k'
- output_dir: null
- save_tmp: false
- read_only: false
- limit: 10000
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 10
  - job_name: 'rldriving_driving.analyse_lat_rl'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: 'caches/workstation-yassine_2025-06-07_19-42-25'
  - force_local: false
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 30 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'SIMPLE'
    - inpaint_mode: 'NONE'
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 12 keys
    - supercombo: 'b92dd772-6ae6-4329-880d-7e1cc60dd9da/700'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - calibrate_speed: false
    - use_allow_throttle: false
    - use_prev_desired_curv: true
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: false
    - use_raw_plan_long: false
    - desired_curv_smooth_seconds: false
    - desired_accel_smooth_seconds: false
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1563 / 1656 aka 94.384%

better :1179 / 1656 aka 71.196%

best (0.5m):512 / 1656 aka 30.918%

global mae x: 0.230

global mae y: 0.226

global mae y of lanecenter (when detected) in GT: 0.209

global mae y of lanecenter (when detected) in Sim: 0.215

avg resets: 0.257

Mean abs curvature: 0.226 deg/m

Mean abs curvature rate: 0.317 deg/ms

Mean abs lateral accel: 0.370 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.499 m/(sss)

Mean abs lateral jerk GT: 0.271 m/(sss)

Easy straight

Metrics

passed :509 / 510 aka 99.804%

better :498 / 510 aka 97.647%

best (0.5m):305 / 510 aka 59.804%

global mae x: 0.159

global mae y: 0.172

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.179

avg resets: 0.016

Mean abs curvature: 0.048 deg/m

Mean abs curvature rate: 0.088 deg/ms

Mean abs lateral accel: 0.176 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.321 m/(sss)

Mean abs lateral jerk GT: 0.171 m/(sss)

Low speed

Metrics

passed :326 / 347 aka 93.948%

better :208 / 347 aka 59.942%

best (0.5m):82 / 347 aka 23.631%

global mae x: 0.312

global mae y: 0.251

global mae y of lanecenter (when detected) in GT: 0.266

global mae y of lanecenter (when detected) in Sim: 0.247

avg resets: 0.461

Mean abs curvature: 0.441 deg/m

Mean abs curvature rate: 0.640 deg/ms

Mean abs lateral accel: 0.316 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.461 m/(sss)

Mean abs lateral jerk GT: 0.228 m/(sss)

No desire segments

Metrics

passed :922 / 937 aka 98.399%

better :832 / 937 aka 88.794%

best (0.5m):436 / 937 aka 46.531%

global mae x: 0.179

global mae y: 0.195

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.190

avg resets: 0.072

Mean abs curvature: 0.121 deg/m

Mean abs curvature rate: 0.169 deg/ms

Mean abs lateral accel: 0.321 m/(ss)

Mean abs lateral accel GT: 0.284 m/(ss)

Mean abs lateral jerk: 0.412 m/(sss)

Mean abs lateral jerk GT: 0.235 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :123 / 133 aka 92.481%

better :73 / 133 aka 54.887%

best (0.5m):18 / 133 aka 13.534%

global mae x: 0.256

global mae y: 0.264

global mae y of lanecenter (when detected) in GT: 0.261

global mae y of lanecenter (when detected) in Sim: 0.235

avg resets: 0.211

Mean abs curvature: 0.296 deg/m

Mean abs curvature rate: 0.352 deg/ms

Mean abs lateral accel: 0.669 m/(ss)

Mean abs lateral accel GT: 0.645 m/(ss)

Mean abs lateral jerk: 0.674 m/(sss)

Mean abs lateral jerk GT: 0.396 m/(sss)

Left lane change segments

114/120 aka 95.00%

better: 90/120 aka 75.00%

best (0.5m): 29/120 aka 24.17%

mean max error C3: 0.742533

mean max error C2: 0.885997

global mae: 0.213187

avg resets: 0.058333

Right lane change segments

146/150 aka 97.33%

better: 110/150 aka 73.33%

best (0.5m): 33/150 aka 22.00%

mean max error C3: 0.822491

mean max error C2: 0.874941

global mae: 0.230550

avg resets: 0.113333

Left turn segments

66/79 aka 83.54%

better: 26/79 aka 32.91%

best (0.5m): 3/79 aka 3.80%

mean max error C3: 1.361728

mean max error C2: 1.318624

global mae: 0.280616

avg resets: 0.797468

Right turn segments

79/91 aka 86.81%

better: 29/91 aka 31.87%

best (0.5m): 2/91 aka 2.20%

mean max error C3: 1.390341

mean max error C2: 1.356979

global mae: 0.286357

avg resets: 0.890110

Left keep segments

30/33 aka 90.91%

better: 13/33 aka 39.39%

best (0.5m): 1/33 aka 3.03%

mean max error C3: 1.308544

mean max error C2: 1.273226

global mae: 0.260673

avg resets: 0.272727

Right keep segments

23/26 aka 88.46%

better: 13/26 aka 50.00%

best (0.5m): 2/26 aka 7.69%

mean max error C3: 1.288925

mean max error C2: 1.177488

global mae: 0.276998

avg resets: 0.269231

Plan y error in turns (> 6°/s)

Mean path error : -0.016 meters

Mean absolute path error : 0.455 meters

Turn bias test (y_pos at 50m > 3m)

got 28874 left samples

got 30651 right samples

mean left gt: -0.05176308

mean right gt: 0.016140223

mean turn lanecenter error (GT pos, GT ll): left = 0.2170194685459137, right = -0.24659454822540283

mean turn lanecenter error (sim pos, GT ll): left = 0.18553046882152557, right = -0.23637446761131287

mean turn lanecenter error (GT pos, model ll): left = 0.2148418128490448, right = -0.2401113510131836

mean turn lanecenter error (sim pos, model ll): left = 0.15512578189373016, right = -0.22132596373558044

Mean abs curvature: 0.214273 deg/m

Mean abs curvature rate: 0.212986 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.173742 Hz, period 5.755661 s

Mean amplitude: 0.121969 m

Mean ping-pong significance (frequency x amplitude): 2.050613 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	165
turn_right	179
lane_change_left	234
lane_change_right	285
keep_left	79
keep_right	67

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-1.531m
turn_right	+0.915m
lane_change_left	-1.668m
lane_change_right	+1.586m
keep_left	-0.694m
keep_right	+0.558m

Mean desire duration - ground truth

Command	Duration
turn_left	6.816s
turn_right	6.226s
lane_change_left	3.316s
lane_change_right	3.496s
keep_left	3.728s
keep_right	3.840s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	3.27s	4.10s	4.80s	5.57s
turn_right	3.45s	4.06s	4.76s	5.63s
lane_change_left	0.48s	1.22s	2.13s	3.72s
lane_change_right	0.67s	1.35s	2.23s	3.71s
keep_left	3.49s	5.04s	5.99s	7.04s
keep_right	3.31s	4.66s	5.88s	7.18s

Group	Gas prob	Brake prob	Steer prob
All segments	0.010	0.023	0.264
Easy straight	0.006	0.009	0.143
Low speed	0.017	0.036	0.371
No desire segments	0.007	0.014	0.205
No desire turns (max lat accel > 2.5 m/ss)	0.011	0.028	0.360