Model analysis report for

▶
root: {} 11 keys
- report_name: 'rldriving_analyselat'
- output_dir: null
- save_tmp: false
- read_only: false
- limit: 10000
- shuffle: false
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 1000
  - job_name: 'rldriving_analyselat'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: 'caches/workstation-yassine_2025-05-13_10-13-43'
  - force_local: false
- route_list: '/home/batman/xx/routes/default_test_2k'
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 30 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'SIMPLE'
    - inpaint_mode: 'NONE'
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 10 keys
    - supercombo: 'e30305e9-5fb8-48bd-a278-f714e880aef1/700'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - calibrate_speed: false
    - use_allow_throttle: false
    - use_prev_desired_curv: true
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: false
    - use_raw_plan_long: false
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1548 / 1658 aka 93.366%

better :1168 / 1658 aka 70.446%

best (0.5m):431 / 1658 aka 25.995%

global mae x: 0.229

global mae y: 0.231

global mae y of lanecenter (when detected) in GT: 0.209

global mae y of lanecenter (when detected) in Sim: 0.220

avg resets: 0.249

Mean abs curvature: 0.230 deg/m

Mean abs curvature rate: 0.320 deg/ms

Mean abs lateral accel: 0.374 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.497 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :509 / 510 aka 99.804%

better :489 / 510 aka 95.882%

best (0.5m):262 / 510 aka 51.373%

global mae x: 0.159

global mae y: 0.176

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.185

avg resets: 0.016

Mean abs curvature: 0.049 deg/m

Mean abs curvature rate: 0.089 deg/ms

Mean abs lateral accel: 0.181 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.321 m/(sss)

Mean abs lateral jerk GT: 0.171 m/(sss)

Low speed

Metrics

passed :320 / 347 aka 92.219%

better :202 / 347 aka 58.213%

best (0.5m):73 / 347 aka 21.037%

global mae x: 0.304

global mae y: 0.252

global mae y of lanecenter (when detected) in GT: 0.266

global mae y of lanecenter (when detected) in Sim: 0.252

avg resets: 0.438

Mean abs curvature: 0.452 deg/m

Mean abs curvature rate: 0.632 deg/ms

Mean abs lateral accel: 0.319 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.453 m/(sss)

Mean abs lateral jerk GT: 0.228 m/(sss)

No desire segments

Metrics

passed :921 / 938 aka 98.188%

better :813 / 938 aka 86.674%

best (0.5m):372 / 938 aka 39.659%

global mae x: 0.179

global mae y: 0.198

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.195

avg resets: 0.072

Mean abs curvature: 0.123 deg/m

Mean abs curvature rate: 0.171 deg/ms

Mean abs lateral accel: 0.325 m/(ss)

Mean abs lateral accel GT: 0.284 m/(ss)

Mean abs lateral jerk: 0.409 m/(sss)

Mean abs lateral jerk GT: 0.236 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :122 / 132 aka 92.424%

better :68 / 132 aka 51.515%

best (0.5m):18 / 132 aka 13.636%

global mae x: 0.254

global mae y: 0.263

global mae y of lanecenter (when detected) in GT: 0.260

global mae y of lanecenter (when detected) in Sim: 0.239

avg resets: 0.189

Mean abs curvature: 0.294 deg/m

Mean abs curvature rate: 0.342 deg/ms

Mean abs lateral accel: 0.671 m/(ss)

Mean abs lateral accel GT: 0.649 m/(ss)

Mean abs lateral jerk: 0.659 m/(sss)

Mean abs lateral jerk GT: 0.398 m/(sss)

Left lane change segments

118/120 aka 98.33%

better: 93/120 aka 77.50%

best (0.5m): 31/120 aka 25.83%

mean max error C3: 0.689156

mean max error C2: 0.855447

global mae: 0.213621

avg resets: 0.066667

Right lane change segments

144/150 aka 96.00%

better: 104/150 aka 69.33%

best (0.5m): 17/150 aka 11.33%

mean max error C3: 0.871655

mean max error C2: 0.938028

global mae: 0.241029

avg resets: 0.126667

Left turn segments

63/79 aka 79.75%

better: 30/79 aka 37.97%

best (0.5m): 1/79 aka 1.27%

mean max error C3: 1.357153

mean max error C2: 1.338654

global mae: 0.292253

avg resets: 0.772152

Right turn segments

75/91 aka 82.42%

better: 29/91 aka 31.87%

best (0.5m): 3/91 aka 3.30%

mean max error C3: 1.379683

mean max error C2: 1.354860

global mae: 0.291429

avg resets: 0.890110

Left keep segments

28/33 aka 84.85%

better: 15/33 aka 45.45%

best (0.5m): 0/33 aka 0.00%

mean max error C3: 1.348786

mean max error C2: 1.274081

global mae: 0.259876

avg resets: 0.242424

Right keep segments

23/26 aka 88.46%

better: 15/26 aka 57.69%

best (0.5m): 1/26 aka 3.85%

mean max error C3: 1.464851

mean max error C2: 1.152520

global mae: 0.271027

avg resets: 0.269231

Plan y error in turns (> 6°/s)

Mean path error : -0.029 meters

Mean absolute path error : 0.436 meters

Turn bias test (y_pos at 50m > 3m)

got 29069 left samples

got 30815 right samples

mean left gt: -0.033501696

mean right gt: 0.026990766

mean turn lanecenter error (GT pos, GT ll): left = 0.21726641058921814, right = -0.24762772023677826

mean turn lanecenter error (sim pos, GT ll): left = 0.18915356695652008, right = -0.23334111273288727

mean turn lanecenter error (GT pos, model ll): left = 0.20967184007167816, right = -0.23950721323490143

mean turn lanecenter error (sim pos, model ll): left = 0.1603313833475113, right = -0.20357440412044525

Mean abs curvature: 0.214717 deg/m

Mean abs curvature rate: 0.211588 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.175749 Hz, period 5.689929 s

Mean amplitude: 0.130329 m

Mean ping-pong significance (frequency x amplitude): 2.198215 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	166
turn_right	179
lane_change_left	234
lane_change_right	285
keep_left	80
keep_right	67

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-0.863m
turn_right	+0.618m
lane_change_left	-1.659m
lane_change_right	+1.544m
keep_left	-0.613m
keep_right	+0.964m

Mean desire duration - ground truth

Command	Duration
turn_left	6.802s
turn_right	6.226s
lane_change_left	3.316s
lane_change_right	3.496s
keep_left	3.741s
keep_right	3.840s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	3.21s	4.01s	4.71s	5.58s
turn_right	3.38s	4.04s	4.60s	5.58s
lane_change_left	0.52s	1.21s	2.10s	3.51s
lane_change_right	0.65s	1.32s	2.19s	3.91s
keep_left	3.47s	5.00s	5.65s	7.26s
keep_right	3.40s	4.57s	6.05s	7.10s

Group	Gas prob	Brake prob	Steer prob
All segments	0.010	0.023	0.268
Easy straight	0.006	0.009	0.147
Low speed	0.017	0.036	0.374
No desire segments	0.007	0.014	0.209
No desire turns (max lat accel > 2.5 m/ss)	0.011	0.028	0.364