Model analysis report for

▶
root: {} 11 keys
- report_name: 'rldriving_analyselat'
- output_dir: null
- save_tmp: false
- read_only: false
- limit: 10000
- shuffle: false
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 1000
  - job_name: 'rldriving_analyselat'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: 'caches/workstation-yassine_2025-05-09_19-49-00'
  - force_local: false
- route_list: '/home/batman/xx/routes/default_test_2k'
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 30 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'DEPTH'
    - inpaint_mode: 'NONE'
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 10 keys
    - supercombo: '1496451e-897b-4a1b-a284-37d244bfddb3/400'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - calibrate_speed: false
    - use_allow_throttle: false
    - use_prev_desired_curv: false
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: true
    - use_raw_plan_long: true
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1442 / 1657 aka 87.025%

better :1054 / 1657 aka 63.609%

best (0.5m):342 / 1657 aka 20.640%

global mae x: 0.231

global mae y: 0.269

global mae y of lanecenter (when detected) in GT: 0.209

global mae y of lanecenter (when detected) in Sim: 0.237

avg resets: 0.319

Mean abs curvature: 0.200 deg/m

Mean abs curvature rate: 0.280 deg/ms

Mean abs lateral accel: 0.362 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.465 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :502 / 510 aka 98.431%

better :469 / 510 aka 91.961%

best (0.5m):216 / 510 aka 42.353%

global mae x: 0.160

global mae y: 0.197

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.191

avg resets: 0.024

Mean abs curvature: 0.039 deg/m

Mean abs curvature rate: 0.083 deg/ms

Mean abs lateral accel: 0.175 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.342 m/(sss)

Mean abs lateral jerk GT: 0.171 m/(sss)

Low speed

Metrics

passed :292 / 347 aka 84.150%

better :185 / 347 aka 53.314%

best (0.5m):57 / 347 aka 16.427%

global mae x: 0.302

global mae y: 0.304

global mae y of lanecenter (when detected) in GT: 0.266

global mae y of lanecenter (when detected) in Sim: 0.262

avg resets: 0.545

Mean abs curvature: 0.391 deg/m

Mean abs curvature rate: 0.566 deg/ms

Mean abs lateral accel: 0.306 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.397 m/(sss)

Mean abs lateral jerk GT: 0.228 m/(sss)

No desire segments

Metrics

passed :885 / 938 aka 94.350%

better :775 / 938 aka 82.623%

best (0.5m):301 / 938 aka 32.090%

global mae x: 0.180

global mae y: 0.226

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.204

avg resets: 0.095

Mean abs curvature: 0.106 deg/m

Mean abs curvature rate: 0.146 deg/ms

Mean abs lateral accel: 0.318 m/(ss)

Mean abs lateral accel GT: 0.284 m/(ss)

Mean abs lateral jerk: 0.406 m/(sss)

Mean abs lateral jerk GT: 0.236 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :108 / 133 aka 81.203%

better :68 / 133 aka 51.128%

best (0.5m):16 / 133 aka 12.030%

global mae x: 0.250

global mae y: 0.308

global mae y of lanecenter (when detected) in GT: 0.261

global mae y of lanecenter (when detected) in Sim: 0.269

avg resets: 0.271

Mean abs curvature: 0.269 deg/m

Mean abs curvature rate: 0.288 deg/ms

Mean abs lateral accel: 0.654 m/(ss)

Mean abs lateral accel GT: 0.646 m/(ss)

Mean abs lateral jerk: 0.578 m/(sss)

Mean abs lateral jerk GT: 0.396 m/(sss)

Left lane change segments

104/120 aka 86.67%

better: 72/120 aka 60.00%

best (0.5m): 17/120 aka 14.17%

mean max error C3: 0.957434

mean max error C2: 1.161669

global mae: 0.270780

avg resets: 0.100000

Right lane change segments

129/150 aka 86.00%

better: 83/150 aka 55.33%

best (0.5m): 13/150 aka 8.67%

mean max error C3: 1.055169

mean max error C2: 1.270563

global mae: 0.286582

avg resets: 0.240000

Left turn segments

63/79 aka 79.75%

better: 23/79 aka 29.11%

best (0.5m): 0/79 aka 0.00%

mean max error C3: 1.505431

mean max error C2: 1.426323

global mae: 0.321605

avg resets: 0.822785

Right turn segments

62/90 aka 68.89%

better: 22/90 aka 24.44%

best (0.5m): 2/90 aka 2.22%

mean max error C3: 1.612009

mean max error C2: 1.639113

global mae: 0.386538

avg resets: 1.022222

Left keep segments

22/33 aka 66.67%

better: 14/33 aka 42.42%

best (0.5m): 1/33 aka 3.03%

mean max error C3: 1.588285

mean max error C2: 1.475056

global mae: 0.311958

avg resets: 0.545455

Right keep segments

22/26 aka 84.62%

better: 13/26 aka 50.00%

best (0.5m): 4/26 aka 15.38%

mean max error C3: 1.168476

mean max error C2: 1.181319

global mae: 0.311767

avg resets: 0.192308

Plan y error in turns (> 6°/s)

Mean path error : 0.010 meters

Mean absolute path error : 0.507 meters

Turn bias test (y_pos at 50m > 3m)

got 28799 left samples

got 30815 right samples

mean left gt: -0.025579425

mean right gt: 0.019240947

mean turn lanecenter error (GT pos, GT ll): left = 0.21541060507297516, right = -0.24762769043445587

mean turn lanecenter error (sim pos, GT ll): left = 0.1925777643918991, right = -0.22886568307876587

mean turn lanecenter error (GT pos, model ll): left = 0.20377743244171143, right = -0.2508736252784729

mean turn lanecenter error (sim pos, model ll): left = 0.16814389824867249, right = -0.20053966343402863

Mean abs curvature: 0.213566 deg/m

Mean abs curvature rate: 0.181357 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.170443 Hz, period 5.867061 s

Mean amplitude: 0.138051 m

Mean ping-pong significance (frequency x amplitude): 2.230169 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	166
turn_right	178
lane_change_left	234
lane_change_right	285
keep_left	80
keep_right	67

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-5.395m
turn_right	+5.460m
lane_change_left	-1.484m
lane_change_right	+1.220m
keep_left	-0.792m
keep_right	+0.746m

Mean desire duration - ground truth

Command	Duration
turn_left	6.802s
turn_right	6.235s
lane_change_left	3.316s
lane_change_right	3.496s
keep_left	3.741s
keep_right	3.840s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	2.47s	3.26s	3.91s	4.50s
turn_right	2.43s	3.25s	3.78s	4.46s
lane_change_left	1.15s	1.72s	2.46s	3.78s
lane_change_right	1.21s	1.83s	2.66s	4.32s
keep_left	1.61s	2.30s	3.36s	5.16s
keep_right	1.86s	2.96s	3.88s	5.23s

Group	Gas prob	Brake prob	Steer prob
All segments	0.010	0.024	0.269
Easy straight	0.007	0.009	0.147
Low speed	0.018	0.037	0.376
No desire segments	0.008	0.014	0.209
No desire turns (max lat accel > 2.5 m/ss)	0.011	0.028	0.363