Model analysis report for

▶
root: {} 10 keys
- report_name: 'rldriving_driving.analyse_lat_rl'
- route_list: '/home/batman/xx/routes/default_test_2k'
- output_dir: null
- save_tmp: false
- read_only: false
- limit: 10000
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 10
  - job_name: 'rldriving_driving.analyse_lat_rl'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: 'caches/workstation-yassine_2025-06-09_15-43-38'
  - force_local: false
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 30 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'SIMPLE'
    - inpaint_mode: 'NONE'
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 12 keys
    - supercombo: '6a8a3da8-c264-4f91-b0a6-d04722cccfce/700'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - calibrate_speed: false
    - use_allow_throttle: false
    - use_prev_desired_curv: true
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: false
    - use_raw_plan_long: false
    - desired_curv_smooth_seconds: false
    - desired_accel_smooth_seconds: false
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1580 / 1658 aka 95.296%

better :1203 / 1658 aka 72.557%

best (0.5m):506 / 1658 aka 30.519%

global mae x: 0.165

global mae y: 0.224

global mae y of lanecenter (when detected) in GT: 0.209

global mae y of lanecenter (when detected) in Sim: 0.216

avg resets: 0.237

Mean abs curvature: 0.223 deg/m

Mean abs curvature rate: 0.303 deg/ms

Mean abs lateral accel: 0.372 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.485 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :510 / 510 aka 100.000%

better :496 / 510 aka 97.255%

best (0.5m):315 / 510 aka 61.765%

global mae x: 0.106

global mae y: 0.173

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.180

avg resets: 0.018

Mean abs curvature: 0.048 deg/m

Mean abs curvature rate: 0.087 deg/ms

Mean abs lateral accel: 0.175 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.311 m/(sss)

Mean abs lateral jerk GT: 0.171 m/(sss)

Low speed

Metrics

passed :325 / 347 aka 93.660%

better :213 / 347 aka 61.383%

best (0.5m):78 / 347 aka 22.478%

global mae x: 0.221

global mae y: 0.247

global mae y of lanecenter (when detected) in GT: 0.266

global mae y of lanecenter (when detected) in Sim: 0.245

avg resets: 0.441

Mean abs curvature: 0.434 deg/m

Mean abs curvature rate: 0.609 deg/ms

Mean abs lateral accel: 0.319 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.448 m/(sss)

Mean abs lateral jerk GT: 0.228 m/(sss)

No desire segments

Metrics

passed :927 / 938 aka 98.827%

better :834 / 938 aka 88.913%

best (0.5m):444 / 938 aka 47.335%

global mae x: 0.124

global mae y: 0.193

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.191

avg resets: 0.068

Mean abs curvature: 0.120 deg/m

Mean abs curvature rate: 0.163 deg/ms

Mean abs lateral accel: 0.321 m/(ss)

Mean abs lateral accel GT: 0.284 m/(ss)

Mean abs lateral jerk: 0.399 m/(sss)

Mean abs lateral jerk GT: 0.236 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :126 / 133 aka 94.737%

better :83 / 133 aka 62.406%

best (0.5m):17 / 133 aka 12.782%

global mae x: 0.210

global mae y: 0.249

global mae y of lanecenter (when detected) in GT: 0.261

global mae y of lanecenter (when detected) in Sim: 0.235

avg resets: 0.165

Mean abs curvature: 0.294 deg/m

Mean abs curvature rate: 0.345 deg/ms

Mean abs lateral accel: 0.668 m/(ss)

Mean abs lateral accel GT: 0.646 m/(ss)

Mean abs lateral jerk: 0.647 m/(sss)

Mean abs lateral jerk GT: 0.396 m/(sss)

Left lane change segments

117/120 aka 97.50%

better: 89/120 aka 74.17%

best (0.5m): 21/120 aka 17.50%

mean max error C3: 0.710322

mean max error C2: 0.873602

global mae: 0.214106

avg resets: 0.041667

Right lane change segments

146/150 aka 97.33%

better: 108/150 aka 72.00%

best (0.5m): 28/150 aka 18.67%

mean max error C3: 0.833319

mean max error C2: 0.876379

global mae: 0.232108

avg resets: 0.113333

Left turn segments

68/79 aka 86.08%

better: 29/79 aka 36.71%

best (0.5m): 0/79 aka 0.00%

mean max error C3: 1.337763

mean max error C2: 1.310509

global mae: 0.281457

avg resets: 0.734177

Right turn segments

82/91 aka 90.11%

better: 33/91 aka 36.26%

best (0.5m): 2/91 aka 2.20%

mean max error C3: 1.233751

mean max error C2: 1.271191

global mae: 0.273978

avg resets: 0.791209

Left keep segments

29/33 aka 87.88%

better: 13/33 aka 39.39%

best (0.5m): 1/33 aka 3.03%

mean max error C3: 1.337232

mean max error C2: 1.358332

global mae: 0.266453

avg resets: 0.121212

Right keep segments

22/26 aka 84.62%

better: 14/26 aka 53.85%

best (0.5m): 3/26 aka 11.54%

mean max error C3: 1.399103

mean max error C2: 1.147612

global mae: 0.289168

avg resets: 0.192308

Plan y error in turns (> 6°/s)

Mean path error : -0.001 meters

Mean absolute path error : 0.421 meters

Turn bias test (y_pos at 50m > 3m)

got 29069 left samples

got 30815 right samples

mean left gt: -0.029147478

mean right gt: 0.009959127

mean turn lanecenter error (GT pos, GT ll): left = 0.21726644039154053, right = -0.24762769043445587

mean turn lanecenter error (sim pos, GT ll): left = 0.2041744589805603, right = -0.24059267342090607

mean turn lanecenter error (GT pos, model ll): left = 0.2218925952911377, right = -0.2388170212507248

mean turn lanecenter error (sim pos, model ll): left = 0.17497864365577698, right = -0.20996639132499695

Mean abs curvature: 0.213938 deg/m

Mean abs curvature rate: 0.201793 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.177003 Hz, period 5.649615 s

Mean amplitude: 0.125843 m

Mean ping-pong significance (frequency x amplitude): 2.149002 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	166
turn_right	179
lane_change_left	234
lane_change_right	285
keep_left	80
keep_right	67

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-0.936m
turn_right	+1.117m
lane_change_left	-1.675m
lane_change_right	+1.570m
keep_left	-0.546m
keep_right	+0.730m

Mean desire duration - ground truth

Command	Duration
turn_left	6.802s
turn_right	6.226s
lane_change_left	3.316s
lane_change_right	3.496s
keep_left	3.741s
keep_right	3.840s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	3.37s	4.20s	4.97s	5.80s
turn_right	3.26s	4.07s	4.59s	5.36s
lane_change_left	0.54s	1.20s	2.04s	3.47s
lane_change_right	0.66s	1.28s	2.13s	3.77s
keep_left	3.83s	5.07s	6.19s	7.32s
keep_right	3.65s	5.03s	5.99s	7.04s

Group	Gas prob	Brake prob	Steer prob
All segments	0.010	0.023	0.266
Easy straight	0.006	0.009	0.144
Low speed	0.017	0.035	0.372
No desire segments	0.007	0.014	0.206
No desire turns (max lat accel > 2.5 m/ss)	0.011	0.027	0.361