Model analysis report for

▶
root: {} 10 keys
- report_name: 'rldriving_driving.analyse_lat_rl'
- route_list: '/home/batman/xx/routes/default_test_2k'
- output_dir: null
- save_tmp: false
- read_only: false
- limit: 10000
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 10
  - job_name: 'rldriving_driving.analyse_lat_rl'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: 'caches/workstation-yassine_2025-06-20_19-45-51'
  - force_local: false
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 30 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'SIMPLE'
    - inpaint_mode: 'NONE'
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 12 keys
    - supercombo: '986745e3-b382-41a7-b15a-2cdcb664d072/700'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - calibrate_speed: false
    - use_allow_throttle: false
    - use_prev_desired_curv: true
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: false
    - use_raw_plan_long: false
    - desired_curv_smooth_seconds: false
    - desired_accel_smooth_seconds: false
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1556 / 1656 aka 93.961%

better :1182 / 1656 aka 71.377%

best (0.5m):512 / 1656 aka 30.918%

global mae x: 0.166

global mae y: 0.224

global mae y of lanecenter (when detected) in GT: 0.209

global mae y of lanecenter (when detected) in Sim: 0.219

avg resets: 0.224

Mean abs curvature: 0.221 deg/m

Mean abs curvature rate: 0.296 deg/ms

Mean abs lateral accel: 0.371 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.482 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :510 / 510 aka 100.000%

better :492 / 510 aka 96.471%

best (0.5m):311 / 510 aka 60.980%

global mae x: 0.106

global mae y: 0.172

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.182

avg resets: 0.016

Mean abs curvature: 0.048 deg/m

Mean abs curvature rate: 0.081 deg/ms

Mean abs lateral accel: 0.176 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.309 m/(sss)

Mean abs lateral jerk GT: 0.171 m/(sss)

Low speed

Metrics

passed :319 / 346 aka 92.197%

better :208 / 346 aka 60.116%

best (0.5m):84 / 346 aka 24.277%

global mae x: 0.223

global mae y: 0.248

global mae y of lanecenter (when detected) in GT: 0.266

global mae y of lanecenter (when detected) in Sim: 0.251

avg resets: 0.399

Mean abs curvature: 0.432 deg/m

Mean abs curvature rate: 0.597 deg/ms

Mean abs lateral accel: 0.317 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.438 m/(sss)

Mean abs lateral jerk GT: 0.228 m/(sss)

No desire segments

Metrics

passed :927 / 937 aka 98.933%

better :818 / 937 aka 87.300%

best (0.5m):435 / 937 aka 46.425%

global mae x: 0.125

global mae y: 0.191

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.193

avg resets: 0.060

Mean abs curvature: 0.120 deg/m

Mean abs curvature rate: 0.157 deg/ms

Mean abs lateral accel: 0.321 m/(ss)

Mean abs lateral accel GT: 0.284 m/(ss)

Mean abs lateral jerk: 0.396 m/(sss)

Mean abs lateral jerk GT: 0.236 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :126 / 133 aka 94.737%

better :72 / 133 aka 54.135%

best (0.5m):21 / 133 aka 15.789%

global mae x: 0.214

global mae y: 0.244

global mae y of lanecenter (when detected) in GT: 0.261

global mae y of lanecenter (when detected) in Sim: 0.240

avg resets: 0.180

Mean abs curvature: 0.297 deg/m

Mean abs curvature rate: 0.349 deg/ms

Mean abs lateral accel: 0.666 m/(ss)

Mean abs lateral accel GT: 0.646 m/(ss)

Mean abs lateral jerk: 0.639 m/(sss)

Mean abs lateral jerk GT: 0.396 m/(sss)

Left lane change segments

115/120 aka 95.83%

better: 94/120 aka 78.33%

best (0.5m): 33/120 aka 27.50%

mean max error C3: 0.682686

mean max error C2: 0.872204

global mae: 0.212922

avg resets: 0.033333

Right lane change segments

141/149 aka 94.63%

better: 108/149 aka 72.48%

best (0.5m): 31/149 aka 20.81%

mean max error C3: 0.836216

mean max error C2: 0.904792

global mae: 0.235787

avg resets: 0.087248

Left turn segments

68/79 aka 86.08%

better: 30/79 aka 37.97%

best (0.5m): 3/79 aka 3.80%

mean max error C3: 1.353090

mean max error C2: 1.298792

global mae: 0.274721

avg resets: 0.759494

Right turn segments

79/91 aka 86.81%

better: 37/91 aka 40.66%

best (0.5m): 2/91 aka 2.20%

mean max error C3: 1.225290

mean max error C2: 1.259042

global mae: 0.276929

avg resets: 0.769231

Left keep segments

30/33 aka 90.91%

better: 11/33 aka 33.33%

best (0.5m): 1/33 aka 3.03%

mean max error C3: 1.363814

mean max error C2: 1.286570

global mae: 0.260888

avg resets: 0.242424

Right keep segments

22/26 aka 84.62%

better: 9/26 aka 34.62%

best (0.5m): 1/26 aka 3.85%

mean max error C3: 1.342044

mean max error C2: 1.220375

global mae: 0.287177

avg resets: 0.230769

Plan y error in turns (> 6°/s)

Mean path error : 0.001 meters

Mean absolute path error : 0.423 meters

Turn bias test (y_pos at 50m > 3m)

got 29067 left samples

got 30815 right samples

mean left gt: -0.017145127

mean right gt: 0.016126117

mean turn lanecenter error (GT pos, GT ll): left = 0.21726644039154053, right = -0.24762772023677826

mean turn lanecenter error (sim pos, GT ll): left = 0.21795448660850525, right = -0.2349650114774704

mean turn lanecenter error (GT pos, model ll): left = 0.21736665070056915, right = -0.22390465438365936

mean turn lanecenter error (sim pos, model ll): left = 0.19643644988536835, right = -0.20575398206710815

Mean abs curvature: 0.213673 deg/m

Mean abs curvature rate: 0.198955 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.172980 Hz, period 5.781002 s

Mean amplitude: 0.124398 m

Mean ping-pong significance (frequency x amplitude): 2.072584 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	166
turn_right	179
lane_change_left	234
lane_change_right	284
keep_left	80
keep_right	67

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-1.363m
turn_right	+0.602m
lane_change_left	-1.603m
lane_change_right	+1.566m
keep_left	-0.572m
keep_right	+0.517m

Mean desire duration - ground truth

Command	Duration
turn_left	6.802s
turn_right	6.226s
lane_change_left	3.316s
lane_change_right	3.501s
keep_left	3.741s
keep_right	3.840s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	3.33s	4.14s	4.80s	5.52s
turn_right	2.93s	4.03s	4.83s	5.53s
lane_change_left	0.57s	1.22s	2.10s	3.65s
lane_change_right	0.69s	1.29s	2.21s	3.85s
keep_left	3.42s	4.75s	5.99s	7.12s
keep_right	3.44s	4.61s	5.83s	7.22s

Group	Gas prob	Brake prob	Steer prob
All segments	0.010	0.023	0.273
Easy straight	0.006	0.009	0.150
Low speed	0.017	0.037	0.378
No desire segments	0.008	0.014	0.214
No desire turns (max lat accel > 2.5 m/ss)	0.013	0.028	0.372