Model analysis report for

▶
root: {} 11 keys
- report_name: 'rldriving_analyselat'
- output_dir: null
- save_tmp: false
- read_only: false
- limit: 10000
- shuffle: false
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 1000
  - job_name: 'rldriving_analyselat'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: 'caches/workstation-harald_2025-04-29_11-17-03'
  - force_local: false
- route_list: '/home/batman/xx/routes/default_test_2k'
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 30 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: false
    - future_observation_range_seconds: null
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'DEPTH'
    - inpaint_mode: 'NONE'
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 10 keys
    - supercombo: '8d0a1b3b-9972-4e53-b9c5-3e13e5e3e404/400'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - calibrate_speed: false
    - use_allow_throttle: false
    - use_prev_desired_curv: false
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: true
    - use_raw_plan_long: true
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1507 / 1657 aka 90.947%

better :1143 / 1657 aka 68.980%

best (0.5m):499 / 1657 aka 30.115%

global mae x: 0.222

global mae y: 0.234

global mae y of lanecenter (when detected) in GT: 0.209

global mae y of lanecenter (when detected) in Sim: 0.209

avg resets: 0.364

Mean abs curvature: 0.222 deg/m

Mean abs curvature rate: 0.472 deg/ms

Mean abs lateral accel: 0.429 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.964 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :504 / 510 aka 98.824%

better :485 / 510 aka 95.098%

best (0.5m):311 / 510 aka 60.980%

global mae x: 0.156

global mae y: 0.170

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.163

avg resets: 0.018

Mean abs curvature: 0.054 deg/m

Mean abs curvature rate: 0.176 deg/ms

Mean abs lateral accel: 0.284 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.950 m/(sss)

Mean abs lateral jerk GT: 0.171 m/(sss)

Low speed

Metrics

passed :303 / 347 aka 87.320%

better :203 / 347 aka 58.501%

best (0.5m):69 / 347 aka 19.885%

global mae x: 0.293

global mae y: 0.276

global mae y of lanecenter (when detected) in GT: 0.266

global mae y of lanecenter (when detected) in Sim: 0.248

avg resets: 0.602

Mean abs curvature: 0.434 deg/m

Mean abs curvature rate: 0.945 deg/ms

Mean abs lateral accel: 0.337 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.709 m/(sss)

Mean abs lateral jerk GT: 0.228 m/(sss)

No desire segments

Metrics

passed :903 / 938 aka 96.269%

better :804 / 938 aka 85.714%

best (0.5m):443 / 938 aka 47.228%

global mae x: 0.174

global mae y: 0.195

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.176

avg resets: 0.099

Mean abs curvature: 0.122 deg/m

Mean abs curvature rate: 0.267 deg/ms

Mean abs lateral accel: 0.394 m/(ss)

Mean abs lateral accel GT: 0.284 m/(ss)

Mean abs lateral jerk: 0.927 m/(sss)

Mean abs lateral jerk GT: 0.236 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :118 / 133 aka 88.722%

better :73 / 133 aka 54.887%

best (0.5m):18 / 133 aka 13.534%

global mae x: 0.255

global mae y: 0.267

global mae y of lanecenter (when detected) in GT: 0.261

global mae y of lanecenter (when detected) in Sim: 0.231

avg resets: 0.316

Mean abs curvature: 0.282 deg/m

Mean abs curvature rate: 0.471 deg/ms

Mean abs lateral accel: 0.681 m/(ss)

Mean abs lateral accel GT: 0.645 m/(ss)

Mean abs lateral jerk: 0.958 m/(sss)

Mean abs lateral jerk GT: 0.396 m/(sss)

Left lane change segments

111/120 aka 92.50%

better: 86/120 aka 71.67%

best (0.5m): 22/120 aka 18.33%

mean max error C3: 0.854086

mean max error C2: 1.052632

global mae: 0.229405

avg resets: 0.116667

Right lane change segments

137/150 aka 91.33%

better: 90/150 aka 60.00%

best (0.5m): 21/150 aka 14.00%

mean max error C3: 0.911385

mean max error C2: 1.106039

global mae: 0.250478

avg resets: 0.213333

Left turn segments

65/79 aka 82.28%

better: 32/79 aka 40.51%

best (0.5m): 1/79 aka 1.27%

mean max error C3: 1.379895

mean max error C2: 1.277219

global mae: 0.299376

avg resets: 0.987342

Right turn segments

77/90 aka 85.56%

better: 34/90 aka 37.78%

best (0.5m): 4/90 aka 4.44%

mean max error C3: 1.204948

mean max error C2: 1.389704

global mae: 0.296788

avg resets: 1.377778

Left keep segments

24/33 aka 72.73%

better: 14/33 aka 42.42%

best (0.5m): 2/33 aka 6.06%

mean max error C3: 1.573699

mean max error C2: 1.507357

global mae: 0.300504

avg resets: 0.333333

Right keep segments

23/26 aka 88.46%

better: 12/26 aka 46.15%

best (0.5m): 2/26 aka 7.69%

mean max error C3: 1.184829

mean max error C2: 1.217190

global mae: 0.280324

avg resets: 0.269231

Plan y error in turns (> 6°/s)

Mean path error : -0.032 meters

Mean absolute path error : 0.444 meters

Turn bias test (y_pos at 50m > 3m)

got 28799 left samples

got 30815 right samples

mean left gt: 0.0062582283

mean right gt: -0.023807894

mean turn lanecenter error (GT pos, GT ll): left = 0.21541060507297516, right = -0.24762769043445587

mean turn lanecenter error (sim pos, GT ll): left = 0.23029622435569763, right = -0.2675381898880005

mean turn lanecenter error (GT pos, model ll): left = 0.20206281542778015, right = -0.24821825325489044

mean turn lanecenter error (sim pos, model ll): left = 0.20917993783950806, right = -0.2344406098127365

Mean abs curvature: 0.213865 deg/m

Mean abs curvature rate: 0.276995 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.177243 Hz, period 5.641986 s

Mean amplitude: 0.111525 m

Mean ping-pong significance (frequency x amplitude): 1.857574 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	166
turn_right	178
lane_change_left	234
lane_change_right	285
keep_left	80
keep_right	67

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-5.757m
turn_right	+5.192m
lane_change_left	-1.528m
lane_change_right	+1.276m
keep_left	-1.052m
keep_right	+1.229m

Mean desire duration - ground truth

Command	Duration
turn_left	6.802s
turn_right	6.235s
lane_change_left	3.316s
lane_change_right	3.496s
keep_left	3.741s
keep_right	3.840s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	2.40s	3.10s	3.70s	4.28s
turn_right	2.23s	3.09s	3.78s	4.39s
lane_change_left	1.09s	1.52s	2.20s	3.64s
lane_change_right	1.05s	1.69s	2.47s	4.30s
keep_left	1.50s	2.31s	3.26s	4.60s
keep_right	1.77s	2.69s	3.66s	5.41s

Group	Gas prob	Brake prob	Steer prob
All segments	0.010	0.023	0.265
Easy straight	0.006	0.009	0.147
Low speed	0.017	0.037	0.373
No desire segments	0.008	0.014	0.205
No desire turns (max lat accel > 2.5 m/ss)	0.011	0.028	0.356