Model analysis report for

▶
root: {} 11 keys
- report_name: 'rldriving_analyselat'
- output_dir: null
- save_tmp: false
- read_only: false
- limit: 10000
- shuffle: false
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 1000
  - job_name: 'rldriving_analyselat'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: 'caches/workstation-yassine_2025-05-01_09-40-16'
  - force_local: false
- route_list: '/home/batman/xx/routes/default_test_2k'
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 30 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: false
    - future_observation_range_seconds: null
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'DEPTH'
    - inpaint_mode: 'NONE'
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 10 keys
    - supercombo: 'e9237324-4b92-48f5-acaa-ebdf7fe46339/400'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - calibrate_speed: false
    - use_allow_throttle: false
    - use_prev_desired_curv: false
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: false
    - use_raw_plan_long: true
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1441 / 1657 aka 86.964%

better :1045 / 1657 aka 63.066%

best (0.5m):421 / 1657 aka 25.407%

global mae x: 0.233

global mae y: 0.267

global mae y of lanecenter (when detected) in GT: 0.209

global mae y of lanecenter (when detected) in Sim: 0.225

avg resets: 0.305

Mean abs curvature: 0.197 deg/m

Mean abs curvature rate: 0.265 deg/ms

Mean abs lateral accel: 0.375 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.540 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :501 / 510 aka 98.235%

better :478 / 510 aka 93.725%

best (0.5m):265 / 510 aka 51.961%

global mae x: 0.159

global mae y: 0.186

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.176

avg resets: 0.027

Mean abs curvature: 0.042 deg/m

Mean abs curvature rate: 0.095 deg/ms

Mean abs lateral accel: 0.199 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.469 m/(sss)

Mean abs lateral jerk GT: 0.171 m/(sss)

Low speed

Metrics

passed :283 / 347 aka 81.556%

better :169 / 347 aka 48.703%

best (0.5m):60 / 347 aka 17.291%

global mae x: 0.315

global mae y: 0.326

global mae y of lanecenter (when detected) in GT: 0.266

global mae y of lanecenter (when detected) in Sim: 0.273

avg resets: 0.533

Mean abs curvature: 0.376 deg/m

Mean abs curvature rate: 0.512 deg/ms

Mean abs lateral accel: 0.305 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.386 m/(sss)

Mean abs lateral jerk GT: 0.228 m/(sss)

No desire segments

Metrics

passed :879 / 938 aka 93.710%

better :771 / 938 aka 82.196%

best (0.5m):367 / 938 aka 39.126%

global mae x: 0.180

global mae y: 0.220

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.191

avg resets: 0.099

Mean abs curvature: 0.108 deg/m

Mean abs curvature rate: 0.150 deg/ms

Mean abs lateral accel: 0.335 m/(ss)

Mean abs lateral accel GT: 0.284 m/(ss)

Mean abs lateral jerk: 0.506 m/(sss)

Mean abs lateral jerk GT: 0.236 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :104 / 133 aka 78.195%

better :57 / 133 aka 42.857%

best (0.5m):15 / 133 aka 11.278%

global mae x: 0.252

global mae y: 0.314

global mae y of lanecenter (when detected) in GT: 0.261

global mae y of lanecenter (when detected) in Sim: 0.259

avg resets: 0.293

Mean abs curvature: 0.269 deg/m

Mean abs curvature rate: 0.268 deg/ms

Mean abs lateral accel: 0.657 m/(ss)

Mean abs lateral accel GT: 0.646 m/(ss)

Mean abs lateral jerk: 0.603 m/(sss)

Mean abs lateral jerk GT: 0.396 m/(sss)

Left lane change segments

107/120 aka 89.17%

better: 71/120 aka 59.17%

best (0.5m): 23/120 aka 19.17%

mean max error C3: 0.951576

mean max error C2: 1.157070

global mae: 0.261058

avg resets: 0.116667

Right lane change segments

133/150 aka 88.67%

better: 84/150 aka 56.00%

best (0.5m): 18/150 aka 12.00%

mean max error C3: 1.079263

mean max error C2: 1.155791

global mae: 0.274909

avg resets: 0.186667

Left turn segments

64/79 aka 81.01%

better: 18/79 aka 22.78%

best (0.5m): 1/79 aka 1.27%

mean max error C3: 1.521963

mean max error C2: 1.462429

global mae: 0.341138

avg resets: 0.848101

Right turn segments

63/90 aka 70.00%

better: 17/90 aka 18.89%

best (0.5m): 1/90 aka 1.11%

mean max error C3: 1.471113

mean max error C2: 1.789128

global mae: 0.392616

avg resets: 0.811111

Left keep segments

26/33 aka 78.79%

better: 16/33 aka 48.48%

best (0.5m): 3/33 aka 9.09%

mean max error C3: 1.435164

mean max error C2: 1.360384

global mae: 0.305450

avg resets: 0.333333

Right keep segments

20/26 aka 76.92%

better: 15/26 aka 57.69%

best (0.5m): 3/26 aka 11.54%

mean max error C3: 1.549318

mean max error C2: 1.259095

global mae: 0.319327

avg resets: 0.153846

Plan y error in turns (> 6°/s)

Mean path error : 0.002 meters

Mean absolute path error : 0.525 meters

Turn bias test (y_pos at 50m > 3m)

got 28799 left samples

got 30815 right samples

mean left gt: -0.0121567855

mean right gt: -0.0058193086

mean turn lanecenter error (GT pos, GT ll): left = 0.21541060507297516, right = -0.24762769043445587

mean turn lanecenter error (sim pos, GT ll): left = 0.21068792045116425, right = -0.24904362857341766

mean turn lanecenter error (GT pos, model ll): left = 0.20575878024101257, right = -0.25365376472473145

mean turn lanecenter error (sim pos, model ll): left = 0.192451611161232, right = -0.22147683799266815

Mean abs curvature: 0.213414 deg/m

Mean abs curvature rate: 0.188600 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.171642 Hz, period 5.826094 s

Mean amplitude: 0.126505 m

Mean ping-pong significance (frequency x amplitude): 2.080166 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	166
turn_right	178
lane_change_left	234
lane_change_right	285
keep_left	80
keep_right	67

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-4.684m
turn_right	+5.095m
lane_change_left	-1.427m
lane_change_right	+1.314m
keep_left	-0.929m
keep_right	+0.984m

Mean desire duration - ground truth

Command	Duration
turn_left	6.802s
turn_right	6.235s
lane_change_left	3.316s
lane_change_right	3.496s
keep_left	3.741s
keep_right	3.840s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	2.39s	3.04s	3.72s	4.19s
turn_right	2.40s	3.07s	3.56s	4.05s
lane_change_left	1.14s	1.72s	2.54s	4.14s
lane_change_right	1.19s	1.83s	2.55s	4.41s
keep_left	1.55s	2.33s	3.50s	4.80s
keep_right	1.63s	2.35s	3.51s	5.06s

Group	Gas prob	Brake prob	Steer prob
All segments	0.010	0.024	0.267
Easy straight	0.006	0.009	0.144
Low speed	0.017	0.037	0.376
No desire segments	0.008	0.014	0.206
No desire turns (max lat accel > 2.5 m/ss)	0.011	0.028	0.364