Model analysis report for

▶
root: {} 11 keys
- report_name: 'rldriving_analyselat'
- output_dir: null
- save_tmp: false
- read_only: false
- limit: 10000
- shuffle: false
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 1000
  - job_name: 'rldriving_analyselat'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: 'caches/workstation-harald_2025-04-25_17-31-30'
  - force_local: false
- route_list: '/home/batman/xx/routes/default_test_2k'
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 29 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - future_observation_range_seconds: null
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'DEPTH'
    - inpaint_mode: 'NONE'
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 10 keys
    - supercombo: '846c53a4-b886-4942-a4d3-7ea3aff5be8d/400'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - calibrate_speed: false
    - use_allow_throttle: false
    - use_prev_desired_curv: false
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: false
    - use_raw_plan_long: true
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1523 / 1657 aka 91.913%

better :1146 / 1657 aka 69.161%

best (0.5m):467 / 1657 aka 28.183%

global mae x: 0.227

global mae y: 0.234

global mae y of lanecenter (when detected) in GT: 0.209

global mae y of lanecenter (when detected) in Sim: 0.219

avg resets: 0.277

Mean abs curvature: 0.212 deg/m

Mean abs curvature rate: 0.362 deg/ms

Mean abs lateral accel: 0.392 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.696 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :506 / 510 aka 99.216%

better :479 / 510 aka 93.922%

best (0.5m):278 / 510 aka 54.510%

global mae x: 0.155

global mae y: 0.178

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.180

avg resets: 0.016

Mean abs curvature: 0.050 deg/m

Mean abs curvature rate: 0.142 deg/ms

Mean abs lateral accel: 0.207 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.554 m/(sss)

Mean abs lateral jerk GT: 0.171 m/(sss)

Low speed

Metrics

passed :302 / 347 aka 87.032%

better :200 / 347 aka 57.637%

best (0.5m):81 / 347 aka 23.343%

global mae x: 0.302

global mae y: 0.267

global mae y of lanecenter (when detected) in GT: 0.266

global mae y of lanecenter (when detected) in Sim: 0.242

avg resets: 0.476

Mean abs curvature: 0.410 deg/m

Mean abs curvature rate: 0.743 deg/ms

Mean abs lateral accel: 0.335 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.639 m/(sss)

Mean abs lateral jerk GT: 0.228 m/(sss)

No desire segments

Metrics

passed :911 / 938 aka 97.122%

better :809 / 938 aka 86.247%

best (0.5m):406 / 938 aka 43.284%

global mae x: 0.174

global mae y: 0.199

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.191

avg resets: 0.074

Mean abs curvature: 0.117 deg/m

Mean abs curvature rate: 0.212 deg/ms

Mean abs lateral accel: 0.344 m/(ss)

Mean abs lateral accel GT: 0.284 m/(ss)

Mean abs lateral jerk: 0.623 m/(sss)

Mean abs lateral jerk GT: 0.236 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :121 / 133 aka 90.977%

better :79 / 133 aka 59.398%

best (0.5m):26 / 133 aka 19.549%

global mae x: 0.247

global mae y: 0.257

global mae y of lanecenter (when detected) in GT: 0.261

global mae y of lanecenter (when detected) in Sim: 0.237

avg resets: 0.218

Mean abs curvature: 0.281 deg/m

Mean abs curvature rate: 0.371 deg/ms

Mean abs lateral accel: 0.674 m/(ss)

Mean abs lateral accel GT: 0.646 m/(ss)

Mean abs lateral jerk: 0.822 m/(sss)

Mean abs lateral jerk GT: 0.396 m/(sss)

Left lane change segments

112/120 aka 93.33%

better: 83/120 aka 69.17%

best (0.5m): 28/120 aka 23.33%

mean max error C3: 0.815671

mean max error C2: 1.041204

global mae: 0.227419

avg resets: 0.075000

Right lane change segments

138/150 aka 92.00%

better: 96/150 aka 64.00%

best (0.5m): 20/150 aka 13.33%

mean max error C3: 1.021437

mean max error C2: 1.053544

global mae: 0.245807

avg resets: 0.160000

Left turn segments

70/79 aka 88.61%

better: 32/79 aka 40.51%

best (0.5m): 3/79 aka 3.80%

mean max error C3: 1.278615

mean max error C2: 1.292744

global mae: 0.282114

avg resets: 0.860759

Right turn segments

72/90 aka 80.00%

better: 31/90 aka 34.44%

best (0.5m): 5/90 aka 5.56%

mean max error C3: 1.271337

mean max error C2: 1.436895

global mae: 0.294269

avg resets: 0.900000

Left keep segments

28/33 aka 84.85%

better: 17/33 aka 51.52%

best (0.5m): 0/33 aka 0.00%

mean max error C3: 1.256868

mean max error C2: 1.302682

global mae: 0.258426

avg resets: 0.424242

Right keep segments

24/26 aka 92.31%

better: 13/26 aka 50.00%

best (0.5m): 2/26 aka 7.69%

mean max error C3: 1.036347

mean max error C2: 1.094186

global mae: 0.263992

avg resets: 0.115385

Plan y error in turns (> 6°/s)

Mean path error : -0.044 meters

Mean absolute path error : 0.435 meters

Turn bias test (y_pos at 50m > 3m)

got 28799 left samples

got 30815 right samples

mean left gt: -0.0064031268

mean right gt: -0.006257103

mean turn lanecenter error (GT pos, GT ll): left = 0.21541060507297516, right = -0.24762769043445587

mean turn lanecenter error (sim pos, GT ll): left = 0.21259412169456482, right = -0.2503962814807892

mean turn lanecenter error (GT pos, model ll): left = 0.2088928520679474, right = -0.2521941363811493

mean turn lanecenter error (sim pos, model ll): left = 0.20313960313796997, right = -0.22826199233531952

Mean abs curvature: 0.214646 deg/m

Mean abs curvature rate: 0.248959 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.171451 Hz, period 5.832580 s

Mean amplitude: 0.116109 m

Mean ping-pong significance (frequency x amplitude): 1.876875 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	166
turn_right	178
lane_change_left	234
lane_change_right	285
keep_left	80
keep_right	67

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-4.338m
turn_right	+4.746m
lane_change_left	-1.467m
lane_change_right	+1.383m
keep_left	-1.203m
keep_right	+0.979m

Mean desire duration - ground truth

Command	Duration
turn_left	6.802s
turn_right	6.235s
lane_change_left	3.316s
lane_change_right	3.496s
keep_left	3.741s
keep_right	3.840s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	2.38s	3.18s	3.72s	4.18s
turn_right	2.19s	2.89s	3.56s	4.18s
lane_change_left	1.03s	1.53s	2.22s	3.82s
lane_change_right	1.01s	1.57s	2.41s	3.89s
keep_left	1.40s	2.15s	2.93s	4.84s
keep_right	1.60s	2.29s	3.43s	5.16s

Group	Gas prob	Brake prob	Steer prob
All segments	0.010	0.023	0.261
Easy straight	0.006	0.009	0.139
Low speed	0.017	0.036	0.371
No desire segments	0.007	0.014	0.200
No desire turns (max lat accel > 2.5 m/ss)	0.011	0.028	0.355