Model analysis report for

▶
root: {} 11 keys
- report_name: 'rldriving_analyselat'
- output_dir: null
- save_tmp: false
- read_only: false
- limit: 10000
- shuffle: false
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 1000
  - job_name: 'rldriving_analyselat'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: 'caches/workstation-yassine_2025-05-11_17-17-52'
  - force_local: false
- route_list: '/home/batman/xx/routes/default_test_2k'
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 30 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'DEPTH'
    - inpaint_mode: 'NONE'
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 10 keys
    - supercombo: '5e4cb3d3-b9cc-45c7-a476-38083e75029c/400'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - calibrate_speed: false
    - use_allow_throttle: false
    - use_prev_desired_curv: false
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: true
    - use_raw_plan_long: true
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1484 / 1657 aka 89.559%

better :1098 / 1657 aka 66.264%

best (0.5m):393 / 1657 aka 23.718%

global mae x: 0.231

global mae y: 0.253

global mae y of lanecenter (when detected) in GT: 0.209

global mae y of lanecenter (when detected) in Sim: 0.230

avg resets: 0.344

Mean abs curvature: 0.203 deg/m

Mean abs curvature rate: 0.346 deg/ms

Mean abs lateral accel: 0.367 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.559 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :502 / 510 aka 98.431%

better :475 / 510 aka 93.137%

best (0.5m):243 / 510 aka 47.647%

global mae x: 0.159

global mae y: 0.189

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.186

avg resets: 0.024

Mean abs curvature: 0.040 deg/m

Mean abs curvature rate: 0.094 deg/ms

Mean abs lateral accel: 0.177 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.385 m/(sss)

Mean abs lateral jerk GT: 0.171 m/(sss)

Low speed

Metrics

passed :301 / 347 aka 86.744%

better :193 / 347 aka 55.620%

best (0.5m):64 / 347 aka 18.444%

global mae x: 0.305

global mae y: 0.280

global mae y of lanecenter (when detected) in GT: 0.266

global mae y of lanecenter (when detected) in Sim: 0.245

avg resets: 0.582

Mean abs curvature: 0.403 deg/m

Mean abs curvature rate: 0.708 deg/ms

Mean abs lateral accel: 0.316 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.515 m/(sss)

Mean abs lateral jerk GT: 0.228 m/(sss)

No desire segments

Metrics

passed :898 / 938 aka 95.736%

better :785 / 938 aka 83.689%

best (0.5m):345 / 938 aka 36.780%

global mae x: 0.182

global mae y: 0.214

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.200

avg resets: 0.098

Mean abs curvature: 0.108 deg/m

Mean abs curvature rate: 0.175 deg/ms

Mean abs lateral accel: 0.320 m/(ss)

Mean abs lateral accel GT: 0.284 m/(ss)

Mean abs lateral jerk: 0.475 m/(sss)

Mean abs lateral jerk GT: 0.236 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :114 / 133 aka 85.714%

better :67 / 133 aka 50.376%

best (0.5m):16 / 133 aka 12.030%

global mae x: 0.267

global mae y: 0.289

global mae y of lanecenter (when detected) in GT: 0.261

global mae y of lanecenter (when detected) in Sim: 0.262

avg resets: 0.301

Mean abs curvature: 0.267 deg/m

Mean abs curvature rate: 0.368 deg/ms

Mean abs lateral accel: 0.659 m/(ss)

Mean abs lateral accel GT: 0.645 m/(ss)

Mean abs lateral jerk: 0.714 m/(sss)

Mean abs lateral jerk GT: 0.396 m/(sss)

Left lane change segments

109/120 aka 90.83%

better: 78/120 aka 65.00%

best (0.5m): 21/120 aka 17.50%

mean max error C3: 0.900414

mean max error C2: 1.112690

global mae: 0.251370

avg resets: 0.108333

Right lane change segments

132/150 aka 88.00%

better: 89/150 aka 59.33%

best (0.5m): 13/150 aka 8.67%

mean max error C3: 1.033447

mean max error C2: 1.164000

global mae: 0.280310

avg resets: 0.200000

Left turn segments

65/79 aka 82.28%

better: 24/79 aka 30.38%

best (0.5m): 0/79 aka 0.00%

mean max error C3: 1.422115

mean max error C2: 1.436168

global mae: 0.300767

avg resets: 0.835443

Right turn segments

75/90 aka 83.33%

better: 29/90 aka 32.22%

best (0.5m): 1/90 aka 1.11%

mean max error C3: 1.328896

mean max error C2: 1.384656

global mae: 0.327219

avg resets: 1.200000

Left keep segments

23/33 aka 69.70%

better: 13/33 aka 39.39%

best (0.5m): 3/33 aka 9.09%

mean max error C3: 1.601592

mean max error C2: 1.417103

global mae: 0.306841

avg resets: 0.515152

Right keep segments

21/26 aka 80.77%

better: 14/26 aka 53.85%

best (0.5m): 4/26 aka 15.38%

mean max error C3: 1.226505

mean max error C2: 1.217571

global mae: 0.309486

avg resets: 0.192308

Plan y error in turns (> 6°/s)

Mean path error : -0.038 meters

Mean absolute path error : 0.461 meters

Turn bias test (y_pos at 50m > 3m)

got 28799 left samples

got 30815 right samples

mean left gt: -0.0012418638

mean right gt: -0.016655212

mean turn lanecenter error (GT pos, GT ll): left = 0.21541060507297516, right = -0.24762772023677826

mean turn lanecenter error (sim pos, GT ll): left = 0.2137027233839035, right = -0.26721566915512085

mean turn lanecenter error (GT pos, model ll): left = 0.20182618498802185, right = -0.2499084621667862

mean turn lanecenter error (sim pos, model ll): left = 0.18343551456928253, right = -0.2373652458190918

Mean abs curvature: 0.213591 deg/m

Mean abs curvature rate: 0.212998 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.165705 Hz, period 6.034825 s

Mean amplitude: 0.135307 m

Mean ping-pong significance (frequency x amplitude): 2.140613 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	166
turn_right	178
lane_change_left	234
lane_change_right	285
keep_left	80
keep_right	67

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-5.374m
turn_right	+5.288m
lane_change_left	-1.410m
lane_change_right	+1.314m
keep_left	-1.072m
keep_right	+0.866m

Mean desire duration - ground truth

Command	Duration
turn_left	6.802s
turn_right	6.235s
lane_change_left	3.316s
lane_change_right	3.496s
keep_left	3.741s
keep_right	3.840s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	2.24s	2.89s	3.50s	4.25s
turn_right	2.29s	3.07s	3.76s	4.40s
lane_change_left	1.10s	1.67s	2.45s	3.98s
lane_change_right	1.11s	1.74s	2.55s	4.08s
keep_left	1.47s	2.32s	3.36s	4.64s
keep_right	1.67s	2.64s	3.65s	4.87s

Group	Gas prob	Brake prob	Steer prob
All segments	0.010	0.024	0.268
Easy straight	0.007	0.009	0.145
Low speed	0.018	0.037	0.376
No desire segments	0.008	0.014	0.207
No desire turns (max lat accel > 2.5 m/ss)	0.011	0.029	0.362