Model analysis report for

▶
root: {} 10 keys
- report_name: 'rldriving_driving.analyse_lat_rl'
- route_list: '/home/batman/xx/routes/default_test_2k'
- output_dir: null
- save_tmp: false
- read_only: false
- limit: 10000
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 10
  - job_name: 'rldriving_driving.analyse_lat_rl'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: 'caches/workstation-yassine_2025-06-20_19-46-11'
  - force_local: false
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 30 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'SIMPLE'
    - inpaint_mode: 'NONE'
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 12 keys
    - supercombo: 'b2714021-a7bc-41d6-8e1c-bfd20e59cc75/700'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - calibrate_speed: false
    - use_allow_throttle: false
    - use_prev_desired_curv: true
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: false
    - use_raw_plan_long: false
    - desired_curv_smooth_seconds: false
    - desired_accel_smooth_seconds: false
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1561 / 1652 aka 94.492%

better :1196 / 1652 aka 72.397%

best (0.5m):517 / 1652 aka 31.295%

global mae x: 0.166

global mae y: 0.224

global mae y of lanecenter (when detected) in GT: 0.209

global mae y of lanecenter (when detected) in Sim: 0.217

avg resets: 0.228

Mean abs curvature: 0.220 deg/m

Mean abs curvature rate: 0.298 deg/ms

Mean abs lateral accel: 0.370 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.476 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :507 / 508 aka 99.803%

better :491 / 508 aka 96.654%

best (0.5m):314 / 508 aka 61.811%

global mae x: 0.105

global mae y: 0.173

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.178

avg resets: 0.014

Mean abs curvature: 0.048 deg/m

Mean abs curvature rate: 0.082 deg/ms

Mean abs lateral accel: 0.176 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.303 m/(sss)

Mean abs lateral jerk GT: 0.171 m/(sss)

Low speed

Metrics

passed :327 / 344 aka 95.058%

better :211 / 344 aka 61.337%

best (0.5m):82 / 344 aka 23.837%

global mae x: 0.222

global mae y: 0.246

global mae y of lanecenter (when detected) in GT: 0.265

global mae y of lanecenter (when detected) in Sim: 0.250

avg resets: 0.416

Mean abs curvature: 0.428 deg/m

Mean abs curvature rate: 0.602 deg/ms

Mean abs lateral accel: 0.315 m/(ss)

Mean abs lateral accel GT: 0.304 m/(ss)

Mean abs lateral jerk: 0.434 m/(sss)

Mean abs lateral jerk GT: 0.227 m/(sss)

No desire segments

Metrics

passed :926 / 935 aka 99.037%

better :826 / 935 aka 88.342%

best (0.5m):439 / 935 aka 46.952%

global mae x: 0.124

global mae y: 0.191

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.191

avg resets: 0.058

Mean abs curvature: 0.120 deg/m

Mean abs curvature rate: 0.158 deg/ms

Mean abs lateral accel: 0.321 m/(ss)

Mean abs lateral accel GT: 0.284 m/(ss)

Mean abs lateral jerk: 0.392 m/(sss)

Mean abs lateral jerk GT: 0.236 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :128 / 132 aka 96.970%

better :76 / 132 aka 57.576%

best (0.5m):19 / 132 aka 14.394%

global mae x: 0.210

global mae y: 0.245

global mae y of lanecenter (when detected) in GT: 0.262

global mae y of lanecenter (when detected) in Sim: 0.243

avg resets: 0.174

Mean abs curvature: 0.295 deg/m

Mean abs curvature rate: 0.342 deg/ms

Mean abs lateral accel: 0.670 m/(ss)

Mean abs lateral accel GT: 0.649 m/(ss)

Mean abs lateral jerk: 0.643 m/(sss)

Mean abs lateral jerk GT: 0.398 m/(sss)

Left lane change segments

115/120 aka 95.83%

better: 93/120 aka 77.50%

best (0.5m): 30/120 aka 25.00%

mean max error C3: 0.676108

mean max error C2: 0.883412

global mae: 0.213274

avg resets: 0.050000

Right lane change segments

143/149 aka 95.97%

better: 104/149 aka 69.80%

best (0.5m): 30/149 aka 20.13%

mean max error C3: 0.847020

mean max error C2: 0.902428

global mae: 0.235243

avg resets: 0.093960

Left turn segments

69/78 aka 88.46%

better: 30/78 aka 38.46%

best (0.5m): 3/78 aka 3.85%

mean max error C3: 1.274636

mean max error C2: 1.294550

global mae: 0.275893

avg resets: 0.782051

Right turn segments

77/90 aka 85.56%

better: 36/90 aka 40.00%

best (0.5m): 4/90 aka 4.44%

mean max error C3: 1.293239

mean max error C2: 1.287907

global mae: 0.280276

avg resets: 0.766667

Left keep segments

29/33 aka 87.88%

better: 13/33 aka 39.39%

best (0.5m): 2/33 aka 6.06%

mean max error C3: 1.194496

mean max error C2: 1.249827

global mae: 0.260409

avg resets: 0.181818

Right keep segments

22/26 aka 84.62%

better: 12/26 aka 46.15%

best (0.5m): 2/26 aka 7.69%

mean max error C3: 1.293227

mean max error C2: 1.177867

global mae: 0.273350

avg resets: 0.230769

Plan y error in turns (> 6°/s)

Mean path error : -0.006 meters

Mean absolute path error : 0.424 meters

Turn bias test (y_pos at 50m > 3m)

got 29069 left samples

got 30815 right samples

mean left gt: -0.002993583

mean right gt: 0.003676542

mean turn lanecenter error (GT pos, GT ll): left = 0.21726641058921814, right = -0.24762772023677826

mean turn lanecenter error (sim pos, GT ll): left = 0.2285427749156952, right = -0.2545383870601654

mean turn lanecenter error (GT pos, model ll): left = 0.21672022342681885, right = -0.23201343417167664

mean turn lanecenter error (sim pos, model ll): left = 0.19700108468532562, right = -0.22559741139411926

Mean abs curvature: 0.213259 deg/m

Mean abs curvature rate: 0.200585 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.174816 Hz, period 5.720307 s

Mean amplitude: 0.125102 m

Mean ping-pong significance (frequency x amplitude): 2.094779 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	165
turn_right	177
lane_change_left	234
lane_change_right	283
keep_left	80
keep_right	67

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-1.115m
turn_right	-0.193m
lane_change_left	-1.612m
lane_change_right	+1.542m
keep_left	-0.649m
keep_right	+0.606m

Mean desire duration - ground truth

Command	Duration
turn_left	6.792s
turn_right	6.220s
lane_change_left	3.316s
lane_change_right	3.497s
keep_left	3.741s
keep_right	3.840s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	3.32s	4.28s	4.97s	5.71s
turn_right	3.19s	4.06s	4.66s	5.40s
lane_change_left	0.56s	1.20s	2.13s	3.61s
lane_change_right	0.60s	1.29s	2.18s	3.62s
keep_left	3.64s	4.91s	5.87s	7.10s
keep_right	3.33s	4.56s	5.95s	7.10s

Group	Gas prob	Brake prob	Steer prob
All segments	0.010	0.023	0.273
Easy straight	0.006	0.008	0.150
Low speed	0.017	0.036	0.377
No desire segments	0.008	0.014	0.214
No desire turns (max lat accel > 2.5 m/ss)	0.013	0.028	0.373