Model analysis report for

▶
root: {} 11 keys
- report_name: 'rldriving_analyselat'
- output_dir: null
- save_tmp: false
- read_only: false
- limit: 10000
- shuffle: false
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 1000
  - job_name: 'rldriving_analyselat'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: 'caches/workstation-harald_2025-04-28_22-59-34'
  - force_local: false
- route_list: '/home/batman/xx/routes/default_test_2k'
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 30 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: false
    - future_observation_range_seconds: null
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'DEPTH'
    - inpaint_mode: 'NONE'
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 10 keys
    - supercombo: '154640c6-9151-44f9-bd44-c3d39aaf52d7/400'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - calibrate_speed: false
    - use_allow_throttle: false
    - use_prev_desired_curv: false
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: false
    - use_raw_plan_long: true
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1506 / 1657 aka 90.887%

better :1143 / 1657 aka 68.980%

best (0.5m):500 / 1657 aka 30.175%

global mae x: 0.232

global mae y: 0.235

global mae y of lanecenter (when detected) in GT: 0.209

global mae y of lanecenter (when detected) in Sim: 0.212

avg resets: 0.288

Mean abs curvature: 0.209 deg/m

Mean abs curvature rate: 0.346 deg/ms

Mean abs lateral accel: 0.401 m/(ss)

Mean abs lateral accel GT: 0.336 m/(ss)

Mean abs lateral jerk: 0.737 m/(sss)

Mean abs lateral jerk GT: 0.272 m/(sss)

Easy straight

Metrics

passed :505 / 510 aka 99.020%

better :484 / 510 aka 94.902%

best (0.5m):314 / 510 aka 61.569%

global mae x: 0.159

global mae y: 0.172

global mae y of lanecenter (when detected) in GT: 0.161

global mae y of lanecenter (when detected) in Sim: 0.168

avg resets: 0.016

Mean abs curvature: 0.049 deg/m

Mean abs curvature rate: 0.138 deg/ms

Mean abs lateral accel: 0.224 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.644 m/(sss)

Mean abs lateral jerk GT: 0.171 m/(sss)

Low speed

Metrics

passed :295 / 347 aka 85.014%

better :199 / 347 aka 57.349%

best (0.5m):80 / 347 aka 23.055%

global mae x: 0.310

global mae y: 0.274

global mae y of lanecenter (when detected) in GT: 0.266

global mae y of lanecenter (when detected) in Sim: 0.249

avg resets: 0.507

Mean abs curvature: 0.404 deg/m

Mean abs curvature rate: 0.680 deg/ms

Mean abs lateral accel: 0.329 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.571 m/(sss)

Mean abs lateral jerk GT: 0.228 m/(sss)

No desire segments

Metrics

passed :900 / 938 aka 95.949%

better :814 / 938 aka 86.780%

best (0.5m):444 / 938 aka 47.335%

global mae x: 0.180

global mae y: 0.198

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.181

avg resets: 0.075

Mean abs curvature: 0.116 deg/m

Mean abs curvature rate: 0.202 deg/ms

Mean abs lateral accel: 0.358 m/(ss)

Mean abs lateral accel GT: 0.284 m/(ss)

Mean abs lateral jerk: 0.685 m/(sss)

Mean abs lateral jerk GT: 0.236 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :113 / 133 aka 84.962%

better :74 / 133 aka 55.639%

best (0.5m):21 / 133 aka 15.789%

global mae x: 0.255

global mae y: 0.274

global mae y of lanecenter (when detected) in GT: 0.261

global mae y of lanecenter (when detected) in Sim: 0.239

avg resets: 0.218

Mean abs curvature: 0.276 deg/m

Mean abs curvature rate: 0.341 deg/ms

Mean abs lateral accel: 0.672 m/(ss)

Mean abs lateral accel GT: 0.646 m/(ss)

Mean abs lateral jerk: 0.773 m/(sss)

Mean abs lateral jerk GT: 0.396 m/(sss)

Left lane change segments

109/120 aka 90.83%

better: 76/120 aka 63.33%

best (0.5m): 20/120 aka 16.67%

mean max error C3: 0.880566

mean max error C2: 1.149015

global mae: 0.233687

avg resets: 0.108333

Right lane change segments

138/150 aka 92.00%

better: 96/150 aka 64.00%

best (0.5m): 21/150 aka 14.00%

mean max error C3: 0.938702

mean max error C2: 1.058813

global mae: 0.247212

avg resets: 0.166667

Left turn segments

69/79 aka 87.34%

better: 26/79 aka 32.91%

best (0.5m): 2/79 aka 2.53%

mean max error C3: 1.276861

mean max error C2: 1.340555

global mae: 0.273898

avg resets: 0.898734

Right turn segments

72/90 aka 80.00%

better: 33/90 aka 36.67%

best (0.5m): 5/90 aka 5.56%

mean max error C3: 1.216388

mean max error C2: 1.465441

global mae: 0.304161

avg resets: 0.922222

Left keep segments

26/33 aka 78.79%

better: 16/33 aka 48.48%

best (0.5m): 1/33 aka 3.03%

mean max error C3: 1.381117

mean max error C2: 1.291913

global mae: 0.292919

avg resets: 0.303030

Right keep segments

21/26 aka 80.77%

better: 16/26 aka 61.54%

best (0.5m): 2/26 aka 7.69%

mean max error C3: 1.251559

mean max error C2: 1.176989

global mae: 0.274849

avg resets: 0.230769

Plan y error in turns (> 6°/s)

Mean path error : -0.027 meters

Mean absolute path error : 0.438 meters

Turn bias test (y_pos at 50m > 3m)

got 28799 left samples

got 30815 right samples

mean left gt: 0.020500297

mean right gt: 0.020110112

mean turn lanecenter error (GT pos, GT ll): left = 0.21541060507297516, right = -0.24762769043445587

mean turn lanecenter error (sim pos, GT ll): left = 0.2410910725593567, right = -0.232634037733078

mean turn lanecenter error (GT pos, model ll): left = 0.20335346460342407, right = -0.250313401222229

mean turn lanecenter error (sim pos, model ll): left = 0.22427427768707275, right = -0.2144901305437088

Mean abs curvature: 0.214245 deg/m

Mean abs curvature rate: 0.230521 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.176657 Hz, period 5.660694 s

Mean amplitude: 0.112712 m

Mean ping-pong significance (frequency x amplitude): 1.881242 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	166
turn_right	178
lane_change_left	234
lane_change_right	285
keep_left	80
keep_right	67

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-5.340m
turn_right	+4.881m
lane_change_left	-1.468m
lane_change_right	+1.318m
keep_left	-1.174m
keep_right	+0.860m

Mean desire duration - ground truth

Command	Duration
turn_left	6.802s
turn_right	6.235s
lane_change_left	3.316s
lane_change_right	3.496s
keep_left	3.741s
keep_right	3.840s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	2.30s	3.04s	3.53s	4.13s
turn_right	2.20s	3.02s	3.68s	4.40s
lane_change_left	1.03s	1.57s	2.34s	4.07s
lane_change_right	1.09s	1.61s	2.50s	4.22s
keep_left	1.42s	2.33s	3.28s	4.90s
keep_right	1.61s	2.57s	3.75s	5.15s

Group	Gas prob	Brake prob	Steer prob
All segments	0.010	0.023	0.264
Easy straight	0.006	0.009	0.142
Low speed	0.017	0.037	0.373
No desire segments	0.008	0.014	0.203
No desire turns (max lat accel > 2.5 m/ss)	0.011	0.028	0.357