Model analysis report for

▶
root: {} 12 keys
- report_name: 'mlsimdriving_driving.analyse_lat_mlsim'
- route_list: '/code.nfs/branches/caches/workstation-harald_2026-01-07_09-23-07/xx/datasets/lists/train_500k_20240617_test_2k.txt'
- output_dir: null
- save_tmp: false
- read_only: false
- shuffle: false
- limit: 10000
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 10
  - job_name: 'mlsimdriving_driving.analyse_lat_mlsim'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - unique_id: null
  - codedir: '/code.nfs/branches/caches/workstation-harald_2026-01-07_09-23-07/xx'
  - use_local_codedir: false
- ▶
  filter: {} 3 keys
  - min_hist: 100
  - max_hist: -60
  - max_fidx: 1200
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 34 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2025_07'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - ▶
      physics: {} 3 keys
      - type: 'BICYCLE'
      - ▶
        bicycle_model_config: {} 1 keys
        
        vehicle_platform: 'HONDA_CIVIC'
      - ▶
        ml_model_config: {} 11 keys
        
        platform: 'TOYOTA_COROLLA_TSS2'
        
        eid: 'effd03d9-472d-44a1-910a-6d49a41eb63f/50'
        
        y_range: [-3.0, 3.0]
        
        vocab_size: 1024
        
        temperature: 0.5
        
        use_gt_features: false
        
        platform_vec_size: 256
        
        platform_map_path: '/code.nfs/branches/caches/workstation-harald_2026-01-07_09-23-07/xx/arena/envs/lib/physics_models/assets/platform_map.txt'
        
        lateral_tuning_type: 'AUTO'
        
        ▶
        lateral_tuning_params: {} 0 keys
        
        ▶
        lateral_initial_state_params: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_decoder_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'DEPTH'
    - inpaint_mode: 'NONE'
    - initial_speed: null
    - allow_any_frame: false
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 8 keys
    - supercombo: 'f1d30a23-4122-400a-80a6-557502284c36/200'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - use_allow_throttle: false
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - desired_curv_smooth_seconds: false
    - desired_accel_smooth_seconds: false
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1455 / 1690 aka 86.095%

better :1069 / 1690 aka 63.254%

best (0.5m):385 / 1690 aka 22.781%

global mae x: 0.171

global mae y: 0.272

global mae y of lanecenter (when detected) in GT: 0.210

global mae y of lanecenter (when detected) in Sim: 0.233

avg resets: 0.309

Mean abs curvature: 0.192 deg/m

Mean abs curvature rate: 0.260 deg/ms

Mean abs lateral accel: 0.367 m/(ss)

Mean abs lateral accel GT: 0.337 m/(ss)

Mean abs lateral jerk: 0.446 m/(sss)

Mean abs lateral jerk GT: 0.271 m/(sss)

Easy straight

Metrics

passed :509 / 515 aka 98.835%

better :484 / 515 aka 93.981%

best (0.5m):251 / 515 aka 48.738%

global mae x: 0.107

global mae y: 0.192

global mae y of lanecenter (when detected) in GT: 0.160

global mae y of lanecenter (when detected) in Sim: 0.185

avg resets: 0.019

Mean abs curvature: 0.041 deg/m

Mean abs curvature rate: 0.083 deg/ms

Mean abs lateral accel: 0.182 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.338 m/(sss)

Mean abs lateral jerk GT: 0.170 m/(sss)

Low speed

Metrics

passed :282 / 361 aka 78.116%

better :181 / 361 aka 50.139%

best (0.5m):67 / 361 aka 18.560%

global mae x: 0.232

global mae y: 0.323

global mae y of lanecenter (when detected) in GT: 0.264

global mae y of lanecenter (when detected) in Sim: 0.281

avg resets: 0.529

Mean abs curvature: 0.365 deg/m

Mean abs curvature rate: 0.502 deg/ms

Mean abs lateral accel: 0.308 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.379 m/(sss)

Mean abs lateral jerk GT: 0.229 m/(sss)

No desire segments

Metrics

passed :892 / 949 aka 93.994%

better :779 / 949 aka 82.086%

best (0.5m):344 / 949 aka 36.249%

global mae x: 0.126

global mae y: 0.223

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.202

avg resets: 0.093

Mean abs curvature: 0.103 deg/m

Mean abs curvature rate: 0.135 deg/ms

Mean abs lateral accel: 0.326 m/(ss)

Mean abs lateral accel GT: 0.287 m/(ss)

Mean abs lateral jerk: 0.393 m/(sss)

Mean abs lateral jerk GT: 0.237 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :109 / 134 aka 81.343%

better :62 / 134 aka 46.269%

best (0.5m):14 / 134 aka 10.448%

global mae x: 0.209

global mae y: 0.302

global mae y of lanecenter (when detected) in GT: 0.252

global mae y of lanecenter (when detected) in Sim: 0.247

avg resets: 0.246

Mean abs curvature: 0.255 deg/m

Mean abs curvature rate: 0.247 deg/ms

Mean abs lateral accel: 0.676 m/(ss)

Mean abs lateral accel GT: 0.667 m/(ss)

Mean abs lateral jerk: 0.558 m/(sss)

Mean abs lateral jerk GT: 0.407 m/(sss)

Left lane change segments

113/128 aka 88.28%

better: 85/128 aka 66.41%

best (0.5m): 16/128 aka 12.50%

mean max error C3: 1.004249

mean max error C2: 1.085501

global mae: 0.253134

avg resets: 0.125000

Right lane change segments

127/145 aka 87.59%

better: 84/145 aka 57.93%

best (0.5m): 15/145 aka 10.34%

mean max error C3: 1.107439

mean max error C2: 1.133649

global mae: 0.272905

avg resets: 0.206897

Left turn segments

63/85 aka 74.12%

better: 21/85 aka 24.71%

best (0.5m): 0/85 aka 0.00%

mean max error C3: 1.665635

mean max error C2: 1.588818

global mae: 0.341797

avg resets: 1.035294

Right turn segments

63/89 aka 70.79%

better: 23/89 aka 25.84%

best (0.5m): 3/89 aka 3.37%

mean max error C3: 1.565196

mean max error C2: 1.583843

global mae: 0.359498

avg resets: 0.955056

Left keep segments

30/36 aka 83.33%

better: 15/36 aka 41.67%

best (0.5m): 1/36 aka 2.78%

mean max error C3: 1.038309

mean max error C2: 1.424945

global mae: 0.330475

avg resets: 0.250000

Right keep segments

19/25 aka 76.00%

better: 11/25 aka 44.00%

best (0.5m): 3/25 aka 12.00%

mean max error C3: 1.679107

mean max error C2: 1.253022

global mae: 0.350097

avg resets: 0.200000

Plan y error in turns (> 6°/s)

Mean path error : 0.033 meters

Mean absolute path error : 0.516 meters

Turn bias test (y_pos at 50m > 3m)

got 29747 left samples

got 31136 right samples

mean left gt: -0.013104751

mean right gt: 0.046767514

mean turn lanecenter error (GT pos, GT ll): left = 0.21865172684192657, right = -0.2586999833583832

mean turn lanecenter error (sim pos, GT ll): left = 0.23070424795150757, right = -0.20724456012248993

mean turn lanecenter error (GT pos, model ll): left = 0.207358255982399, right = -0.2414860874414444

mean turn lanecenter error (sim pos, model ll): left = 0.1981988549232483, right = -0.1781398504972458

Mean abs curvature: 0.212126 deg/m

Mean abs curvature rate: 0.165975 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.172616 Hz, period 5.793197 s

Mean amplitude: 0.130660 m

Mean ping-pong significance (frequency x amplitude): 2.156789 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	176
turn_right	190
lane_change_left	239
lane_change_right	279
keep_left	88
keep_right	63

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-5.900m
turn_right	+4.992m
lane_change_left	-1.457m
lane_change_right	+1.496m
keep_left	-1.178m
keep_right	+1.140m

Mean desire duration - ground truth

Command	Duration
turn_left	6.828s
turn_right	6.370s
lane_change_left	3.324s
lane_change_right	3.497s
keep_left	3.943s
keep_right	3.664s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	2.36s	3.12s	3.54s	4.19s
turn_right	2.31s	3.11s	3.81s	4.56s
lane_change_left	1.11s	1.62s	2.25s	3.69s
lane_change_right	1.11s	1.62s	2.28s	3.72s
keep_left	1.45s	2.16s	2.81s	4.16s
keep_right	1.68s	2.48s	3.43s	5.10s

Group	Gas prob	Brake prob	Steer prob
All segments	0.009	0.026	0.293
Easy straight	0.006	0.011	0.173
Low speed	0.014	0.039	0.388
No desire segments	0.007	0.016	0.233
No desire turns (max lat accel > 2.5 m/ss)	0.010	0.028	0.386