Model analysis report for

▶
root: {} 11 keys
- report_name: 'mlsimdriving_driving.analyse_lat_mlsim'
- route_list: '/code.nfs/branches/caches/workstation-yassine_2025-10-06_15-35-15/xx/datasets/lists/train_500k_20240617_test_2k.txt'
- output_dir: null
- save_tmp: false
- read_only: false
- shuffle: false
- limit: 10000
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 10
  - job_name: 'mlsimdriving_driving.analyse_lat_mlsim'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: '/code.nfs/branches/caches/workstation-yassine_2025-10-06_15-35-15/xx'
  - use_local_codedir: false
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 32 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2025_07'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - ▶
      physics: {} 3 keys
      - type: 'BICYCLE'
      - ▶
        bicycle_model_config: {} 1 keys
        
        vehicle_platform: 'HONDA_CIVIC'
      - ▶
        ml_model_config: {} 11 keys
        
        platform: 'TOYOTA_COROLLA_TSS2'
        
        eid: 'effd03d9-472d-44a1-910a-6d49a41eb63f/50'
        
        y_range: [-3.0, 3.0]
        
        vocab_size: 1024
        
        temperature: 0.5
        
        use_gt_features: false
        
        platform_vec_size: 256
        
        platform_map_path: '/code.nfs/branches/caches/workstation-yassine_2025-10-06_15-35-15/xx/arena/envs/lib/physics_models/assets/platform_map.txt'
        
        lateral_tuning_type: 'AUTO'
        
        ▶
        lateral_tuning_params: {} 0 keys
        
        ▶
        lateral_initial_state_params: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_decoder_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'DEPTH'
    - inpaint_mode: 'NONE'
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 9 keys
    - supercombo: 'e2d9c622-25a8-4ccd-8c8e-c62537b7aa0c/400'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - use_allow_throttle: false
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: true
    - desired_curv_smooth_seconds: false
    - desired_accel_smooth_seconds: false
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1458 / 1690 aka 86.272%

better :1067 / 1690 aka 63.136%

best (0.5m):314 / 1690 aka 18.580%

global mae x: 0.171

global mae y: 0.283

global mae y of lanecenter (when detected) in GT: 0.210

global mae y of lanecenter (when detected) in Sim: 0.258

avg resets: 0.266

Mean abs curvature: 0.191 deg/m

Mean abs curvature rate: 0.246 deg/ms

Mean abs lateral accel: 0.363 m/(ss)

Mean abs lateral accel GT: 0.337 m/(ss)

Mean abs lateral jerk: 0.413 m/(sss)

Mean abs lateral jerk GT: 0.271 m/(sss)

Easy straight

Metrics

passed :511 / 515 aka 99.223%

better :467 / 515 aka 90.680%

best (0.5m):204 / 515 aka 39.612%

global mae x: 0.107

global mae y: 0.207

global mae y of lanecenter (when detected) in GT: 0.160

global mae y of lanecenter (when detected) in Sim: 0.212

avg resets: 0.014

Mean abs curvature: 0.039 deg/m

Mean abs curvature rate: 0.073 deg/ms

Mean abs lateral accel: 0.175 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.291 m/(sss)

Mean abs lateral jerk GT: 0.170 m/(sss)

Low speed

Metrics

passed :289 / 361 aka 80.055%

better :197 / 361 aka 54.571%

best (0.5m):72 / 361 aka 19.945%

global mae x: 0.232

global mae y: 0.318

global mae y of lanecenter (when detected) in GT: 0.264

global mae y of lanecenter (when detected) in Sim: 0.281

avg resets: 0.479

Mean abs curvature: 0.361 deg/m

Mean abs curvature rate: 0.484 deg/ms

Mean abs lateral accel: 0.307 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.366 m/(sss)

Mean abs lateral jerk GT: 0.229 m/(sss)

No desire segments

Metrics

passed :898 / 949 aka 94.626%

better :778 / 949 aka 81.981%

best (0.5m):275 / 949 aka 28.978%

global mae x: 0.125

global mae y: 0.234

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.226

avg resets: 0.075

Mean abs curvature: 0.103 deg/m

Mean abs curvature rate: 0.124 deg/ms

Mean abs lateral accel: 0.321 m/(ss)

Mean abs lateral accel GT: 0.287 m/(ss)

Mean abs lateral jerk: 0.357 m/(sss)

Mean abs lateral jerk GT: 0.237 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :113 / 134 aka 84.328%

better :71 / 134 aka 52.985%

best (0.5m):11 / 134 aka 8.209%

global mae x: 0.200

global mae y: 0.288

global mae y of lanecenter (when detected) in GT: 0.253

global mae y of lanecenter (when detected) in Sim: 0.261

avg resets: 0.201

Mean abs curvature: 0.255 deg/m

Mean abs curvature rate: 0.239 deg/ms

Mean abs lateral accel: 0.673 m/(ss)

Mean abs lateral accel GT: 0.667 m/(ss)

Mean abs lateral jerk: 0.539 m/(sss)

Mean abs lateral jerk GT: 0.407 m/(sss)

Left lane change segments

110/128 aka 85.94%

better: 77/128 aka 60.16%

best (0.5m): 11/128 aka 8.59%

mean max error C3: 0.959625

mean max error C2: 1.204538

global mae: 0.285074

avg resets: 0.085938

Right lane change segments

125/145 aka 86.21%

better: 80/145 aka 55.17%

best (0.5m): 15/145 aka 10.34%

mean max error C3: 1.131887

mean max error C2: 1.192636

global mae: 0.300298

avg resets: 0.151724

Left turn segments

63/85 aka 74.12%

better: 24/85 aka 28.24%

best (0.5m): 1/85 aka 1.18%

mean max error C3: 1.577201

mean max error C2: 1.558565

global mae: 0.344103

avg resets: 0.882353

Right turn segments

67/89 aka 75.28%

better: 32/89 aka 35.96%

best (0.5m): 3/89 aka 3.37%

mean max error C3: 1.442316

mean max error C2: 1.478223

global mae: 0.349164

avg resets: 0.752809

Left keep segments

27/36 aka 75.00%

better: 16/36 aka 44.44%

best (0.5m): 3/36 aka 8.33%

mean max error C3: 1.184655

mean max error C2: 1.504330

global mae: 0.312294

avg resets: 0.222222

Right keep segments

19/25 aka 76.00%

better: 9/25 aka 36.00%

best (0.5m): 3/25 aka 12.00%

mean max error C3: 1.664139

mean max error C2: 1.336778

global mae: 0.367507

avg resets: 0.200000

Plan y error in turns (> 6°/s)

Mean path error : -0.001 meters

Mean absolute path error : 0.503 meters

Turn bias test (y_pos at 50m > 3m)

got 29747 left samples

got 31135 right samples

mean left gt: 0.012828226

mean right gt: 0.05324916

mean turn lanecenter error (GT pos, GT ll): left = 0.2186366617679596, right = -0.2586999237537384

mean turn lanecenter error (sim pos, GT ll): left = 0.26273003220558167, right = -0.19360145926475525

mean turn lanecenter error (GT pos, model ll): left = 0.1969698816537857, right = -0.22279231250286102

mean turn lanecenter error (sim pos, model ll): left = 0.22307425737380981, right = -0.13566674292087555

Mean abs curvature: 0.213398 deg/m

Mean abs curvature rate: 0.166391 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.170514 Hz, period 5.864629 s

Mean amplitude: 0.142763 m

Mean ping-pong significance (frequency x amplitude): 2.337816 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	176
turn_right	190
lane_change_left	239
lane_change_right	279
keep_left	88
keep_right	63

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-5.089m
turn_right	+5.828m
lane_change_left	-1.416m
lane_change_right	+1.394m
keep_left	-1.097m
keep_right	+1.137m

Mean desire duration - ground truth

Command	Duration
turn_left	6.828s
turn_right	6.370s
lane_change_left	3.324s
lane_change_right	3.497s
keep_left	3.943s
keep_right	3.664s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	2.43s	3.02s	3.63s	4.39s
turn_right	2.30s	3.00s	3.51s	4.11s
lane_change_left	1.11s	1.63s	2.36s	3.71s
lane_change_right	1.14s	1.69s	2.47s	4.01s
keep_left	1.57s	2.24s	3.21s	4.94s
keep_right	1.73s	2.62s	3.79s	5.17s

Group	Gas prob	Brake prob	Steer prob
All segments	0.009	0.025	0.273
Easy straight	0.007	0.010	0.155
Low speed	0.014	0.040	0.362
No desire segments	0.007	0.015	0.215
No desire turns (max lat accel > 2.5 m/ss)	0.010	0.028	0.365