Model analysis report for

▶
root: {} 12 keys
- report_name: 'analyse_lat_mlsim_realistic_noise'
- route_list: '/code.nfs/branches/caches/tinytower-haraschax_2026-09-14_10-41-51/xx/comma_data/lists/train_10m_20260826_test_2k.txt'
- output_dir: null
- save_tmp: false
- read_only: false
- shuffle: false
- limit: 10000
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 10 keys
  - priority: 3
  - job_name: 'analyse_lat_mlsim_realistic_noise'
  - queue_name: 'remote_v3'
  - redis_host: 'redis.comma.internal'
  - codedir: '/code.nfs/branches/caches/tinytower-haraschax_2026-09-14_10-41-51/xx'
  - use_local_codedir: false
  - ▶
    limits: {} 7 keys
    - cpu_threads: true
    - memory: true
    - small_gpu_memory: false
    - big_gpu_memory: false
    - timeout_seconds: 60
    - triton: true
    - node_whitelist: null
  - ▶
    env: {} 0 keys
  - queue_timeout: 259200
  - job_group: 'rldriving_validation_f78ed37d-afad-4dbc-8050-40ea885eedde'
- ▶
  filter: {} 3 keys
  - min_hist: 100
  - max_hist: -60
  - max_fidx: 1200
- ▶
  rollout: {} 9 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 35 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 5 keys
      - segment: ''
      - depth_type: 'EASYDEPTH'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2026_02'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - ▶
      physics: {} 3 keys
      - type: 'BICYCLE'
      - ▶
        bicycle_model_config: {} 1 keys
        
        vehicle_platform: 'HONDA_CIVIC'
      - ▶
        ml_model_config: {} 11 keys
        
        platform: 'TOYOTA_COROLLA_TSS2'
        
        eid: 'effd03d9-472d-44a1-910a-6d49a41eb63f/50'
        
        y_range: [-3.0, 3.0]
        
        vocab_size: 1024
        
        temperature: 0.5
        
        use_gt_features: false
        
        platform_vec_size: 256
        
        platform_map_path: '/code.nfs/branches/caches/tinytower-haraschax_2026-09-14_10-41-51/xx/arena/envs/lib/physics_models/assets/platform_map.txt'
        
        lateral_tuning_type: 'AUTO'
        
        ▶
        lateral_tuning_params: {} 0 keys
        
        ▶
        lateral_initial_state_params: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_decoder_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'DEPTH'
    - inpaint_mode: 'NONE'
    - initial_speed: null
    - allow_any_frame: false
    - lat_lag_override: null
    - lat_action_bias_override: null
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - avoid_hood: false
    - cache_granularity: true
  - ▶
    agent: {} 8 keys
    - supercombo: 'f78ed37d-afad-4dbc-8050-40ea885eedde/12864'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - use_allow_throttle: false
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - desired_curv_smooth_seconds: false
    - desired_accel_smooth_seconds: false

Plan analysis

All segments

Metrics

passed :1539 / 1772 aka 86.85101580135441%

better :1118 / 1772 aka 63.092550790067726%

best (0.5m):241 / 1772 aka 13.600451467268623%

global bias y: +0.131

global mae y: 0.301

global mae y of lanecenter (when detected) in GT: 0.211

global mae y of lanecenter (when detected) in Sim: 0.280

avg resets: 0.269

Mean abs curvature: 0.139 deg/m

Mean abs curvature rate: 0.149 deg/ms

Mean abs lateral accel: 0.306 m/(ss)

Mean abs lateral accel GT: 0.291 m/(ss)

Mean abs lateral jerk: 0.245 m/(sss)

Mean abs lateral jerk GT: 0.249 m/(sss)

Easy straight

Metrics

passed :619 / 644 aka 96.11801242236024%

better :565 / 644 aka 87.73291925465838%

best (0.5m):161 / 644 aka 25.0%

global bias y: +0.135

global mae y: 0.250

global mae y of lanecenter (when detected) in GT: 0.173

global mae y of lanecenter (when detected) in Sim: 0.238

avg resets: 0.045

Mean abs curvature: 0.043 deg/m

Mean abs curvature rate: 0.062 deg/ms

Mean abs lateral accel: 0.144 m/(ss)

Mean abs lateral accel GT: 0.108 m/(ss)

Mean abs lateral jerk: 0.171 m/(sss)

Mean abs lateral jerk GT: 0.169 m/(sss)

Low speed

Metrics

passed :217 / 267 aka 81.27340823970037%

better :139 / 267 aka 52.05992509363296%

best (0.5m):49 / 267 aka 18.352059925093634%

global bias y: +0.129

global mae y: 0.335

global mae y of lanecenter (when detected) in GT: 0.231

global mae y of lanecenter (when detected) in Sim: 0.284

avg resets: 0.543

Mean abs curvature: 0.305 deg/m

Mean abs curvature rate: 0.363 deg/ms

Mean abs lateral accel: 0.222 m/(ss)

Mean abs lateral accel GT: 0.236 m/(ss)

Mean abs lateral jerk: 0.213 m/(sss)

Mean abs lateral jerk GT: 0.181 m/(sss)

No desire segments

Metrics

passed :975 / 1044 aka 93.39080459770115%

better :848 / 1044 aka 81.22605363984674%

best (0.5m):207 / 1044 aka 19.82758620689655%

global bias y: +0.129

global mae y: 0.264

global mae y of lanecenter (when detected) in GT: 0.187

global mae y of lanecenter (when detected) in Sim: 0.249

avg resets: 0.096

Mean abs curvature: 0.072 deg/m

Mean abs curvature rate: 0.077 deg/ms

Mean abs lateral accel: 0.264 m/(ss)

Mean abs lateral accel GT: 0.238 m/(ss)

Mean abs lateral jerk: 0.209 m/(sss)

Mean abs lateral jerk GT: 0.219 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :52 / 77 aka 67.53246753246754%

better :29 / 77 aka 37.66233766233766%

best (0.5m):1 / 77 aka 1.2987012987012987%

global bias y: +0.139

global mae y: 0.350

global mae y of lanecenter (when detected) in GT: 0.264

global mae y of lanecenter (when detected) in Sim: 0.326

avg resets: 0.571

Mean abs curvature: 0.262 deg/m

Mean abs curvature rate: 0.243 deg/ms

Mean abs lateral accel: 0.593 m/(ss)

Mean abs lateral accel GT: 0.612 m/(ss)

Mean abs lateral jerk: 0.362 m/(sss)

Mean abs lateral jerk GT: 0.374 m/(sss)

Left lane change segments

125/138 aka 90.58%

better: 82/138 aka 59.42%

best (0.5m): 17/138 aka 12.32%

global mae: 0.283236

avg resets: 0.159420

Right lane change segments

146/164 aka 89.02%

better: 65/164 aka 39.63%

best (0.5m): 4/164 aka 2.44%

global mae: 0.325868

avg resets: 0.164634

Left turn segments

53/65 aka 81.54%

better: 23/65 aka 35.38%

best (0.5m): 3/65 aka 4.62%

global mae: 0.335516

avg resets: 0.892308

Right turn segments

58/80 aka 72.50%

better: 22/80 aka 27.50%

best (0.5m): 0/80 aka 0.00%

global mae: 0.379429

avg resets: 0.725000

Left keep segments

27/37 aka 72.97%

better: 18/37 aka 48.65%

best (0.5m): 7/37 aka 18.92%

global mae: 0.341551

avg resets: 0.405405

Right keep segments

29/40 aka 72.50%

better: 21/40 aka 52.50%

best (0.5m): 2/40 aka 5.00%

global mae: 0.373180

avg resets: 0.300000

Plan y error in turns (> 6°/s)

Mean path error : 0.114 meters

Mean absolute path error : 0.548 meters

Turn bias test (y_pos at 50m > 3m)

got 28461 left samples

got 31783 right samples

mean left gt: 0.12864402

mean right gt: 0.07733666

mean turn lanecenter error (GT pos, GT ll): left = 0.11865702271461487, right = -0.27515867352485657

mean turn lanecenter error (sim pos, GT ll): left = 0.25164297223091125, right = -0.20851314067840576

mean turn lanecenter error (GT pos, model ll): left = 0.12616051733493805, right = -0.2503855228424072

mean turn lanecenter error (sim pos, model ll): left = 0.2203679382801056, right = -0.20499028265476227

Mean abs curvature: 0.184910 deg/m

Mean abs curvature rate: 0.092105 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.172263 Hz, period 5.805084 s

Mean amplitude: 0.142330 m

Mean ping-pong significance (frequency x amplitude): 2.357067 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	112
turn_right	163
lane_change_left	283
lane_change_right	302
keep_left	67
keep_right	77

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-4.167m
turn_right	+3.332m
lane_change_left	-1.151m
lane_change_right	+1.337m
keep_left	-1.026m
keep_right	+1.319m

Mean desire duration - ground truth

Command	Duration
turn_left	6.405s
turn_right	6.198s
lane_change_left	3.345s
lane_change_right	3.320s
keep_left	3.497s
keep_right	3.821s

Elapsed simulator displacement in command direction after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	2.76s (78/112)	3.19s (73/112)	3.44s (68/112)	3.83s (58/112)
turn_right	2.09s (141/163)	2.55s (134/163)	2.94s (126/163)	3.42s (112/163)
lane_change_left	1.21s (230/283)	1.64s (213/283)	2.15s (194/283)	2.91s (127/283)
lane_change_right	1.32s (241/302)	1.79s (229/302)	2.29s (198/302)	2.94s (117/302)

Meta analysis

Group	Gas prob	Brake prob	Steer prob
All segments	0.004	0.014	0.318
Easy straight	0.003	0.010	0.219
Low speed	0.007	0.023	0.417
No desire segments	0.003	0.011	0.257
No desire turns (max lat accel > 2.5 m/ss)	0.006	0.024	0.445