Model analysis report for

▶
root: {} 11 keys
- report_name: 'mlsimdriving_driving.analyse_lat_mlsim'
- route_list: '/home/batman/xx/datasets/data_lists/default_test_2k'
- output_dir: null
- save_tmp: false
- read_only: false
- shuffle: false
- limit: 10000
- use_virtual_webgl: false
- format: 'HTML'
- ▶
  miniray: {} 8 keys
  - priority: 10
  - job_name: 'mlsimdriving_driving.analyse_lat_mlsim'
  - queue_name: 'remote2'
  - submit_host: 'redis.comma.internal'
  - result_host: 'data-ram.comma.internal'
  - no_unique_id: false
  - codedir: 'caches/workstation-harald_2025-08-31_00-41-12'
  - force_local: false
- ▶
  rollout: {} 10 keys
  - type: 'TinySimRL'
  - nowindow: false
  - debug: false
  - debug_render: false
  - save_observation_keys: []
  - output_dir: null
  - rerun: false
  - ▶
    env: {} 32 keys
    - debug_window_size: [512, 512]
    - ▶
      ep: {} 7 keys
      - segment: ''
      - augment_desire: false
      - depth_type: 'EASYDEPTH'
      - calibration_source: 'LOG'
      - pipeline_dir: 'http://data-ssd.comma.life/runner/training_2025_07'
      - feature_dir: null
      - ▶
        input_paths: {} 0 keys
    - start: null
    - end: null
    - env: 'xx.arena.envs.tinysim_env.TinySimEnv'
    - get_imgs: true
    - allow_resets: true
    - zero_desire: false
    - fps: 20
    - allow_lat: true
    - allow_long: false
    - min_hist: 40
    - compressor_model: null
    - compressor_decoder_model: null
    - compressor_mode: 'NONE'
    - quantized_compressor_latents: false
    - yolo_model: false
    - lpips_model: false
    - sharpness_metric: true
    - future_observation_range_seconds: null
    - ▶
      physics: {} 3 keys
      - type: 'BICYCLE'
      - ▶
        bicycle_model_config: {} 1 keys
        
        vehicle_platform: 'HONDA_CIVIC'
      - ▶
        ml_model_config: {} 8 keys
        
        platform: 'TOYOTA_COROLLA_TSS2'
        
        eid: '3bec45a4-9512-4c7f-bdbc-9415e5890fb6/100'
        
        y_range: [-5.0, 5.0]
        
        vocab_size: 2048
        
        temperature: 0.5
        
        use_gt_features: false
        
        platform_vec_size: 256
        
        platform_map_path: '/code.nfs/branches/caches/workstation-harald_2025-08-31_00-41-12/xx/arena/envs/lib/physics_models/assets/platform_map.txt'
    - max_x_error: 4.0
    - max_y_error: 3.0
    - max_yaw_error: 0.2
    - reset_every: null
    - noise_model: 'REALISTIC'
    - photo_noise_model: 'NONE'
    - augment_mode: 'DEPTH'
    - inpaint_mode: 'NONE'
    - avoid_hood: false
    - cache_granularity: true
    - worldmodel: null
  - ▶
    agent: {} 10 keys
    - supercombo: '816ce390-c41a-42fa-a5df-f393cbe2dcc4/400'
    - restrict_outputs: null
    - exclude_outputs: ['imgs']
    - use_allow_throttle: false
    - use_prev_desired_curv: false
    - model_trained_fps: 5
    - use_worldmodel_as_agent: false
    - use_raw_plan_lat: true
    - desired_curv_smooth_seconds: false
    - desired_accel_smooth_seconds: false
  - ▶
    shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1484 / 1690 aka 87.811%

better :1057 / 1690 aka 62.544%

best (0.5m):304 / 1690 aka 17.988%

global mae x: 0.170

global mae y: 0.284

global mae y of lanecenter (when detected) in GT: 0.210

global mae y of lanecenter (when detected) in Sim: 0.250

avg resets: 0.269

Mean abs curvature: 0.193 deg/m

Mean abs curvature rate: 0.252 deg/ms

Mean abs lateral accel: 0.360 m/(ss)

Mean abs lateral accel GT: 0.337 m/(ss)

Mean abs lateral jerk: 0.405 m/(sss)

Mean abs lateral jerk GT: 0.271 m/(sss)

Easy straight

Metrics

passed :509 / 515 aka 98.835%

better :468 / 515 aka 90.874%

best (0.5m):202 / 515 aka 39.223%

global mae x: 0.107

global mae y: 0.210

global mae y of lanecenter (when detected) in GT: 0.160

global mae y of lanecenter (when detected) in Sim: 0.202

avg resets: 0.016

Mean abs curvature: 0.039 deg/m

Mean abs curvature rate: 0.073 deg/ms

Mean abs lateral accel: 0.171 m/(ss)

Mean abs lateral accel GT: 0.130 m/(ss)

Mean abs lateral jerk: 0.284 m/(sss)

Mean abs lateral jerk GT: 0.170 m/(sss)

Low speed

Metrics

passed :298 / 361 aka 82.548%

better :197 / 361 aka 54.571%

best (0.5m):64 / 361 aka 17.729%

global mae x: 0.228

global mae y: 0.324

global mae y of lanecenter (when detected) in GT: 0.264

global mae y of lanecenter (when detected) in Sim: 0.278

avg resets: 0.485

Mean abs curvature: 0.367 deg/m

Mean abs curvature rate: 0.491 deg/ms

Mean abs lateral accel: 0.308 m/(ss)

Mean abs lateral accel GT: 0.305 m/(ss)

Mean abs lateral jerk: 0.360 m/(sss)

Mean abs lateral jerk GT: 0.229 m/(sss)

No desire segments

Metrics

passed :904 / 949 aka 95.258%

better :773 / 949 aka 81.454%

best (0.5m):272 / 949 aka 28.662%

global mae x: 0.125

global mae y: 0.235

global mae y of lanecenter (when detected) in GT: 0.180

global mae y of lanecenter (when detected) in Sim: 0.217

avg resets: 0.075

Mean abs curvature: 0.103 deg/m

Mean abs curvature rate: 0.126 deg/ms

Mean abs lateral accel: 0.317 m/(ss)

Mean abs lateral accel GT: 0.287 m/(ss)

Mean abs lateral jerk: 0.348 m/(sss)

Mean abs lateral jerk GT: 0.237 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :117 / 134 aka 87.313%

better :71 / 134 aka 52.985%

best (0.5m):9 / 134 aka 6.716%

global mae x: 0.200

global mae y: 0.292

global mae y of lanecenter (when detected) in GT: 0.252

global mae y of lanecenter (when detected) in Sim: 0.253

avg resets: 0.216

Mean abs curvature: 0.255 deg/m

Mean abs curvature rate: 0.244 deg/ms

Mean abs lateral accel: 0.671 m/(ss)

Mean abs lateral accel GT: 0.667 m/(ss)

Mean abs lateral jerk: 0.531 m/(sss)

Mean abs lateral jerk GT: 0.408 m/(sss)

Left lane change segments

112/128 aka 87.50%

better: 76/128 aka 59.38%

best (0.5m): 8/128 aka 6.25%

mean max error C3: 0.971864

mean max error C2: 1.245369

global mae: 0.289490

avg resets: 0.093750

Right lane change segments

130/145 aka 89.66%

better: 79/145 aka 54.48%

best (0.5m): 12/145 aka 8.28%

mean max error C3: 1.085675

mean max error C2: 1.150743

global mae: 0.291360

avg resets: 0.124138

Left turn segments

64/85 aka 75.29%

better: 23/85 aka 27.06%

best (0.5m): 2/85 aka 2.35%

mean max error C3: 1.513822

mean max error C2: 1.541938

global mae: 0.352756

avg resets: 0.882353

Right turn segments

72/89 aka 80.90%

better: 32/89 aka 35.96%

best (0.5m): 2/89 aka 2.25%

mean max error C3: 1.354535

mean max error C2: 1.475284

global mae: 0.361925

avg resets: 0.752809

Left keep segments

29/36 aka 80.56%

better: 14/36 aka 38.89%

best (0.5m): 2/36 aka 5.56%

mean max error C3: 1.202480

mean max error C2: 1.455046

global mae: 0.321734

avg resets: 0.222222

Right keep segments

17/25 aka 68.00%

better: 8/25 aka 32.00%

best (0.5m): 2/25 aka 8.00%

mean max error C3: 1.740872

mean max error C2: 1.422334

global mae: 0.354469

avg resets: 0.200000

Plan y error in turns (> 6°/s)

Mean path error : -0.013 meters

Mean absolute path error : 0.508 meters

Turn bias test (y_pos at 50m > 3m)

got 29747 left samples

got 31135 right samples

mean left gt: -0.052804686

mean right gt: -0.02361284

mean turn lanecenter error (GT pos, GT ll): left = 0.2186366468667984, right = -0.258699893951416

mean turn lanecenter error (sim pos, GT ll): left = 0.1987944394350052, right = -0.24573995172977448

mean turn lanecenter error (GT pos, model ll): left = 0.20726783573627472, right = -0.21535371243953705

mean turn lanecenter error (sim pos, model ll): left = 0.15213976800441742, right = -0.1932547390460968

Mean abs curvature: 0.213076 deg/m

Mean abs curvature rate: 0.164991 deg/ms

Mean gt pos in left and right turns

Lane position frequency-domain analysis

Mean frequency: 0.172845 Hz, period 5.785532 s

Mean amplitude: 0.143545 m

Mean ping-pong significance (frequency x amplitude): 2.368447 %

Desire analysis

Desire pulses by type

Command	Count
turn_left	176
turn_right	190
lane_change_left	239
lane_change_right	279
keep_left	88
keep_right	63

Mean difference in path y-coord after desire pulse

Command	Diff
turn_left	-5.036m
turn_right	+5.450m
lane_change_left	-1.437m
lane_change_right	+1.344m
keep_left	-1.128m
keep_right	+1.086m

Mean desire duration - ground truth

Command	Duration
turn_left	6.828s
turn_right	6.370s
lane_change_left	3.324s
lane_change_right	3.497s
keep_left	3.943s
keep_right	3.664s

Mean times for 20/50/100/200cm deviation in plan after desire pulse

Command	20cm	50cm	100cm	200cm
turn_left	2.43s	3.10s	3.70s	4.44s
turn_right	2.30s	3.09s	3.59s	4.34s
lane_change_left	1.24s	1.83s	2.58s	4.06s
lane_change_right	1.21s	1.80s	2.57s	4.09s
keep_left	1.70s	2.37s	3.33s	4.96s
keep_right	1.82s	2.62s	3.56s	5.03s

Group	Gas prob	Brake prob	Steer prob
All segments	0.009	0.025	0.273
Easy straight	0.007	0.010	0.155
Low speed	0.014	0.040	0.364
No desire segments	0.007	0.015	0.215
No desire turns (max lat accel > 2.5 m/ss)	0.009	0.028	0.365