Model analysis report for

• ▶
  root: {} 11 keys
  • report_name: 'analyse_lat_worldmodel_new_upweight_nonan_realistic'
  • route_list: '/home/batman/xx/datasets/lists/train_500k_20240617_test_2k.txt'
  • output_dir: '/raid.unprotected/tmp/runner/tmp_11bfwyo7'
  • save_tmp: true
  • read_only: false
  • shuffle: false
  • limit: 10000
  • use_virtual_webgl: false
  • format: 'FILE'
  • ▶
    miniray: {} 8 keys
    • priority: 10
    • job_name: 'analyse_lat_worldmodel_new_upweight_nonan_realistic'
    • queue_name: 'remote2'
    • submit_host: 'redis.comma.internal'
    • result_host: 'data-ram.comma.internal'
    • no_unique_id: false
    • codedir: null
    • use_local_codedir: true
  • ▶
    rollout: {} 10 keys
    • type: 'TinySimWorldModel'
    • nowindow: false
    • debug: false
    • debug_render: false
    • save_observation_keys: []
    • output_dir: null
    • rerun: false
    • ▶
      env: {} 32 keys
      • debug_window_size: [512, 512]
      • ▶
        ep: {} 7 keys
        • segment: ''
        • augment_desire: false
        • depth_type: 'EASYDEPTH'
        • calibration_source: 'LOG'
        • pipeline_dir: 'http://data-ssd.comma.life/runner/training_2025_07'
        • feature_dir: null
        • ▶
          input_paths: {} 0 keys
      • ▶
        physics: {} 3 keys
        • type: 'BICYCLE'
        • ▶
          bicycle_model_config: {} 1 keys
          • vehicle_platform: 'HONDA_CIVIC'
        • ▶
          ml_model_config: {} 11 keys
          • platform: 'TOYOTA_COROLLA_TSS2'
          • eid: 'effd03d9-472d-44a1-910a-6d49a41eb63f/50'
          • y_range: [-3.0, 3.0]
          • vocab_size: 1024
          • temperature: 0.5
          • use_gt_features: false
          • platform_vec_size: 256
          • platform_map_path: '/home/batman/xx/arena/envs/lib/physics_models/assets/platform_map.txt'
          • lateral_tuning_type: 'AUTO'
          • ▶
            lateral_tuning_params: {} 0 keys
          • ▶
            lateral_initial_state_params: {} 0 keys
      • start: null
      • end: 1000
      • env: 'xx.arena.envs.tinysim_env.TinySimEnv'
      • get_imgs: true
      • allow_resets: true
      • zero_desire: false
      • fps: 20
      • allow_lat: true
      • allow_long: false
      • min_hist: 40
      • compressor_model: '6355d081-a4be-4eb3-ab97-e9ef9cdade1f'
      • compressor_decoder_model: null
      • compressor_mode: 'NONE'
      • quantized_compressor_latents: false
      • yolo_model: false
      • lpips_model: false
      • sharpness_metric: true
      • future_observation_range_seconds: [2, 3]
      • noise_model: 'REALISTIC'
      • photo_noise_model: 'NONE'
      • augment_mode: 'DEPTH'
      • inpaint_mode: 'NONE'
      • max_x_error: 4.0
      • max_y_error: 3.0
      • max_yaw_error: 0.2
      • reset_every: null
      • avoid_hood: false
      • cache_granularity: true
      • worldmodel: null
    • ▶
      agent: {} 14 keys
      • model: '923eee54-b95d-465c-a9d7-8c1064170270/90'
      • compressor_model: '6355d081-a4be-4eb3-ab97-e9ef9cdade1f'
      • vae_scale_factor: 5.79787449920859
      • restrict_outputs: null
      • exclude_outputs: ['imgs', 'tokens', 'latents']
      • model_trained_fps: 5
      • use_worldmodel_as_agent: false
      • context_size_seconds: 2
      • future_size_seconds: true
      • future_frame_start_seconds: 2
      • advance_to_future: false
      • use_raw_plan_lat: true
      • desired_curv_smooth_seconds: false
      • desired_accel_smooth_seconds: false
    • ▶
      shadow_agent: {} 0 keys

Plan analysis

All segments

Metrics

passed :1670 / 1680 aka 99.405%

better :1584 / 1680 aka 94.286%

best (0.5m):1264 / 1680 aka 75.238%

global mae x: 0.162

global mae y: 0.115

global mae y of lanecenter (when detected) in GT: 0.211

global mae y of lanecenter (when detected) in Sim: 0.236

avg resets: 0.046

Mean abs curvature: 0.210 deg/m

Mean abs curvature rate: 0.210 deg/ms

Mean abs lateral accel: 0.394 m/(ss)

Mean abs lateral accel GT: 0.339 m/(ss)

Mean abs lateral jerk: 0.509 m/(sss)

Mean abs lateral jerk GT: 0.273 m/(sss)

Easy straight

Metrics

passed :587 / 587 aka 100.000%

better :586 / 587 aka 99.830%

best (0.5m):571 / 587 aka 97.274%

global mae x: 0.110

global mae y: 0.089

global mae y of lanecenter (when detected) in GT: 0.164

global mae y of lanecenter (when detected) in Sim: 0.189

avg resets: 0.007

Mean abs curvature: 0.051 deg/m

Mean abs curvature rate: 0.098 deg/ms

Mean abs lateral accel: 0.217 m/(ss)

Mean abs lateral accel GT: 0.128 m/(ss)

Mean abs lateral jerk: 0.473 m/(sss)

Mean abs lateral jerk GT: 0.171 m/(sss)

Low speed

Metrics

passed :376 / 377 aka 99.735%

better :349 / 377 aka 92.573%

best (0.5m):250 / 377 aka 66.313%

global mae x: 0.215

global mae y: 0.115

global mae y of lanecenter (when detected) in GT: 0.268

global mae y of lanecenter (when detected) in Sim: 0.278

avg resets: 0.048

Mean abs curvature: 0.402 deg/m

Mean abs curvature rate: 0.385 deg/ms

Mean abs lateral accel: 0.328 m/(ss)

Mean abs lateral accel GT: 0.309 m/(ss)

Mean abs lateral jerk: 0.324 m/(sss)

Mean abs lateral jerk GT: 0.230 m/(sss)

No desire segments

Metrics

passed :1025 / 1027 aka 99.805%

better :1009 / 1027 aka 98.247%

best (0.5m):894 / 1027 aka 87.050%

global mae x: 0.123

global mae y: 0.102

global mae y of lanecenter (when detected) in GT: 0.182

global mae y of lanecenter (when detected) in Sim: 0.207

avg resets: 0.013

Mean abs curvature: 0.117 deg/m

Mean abs curvature rate: 0.132 deg/ms

Mean abs lateral accel: 0.354 m/(ss)

Mean abs lateral accel GT: 0.288 m/(ss)

Mean abs lateral jerk: 0.495 m/(sss)

Mean abs lateral jerk GT: 0.238 m/(sss)

No desire turns (max lat accel > 2.5 m/ss)

Metrics

passed :132 / 133 aka 99.248%

better :123 / 133 aka 92.481%

best (0.5m):64 / 133 aka 48.120%

global mae x: 0.204

global mae y: 0.136

global mae y of lanecenter (when detected) in GT: 0.258

global mae y of lanecenter (when detected) in Sim: 0.275

avg resets: 0.045

Mean abs curvature: 0.291 deg/m

Mean abs curvature rate: 0.244 deg/ms

Mean abs lateral accel: 0.721 m/(ss)

Mean abs lateral accel GT: 0.696 m/(ss)

Mean abs lateral jerk: 0.590 m/(sss)

Mean abs lateral jerk GT: 0.424 m/(sss)

Left lane change segments

115/115 aka 100.00%

better: 112/115 aka 97.39%

best (0.5m): 91/115 aka 79.13%

mean max error C3: 0.402692

mean max error C2: 0.381979

global mae: 0.114531

avg resets: 0.000000

Right lane change segments

131/131 aka 100.00%

better: 125/131 aka 95.42%

best (0.5m): 102/131 aka 77.86%

mean max error C3: 0.413918

mean max error C2: 0.432517

global mae: 0.114975

avg resets: 0.076336

Left turn segments

78/80 aka 97.50%

better: 64/80 aka 80.00%

best (0.5m): 25/80 aka 31.25%

mean max error C3: 0.763796

mean max error C2: 0.738686

global mae: 0.150881

avg resets: 0.150000

Right turn segments

89/89 aka 100.00%

better: 73/89 aka 82.02%

best (0.5m): 31/89 aka 34.83%

mean max error C3: 0.772689

mean max error C2: 0.650564

global mae: 0.152147

avg resets: 0.134831

Left keep segments

35/35 aka 100.00%

better: 35/35 aka 100.00%

best (0.5m): 22/35 aka 62.86%

mean max error C3: 0.354925

mean max error C2: 0.510137

global mae: 0.126492

avg resets: 0.000000

Right keep segments

22/22 aka 100.00%

better: 22/22 aka 100.00%

best (0.5m): 16/22 aka 72.73%

mean max error C3: 0.437077

mean max error C2: 0.449999

global mae: 0.112644

avg resets: 0.000000

Plan y error in turns (> 6°/s)

Mean path error : -0.011 meters

Mean absolute path error : 0.280 meters

Turn bias test (y_pos at 50m > 3m)

got 24440 left samples

got 25567 right samples

mean left gt: 0.03468685

mean right gt: -0.027780661

mean turn lanecenter error (GT pos, GT ll): left = 0.22221218049526215, right = -0.26461705565452576

mean turn lanecenter error (sim pos, GT ll): left = 0.2708660364151001, right = -0.2814706265926361

Mean abs curvature: 0.212876 deg/m

Mean abs curvature rate: 0.150009 deg/ms

Mean gt pos in left and right turns