Model analysis report for

• ▶
  root: {} 14 keys
  • report_name: 'analyse_worldmodel_923eee54_cfg'
  • route_list: '/home/batman/xx/datasets/lists/train_500k_20240617_test_2k.txt'
  • output_dir: '/raid.unprotected/tmp/runner/tmp_xxwpmgfc'
  • save_tmp: true
  • read_only: false
  • shuffle: false
  • limit: 10000
  • use_virtual_webgl: false
  • format: 'FILE'
  • ▶
    miniray: {} 8 keys
    • priority: 10
    • job_name: 'analyse_worldmodel_923eee54_cfg'
    • queue_name: 'remote2'
    • submit_host: 'redis.comma.internal'
    • result_host: 'data-ram.comma.internal'
    • no_unique_id: false
    • codedir: null
    • use_local_codedir: true
  • rollout_noise_types: ['gt', 'lat', 'long']
  • ▶
    rollout: {} 10 keys
    • type: 'MLSimRL'
    • nowindow: false
    • debug: false
    • debug_render: false
    • save_observation_keys: []
    • output_dir: null
    • rerun: false
    • ▶
      env: {} 41 keys
      • debug_window_size: [512, 712]
      • ▶
        ep: {} 7 keys
        • segment: ''
        • augment_desire: false
        • depth_type: 'EASYDEPTH'
        • calibration_source: 'MESH_3D'
        • pipeline_dir: 'http://data-ssd.comma.life/runner/training_2025_07'
        • feature_dir: 'http://data-ssd.comma.life/vae_model_features'
        • ▶
          input_paths: {} 0 keys
      • ▶
        physics: {} 3 keys
        • type: 'BICYCLE'
        • ▶
          bicycle_model_config: {} 1 keys
          • vehicle_platform: 'HONDA_CIVIC'
        • ▶
          ml_model_config: {} 11 keys
          • platform: 'TOYOTA_COROLLA_TSS2'
          • eid: 'effd03d9-472d-44a1-910a-6d49a41eb63f/50'
          • y_range: [-3.0, 3.0]
          • vocab_size: 1024
          • temperature: 0.5
          • use_gt_features: false
          • platform_vec_size: 256
          • platform_map_path: '/home/batman/xx/arena/envs/lib/physics_models/assets/platform_map.txt'
          • lateral_tuning_type: 'AUTO'
          • ▶
            lateral_tuning_params: {} 0 keys
          • ▶
            lateral_initial_state_params: {} 0 keys
      • start: null
      • end: null
      • env: 'xx.arena.envs.mlsim_env.MLSimEnv'
      • get_imgs: true
      • allow_resets: false
      • zero_desire: false
      • fps: 5
      • allow_lat: false
      • allow_long: false
      • min_hist: 40
      • compressor_model: '6355d081-a4be-4eb3-ab97-e9ef9cdade1f'
      • compressor_decoder_model: null
      • compressor_mode: 'DECODE'
      • quantized_compressor_latents: false
      • yolo_model: true
      • lpips_model: true
      • sharpness_metric: true
      • future_observation_range_seconds: null
      • debug_plot_keys: ['agent_action/curv', 'agent_action/accel']
      • noise_model: 'NONE'
      • photo_noise_model: 'NONE'
      • max_rollout_frames: 50
      • context_size_frames: 10
      • future_size_frames: 5
      • vae_scale_factor: 5.79787449920859
      • worldmodel: '923eee54-b95d-465c-a9d7-8c1064170270/90'
      • plan_model: null
      • pose_model: null
      • pose_source: 'AGENT'
      • pre_worldmodel_warmup_frames: 10
      • sampling_steps: 15
      • use_tinysim_for_images: false
      • tinysim_compressor_mode: 'ENCODE_AND_DECODE'
      • init_deviate: false
      • worldmodel_triton_max_bs: 5
      • cfg: 2.0
      • inference_schedule: 'linear'
      • scheduler_type: 'rf'
    • ▶
      agent: {} 9 keys
      • supercombo: '91e5997f-e410-48f6-a67a-35f1ce53c831/500'
      • restrict_outputs: null
      • exclude_outputs: ['imgs']
      • use_allow_throttle: false
      • model_trained_fps: 5
      • use_worldmodel_as_agent: false
      • use_raw_plan_lat: false
      • desired_curv_smooth_seconds: false
      • desired_accel_smooth_seconds: false
    • ▶
      shadow_agent: {} 0 keys
  • ▶
    raw_video_rollout: {} 10 keys
    • type: 'RawVideoRL'
    • nowindow: false
    • debug: false
    • debug_render: false
    • save_observation_keys: []
    • output_dir: null
    • rerun: false
    • ▶
      env: {} 22 keys
      • debug_window_size: [512, 512]
      • ▶
        ep: {} 7 keys
        • segment: ''
        • augment_desire: false
        • depth_type: 'EASYDEPTH'
        • calibration_source: 'LOG'
        • pipeline_dir: 'http://data-ssd.comma.life/runner/training_2025_07'
        • feature_dir: 'http://data-ssd.comma.life/vae_model_features'
        • ▶
          input_paths: {} 0 keys
      • ▶
        physics: {} 3 keys
        • type: 'BICYCLE'
        • ▶
          bicycle_model_config: {} 1 keys
          • vehicle_platform: 'HONDA_CIVIC'
        • ▶
          ml_model_config: {} 11 keys
          • platform: 'TOYOTA_COROLLA_TSS2'
          • eid: 'effd03d9-472d-44a1-910a-6d49a41eb63f/50'
          • y_range: [-3.0, 3.0]
          • vocab_size: 1024
          • temperature: 0.5
          • use_gt_features: false
          • platform_vec_size: 256
          • platform_map_path: '/home/batman/xx/arena/envs/lib/physics_models/assets/platform_map.txt'
          • lateral_tuning_type: 'AUTO'
          • ▶
            lateral_tuning_params: {} 0 keys
          • ▶
            lateral_initial_state_params: {} 0 keys
      • start: null
      • end: null
      • env: 'xx.arena.envs.rawvideo_env.RawVideoEnv'
      • get_imgs: true
      • allow_resets: true
      • zero_desire: false
      • fps: 5
      • allow_lat: false
      • allow_long: false
      • min_hist: 40
      • compressor_model: null
      • compressor_decoder_model: null
      • compressor_mode: 'NONE'
      • quantized_compressor_latents: false
      • yolo_model: false
      • lpips_model: false
      • sharpness_metric: true
      • future_observation_range_seconds: null
      • with_gt: false
    • ▶
      agent: {} 9 keys
      • supercombo: '91e5997f-e410-48f6-a67a-35f1ce53c831/500'
      • restrict_outputs: null
      • exclude_outputs: ['imgs']
      • use_allow_throttle: false
      • model_trained_fps: 5
      • use_worldmodel_as_agent: false
      • use_raw_plan_lat: false
      • desired_curv_smooth_seconds: false
      • desired_accel_smooth_seconds: false
    • ▶
      shadow_agent: {} 0 keys
  • num_render_vids: 16

Recovery metrics

all segments are run with consistent left and right noise

Worldmodel desired lateral accel (m/s**2) recovery over next 0.5s: left 0.152 and right -0.024

Worldmodel desired lateral (m) recovery in 5s: left 0.798 and right -0.593

average current deviation (m) from GT (max expected recovery): left 0.341 and right -0.340

Long recovery metrics

all segments are run with consistent slowdown and speedup noise

Worldmodel desired accel (m/s**2) recovery over next 0.5s: slow -0.010 and fast -0.009

Worldmodel desired speed - gt speed in 5s (m/s): slow -0.036 and gast -0.170

average current speed (m/s) deviation from GT (max expected recovery): slow -0.052 and fast 0.195

Pose errors

x speed mae: 0.732 - bias: -0.073

y speed mae: 0.051 - bias: 0.010

z speed mae: 0.053 - bias: 0.015

roll rate mae: 0.006 - bias: -0.000

pitch rate mae: 0.006 - bias: -0.000

yaw rate mae: 0.003 - bias: -0.000

LPIPS w.r.t. GT images, Image sharpness, and Yolo entropy

average lpips: 0.256

average big_lpips: 0.226

average sharpness: 233.390

average big_sharpness: 274.468

average yolo_conf: 0.396

average big_yolo_conf: 0.367

Video metrics

Features shape: (1690, 1024)

Frechet distance: 3.306e-02

Maximum-Mean Discrepancy: 2.335e-03

video samples

average video compression rate: 119.379