Hugging Report

• ▶
  root: {} 12 keys
  • report_name: 'analyse_hugging_rl'
  • output_dir: null
  • save_tmp: false
  • read_only: false
  • limit: 10000
  • shuffle: false
  • use_virtual_webgl: false
  • format: 'HTML'
  • ▶
    miniray: {} 8 keys
    • priority: 1000
    • job_name: 'analyse_hugging_rl'
    • queue_name: 'remote2'
    • submit_host: 'redis.comma.internal'
    • result_host: 'data-ram.comma.internal'
    • no_unique_id: false
    • codedir: 'caches/workstation-harald_2025-04-25_17-31-30'
    • force_local: false
  • max_length: null
  • assert_all_passed: false
  • ▶
    rollout: {} 10 keys
    • type: 'MetaDriveRL'
    • nowindow: false
    • debug: false
    • debug_render: true
    • save_observation_keys: ['step_idx', 'debug_rendered']
    • output_dir: null
    • rerun: false
    • ▶
      env: {} 27 keys
      • debug_window_size: [512, 512]
      • ▶
        ep: {} 7 keys
        • segment: ''
        • augment_desire: false
        • depth_type: 'EASYDEPTH'
        • calibration_source: 'LOG'
        • pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
        • feature_dir: null
        • ▶
          input_paths: {} 0 keys
      • start: null
      • end: null
      • env: 'xx.arena.envs.metadrive_env.MetaDriveSimEnv'
      • get_imgs: true
      • allow_resets: true
      • zero_desire: false
      • fps: 20
      • allow_lat: true
      • allow_long: false
      • min_hist: 150
      • compressor_model: null
      • compressor_mode: 'NONE'
      • quantized_compressor_latents: false
      • yolo_model: false
      • lpips_model: false
      • future_observation_range_seconds: null
      • seed: 2927071513
      • port_idx: false
      • noise_model: 'NONE'
      • photo_noise_model: 'NONE'
      • augment_mode: 'SIMPLE'
      • inpaint_mode: 'NONE'
      • ▶
        desire: {} 0 keys
      • future_target_velocity: null
      • ▶
        scenario: {} 6 keys
        • name: 'straight_road'
        • description: ''
        • length: 600
        • ▶
          ego_spawn: {} 5 keys
          • position: [0.0, 0.0]
          • heading: false
          • velocity: 15.0
          • policy: 'ZERO'
          • policy_seed: null
        • ▶
          scene: {} 5 keys
          • traffic_light_plan: []
          • lane_width: 3.8
          • lane_number: 2
          • start_position: [0.0, 0.0]
          • map_pg_blocks: [{'id': 'I', 'pre_block_socket_index': None}, {'id': 'S', 'pre_block_socket_index': 0, 'length': 2048}]
        • npc_spawns: []
    • ▶
      agent: {} 10 keys
      • supercombo: '846c53a4-b886-4942-a4d3-7ea3aff5be8d/400'
      • restrict_outputs: null
      • exclude_outputs: ['imgs']
      • calibrate_speed: false
      • use_allow_throttle: false
      • use_prev_desired_curv: false
      • model_trained_fps: 5
      • use_worldmodel_as_agent: false
      • use_raw_plan_lat: false
      • use_raw_plan_long: true
    • ▶
      shadow_agent: {} 0 keys

Metadrive convergence to lane-center test at 15.0 m/s on straight_road

TEST PASSED ✅

Note: hugging(offset=k) = sim_local_pos(offset=k) - sim_local_pos(offset=0)

Note: recovery = 1 - | hugging(offset) / offset |

Overall mean hugging/offset % = 14.78%, Overall mean recovery % = 85.22%

Metadrive convergence to lane-center test at 2.5 m/s on straight_road

TEST FAILED ❌

Note: hugging(offset=k) = sim_local_pos(offset=k) - sim_local_pos(offset=0)

Note: recovery = 1 - | hugging(offset) / offset |

Overall mean hugging/offset % = 46.81%, Overall mean recovery % = 53.19%

Metadrive convergence to lane-center test at 15.0 m/s on left_turn

TEST PASSED ✅

Note: hugging(offset=k) = sim_local_pos(offset=k) - sim_local_pos(offset=0)

Note: recovery = 1 - | hugging(offset) / offset |

Overall mean hugging/offset % = 10.94%, Overall mean recovery % = 89.06%

Metadrive convergence to lane-center test at 2.5 m/s on left_turn

TEST FAILED ❌

Note: hugging(offset=k) = sim_local_pos(offset=k) - sim_local_pos(offset=0)

Note: recovery = 1 - | hugging(offset) / offset |

Overall mean hugging/offset % = 46.12%, Overall mean recovery % = 53.88%

Metadrive convergence to lane-center test at 15.0 m/s on right_turn

TEST PASSED ✅

Note: hugging(offset=k) = sim_local_pos(offset=k) - sim_local_pos(offset=0)

Note: recovery = 1 - | hugging(offset) / offset |

Overall mean hugging/offset % = 7.22%, Overall mean recovery % = 92.78%

Metadrive convergence to lane-center test at 2.5 m/s on right_turn

TEST FAILED ❌

Note: hugging(offset=k) = sim_local_pos(offset=k) - sim_local_pos(offset=0)

Note: recovery = 1 - | hugging(offset) / offset |

Overall mean hugging/offset % = 44.5%, Overall mean recovery % = 55.5%

Traffic Lights Report 🚦

• ▶
  root: {} 12 keys
  • report_name: 'analyse_traffic_lights'
  • output_dir: null
  • save_tmp: false
  • read_only: false
  • limit: 10000
  • shuffle: false
  • use_virtual_webgl: false
  • format: 'HTML'
  • ▶
    miniray: {} 8 keys
    • priority: 1000
    • job_name: 'analyse_traffic_lights'
    • queue_name: 'remote2'
    • submit_host: 'redis.comma.internal'
    • result_host: 'data-ram.comma.internal'
    • no_unique_id: false
    • codedir: 'caches/workstation-harald_2025-04-25_17-31-30'
    • force_local: false
  • max_length: null
  • assert_all_passed: false
  • ▶
    rollout: {} 10 keys
    • type: 'MetaDriveRL'
    • nowindow: false
    • debug: false
    • debug_render: true
    • save_observation_keys: ['step_idx', 'debug_rendered']
    • output_dir: null
    • rerun: false
    • ▶
      env: {} 27 keys
      • debug_window_size: [512, 512]
      • ▶
        ep: {} 7 keys
        • segment: ''
        • augment_desire: false
        • depth_type: 'EASYDEPTH'
        • calibration_source: 'LOG'
        • pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
        • feature_dir: null
        • ▶
          input_paths: {} 0 keys
      • start: null
      • end: null
      • env: 'xx.arena.envs.metadrive_env.MetaDriveSimEnv'
      • get_imgs: true
      • allow_resets: true
      • zero_desire: false
      • fps: 20
      • allow_lat: true
      • allow_long: true
      • min_hist: 40
      • compressor_model: null
      • compressor_mode: 'NONE'
      • quantized_compressor_latents: false
      • yolo_model: false
      • lpips_model: false
      • future_observation_range_seconds: null
      • seed: 118664988
      • port_idx: false
      • noise_model: 'NONE'
      • photo_noise_model: 'NONE'
      • augment_mode: 'SIMPLE'
      • inpaint_mode: 'NONE'
      • ▶
        desire: {} 0 keys
      • future_target_velocity: null
      • ▶
        scenario: {} 6 keys
        • name: 'straight_road'
        • description: ''
        • length: 600
        • ▶
          ego_spawn: {} 5 keys
          • position: [0.0, 0.0]
          • heading: false
          • velocity: 15.0
          • policy: 'ZERO'
          • policy_seed: null
        • ▶
          scene: {} 5 keys
          • traffic_light_plan: []
          • lane_width: 3.8
          • lane_number: 2
          • start_position: [0.0, 0.0]
          • map_pg_blocks: [{'id': 'I', 'pre_block_socket_index': None}, {'id': 'S', 'pre_block_socket_index': 0, 'length': 2048}]
        • npc_spawns: []
    • ▶
      agent: {} 10 keys
      • supercombo: '846c53a4-b886-4942-a4d3-7ea3aff5be8d/400'
      • restrict_outputs: null
      • exclude_outputs: ['imgs']
      • calibrate_speed: false
      • use_allow_throttle: false
      • use_prev_desired_curv: false
      • model_trained_fps: 5
      • use_worldmodel_as_agent: false
      • use_raw_plan_lat: false
      • use_raw_plan_long: true
    • ▶
      shadow_agent: {} 0 keys

Scenario: red traffic light without NPC

Noise model: Noise.NONE

Stopping success rate: 15 out of 36 (41.67%)

Reasonable (max 3.0m off) success rate: 6 out of 36 (16.67%)

Average distance from stopline: 6.114296648271884

Average absolute distance from stopline: 6.114296648271884

Noise model: Noise.TRAIN_ALL_MLSIM

Stopping success rate: 9 out of 36 (25.00%)

Reasonable (max 3.0m off) success rate: 2 out of 36 (5.56%)

Average distance from stopline: 8.306438936278315

Average absolute distance from stopline: 8.306438936278315

Results per scenario

Final thumbnails for each scene

Note 1: This doesn't cover all the test cases, only the first one for each scene. Use only as visual reference of the map.

Scene: 1 lane[s] | crosswalks | no-npc

Scene: 1 lane[s] | no-crosswalks | no-npc

Scene: 2 lane[s] | crosswalks | no-npc

Scene: 2 lane[s] | no-crosswalks | no-npc

Scene: 3 lane[s] | crosswalks | no-npc

Scene: 3 lane[s] | no-crosswalks | no-npc

Scene: 1 lane[s] | crosswalks | no-npc

Scene: 1 lane[s] | no-crosswalks | no-npc

Scene: 2 lane[s] | crosswalks | no-npc

Scene: 2 lane[s] | no-crosswalks | no-npc

Scene: 3 lane[s] | crosswalks | no-npc

Scene: 3 lane[s] | no-crosswalks | no-npc

Scene: 1 lane[s] | crosswalks | no-npc

Scene: 1 lane[s] | no-crosswalks | no-npc

Scene: 2 lane[s] | crosswalks | no-npc

Scene: 2 lane[s] | no-crosswalks | no-npc

Scene: 3 lane[s] | crosswalks | no-npc

Scene: 3 lane[s] | no-crosswalks | no-npc

Speed Convergence Report

• ▶
  root: {} 12 keys
  • report_name: 'analyse_speed_convergence_rl'
  • output_dir: null
  • save_tmp: false
  • read_only: false
  • limit: 10000
  • shuffle: false
  • use_virtual_webgl: false
  • format: 'HTML'
  • ▶
    miniray: {} 8 keys
    • priority: 1000
    • job_name: 'analyse_speed_convergence_rl'
    • queue_name: 'remote2'
    • submit_host: 'redis.comma.internal'
    • result_host: 'data-ram.comma.internal'
    • no_unique_id: false
    • codedir: 'caches/workstation-harald_2025-04-25_17-31-30'
    • force_local: false
  • max_length: null
  • assert_all_passed: false
  • ▶
    rollout: {} 10 keys
    • type: 'MetaDriveRL'
    • nowindow: false
    • debug: false
    • debug_render: true
    • save_observation_keys: ['step_idx', 'debug_rendered']
    • output_dir: null
    • rerun: false
    • ▶
      env: {} 27 keys
      • debug_window_size: [512, 512]
      • ▶
        ep: {} 7 keys
        • segment: ''
        • augment_desire: false
        • depth_type: 'EASYDEPTH'
        • calibration_source: 'LOG'
        • pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
        • feature_dir: null
        • ▶
          input_paths: {} 0 keys
      • start: null
      • end: null
      • env: 'xx.arena.envs.metadrive_env.MetaDriveSimEnv'
      • get_imgs: true
      • allow_resets: true
      • zero_desire: false
      • fps: 20
      • allow_lat: false
      • allow_long: true
      • min_hist: 40
      • compressor_model: null
      • compressor_mode: 'NONE'
      • quantized_compressor_latents: false
      • yolo_model: false
      • lpips_model: false
      • future_observation_range_seconds: null
      • seed: 5845776
      • port_idx: false
      • noise_model: 'NONE'
      • photo_noise_model: 'NONE'
      • augment_mode: 'SIMPLE'
      • inpaint_mode: 'NONE'
      • ▶
        desire: {} 0 keys
      • future_target_velocity: null
      • ▶
        scenario: {} 6 keys
        • name: 'straight_road'
        • description: ''
        • length: 600
        • ▶
          ego_spawn: {} 5 keys
          • position: [0.0, 0.0]
          • heading: false
          • velocity: 15.0
          • policy: 'ZERO'
          • policy_seed: null
        • ▶
          scene: {} 5 keys
          • traffic_light_plan: []
          • lane_width: 3.8
          • lane_number: 2
          • start_position: [0.0, 0.0]
          • map_pg_blocks: [{'id': 'I', 'pre_block_socket_index': None}, {'id': 'S', 'pre_block_socket_index': 0, 'length': 2048}]
        • npc_spawns: []
    • ▶
      agent: {} 10 keys
      • supercombo: '846c53a4-b886-4942-a4d3-7ea3aff5be8d/400'
      • restrict_outputs: null
      • exclude_outputs: ['imgs']
      • calibrate_speed: false
      • use_allow_throttle: false
      • use_prev_desired_curv: false
      • model_trained_fps: 5
      • use_worldmodel_as_agent: false
      • use_raw_plan_lat: false
      • use_raw_plan_long: true
    • ▶
      shadow_agent: {} 0 keys

straight_road (Number of lanes: 1)

Convergence rate: 2/9 (22.22%)

Note 1: All speeds in meters per second

Note 2: Initial longitudinal history is 40 steps

Note 3: If some test case ends prematurely, it either was reset or went out of map bounds (most likely for high-speed test cases)

Note 4: Convergence is determined by the standard deviation of speed at the last 200 frames. Which means the speed was stable at the end and not that it converged to some specific value.

Scene: empty (opposite-way) | ego | shoulder

Wide camera view of the scene (at the start of the test case)

straight_road (Number of lanes: 3)

Convergence rate: 4/27 (14.81%)

Note 1: All speeds in meters per second

Note 2: Initial longitudinal history is 40 steps

Note 3: If some test case ends prematurely, it either was reset or went out of map bounds (most likely for high-speed test cases)

Note 4: Convergence is determined by the standard deviation of speed at the last 200 frames. Which means the speed was stable at the end and not that it converged to some specific value.

Scene: empty (opposite-way) | empty (same-way) | empty (same-way) | ego | shoulder

Wide camera view of the scene (at the start of the test case)

Scene: empty (opposite-way) | ego | empty (same-way) | empty (same-way) | shoulder

Wide camera view of the scene (at the start of the test case)

Scene: empty (opposite-way) | empty (same-way) | ego | empty (same-way) | shoulder

Wide camera view of the scene (at the start of the test case)

Lane Change Report

• ▶
  root: {} 12 keys
  • report_name: 'analyse_lane_change_rl'
  • output_dir: null
  • save_tmp: false
  • read_only: false
  • limit: 10000
  • shuffle: false
  • use_virtual_webgl: false
  • format: 'HTML'
  • ▶
    miniray: {} 8 keys
    • priority: 1000
    • job_name: 'analyse_lane_change_rl'
    • queue_name: 'remote2'
    • submit_host: 'redis.comma.internal'
    • result_host: 'data-ram.comma.internal'
    • no_unique_id: false
    • codedir: 'caches/workstation-harald_2025-04-25_17-31-30'
    • force_local: false
  • max_length: null
  • assert_all_passed: false
  • ▶
    rollout: {} 10 keys
    • type: 'MetaDriveRL'
    • nowindow: false
    • debug: false
    • debug_render: true
    • save_observation_keys: ['step_idx', 'sm', 'debug_rendered']
    • output_dir: null
    • rerun: false
    • ▶
      env: {} 27 keys
      • debug_window_size: [512, 512]
      • ▶
        ep: {} 7 keys
        • segment: ''
        • augment_desire: false
        • depth_type: 'EASYDEPTH'
        • calibration_source: 'LOG'
        • pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
        • feature_dir: null
        • ▶
          input_paths: {} 0 keys
      • start: null
      • end: null
      • env: 'xx.arena.envs.metadrive_env.MetaDriveSimEnv'
      • get_imgs: true
      • allow_resets: true
      • zero_desire: false
      • fps: 20
      • allow_lat: true
      • allow_long: false
      • min_hist: 150
      • compressor_model: null
      • compressor_mode: 'NONE'
      • quantized_compressor_latents: false
      • yolo_model: false
      • lpips_model: false
      • future_observation_range_seconds: null
      • seed: 451135662
      • port_idx: false
      • noise_model: 'NONE'
      • photo_noise_model: 'NONE'
      • augment_mode: 'SIMPLE'
      • inpaint_mode: 'NONE'
      • ▶
        desire: {} 0 keys
      • future_target_velocity: null
      • ▶
        scenario: {} 6 keys
        • name: 'straight_road'
        • description: ''
        • length: 600
        • ▶
          ego_spawn: {} 5 keys
          • position: [0.0, 0.0]
          • heading: false
          • velocity: 15.0
          • policy: 'ZERO'
          • policy_seed: null
        • ▶
          scene: {} 5 keys
          • traffic_light_plan: []
          • lane_width: 3.8
          • lane_number: 2
          • start_position: [0.0, 0.0]
          • map_pg_blocks: [{'id': 'I', 'pre_block_socket_index': None}, {'id': 'S', 'pre_block_socket_index': 0, 'length': 2048}]
        • npc_spawns: []
    • ▶
      agent: {} 10 keys
      • supercombo: '846c53a4-b886-4942-a4d3-7ea3aff5be8d/400'
      • restrict_outputs: null
      • exclude_outputs: ['imgs']
      • calibrate_speed: false
      • use_allow_throttle: false
      • use_prev_desired_curv: false
      • model_trained_fps: 5
      • use_worldmodel_as_agent: false
      • use_raw_plan_lat: false
      • use_raw_plan_long: true
    • ▶
      shadow_agent: {} 0 keys

Map: straight_road, Speed: 15.0

TEST PASSED ✅

Note 1: Initial lateral history is 150 steps (7.5 sec)

Note 2: Lane change is considered successful if the last 20.0% of the episode is spent in the target lane, and the mean offset from the target lane-center is at most 1.0m

Note 3: If some test case ends prematurely, it either was reset or went out of map bounds

Scene: left lane change

Scene: right lane change

Map: straight_road, Speed: 25.0

TEST PASSED ✅

Note 1: Initial lateral history is 150 steps (7.5 sec)

Note 2: Lane change is considered successful if the last 20.0% of the episode is spent in the target lane, and the mean offset from the target lane-center is at most 1.0m

Note 3: If some test case ends prematurely, it either was reset or went out of map bounds

Scene: left lane change

Scene: right lane change

Oscilllation Report

• ▶
  root: {} 12 keys
  • report_name: 'analyse_oscillation'
  • output_dir: null
  • save_tmp: false
  • read_only: false
  • limit: 10000
  • shuffle: false
  • use_virtual_webgl: false
  • format: 'HTML'
  • ▶
    miniray: {} 8 keys
    • priority: 1000
    • job_name: 'analyse_oscillation'
    • queue_name: 'remote2'
    • submit_host: 'redis.comma.internal'
    • result_host: 'data-ram.comma.internal'
    • no_unique_id: false
    • codedir: 'caches/workstation-harald_2025-04-25_17-31-30'
    • force_local: false
  • max_length: null
  • assert_all_passed: false
  • ▶
    rollout: {} 10 keys
    • type: 'MetaDriveRL'
    • nowindow: false
    • debug: false
    • debug_render: true
    • save_observation_keys: ['step_idx', 'sim_lat_lag', 'sim_lat_antilag', 'debug_rendered']
    • output_dir: null
    • rerun: false
    • ▶
      env: {} 27 keys
      • debug_window_size: [512, 512]
      • ▶
        ep: {} 7 keys
        • segment: ''
        • augment_desire: false
        • depth_type: 'EASYDEPTH'
        • calibration_source: 'LOG'
        • pipeline_dir: 'http://data-ssd.comma.life/runner/training_2024_06'
        • feature_dir: null
        • ▶
          input_paths: {} 0 keys
      • start: null
      • end: null
      • env: 'xx.arena.envs.metadrive_env.MetaDriveSimEnv'
      • get_imgs: true
      • allow_resets: true
      • zero_desire: false
      • fps: 20
      • allow_lat: true
      • allow_long: false
      • min_hist: 40
      • compressor_model: null
      • compressor_mode: 'NONE'
      • quantized_compressor_latents: false
      • yolo_model: false
      • lpips_model: false
      • future_observation_range_seconds: null
      • seed: false
      • port_idx: false
      • noise_model: 'REALISTIC'
      • photo_noise_model: 'NONE'
      • augment_mode: 'SIMPLE'
      • inpaint_mode: 'NONE'
      • ▶
        desire: {} 0 keys
      • future_target_velocity: null
      • ▶
        scenario: {} 6 keys
        • name: 'straight_road'
        • description: ''
        • length: 600
        • ▶
          ego_spawn: {} 5 keys
          • position: [0.0, 0.0]
          • heading: false
          • velocity: 15.0
          • policy: 'ZERO'
          • policy_seed: null
        • ▶
          scene: {} 5 keys
          • traffic_light_plan: []
          • lane_width: 3.8
          • lane_number: 2
          • start_position: [0.0, 0.0]
          • map_pg_blocks: [{'id': 'I', 'pre_block_socket_index': None}, {'id': 'S', 'pre_block_socket_index': 0, 'length': 2048}]
        • npc_spawns: []
    • ▶
      agent: {} 10 keys
      • supercombo: '846c53a4-b886-4942-a4d3-7ea3aff5be8d/400'
      • restrict_outputs: null
      • exclude_outputs: ['imgs']
      • calibrate_speed: false
      • use_allow_throttle: false
      • use_prev_desired_curv: false
      • model_trained_fps: 5
      • use_worldmodel_as_agent: false
      • use_raw_plan_lat: false
      • use_raw_plan_long: true
    • ▶
      shadow_agent: {} 0 keys

Metadrive lane-center oscillation test at 15.0 m/s on straight_road

TEST FAILED ❌

Standard deviation of lateral position: 0.093 m

Oscillation frequency: mean = 0.156 Hz, min = 0.116 Hz, max = 0.302 Hz

Oscillation amplitude: mean = 0.035 m, min = 0.011 m, max = 0.058 m

Oscillation significance: 0.535; just finished cases: 0.535

Number of finished cases: 50/50; passed cases: 49/50

Note 1: Oscillation significance is the product of frequency and amplitude (x100). A value of 1 or less is considered acceptable.

Note 2: The green shaded area in the frequency domain plot represents the acceptable frequency range (0.1-1.0 Hz)

Note 3: nan values indicate that the case ended prematurely due to a reset or out-of-bounds

Metadrive lane-center oscillation test at 30.0 m/s on straight_road

TEST FAILED ❌

Standard deviation of lateral position: 0.175 m

Oscillation frequency: mean = 0.146 Hz, min = 0.116 Hz, max = 0.256 Hz

Oscillation amplitude: mean = 0.064 m, min = 0.023 m, max = 0.120 m

Oscillation significance: 0.922; just finished cases: 0.922

Number of finished cases: 50/50; passed cases: 31/50

Note 1: Oscillation significance is the product of frequency and amplitude (x100). A value of 1 or less is considered acceptable.

Note 2: The green shaded area in the frequency domain plot represents the acceptable frequency range (0.1-1.0 Hz)

Note 3: nan values indicate that the case ended prematurely due to a reset or out-of-bounds