I'm trying to create a panning control for a camera in bevy, but I can't seem to get the panning logic right when the camera is rotated. It works fine if the camera transform is directly facing the XY plane. This works with any axis as the "up" direction:

Transform::from_xyz(0., 0., 1.).looking_at(Vec3::ZERO, Vec3::X)

But, it doesn't move correctly when the camera isn't aligned to the plane, like so:

Transform::from_xyz(1., -2., 2.).looking_at(Vec3::ZERO, Vec3::Z)

I want the camera to pan over the XY plane at a fixed Z height. How can I convert this 2D implementation to a proper 3D implementation?

Here's the pseudocode for my current logic, for those not familiar with bevy:

Vec2 delta_mouse = get_mouse_movement_since_last_frame();
Angle camera_rotation = camera.get_rotation_axis_and_angle().angle;
Vec2 rotation_mat = { x: cos(camera_rotation), y: sin(camera_rotation) };
Vec2 rotated_delta_mouse = {
    x: delta_mouse.x * rotation_mat.x - delta_mouse.y * rotation_mat.y,
    y: delta_mouse.y * rotation_mat.x + delta_mouse.x * rotation_mat.y
};
camera.translation += rotated_delta_mouse;

And the full rust code I am using now:

fn drag_camera(
    input: Res<ButtonInput<MouseButton>>,
    mut camera_query: Query<&mut Transform, With<Camera>>,
    window_query: Query<&Window>,
    mut ev_motion: EventReader<MouseMotion>,
    camera_info: Res<CameraSettings>,
) {
    let pan_button = MouseButton::Left;
    if !input.pressed(pan_button) {
        return;
    }
    let mut pan = Vec2::ZERO;
    for ev in ev_motion.read() {
        pan += ev.delta;
    }
    let mut scale_factor = camera_info.zoom_scale * 2.;
    let window = window_query.single();
    scale_factor /= window.resolution.physical_height() as f32;

    let mut transform = camera_query.single_mut();
    let rotation_angle = transform.rotation.to_axis_angle().1;
    let pan_rotated = pan.rotate(Vec2::from_angle(rotation_angle));

    transform.translation.x -= pan_rotated.x * scale_factor;
    transform.translation.y += pan_rotated.y * scale_factor;
}