Most VR headsets are pretty far off from being comfortable to wear all day, in terms of extra weight on your neck, airflow fo your face/eyes/ears, and eye strain from the dual displays. The models I’ve used are comparatively low resolution, too. That would be my major worry with this approach, followed by fast and precise input like you mentioned.

I have purchased not a VR headset but a headset for drone control: DJI - The World Leader in Camera Drones/Quadcopters for Aerial Photography

This thing has 2x 1920×1080 displays, one per each eye, and doesn’t do any 3D / VR. Basically it connects with HDMI and you can see your screen. That kinda does the job, and you can, if you really try hard to work in that. The eyes still struggle after an hour or so, so that’s not quite there yet I think.

For the input I use https://ultimatehackingkeyboard.com/, and I put it on the board on my laps when I am seated. You don’t have to use mouse with it as it has mouse layer so you control cursor with a mouse.

So yeah, I tried something similar and it kind of works. The weakest link is the headset technology, the VR headset is nowhere close to displaying crisp fonts enough, and the drone ones are better but still not 100%.