• Hey, guest user. Hope you're enjoying NeoGAF! Have you considered registering for an account? Come join us and add your take to the daily discourse.

positional tracking: What it is, how its done, and why Valves tracker is a revolution

Krejlooc

Banned
....

What the hell are controllers for?

Why does it need any tracking at all? Cant I move with my fingers?

The way "motion controls" work in VR is very different than how they were used on the Wii or Playstation Move or Kinect. Mostly, in the past, motion controls were merely ways to interpret gesture recognition, i.e. "move your arm along this path and if it's close enough to a path we expect, we will trigger a reaction." This amounted, often, to little more than the same sort of feedback that pressing a button gave. Even when this wasn't the case and they displayed positional tracking (i.e. Pointer controls), there was a large degree of separation between what you did with your body and what you saw happening across the room.

VR engages our ability to recognize the position of our limbs in 3d, without looking at them. This ability is called proprioception. It's how we can touch our hands to our noses with our eyes closed - we have enough cognizance to know where our hands are in 3D space without looking at them. We have varying degrees of proprioception in our bodyparts, but it's especially strong in the hands (and incredibly strong in our head).

Motion controls in VR engage our proprioception. We gain an intimate understanding of where our "inputs" are in space around us. We have entirely new concepts that get expressed to us in VR. As an example - some people can get really immersed in a game and begin to recognize the spatial mapping of the character they represent, even when playing a game across the room. Hence, when the avatar for the player is next to a door, someone can say "the door is to your right" and the player will basically immediately parse the sentence and realize which direction is "right" relative to the player they are controlling. But if, for example, someone said "the key is to the left of your right hand," most people don't have a degree of connection with their avatar across the room to the point where that location in space is instantly recognized.

In VR, it is. If you are looking at something else very intensely, say watching a gauge and you can't take your eyes off of it, while your hands are on some controls, and i tell you that you need to hit the switch to the right of your left hand, you instinctively know where that switch is in relation to not just your hand, but also your entire body. And that switch doesn't even exist!

Motion controls in VR are very, very cool stuff. This is an entire field of study unto itself. Endlessly fascinating.

That's why.

A swivel chair?

Yeah.

This is going to take off like 3d gaming and movies.

If you thought having to wear terrible glasses was bad, now imagine a terrible headset for long periods of time where you have to spin in a chair! Ill take 12!

I don't have to imagine, unlike you.
 

Krejlooc

Banned
True, I was just focusing on the "walking around" pics when it's obviously applicable to any range of motion with your body. Though, is it literal light that tracks your motion? Because in that case I think there would be issues with shadows or positioning, if your hands were obscured by other parts of your body when pivoting, etc.

It's a camera watching for light. The light doesn't track, the light is a landmark for the camera itself to track.
 
....

What the hell are controllers for?

Why does it need any tracking at all? Cant I move with my fingers?

A swivel chair?

Yeah.

This is going to take off like 3d gaming and movies.

If you thought having to wear terrible glasses was bad, now imagine a terrible headset for long periods of time where you have to spin in a chair! Ill take 12!

I just want you to know that you are misunderstanding this entire thread, on a fundamental level.
 
I don't see this as a living room device, and I can't imagine people having dedicated dark rooms void of furniture just to play VR games in this way. It's interesting, but there are just too many levels of inconvenience to use for any sort of widespread adoption or use.

But hey, maybe there will be dedicated VRcades we can go to in the future.

Even in a VRcade (love the name) it still seems impractical. You'd need at least a classroom sized space for each person and only allow walking. although I could envision a nice dungeon crawler with this approach. That is why I think you need to do a combinations of approaches. Have forward movement automatically done with a button press, while everything else tracked 1-to-1 with real movement.
 

injurai

Banned
This is a concept Oculus has been high on for a while. I've done work with it, too - it is really cool stuff. This is definitely something big. Here's a cool representation of it:

eTFBf3F.jpg


The path on the left is what the user thinks he is walking. The path on the right is what he actually walks.

So If I just need to walk in a straight line how will it handle that? Or is the idea forward movement will be mostly handled by controller input i.e. purely simulated.
 
You can do really cool stuff with VR. I don't know if many grasp what it actually does - it's taking over an entire sense. Being a man of science, I believe we are nothing more than series of impulses in our brain, fed by our senses. Hijacking a major sense like sight gives us the power to alter our ability to perceive. We create realities in our heads.

This is the coolest of cool techs, IMO.

The moment VR can hijack our tactile sensors (touch, heat, pressure, weight, texture) alongside moving freely in a 3d mesh (say physically run from whiterun to riverwood in skyrim and feeling slopes and hills) is when I will officially and totally lose my shit.
 
So If I just need to walk in a straight line how will it handle that? Or is the idea forward movement will be mostly handled by controller input i.e. purely simulated.

Here's the cool thing: when you walk forward, you move towards an object; your body doesn't know a "perfectly" straight direction. If you close your eyes, because you lack the visual input of the object you are walking towards, you will not walk in a straight line. So if the headset keeps making subtle adjustments to your view, your body will "correct" it, thus making you move in a circle. It's less of a pronounced effect, but it will give you some room.

Even in a VRcade (love the name) it still seems impractical. You'd need at least a classroom sized space for each person and only allow walking. although I could envision a nice dungeon crawler with this approach. That is why I think you need to do a combinations of approaches. Have forward movement automatically done with a button press, while everything else tracked 1-to-1 with real movement.
This is the ideal approach, I feel. Have major distances of movement still be done via a button, and then directional orientation and smaller amounts of movement done by the player themselves.
 

Krejlooc

Banned
Even in a VRcade (love the name) it still seems impractical. You'd need at least a classroom sized space for each person and only allow walking. although I could envision a nice dungeon crawler with this approach. That is why I think you need to do a combinations of approaches. Have forward movement automatically done with a button press, while everything else tracked 1-to-1 with real movement.

With regards to locomotion in VR, NASA has some really cool, very impractical mechanical tracking rigs for motion. They are very accurate and let you walk in place - not on a treadmill but actually moving small platforms along your feet. But, like I said, it's not really feasible for mass market.

This is how ideas get started, though. When you are faced with a problem, your first step is to break it down into bite sized chunks, and figure out how to solve each individual problem regardless of feasibility or practicality. Once you know how to solve your large problem "correctly," you then work on cheating and making the infeasible, feasible.

So right now, with regards to locomotion, we know how to do it right. Now we just need to engineer a way to do it right easily.
 

Arkage

Banned
It's a camera watching for light. The light doesn't track, the light is a landmark for the camera itself to track.

Hmmm, still not sure how a camera would track light in a 1:1 way for your entire body - a camera is only viewing a 2D profile of yourself. For example, how would this track the motion of your thumb if it's behind your fingers, and the camera is in front of your fingers?
 

Krejlooc

Banned
So If I just need to walk in a straight line how will it handle that? Or is the idea forward movement will be mostly handled by controller input i.e. purely simulated.

Same method, just with a bigger turning radius.

Damn, and Gabe plans to give away the tech for other companies to use it. It's pretty awesome

Exactly, I was floored. Granted, I'm obviously a fan of Valve in general - but I don't really play fanboy favorites with VR stuff. It's all the same medium to me. Giving away this technology really bolsters the medium. This is the type of awesome stuff you always wished would happen in business, but logically never would.
 
Hmmm, still not sure how a camera would track light in a 1:1 way for your entire body - a camera is only viewing a 2D profile of yourself. For example, how would this track the motion of your thumb if it's behind your fingers, and the camera is in front of your fingers?

This doesn't track your body. Rather, the headset and the controllers track themselves, using an entire room, and obviously you are wearing the headset to tell the game where YOU are.

The cameras are ON THE HEADSET AND CONTROLLERS. Your entire room becomes one big marker for them to track, FROM ALL ANGLES.
 
That's because Kinect isn't really using optical tracking, not like I'm describing. Kinect uses time of flight cameras to measure how long it takes for a fired beam of light to reach a surface and reflect back. By counting the fractions of a second it takes to complete this process, they can map a single point of "depth." By doing this hundreds of thousands of times a second, you can form a depth map from this data. From this 3D depth map, Microsoft then does lots of post-image processing to try and figure out the skeletal shape of the figure being captured, and then maps position onto that skeleton.

It's not a very accurate or quick process.

There is also magnetic induction tracking, which the razer hydras and sixense stems use, but those have problems as well. I was mainly concentrating on the 3 biggest methods of positional tracking used - pure IMU data, outside-in positional tracking, and inside-out positional tracking.

As far as our collective knowledge about positional tracking is concerned, inside-out positional tracking is the current apex of positional tracking outside of things like mechanically tracking your limbs (i.e. using servos attached to your body).

Ah I see. Thanks for the explanation.
 

injurai

Banned
Here's the cool thing: when you walk forward, you move towards an object; your body doesn't know a "perfectly" straight direction. If you close your eyes, because you lack the visual input of the object you are walking towards, you will not walk in a straight line. So if the headset keeps making subtle adjustments to your view, your body will "correct" it, thus making you move in a circle. It's less of a pronounced effect, but it will give you some room.
.

Same method, just with a bigger turning radius.

Yeah I sort of expected as much. Sort of diminishing returns for anything that must be handled more gradually.

Imagine having an entire gymnasium though to play around in.
 

Krejlooc

Banned
Hmmm, still not sure how a camera would track light in a 1:1 way for your entire body - a camera is only viewing a 2D profile of yourself. For example, how would this track the motion of your thumb if it's behind your fingers, and the camera is in front of your fingers?

It doesn't track for your entire body, it tracks for that specific point, which you then attach to a point on your body that you want tracked. For example - if you want to track your head, and both hands, you need 3 cameras watching light independently.

there are other ways you can glean the position of your fingers going forward - using stuff like forward kinematics relative to your body's absolute position. These concepts make a bit more sense if you're familiar with 3D modeling - you can basically treat parts of your body as "object space" with your body (being tracked in absolution at certain points) being world-space.
 
I really VR to work, but no thanks to any form of locomotion in a room. It's just not going to work outside of commercial applications. Give me a couple single-handed controllers - hell give me a 360 controller - anything but standing and moving around. I don't want to even jog in place or anything.
 

Krejlooc

Banned
I really VR to work, but no thanks to any form of locomotion in a room. It's just not going to work outside of commercial applications. Give me a couple single-handed controllers - hell give me a 360 controller - anything but standing and moving around. I don't want to even jog in place or anything.

truthfully, there is no real solution to yaw rotation in VR. Some people, like myself, are lucky and have iron stomachs. We are rare. This is why gear VR advocates using a swivvel chair - rotating without affecting your vestibular system makes people sick. The only ways to do that is to either rely on a person actually turning, or relying on galvanic vestibular stimulation - attaching small electrodes to the back of your ear and shocking your brain into making you think you are spinning. I am a huge proponent of VR, and even I won't try galvanic vestibular stimulation.

That said, you can make a game where you don't need to turn around, and having a cockpit landmark to keep your orientation in space feels fine (i.e. turning a car in VR) but normal FPS controls as you know them? Not going to be fun in VR.
 
I really VR to work, but no thanks to any form of locomotion in a room. It's just not going to work outside of commercial applications. Give me a couple single-handed controllers - hell give me a 360 controller - anything but standing and moving around. I don't want to even jog in place or anything.

Haha for me it's the other way around; I hope someday I will be able to actually "walk / run" in those great open world game with jaw dropping scenery. Getting puffed from travelling from one town to another is my dream game :p
 
For some reason I have an urge to knock over someone's Lighthouse base station while they're in VR, just to see the reaction. I assume it'll be pretty dramatic.
Damn, and Gabe plans to give away the tech for other companies to use it. It's pretty awesome

I'm sure there are caveats (most likely to do with Steam or APIs), but it's in their best interest to increase adoption and make it some sort of industry standard
 
I really VR to work, but no thanks to any form of locomotion in a room. It's just not going to work outside of commercial applications. Give me a couple single-handed controllers - hell give me a 360 controller - anything but standing and moving around. I don't want to even jog in place or anything.

You have single handed controllers. Two of them in fact. You do not NEED to walk around with this. The major advantage of this is say you are playing a VR game, and you come up to a desk. Instead of constantly using the trackpad/joystick to adjust your position, you can simply lean over, take a step here, take a step there, etc.

In fact, this has some REALLY HUGE advantages over the current DK2 method. I currently have the DK2 and the Razer Hydra. But because both of these have a "home location", something they need to be pointed at, I am restricted from being able to turn around in my nice chair, or get up and walk, or do any number of things. So, I have to rely purely on my controller to rotate myself, which in VR is very uncomfortable. With the Vive, given a rotating chair, I would NEVER have to worry about whether I could or could not turn. I could lean infinitely without worrying about falling out of view of a camera.

This is huge, even without considering walking around.
 

Krejlooc

Banned
For some reason I have an urge to knock over someone's Lighthouse base station while they're in VR, just to see the reaction. I assume it'll be pretty dramatic.

It wouldn't do anything.


I'm sure there are caveats (most likely to do with Steam or APIs), but it's in their best interest to increase adoption and make it some sort of industry standard

It's really just a thing that projects landmarks.
 

injurai

Banned
Haha for me it's the other way around; I hope someday I will be able to actually "walk / run" in those great open world game with jaw dropping scenery. Getting puffed from travelling from one town to another is my dream game :p

I'm imagining full body haptic feed back suits. Where you reach for things that aren't there and negative resistance feels like the positive normal force of an object.
 

Tain

Member
So I'm a little uncertain on this:

I read that Valve's solution can read and display walls around you. Does that mean it reads anything in the 3D space you're occupying? Can your body become a natural collision volume?
 

Ishan

Junior Member
blue pill or the red pill?

EDIT: This seems interesting going to bookmark this and return to this after this weekend when im not so swamped. (I've worked with the kinect so I'm aware of some of the tech in it and got knows occlusion is such a pain especially when we were trying to use it to track human joints for medical reasons)
 

Krejlooc

Banned
If the landmarks move in unison wouldn't the sensors assume the position is moving?

oh, I thought you we thinking it'd do something like destroy a user's brain or other silly suggestions I've seen. Yes, it would feel akin to someone bumping their head.

So I'm a little uncertain on this:

I read that Valve's solution can read and display walls around you. Does that mean it reads anything in the 3D space you're occupying? Can your body become a natural collision volume?

that's on the programmer, but certainly.
 

Krejlooc

Banned
I'm imagining full body haptic feed back suits. Where you reach for things that aren't there and negative resistance feels like the positive normal force of an object.

This is dexmo:

AnchPjH.png


The spider-like appendages are break points. They don't pull back on your fingers (that's a big no-no) but rather prevent your fingers from moving forward. By tracking the position of your fingers in 3D space using forward kinematics, you can watch for collision between them and in-game objects, and cause the breaks to kick in when they intersect. By doing this, you can feel objects that don't exist - spheres and walls and such.

The obvious problem at the moment is durability and that touch with this thing is binary - you don't feel texture, merely only blunt "existence" or "non-existence."

But it's an incredibly cool first step. This stuff is cheap, too - they had planned to sell them at $100 for a pair. They went into hibernation a while back to let their concept cook a bit more.
 

injurai

Banned
This thread is actually getting more excited for VR than any of the more tangible things being shown. Guess my imagination is just stronger, but the prospects for the future are here. Who knows where VR will end up as it becomes a mature facet of the gaming experience.

edit: ^ oh wow. That is exactly what I had it mind. Albeit it thresholds seem better than forceful resistance, I could see that causing harm now.
 
This is dexmo:

AnchPjH.png


The spider-like appendages are break points. They don't pull back on your fingers (that's a big no-no) but rather prevent your fingers from moving forward. By tracking the position of your fingers in 3D space using forward kinematics, you can watch for collision between them and in-game objects, and cause the breaks to kick in when they intersect. By doing this, you can feel objects that don't exist - spheres and walls and such.

The obvious problem at the moment is durability and that touch with this thing is binary - you don't feel texture, merely only blunt "existence" or "non-existence."

But it's an incredibly cool first step. This stuff is cheap, too - they had planned to sell them at $100 for a pair. They went into hibernation a while back to let their concept cook a bit more.
Theoretically does it need to be in that form factor (large spider on your hand) or could it be made as a glove?
 

Krejlooc

Banned
This thread is actually getting more excited for VR than any of the more tangible things being shown. Guess my imagination is just stronger, but the prospects for the future are here. Who knows where VR will end up as it becomes a mature facet of the gaming experience.

In terms of VR, gaming isn't even very high up on my list of priorities. I think this is fundamentally cool technology if people will just think a bit out of the box. I do work with MD Anderson using VR, we are doing some incredibly cool stuff. VR Pain managment, as an example, is an emerging field.

This tech can do wonderful things, well beyond merely providing targets to shoot at with virtual guns.

Theoretically does it need to be in that form factor (large spider on your hand) or could it be made as a glove?

As it works currently, it needs to be in that shape.

There are also entire sleeves you can put your arm into to track your whole arm and make it stop. I've seen people lean on virtual bar tops before.
 
I brought this up in another thread, but is there a calibration process for the Vive/lighthouse?

Because, to my understanding of how this tech works, unless your computer knows the dimensions of the entire room, the tracking is only going to be relative to last position - not full 3D positioning within a space.

If my wand is pointing at the right lounge room wall, bathed in dots, without knowing exactly where those dots are in 3D space, you can't then determine where the wand is.
 
With regards to locomotion in VR, NASA has some really cool, very impractical mechanical tracking rigs for motion. They are very accurate and let you walk in place - not on a treadmill but actually moving small platforms along your feet. But, like I said, it's not really feasible for mass market.

This is how ideas get started, though. When you are faced with a problem, your first step is to break it down into bite sized chunks, and figure out how to solve each individual problem regardless of feasibility or practicality. Once you know how to solve your large problem "correctly," you then work on cheating and making the infeasible, feasible.

So right now, with regards to locomotion, we know how to do it right. Now we just need to engineer a way to do it right easily.

Agreed, but the Vive is releasing this holiday season so we need a solution, even if it is not perfect, right now.

I haven't ever used VR but the question I'd like to know the answer to is, how disorienting is it to be moving forward in VR but have your body stay still in real life? I suppose/hope that it wouldn't be that bad since if you were moving at constant speed you wouldn't feel any acceleration in real life anyway. So I guess the real question is how jarring is it to go from a standstill to full running speed in an instant in VR?
 
I brought this up in another thread, but is there a calibration process for the Vive/lighthouse?

Because, to my understanding of how this tech works, unless your computer knows the dimensions of the entire room, the tracking is only going to be relative to last position - not full 3D positioning within a space.

If my wand is pointing at the right lounge room wall, bathed in dots, without knowing exactly where those dots are in 3D space, you can't then determine where the wand is.

If there's a special marker on the base station or on the headset then it should be able to orient itself, no?
 

Krejlooc

Banned
Aren't you severely limited by your HDMI cable length and positioning? Or am I picturing this whole setup wrong?

The convention currently is to use boom mic stands to run the cables down from above you. So yes, HDMI length matters.

I brought this up in another thread, but is there a calibration process for the Vive/lighthouse?

Because, to my understanding of how this tech works, unless your computer knows the dimensions of the entire room, the tracking is only going to be relative to last position - not full 3D positioning within a space.

If my wand is pointing at the right lounge room wall, bathed in dots, without knowing exactly where those dots are in 3D space, you can't then determine where the wand is.

if you were attempting to demonstrate where an object is in a real room, yes. But with virtual reality, you have no frame of reference for the real world, so that doesn't really matter. All position in VR is relative, even if you arrive at that relative position using absolute positioning IRL.

Agreed, but the Vive is releasing this holiday season so we need a solution, even if it is not perfect, right now.

I haven't ever used VR but the question I'd like to know the answer to is, how disorienting is it to be moving forward in VR but have your body stay still in real life? I suppose/hope that it wouldn't be that bad since if you were moving at constant speed you wouldn't feel any acceleration in real life anyway. So I guess the real question is how jarring is it to go from a standstill to full running speed in an instant in VR?

When you move forward in VR, you experience a slight "hit" to your stomach at the start of your motion, but it's not continuous. Motion along a forward or backwards vector isn't bad. What is killer is yaw rotation. That alone is probably the biggest challenge facing VR.
 

JaggedSac

Member
You can pull off redirected walking in the space of a bedroom - I did. But yes, you won't be running with this stuff. That falls on the developer to recognize good design practices.

An empty bedroom? Also, wires will mess with moving and turning unless wireless. Regardless, I don't think moving around in real life will be useful during these first generation of devices. Might be cool to go somewhere that has empty spaces for you to play this stuff in. I sure am not moving stuff around to use thus stuff.
 
If there's a special marker on the base station or on the headset then it should be able to orient itself, no?

Pretty sure that's not how it works. It's looking at the constellation on your walls for position - to avoid issues with occlusion.

PS Move works, because the PS knows exactly how big the ball is.
But a dot projected onto a wall at X distance, will be a different size.
 

Krejlooc

Banned
blue pill or the red pill?

EDIT: This seems interesting going to bookmark this and return to this after this weekend when im not so swamped. (I've worked with the kinect so I'm aware of some of the tech in it and got knows occlusion is such a pain especially when we were trying to use it to track human joints for medical reasons)

Leap Motion has very similar problems, so will Nimble (oculus bought them).

I'm just not big on using 3D cameras for positional tracking. Now, using 3D cameras as a means to capture 3D space? That shit is amazing. Kinect is so cool, it's an awesome camera, but a crappy input device.
 
Just read the OP and nothing more and I want to say "great thread op".

This was way more interesting than I anticipated when I clicked on the thread.
 

Krejlooc

Banned
Pretty sure that's not how it works. It's looking at the constellation on your walls for position - to avoid issues with occlusion.

PS Move works, because the PS knows exactly how big the ball is.
But a dot projected onto a wall at X distance, will be a different size.

Stuff like this is precisely why, as I said earlier in the thread, knowing the process is easy, but actually pulling it off is hard. I honestly have no idea how valve gets around this problem. I suspect we'll learn soon enough, however.

They flat-out have some smart people working there.
 
Pretty sure that's not how it works. It's looking at the constellation on your walls for position - to avoid issues with occlusion.

PS Move works, because the PS knows exactly how big the ball is.
But a dot projected onto a wall at X distance, will be a different size.

I'm assuming that there's two sets of sensors (one on the headset and one on the controller) so if they all see each other they can do some sort of triangulation about where each piece is relative to the space
 

JaggedSac

Member
Kind of. My bedrooms tend to have the middle of them open, it's just how I've always grown up. Shelves along the walls, not much in the middle of the room.

What dimensions we talking here?

Also, is there a link to Valve discussing this stuff?
 
When you move forward in VR, you experience a slight "hit" to your stomach at the start of your motion, but it's not continuous. Motion along a forward or backwards vector isn't bad. What is killer is yaw rotation. That alone is probably the biggest challenge facing VR.

To add to this, using the Vive's full room tracking will allow you to spin indefinitely, removing the need to turn virtually as opposed to turning your body -- so it should fix this.
 

JaggedSac

Member
To add to this, using the Vive's full room tracking will allow you to spin indefinitely, removing the need to turn virtually as opposed to turning your body -- so it should fix this.

Wires. You gonna be wrapped like a mummy if you spin indefinitely.

aw jeeze, I'm terrible at estimating distance. Maybe 20' x 20'?

That's a decent chunk of space.


Your enthusiasm is cool. Do work and make cool shit man.
 
if you were attempting to demonstrate where an object is in a real room, yes. But with virtual reality, you have no frame of reference for the real world, so that doesn't really matter. All position in VR is relative, even if you arrive at that relative position using absolute positioning IRL.

But we're just talking about tracking the position of the wand in 3D space, real or otherwise.

The Vive has 3 devices, each tracked independently with inside-out tracking.
If there's no overlap in the dots they're tracking for their respective position, then there would be an issue with positioning each device relative to one another. ie your hands could be further away from your head than they're supposed to be.

Unless, your PC has a complete picture of where the dots are in the room.
 

Krejlooc

Banned
I can't imagine that being a part of any consumer version.

It's pretty much how VR is done at the moment, it's a natural limitation of the tech. It's either that, or let it drag behind you like a tail.

GOOD NEWS, THOUGH - Valve is turning their concentration on adapting their Steam Link tech to make VR wireless. I honestly expect it'll take them a decade of research, but valve has a propensity of surprising the shit out of people.
 
Top Bottom