Gesture or Action: Experiment

There are two potential experiments that could help detect and understand the difference between gesture and action in IVEs.

Gesture, Gestue w/ Wand, Action

In this experiment, learners are asked to memorise 30 words. Each word is a verb, related to a different object presented in front of them on a podium. There are three groups, and each group is a different method for interacting in the scenario: gesture, gesture with wand, and action. They differ as follows:

  • Gesture: Input is made by doing the correct gesture in the air, away from the object
  • Wand Gesture: Input is made by grabbing the “gesture detection object” in the IVE, and making a gesture with it
  • Action: Input is made by grabbing the actual VR object, and doing the correct gesture with it.

Question regaring action: do we show feedback from the action? Like, milk getting stirred into a cup, or dirt being wiped away?

The results from this would allow us to understand the differences between gesture and action, as follows:

  • If Gesture is better: Gripping something or acting on another object harms memorisation
  • If Wand Gesture is better: Weird result.
  • If Action is better: Action is best-suited for memorisation than gesture
  • If Gesture AND Wand Gesture is better: Acting on the context-specific object harms learning!
  • If Action AND Gesture is better: Using a wand to gesture is an obscure distraction?
  • If Action AND Wand Gesture is better: The act of gripping/grabbing something (interactionalist theory?) in the IVE helps learning, OR applying a gesture to a/any object helps learning

The wand is included to allow a state that has interaction with an object in the IVE, but not interaction with the object.

There is an argument to say that we could remove Wand condition, or remove Gesture without the wand condition.

Parameters

Within-subject design. 10 words per scenario. 30 words total.

Each scenario will:

  • Pre-test
  • Scenario interaction introduction
  • Go through the words 3 times
  • Post-test
  • Rate how-well they felt they learned, how they enjoyed it
  • Store time per session, time per word

Gesture and Action on Verbs, Gesture and Action on Nouns

In this experiment, there are two stages.

In the first, learners are asked to memorise 20 words. Each word is a verb, related to a different object presented in front of them on a podium. There are two groups, and each group is a different method for interacting in the scenario: gesture, and action. They differ as follows:

  • Gesture: Input is made by doing the correct gesture in the air, away from the object
  • Action: Input is made by grabbing the actual VR object, and doing the correct gesture with it.

If there is a memorisation difference, then the cause is either:

  • (A) because of touching the object and not
  • (B) because of difference between gesture and action

The second stage, participants are asked to use gesture in air to rotate object, or actually move object, to memorise a noun.

If there is a memorisation difference, it is caused by:

  • (A) because of “touching” or not “touching” the object

Therefore with the combination of results between the to stages, if:

  • 1 (has difference) + 2 (has difference) = “physically” manipulating an object makes a difference to learning (and there could also be a difference in the impact of gesture, but we can't know)
  • 1 (has difference) + 2 (has no difference) = bodily action on an object makes a learning difference compared with gesture
  • 1 (has no difference) + 2 (has no difference) = “physically” manipulating does not make a difference (and there could also be a difference in the impact of gesture, but we can't know)
  • 1 (has no difference) + 2 (has difference) = bodily action on an object makes a learning difference compared with no action, but gesture makes up for it