Neural Network Learning of Context-Dependent Affordances

Luca Simione

¹ Istituto di Scienze e Tecnologie della Cognizione, CNR, Rome, Italy

² Dipartimento di Psicologia Dinamica, Clinica e Salute, Sapienza, Università di Roma, Rome, Italy

^a[email protected], ^b[email protected], ^c[email protected]

^*Corresponding Author

In this paper, we investigated whether affordances are activated automatically, independently of the context in which they are experienced, or not. The first hypothesis postulates that stimuli affording different actions in different contexts tend to activate all actions initially. The action appropriate to the current context is later selected through a competitive process. The second hypothesis instead postulates that only the action appropriate to the current context is activated. The apparent tension between these two alternative hypotheses constitutes an open issue since, in some cases, experimental evidence supports the context-independent hypothesis, while in other cases it supports the context-dependent hypothesis.  

To study this issue, we trained a deep neural network with stimuli in which action inputs co-varied systematically with visual inputs. The neural network included two separate pathways for encoding visual and action inputs with two hidden layers each, and then a common hidden layer. The training was realized through an auto-associative unsupervised learning algorithm and the testing was conducted by presenting only part of the stimulus to the neural network, to study its generative properties.

As a result of the training process, the network formed visual-action affordances. Furthermore, we conducted the training process in different contexts in which the relation between stimuli and actions varied. The analysis of the obtained results indicates that the network displays both a context-dependent activation of affordances (i.e., the action appropriate to the current context tends to be more activated than the alternative action) and a competitive process that refines action selection (i.e., that increases the offset between the activation of the appropriate and unappropriate actions). Overall, this suggests that the apparent contradiction between the two hypotheses can be resolved. Moreover, our analysis indicates that the greater facility with which colour-action associations are acquired with respect to shape-action associations is because the representation of surface features, such as colour, tends to be more readily available for deeper features, such as shape.

Our results support the feasibility of human-like affordance acquisition in artificial neural networks trained using a deep learning algorithm. This model could be further applied to a number of robotic and applicative scenarios. 

Deep learning, neural network, affordance, context, action

Luca Simione, Anna M Borghi, Stefano Nolfi (2022). Neural Network Learning of Context-Dependent Affordances. Journal of Artificial Intelligence and Systems, 4, 83–106. https://doi.org/10.33969/AIS.2022040106.

[1] J. J. Gibson, The Ecological Approach to Visual Perception, vol. 40, no. 1. Boston: Houghton Mifflin, 1979. doi: 10.2307/989638.

[2] S. Thill, D. Caligiore, A. M. Borghi, T. Ziemke, and G. Baldassarre, “Theories and computational models of affordance and mirror systems: An integrative review,” Neurosci Biobehav Rev, vol. 37, no. 3, pp. 491–521, Mar. 2013, doi: 10.1016/j.neubiorev.2013.01.012.

[3] L. W. Barsalou, “Grounded cognition.,” Annu Rev Psychol, vol. 59, pp. 617–45, Jan. 2008, doi: 10.1146/annurev.psych.59.103006.093639.

[4] A. M. Borghi and D. Pecher, “Introduction to the special topic embodied and grounded cognition.,” Front Psychol, vol. 2, p. 187, Jan. 2011, doi: 10.3389/fpsyg.2011.00187.

[5] J. I. Davis and A. B. Markman, “Embodied cognition as a practical paradigm: introduction to the topic, the future of embodied cognition.,” Top Cogn Sci, vol. 4, no. 4, pp. 685–91, Oct. 2012, doi: 10.1111/j.1756-8765.2012.01227.x.

[6] G. Dove, “Beyond the body?The future of embodied cognition.” Frontiers in Psychology, 2014.

[7] H. E. Matheson and L. W. Barsalou, “Embodiment and Grounding in Cognitive Neuroscience,” Stevens’ Handbook of Experimental Psychology and Cognitive Neuroscience, pp. 1–27, Mar. 2018, doi: 10.1002/9781119170174.EPCN310.

[8] M. Tucker and R. Ellis, “On the relations between seen objects and components of potential actions.,” J Exp Psychol Hum Percept Perform, vol. 24, no. 3, pp. 830–846, Jun. 1998, doi: 10.1037/0096-1523.24.3.830.

[9] M. Tucker and R. Ellis, “The potentiation of grasp types during visual object categorization.,” Vis cogn, vol. 8, pp. 769–800, 2001, doi: 10.1080/13506280042000144.

[10] M. Tucker and R. Ellis, “Action priming by briefly presented objects.,” Acta Psychol (Amst), vol. 116, no. 2, pp. 185–203, Jun. 2004, doi: 10.1016/j.actpsy.2004.01.004.

[11] M. van Elk, H. van Schie, and H. Bekkering, “Action semantics: A unifying conceptual framework for the selective use of multimodal and modality-specific object knowledge,” Phys Life Rev, vol. 11, no. 2, pp. 220–50, Jun. 2013, doi: 10.1016/j.plrev.2013.11.005.

[12] M. Costantini, E. Ambrosini, G. Tieri, C. Sinigaglia, and G. Committeri, “Where does an object trigger an action? An investigation about affordances in space.,” Exp Brain Res, vol. 207, no. 1–2, pp. 95–103, Nov. 2010, doi: 10.1007/s00221-010-2435-8.

[13] M. Costantini, E. Ambrosini, C. Scorolli, and A. M. Borghi, “When objects are close to me: affordances in the peripersonal space.,” Psychon Bull Rev, vol. 18, no. 2, pp. 302–8, Apr. 2011, doi: 10.3758/s13423-011-0054-4.

[14] E. Y. Yoon, G. W. Humphreys, and M. J. Riddoch, “The Paired-Object Affordance Effect.,” J Exp Psychol Hum Percept Perform, vol. 36, no. 4, pp. 812–824, Jul. 2010, Accessed: Jan. 22, 2015. [Online]. Available: http://eric.ed.gov/?id=EJ894200

[15] A. M. Borghi, A. Flumini, N. Natraj, and L. A. Wheaton, “One hand, two objects: emergence of affordance in contexts.,” Brain Cogn, vol. 80, no. 1, pp. 64–73, Oct. 2012, doi: 10.1016/j.bandc.2012.04.007.

[16] S. Kalénine, A. D. Shapiro, A. Flumini, A. M. Borghi, and L. J. Buxbaum, “Visual context modulates potentiation of grasp types during semantic object categorization.,” Psychon Bull Rev, vol. 21, no. 3, pp. 645–51, Jun. 2014, doi: 10.3758/s13423-013-0536-7.

[17] C. Lee, E. Middleton, D. Mirman, S. Kalénine, and L. J. Buxbaum, “Incidental and context-responsive activation of structure- and function-based action features during object identification.,” J Exp Psychol Hum Percept Perform, vol. 39, no. 1, pp. 257–70, Feb. 2013, doi: 10.1037/a0027533.

[18] F. Ferri, G. C. Campione, R. Dalla Volta, C. Gianelli, and M. Gentilucci, “Social requests and social affordances: how they affect the kinematics of motor sequences during interactions between conspecifics.,” PLoS One, vol. 6, no. 1, p. e15855, Jan. 2011, doi: 10.1371/journal.pone.0015855.

[19] L. Sartori, C. Becchio, M. Bulgheroni, and U. Castiello, “Modulation of the action control system by social intention: unexpected social requests override preplanned action.,” J Exp Psychol Hum Percept Perform, vol. 35, no. 5, pp. 1490–500, Oct. 2009, doi: 10.1037/a0015777.

[20] C. Scorolli, M. Miatton, L. A. Wheaton, and A. M. Borghi, “I give you a cup, I get a cup: a kinematic study on social intention.,” Neuropsychologia, vol. 57, pp. 196–204, May 2014, doi: 10.1016/j.neuropsychologia.2014.03.006.

[21] A. M. Borghi, “Affordances, context and sociality,” Synthese, vol. 199, no. 5–6, pp. 12485–12515, Dec. 2021, doi: 10.1007/s11229-018-02044-1.

[22] M. Mustile, F. Giocondo, D. Caligiore, A. M. Borghi, and D. Kourtis, “Motor Inhibition to Dangerous Objects: Electrophysiological Evidence for Task-dependent Aversive Affordances,” J Cogn Neurosci, vol. 33, no. 5, pp. 826–839, Apr. 2021, doi: 10.1162/jocn_a_01690.

[23] M. L. Kellenbach, M. Brett, and K. Patterson, “Actions speak louder than functions: the importance of manipulability and action in tool representation.,” J Cogn Neurosci, vol. 15, no. 1, pp. 30–46, Jan. 2003, doi: 10.1162/089892903321107800.

[24] P. Cisek, “Cortical mechanisms of action selection: the affordance competition hypothesis.,” Philos Trans R Soc Lond B Biol Sci, vol. 362, no. 1485, pp. 1585–99, Sep. 2007, doi: 10.1098/rstb.2007.2054.

[25] G. Pezzulo and P. Cisek, “Navigating the Affordance Landscape: Feedback Control as a Process Model of Behavior and Cognition,” Trends Cogn Sci, vol. 20, no. 6, pp. 414–424, Jun. 2016, doi: 10.1016/j.tics.2016.03.013.

[26] S. P. Tipper, M. a Paul, and A. E. Hayes, “Vision-for-action: the effects of object property discrimination and action state on affordance compatibility effects.,” Psychon Bull Rev, vol. 13, no. 3, pp. 493–8, Jun. 2006, [Online]. Available: http://www.ncbi.nlm.nih.gov/pubmed/17048736.

[27] A. Pellicano, C. Iani, A. M. Borghi, S. Rubichi, and R. Nicoletti, “Simon-like and functional affordance effects with tools: The effects of object perceptual discrimination and object action state,” Q J Exp Psychol, vol. 63, no. 11, pp. 2190–201, 2010, doi: 10.1080/17470218.2010.486903.

[28] G. M. Anderson, D. Heinke, and G. W. Humphreys, “Featural guidance in conjunction search: the contrast between orientation and color.,” J Exp Psychol Hum Percept Perform, vol. 36, no. 5, pp. 1108–27, Oct. 2010, doi: 10.1037/a0017179.

[29] G. Morlino, C. Gianelli, A. M. Borghi, and S. Nolfi, “Learning to Manipulate and Categorize in Human and Artificial Agents.,” Cogn Sci, Jul. 2014, doi: 10.1111/cogs.12130.

[30] L. Simione and S. Nolfi, “The Role of Selective Attention and Action Selection in the Development of Multiple Action Capabilities.,” Conn Sci, vol. 26, no. 4, pp. 389–402, 2014, doi: 10.1080/09540091.2014.942597.

[31] L. Simione and S. Nolfi, “Selection-for-action emerges in neural networks trained to learn spatial associations between stimuli and actions,” Cogn Process, vol. 16, pp. 393–397, 2015, doi: 10.1007/s10339-015-0679-8.

[32] M. Kokic, J. A. Stork, J. A. Haustein, and D. Kragic, “Affordance detection for task-specific grasping using deep learning,” in 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids), Nov. 2017, pp. 91–98. doi: 10.1109/HUMANOIDS.2017.8239542.

[33] T.-T. Do, A. Nguyen, and I. Reid, “AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection,” in 2018 IEEE International Conference on Robotics and Automation (ICRA), May 2018, pp. 1–5. doi: 10.1109/ICRA.2018.8460902.

[34] N. Yamanobe et al., “A Brief Review of Affordance in Robotic Manipulation Research,” Journal of the Robotics Society of Japan, vol. 36, no. 5, pp. 327–337, 2018, doi: 10.7210/JRSJ.36.327.

[35] C. C. Aggarwal, “Neural Networks and Deep Learning,” Neural Networks and Deep Learning, 2018, doi: 10.1007/978-3-319-94463-0.

[36] G. E. Hinton, “Learning multiple layers of representation.,” Trends Cogn Sci, vol. 11, no. 10, pp. 428–34, Oct. 2007, doi: 10.1016/j.tics.2007.09.004.

[37] M. Zorzi, A. Testolin, and I. P. Stoianov, “Modeling language and cognition with deep unsupervised learning: a tutorial overview.,” Front Psychol, vol. 4, no. August, p. 515, Jan. 2013, doi: 10.3389/fpsyg.2013.00515.

[38] G. E. Hinton and R. R. Salakhutdinov, “Reducing the dimensionality of data with neural networks,” Science (1979), vol. 313, no. July, pp. 504–507, 2006, Accessed: Oct. 15, 2013. [Online]. Available: http://www.sciencemag.org/content/313/5786/504.short.

[39] C. Thornton, “Separability is a learner’s best friend,” in 4th Neural Computation and Psychology Workshop, London, 9–11 April 1997, J. Bullinaria, D. Glasspool, and G. Houghton, Eds. London: Springer Verlag, 1997, pp. 40–47. doi: 10.1007/978-1-4471-1546-5_4.

[40] F. Binkofski and L. J. Buxbaum, “Two action systems in the human brain.,” Brain Lang, vol. 127, no. 2, pp. 222–9, Nov. 2013, doi: 10.1016/j.bandl.2012.07.007.

[41] P. Cisek and J. F. Kalaska, “Neural mechanisms for interacting with a world full of action choices.,” Annu Rev Neurosci, vol. 33, no. March, pp. 269–98, Jan. 2010, doi: 10.1146/annurev.neuro.051508.135409.

[42] G. H. de Rosa and J. P. Papa, “Soft-Tempering Deep Belief Networks Parameters Through Genetic Programming,” Journal of Artificial Intelligence and Systems, vol. 1, no. 1, pp. 43–59, 2019, doi: 10.33969/ais.2019.11003.

[43] R. Arkin, Behavior-based robotics. MIT Press, 1998. Accessed: Jul. 12, 2022. [Online]. Available: https://sci-hub.do/https://books.google.com/books?hl=it&lr=&id=mRWT6alZt9oC&oi=fnd&pg=PR11&dq=Arkin+R.+Behavior-based+Robotics.+MIT+Press.+1998&ots=46ZuhfVblx&sig=Dn0hmnMMLwiRYx5yqzzR2TUuOGA.

[44] T. Shu, M. S. Ryoo, and S.-C. Zhu, “Learning Social Affordance for Human-Robot Interaction,” in InternationalJoint Conference on Artificial Intelligence., 2016.

[45] L. Simione and S. Nolfi, “The emergence of selective attention through probabilistic associations between stimuli and actions,” PLoS One, vol. 11, no. 11, p. e0166174, 2016, doi: 10.1371/journal.pone.0166174.

[46] A. Saxena, A. Khanna, and D. Gupta, “Emotion Recognition and Detection Methods: A Comprehensive Survey,” Journal of Artificial Intelligence and Systems, vol. 2, no. 1, pp. 53–79, 2020, doi: 10.33969/AIS.2020.21005.