Policy gradient reinforcement learning with environmental dynamics and action-values in policies

Seiji Ishihara, Harukazu Igarashi

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The knowledge concerning an agent's policies consists of two types: the environmental dynamics for defining state transitions around the agent, and the behavior knowledge for solving a given task. However, these two types of information, which are usually combined into state-value or action-value functions, are learned together by conventional reinforcement learning. If they are separated and learned independently, either might be reused in other tasks or environments. In our previous work, we presented learning rules using policy gradients with an objective function, which consists of two types of parameters representing environmental dynamics and behavior knowledge, to separate the learning for each type. In such a learning framework, state-values were used as an example of the set of parameters corresponding to behavior knowledge. By the simulation results on a pursuit problem, our method properly learned hunter-agent policies and reused either bit of knowledge. In this paper, we adopt action-values as a set of parameters in the objective function instead of state-values and present learning rules for the function. Simulation results on the same pursuit problem as in our previous work show that such parameters and learning rules are also useful.

Original languageEnglish
Title of host publicationKnowledge-Based and Intelligent Information and Engineering Systems - 15th International Conference, KES 2011, Proceedings
Pages120-130
Number of pages11
EditionPART 1
DOIs
Publication statusPublished - 2011 Sep 29
Event15th International Conference on Knowledge-Based and Intelligent Information and Engineering Systems, KES 2011 - Kaiserslautern, Germany
Duration: 2011 Sep 122011 Sep 14

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
NumberPART 1
Volume6881 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference15th International Conference on Knowledge-Based and Intelligent Information and Engineering Systems, KES 2011
CountryGermany
CityKaiserslautern
Period11/9/1211/9/14

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint Dive into the research topics of 'Policy gradient reinforcement learning with environmental dynamics and action-values in policies'. Together they form a unique fingerprint.

  • Cite this

    Ishihara, S., & Igarashi, H. (2011). Policy gradient reinforcement learning with environmental dynamics and action-values in policies. In Knowledge-Based and Intelligent Information and Engineering Systems - 15th International Conference, KES 2011, Proceedings (PART 1 ed., pp. 120-130). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 6881 LNAI, No. PART 1). https://doi.org/10.1007/978-3-642-23851-2_13