Poster: Generating Experiences for Autonomous Network Defense

Reinforcement Learning (RL) offers a promising path toward developing defenses for the next generation of computer networks. The hope is that RL not only helps to automate network defenses, but in addition, RL finds novel solutions to defend networks that adapt to deal with the increasing complexity of networks and threats. Despite the promise, existing work applying RL to cybersecurity trains cyber defenders on rigid and narrow problem definitions with small computer networks.

Inspired by research that demonstrates that open-ended learning helps agents to adapt rapidly and to generalize to tasks never seen before, we hypothesize that similar approaches can offer a path toward practical RL for network defense. We provide evidence to support this hypothesis. A key aspect to enable generalizable learning is our approach for generating experiences for the learning agent--based on a universe of tasks-in a manner that allows the agent to learn to defend increasingly more complex networks. We show that RL agents can learn to master a reasonably complex network defense task by learning to solve tasks with varying degrees of difficulty. Our preliminary results show that in addition to contributing to the feasibility of mastering complex tasks, this type of experience generation may result in more robust policies.

Overall, our research demonstrates that the collection of experiences that we present to the learning agent is a critical aspect for achieving high performance. We share with the research community our approaches for (i) defining distributions over network defense tasks; (ii) updating distributions as the agent learns; and (iii) maintaining key aspects of tasks invariant to preserve knowledge as tasks vary.

Our experiments are enabled by the second version of our Framework for Advanced Reinforcement Learning for Autonomous Network Defense (FARLAND), which integrates support for action representations, dynamic task selection, and validation of policies in simulation and emulation. Our hope is that by sharing our ideas and results, we foster collaborations and innovation toward the creation of increasingly sophisticated gyms to train network defenders.