Sobotify: A Framework for Turning Robots into Social Robots

Sobotify is a software framework, which aims at simplifying the process of using robots in the field of social robotics. This paper delineates the design and usage of the framework. With Sobotify, even non-technical people should be enabled to use robots for their specific purposes, such as teachers in a classroom, therapists in a physiological or psychological therapy or childcare workers in kindergarten. During the development process of Sobotify, feedback from teachers at a vocational school have been taken into account in order to adjust the tools to their needs. The framework is designed to work with different robots including both humanoid (NAO and Pepper) and non-humanoid robots such as toy robots (Cozmo) with advanced abilities as well as very simple toy robots (MyKeepon). The framework was tested by two research works which proved that Sobotify is applicable in different setups. Further development is already planned for the next months, e.g. integration of additional robots and extension of tools.

educational researchers from various backgrounds to successfully utilize robots in teaching, the technical knowledge and skill barriers to adopt robots in the application scenarios act as a big hurdle to overcome.Therefore, we developed a social robotics authoring platform, the Sobotify framework, which aims at allowing any user with basic computer skills to use the robot of their choice for their application scenarios in this feld.To make the platform to respect user's data privacy and to adhere the open source paradigm, Sobotify focuses on utilizing ofine and open-source software.In this paper we describe the design of this framework and three interfaces that users can use: a gesture GUI, an app GUI and the project spreadsheets.The current status of Sobotify already provides a basic set of tools, applications and supported robots, however it is intended to be continuously extended and further developed.Besides the main developer, also the community is invited to contribute new applications and tools or add support for new robots.During this process, a continuous exchange with the target group is planned, in order to adjust to their needs.

PURPOSE
Customizing a robot's behavior usually requires users to use the vendor-specifc software and hardware.When changing to a new robot, the user has to adapt to the new robot and recreate the already designed behavior (e.g.written as software code) for the new robot.Using robots developed by the same company may reduce this issue due to compatibility but as there are various diferent robots and more coming out, one should be able to try and explore diferent robots which suit the needs.However, this means that users should be able to know the hardware and software requirements of the selected robots in their scenarios.This difculty limits not only the usages of robots but also the research in social robotics to be prevalent in using specifc robots.The key operation that Sobotify performs is freeing users from considering robot specifc software environment when utilizing robots in their scenarios.For instance, when users like to use both humanoid robot such as Pepper or NAO and also non-humanoid robots such as Cozmo or MyKeepon, users should be profcient in knowing about the hardware and software requirements of all four diferent robots.Users are limited to the specifc features of each robot and usually applications developed for a robot are usually not transferable to other robots, unless two robots are highly compatible (e.g.Pepper and NAO).In addition, hardware features (e.g.RAM, storage size, camera resolution, joints) difer signifcantly from each other.Not to say the requirement of programming skills or at least for slightly advanced applications technical understanding is necessary.Sobotify aims at overcoming these above mentioned restrictions, by providing a number of predefned applications, which can be customized without programming knowledge.For example, with Sobotify, user can change the content of a quiz on a spreadsheet (e.g. from simple math questions to geography questions) without knowing the programming language nor the specifcations of the robot.Additionally, it provides gesture and speech extraction features for adding user defned gestures for humanoid robots.It supports a growing number of robots and it ofers a simple installation procedure, currently supporting Windows 10 (Linux and MacOS support planned).Furthermore, Sobotify provides many features, such as speech, facial and emotion detection/recognition, independent of the robot's abilities by implementing these features on an external computer/laptop instead of executing them on the robot's system.Also, as robots' computational performance is often too limited to execute advanced processing task, such as demanding machine learning based applications, running these task on an external computer resolves this issue.
For application developers it ofers a unifed API for creating applications for diferent social robots and numerous tools, which are typically required in social robotics, see the list below.
And for software developers, who want to extend the tool set or add new robots, Sobotify provides the ability to add new tools easily by providing a simple standard messaging interface (MQTT) to interconnect tools with each others.Also new robots can be added easily, it requires mainly defning access to the robots' features (such as speak, move and camera and other sensors and actuators).
The framework aims at providing an easy way to program and control diferent robots in a unifed environment.It includes a set of tools typically required in social robotics: • speaking (including text to speech): for verbal interaction with the user • gestures (recording and synchronized playback): for supporting the speech with descriptive gestures • speech recognition: speech-to-text engine, for speech-driven interaction with the robot • face detection and following: detect faces and follow them (e.g. with head movement) for a focused interaction • facial recognition: recognize faces from an internal face database for allowing a personalized interaction with the user • facial emotion recognition: estimate users emotion, e.g. for adjusting content or giving replies of afrmation • logging: automatic logging of all messages passed between tools and the application, recorded images and manually added log messages from the application, for example for research purposes • language tools: grammar and spell checking • chat bot: supports integrating large language models (LLM), e.g.remote LLMs via JSON APIs or local ofine LLMs

CHARACTERISTICS
This section provides a deeper look inside the Sobotify framework.
Figure 1 shows the framework structure, which is separated into four layers: the robot abstraction layer, which provides a robot agnostic interface to the tool layer, which in turn provides numerous tools, as described before.The Sobotify API itself gives a unifed interface between the tools and the applications layer.The top most layer provides easy-to-use graphical user interfaces (GUIs) to start and customize applications by means of a project confguration fle (.xlsx), which can be edited with a spreadsheet program and also for extracting gesture and speech from recorded videos.

Details of Algorithms
Sobotify applies many open source Python packages and external programs, especially for providing the tool functionalities.In the following the most important ones are briefy described.Vosk[3] is an ofine speech recognition tool, which is used here for live speech detection and for speech-to-text when extracting speech from videos (currently for live speech detection a computer or external microphone is used).Deepface [19][20] is a collection of face-related processing tools, including OpenCV, DeepFace, Open-Face and others.In Sobotify deepface is used for face detection, face recognition and facial expression recognition.MediaPipe [8] Pose is a machine learning based tool for detecting landmarks of a human body, which is used here for extracting gestures to control a robot's movement.FFmpeg [5] is a tool collection for video and audio processing.Sobotify uses it for extracting timestamps and audio from video fles as part of the speech and gesture extraction.Mosquitto [17] is a message broker for MQTT, a lightweight message protocol, which is applied here for intercommunication between the tools and the applications.LanguageTool [7] is a tool for grammar and spell checking, which is available as an ofine server, which can be applied here e.g. in language learning applications.Furthermore, some algorithms are directly implemented in Sobotify.
The tools for teleoperation, gesture extraction and the stickman visualization are part of the Python code, and are based on [6], [13] and [12].Also the robot controller, which interacts with the robots abstraction layer, implements many features within Sobotify, for example, synchronization of speech and gesture as well as realtime text replacement.Synchronization uses the timinig information, which is included in the speech text fles (in subtitle format ".srt") and in the gestures fles (time stamps e.g.extracted with fprobe).

Installation Instructions
Sobotify is Python based and most tools are currently running in Python 3.8, however some robots require diferent a Python version (NAO and Pepper require Python 2.7).Therefore the framework uses miniconda for creating the necessary environments.Additionally, some external programs and robots specifc SDKs are required.
The following describes the necessary steps for installation.Sobotify is available on github [10] for download.After copying and unzipping to a Windows 10 based computer, the script install.batcan be started for automatic installation, which includes the following steps : • download the required programs and data: miniconda, FFmpeg, Mosquitto, LanguageTool, Vosk language models and robot specifc items (NAOqi SDK and pycozmo resources) • create a storage repository (".sobotify" folder in the user's profle), for storing gesture and speech data, projects, log fles and downloaded items, see section 4.2 for more details.• initiate installation of miniconda and Mosquitto and extract downloaded fles to Sobotify's storage repository • create two miniconda-based Python environments (3.8 and 2.7) and install the required Python packages, including the local sobotify folder as a package (pip install -e ...) For most of the robots the required SDKs and interfaces are installed during the installation process.But for using the MyKeepon robot, hardware changes are necessary following the instructions cited in [15] and it requires an Arduino frmware [14].

Usage Instructions
Currently Sobotify has two separate GUIs, one for starting applications (Figure 3) and the other one for importing gestures and speech from video fles and for testing them (Figure 4).The GUIs can be started with the corresponding batch fles start_app.batand start_gesture.bat.

3.3.1
The Sobotify App GUI.In Sobotify applications are usually adaptable to the specifc needs of the user, such customized applications are called projects.For example, the quiz application can be customized to diferent languages, an English and a German version of the quiz.These versions are stored as projects in diferent spreadsheet fles (quiz_english.xlsx and quiz_german.xlsx).The English quiz, shown in Figure 2 has multiple sections: The frst section defnes basic information (name and version of the app to be used and the language), the second section defnes general text used in the quiz (e.g.welcome and farewell phrases and the reply on correct and incorrect answers).The following sections defne the content of the quiz, one section for each question and the corresponding answers.If multiple answers are possible (OR conjunction), they can be specifed in additional columns (alternative 1, 2 . . .).If multiple answers are required (AND conjunction), they can be defned in additional rows.

Figure 3: Sobotify GUI for starting applications
The Sobotify App GUI can be used to run a project with a robot.Any project stored in the Sobotify project folder can be selected from the drop-down list.The robot to be used can also be chosen from a drop-down list, see Figure 3 .As applications are (usually) written in a robot agnostic manner, they should be usable with any supported robot.Additional robot specifc settings (robot's IP address for NAO and Pepper or the COM port for MyKeepon), can be specifed in a text feld.Additionally the sound device to be used for speech detection can be chosen, and eventually the project can be started (and stopped) with the corresponding buttons.Projects can be newly created (based on existing projects/templates) and deleted in a separate window, which opens with "new/delete".

3.3.2
The Sobotify Gesture GUI.The Sobotify gesture GUI can be used to extract gestures and speech from a previously recorded video of a human.The extracted data is stored in a local database, and can also be tested with the robot.The apps and also many of the tools support a direct command line interface, e.g. for testing or integrating them in other scripts for automation.The quiz app can be called for example like this from the miniconda prompt: conda activate sobotify python quiz.py--robot_name stickman --language english --project_file quiz_english.xlsxFor testing facial recognition, the facial processing can be called with the following command: conda activate sobotify python facial_processing.py--frame_rate 5 For using facial recognition, facial images need to be placed in the .sobotify/face_dbfolder.

CODE/SOFTWARE
Sobotify has two storage locations on the local disk: the code repository ("sobotify" folder), containing the Python package, and a storage repository (".sobotify" folder), used for storing user content and downloaded programs and data.

Code Repository
The code repository contains the entire Python code and data of the framework downloaded from github.It is structured as follows: • ./(root folder): contains the README, installation scripts and fles and the scripts for starting the GUIs.• ./sobotify:sobotify API in sobotify.pyand GUIs • ./sobotify/apps:with subfolders for each application, which contain the apps' code and example projects • ./sobotify/tools:contains the tools; for each tool one Python fle or a folder, if a tool consists of multiple fles • ./sobotify/robots:contains for each robot one folder, which defnes the robot specifc interfaces and data • ./sobotify/common:common and external functionalities

Storage Repository
The .sobotify folder is created during the installation process and is composed of the following subfolders:

OUTLOOK AND FUTURE WORK
Sobotify was designed for non-technical users and researchers in HRI.To do that, teachers' needs were collected through interviews and workshops.Sobotify was formatively evaluated by the teachers during the development -the GUI and spreadsheet through workshops.Additionally, two separate research projects utilized Sobotify to use diferent robots as social robots.In the frst project, Pepper was designed as a sales assistant to engage students in a conversation [9] and in the second project, Pepper, NAO, Cozmo and MyKeepon were respectively designed as language learning companions to engage learners to converse in foreign language.Furthermore, Sobotify is under active development with more than 150 commits in the last year and will continue to be extended, updated and refactored.The future work entails, 1) adding more robots such as Cue and Bioloid.2) add Linux and MacOS support and a server-based version with a web interface.3) add hardware extension for robots such as camera, speaker and microphone and 4) add support for additional sensors and actuators such as LEDs, buttons and touch areas.We also plan to integrate more chat clients (e.g. based on GPT4All) and to add new applications in the feld of Socially Assistive Robot (SAR) .

ACKNOWLEDGMENTS
This research was partly funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy -EXC 2002/1 "Science of Intelligence" -project number 390523135.We would like to express our special thanks to teachers at the vocational schools in Cologne and Düsseldorf for their invaluable inputs and active participation in creating Sobotify.

Figure 2 :
Figure 2: Example of a Sobotify project fle (spreadsheet format) for defning the content

Figure 4 :
Figure 4: Sobotify GUI for importing (adding) gestures and testing them with a robot Adding New Robots.For adding a new robot a copy of an existing robot folder and Python fle can be renamed to the new robot's name and the required functions need to be adjusted, e.g.• say(self, text): where text is a string to be spoken by the robot, • move(self,line): where line is a list of movement defnition, e.g.joint angles, to be executed by the specifc robot, • get_image(self): return a camera image from the robot.Additionally, gesture extraction needs to be added for generating movement defnitions, e.g. by converting humans landmarks into robot movement commands.And the robot needs to be added in GUIs and the /sobotify/robots/robots.py.4.3.3AddingNew Tools.New tools can be added as Python fles in the tools folder and the corresponding calls in sobotify.pyfor starting the tool and accessing it with API calls.The chatbot.py can be used as example.It shows how the communication interface for a tool is established: for input data the tool subscribes to specifc MQTT messages and for output data MQTT messages are published.