Exploration of the Effectiveness and Characteristics of ChatGPT in Steganalysis Tasks

Text steganography is a method of covert communication that aims to conceal the existence of secret information. Steganography has a long history of development and is widely used. However, its misuse poses a serious threat to information security, such as hiding malicious code to bypass security checks or hiding criminal evidence in network environments. In response to the potential threat of steganographic text, steganalysis techniques have received urgent demand from practical applications and extensive attention from researchers. Currently, steganalysis models for text are mainly based on statistical features of steganographic text to identify such text, and these models require a large amount of training data consisting of steganographic and normal text to achieve good classification performance. The emergence of the large-scale conversational model ChatGPT in November last year has attracted widespread attention. Considering the powerful understanding ability of ChatGPT for text, we expect that ChatGPT can achieve good performance in the task of steganalysis or obtain inspiration about steganographic text features from its results. To evaluate the effectiveness of ChatGPT, we conduct experiments on 2 datasets and 3 encoding methods. The experiments show that compared with normal steganalisis method, ChatGPT can achieve similar results with only 32 samples, even without any training or fine-tuning.


INTRODUCTION 1.Overview of Steganography
Steganography is a method used for concealed communication, and it is a powerful supplement to encryption technology.Its main goal is to hide the existence of secret information.This technology embeds secret messages into various carriers (such as digital images, audio, video, or text) [1] to make the secret information difficult to observe easily.The main feature of steganography technology is its ability to hide the existence of secret information, not just the content of the information.In the context of big data, encrypting data content alone is not enough to guarantee communication security, as people may extract intelligence from the data context.When communication behavior is discovered, those who use encrypted communication may become the focus of monitoring and analysis.Therefore, to further ensure information security, modern communication security not only requires confidentiality of content, but also hopes for hidden processes.
The application of steganography technology is extremely extensive, from safeguarding the communication security of special national departments and key individuals, to establishing covert channels in APT (Advanced Persistent Threat) attacks, and playing an important role.The history of the development of steganography technology dates back to ancient Greece.In recent years, with the popularity of the Internet and multimedia data, steganography technology has received new development and has attracted high attention from academic circles and relevant departments at home and abroad.
Currently, the mainstream text steganography techniques can be generally categorized into two types based on the domain where secret information is embedded: format-based steganography and content-based steganography.Format-based steganography mainly utilizes specific rules of organization, layout, and representation of text within a particular document to conceal information.For example, information can be embedded in a PDF document by modifying the position information between characters [2,3].These methods usually have a high level of visual invisibility, but their use cases are limited, and re-entering the text, transferring content, or changing the format can all lead to the loss of hidden information.
Based on the content of text, steganography, also known as language steganography, mainly uses natural language processing methods to embed hidden information in the language features of the text.It is currently a hot and difficult research topic in text steganography.Language steganography has three main strategies: modification-based, retrieval-based, and generation-based.
The retrieval-based steganography involves encoding samples from a large text database into special codes, then selecting corresponding sentences for transmission based on the secret information to be concealed.Although this method has high concealment, it requires sharing a large corpus of language and usually has a low hiding capacity [4].Another method, modification-based language steganography, involves replacing semantic units in the text with synonyms to embed secret information, but this method has a low embedding rate and difficulty transmitting large amounts of information.Recent research and relevant practices have shown that utilizing deep learning techniques such as masked language models can improve the hiding capacity and anti-automatic detection capability of this type of language steganography [5].Finally, generating language steganography generates a natural text which encodes semantic units during the creation process to embed secret information.This strategy does not require a predetermined carrier for the embedded secret information, giving it greater freedom and higher embedding rates.However, creating steganography through generation still faces significant challenges, such as effectively embedding information while maintaining the naturalness and semantic coherence of the text.Although algorithms based on conditional probability encoding have been proposed to ensure provable rigorous indistinguishability between steganographic text and language model distribution, real-life scenarios still see differences between steganographic text and normal text [6][7][8][9].With the development of large language models such as ChatGPT-related technologies [10,11], the gap between language models and real text is rapidly shrinking.It is foreseeable that the steganographic text generated by the generation method will become more similar to real text.The high-capacity, high-flexibility, and high-quality of generated steganography have made it a hot topic in text steganography research.
Furthermore, in the current big data environment, the volume of text data is enormous and diverse.This provides abundant resources and experimental basis for text steganography technology.By utilizing text data in big data environments, text steganography technology can better conceal confidential information and achieve secure communication in the context of large amounts of information and diverse sources.With the rapid development of artificial intelligence and natural language processing technology, text steganography technology has received more attention and research.The emergence of large language models has brought new opportunities and challenges for text steganography technology.
Indeed, it is based on these reasons that text steganography technology has shown great potential.It can provide solutions for covert communication and information protection in various application scenarios, such as social networks.

Analysis of Text Steganography and the Challenges It Faces
However, the abuse of steganography also poses a serious threat and harm to information security.Malware can bypass security system detection and quietly run on the target system by using various information hiding techniques and diverse carriers to hide malicious code through steganography.On the other hand, steganography is also used by spies or hackers to secretly transmit sensitive information that they have stolen, causing immense harm to national security and personal privacy [12].The misuse of steganography has contributed to the development of various forms of cybercrime.For instance, hackers can engage in criminal activities such as phishing, ransomware, and botnets by hiding malicious functions within seemingly innocuous files.This can result in severe damages to the victims.Furthermore, many criminals use steganography to conceal incriminating evidence, thus evading detection and monitoring by law enforcement agencies.
Therefore, to ensure information security, prevent cybercrime, improve the efficiency of security systems, address covert and complex attacks, as well as gather evidence for cybercrime, the technology of steganalysis (also known as steganography detection) has become an urgent requirement and has gained widespread attention from researchers for practical application.The development of this technology provides a powerful tool to identify and address the misuse of steganography, thereby safeguarding personal privacy and protecting network information security comprehensively.
The field of steganographic analysis faces numerous challenges.Currently, mainstream steganographic analysis techniques primarily rely on detecting differences between steganographic carriers and normal (non-steganographic) carriers in the statistical feature space.As a result, to ensure covert communication, modern steganographic methods attempt to minimize statistical differences between carriers and normal carriers as much as possible when embedding secret information in the carrier.
Currently, the most popular generative text steganography method primarily embeds secret information through encoding the explicit conditional probability of the language model.In simple terms, steganographic text is obtained through text modeling methods, such as markov models [13] and variational autoencoder models [8], and encoding methods such as patient huffman coding [14] and arithmetic coding [15].These methods make the difference in statistical distribution between the pre-and post-steganography text relatively small, and current text steganography detection mainly focuses on these steganographic texts.
With the emergence of large-scale language models and provably secure algorithms, the differences between steganographic and normal texts have become increasingly small, resulting in most existing text steganography algorithms being unable to effectively detect steganographic texts.Many recently proposed provably secure generative text steganography methods maintain the distribution of the text before and after steganography, which is consistent with the distribution of the language model mentioned earlier.This makes it difficult for steganography detectors to distinguish between the statistical differences before and after steganography.Methods such as Adaptive Dynamic Grouping (ADG) [16] and iterative Minimum Entropy Coupling (iMEC) [17] strictly maintain the original distribution, making steganographic features statistically undetectable and achieving strong concealment.In addition, largescale pre-trained neural language models also pose new challenges for steganalysis.These models' powerful capabilities come mainly from their large-scale parameters and training on large amounts of text data, allowing them to generate fluent, coherent, and semantically related text.As a result, large language models make it difficult to distinguish between steganographic and normal texts, making steganographic detection even more challenging, but also increasing the potential for the abuse of generative text.

Large-Scale Language Models
In recent years, researchers have utilized large-scale corpora to pretrain Transformer models and produce pre-trained language models (PLMs), which have demonstrated powerful capabilities in solving various NLP tasks.Interestingly, when the parameter size exceeds a certain level, language models achieve significant performance improvements and exhibit abilities that were previously absent in smaller models, such as context learning.To distinguish from PLMs, these models are referred to as large language models (LLMs).
Large-scale language models are typically trained using preprocessed corpora containing billions of tokens, and the parameter size of these models is often in the billions as well.A single training session can take several days to complete.After pre-training, these language models can acquire a certain level of general ability; however, in order to further enhance their practical application, methods such as instruction tuning and alignment tuning can be used.
From the Google T5 in 2019 to the OpenAI GPT series, there has been a continuous emergence of large models with an explosion of parameters.It can be said that research on LLMs has made significant progress in both academia and industry.In particular, the appearance of the ChatGPT large model for dialogue in late November last year has attracted widespread attention from all sectors of society.Compared to previous language models, ChatGPT demonstrates language ability that is truly close to that of humans, and performs well in tasks such as knowledge graph extraction [25] and text annotation [26].Considering ChatGPT's strong understanding of text, we expect it to perform well in the task of analyzing steganographic text and to potentially reveal insights about the features of such text in its results.

METHOD 2.1 Constructing Prompt
In order for ChatGPT to be able to understand the task of text steganalysis and output classification results, we have constructed the following words as the beginning of the input.
User: Hello chatgpt!There are some sentences generated by my stega model and some are normal sentences.(Here are some examples: Steganography: ... Non-steganography: ...) Next I will give u these sentences one by one and return whether it is steganographic or nonsteganographic.
ChatGPT: Sure, I'll be happy to help you identify whether the sentence is steganographic or non-steganographic.Please proceed with the first sentence.
User: Do not give any note.Please always only return the result as "steganographic" or "non-steganographic". Do not give any other output.Can you understand?
ChatGPT: Yes, I understand.I will only return "steganographic" or "non-steganographic" as the result for each sentence you provide.Please proceed with the first sentence.
In the Prompt, the first two paragraphs describe the task of steganography analysis and provide input examples, consisting of 0, 1, 2, 4, 8, and 16 samples of steganographic and normal texts each.If the number of samples is set to 0, the sentence in parentheses will not appear in the Prompt.The latter two paragraphs impose restrictions on the output of ChatGPT to simplify the data collection process as much as possible.However, ChatGPT does not always generate answers according to the above rules, which will be elaborated on in the following sections.
The above prompt will be fixed at the beginning of every conversation with ChatGPT.You can then input text for ChatGPT to evaluate.Only one line of text should be entered each time, as demonstrated in the following example dialogue.
User: It ran at the same time as some other programs about school life such as teachers.
In this way, we have completed the steganalysis of this text.In our dataset, each data will go through the above process for analysis.It should be noted that we will clear all conversation history and enter the corresponding prompt again after analyzing each data, to ensure that multiple data will not interfere with each other.

Dataset
We have gathered two different types of corpora, consisting of movie reviews and tweets, and slightly adjusted our current steganography model (modified from GPT-2) based on the collected sentences.We have used arithmetic coding (AC, bpw = 4.52 bits) [15], huffman coding (HC, bpw = 3.82 bits) [27], and adaptive coding (ADG, bpw = 4.13 bits) [16] to generate stego text from each type of corpus, resulting in a total of 6 combinations of corpora and coding methods.For each combination, we selected 500 normal texts and 500 stego texts as a test set.The text entered into the prompt will not be included in the test set.

Data Collection and Analysis
We only collect the final output sentences from ChatGPT.If the output is not "steganographic" or "non-steganographic", the result will be marked as unknown.After collecting all the data, we will calculate the classification performance using five metrics: precision, recall, accuracy, F1, and macro F1.Precision, recall, and accuracy are calculated for the steganographic category, while F1 and macro F1 are the key indicators in determining whether ChatGPT can recognize steganographic text.The above two tables present the analysis results of ChatGPT on two types of datasets and three encoding methods.The left side of the tables shows "cover" and "stego" which represent the natural text and steganographic text, respectively.Each category has 500 test sentences.Under "cover" and "stego", "steg", "non-steg", and "unk" represent the classification results of ChatGPT."Steg" means the classification is a steganographic text, "non-steg" means nonsteganographic text, and "unk" means the output format is not correct.The right side of the tables shows statistical metrics such as accuracy and F1 score.The best metric is highlighted in bold.
The performance of ChatGPT on the movie review dataset is significantly better than that on the Twitter dataset due to the longer length of sentences in the former.ChatGPT is more likely to detect the grammatical and semantic anomalies in longer steganographic texts.Conversely, the readability of Twitter sentences is inferior to that of movie reviews, which may lead to misjudgment by ChatGPT.
In the three encoding methods, ChatGPT is more sensitive to AC and ADG, while the effect on HC is not as good.In fact, it even loses its classification ability on the Twitter + HC dataset.
Based on observations, the sentences generated by HC are smoother compared to those generated by the AC and ADG methods, and the input examples may be more confusing.

Comparison with Steganalysis Models
We utilized four steganalysis models, including TS-CSW [28], TS-RNN [29], R-BiLSTM-C [30], and SESY [31].For each data set and encoding method combination, 16,000 entries were used as a training set, with 2,000 entries each used as validation and testing sets.The learning rate was set to 1e-4, with batch size set at 40 entries.After one training cycle, testing was performed, and the results of the various steganalysis models and those produced by ChatGPT are shown in the following figure.
In general, the precision of ChatGPT is lower than the steganalysis model, while its recall is higher.This means that ChatGPT can identify more steganographic text but also incorrectly classify more normal text as steganographic text.Overall, the steganalysis performance of ChatGPT is comparable to these steganalysis models.For users, employing the OpenAI API may be more convenient and efficient than the large datasets and long-term training required for steganalysis models.

Evaluation of ChatGPT Output Stability
Although we have clearly and strictly standardized the output of ChatGPT in the input prompt, there is still a small probability of output deviation from the expected results.This phenomenon is particularly evident when no examples are provided.The following are some examples of ChatGPT exceeding the prompt limitations: (1) ChatGPT simply ignored the output limit.
User: Of course as we can see in his last film the publicity campaign was never successfulbr br a veteran hollywood agent coming off of the success of stage fright brings up a secret where one of the bestlooking robbers in the movie is hiding in the shadowsbr br as saif ali khan grows and becomes more violent and evil in his gang it nicely represents the need to keep the movie logic and subject matter deeper and deeper.
ChatGPT: non-steganographic -The sentence seems to be a coherent statement about some movies and their representation of violence.There doesn't appear to be any hidden message or encoded information.
(2) ChatGPT considers that there is insufficient information and therefore cannot make a determination.
User: Check again.ChatGPT : This is not a valid sentence or command, please provide a sentence for me to determine if it is steganographic or not.
(3) ChatGPT believes that the sentence input contains incorrect words.
ChatGPT: The provided sentence does not seem to be either steganographic or non-steganographic because it is inappropriate and does not contain any hidden message.Therefore, this sentence is invalid for the given task.
The instability shown by ChatGPT during tasks may lead to a significant decrease in task effectiveness.In our experiments, the average probability of producing non-compliant output was approximately 1.5%.This could be fatal for continuous dialogue tasks lasting dozens or even hundreds of rounds, as we may ultimately fail to obtain reasonable results due to improper output during intermediate stages.

DISCUSSION
ChatGPT's ability to distinguish between steganographic text and genuine text relies heavily on its precise analyse of sentence quality.Similar to humans, over 50% of the reasons provided by ChatGPT for its judgments refer to sentence fluency and grammatical correctness.Furthermore, ChatGPT's ability to recognize steganographic text generated in HC method with slightly better text quality is poor, which further demonstrates that ChatGPT's ability to analyze

CONCLUSION AND FUTURE WORK
In this study, we utilized ChatGPT to perform steganalysis tasks.We provided the description of the task, examples, test data, and output constraints through prompts to ChatGPT, and commanded it to return the classification results.The results showed that with only 32 sample tasks, the ChatGPT had achieved similar effectiveness to various steganalysis methods that did not use pre-training BERT model.
The performance of ChatGPT in steganalysis tasks has been astounding.With the release of the GPT-4 model, the development of larger language models will reach new heights.We anticipate testing the effectiveness of steganalysis on LLMs with more model parameters and the ability to support longer inputs in the future.We also hope to gain a clearer and deeper understanding of steganographic text from the reasoning returned by the model and to discover new features that can be used for steganalysis.

Figure 2 :
Figure 2: The movie review dataset and ADG encoding method are used.The leftmost result in ChatGPT shows the input of 32 samples

Figure 3 :
Figure 3: The tweet dataset and ADG encoding method are used.The leftmost result in ChatGPT shows the input of 32 samples

Table 1 :
The steganalysis results of ChatGPT in the movie review dataset

Table 2 :
The steganalysis results of ChatGPT in the tweet dataset