Topic modelling analysis of depression text message therapy: A preliminary study First Author\[1\]\*, Second Author2 (Double Blind Review: Please do not type or edit anything here until final-camera ready submissions) *1First affiliation, City and Country (Please do not type or edit anything here, until final camera-ready paper submission)* *2Second affiliation, City and Country (Please do not type or edit anything here, until final camera-ready paper submission)*
ARTICLE INFO ABSTRACT

Article history:

Received

Revised

Accepted

Online first

Published 1 March 2024

 The coronavirus disease 2019 (COVID-19) that has plagued the world since 2019 has initiated several issues and challenges in the mental health services field. Among the issues are relocation of mental health staff to non-mental health related COVID-19 tasks, some people face movement restrictions to attend face-to-face therapy and fear of public prejudice when seek help. As an alternative, World Health Organisation (WHO) recommended implementing remote mental health services such as telehealth to reach out to patients. Nonetheless, the topics included in depression mobile therapy are scarce, particularly from the text messaging perspective. Topic modelling (TM) extracts topics from a set of documents without manually reading individual documents. It is very useful in searching for topics contained in short texts. This study aims to determine the topics in the text messages sent by mental health practitioners for depression therapy. In this study, three topic modelling techniques, i.e., Biterm Topic Model (BTM), Word Network Topic Model (WNTM), and Latent Feature Dirichlet Multinomial Mixture (LFDMM), were evaluated on 258 text messages of depression therapy. The performance of the TM techniques was evaluated using classification accuracy, clustering, and coherence scores. The findings indicate that the set of text messages comprises five topics. BTM performed better than the other techniques in classification accuracy and clustering in some cases based on the performance measures. Consequently, not much significant difference was found in the coherence score between the three topic modelling.

Keywords:

keyword1

keyword2

keyword3

keyword4

keyword5

keyword6

DOI:

10.24191/jcrinn.v9i1
# Introduction The most common mental disorder is depression. Depression or major depressive disorder is a common but serious illness that usually affects mood and behaviour. Depression contributes to 4.3% of the global disease burden and is one of the most common causes of disability, predominantly among women (World Health Organization, 2021). Although this condition is serious, it can be treated. The development of tools and strategies involving digital technology and mobile self-care has the potential to be part of the support care system services. With the increase in mental health practitioners’ awareness and initiative to reach patients remotely, several mental health mobile therapy tools were developed. Short Message Service (SMS), for example, has been widely used to support change in attitudes of individuals suffering from depression (Agyapong et al., 2021; Almeida et al., 2018; Shariful Islam et al., 2019). However, the study on the topics in the text messages meant to deliver depression therapy is scarce (Kerrigan et al., 2019). Machine learning is a branch of computer science and artificial intelligent which focuses on the use of data and algorithm in predicting outcomes by first learning the way that things work. For example, neural language models and deep neural networks from machine learning was used to develop sarcasm identification framework on social media data (Aytug Onan & Tocoglu, 2021). Another work reported different types of deep neural network model provide promising results on text classification task involving sentiment analysis (Aytuğ Onan, 2022). In machine learning and natural language processing (NLP), topic modelling is a generative model that provides a probability framework for the frequency of occurrence of terms in documents from a corpus. Topic modelling is also guided only by the frequency of terms where word order information is not considered. This is also referred to as the word exchange assumption in a document, and this assumption leads to a bag-of-words model (Grün & Hornik, 2011). Topic modelling methods can be in untrained or semi-trained forms, structured or unstructured data, and have the potential to be applied in various fields such as health, plantation, education, banking, social network opinion analysis, and data or transportation networks. It has been widely used to find latent abstract topics contained in a collection of texts such as documents, short texts, post text on Twitter or Facebook, and user comments on news pages, blogs, and emails (Albalawi et al., 2020). A topic modelling method known as Latent Dirichlet Allocation (LDA) and a package of topic models in R were used in the process of identifying themes related to self-injurious thoughts and behaviour (SITB) in a TeenHelp.org user discussion (Franz et al., 2019). Meanwhile, LDA was used to analyse latent topics and the frequency of depression-related words extracted from post texts on Reddit sites (Tadesse et al., 2019). Due to sparsity problems in the short text, long text topic modelling such as LDA is less efficient in identifying latent topics (Yan et al., 2013). In the effort to overcome the sparsity problem, renewed LDA techniques have been developed by several researchers (Wu et al., 2020; Yan et al., 2013; Yang et al., 2020) to obtain more accurate topic modelling based on the characteristics of simple and short text messages. Each technique developed has different results on different text samples. Self-aggregation based topic model (SATM) was introduced in which short texts are aggregated for data sparsity (Quan et al., 2015). In particular, short texts are the results of randomly shredding lengthy documents produced by a common theme model. However, as the amount of data grows, the number of parameters also grows, increasing the computing cost and risk of overfitting. As a result, Pseudo-Document-Based Topic Modelling (PTM) based on pseudo-documents for short texts was introduced (Zuo et al., 2016). Similar to SATM, PTM automatically aggregates short texts, but it only allows each pseudo document to cover one topic, which cuts down on the processing time. To encourage semantically related terms under related topics, the Generalized Polya urn (GPU) expand the Dirichlet Multinominal Model (DMM) by leveraging word embeddings to produce more coherent and pertinent results (Li et al., 2016). Next, Generalized Polya urn Poisson-based Dirichlet Multinomial Mixture Model (GPUPDMM) was developed and was found to performs well but has a high time complexity due to the Gibbs sampling method used in the method (Tian & Fang, 2019). Previous study conducted experiments on texts from six different sources using nine short text topic modelling techniques (Qiang et al., 2019). They discovered that each short text topic modelling produced different results for each dataset tested. Through reading and to the best of the researcher’s knowledge, the studies focusing on topic identification in depression therapy text messages using short text topic modelling methods are limited. Much short text topic modelling software can be used to identify topics in a document. However, the suitable topic modelling method for the depression therapy text message has not been determined. Thus, this study aims to determine the topics in the text messages sent by mental health practitioners for depression therapy and compare the performance of Biterm Topic Model (BTM), Word Network Topic Model (WNTM) and Latent Feature Dirichlet Multinomial Mixture (LFDMM) on depression text messages. BTM and WNTM are global word co-occurrences based methods whereas LFDMM is Dirichlet Multinomial Mixture based Methods as explained in the next section. # Topic Modelling Techniques ## Biterm Topic Model (BTM) The key concept of BTM (Yan et al., 2013) is to learn topics contained in short texts based on the combination of two words in the whole corpus to solve the issue of sparseness in a single document. BTM assumes that the whole corpus is a mixture of topics, where the two words, known as the biterm, are independently generated from a specific topic. The BTM model topics use two words that co-occur together rather than a single word. The co-occurrence of two words can enhance the topic modelling than using only one word besides improving the topic learning. Moreover, BTM forms biterm from the whole corpus used together to learn topics rather than from a document. The approach helps to improve and uncover the latent topics by fully utilising the rich global word co-occurrence. The main advantage of BTM is that it uses the whole corpus as input, allowing the topic model to solve the issue of document shortness and better reveal the topic-word relations (Pietsch & Lessmann, 2018). The graphical model of BTM is shown in Fig. 1, and the generative process is described below (Yan et al., 2013). Fig. 1. Graphical Model of BTM For each topic *z*, draw a topic-specific word distribution *ϕz ∼* Dir(***θ***) 1. > Draw a topic distribution ***θ** ∼* Dir(***α***) 2. > For each biterm *b* in the biterm set *B* 1. > draw a topic assignment *z ∼* Multi(***θ***) 2. > draw two words: *wi, wj ∼* Multi(*ϕz*) Different from BTM, WNTM models the distribution over topics for each word instead of learning topics for each document. The following explains WNTM generation process. ## Word Network Topic Model (WNTM) WNTM was introduced by Zuo et al. (2016) where the framework implements the same Gibbs sampling with LDA to reveal latent word groups in a word co-occurrence network. In the co-occurrence network, each word is known as a node. The connection between nodes is referred to as an edge. The connections between nodes indicate that these words have co-occurred at least once in the local context. In the original paper, a sliding window value of 10 is fixed to limit the size of the word network and to reserve only the local context for each word. The sliding window is used to scan words through the document; any two different words appearing in the same window would be treated as co-occurred. WNTM creates a new pseudo-document representing the word co-occurrence network to run the Gibbs sampling. A pseudo-document which contains all the words connected to it in the word network is created for every word. The difference between Gibbs sampling in LDA and WNTM is that in the former, LDA uses the original text document as an input, whereas WNTM uses the newly generated pseudo-document as input to topic modelling (Zuo et al., 2016). The graphical model of WNTM is shown in Fig. 2. Fig. 2. Graphical Model of WNTM > 1\. For each latent word group z, draw ϕz ∼ Dir(β), a multinomial distribution over words for z. > > 2\. Draw a latent word group distribution for the adjacent word-list Li of the word wi, Φi ∼ Dir(α). > > 3\. For each word wj ∈ Li: > > a) select a latent word group zj ∼ Φi > > b) select the adjacent word wj ∼ ϕzj In summary, BTM and WNTM learn the latent topics from the global word co-occurrences obtained from the documents, relying on the idea of the closer the two words, the more relevance the they are. Next, the following explains LFDMM generation process. ## Latent Feature Dirichlet Multinomial Mixture (LFDMM) LFDMM combines the latent feature and Dirichlet Multinomial Mixture (DMM) models. This model assumes that there is only one topic in each document. In LFDMM, pre-trained word vectors trained on large corpora were used to incorporate information from large datasets. Therefore, LFDMM surpasses the word sparseness in a short text. Generally, LFDMM has the structure of the original Dirichlet multinomial topic models, with additional two matrices, where τt is the latent feature vector associated with the topic (t) and ωw is the latent feature vector associated with the word (w). To generate a document in a document using LFDMM, a distribution over the topic is sampled for the document collection. Next, LFDMM draws a topic indicator for the whole document. A binary indicator variable is sampled from a Bernoulli distribution, Ber(λ), for each word in the document, which will be used to determine either the Dirichlet multinomial or latent feature component to generate the word. Lastly, the selected component will generate the word from the same topic. The hyper-parameter, λ, is the probability of a word being generated by the latent feature model. The graphical model of LFDMM is shown in Figure 3 and generative process of LFDMM is as follows (Nguyen et al., 2015). **Fig. 3:** Graphical Model of LFDMM > 1\. Topic document distributions is drawn, θ ∼ Dir(α) > > 2\. Topic word distribution is drawn, ϕz ∼ Dir(β) > > 3\. Topic indicator is drawn for the entire document d, zd ∼ Cat(θ) > > 4\. A binary indicator is sampled from a Bernoulli distribution, sdi ∼ Ber(λ) > > 5\. Word generated, wdi ∼ (1 − sdi)Cat(ϕzd) + sdi CatE(τzd ωT) # Methodology This study was conducted in Intelligent Data Analytics lab at XXXX, from October 2023 to November 2023. Several experiments were conducted on the sample of text messages used in depression therapy. These were done to examine the best performing topic models for the type of text message samples in this study. The experiments were conducted on a Windows 10 Pro with Intel(R) Core(TM) i5-6200U CPU and 8GB RAM. The data source and pre-processing tasks are presented in the following section. ## Data Set This study focuses on text messages used by professionals in treating depression, where the treatment targets different types of primary and secondary outcomes. The dataset was developed by collecting text messages for depression therapy from previous studies. The text messages were collected from published studies between 2009 and 2020 (Aguilera et al., 2017; Agyapong et al., 2018; Almeida et al., 2018; Anstiss & Davies, 2015; Barrera et al., 2020; Hartnett et al., 2017; Kraft et al., 2017; Ranney et al., 2017; Shariful Islam et al., 2019). Text messages can be found either in the main content section of the article or the appendix section. Consequently, 258 text messages made up the dataset for this preliminary study. The number of topics (value of K) needs to be determined before the text messages can be used as the dataset in the experiment. One way is using the manual topic distribution through a qualitative approach. In this process, each text message content was analysed by an expert in mental health counselling. The interpretation of the text messages depends on the expert’s judgement and knowledge in the mental health therapy context, psychology theories and practices. Each of the text messages were assigned to at least one label. Meanwhile, some text messages have more than one label and share the same label with other text messages (Fig. 4). This leads to the nine labels assigned to the text messages dataset, as shown in Fig. 5. Table 1 provides a short description of each label. Some pre-processing steps are performed, including eliminating stop words, removing numbers, converting all text to lowercase, and lemmatisation. The statistics of the dataset include 2656 words, 377 unique words, maximum length of a sentence is 14 words, 9.14 words per sentence on average, 1438 letters and 51.36 letters per sentence. \[CHART\] Fig. 4. Number of Label Assigned to Each Text Message \[CHART\] Fig. 5. Number of Text Messages Assigned to Each Label Table . Short description for each label | | | | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | Label | Short description | | Cognitive, mood or behaviour monitoring | Monitoring on thinking pattern, current mood or behaviour | | Medication | Anything related to medication such as adherence, appointment, reminder or side effect | | Practice healthy social or physical activities | Content talks about activities that help | | Future and life opportunity | Pay attention to things that can change for a better future by taking opportunities. Forget about yesterday and the past. | | Health information fact | Fact about depression, health, treatments, symptoms, drinking habit etc. | | Coping plan | Method to control or change mood and behaviour | | Managing thoughts | Method to shift negative thinking to a positive one | | Self-reward | Appreciate self-effort and hard work by using any form reward | | Seek Help | Provide help care contact or encourage patient to ask for support from family/friends/professionals. | ## Parameter Settings This study applied the open-source Short Text Topic Modelling (STTM) with a JAVA-based library (Qiang et al., 2019). The software contains all three short text topic modelling techniques: BTM, WNTM, and LFDMM. For the hyperparameter settings, two studies used smaller α values (α = 0.05) within the LDA experiments.(Cheng et al., 2014; Schneider & Vlachos, 2018) In contrast, other study used α = 0.1.(Wang et al., 2016) BTM also worked well when α = 50/K (Cheng et al., 2014; Qiang et al., 2019). Moreover, the smaller value of β = 0.01 was also used in some studies(Cheng et al., 2014; Nguyen et al., 2015; Qiang et al., 2019; Wang et al., 2016; Yan et al., 2013) when applied to the short text. Based on these values, the smaller value of hyperparameter settings is assumed to be suitable for short texts. The small value of α indicates that short text documents contain less topic, while the smaller β value is associated with topic word-sparseness (Pietsch & Lessmann, 2018). The same hyperparameter settings were used to compare the three models in this experiment, with α = 0.05 and β = 0.01. For BTM, each document is treated as one window. For WNTM, the window size is set to 10 words. BTM, WNTM and LFDMM run for 2000 iterations, which is generally sufficient for convergence. LFDMM iteration baseline model was run 1500 times and use the sample produced by the baseline model to initialize the LFDMM with further 500 iterations (Nguyen et al., 2015). For LFDMM, λ is set to 0.6, which has shown in the past study that after combines data from bigger external corpus with corpus-specific topic-word multinomials, better word-topic distributions were produced in this way (Nguyen et al., 2015). In addition, LFDMM techniques are based on the concept of word embedding. In this experiment, “glove.6B.300d.txt” pre-trained word embeddings were used. Subsequently, each model was run with 6 different number of topics (K), from 5 to 20, with a step size of 3. The range for K was selected based on the number of labels assigned by an expert from the previous stage. The minimum value of K is very close to nine, i.e., the original number of the labels. On the contrary, the maximum limit value of the topic for this experiment is 20, considering the significant differences observed between the considerable K values and the ability of the researcher to assess the topic manually. The step size of three was chosen because each document has at least one label and three labels at the most. This procedure is also used by Pietsch and Lessmann (Pietsch & Lessmann, 2018). ## Statistical Analysis According to Asmussen & Moller (2019) the outcome produced by the model can be used to assign labels to the topic group. Likewise, other studies also implemented text processing algorithms to measure classification accuracy, prediction score, coherence, and clustering (Howes et al., 2013; Nguyen et al., 2015; Pietsch & Lessmann, 2018; Qiang et al., 2019). In this study, both quantitative and qualitative perspectives are used. From the qualitative perspective, a manual analysis of the words extracted from the experiment for each topic was done to interpret their meaning and assign a topic label. The output of the models is the list of the most probable words in a topic. The top 20 most probable words in each topic group are presented to a group of 6 experts for interpretation and labelling. These experts have at least five years’ experience in depression counselling. They were invited through email, and a consent form was filled out as an agreement to partake in the manual analysis. Next, an online meeting was conducted with the experts to analyse the top 20 words in each topic and discuss the suitable topic to be assigned to each topic group. The discussion lasted for two hours. The topical document evaluations were conducted using document classification accuracy, clustering, and coherence for the quantitative measurement, as included in the STTM package (Qiang et al., 2019). Classification accuracy is used to measure document-topic distribution. Ideally, each document (text message in this study) is assigned to a topic with the highest probability using the document-topic distribution p( z | d ). The text classification accuracy is used to evaluate the performance of BTM, LFDMM, and WNTM. In the STTM package, the default setting of the linear kernel Support Vector Machine (SVM) classifier in LIBLINEAR (https://liblinear.bwaldvogel.de/) was used. Fivefold cross-validation was performed on the dataset to measure its classification accuracy. A higher accuracy indicates that the model could better discriminate the learned topic. It also means that the learned topics are more related to the real dataset topics. Clustering is another important evaluation that directly measures the effectiveness of a topic model without the influence of external methods (Yan et al., 2013). In document clustering, each topic is treated as a cluster. After the topic probability calculation was performed, each document (d) was assigned to the topic (z) with the highest probability p( z | d ). The two most common clustering metrics, Purity and Normalised Mutual Information (NMI), are used in this study for clustering evaluation. The good clustering value is close to 1, while bad clustering is close to 0. Next, coherence evaluation is conducted to measure the quality of topic-word distribution. A coherence score is based on the concept that words belonging to the same topic will co-occur in a document. It is noteworthy that the coherence score is only reliable for measuring frequently found words in a document. This is because the less frequent word is unreliable in calculating the topic-word distribution (Yan et al., 2013). In this experiment, one Wikipedia meta-document was used to compute the score of topic coherence. Point-wise Mutual Information (PMI) of each word pair was calculated using a 10-word size sliding window and considering the word co-occurrence from the entire dataset of Wikipedia articles. # Result Six different topics were run using hyperparameter combinations of α = 0.05 and β = 0.01 for each topic modelling. Each model was run 20 times, and the mean performance of classification, clustering, and topic coherence was reported. ## Classification Accuracy The classification accuracy of text messages for depression therapy using three topic modelling is shown in Table 2. LFDMM exhibits a higher accuracy score than the WNTM method. However, BTM outperforms the other two methods except with LFDMM when K = 5. Overall, the results show that the biterm word co-occurrence performs better than the word embedding and co-occurrence network method for the depression domain dataset. Although BTM and WNTM are word co-occurrence methods, the better performance of BTM could be due to the use of the whole corpus as an input, while WNTM generates document-topic distribution learned from pseudo-document. This is in line with other findings stating that BTM achieved better classification accuracy than WNTM (Qiang et al., 2019; Schneider & Vlachos, 2018). Thus, this shows that generating a new pseudo-document based on the word context does not necessarily mean it is sufficient to overcome the tendency of the topic inference under data sparseness (Schneider & Vlachos, 2018). In contrast, other studies report that WNTM performed better than BTM in classification accuracy (Pietsch & Lessmann, 2018; Yi et al., 2020). It indicates that the performance of topic modelling is dataset-dependent. Overall, BTM and LFDMM show an upward trend as the value of K increases, whereas WNTM shows an unstable up and down trend across the K values. It signifies that WNTM is less reliable in making document-topic distribution for depression datasets than the other two methods. Notably, the classification accuracy scores for both BTM and LFDMM decrease when K = 11 despite the comparatively small differences. Table . Mean classification accuracy of three topic modelling with alpha = 0.05, beta = 0.01 and different values of K | | | | | | ------------------------ | -------- | -------- | --------- | | **Number of topics (K)** | **BTM** | **WNTM** | **LFDMM** | | 5 | 10.71429 | 7.678575 | 13.21429 | | 8 | 18.39285 | 6.964289 | 13.75000 | | 11 | 18.03571 | 9.821428 | 13.57143 | | 14 | 19.28571 | 8.035713 | 14.28571 | | 17 | 21.42857 | 9.285723 | 18.21429 | | 20 | 20.71427 | 8.214291 | 17.32143 | ## Clustering Short text clustering is one of the most important aspects of short text topic modelling as it helps find targeted patterns from the dataset. The result of the clustering evaluation is reported in Table 3. The data report the average purity and NMI scores of each modelling on depression datasets. Purity and NMI scores range from 0 to 1, and the closer the value to 1.0, the better its clustering performance (Nguyen et al., 2015). The highest score of purity and NMI of each modelling comprising different K values are highlighted in bold. Overall, all models produced upward trends with an increasing number of topics. Although BTM achieves the highest accuracy score in most K values, its purity and NMI scores are the lowest among all methods. It signifies that BTM is the worst in labelling sample text messages into their cluster compared to the golden label among all methods. The differences in purity and NMI scores among the methods are comparatively small. Although WNTM performs best on clustering evaluation, it performs the worst in classification accuracy evaluation. Table . Mean clustering score of three model with alpha = 0.05, beta = 0.01 and different values of K. | **Number of topics (K)** | **BTM** | **WNTM** | **LFDMM** | | | | | ------------------------ | ------------ | ---------- | ------------ | ---------- | ------------ | ---------- | | | ***Purity*** | ***NMI*** | ***Purity*** | ***NMI*** | ***Purity*** | ***NMI*** | | 5 | 0.4142 | 0.4163 | 0.3928 | 0.4147 | 0.3839 | 0.4224 | | 8 | 0.4803 | 0.5208 | 0.4678 | 0.5016 | 0.4482 | 0.4988 | | 11 | 0.5410 | 0.5716 | 0.5410 | 0.5710 | 0.5375 | 0.5541 | | 14 | 0.6250 | 0.6206 | 0.6375 | 0.6335 | 0.5964 | 0.5891 | | 17 | 0.6946 | 0.6574 | 0.7428 | 0.6875 | 0.6857 | 0.6404 | | 20 | **0.7464** | **0.6788** | **0.7946** | **0.6977** | **0.7678** | **0.6812** | ## Topic Coherence Topic coherence evaluates the quality of topic-word association by measuring how well the word is assigned to a topic (Nguyen et al., 2015; Qiang et al., 2019). Based on Table 4, it can be observed that WNTM and LFDMM show higher coherence scores than BTM in most K values. However, the performance is inconsistent, where the scores increase when K = 17 and decrease again when K = 20. Overall, the three methods produce downward trends with an increasing number of topics. This could be assumed that not many good topics can be produced when K increases as the sample data used were all short text messages consisting of limited words. Another reason is that the increase in K will only lead to a meaningless topic and less coherence. Therefore, K = 5 will be considered the best number of topics for sample data on depression in this study. Table . Mean coherence score of three models with alpha = 0.05, beta = 0.01 and different values of K. | **Number of topics (K)** | **BTM** | **WNTM** | **LFDMM** | | ------------------------ | -------- | -------- | ---------- | | 5 | 0.592458 | 0.596730 | 0.600214 | | 8 | 0.588065 | 0.593505 | 0.598672 | | 11 | 0.587301 | 0.590957 | 0.580718 | | 14 | 0.583307 | 0.582768 | 0.579854 | | 17 | 0.577075 | 0.695082 | 0\. 598182 | | 20 | 0.571340 | 0.569066 | 0\. 594948 | A notable huge distance between BTM classification accuracy scores and the other two methods indicates that BTM is a suitable option for topic modelling for the depression dataset in this study. An acceptable clustering score and a consistent performance in coherence evaluation indicate that BTM is a promising topic modelling method for the depression dataset of this study. The coherence evaluation also shows decreasing scores for all methods with an increasing number of topics. It can be said that there is still one good topic found in those methods regardless of the number of topics. On the contrary, the decrease of coherence scores for all methods when K \>= 8 signifies that the higher the number of topics, the poorer the topic-word distribution. Besides, the value of K = 5 is also significant to be considered the most suitable number of topics because it is close to the original label, which is nine. ## Efficiency This section compares the efficiency of the three methods on depression text messages. Table 5 reports the mean initiation time and per iteration time in milliseconds for each method. From the results, BTM performs the best. As expected, LFDMM performs the slowest because it needs more time to optimise topic vectors and runs more iterations than the other two methods. WNTM is slower than BTM possibly due to the time needed to find word co-occurrence and the process of creating pseudo-document. According to Qiang et al. (2019) the word embedding method (LFDMM) has the slowest initiation time due to computational steps for the similarity between words. Therefore, it is not surprising that LFDMM performs the slowest in this study. Table . Topics in text messages for depression therapy produced by BTM. | **Topic Model** | **Initiation Time (millisecond)** | **Per Iteration Time (millisecond)** | | --------------- | --------------------------------- | ------------------------------------ | | BTM | 7.0 | 0.144 | | WNTM | 62.0 | 0.355 | | LFDMM | 156.0 | 35.988 | ## Topic Labelling – Qualitative This section reports the interpretability of the topics using a qualitative approach. A group of six experts from the depression counselling domain was involved. As discussed in the previous section, the topic modelling method used for topic interpretation is BTM with the K value of 5. In this process, the experts were given a list of five topics, where each topic has the top 20-word order by word probability. The first word has the highest probability, and the last word has the lowest probability of being generated by the topic. Table 6 presents the model’s five topics, including the top 20 words. Some words appear in every topic (as underlined), but only a few words are unique to one topic, such as “work,” “far,” and “reward” for Topic 1. The interpretation and topic labelling were conducted in a 2-hour online meeting, where all six experts discussed and shared their opinions on the words listed for each topic. The experts interpreted the topics by analysing the top 20 words. During the discussion, the experts identified common words from the list they had always used during depression text-based therapy. The experts also related coherent words to the psychotherapy element they have been practising. Two to three labels were suggested during the discussion, with the most relevant words related to it listed together to get a clearer justification. Some of the words could not fit into any topic label suggested by the experts. Thus, those words are considered less coherent to the topic. The experts then discuss each suggested label and share their opinions before selecting the most suitable label. Table 6 shows the five topics with their labelling. For example, topic 1 is about Life Style Remedies with words such as “do,” “friends,” “work,” “far,” “fun,” “reward,” “coffee,” “activity,” “movie,” “day,” “place,” and “call.” Table . Topics in text messages for depression therapy produced by BTM. | **Topic** | **Topic Labeling** | **Top 20 words** | | --------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 1 | Life Style Remedies | do friends work far reward hard bad think plan something remember fun get coffee activity go movie day place call | | 2 | Close Monitoring | do today think tomorrow feel stress present change take opportunity determine rise up advantage mood something scale sad angry piss | | 3 | Cure, Recovery and Treatment | activity mood think improve positive today low feel attention impact note refer hit point try thing good negative thought value | | 4 | Treatment Adherence | first take medication feel go day help thought stress side effects stick initial discomfort last few plan little future sometimes | | 5 | Support and Motivation | lie success today stumble block become step stone better life adversity opportunity discourage problem turn behind before tiny matter compare | # Discussion This paper aimed to identify the content of text messages sent by a mental health professional as depression therapy. Previous studies have focused on lifestyle and remedies in depression therapy, such as nutrition and physical activity in depression therapy interventions (Looijmans et al., 2017; Navarro et al., 2020; Young et al., 2022). A longitudinal study (Velten et al., 2018) has proven that irregular social rhythms, smoking, and a vegetarian diet contributes to mental health problem among German and Chinese students. Thus, text message therapy focusing on lifestyle seems necessary and relevant to improving mental health. The Close Monitoring topic covers the text content that requires patients to rate their current feeling and mood based on a certain scale. Some text messages sent to patients also encourage patients to text back about their thoughts and activity (Aguilera et al., 2017; Ranney et al., 2017). Moreover, text messages are useful in self-monitoring depressive symptoms (Keding et al., 2015). Based on Table 6, all words listed are considered representable of Topic 2. Next, the experts in this study agreed that the words listed are all meaningful and related to Cure, Recovery, and Treatment, as shown in Topic 3. The experts also mentioned using those words in the depression therapy sessions. Some text messages in this study were constructed in the context of treatment adherence. For example, some text messages encourage patients to continue with clinic appointments, taking medicine, treatment follow-up, and consultation with professionals (Aguilera et al., 2017; Almeida et al., 2018; Välimäki et al., 2017). Text messages that embed supportive and motivational senses help manage follow-up care and medical appointments (Agyapong et al., 2016). Supportive and motivational text messages also leave positive sentiment in patients, motivating them to persevere with the treatment, grab any opportunity, be open to taking part in new activities, and promote positive thinking. Overall, the experts in depression counselling domain found that the top 20 words produced by topic modelling are interpretable into topics. The qualitative evaluation implies that topic labelling can vary depending on human judgement and knowledge in the area. One expert might not have the same idea of topic labelling as the others. Furthermore, the involvement of experts in qualitative evaluations through group discussion can reduce human biases in topic labelling. Nonetheless, topic modelling has not been used entirely for document analysis and still requires human coding for some factual reasons. First, the data sparseness under the context of short text data is a challenge in finding document-topic distributions caused by limited word co-occurrence information (Shi et al., 2019; Yang et al., 2020; Yi et al., 2020). Second, limited information in the context makes it difficult to find the meaning of ambiguous words in the short texts. Third, the word co-occurrence does not fully capture the semantic relationship and structure between words (Yi et al., 2020). This also contributes to difficulty in identifying the hidden meaning of a sentence. Nevertheless, topic modelling is useful in identifying topics in depression therapy text messages. Most importantly, using a topic modelling algorithm reduces the time and effort of going through a large number of documents (Asmussen & Moller, 2019). Besides, the use of algorithms provides more consistency in predicting topics. Human coders such as the experts in this study tend to suggest different topics based on their background, knowledge and experiences, which is inconsistent among the experts. # Conclusion This paper investigates the potential of three topic modelling methods to identify topics in text messages sent by mental health professionals as depression therapy. Based on this study, topic modelling is feasible in identifying hidden topics in depression therapy text messages. Although the three topic modelling methods used in this study have different capabilities in classification, clustering, and coherency, with BTM exhibiting the best accuracy score, WNTM performs better in clustering, while WNTM and LFDMM showed a higher score in coherence than BTM, the differences are comparatively small. Due to this, careful analysis and interpretation must be conducted. With the justifications discussed earlier, BTM is the most suitable method for identifying topics for the dataset of this study. It depends on the practical requirement to say whether topic modelling is appropriate to discover topics in documents. The qualitative approach used in this study shows that the top 20 words listed by BTM are interpretable. From the experiment, five topics are found to represent the 258 text messages in this study. It is assumed that the number of topics discovered by topic modelling can be more when using more text message samples in an experiment. Apart from that, a preliminary study needs to be conducted with different parameter settings for the researcher to decide the parameter combinations with the most efficient in making topic modelling before an actual experiment is carried out with a bigger sample dataset. # Acknowledgements The authors would like to thank all the experts involved in the qualitative analysis. # Conflict of interest statement The authors have no competing interests to declare. # Authors’ contributions XXX: Conceptualization; Data curation; Formal analysis; Writing - original draft; Investigation; Methodology; Project administration; Validation; Visualization; Resources; Software; YYY: Conceptualization; Writing - original draft; Investigation; AL: Writing - original draft; Visualization; NNNN: Supervision; Project administration; Resources; Writing - review & editing # References Aguilera, A., Bruehlman-Senecal, E., Demasi, O., & Avila, P. (2017). Automated Text Messaging as an Adjunct to Cognitive Behavioral Therapy for Depression:A Clinical Trial. Journal of Medical Internet Research, 19(5). https://doi.org/10.2196/jmir.6914 Agyapong, V. I. O., Juhás, M., Mrklas, K., Hrabok, M., Omeje, J., Gladue, I., Kozak, J., Leslie, M., Chue, P., & Greenshaw, A. J. (2018). Randomized controlled pilot trial of supportive text messaging for alcohol use disorder patients. Journal of Substance Abuse Treatment, 94(2018), 74–80. https://doi.org/10.1016/j.jsat.2018.08.014 Agyapong, V. I. O., Mrklas, K., Juhás, M., Omeje, J., Ohinmaa, A., Dursun, S. M., & Greenshaw, A. J. (2016). Cross-sectional survey evaluating Text4Mood: Mobile health program to reduce psychological treatment gap in mental healthcare in Alberta through daily supportive text messages. BMC Psychiatry, 16(1), 1–12. https://doi.org/10.1186/s12888-016-1104-2 Agyapong, V. I. O., Shalaby, R., Hrabok, M., Vuong, W., Noble, J. M., Gusnowski, A., Mrklas, K., Li, D., Snaterse, M., Surood, S., Cao, B., Li, X. M., Greiner, R., & Greenshaw, A. J. (2021). Mental health outreach via supportive text messages during the covid-19 pandemic: Improved mental health and reduced suicidal ideation after six weeks in subscribers of text4hope compared to a control population. International Journal of Environmental Research and Public Health, 18(4), 1–13. https://doi.org/10.3390/ijerph18042157 Albalawi, R., Yeap, T. H., & Benyoucef, M. (2020). Using Topic Modeling Methods for Short-Text Data: A Comparative Analysis. Frontiers in Artificial Intelligence, 3(July), 1–14. https://doi.org/10.3389/frai.2020.00042 Almeida, A. M. P., Almeida, H. S., & Figueiredo-Braga, M. (2018). Mobile solutions in depression: Enhancing communication with patients using an SMS-based intervention. Procedia Computer Science, 138(2018), 89–96. https://doi.org/10.1016/j.procs.2018.10.013 Anstiss, D., & Davies, A. (2015). “Reach Out, Rise Up”: The efficacy of text messaging in an intervention package for anxiety and depression severity in young people. Children and Youth Services Review, 58, 99–103. https://doi.org/10.1016/j.childyouth.2015.09.011 Asmussen, C. B., & Moller, C. (2019). Smart literature review: a practical topic modelling approach to exploratory literature review. Journal of Big Data, 6. https://doi.org/10.1186/s40537-019-0255-7 Barrera, A. Z., Aguilera, A., Inlow, N., & Servin, J. (2020). A preliminary study on the acceptability of a brief SMS program for perinatal women. Health Informatics Journal, 26(2), 1079–1087. https://doi.org/10.1177/1460458219866560 Cheng, X., Yan, X., Lan, Y., & Guo, J. (2014). BTM: Topic modeling over short texts. IEEE Transactions on Knowledge and Data Engineering, 26(12), 2928–2941. https://doi.org/10.1109/TKDE.2014.2313872 Franz, P. J., Nook, E. C., Mair, P., & Nock, M. K. (2019). Using Topic Modeling to Detect and Describe Self-Injurious and Related Content on a Large-Scale Digital Platform. Suicide and Life-Threatening Behavior, 50(1), 5–18. https://doi.org/10.1111/sltb.12569 Grün, B., & Hornik, K. (2011). topicmodels: An R Package for Fitting Topic Models. Journal of Statistical Software., 40(13), 1–30. Hartnett, D., Murphy, E., Kehoe, E., Agyapong, V., Mcloughlin, D. M., & Farren, C. (2017). Supportive text messages for patients with alcohol use disorder and a comorbid depression : a protocol for a single-blind randomised controlled aftercare trial. BMJ Open, 7:e013587. https://doi.org/10.1136/bmjopen-2016-013587 Howes, C., Purver, M., & McCabe, R. (2013). Investigating Topic Modelling for therapy dialogue analysis. Proceedings of the IWCS 2013 Workshop on Computational Semantics in Clinical Text (CSCT 2013), 7–16. https://www.eecs.qmul.ac.uk/\~mpurver/papers/howes-et-al13csct.pdf Keding, A., Böhnke, J. R., Croudace, T. J., Richmond, S. J., & MacPherson, H. (2015). Validity of single item responses to short message service texts to monitor depression: An mHealth sub-study of the UK ACUDep trial. BMC Medical Research Methodology, 15(1), 1–10. https://doi.org/10.1186/s12874-015-0054-6 Kerrigan, A., Kaonga, N. N., Tang, A. M., Jordan, M. R., Steven, Y., & Diseases, I. (2019). Content Guidance for Mobile Phones Short Message Service (SMS)-Based Antiretroviral Therapy Adherence and Appointmnet Reminders : A Review of the Literature. AIDS Care, 31(5), 636–646. https://doi.org/10.1080/09540121.2018.1549723.Content Kraft, S., Wolf, M., Klein, T., Becker, T., Bauer, S., & Puschner, B. (2017). Text message feedback to support mindfulness practice in people with depressive symptoms: A pilot randomized controlled trial. JMIR MHealth and UHealth, 5(5). https://doi.org/10.2196/mhealth.7095 Li, C., Wang, H., Zhang, Z., Sun, A., & Ma, Z. (2016). Topic modeling for short texts with auxiliary word embeddings. SIGIR 2016 - Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, 165–174. https://doi.org/10.1145/2911451.2911499 Looijmans, A., Jörg, F., Bruggeman, R., Schoevers, R., & Corpeleijn, E. (2017). Design of the Lifestyle Interventions for severe mentally ill Outpatients in the Netherlands (LION) trial; a cluster randomised controlled study of a multidimensional web tool intervention to improve cardiometabolic health in patients with severe mental i. BMC Psychiatry, 17(1), 1–14. https://doi.org/10.1186/s12888-017-1265-7 Navarro, C., Yáñez, A. M., Garcia, A., Seguí, A., Gazquez, F., Marino, J. A., Ibarra, O., Serrano-Ripoll, M. J., Gomez-Juanes, R., Bennasar-Veny, M., Salva, J., Oliván, B., Roca, M., Gili, M., & Garcia-Toro, M. (2020). Effectiveness of a healthy lifestyle promotion program as adjunctive teletherapy for treatment-resistant major depression during COVID 19 pandemic: A randomized clinical trial protocol. Medicine, 99(45), e22958. https://doi.org/10.1097/MD.0000000000022958 Nguyen, D. Q., Billingsley, R., Du, L., & Johnson, M. (2015). Improving Topic Models with Latent Feature Word Representations. Transactions of the Association for Computational Linguistics, 3, 598–599. https://doi.org/10.1162/tacl\_a\_00245 Onan, Aytuğ. (2022). Bidirectional convolutional recurrent neural network architecture with group-wise enhancement mechanism for text sentiment classification. Journal of King Saud University - Computer and Information Sciences, 34(5), 2098–2117. https://doi.org/10.1016/j.jksuci.2022.02.025 Onan, Aytug, & Tocoglu, M. A. (2021). A Term Weighted Neural Language Model and Stacked Bidirectional LSTM Based Framework for Sarcasm Identification. IEEE Access, 9, 7701–7722. https://doi.org/10.1109/ACCESS.2021.3049734 Pietsch, A. S., & Lessmann, S. (2018). Topic modeling for analyzing open-ended survey responses. Journal of Business Analytics, 1(2), 93–116. https://doi.org/10.1080/2573234X.2019.1590131 Qiang, J., Qian, Z., Li, Y., Yuan, Y., & Wu, X. (2019). Short text topic modeling techniques, applications, and performance: A survey. Journal of Latex Class Files, 14(8), 1–17. https://doi.org/10.1109/tkde.2020.2992485 Quan, X., Kit, C., Ge, Y., & Pan, S. J. (2015). Short and sparse text topic modeling via self-aggregation. Twenty-Fourth International Joint Conference on Artificial Intelligence, 2270–2276. Ranney, M. L., Freeman, J. R., Connell, G., Boyer, E., Walton, M., Guthrie, K., Cunningham, R. M., Behavior, H., G-, B., Worcester, N., & Arbor, A. (2017). A Depression Prevention Intervention for Adolescents in the Emergency Department. Journal of Adolescent Health, 59(4), 401–410. https://doi.org/10.1016/j.jadohealth.2016.04.008.A Schneider, J., & Vlachos, M. (2018). Topic modeling based on keywords and context. Proceedings of the 2018 SIAM International Conference on Data Mining, May, 369–377. https://doi.org/10.1137/1.9781611975321.42 Shariful Islam, S. M., Chow, C. K., Redfern, J., Kok, C., Rådholm, K., Stepien, S., Rodgers, A., & Hackett, M. L. (2019). Effect of text messaging on depression in patients with coronary heart disease: A substudy analysis from the TEXT ME randomised controlled trial. BMJ Open, 9(2), 1–7. https://doi.org/10.1136/bmjopen-2018-022637 Shi, L., Cheng, G., Xie, S. R., & Xie, G. (2019). A word embedding topic model for topic detection and summary in social networks. Measurement and Control (United Kingdom), 52(9–10), 1289–1298. https://doi.org/10.1177/0020294019865750 Tadesse, M. M., Lin, H., Xu, B., & Yang, L. (2019). Detection of depression-related posts in reddit social media forum. IEEE Access, 7, 44883–44893. https://doi.org/10.1109/ACCESS.2019.2909180 Tian, T., & Fang, Z. (2019). Attention-based Autoencoder Topic Model for Short Texts. Procedia Computer Science, 151, 1134–1139. https://doi.org/10.1016/j.procs.2019.04.161 Välimäki, M., Kannisto, K. A., Vahlberg, T., Hätönen, H., & Adams, C. E. (2017). Short text messages to encourage adherence to medication and follow-up for people with psychosis (mobile.net): Randomized controlled trial in Finland. Journal of Medical Internet Research, 19(7). https://doi.org/10.2196/jmir.7028 Velten, J., Bieda, A., Scholten, S., Wannemüller, A., & Margraf, J. (2018). Lifestyle choices and mental health: A longitudinal survey with German and Chinese students. BMC Public Health, 18(1), 1–15. https://doi.org/10.1186/s12889-018-5526-2 Wang, S. H., Ding, Y., Zhao, W., Huang, Y. H., Perkins, R., Zou, W., & Chen, J. J. (2016). Text mining for identifying topics in the literatures about adolescent substance use and depression. BMC Public Health, 16(1), 4–11. https://doi.org/10.1186/s12889-016-2932-1 World Health Organization. (2021). Comprehensive Mental Health Action Plan 2013 - 2030. Wu, X., Li, C., Zhu, Y., & Miao, Y. (2020). Short Text Topic Modeling with Topic Distribution Quantization and Negative Sampling Decoder. In B. Webber, Y. He, & Y. Liu (Eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (pp. 1772–1782). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.138 Yan, X., Guo, J., Lan, Y., & Cheng, X. (2013). A biterm topic model for short texts. WWW 2013 - Proceedings of the 22nd International Conference on World Wide Web, May, 1445–1455. https://doi.org/10.1145/2488388.2488514 Yang, Y., Wang, H., Zhu, J., Wu, Y., Jiang, K., Guo, W., & Shi, W. (2020). Dataless short text classification based on biterm topic model and word embeddings. IJCAI International Joint Conference on Artificial Intelligence, 3969–3975. https://doi.org/10.24963/ijcai.2020/549 Yi, F., Jiang, B., & Wu, J. (2020). Topic Modeling for Short Texts via Word Embedding and Document Correlation. IEEE Access, 8, 30692–30705. https://doi.org/10.1109/ACCESS.2020.2973207 Young, L. M., Moylan, S., John, T., Turner, M., Opie, R., Hockey, M., Saunders, D., Bruscella, C., Jacka, F., Teychenne, M., Rosenbaum, S., Banker, K., Mahoney, S., Tembo, M., Lai, J., Mundell, N., McKeon, G., Yucel, M., Speight, J., … O’Neil, A. (2022). Evaluating telehealth lifestyle therapy versus telehealth psychotherapy for reducing depression in adults with COVID-19 related distress: the curbing anxiety and depression using lifestyle medicine (CALM) randomised non-inferiority trial protocol. BMC Psychiatry, 22(1), 1–12. https://doi.org/10.1186/s12888-022-03840-3 Zuo, Y., Wu, J., Xhang, H., Lin, H., Wang, F., Xu, K., & Xiong, H. (2016). Topic Modeling of Short Texts: A Pseudo-Document View. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2105–2114. https://doi.org/10.1109/TKDE.2021.3073195 Zuo, Y., Zhao, J., & Xu, K. (2016). Word network topic model: a simple but general solution for short and imbalanced texts. Knowledge and Information Systems, 48(2), 379–398. https://doi.org/10.1007/s10115-015-0882-z.

A picture containing text, clipart Description automatically generated

 © 2024 by the authors. Submitted for open access publication under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
1. \* Corresponding author. *E-mail address*: (Please do not type or edit anything here, our editors will do the work for you)