Capturing Political Communication Online Using Image and Text Data: A Deep Learning Approach

Abstract

Social media data enables political scientists to observe phenomena that have been otherwise difficult to capture. The scale and structure of such data is problematic, however, as sorting social media posts by hand is a prohibitively costly endeavor. For instance, there are over 500 million tweets posted per day, consisting of text, image, gif, and video content. This has created a technology gap between what social scientists want to do, conceptually, and what they can do, computationally. This study develops text- and multimodal (text and image) classification technology. Such methods are used to investigate questions in algorithmic bias, election experiences, and Black Lives Matter protest activity. Multiple machine learning algorithms -- called convolutional neural networks or textit{deep learning} models -- were developed. These models were trained on facial images and tweet text. Results indicate that deep learning achieves high accuracy on training data; performance declines when the machine attempts to predict the previously unseen validation set. These algorithms can lack predictive power. Deep learning shows promise for automated content analysis, but more work must be done to curate theoretically motivated training data. Social scientists should focus on features in the data that best differentiate categories of interest. This study contributes to larger trends in computational social science that seek to apply machine learning methods to problems in political science. Even the most advanced methodology, however, must be wrapped in strong theory and substantively interesting questions.