Multimodal model based on image and text for predicting users' interest on the pinterest social network

Abstract

The increased use of social media has led to a substantial growth in user-generated information in recent years. Revealing user interests within these platforms represents an opportunity to generate recommendation systems, conduct market research, and take advantage of this massive amount of information. This thesis proposes developing a multimodal model based on images and text to predict user interests within the social network, Pinterest. In this model, text and images are transformed through Word-embeddings and Deep-learning models to optimize a logistic regression classifier for each model and modality independently. The construction of the Mix-Modality model is performed through the combination of six different models. Of these, four are based on selecting the logistic regression models with the best score (two for images, two for text) and the remaining two on fine-tuned models of BERT and RoBETa used exclusively as text classifiers. The combination of the models based on late fusion is generated through a weighted sum according to the effectiveness of each model for predicting user interests. In addition, a comparison with two other fusion methods discussed in the literature is presented, where the fusion is made by applying a lambda factor that affects images directly or using a feature vector crossing technique. Considering the top-k accuracy metric, the results show the Mix-Modality model consistent with better results than the unimodal models and the two fusion methods.