Summary

This dissertation investigates how abstract attributes, such as restaurant atmosphere or a firm’s sustainability commitment, can be computationally inferred from large-scale social multimedia and linked to real-world outcomes. It advances a general framework that (i) aggregates social multimedia (text, images, interactions), (ii) detects abstract attributes using deep learning with weak or indirect supervision, and (iii) validates these signals through prediction of concrete, real-world performance indicators.

In experiential services, the work models restaurants within a heterogeneous information network to capture both concrete (e.g., price, cuisine, location) and abstract characteristics. Using metapath-guided embeddings and attention-based fusion, the approach predicts organic success proxies (review volume), revealing complementary relational views that enhance performance. Shifting to user-generated imagery, the dissertation shows that visual content captures gradations in abstract service qualities (food and service quality, value, atmosphere). It leverages group-level weak supervision from ratings and an inverted question-answering scheme to infer attribute relevance without costly annotations, using gradient-based visualizations to reveal discriminative cues.

In corporate sustainability, the thesis addresses a key data gap with GreenScreen, a large-scale multimodal dataset of corporate social media posts and images from major publicly listed firms, paired with ESG risk assessments. Embedding-based models learn associations between sustainability messaging and ESG performance. Finally, an ensemble of open-weight (vision-)language models enables zero-shot SDG relevance annotations, supporting analyses of temporal trends, sectoral communication patterns, and visual themes that correlate with audience engagement and ESG risk.

Collectively, the dissertation shows that abstract and subjectively perceived attributes can be operationalized from multimedia at scale.