Category Archives: Deep Learning

CAPTAIN: Comprehensive Composition Assistance for Photo Taking

Abstract: Many people are interested in taking astonishing photos and sharing with others. Emerging high-tech hardware and software facilitate ubiquitousness and functionality of digital photography. Because composition matters in photography, researchers have leveraged some common composition techniques, such as the rule of thirds, the triangle technique, and the perspective-related techniques, to assess the aesthetic quality of photos computationally. However, composition techniques developed by professionals are far more diverse than well-documented techniques can cover. We leverage the vast under-explored innovations in photography for computational composition assistance, and there is a lack of a holistic framework to capture important aspects of a given scene and help individuals by constructive clues to take a better shot in their adventure. We propose a comprehensive framework, named CAPTAIN (Composition Assistance for Photo Taking), containing integrated deep-learned semantic detectors, sub-genre categorization, artistic pose clustering, personalized aesthetics-based image retrieval, and style set matching. The framework is backed by a large dataset crawled from a photo-sharing Website with mostly photography enthusiasts and professionals.
The work proposes a sequence of steps that have not been explored in the past by researchers.
The work addresses personal preferences for composition through presenting a ranked-list of photographs to the user based on user-specified weights in the similarity measure. We believe our design leveraging user-defined preferences. Our framework extracts ingredients of a given snapshot of a scene (i.e. the scene that the user is interested in taking a picture of) as a set of composition-related features ranging from low-level features such as color, pattern, and texture to high-level features such as pose, category, rating, gender, and object. Our composition model, indexed offline, is used to provide visual exemplars as recommendations for the scene, which is a novel model for aesthetics-related image retrieval. We believe our design leveraging user-defined preferences The matching algorithm recognizes the best shot among a sequence of shots with respect to the user’s preferred style set. We have conducted a number of experiments on the newly proposed components and reported findings. A user study demonstrates that the work is useful to those taking photos.

Keywords: Computational Composition, Image Aesthetics, Photography, Deep Learning, Image Retrieval


Deep-learned Models and Photography Idea Retrieval

Intelligent Portrait Composition Assistance (IPCA)
Farshid Farhat, Mohammad Kamani, Sahil Mishra, James Wang
ACM Multimedia 2017, Mountain View, CA, USA
(Acceptance rate = 64/495 = 12.93%)

ABSTRACT: Retrieving photography ideas corresponding to a given location facilitates the usage of smart cameras, where there is a high interest among amateurs and enthusiasts to take astonishing photos at anytime and in any location. Existing research captures some aesthetic techniques such as the rule of thirds, triangle, and perspective, and retrieves useful feedbacks based on one technique. However, they are restricted to a particular technique and the retrieved results have room to improve as they can be limited to the quality of the query. There is a lack of a holistic framework to capture important aspects of a given scene and give a novice photographer informative feedback to take a better shot in his/her photography adventure. This work proposes an intelligent framework of portrait composition using our deep-learned models and image retrieval methods. A highly-rated web-crawled portrait dataset is exploited for retrieval purposes. Our framework detects and extracts ingredients of a given scene representing as a correlated hierarchical model. It then matches extracted semantics with the dataset of aesthetically composed photos to investigate a ranked list of photography ideas, and gradually optimizes the human pose and other artistic aspects of the composed scene supposed to be captured. The conducted user study demonstrates that our approach is more helpful than the other constructed feedback retrieval systems.

Please cite our paper if you are using our professional portrait dataset.

author = {Farhat, Farshid and Kamani, Mohammad Mahdi and Mishra, Sahil and Wang, James Z.},
title = {Intelligent Portrait Composition Assistance: Integrating Deep-learned Models and Photography Idea Retrieval},
booktitle = {Proceedings of the on Thematic Workshops of ACM Multimedia 2017},
series = {Thematic Workshops ’17},
year = {2017},
isbn = {978-1-4503-5416-5},
location = {Mountain View, California, USA},
pages = {17–25},
numpages = {9},
url = {},
doi = {10.1145/3126686.3126710},
acmid = {3126710},
publisher = {ACM},
address = {New York, NY, USA},
keywords = {deep learning, image aesthetics, image retrieval., photographic composition, portrait photography, smart camera},


Skeleton Matching for Severe Weather Detection

Title: Skeleton Matching with Applications in Severe Weather Detection

Authors: Mohammad Mahdi Kamani, Farshid Farhat, Stephen Wistar and James Z. Wang.

Elsevier Journal: Applied Soft Computing, ~27 pages, May 2017.


Severe weather conditions cause an enormous amount of damages around the globe. Bow echo patterns in radar images are associated with a number of these destructive conditions such as damaging winds, hail, thunderstorms, and tornadoes. They are detected manually by meteorologists. In this paper, we propose an automatic framework to detect these patterns with high accuracy by introducing novel skeletonization and shape matching approaches. In this framework, first we extract regions with high probability of occurring bow echo from radar images and apply our skeletonization method to extract the skeleton of those regions. Next, we prune these skeletons using our innovative pruning scheme with fuzzy logic. Then, using our proposed shape descriptor, Skeleton Context, we can extract bow echo features from these skeletons in order to use them in shape matching algorithm and classification step. The output of classification indicates whether these regions are bow echo with over 97% accuracy.

Full Text: Skeleton_Matching_Severe_Weather_Forecasting

NEWS Highlights:

Big data provides deadly storm alert