Service-based Processing and Provisioning of Image-Abstraction Techniques

M. RICHTER, M. SÖCHTING, A. SEMMO, J. DÖLLNER, M. TRAPP

User generated content (UGC) as worldwide central source and medium for information,
entertainment, documentation, and communication
Images shared [kbcp2016]
[https://www.instagram.com/explore/tags/nature/]

Observation
▪ Image abstraction techniques are widely used for entertainment purposes
▪ Artistic image abstractions are more likely to be distributed and shared
than photorealistic images [Bakhshi et al. 2015]
▪ Image abstraction is a typical approach to share images with sensitive content
Objective
▪ Support image enhancement and image abstraction techniques
▪ Produce artistic effects such as obtained by style transfer
[Gatys et al., 2016]
▪ Provide interactive image manipulation and
give control over displayed content
with an adjustable degree of abstraction and stylization

▪ Mobile on-device image stylization application with high degree of interactivity and parameterization utilizing the
generic image effect representation [Dürschmid et al., 2017]
▪ Limitations: (1) Design, implementation, testing and deployment are cost-intensive; (2) device heterogeneity;
(3) image resolution and processing time

▪ Server-side processing: heterogeneous clients can be attached to provide hardware-independent cross-
platform solutions
▪ Service-Oriented Architecture (SOA) pattern is predominant design pattern [Winkler et al., 2013]
▪ Outsourcing of intensive processing tasks to the server
▪ Clients only trigger processing, download, and display results (thin clients)
▪ Low battery consumption of mobile clients
▪ Images mostly reside on cloud storages (e.g., Instagram, Facebook)
▪ Image management and processing is feasible without roundtrip over client device

1. Service-based system for off-device image processing by compositing atomic low- to high-level services
2. Provisioning of platform-independent representations of image effects for customizable configuration
3. Demonstration for Android and desktop systems, and integration into the Google Office Suite

Service-based Processing and Provisioning of Image-Abstraction Techniques

▪ Service components of server infrastructure

▪ Orchestration Service
▪ API gateway providing the service endpoints
▪ Composition of low-level services to high-level service (Orchestration)

▪ Image Processing Service
▪ Central component for processing images
▪ Requires an image and an image effect and outputs a stylized image

▪ Effect Service
▪ Storing and provisioning of image effects
▪ Image effects comprise platform-independent information (e.g.,
parameters, presets) and implementation-specific assets (shader)
Shader
XML-based effect definition

▪ Effect Service
▪ Storing and provisioning of image effects
▪ Image effects comprise platform-independent information (e.g.,
parameters, presets) and implementation-specific assets (shader)
▪ Resource Management Service
▪ Responsible for storing and provision of resource data (images)
▪ Caching of intermediate results

▪ Wide range of clients can use the services via the language-agnostic interface
of the Orchestration Service

▪ Two major factors affect runtime performance:
▪ Data transmission
▪ Data processing
▪ Runtime of image processing linearly depends
on resolution
▪ High-resolution images (256 megapixels) can be
processed but requires more than 4 GB RAM
▪ The implementation is not real-time capable yet
▪ Processing of common HD resolutions takes 0.3-0.7s
▪ Effect loading and image decoding are bottlenecks

Service-based image-processing system with the following properties:
▪ Atomic services allow composition and creation of higher-level services
▪ Execution of resource intensive tasks
▪ Provisioning of platform-independent effect representations with highly customizable configuration
▪ Demonstration for research and industrial applications
Future Work
▪ Extend service-based processing approach for video processing
▪ Speed up and scale up the image processing service
▪ Support tile-based rendering to process images with resolutions higher than 16k2 pixels

Contact
▪ Marvin Richter (marvin.richter@hpi.de)
▪ HPI Computer Graphics Systems Group (Image & Video Processing)
▪ This work was partly funded by the
Federal Ministry of Education and Research (BMBF)
for the AVA project 01IS15041B (https://ava.hpi3d.de)
▪ Industry Partner:
Digital Masterpieces GmbH (http://www.digitalmasterpieces.com)

▪ [kbcp2016] https://www.scoopnest.com/user/ValaAfshar/738076291874889728
▪ [Winkler et al., 2013] Robert P. Winkler and Chris Schlesiger. Image processing rest web services. Technical Report
ARL-TR-6393, Army Research Laboraty, Adelphi, MD 20783-119, 2013.
▪ [Bakhshi et al. 2015] Saeideh Bakhshi, David A Shamma, Lyndon Kennedy, and Eric Gilbert. 2015. Why We Filter Our
Photos and How It Impacts Engagement. In Proc. ICWSM. 12–21.
▪ [Gatys et al., 2016] Leon A. Gatys, Matthias Bethge, Aaron Hertzmann, and Eli Shechtman. 2016a. Preserving Color
in Neural Artistic Style Transfer. arXiv.org report 1606.05897. https://arxiv.org/abs/1606.05897
▪ [Dürschmid et al., 2017] Tobias Dürschmid, Maximilian Söchting, Amir Semmo, Matthias Trapp, and Jürgen Döllner.
Prosumerfx: Mobile design of image stylization components. In SIGGRAPH Asia 2017 Mobile Graphics & Interactive
Applications, SA ’17, pages 1:1–1:8, New York, NY, USA, 2017. ACM.

• To tackle more complex use cases and build higher level services
for divers visual media applications composition is a critical question
• Two composition strategies are prevelant:
• Orchestration: single centralized process coordinates the interaction
among different services
• Choreography: direct decentralized communication between services
• Examples of orchestration metaservices:
• 1) image processing (server-side orchestration)
• 2) effect composition in android client (client-side orchestration)
Client-side composition
of effects
Orchestration vs.
Choreography

• Number of users increases exponetially → Deploy services on server farms (AWS, Hetzner)
• Goal
• Make the system scalable
• Dynamic increase of service instances
• Load balancing
• Resource management on distributed data
Microservices
Scaling

1. Services should be small and focus on one specific thing (Keep scope and functionality small)
2. Services should run in their own containers/environments
3. Each service owns its own data (individual database)
4. Services communiate via language-agnostic APIs (e.g., REST, SOAP, Message Broker such as RabbitMQ
and Apache Kafka, …)
5. Services are independently deployable and scalable

Service-based Processing and Provisioning of Image-Abstraction Techniques

More Related Content

Similar to Service-based Processing and Provisioning of Image-Abstraction Techniques (20)

More from Matthias Trapp (20)

Recently uploaded (20)

Service-based Processing and Provisioning of Image-Abstraction Techniques