Ahsoka's In-Camera VFX: Art Direction Breakdown

An in-depth look at the virtual set design process used to create stunning in-camera VFX environments for Ahsoka.

Written by: Felix Jorge - Collaborator: Safari Sosebee

I wanted to share a breakdown of the virtual art department process used in Lucasfilm’s Ahsoka, building on techniques from The Mandalorian Season 1. We pushed Unreal Engine scenes in pre-production up to camera-ready, allowing for real-time environments that were both visually stunning and integrated with physical sets.

The team leveraged an extensive Unreal Engine asset library from projects like The Mandalorian and Obi-Wan Kenobi, speeding up the set decoration process. By working closely with the art department, we designed, lit, and refined multiple virtual sets in collaboration with other key creatives. Delivering final pixel scenes to an Unreal Engine stage process, Helios and traditional VFX.

At a Glance:

Schedule

• Concept Art and Design: 8 weeks
• VAD Construction: 9 months
• Volume Integration and Testing: 4 months

Resources

• Number of Real-time VAD Artists: 14
• Number of Engineers: 3

Deliverables

• Hero Location Sets With Multiple Variants: 17 
• Set Variants: 65+
• Set Dressing Assets photogrammetry: 30+
• Lighting Scenarios: 85+
• VScout Storyboard-Frames: 300+

Partner Reviews

• Virtual Stage Walks With Production Designer: 190+
• Prelight & Camera Blocking Sessions With DP: 65+
• Hosted Creative Reviews (VScouts): 42+

Partner Objectives and Challenges

The project aimed to create Unreal Engine virtual sets that were as close to the final image as possible, marking a shift from the usual workflow that relied heavily on post-production for further refinement. Unlike earlier seasons, there was a higher confidence in the team and process. While we can't speak to the differences between Unreal Engine and Helios, we will focus on the advancements and techniques we were driving.

To achieve high-quality sets at the speed of previs, we restructured the team and trained VFX sculptors and lighters in real-time asset creation and lighting. Regular communication with the production team and virtual location scouting sessions allowed us to direct our work where the camera was looking, focusing efforts where they were most needed.

Getting Started: Starting the Initial Blockout Set Model

It all started with meetings between the production designer, Andrew Jones, and VAD Art Director, Safari Sosebee, to review reference and concept art. At this stage, we incorporated set designers' 3D models, VFX assets, and various asset libraries into a unified Perforce database, configuring server permissions to ensure secure access for each department based on the show's requirements.

These initial block-outs, created using simple shapes, were crucial as they laid the groundwork for everything that followed. Although this step might seem basic, it was essential for setting the foundation of the environments and ensuring a cohesive vision moving forward.

Set Decoration: Asset Development for the Sets

The development of a set goes through several stages to help creatives stay on time and schedule. We begin with block-outs, move to previs, then progress to look-development, and finally reach the final version.

The final version is carefully vetted, tested on stage, and optimized. Passing off an Unreal Engine file for each scene that includes multiple lighting scenarios. The sets are handed over to the stage teams, who save them, ready for the shoot, and or last-minute tweaks.

Hosting Group Creative Reviews: Virtual Location Scouts

On The Mandalorian Season 1, we used Virtual Location Scouts. For Ahsoka, we introduced remotely driven, multi-user Virtual Location Scouts, which were crucial for keeping everyone aligned.

We hosted regular sessions with directors, production designers, DPs, storyboard artists, writers, and producers to review locations in real-time, place cameras, and create still frames. These frames guided focus and were utilized across all departments, including previs, VAD, VFX, and physical construction.

These multi-user virtual location scouts lasted 1 to 2.5 hours and produced 30 to 80 usable frames per session, which were immediately shared with key production members.

We also provided remote access for creatives to explore environments from their homes whenever they wanted. Facilitating cloud-based meetings allowed them to collaborate on a virtual location and work on their story together, ensuring all decision-makers remained aligned.

Ahsoka Key Creative Team:

• Andrew Jones - Production Designer
• Doug Chiang - Production Designer
• Jon Favreau - Creator/Executive Producer/Writer
• Dave Filoni - Creator/Executive Producer/Director/Writer
• Eric Steelberg - Director of Photography
• Q Tran - Director of Photography
• Rachel - Art Director
• Clint Spillers - VP Producer

Capturing Set Decoration and Model Makers Miniatures

Photogrammetry of life-size props and miniatures, along with set decorations, allowed for the production designer and set decorator to drive the look and marry the physical and virtual props. These scanned assets go through a cleanup process, where the textures go through a flattening process, and low, medium, and high assets are created.  

Whenever we scanned a miniature, we collaborated closely with model makers to scale these miniatures into life-size sets, using them as foundations for block-outs. An example of this is the Seatos StoneHenge above.

Out of the 17 hero locations, 7 of them utilized photogrammetry captured with the art department.

Lighting and Rendering: Hitting the Final Look

Lighting is a vital component of in-camera VFX. Under the guidance of Safari, Narwhal’s studio Art Director and an experienced lighter with a strong background in real-time and VFX, the lighting for the sets was continuously refined throughout the project.

Each week, the lighters collaborated with directors of photography Eric Steelberg and Q Tran to establish base lighting for scenes, creating multiple scenarios that were fine-tuned for virtual location scouts. Through trial and error, and close collaboration with the physical construction, set decoration, stage, and VFX teams, we kept adjusting the lighting until the key creatives were satisfied.

Lighting Tips from Safari Sosebee

Here are some tips from Safari: Start by understanding the action and mood, and gather references to guide your work. Begin lighting with the major shapes in place, using broad strokes to establish key angles and camera placements. As you refine the lighting, focus on sculpting light to highlight areas of interest, and adjust light color and intensity as textures and materials are introduced. Finally, fine-tune the placement of lights and camera settings to achieve the desired complexity and visual impact.

Once the lighting was approved through the lens on the stage, we handed it over to the stage team. Before this handoff, we conducted stage tests and incorporated feedback to ensure everything met the required standards.

Turnover Process: Art Department Techvis

The turnover process was a very important part of planning the shoot, involving Techvis developed in close collaboration with the art department, stage and vfx teams.

Night Variant

For the Per Henge Volume Set, various lighting scenarios and volume placements were used to accommodate different scenes.

Late Morning Variant

Each set variant, such as night, late morning, and late afternoon, requires specific adjustments like set decoration, and lighting. These elements are planned and coordinated before filming to ensure a seamless production process.

Late Afternoon Variant

The location of each volume in the Unreal Engine scene labeled A1, A2, B1, B2, etc., dictates the content displayed on the screens. While some lighting scenarios and set decorations are consistent across scenes, others require specific alterations to match the story beats.

Day Variant

Regular LED-stage testing was conducted with the director of photography and other key creatives to ensure the virtual sets appeared realistic and met the creative vision. Feedback from these tests was used to refine the environments and passed off to the ILM team.

Afternoon Variant

The turnover process included detailed documentation, including techvis, volume placements, virtual scout cameras, set decoration, reference and concept art. This method mirrors the coordination that typically takes place by physical construction teams in the art department, adapted to the virtual environment.

Coordinating and Crafting Multiple Locations Simultaneously

Each environment for Ahsoka had unique challenges and required a tailored approach to effectively tell the story. To manage this, we formed pods with an art director, team lead, set owners, and VAD and PHG artists, running up to four sets at once.

Set owners worked with generalist VAD artists to finalize designs, guided by the VAD Art Director and Lead, who prioritized tasks based on feedback from the VP Producer and Production Designer. This process focused on reviewing still frames from weekly virtual location scouts to address the most critical elements.

All of this was managed in Shotgrid, which we heavily customized to suit the virtual art department workflow. We programmed specific functionalities to easily adapt to daily sprint plans that would drive regular changes.

Daily sprint planning ensured our focus was on the most crucial tasks. Updates were then communicated through Shotgrid and Mattermost, a secure, server-hosted chat platform, keeping everything off the internet.

Virtual Set Challenge Examples

Designing virtual sets for Ahsoka came with unique challenges. Each environment needed tailored strategies to achieve visual fidelity while capturing all the necessary story beats.

The Shipyard Location

The shipyard was one of the most ambitious sets for Ahsoka due to its moving sequence through a long corridor. Starting with simple geometric block-outs, we added complexity with submarine factory models to enhance depth and realism.

Finding the right scale was a challenge, so we experimented with different sizes, from a single ship to dozens of parts. With constant feedback from Production Designer Andrew Jones and Director of Photography Eric Steelberg, we achieved a balanced and visually stunning set that met the creative vision and set the foundation for future shots.

Eye of Sion Location

For the Eye of Sion environment, the script called for an animated pulsing effect when the generators were activated, but the creatives had no specific visual concept. Safari Sosebee and Grant Anderson presented multiple options, experimenting with various designs and effects.

After several rounds of feedback and adjustments, they finalized a design that captured the desired look and energy for the scene. The animated lights can be seen in the final shot, behind the actress.

Peridea Henge Location

The Peridea Henge set required over 6 lighting variants, each tailored to different scenes as characters moved through the space and action unfolded in various areas. To accommodate these changes seamlessly, the lighting needed to be flexible and adaptable.

We matched the lighting variants to set decoration change-overs and provided five different lighting setups that could be used on the day of principal photography. This approach allowed the team to be responsive and adjust the lighting as needed during filming.

Noti Reveal Location

The Noti Reveal set began as a simple misty landscape, but as production progressed, it became clear that it needed a new look to better fit the evolving story. The creative team decided to create multiple designs, and with only a few weeks before the turnover deadline, our team quickly adapted to implement the changes.

This is a prime example of the kind of last-minute adjustment that might arise, even when there’s alignment across all creatives. This came down to the story, and so we re-designed the virtual set to drive more of a visual impact and fit the description closely in the script.

Lessons Learned

The Ahsoka project provided key lessons in optimizing workflows and achieving the desired visual quality. Here are a few lessons learned:

• Engaged Production Team: A dedicated and interested production team is essential for embracing this process.
• Clear Roles: Defining VAD roles and thorough onboarding are crucial, especially for new team members. A buddy system and regular training helped maintain smooth collaboration.
• Project Management: Strong coordination and communication were vital for managing the project’s scale with a small team, ensuring high quality.
• Lighting and Assets: Efficiently managing hundreds of assets and lighting scenarios is essential for scalability and consistent quality.
• Efficiency at Speed: Producing high-quality scenes quickly required focused efforts and flexible scheduling, supported by virtual location scouts.
• Proactive Solutions: Close collaboration with art directors and the virtual art department, and being proactive at problem-solving were key to meeting project goals.

Conclusion

As the project wrapped up, Safari shared this image by Lee Siassi, our scout artist at the time. Although the team has since moved on to different ventures, I send them good vibes and hope they carry these valuable lessons with them.

Really happy about the lessons learned. For those interested real-time design, I hope this shares some ideas on how to use the process. I welcome any feedback, thank you for reading.