OpenAI’s Sora is coming. Early testers reveal the pros and cons of the text-to-video generator.

OpenAI gave access to several visual artists, designers, and filmmakers to get feedback on how to advance the model in a way that would be most helpful for creative professionals.

Four of the artists and early testers of Sora spoke to Business Insider about their experiences using the model and some of the tool’s positive aspects, as well as its drawbacks.

Charlotte Triebus, a performance artist and choreographer, has been testing out Sora for about four months.

She prompted Sora to generate a video of human drooling but said that Sora refused.

Triebus finds that it can generate people of diverse genders. However, it still has issues with physics, which she finds interesting artistically because it gives her a new way of thinking about the human body.

The German artist uses Sora to create ideas or sketch visions that she can then use in her choreography with dancers. The video she generated with Sora, which OpenAI shared on YouTube, depicts a human body with extra limbs.

Triebus said Sora would display a diverse group of people when she first entered a prompt, but then it would turn them into other ethnicities or mostly white people after she entered other prompts.

This topic was raised when Triebus delivered feedback to OpenAI, and she was told the company was working to resolve the issue.

Overall, the artist is very excited about Sora’s possibilities: “Artistically, it is something that can really assist us in visualizing things and exploring things.”

OpenAI says on its blog that it’s working with “red teamers” to test Sora in areas including bias, hateful content and misinformation. It is also developing tools to detect if a video was generated by Sora.

It’s helpful for exploring ideas

Architect Tim Fu has used AI to experiment with architectural design for about three years and said it has automated the processes involved.

A few years ago, Fu started using models such as Midjourney, which he said has helped create an entirely different mode of practice. AI has turned the artist into more of an art curator, as tools like Sora can help visualize ideas that can be tweaked until the desired outcome is reached.

“The idea is to get an experiential perspective of what is a new way of architecture, and how AI can help us design and create a new world, and how we get new experiences,” Fu said.

Limitations

In Fu’s view, using Sora in architectural design has its limitations.

“It’s regurgitating its database, and you can tell that some of it is dated, especially in urban design,” he said.

Fu’s studio wants to simulate what urban design will look like in 100 years or so.

“But as you type something futuristic, you either get a sci-fi database or a super CGI-looking database, which obviously is the type of stuff that exists in the database, probably from sci-fi films or CGI renders, or it becomes ultra-realistic camera angles and resembles films of the past,” he said.

“There’s no sweet spot in between, and there are definite limitations to what’s in the database and what we want to create.”

OpenAI told BI that a diffusion model like Sora was not designed to regurgitate videos. Instead, it learns concepts and then creates original videos based on its understanding of the physical world.

Fu thinks OpenAI should release Sora soon as “it’s no longer that special, with competitors catching up.”

Rivals Luma AI and Runway have already released their models, which allow users to upload their own visual materials as prompts.

He said Sora’s capabilities in generating urban architecture were “kind of disappointing,” and finds Runway better because of its image-to-video feature.

Democratize storytelling

Manuel Sainsily, a TED speaker, artist, and extended reality (XR) instructor at McGill University, and William Selviz, an immersive and virtual production director and designer, collaborated on a project called Protopica to explore how Sora could leverage stories from around the world.

Sainsily, from Guadeloupe, and Selviz, who is Venezuelan and grew up in Kuwait, created a short film to do just that.

In their view, Sora is helpful in telling stories they want to share that are hard to picture or even film because of accessibility.

“The goal for us was really to tell stories that resonate with our cultures and that can put a little bit of a pin on the map for all those stories that don’t have a chance to be told without those tools,” Sainsily said.