What do you get if you instruct an AI to turn a house into the most haunted house in the world? What if you ask it for the LEAST haunted house? How does an AI know what "haunted" looks like, anyways?

I did some experiments with CLIP+VQGAN (link and tutorial here). This makes use of CLIP, which learned what kinds of images and text are associated with one another on the internet. Since CLIP has probably seen lots of things labeled "haunted", it can probably guide VQGAN into making some seriously spooky alterations to this lovely 1894 Victorian house:

A beige wooden Victorian house with two turrets, long upper and lower balconies, and intricate wooden decorations. It sits on a green lawn surrounded by thin-leafed trees under an overcast sky.
Image by Josephyurko, cc-by SA 4.0

For my first try, I had it start with this image and alter it to look more like "A haunted Victorian house".

Ominous red glows from within the ground floor of a white stone house, while black draperies or possibly cobwebs, or possibly bones, drop from green-tinted balconies.
A haunted Victorian house

The wood has turned to stone, the yard has proliferated more iron fences (and become greener), and one of the turrets has changed to a chimney pot. Other than that, it looks pretty ordinary.

I decided to kick it up a notch, and told the model to head toward "An extremely haunted Victorian house | dramatic atmospheric night photography."

A house made mostly of grey stone, rickety balconies, and moonlight. A smaller two-windowed 1950s bungalow peers from beneath a floating turret.

The walls became wobblier, another of the turrets changed to a chimney, and curiously, there now seems to be a smaller more modern house inside the larger house.

It turns out a lot of this is the "dramatic atmospheric night photography".

Left and right images show a similar stone house beneath moonlight with many ridiculous columns and layered balconies. The “normal very not haunted” house is a little bluer, and has less of the inviting golden light glimmering from its windows.
Left: A victorian house, dramatic atmospheric night photography. Right: A normal very not haunted victorian house, dramatic atmospheric night photography. 

But the interface I'm using lets me assign negative weights to things, so I can actually subtract "haunted" from the house I'm trying to make. It sort of works! Once I subtract enough "haunted."

A fancy house in golden stone surrounded by sprays of blue and red flowers. The light of a sunset storm illuminates its complicated pillars and balconies, and from the red brick rooms on the ground floor there are several invitingly lit cases that seem like they would contain ice cream and snacks.
Specifically, this is minus two "haunted"

Apparently CLIP has learned that if you want to make things less haunted, add flowers, street lights, and display counters full of snacks.

This, in case you're interested, is the least haunted place in the world. It's what you get when you give CLIP+VQGAN no instructions except for a negative penalty to "haunted".

It appears to be a baseball diamond in an office park on a partly cloudy day. There are tropical flowers and stadium advertisements and possibly the cab of a starfish-textured semi truck.

I'm not sure what most of this stuff is supposed to be, but if you can find it and bring it to the middle of a baseball diamond in an office park, you will be safe.

I also tried haunting a Waffle House.

The moon shines from the clouds onto a Waffle House, brightly lit from within by cold white chandeliers. There’s mist everywhere, and it has just rained. The words “waffle house” are distorted almost beyond legibility.
"World's most haunted waffle house", starting from this image by MBrickn.(cc-by-SA)

Bonus content: a haunted bookstore, a haunted bakery, and a haunted grocery store.

Subscribe now