Drawception Data Science Project Part 1
I had several goals when selecting a topic for my capstone project. I wanted a project that demonstrates my skills, had a clearly defined data set, had a scalable range of success and most of all was fun and exciting. The topic I chose was predicting reactions on images from the website drawception.com. This topic has lots going for it. To complete my capstone, I have to demonstrate web scraping, computer vision with neural networks and possibly natural language processing. The data for the project will come directly from the website without any need to worry about finding public data sources. The goal of predicting reactions can be expanded to predicting image captions and possibly even producing computer generated images. Best of all I get to spend my time looking at silly pictures on the internet and call it work.
Before I get into the details, let me explain Drawception. The idea is that you are given a prompt to draw and a simple web art tool to draw it. Someone else will see your image and write a prompt based on it, and the process continues. A single game typically has 12 players, with 6 drawing and 6 writing captions. You can see an example of an image I made below.
After a game is finished anyone can view the game. With an account anyone can comment and react to images and captions. There are several different emojis you can give as a reaction, in the example above my crow got a smile, 2 loves and a wow. Games can have special modifiers that include extra players, alternate color palettes like grayscale, and the games can get featured as a top game.
So what have I done already, and where am I going? The data collection is complete. For every image I have lots of extra data. I have the author, any special game tags, the captions before and after, and the number of each reaction. Collecting the data was a multi-step process. First I collected the links from the browse games section of the site. Inside each game I collected the image urls, reactions and captions. Finally I needed to get a local copy of all the images. At time of writing I have over 20,000 images collected taking up 800MB of disk space. All of my images were created between 12/18/20 and 12/26/20 so there is likely a lot of Christmas specific images and captions. A final note, the browse games section of the site only shows the last 2100 games played, which only takes a few days to overwrite. The game url’s remain active but it’s harder to find them once they get pushed off the browse games pages.
Now that I have my data the next step is to start modelling. For the reaction prediction I’m going to use a convolutional neural network. Each reaction will have its own model because the reactions are not all equal. The smile reaction is the default, which you might think makes it the most common, but in practice love/haha/wow seem to be more popular. Finally there is the duck reaction, which costs money to actually give and is the rarest reaction. I expect the haha reaction will be more difficult to predict because even a rough stick figure can get big laughs if the context is right, and the model isn’t going to be able to get the joke. The love reaction on the other hand should be easier to model because more detail in the drawing should be noticeable and doesn’t rely on any context.
If all that goes well I will move on to creating my own captions for new drawings. Every image gets two captions so I have extra text on every image. This task is hard even for humans to perform, and is actually the core of the game. If every image was perfectly captioned every time then the game wouldn’t be as exciting. That means if my caption isn’t perfect, it still might pass as a user created caption.
Look forward to the followup to this blog post where we will see which images my model said are funny, lovely and duckworthy!
