Mineral image identifier

TL;DR

A collaboration project on a web app mineralid which helps people identify mineral type using an uploaded image and other mineral features.

How we came about this idea?

A friend of mine, say Sam, discovered that there are dedicated communities, e.g., whatisthisrock, which help people identify the mineral type of rocks that they might have found during hikes or other adventures. This seemed like an ideal problem for deep learning!

Some of the images that we have received, from left to right: serpentine, labradorite, brookite. (click to enlarge)

So we ended up building a web app to tackle this problem: mineralid. To separate the work, I built the machine learning part which consists of a JSON API and my friend built the web app that communicates with the API and includes human input to drastically improve the predictions.

Tech stack

• Tensorflow: we chose tensorflow over other frameworks due to the ease of launching a predictive model in a production environment using tensorflow serving (though the client part for tensorflow serving was not exactly a walk in the park).
• Docker: I use docker to manage my everyday working environment but this was the first time for me to use docker-compose to orchestrate multiple services in a production environment. This was a key component to being able to get our product off the ground quickly.
• Django: used to build the JSON API.
• Nginx: reverse proxy server so that the JSON API is scalable, we also used gunicorn as the Python WSGI between nginx and the django app.
• Ruby + postgres: cannot comment much more on this since Sam built the web app…

Domain challenges

Other than the technical challenges of launching a machine learning product in production environment, we found that the task of identifying minerals is non-trivial.

At the beginning, I thought it was a straight forward job of throwing a bunch of images at a neural network and voila! >99% precision and recall. But it turns out not to be the case. I will probably write some other posts on this but in summary, we faced three main challenges:

Photos illustrating challenges of mineral identification in the wild.

• Not enough data: we only managed to get sufficient data, i.e., >500 images per class, for around ~1000 mineral classes. This meant that our image classifier would not work on rare minerals like moissanite (the first black rock).
• Multi-label problem: mineral identification outside the lab is inherently a multi-label problem since minerals might not naturally form in isolation of other minerals. For example, the gold image (second photo) that we received is actually mixed with some kind of quartz, so does the user want to know the golden mineral or the white mineral?
• Noise: anything can happen in real life… even if the client really uploaded an image of a mineral, it can be in the presence of other minerals or even a person!