Leafsnap is a project I've been working on for about 2 years and we finally launched this week! It's a free app for the iPhone (with an iPad version coming next week and an Android version later in the summer) that contains very high-quality images and descriptions of the tree species of the Northeast US. It also has an automatic-recognition feature that uses computer vision to help users identify trees by taking a photo of a leaf.
This project is a joint collaboration between the computer science departments at Columbia University and the University of Maryland, and botanists at the Smithsonian Institution.
Please let me know what you think! Also, I'd be happy to answer any questions about the app or the recognition technology behind it.
Congratulations and well done. I'm just downloading it now. My first thoughts were if it was able to identify the 320+ species of native trees we have here in NZ. My second thought was to wonder if it can be trained by the users to learn new species? (I'm still waiting for the app to finish downloading so have not tried it yet). It would be great to expand your knowledge base by having user contributions, perhaps in a curated or weighted submission process.
Thanks! We currently have coverage of about 200 species from the northeast US. We're focusing on expanding the coverage to the rest of the US (~500 more species)...and then the world!
We definitely want to take advantage of user contributions in some way, but we're still trying to figure out the best thing to do. Also, the app contains several very high-quality images of each species taken in a particular way by a talented group called FindingSpecies, and to keep the visual integrity of the app, we would want very similar photos for all species that we add. This might be difficult to outsource to the crowd.
It's not something that can be readily outsourced to "the crowd" in general but it is something that can achieved using a hierarchy of interested/invested self checking power users.
It's a similar problem to that faced by, say, open source projects that aim to maintain a certain level of code quality and style matching in contributed source code.
I think this is awesome, I've been waiting for this technology for a long time. It's one of the first CV+Mobile apps I imagined having some day, and when you release the android version I'll finally get my wish! Thank you!
Question: Any thoughts to partnering with IQEngines to bring their adaptive image recognition engine and database together with yours?
We're computer vision people ourselves, so it's not an immediate priority =) Also, there was a lot of computer vision work we had to do specifically for this project to get the recognition to work. See elsewhere for my description of that.
I think it is an outstanding idea, CONGRATS for having done that!!.
It solves a real problem, maybe you can extend it to mushrooms with HUGE warnings(you don't want people eating something poisonous a computer recognized).
I have this little tree in my yard, I do not know for sure what it is, it seems it is a cherry tree but I'm dubious.
Mega Congrats! Any chance of developing a similar app (paid would be fine) for identifying wildflowers? I have thousands of pictures of wilderness areas across the Western US, and I get tired of saying "and here's an example of... a yellow flower I saw a lot of that day" :-)
We've thought about flowers, but since they are present on trees for a much shorter time than leaves, it would be less useful overall. Since we're somewhat constrained in our resources (most of the work has been done on a volunteer basis), we're focusing on expanding coverage first rather than integrating additional cues for recognition (which will be a very tricky computer vision problem that will likely require a lot of effort).
I have a family member that taught dendrology for 30 years.
He amazes me by being able to look at a leaf, bark or chunk of wood and tell what it is near instantly. If accurate, this is the sort of thing that will change the teaching process. I will tell him about it.
1. Segment the leaf from the background. Even though we require users to place the leaf on a white background, this is still a very tough problem because of lighting, shadows, viewpoint, blur, focus, etc. We've tried all published segmentation algorithms and nothing works. We wrote our own, using EM on the HSV colorspace as the basis for it (plus lots of heuristics for leaves). It works okay, but it definitely needs the most work.
2. Extract some features from the contour of the leaf. We use histograms of curvature over scale. The idea is that some leaves have the same rough shape, but differ at a fine scale (e.g., serrated vs. smooth leaves). Others are similar at a fine-scale but look quite different overall. To distinguish all of these cases, we estimate "curvature at a scale" at each point on the contour and build a histogram of curvature values. We then repeat this at coarser and coarser scales and concatenate all histograms together to get a large feature vector.
3. Recognition/matching: We use a simple nearest-neighbors classifier to match the feature vectors to our ground-truth database of labeled leaves. We return the top 20 ranked species or so.
A whole other approach, but I wonder if segmentation might be easier using a video of a leaf moving against a stationary background. The leaf is the part that is changing.
I've often wondered about identifying local trees; I've looked up site of trees, but often leaves seem very similar to several different trees.
Great to have it for other countries, and an edible plant guide.
Color is very tricky to use, because you have to worry about white-balancing, illumination conditions, and of course leaves have a large variation in color over the course of their life.
This is really cool! I like it because I think it will actually encourage kids especially, and generally everyone, to take an interest in nature and the plants around them. Also, I'm sure it could be useful in identifying dangerous and safe plants. Keep up the good work!
Leafsnap is a project I've been working on for about 2 years and we finally launched this week! It's a free app for the iPhone (with an iPad version coming next week and an Android version later in the summer) that contains very high-quality images and descriptions of the tree species of the Northeast US. It also has an automatic-recognition feature that uses computer vision to help users identify trees by taking a photo of a leaf.
This project is a joint collaboration between the computer science departments at Columbia University and the University of Maryland, and botanists at the Smithsonian Institution.
Please let me know what you think! Also, I'd be happy to answer any questions about the app or the recognition technology behind it.