Turning Handwritten Notes into a Pre-filled Cart 🛒

In the age of convenience, we all want a seamless shopping experience. Inspired by Swiggy Instamart’s new “Shopping Lists” feature, I set out to create a similar functionality: upload a handwritten shopping list, extract the items from it, and generate a cart with recommended products. This post dives into the tech behind this feature and how it can make your shopping journey smoother than ever.

Why Build a Shopping List Feature?

Imagine you scribbled a quick shopping list on paper. Instead of manually searching for each item online, you could snap a photo, upload it, and instantly see a cart with all the items filled in. For busy users, this feature saves time and reduces friction, while for platforms, it enhances user engagement and convenience. Let’s break down how I developed it using Python, Tesseract OCR, and Streamlit.

Step 1: Extracting Text from Handwritten Images

The first hurdle was to turn handwritten text into something a computer can understand. For this, I used Optical Character Recognition (OCR) with Tesseract, a powerful open-source OCR engine. Tesseract can identify and convert characters from an image to text, whether typed or handwritten.

Using the pytesseract library, I wrote a small function to load and process the image:

import pytesseract
from PIL import Image

def extract_text_from_image(image_path):
    image = Image.open(image_path)
    return pytesseract.image_to_string(image)

Step 2: Parsing the Extracted Text

OCR results can be messy, especially with handwritten text. After extracting text, I needed to clean it up — removing any extraneous text and standardizing item names. My goal was to parse each line to isolate product names and quantities for a clean, usable shopping list.

# Example parsing code
shopping_list_text = extract_text_from_image("shopping_list.png")
items = [line for line in shopping_list_text.splitlines() if line.strip() != ""]

At this point, I had a structured list of items with quantities, such as:

Paneer 2kg
Basmati Rice 1kg
Chicken Boneless 1kg

Step 3: Generating Product Recommendations

With the items extracted, the next challenge was to provide product recommendations for each item. For this, I created a dictionary with recommended products based on each item name. This dictionary includes details like product name, price, and an image URL for each item.

recommended_products = {
    "Paneer": [
        {"name": "Amul Fresh Paneer", "price": "₹275", "image": "https://link-to-image1"},
        {"name": "Milky Mist Paneer", "price": "₹250", "image": "https://link-to-image2"}
    ],
    # Additional items...
}

This allowed me to dynamically pull up relevant options for each item on the list, providing users with a choice of brands and quantities similar to a real shopping experience.

Step 4: Creating a User Interface with Streamlit

To build an interactive UI, I used Streamlit, an open-source app framework for data science. Streamlit enabled me to create a simple, intuitive web app where users can upload an image, view extracted items, and see recommendations in a familiar format.

Here’s how the layout works:

1. Upload Section: Users upload an image of their handwritten shopping list.
2. Extracted List: A checklist displays the extracted items, allowing users to confirm or deselect items.
3. Recommendations: For each item, a row of recommended products is displayed, each with an “Add” button to simulate adding the item to a cart.

import streamlit as st

# Display extracted items
st.title("Your Shopping List")
for item in items:
    st.checkbox(item, value=True)

# Show recommendations
st.subheader("Shopping Cart Recommendations")
for item in items:
    st.write(f"Recommendations for {item}:")
    for recommendation in recommended_products.get(item.split()[0], []):
        st.image(recommendation["image"], width=100)
        st.write(f"{recommendation['name']} - {recommendation['price']}")
        st.button("Add", key=f"{recommendation['name']}")

With this setup, I could mimic a shopping platform where users select items based on OCR results and view multiple brand options per product.

check the complete code here :

https://github.com/shankarsharma8089/Turning-Handwritten-Notes-into-a-Pre-filled-Cart/tree/main

Final Thoughts

Creating this feature was not only a fun exercise but also a practical solution with real-world applications. By automating shopping list conversions, we can provide users with a smoother, faster way to prepare their carts and streamline their shopping experience.