S
Sanya Kapoor
Guest
Most price comparison sites do a decent job of finding prices. The problem is figuring out whether those prices are actually comparable.
You can pull prices from Amazon, Walmart, and Target, put them in a table, and sort from cheapest to most expensive. Everything might look correct, but that doesn't necessarily mean the cheapest product is the best deal.
Take diapers, for example. If someone is shopping for size 4 diapers, they probably care more about the cost per diaper than the price of the entire box. A box of 120 diapers for $39.99 is a better deal than 74 diapers for $27.49, even though the second box is cheaper.
The math is simple. The difficult part is consistently figuring out how many diapers are actually in each package.
Retailers don't follow a consistent format for product titles. Even products in the same category can describe their quantities in completely different ways.
For example:
At first glance, these look like three similar products. But double rolls, triple rolls, and mega rolls aren't standardized measurements. One brand's double roll might have fewer sheets than another brand's regular roll.
Even comparing the cost per sheet isn't always accurate because sheet sizes can vary.
Ideally, you'd compare the cost per square foot, but that requires knowing the number of sheets and the dimensions of each sheet. Some listings provide that information, while others don't.
This is one of the challenges behind our paper towel price comparison. Collecting the prices is relatively straightforward. Getting all the products into a format where they can be compared fairly takes much more work.
We ran into similar issues when comparing whey protein products.
One manufacturer lists a 5 lb container. Another uses 2.27 kg. A third advertises 74 servings, but you have to look at the nutrition label to find out how much protein is actually in each serving.
For example, a 30 g scoop might contain only 22 g of protein. The remaining weight comes from other ingredients.
If you compare these products based on price per pound, you're including everything in the container, not just the protein. A cheaper product with less actual protein could end up ranking higher.
A more useful comparison is the price per gram of protein.
That means extracting the serving size, protein per serving, and total number of servings. Sometimes that information is in the product description, but often it's only available on the nutrition label as an image.
When we worked on our whey protein comparison across Amazon, Walmart, and Target, we found that comparing actual protein content instead of total container weight changed which products offered the best value.
The process we've been working with is fairly straightforward in theory.
The fourth step is particularly important for supplements.
Someone looking for fish oil with at least 1,000 mg of EPA and DHA might see a bottle advertising 1,200 mg of fish oil. But that doesn't mean it contains 1,200 mg of EPA and DHA. The actual combined amount could be much lower.
If the system doesn't distinguish between total fish oil and its active ingredients, it can recommend a product that doesn't meet the shopper's requirements.
This is where LLMs have been useful. Traditional parsers built around regular expressions can handle predictable formats, but product listings are rarely predictable.
For example, given a title like 6 Double Rolls = 12 Regular Rolls, an LLM can extract something like:
{"rolls": 6, "regular_equivalent": 12}
That makes it easier to handle variations without writing a separate parsing rule for every retailer or brand.
Of course, the model's output still needs validation. We use the model to interpret the product information, while the actual calculations and consistency checks should remain deterministic.
After all that processing, what the shopper sees is usually a pretty simple table showing products ranked by cost per diaper, cost per gram of protein, or another relevant unit.
Our diaper price comparison by size is a good example. There's nothing particularly complicated about the final presentation. The important part is making sure every product is being compared using the same measurement.
And that's something we've come to appreciate while working on Popgot. A lot of the work involved in building a useful product search experience has less to do with finding products and more to do with cleaning up the information retailers provide.
Prices are easy enough to collect. Figuring out what those prices actually represent is where most of the engineering happens.
This story was distributed as a release by Sanya Kapoor under HackerNoon’s Business Blogging Program
You can pull prices from Amazon, Walmart, and Target, put them in a table, and sort from cheapest to most expensive. Everything might look correct, but that doesn't necessarily mean the cheapest product is the best deal.
Take diapers, for example. If someone is shopping for size 4 diapers, they probably care more about the cost per diaper than the price of the entire box. A box of 120 diapers for $39.99 is a better deal than 74 diapers for $27.49, even though the second box is cheaper.
The math is simple. The difficult part is consistently figuring out how many diapers are actually in each package.
Product titles make things complicated
Retailers don't follow a consistent format for product titles. Even products in the same category can describe their quantities in completely different ways.
For example:
- Brand X Paper Towels, 6 Double Rolls = 12 Regular Rolls
- Brand Y Select-A-Size, 8 Triple Rolls, 110 Sheets per Roll
- Brand Z Paper Towels Pack of 12 (Mega)
At first glance, these look like three similar products. But double rolls, triple rolls, and mega rolls aren't standardized measurements. One brand's double roll might have fewer sheets than another brand's regular roll.
Even comparing the cost per sheet isn't always accurate because sheet sizes can vary.
Ideally, you'd compare the cost per square foot, but that requires knowing the number of sheets and the dimensions of each sheet. Some listings provide that information, while others don't.
This is one of the challenges behind our paper towel price comparison. Collecting the prices is relatively straightforward. Getting all the products into a format where they can be compared fairly takes much more work.
Supplements are even harder to compare
We ran into similar issues when comparing whey protein products.
One manufacturer lists a 5 lb container. Another uses 2.27 kg. A third advertises 74 servings, but you have to look at the nutrition label to find out how much protein is actually in each serving.
For example, a 30 g scoop might contain only 22 g of protein. The remaining weight comes from other ingredients.
If you compare these products based on price per pound, you're including everything in the container, not just the protein. A cheaper product with less actual protein could end up ranking higher.
A more useful comparison is the price per gram of protein.
That means extracting the serving size, protein per serving, and total number of servings. Sometimes that information is in the product description, but often it's only available on the nutrition label as an image.
When we worked on our whey protein comparison across Amazon, Walmart, and Target, we found that comparing actual protein content instead of total container weight changed which products offered the best value.
How we handle the data
The process we've been working with is fairly straightforward in theory.
- Collect the product title, price, description, available attributes, and nutrition or packaging information.
- Extract quantities such as package count, total weight, serving size, and active ingredients.
- Convert everything into consistent units, accounting for multi-packs and different measurement systems.
- Check whether the product actually meets the shopper's requirements.
- Calculate the relevant unit price and rank the products.
The fourth step is particularly important for supplements.
Someone looking for fish oil with at least 1,000 mg of EPA and DHA might see a bottle advertising 1,200 mg of fish oil. But that doesn't mean it contains 1,200 mg of EPA and DHA. The actual combined amount could be much lower.
If the system doesn't distinguish between total fish oil and its active ingredients, it can recommend a product that doesn't meet the shopper's requirements.
This is where LLMs have been useful. Traditional parsers built around regular expressions can handle predictable formats, but product listings are rarely predictable.
For example, given a title like 6 Double Rolls = 12 Regular Rolls, an LLM can extract something like:
{"rolls": 6, "regular_equivalent": 12}
That makes it easier to handle variations without writing a separate parsing rule for every retailer or brand.
Of course, the model's output still needs validation. We use the model to interpret the product information, while the actual calculations and consistency checks should remain deterministic.
The final result is just a table
After all that processing, what the shopper sees is usually a pretty simple table showing products ranked by cost per diaper, cost per gram of protein, or another relevant unit.
Our diaper price comparison by size is a good example. There's nothing particularly complicated about the final presentation. The important part is making sure every product is being compared using the same measurement.
And that's something we've come to appreciate while working on Popgot. A lot of the work involved in building a useful product search experience has less to do with finding products and more to do with cleaning up the information retailers provide.
Prices are easy enough to collect. Figuring out what those prices actually represent is where most of the engineering happens.
This story was distributed as a release by Sanya Kapoor under HackerNoon’s Business Blogging Program