Separate 5.56x45 NATO and .223 Remington into distinct calibers
Previously combined as a single "556-223" caliber. Now tracked separately with dedicated scraper URLs per retailer. Also fixes broken URLs for Lucky Gunner, True Shot Ammo, and Palmetto State Armory.
This commit is contained in:
parent
0340fc85be
commit
0171ef1570
13 changed files with 185 additions and 44 deletions
92
CLAUDE.md
92
CLAUDE.md
|
|
@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
|||
|
||||
## Project Overview
|
||||
|
||||
Ammoprices is a Phoenix 1.8 web application (Elixir ~> 1.15) with LiveView, Ecto/PostgreSQL, and Tailwind CSS v4 with daisyUI. Currently a fresh scaffold with no domain logic implemented yet.
|
||||
Ammoprices is a Phoenix 1.8 web application that scrapes ammunition prices from multiple online retailers, tracks price history, and displays analytics. Built with Elixir ~> 1.15, LiveView, Ecto/PostgreSQL, Tailwind CSS v4, and daisyUI.
|
||||
|
||||
## Common Commands
|
||||
|
||||
|
|
@ -27,46 +27,88 @@ mix ecto.reset # Drop and recreate database
|
|||
|
||||
## Architecture
|
||||
|
||||
### Application Structure
|
||||
### Domain Contexts
|
||||
|
||||
- **`Ammoprices.Application`** — Supervision tree: Telemetry, Repo, DNSCluster, PubSub, Endpoint
|
||||
- **`Ammoprices.Repo`** — Ecto repo using PostgreSQL adapter
|
||||
- **`Ammoprices.Mailer`** — Swoosh mailer (local adapter in dev, viewable at `/dev/mailbox`)
|
||||
**Catalog** (`lib/ammoprices/catalog/`) — Retailers, calibers, and products.
|
||||
- `Retailer`: name, slug (unique), base_url, enabled flag, last_scraped_at
|
||||
- `Caliber`: name, slug (unique), category (handgun/rifle/rimfire/shotgun), aliases array
|
||||
- `Product`: belongs_to retailer & caliber. Fields: title, url (unique per retailer), brand, grain_weight, round_count, casing (brass/steel/aluminum/alloy/composite), subsonic, in_stock. Upserted on (retailer_id, url) conflict
|
||||
|
||||
### Web Layer (`lib/ammoprices_web/`)
|
||||
**Prices** (`lib/ammoprices/prices/`) — Immutable price snapshots and analytics.
|
||||
- `PriceSnapshot`: price_cents, price_per_round_cents, in_stock, recorded_at. Append-only (no updated_at)
|
||||
- Queries: latest prices per product (with filters), daily averages, price stats (min/max/avg), cheapest per caliber
|
||||
|
||||
- **`AmmopricesWeb`** (`ammoprices_web.ex`) — Defines `use` macros for `:router`, `:controller`, `:live_view`, `:live_component`, `:html`. The `html_helpers/0` block imports `CoreComponents`, aliases `Layouts` and `Phoenix.LiveView.JS`, and sets up verified routes
|
||||
- **`AmmopricesWeb.Router`** — Browser pipeline only. Currently just `GET / → PageController.home`
|
||||
- **`AmmopricesWeb.CoreComponents`** — UI component library using daisyUI + Tailwind. Includes `flash/1`, `button/1`, `icon/1`, standard form/table components
|
||||
- **`AmmopricesWeb.Layouts`** — `app/1` layout with navbar and theme toggle (light/dark/system). Templates wrap content with `<Layouts.app flash={@flash}>`
|
||||
### Scraping Pipeline (`lib/ammoprices/scraping/`)
|
||||
|
||||
### Config
|
||||
```
|
||||
ScrapeJob (Oban, every 4h) → Runner → Scraper (per retailer) → HttpClient → Floki/JSON parse
|
||||
↓
|
||||
Upsert products → Create snapshots → PubSub broadcast
|
||||
```
|
||||
|
||||
- **`config/config.exs`** — Generator defaults: UTC timestamps, binary IDs
|
||||
- **`config/dev.exs`** — DB: `ammoprices_dev`, live reload watchers for esbuild/tailwind
|
||||
- **`config/test.exs`** — DB: `ammoprices_test`, SQL sandbox mode
|
||||
- **Scraper behaviour**: Required callbacks `retailer_slug/0`, `category_url/1`, `parse_products/1`. Optional `fetch/2` for custom HTTP (e.g., GraphQL)
|
||||
- **Runner**: Checks `function_exported?` for `fetch/2` — uses it if available, otherwise standard HTTP GET + parse flow
|
||||
- **HttpClient**: Req-based with realistic Chrome headers, retries. Has `get/1` and `post_json/3`
|
||||
- **TextDetector**: Regex detection of subsonic and casing type from product titles
|
||||
- **CaliberMatcher**: Matches products to calibers via name/alias lookup
|
||||
- **ScrapeJob**: Oban worker, queue `:scraping` (concurrency 2), 5-15 min random delay between requests in prod, 0 in test
|
||||
|
||||
### Assets (`assets/`)
|
||||
**Retailers** (7 scrapers in `lib/ammoprices/scraping/retailers/`):
|
||||
| Retailer | Type | Notes |
|
||||
|---|---|---|
|
||||
| Lucky Gunner | HTML (Floki) | Slug-to-URL mapping per caliber |
|
||||
| SG Ammo | HTML (Floki) | Standard HTML parsing |
|
||||
| Target Sports USA | HTML (Floki) | Standard HTML parsing |
|
||||
| True Shot Ammo | JSON API (Shopify) | `/collections/{handle}/products.json`, `parse_products` accepts decoded map |
|
||||
| Palmetto State Armory | HTML (Magento) | `data-price-amount` attrs, brand = first word of title |
|
||||
| Bulk Ammo | HTML (Magento) | `a.product-name`, brand via regex from title |
|
||||
| Natchez | GraphQL | Uses `fetch/2` callback, `HttpClient.post_json/3`, requires `Store: default` header |
|
||||
|
||||
- **Tailwind CSS v4** with `@import "tailwindcss"` syntax (no `tailwind.config.js`)
|
||||
- **daisyUI** plugin with custom light/dark themes defined in `app.css`
|
||||
- **esbuild** bundles `app.js` — only `app.js` and `app.css` bundles are supported
|
||||
- Vendor libs: `topbar.js`, `heroicons.js`, `daisyui.js`, `daisyui-theme.js`
|
||||
- JS hooks for LiveView use colocated hook syntax (names prefixed with `.`)
|
||||
### Web Layer
|
||||
|
||||
### Test Support (`test/support/`)
|
||||
**Routes:**
|
||||
- `GET /` → `HomeLive` — Calibers grouped by category with cheapest CPR per caliber
|
||||
- `GET /calibers/:slug` → `CaliberLive.Show` — Price table, Chart.js chart, filter bar, stats
|
||||
|
||||
- **`ConnCase`** — For controller/LiveView tests, sets up conn and DB sandbox
|
||||
- **`DataCase`** — For context/schema tests, provides `errors_on/1` helper
|
||||
- **`LazyHTML`** — Available in tests for HTML assertion selectors
|
||||
**CaliberLive.Show filters:** in_stock (toggle), casing (mutually exclusive), grain_weight (mutually exclusive), subsonic (toggle, only shown when subsonic products exist). Chart range: 7d/30d/90d/1y/all.
|
||||
|
||||
**Real-time:** PubSub on `"prices:updated"` topic. CaliberLive.Show subscribes on connect; broadcasts trigger reload of filter options, product stream, chart, and stats.
|
||||
|
||||
**Components:**
|
||||
- `PriceComponents`: `price_per_round/1` (formats cents as $0.38), `stock_badge/1`, `retailer_link/1`
|
||||
- `CoreComponents`: Standard Phoenix component library (daisyUI + Tailwind)
|
||||
|
||||
### Assets
|
||||
|
||||
- **Tailwind CSS v4** — `@import "tailwindcss"` syntax in `app.css`, no `tailwind.config.js`
|
||||
- **daisyUI** — Custom light/dark themes defined in `app.css`
|
||||
- **Chart.js** — Vendored in `assets/vendor/chart.js`, used via colocated `.PriceChart` hook
|
||||
- **esbuild** — Bundles `app.js` only. `@` alias maps to `assets/vendor/` for imports
|
||||
- Colocated hook imports **must** use `@/vendor/...` alias, NOT relative paths (hooks compile to `_build/`)
|
||||
|
||||
### Database
|
||||
|
||||
- Binary UUIDs, UTC timestamps throughout
|
||||
- Products use `on_conflict: {:replace, [fields...]}` for upserts on (retailer_id, url)
|
||||
- Product filter indexes: (caliber_id, in_stock), (caliber_id, casing), (caliber_id, grain_weight), (caliber_id, subsonic)
|
||||
- Retailers and calibers seeded via database migration (not seeds.exs)
|
||||
|
||||
### Test Infrastructure
|
||||
|
||||
- **Fixtures**: `test/support/fixtures.ex` — factory functions (retailer_fixture, caliber_fixture, product_fixture, snapshot_fixture)
|
||||
- **HTML/JSON fixtures**: `test/fixtures/{retailer_name}/` — scraped response samples for each retailer
|
||||
- **HTTP stubbing**: `Req.Test` with plug config in test.exs
|
||||
- **Oban**: `:manual` mode in test (jobs don't auto-run)
|
||||
- **Scrape delay**: `{0, 0}` in test config
|
||||
|
||||
## Key Conventions
|
||||
|
||||
- **HTTP client**: Use `Req` (already included). Never use HTTPoison, Tesla, or `:httpc`
|
||||
- **Prices**: Display in dollar format ($0.38/rd), stored as integer cents internally
|
||||
- **LiveView templates**: Always begin with `<Layouts.app flash={@flash}>` wrapper
|
||||
- **Collections**: Always use LiveView streams, never assign raw lists
|
||||
- **Icons**: Use `<.icon name="hero-x-mark" />` component, never Heroicons modules
|
||||
- **Forms**: Always use `to_form/2` assigned in LiveView, access via `@form[:field]` in templates
|
||||
- **Collections**: Always use LiveView streams, never assign raw lists
|
||||
- **JS in templates**: Use colocated hook `<script :type={Phoenix.LiveView.ColocatedHook}>` with `.` prefixed names, never raw `<script>` tags
|
||||
- **No inline scripts**: Import vendor deps into `app.js`/`app.css`, no external `src`/`href` in layouts
|
||||
- **Tailwind classes**: Use list syntax `class={["base", condition && "extra"]}` for conditional classes
|
||||
|
|
|
|||
|
|
@ -12,7 +12,8 @@ defmodule Ammoprices.Scraping.Retailers.BulkAmmo do
|
|||
"38-special" => "/handgun/bulk-38-special-ammo",
|
||||
"357-magnum" => "/handgun/bulk-357-mag-ammo",
|
||||
"10mm-auto" => "/handgun/bulk-10mm-ammo",
|
||||
"556-223" => "/rifle/bulk-223-remington-ammo",
|
||||
"223-rem" => "/rifle/bulk-.223-ammo",
|
||||
"556-nato" => "/rifle/bulk-5.56x45-ammo",
|
||||
"308-win" => "/rifle/bulk-308-ammo",
|
||||
"762x39" => "/rifle/bulk-7.62x39-ammo",
|
||||
"30-06" => "/rifle/bulk-30-06-ammo",
|
||||
|
|
|
|||
|
|
@ -12,7 +12,8 @@ defmodule Ammoprices.Scraping.Retailers.LuckyGunner do
|
|||
"38-special" => "/handgun/38-special-ammo",
|
||||
"357-magnum" => "/handgun/357-mag-ammo",
|
||||
"10mm-auto" => "/handgun/10mm-ammo",
|
||||
"556-223" => "/rifle/223-ammo",
|
||||
"223-rem" => "/rifle/223-remington-ammo",
|
||||
"556-nato" => "/rifle/5.56x45-ammo",
|
||||
"308-win" => "/rifle/308-ammo",
|
||||
"762x39" => "/rifle/7.62x39-ammo",
|
||||
"30-06" => "/rifle/30-06-ammo",
|
||||
|
|
|
|||
|
|
@ -13,7 +13,8 @@ defmodule Ammoprices.Scraping.Retailers.Natchez do
|
|||
"38-special" => "38 Spl",
|
||||
"357-magnum" => "357 Mag",
|
||||
"10mm-auto" => "10mm Auto",
|
||||
"556-223" => "223 Rem",
|
||||
"223-rem" => "223 Rem",
|
||||
"556-nato" => "5.56mm",
|
||||
"308-win" => "308 Win",
|
||||
"762x39" => "7.62x39mm",
|
||||
"30-06" => "30-06 Sprg",
|
||||
|
|
|
|||
|
|
@ -12,7 +12,8 @@ defmodule Ammoprices.Scraping.Retailers.PalmettoStateArmory do
|
|||
"38-special" => "/38-special-ammo.html",
|
||||
"357-magnum" => "/357-magnum-ammo.html",
|
||||
"10mm-auto" => "/10mm-auto-ammo.html",
|
||||
"556-223" => "/223-remington-ammo.html",
|
||||
"223-rem" => "/223-ammo.html",
|
||||
"556-nato" => "/5-56-ammo.html",
|
||||
"308-win" => "/308-winchester-ammo.html",
|
||||
"762x39" => "/7-62x39mm-ammo.html",
|
||||
"30-06" => "/30-06-springfield-ammo.html",
|
||||
|
|
|
|||
|
|
@ -12,7 +12,8 @@ defmodule Ammoprices.Scraping.Retailers.SgAmmo do
|
|||
"38-special" => "/catalog/pistol-ammo-for-sale/38-special-ammo",
|
||||
"357-magnum" => "/catalog/pistol-ammo-for-sale/357-magnum-ammo",
|
||||
"10mm-auto" => "/catalog/pistol-ammo-for-sale/10mm-ammo",
|
||||
"556-223" => "/catalog/rifle-ammo-for-sale/223-556mm-ammo",
|
||||
"223-rem" => "/catalog/rifle-ammo-for-sale/223-556mm-ammo",
|
||||
"556-nato" => "/catalog/rifle-ammo-for-sale/223-556mm-ammo",
|
||||
"308-win" => "/catalog/rifle-ammo-for-sale/308-762mm-ammo",
|
||||
"762x39" => "/catalog/rifle-ammo-for-sale/762x39-ammo",
|
||||
"30-06" => "/catalog/rifle-ammo-for-sale/30-06-ammo",
|
||||
|
|
|
|||
|
|
@ -12,7 +12,8 @@ defmodule Ammoprices.Scraping.Retailers.TargetSportsUsa do
|
|||
"38-special" => "/38-special-ammo-c-56.aspx",
|
||||
"357-magnum" => "/357-magnum-ammo-c-57.aspx",
|
||||
"10mm-auto" => "/10mm-auto-ammo-c-60.aspx",
|
||||
"556-223" => "/223-remington-ammo-c-83.aspx",
|
||||
"223-rem" => "/223-remington-ammo-c-83.aspx",
|
||||
"556-nato" => "/556mm-nato-ammo-c-2719.aspx",
|
||||
"308-win" => "/308-winchester-ammo-c-101.aspx",
|
||||
"762x39" => "/762x39mm-ammo-c-108.aspx",
|
||||
"30-06" => "/30-06-springfield-ammo-c-105.aspx",
|
||||
|
|
|
|||
|
|
@ -12,7 +12,8 @@ defmodule Ammoprices.Scraping.Retailers.TrueShotAmmo do
|
|||
"38-special" => "ammunition-pistol-ammo-38-special",
|
||||
"357-magnum" => "ammunition-pistol-ammo-357-magnum",
|
||||
"10mm-auto" => "ammunition-pistol-ammo-10mm",
|
||||
"556-223" => "ammunition-rifle-ammo-223-5-56",
|
||||
"223-rem" => "ammunition-rifle-ammo-223-rem",
|
||||
"556-nato" => "ammunition-rifle-ammo-5-56x45mm",
|
||||
"308-win" => "ammunition-rifle-ammo-308-7-62x51",
|
||||
"762x39" => "ammunition-rifle-ammo-7-62x39",
|
||||
"300-blackout" => "ammunition-rifle-ammo-300-blackout",
|
||||
|
|
|
|||
|
|
@ -0,0 +1,71 @@
|
|||
defmodule Ammoprices.Repo.Migrations.Split556223IntoSeparateCalibers do
|
||||
use Ecto.Migration
|
||||
|
||||
def up do
|
||||
now = DateTime.utc_now() |> DateTime.truncate(:second) |> DateTime.to_naive()
|
||||
|
||||
# Insert .223 Remington
|
||||
execute("""
|
||||
INSERT INTO calibers (id, name, slug, category, aliases, inserted_at, updated_at)
|
||||
VALUES (gen_random_uuid(), '.223 Remington', '223-rem', 'rifle',
|
||||
'{"223 remington",".223 rem",".223 remington",".223"}',
|
||||
'#{now}', '#{now}')
|
||||
ON CONFLICT (slug) DO NOTHING
|
||||
""")
|
||||
|
||||
# Insert 5.56x45 NATO
|
||||
execute("""
|
||||
INSERT INTO calibers (id, name, slug, category, aliases, inserted_at, updated_at)
|
||||
VALUES (gen_random_uuid(), '5.56x45 NATO', '556-nato', 'rifle',
|
||||
'{"5.56x45","5.56 nato","5.56x45 nato","5.56"}',
|
||||
'#{now}', '#{now}')
|
||||
ON CONFLICT (slug) DO NOTHING
|
||||
""")
|
||||
|
||||
# Reassign products with "5.56" in the title to 556-nato
|
||||
execute("""
|
||||
UPDATE products
|
||||
SET caliber_id = (SELECT id FROM calibers WHERE slug = '556-nato'),
|
||||
updated_at = '#{now}'
|
||||
WHERE caliber_id = (SELECT id FROM calibers WHERE slug = '556-223')
|
||||
AND (title ILIKE '%5.56%' OR title ILIKE '%5.56x45%')
|
||||
""")
|
||||
|
||||
# Reassign remaining products (assumed .223) to 223-rem
|
||||
execute("""
|
||||
UPDATE products
|
||||
SET caliber_id = (SELECT id FROM calibers WHERE slug = '223-rem'),
|
||||
updated_at = '#{now}'
|
||||
WHERE caliber_id = (SELECT id FROM calibers WHERE slug = '556-223')
|
||||
""")
|
||||
|
||||
# Delete the old combined caliber
|
||||
execute("DELETE FROM calibers WHERE slug = '556-223'")
|
||||
end
|
||||
|
||||
def down do
|
||||
now = DateTime.utc_now() |> DateTime.truncate(:second) |> DateTime.to_naive()
|
||||
|
||||
# Re-insert the combined caliber
|
||||
execute("""
|
||||
INSERT INTO calibers (id, name, slug, category, aliases, inserted_at, updated_at)
|
||||
VALUES (gen_random_uuid(), '5.56x45 / .223 Rem', '556-223', 'rifle',
|
||||
'{"5.56","5.56x45",".223",".223 rem","223 remington","5.56 nato"}',
|
||||
'#{now}', '#{now}')
|
||||
ON CONFLICT (slug) DO NOTHING
|
||||
""")
|
||||
|
||||
# Move products back to combined caliber
|
||||
execute("""
|
||||
UPDATE products
|
||||
SET caliber_id = (SELECT id FROM calibers WHERE slug = '556-223'),
|
||||
updated_at = '#{now}'
|
||||
WHERE caliber_id IN (
|
||||
SELECT id FROM calibers WHERE slug IN ('223-rem', '556-nato')
|
||||
)
|
||||
""")
|
||||
|
||||
# Delete the split calibers
|
||||
execute("DELETE FROM calibers WHERE slug IN ('223-rem', '556-nato')")
|
||||
end
|
||||
end
|
||||
|
|
@ -45,10 +45,16 @@ calibers = [
|
|||
|
||||
# Rifle
|
||||
%{
|
||||
name: "5.56x45 / .223 Rem",
|
||||
slug: "556-223",
|
||||
name: ".223 Remington",
|
||||
slug: "223-rem",
|
||||
category: "rifle",
|
||||
aliases: ["5.56", "5.56x45", ".223", ".223 rem", "223 remington", "5.56 nato"]
|
||||
aliases: ["223 remington", ".223 rem", ".223 remington", ".223"]
|
||||
},
|
||||
%{
|
||||
name: "5.56x45 NATO",
|
||||
slug: "556-nato",
|
||||
category: "rifle",
|
||||
aliases: ["5.56x45", "5.56 nato", "5.56x45 nato", "5.56"]
|
||||
},
|
||||
%{
|
||||
name: ".308 Win / 7.62x51",
|
||||
|
|
|
|||
|
|
@ -17,9 +17,14 @@ defmodule Ammoprices.Scraping.Retailers.BulkAmmoTest do
|
|||
assert BulkAmmo.category_url(caliber) == "/handgun/bulk-9mm-ammo"
|
||||
end
|
||||
|
||||
test "maps 556-223 caliber to correct URL" do
|
||||
caliber = %{slug: "556-223"}
|
||||
assert BulkAmmo.category_url(caliber) == "/rifle/bulk-223-remington-ammo"
|
||||
test "maps 223-rem caliber to correct URL" do
|
||||
caliber = %{slug: "223-rem"}
|
||||
assert BulkAmmo.category_url(caliber) == "/rifle/bulk-.223-ammo"
|
||||
end
|
||||
|
||||
test "maps 556-nato caliber to correct URL" do
|
||||
caliber = %{slug: "556-nato"}
|
||||
assert BulkAmmo.category_url(caliber) == "/rifle/bulk-5.56x45-ammo"
|
||||
end
|
||||
|
||||
test "returns nil for unmapped caliber" do
|
||||
|
|
|
|||
|
|
@ -17,9 +17,14 @@ defmodule Ammoprices.Scraping.Retailers.PalmettoStateArmoryTest do
|
|||
assert PalmettoStateArmory.category_url(caliber) == "/9mm-ammo.html"
|
||||
end
|
||||
|
||||
test "maps 556-223 caliber to correct URL" do
|
||||
caliber = %{slug: "556-223"}
|
||||
assert PalmettoStateArmory.category_url(caliber) == "/223-remington-ammo.html"
|
||||
test "maps 223-rem caliber to correct URL" do
|
||||
caliber = %{slug: "223-rem"}
|
||||
assert PalmettoStateArmory.category_url(caliber) == "/223-ammo.html"
|
||||
end
|
||||
|
||||
test "maps 556-nato caliber to correct URL" do
|
||||
caliber = %{slug: "556-nato"}
|
||||
assert PalmettoStateArmory.category_url(caliber) == "/5-56-ammo.html"
|
||||
end
|
||||
|
||||
test "returns nil for unmapped caliber" do
|
||||
|
|
|
|||
|
|
@ -17,9 +17,14 @@ defmodule Ammoprices.Scraping.Retailers.TrueShotAmmoTest do
|
|||
assert TrueShotAmmo.category_url(caliber) == "/collections/ammunition-pistol-ammo-9mm/products.json"
|
||||
end
|
||||
|
||||
test "maps 556-223 caliber to correct URL" do
|
||||
caliber = %{slug: "556-223"}
|
||||
assert TrueShotAmmo.category_url(caliber) == "/collections/ammunition-rifle-ammo-223-5-56/products.json"
|
||||
test "maps 223-rem caliber to correct URL" do
|
||||
caliber = %{slug: "223-rem"}
|
||||
assert TrueShotAmmo.category_url(caliber) == "/collections/ammunition-rifle-ammo-223-rem/products.json"
|
||||
end
|
||||
|
||||
test "maps 556-nato caliber to correct URL" do
|
||||
caliber = %{slug: "556-nato"}
|
||||
assert TrueShotAmmo.category_url(caliber) == "/collections/ammunition-rifle-ammo-5-56x45mm/products.json"
|
||||
end
|
||||
|
||||
test "returns nil for unmapped caliber" do
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue