Dynamic sitemap for phoenix apps

Below is a deep-dive blog post on building and integrating a custom Phoenix sitemap generator using your NctWebsiteWeb.SitemapController. It covers why you’d want a dynamic sitemap, how each part of the controller works, and suggestions for enhancement and production hardening.

Why a Dynamic Sitemap?

A well-formed sitemap.xml is a best practice for SEO: it helps search engines discover pages, understand update cadence, and prioritize crawling. While you can hand-craft a static sitemap, keeping it in sync with a growing blog or a changing site becomes tedious. By generating your sitemap on the fly:

• Freshness: Newly published blog posts and pages instantly show up.
• Reduced Ops: No manual updates or build-pipeline steps.
• Flexibility: You can filter out admin routes, include only certain resource types, or even add custom metadata per URL.

Your SitemapController uses Ecto and the Router’s route list to build a complete, up-to-date sitemap whenever it’s requested.

Controller Overview

defmodule NctWebsiteWeb.SitemapController do
  use NctWebsiteWeb, :controller
  alias NctWebsite.Blogs.Blog
  alias NctWebsite.Repo

  @doc """
  Renders a dynamically generated sitemap.xml by fetching all blog posts.
  """
  def sitemap(conn, _params) do
    posts = Repo.all(Blog)
    xml   = build_sitemap(posts)

    conn
    |> put_resp_content_type("application/xml")
    |> send_resp(200, xml)
  end

  # … private helpers …
end

• Route: You’d map this under something like get "/sitemap.xml", SitemapController, :sitemap in your Phoenix router.
• Data fetch: Repo.all(Blog) loads every post—simple and effective for small-to-medium catalogs.
• Response: Sets content_type to application/xml and streams back the generated XML.

Building the XML Document

defp build_sitemap(posts) do
  header        = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"
  urlset_open   = "<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\">\n"
  urlset_close  = "</urlset>"

  urls =
    posts
    |> Enum.map(&url_entry/1)
    |> then(&(base_routes() ++ &1))
    |> Enum.join("\n")

  [header, urlset_open, urls, "\n", urlset_close]
  |> IO.iodata_to_binary()
end

1. XML prolog & namespace: Establishes the standard sitemap schema.
2. base_routes(): Inserts your “static” or non-blog pages first (homepage, about, contact, etc.).
3. url_entry/1: Transforms each blog post into a <url> block.
4. IO data: Concatenates iodata lists efficiently into one binary for the response.

Crafting Each <url> Entry

defp url_entry(%Blog{slug: slug, updated_at: updated_at}) do
  loc     = NctWebsiteWeb.Endpoint.url <> "/blogs/#{slug}"
  lastmod = updated_at |> DateTime.to_date() |> Date.to_iso8601()

  ~s(  <url>\n    <loc>#{loc}</loc>\n    <lastmod>#{lastmod}</lastmod>\n    <changefreq>weekly</changefreq>\n    <priority>0.8</priority>\n  </url>)
end

• <loc>: Absolute URL to the blog post.
• <lastmod>: ISO-formatted date of the last update—crucial for search engines to know which pages changed.
• <changefreq> & <priority>: Hints (not guarantees) to crawlers about how often you expect the content to change, and relative importance.

Feel free to tweak changefreq per resource type (e.g. daily for news, monthly for docs) or even make it dynamic based on post age or traffic.

Injecting Base (Static) Routes

defp base_routes do
  lastmod = DateTime.utc_now() |> DateTime.to_date() |> Date.to_iso8601()

  Phoenix.Router.routes(NctWebsiteWeb.Router)
  |> Enum.filter(fn route ->
    !String.contains?(route.path, "admin")
    and !String.contains?(route.path, ":")
    and route.path != "/"
  end)
  |> Enum.map(fn route ->
    loc      = NctWebsiteWeb.Endpoint.url <> route.path
    ~s(  <url>\n    <loc>#{loc}</loc>\n    <lastmod>#{lastmod}</lastmod>\n    <changefreq>weekly</changefreq>\n    <priority>0.75</priority>\n  </url>)
  end)
  |> List.insert_at(0, 
    ~s(  <url>\n    <loc>#{NctWebsiteWeb.Endpoint.url}</loc>\n    <lastmod>#{lastmod}</lastmod>\n    <changefreq>weekly</changefreq>\n    <priority>1.00</priority>\n  </url>)
  )
end

• Route discovery: Pulls every path from your router, then filters out:
  • Any path containing "admin" (hidden areas)
  • Dynamic segments (:) which don’t make sense in a sitemap
  • The root path temporarily—then re-inserts it at top with highest priority.
• Timestamps: Uses “now” for static pages, assuming you redeploy when content changes.

Integration & Usage

1. Router

   scope "/", NctWebsiteWeb do
     get "/sitemap.xml", SitemapController, :sitemap
   end

2. Link in <head>

   <link rel="sitemap" type="application/xml" href="/sitemap.xml" />

3. Robots.txt

   Sitemap: https://example.com/sitemap.xml

4. Caching For high-traffic sites, wrap build_sitemap/1 in a cache (ETS or Cachex). Regenerate only when blog data changes or on a timed interval.

Production Hardening & Extensions

• Pagination: If you exceed 50 000 URLs, split into multiple sitemap files (e.g. /sitemap-1.xml, /sitemap-2.xml) and serve a sitemap index.
• SiteMap Index: Generate an index file listing all your smaller sitemaps using <sitemapindex>.
• Priority & Frequency Rules: Compute priority based on page views or business logic; adjust changefreq for truly dynamic endpoints (e.g. e-commerce product inventory).
• Error Handling: If Repo.all/Blog fails (DB outage), return a cached sitemap or a 503 to prevent crawlers from treating sitemap unavailability as an error.
• Concurrent Builds: If your site has thousands of posts, build sitemap in a Task or background job and store the result for quick responses.

Conclusion

With just a few Elixir functions and your existing Phoenix tooling, you’ve built a fully dynamic, database-driven sitemap generator. It automatically discovers both static routes and content-backed URLs, stamps each with a last-modified date, and serves a standards-compliant sitemap.xml. Coupled with caching and pagination strategies, this approach scales from small blogs to enterprise sites, keeping search engines—and your SEO—happy with minimal maintenance.