Hello DEV Community! ๐
I want to share a project I’ve been building: an ultra-fast Web Metadata, OpenGraph & Contact Extractor API.
Here is the interesting part: 100% of the code, architecture, optimizations, and benchmarks were engineered in partnership with Gemini AI (Antigravity).
Initially, heavy web scraping took up to 23+ seconds on bloated websites. Working together with Gemini AI, we refactored and optimized the entire backend step-by-step to achieve enterprise-grade performance under 200ms (and 0.05ms on cache hits!).
๐ ๏ธ The Technical Stack & Optimizations
Here is how Gemini AI brought response times down by 99%:
-
Fast C-Parser (
selectolax): Replaced standard BeautifulSoup parsing withselectolax(written in C based on Lexbor engine), dropping DOM parsing times from 15ms to ~2ms. -
HTTP/2 & 256KB Streaming Limit: Configured
httpxwith native HTTP/2 multiplexing and stream capping at 256KB to retrieve<head>tags instantly without downloading 20MB of unnecessary assets. -
Smart DOM Cleaning: Stripped
<script>,<style>, and<code>tags before running regexes to prevent email false positives (like[email protected]in JS templates). - 15-Min TTL In-Memory Cache: Repeated requests to identical URLs return instantaneously in 0.05 ms.
๐ What the API Extracts in a Single Call:
- ๐ฏ SEO & OpenGraph Metadata: Title, Description, OG Image, Keywords, Author, Language, Favicon.
- ๐ง Clean Contact Discovery: Public emails & phone numbers (cleaned of JS noise).
- ๐ฒ Social Media Profiles: Auto-detects Twitter/X, LinkedIn, Facebook, Instagram, GitHub, YouTube, Telegram, and TikTok.
- ๐ ๏ธ Tech Stack Detector: Identifies WordPress, Shopify, WooCommerce, React, Next.js, Vue, Nuxt, TailwindCSS, Bootstrap.
๐ Free Tier Available for Developers
I opened a free tier of 100 requests/month on RapidAPI so the community can use it for side-projects, lead enrichment, or link preview cards:
- ๐ฆ GitHub Repository & Code Examples (Python, JS, cURL): https://github.com/JosejuX/rapidapi-metadata-extractor
- ๐ RapidAPI Hub: Web Metadata & Contact Extractor on RapidAPI
๐ฌ Final Thoughts & Feedback
Working with Gemini AI to build this API showed me how fast we can take an idea from zero to a production-ready, monetization-capable SaaS.
Iโd love to get your feedback, suggestions, or feature requests! What do you think about AI-driven API development? Let me know in the comments below!
๋ต๊ธ ๋จ๊ธฐ๊ธฐ