중복 콘텐츠가 교차 게시되나요? 먼저 오버랩을 측정했습니다.

If you syndicate the same posts to dev.to, Hashnode, Zenn or Qiita, sooner or later you wonder whether search engines read the copies as duplicate content. I got far enough into that worry to write a remediation plan: unpublish the live copies, set canonical everywhere, start over.

Then I counted the actual overlap. Across 11 articles and 22 files, three prose lines matched between origin and copy — and two of them were --- rules. The plan was unnecessary. What replaced it was adding a canonical URL on the platforms that accept one.

Here is the measurement, and the support matrix I wish I had looked up first.

Count the overlap by line, then split it by kind

Three steps:

1. strip frontmatter
2. normalize whitespace, drop blank lines
3. tag each line by whether it sits inside a code fence

Step 3 is the one that decides the answer. Without it you get a single percentage and no way to interpret it.

import fs from "node:fs";

const norm = (s) => s.replace(/\s+/g, " ").trim();

function bodyOf(file) {
  let t = fs.readFileSync(file, "utf8");
  if (t.startsWith("---")) t = t.slice(t.indexOf("\n---", 3) + 4); // strip frontmatter
  return t;
}

// Split into [normalized line, kind]
function classify(text) {
  const out = [];
  let inFence = false;
  for (const raw of text.split("\n")) {
    const line = norm(raw);
    if (/^```/.test(line)) { inFence = !inFence; out.push([line, "code"]); continue; }
    if (!line) continue;
    if (inFence) { out.push([line, "code"]); continue; }
    if (/^#{1,6}\s/.test(line)) { out.push([line, "heading"]); continue; }
    if (/^\|/.test(line)) { out.push([line, "table"]); continue; }
    out.push([line, "prose"]);
  }
  return out;
}

// Put the origin's lines in a set, walk the syndicated copy
const origin = new Set(classify(bodyOf(originFile)).map(([l]) => l));
const ext = classify(bodyOf(copyFile));
const matched = ext.filter(([l]) => origin.has(l));

const count = (k) => matched.filter(([, t]) => t === k).length;
console.log({
  ext: ext.length,
  matched: matched.length,
  code: count("code"),
  table: count("table"),
  heading: count("heading"),
  prose: count("prose"),
});

Enter fullscreen mode Exit fullscreen mode

Skip the inFence toggle and classify on the leading character alone, and shell comments starting with #, plus YAML and SQL lines starting with |, all land in the prose bucket. That bucket is the number you are trying to trust, so it has to be clean.

Blank lines matter for the same reason: they always match, so keeping them inflates both sides of the ratio.

The result: 762 of the 795 matching lines were code

Run across my 11 articles, counted as 22 pairs because English and Japanese are separate files:

Metric Value Lines in the syndicated bodies 1,658 Lines matching the origin 795 (47.9%) — inside code fences 762 — table rows 13 — headings 17 — prose 3 Prose lines in the syndicated bodies 393 Prose overlap 0.8%

Printing the three matching prose lines: two --- rules and one English bullet.

Overlap tracks code density, not copying. The posts with the most snippets score highest, which makes the raw percentage actively misleading. Code repeats because both versions describe the same implementation — rewriting a CREATE UNIQUE INDEX statement for the copy would just make one of them wrong.

If you write each version separately rather than pasting, run this before deciding anything.

Where you can declare a canonical URL, and where you cannot

Support is uneven. The API payloads tell you quickly:

Platform Canonical Where it goes dev.to Yes article.canonical_url on the articles endpoint Hashnode Yes originalArticleURL on the publishPost mutation Medium Yes canonicalUrl on the posts endpoint, or Import from URL Qiita No no equivalent parameter on the items API Zenn No no such key in article frontmatter note No no field in the editor

The Forem API exposes canonical_url on the article object, so it is one line in the payload:

body: JSON.stringify({
  article: {
    title,
    body_markdown: body,
    published: false,
    tags,
    canonical_url: "https://example.com/blog/<slug>/",
  },
}),

Enter fullscreen mode Exit fullscreen mode

Qiita’s API v2 takes title, body, tags, private, tweet and organization_url_name for item creation — nothing that points at an external original. Zenn’s article frontmatter is title, emoji, type, topics, published, publication_name, same situation.

Gotcha: Hashnode drafts cannot carry a canonical

originalArticleURL exists on the publishPost input but not on createDraft. Create a draft, hit publish in the dashboard, and the canonical never gets attached.

const input = publish
  ? { title, contentMarkdown, publicationId, tags, originalArticleURL: canonical }
  : { title, contentMarkdown, publicationId, tags };   // no field for it on a draft

Enter fullscreen mode Exit fullscreen mode

dev.to fixes the value at post time too, so already-published articles need PUT /articles/{id} or a manual edit in the editor.

On some platforms, “temporarily unpublish” does not exist

Without a canonical field, the way back from unpublishing is posting again: new URL, views reset, inbound links broken. It looks reversible and is not.

Which is the practical reason to measure first. Thirty lines of script, one run, and the answer may well delete the entire task list behind it.

The full story — how I ended up conflating an AdSense decision with search-side duplicate handling, the per-channel traffic numbers behind the reversal, and the per-article overlap breakdown — is on Aulvem → Aulvem | Is cross-posting duplicate content? I measured it before deleting anything