Knowledge-and-Memory-Management v0.0.2: Portable Knowledge Collection and Memory Management

Welcome to the v0.0.2 release of Knowledge-and-Memory-Management, a tool designed for ingesting and managing knowledge from diverse sources. This release marks a clean release, stripping all hardcoded personal paths and replacing them with the portable $AGENT_HOME environment variable. For experienced developers, this version brings consistency and ease of deployment across environments without sacrificing the core functionality of knowledge collection and memory management.

The project focuses on three primary collection pipelines: web, video, and articles. Each pipeline is modular, allowing you to configure, extend, or replace components based on your stack. The memory management layer ensures that collected data is indexed, stored, and retrievable via semantic search, making it practical for building personal knowledge bases or feeding into larger systems like agents or RAG pipelines.

Knowledge Collection

The collection module is source-agnostic at its core but ships with specialized handlers for common content types.

Web collection uses a configurable web scraper that supports depth limits, domain filtering, and content extraction via readability algorithms. You can target specific sections, strip ads, and normalize HTML into markdown. The scraper respects robots.txt and supports session management for authenticated sites.

Video collection transcribes audio using a local or remote ASR model. The pipeline extracts audio tracks, splits them into chunks, and generates timestamped transcripts. This is particularly useful for processing lectures, talks, or screencasts. The transcript is treated as a text document for further processing.

Article collection handles RSS/Atom feeds and direct URLs. It parses feeds, fetches full content using readability engines, and deduplicates entries. Articles are converted into a consistent schema: title, author, published date, body text, and metadata.

All collected data passes through a normalizer that converts content into a standard chunk format—suitable for embedding and memory storage. The normalizer handles language detection, tokenization, and optional summarization (the “S” in the original feature set, referring to semantic segmentation or summarization depending on configuration).

Memory Management

The memory layer stores knowledge chunks in a vector database with associated metadata. Each chunk is assigned an embedding generated from a configurable model (e.g., sentence-transformers). The store supports:

• Incremental insertion (upsert by content hash)
• Time-based or capacity-based garbage collection
• Query by embedding similarity, keyword, or metadata filters
• Batch export for backup or migration

In v0.0.2, the memory management has been refactored to use $AGENT_HOME as the base directory for all persistent files, including the vector store index and logs. This removes any previous machine-specific paths, making replication across development, staging, and production environments trivial.

Portable Paths with $AGENT_HOME

The primary change in this release is the elimination of hardcoded user paths. Previously, configuration files and data directories referenced absolute paths like /home/user/.agent/ or C:\Users\user\.agent\. Now, everything is relative to $AGENT_HOME. Set this variable to the project root, and the system resolves all paths automatically.

export AGENT_HOME=/path/to/project

Enter fullscreen mode Exit fullscreen mode

This change applies to:

• The vector store location
• Log and cache directories
• Collector state files (e.g., last scraped timestamps)
• Configuration defaults

If $AGENT_HOME is unset, the system falls back to the current working directory. This ensures backward compatibility for local runs while enforcing portability for deployment.

Code Example

Below is a minimal example showing how to collect a web article and store it in memory:

from agent_knowledge import Collector, MemoryStore

store = MemoryStore(base_path="$AGENT_HOME/data")
collector = Collector(store=store)

# Fetch and process a web page
result = collector.collect_web("https://example.com/technical-blog")

# The result is automatically chunked, embedded, and saved
print(f"Ingested {len(result.chunks)} chunks")

Enter fullscreen mode Exit fullscreen mode

The base_path parameter accepts $AGENT_HOME directly; the system expands the variable at runtime. After ingestion, you can query the memory store with natural language:

matches = store.search("background on memory management")
for chunk in matches:
    print(chunk.text[:200])

Enter fullscreen mode Exit fullscreen mode

Migration Notes

If you are upgrading from a previous version, move your existing data directory to the new location under $AGENT_HOME or update your environment variable. The v0.0.2 loader detects the old format and migrates it automatically, but the path resolution is now strictly based on $AGENT_HOME.

Summary

v0.0.2 delivers a cleaner, more portable foundation for knowledge collection and memory management. The shift to $AGENT_HOME eliminates a common friction point when moving between systems. The collection pipelines—web, video, and articles—remain stable and performant, while the memory layer provides a solid base for building applications that require persistent, searchable knowledge.

For developers integrating this into larger workflows, the API surface is unchanged, so existing integrations continue to work. Future releases will focus on incremental improvements to the embedding pipeline and support for additional source types. Until then, this release stands as a reliable, portable version of the system.