Prepare web content for Amazon Bedrock Knowledge Bases with Crawl4AI

When you build Retrieval Augmented Generation (RAG) applications, answer quality depends on the data that you ingest. Amazon Bedrock Knowledge Bases is the fully managed RAG capability of Amazon Bedrock. With Amazon Bedrock Managed Knowledge Base , you connect your data sources, and Amazon Bedrock manages ingestion, storage, indexing, and retrieval for you. You don’t provision or scale a vector store, and embedding and reranking use service-managed models by default.

For web content, start with the Amazon Bedrock Web Crawler data source connector for managed knowledge bases. You give it the seed URLs or sitemaps that you’re authorized to crawl, and it crawls within the scope and rate limits that you set. Scheduled syncs keep your knowledge base current as pages change, and the connector can wait for content that loads with JavaScript. There’s no crawler for you to build, host, or operate, so this connector is the starting point for most public website content.

Some sources need more control over what reaches the knowledge base. You might want to index only the main content region of each page, remove navigation and other boilerplate, or attach your own metadata attributes to every document. For that class of source material, you can extend the reach of your knowledge base with a preparation step built with Crawl4AI , an open source Python crawler. The step writes Markdown and a metadata sidecar for each page to Amazon Simple Storage Service (Amazon S3), and the knowledge base ingests both through its Amazon S3 data source.

Start with the Amazon Bedrock Web Crawler when it can reach and index the pages you need. It respects robots.txt , and after the first full crawl, each sync captures only added, updated, and deleted content.

The rest of this post follows the companion sample. Treat its architecture and defaults as a starting point, not as requirements of Amazon Bedrock Knowledge Bases.

By default, the sample runs the AWS Fargate task in a public subnet, where it reaches your configured websites through an internet gateway. Crawl4AI’s role ends once it has prepared the source documents and metadata sidecars. From ingestion onward, Amazon Bedrock Knowledge Bases handles parsing, chunking, embedding, indexing, and retrieval. The following diagram shows the Crawl4AI path from the schedule through to retrieval.

Figure 1: The Crawl4AI-to-Amazon S3 ingestion path for Amazon Bedrock Knowledge Bases

In this post, we show you how to choose the appropriate web ingestion path and implement custom preparation for content that needs additional processing. You then start ingestion into a managed knowledge base and validate retrieval quality. The companion aws-samples repository provides the complete workflow as a sample implementation.

The numbered steps in the diagram map to the following workflow:

Each attribute in the sidecar uses the typed format that both managed and customer-managed knowledge bases accept for Amazon S3 data sources. The following abbreviated sidecar, formatted for readability, shows three of the attributes that the workflow writes:

Because managed and customer-managed knowledge bases both support Amazon S3 data sources, the Crawl4AI path works with either type. This post uses a managed knowledge base and notes where a customer-managed knowledge base differs.

The example passes an integer timestamp, which both knowledge base types accept in number filters. A managed knowledge base doesn’t support the startsWith and stringContains filters. For a customer-managed knowledge base, pass the same filter in vectorSearchConfiguration instead of managedSearchConfiguration . The operators it supports depend on its vector store.

This is batch ingestion of known web properties, not real-time web access at inference time such as Web Grounding with Amazon Nova.

Use the Crawl4AI preparation path when public source material needs one or more of the following before ingestion:

The AWS Cloud Development Kit (AWS CDK) app packages the implementation in a container image during deployment. When the schedule fires, the container runs scripts/container_entrypoint.py and executes crawl, upload, and ingestion in sequence. The following three steps explain the core logic.