Knowledgebases

TaskFlowAI introduces powerful knowledgebase capabilities through the implementation of FAISS (local storage). This knowledgebase enables agents to digest and query information efficiently, serving as persistent memory that allows agents to retain and accumulate knowledge over time.

Overview

Knowledgebases in TaskFlowAI provide a mechanism for agents to access and utilize large amounts of information. They can be used to:

Store and retrieve context-specific information.
Serve as long-term memory for agents, allowing them to remember past interactions or data.
Enhance agents' abilities to generate accurate and contextually relevant responses.

The FAISS knowledgebase offers robust functionality for local storage and retrieval:

FAISS Knowledgebase

The FAISS (Facebook AI Similarity Search) knowledgebase provides fast and efficient similarity search and clustering of dense vectors, allowing agents to perform rapid searches over large collections of data stored locally.

Initialization

You can initialize a FAISS knowledgebase by creating a new one with chunks of text, creating an empty one, or by loading it from a saved index.

Initialize with Chunks

from taskflowai.knowledgebases import FaissKnowledgeBase
 
faiss_kb = FaissKnowledgeBase(
    kb_name="my_knowledgebase",
    embedding_provider="openai",
    embedding_model="text-embedding-3-small",
    chunks=[
        "TaskFlowAI is a framework for building AI-driven task pipelines.",
        "It allows agents to perform tasks using various tools.",
        "Agents can remember information over time using knowledgebases."
    ],
    save_to_filepath="path/to/save/index"  # Optional: save immediately after creation
)

Initialize Empty Knowledgebase

faiss_kb = FaissKnowledgeBase(
    kb_name="empty_knowledgebase",
    embedding_provider="openai",
    embedding_model="text-embedding-3-small"
)

Load from a Saved Index

faiss_kb = FaissKnowledgeBase(
    kb_name="my_knowledgebase",
    load_from_index="path/to/my_knowledgebase.index"
)

Querying the Knowledgebase

Agents can query the knowledgebase to retrieve relevant information based on a textual query.

# Query the FAISS knowledgebase
query = "How can agents remember information?"
results = faiss_kb.query(query, top_k=3)
 
# Display the results
for result in results:
    print(f"ID: {result['id']}")
    print(f"Score: {result['score']}")
    print(f"Content: {result['content']}\n")

Sample Output

ID: 2
Score: 0.95
Content: Agents can remember information over time using knowledgebases.

ID: 0
Score: 0.78
Content: TaskFlowAI is a framework for building AI-driven task pipelines.

ID: 1
Score: 0.65
Content: It allows agents to perform tasks using various tools.

Adding Memories

Agents can add new information to the knowledgebase, allowing them to accumulate knowledge over time.

# Add a new memory to the knowledgebase
faiss_kb.add_memory("Knowledgebases enable agents to store and retrieve information efficiently.")

Saving the Knowledgebase

# Save the index and metadata
faiss_kb.save_index("path/to/save/my_knowledgebase")

Note: The index will be saved in a 'faiss_indexes' subfolder within the specified directory.

Using Knowledgebases with Agents

Integrating knowledgebases with agents enables them to access persistent memories and utilize stored information when performing tasks.

Example: Agent with FAISS Knowledgebase

from taskflowai import Agent, Task
from taskflowai.knowledgebases import FaissKnowledgeBase
 
# Initialize the FAISS knowledgebase
kb = FaissKnowledgeBase(
    kb_name="agent_kb",
    embedding_provider="openai",
    embedding_model="text-embedding-3-small",
    chunks=[
        "TaskFlowAI supports multi-agent teams.",
        "Agents can have persistent memory using knowledgebases."
    ]
)
 
# Define an agent with access to the knowledgebase
agent = Agent(
    role="Assistant",
    goal="Provide detailed information about TaskFlowAI.",
    attributes="Helpful, informative, and knowledgeable.",
    tools={kb.query}  # Assign the query method as a tool
)
 
# Define a task that requires the agent to use the knowledgebase
def answer_question(agent, question):
    return Task.create(
        agent=agent,
        instruction=f"Answer the following question using your knowledgebase: {question}",
        context="Use the knowledgebase tool to find relevant information."
    )
 
# Agent responds to a question
task = answer_question(agent, "How can agents have persistent memory?")
response = task.execute()
 
# Print the agent's response
print(response)

In this example, the agent uses the knowledgebase tool to retrieve information relevant to the question, enabling it to provide a more informed response.

Best Practices

Consistency in Embeddings: Ensure that the same embedding provider and model are used throughout the lifecycle of the knowledgebase to maintain consistency in vector dimensions and representations.
Data Security:
- For FAISS: Secure local storage, especially if sensitive information is stored.
Performance Optimization:
- For FAISS: Choose appropriate index types based on your dataset size and search requirements (e.g., IndexFlatL2, IVF, HNSW).
Error Handling: Implement proper exception handling when interacting with knowledgebases to manage scenarios like network failures, missing indexes, or invalid queries.
Regular Updates: Periodically update the knowledgebase with new information to keep the agent's knowledge current and relevant.

Conclusion

Knowledgebases in TaskFlowAI empower agents with the ability to store, retrieve, and utilize information effectively. By integrating FAISS knowledgebases, agents can have persistent memories, enhancing their capabilities to perform complex tasks and provide informed responses.

Custom Tools FAQ