The Problem:

• PostgreSQL excels at basic transactions, but struggles with complex logistics analytics requiring:
  • Joining real-time GPS data, order details, and customer addresses for route optimization.
  • Joining order timestamps, delivery statuses, and customer information to assess delivery efficiency.
• Extensive joins and aggregations slow down PostgreSQL due to table scans and lookups.

The Scaling Challenge:

• As you add drivers and customers, data volume and complexity explode in PostgreSQL.
• Complex analytical tasks like route optimization and delivery efficiency require even more joins and aggregations.

Why Clickhouse?​

In such a case the ideal scenario for a company would be to set up a real-time replication pipeline from their transactional database (Postgres in this case), to something like a ClickHouse - which is optimised for analytical queries on a large dataset.

In this scenario, using ClickHouse alongside PostgreSQL can be a game-changer for the logistics company's data analytics needs. Here's why:

1. Columnar Data Storage: ClickHouse stores data in a way that's perfect for analytics. It organises data by columns, which makes it super fast to search and retrieve just the bits of data needed for a query. On the other hand, PostgreSQL stores data row by row, which can slow things down, especially when dealing with large datasets - as it requires an entire table scan for performing extensive joins.
2. Materialized Views: ClickHouse offers built-in support for materialized views, which are precomputed query results stored as tables. These materialized views can significantly improve query performance by eliminating the need to repeatedly perform complex calculations or joins on large datasets. This feature streamlines analytical queries, leading to faster insights and more efficient data analysis workflows.
3. Scalability: ClickHouse is designed to handle massive datasets and can scale horizontally by distributing data across multiple nodes. This scalability ensures that as the company grows and its data volumes increase, ClickHouse can seamlessly accommodate the expanding workload without sacrificing performance.

Performance Comparisons​

Let’s take a look at some analytical queries that one would see in a real world scenario like mentioned above, and see how each of these databases perform.

As explained in the problem statement, we have 5 tables namely - orders, jobs, drivers, driver_logs, job_assignments. Now, for the performance comparison between the query execution times in clickhouse and postgres databases, we take an example of an SQL query which can be highly useful in the real time analytics of the scenarios that a logistics company would face.

The SQL query selects rounded off latitude and longitude coordinates from the intermediate table "driver_locations" alongside the count of drivers at each location. It then filters the results to include only records where the number of drivers exceeds the average across all locations.

Essentially, this query identifies high-traffic areas by pinpointing GPS coordinates with an above-average concentration of drivers, aiding in the analysis and decision-making processes for optimizing driver deployment and route planning in logistics operations.

The single-read query would look something like:

-- Query to aggregate driver locations and identify high-traffic areas without using materialized views
WITH driver_locations AS (
    SELECT
        ROUND(gps_latitude, 2) AS latitude,
        ROUND(gps_longitude, 2) AS longitude,
        COUNT(*) AS num_drivers
    FROM
        driver_logs
    GROUP BY
        ROUND(gps_latitude, 2), ROUND(gps_longitude, 2)
)
SELECT
    latitude,
    longitude,
    num_drivers
FROM
    driver_locations
WHERE
    num_drivers > (SELECT AVG(num_drivers) FROM driver_locations);

Querying Postgres VS Querying Clickhouse (Non materialised)​

After conducting a performance comparison between ClickHouse and PostgreSQL databases on a dataset containing approximately 10 million records using the provided SQL query, the outcomes are as follows:

In the case of the ClickHouse database, the query execution time is impressively swift, clocking in at 2300 milliseconds (ms). Conversely, for the PostgreSQL database, the query execution time is noticeably longer, measuring 30174 ms (>30 seconds). This stark difference illustrates that ClickHouse outperforms PostgreSQL by nearly fifteen times in terms of query execution speed.

These findings underscore the significant advantage that ClickHouse offers in handling analytical queries efficiently, particularly in scenarios where real-time insights are crucial for logistics operations.

Querying Postgres VS Querying Clickhouse (Materialised)​

Similarly, we can also try having a materialised view for the situation, and then run a single read query on that view. So the query would look something like:

--Create materialised view to aggregate driver locations

CREATE MATERIALIZED VIEW driver_locations_aggregated ENGINE = AggregatingMergeTree() ORDER BY (latitude, longitude) AS

SELECT
round(gps_latitude, 2) AS latitude,
round(gps_longitude, 2) AS longitude,
count(*) AS num_drivers

FROM driver_logs

GROUP BY round(gps_latitude, 2), round(gps_longitude, 2);

-- Populate the materialised view

INSERT INTO driver_locations_aggregated

SELECT

round(gps_latitude, 2) AS latitude,

round(gps_longitude, 2) AS longitude,

count(*) AS num_drivers

FROM driver_logs

GROUP BY round(gps_latitude, 2), round(gps_longitude, 2);

-- Query to identify high-traffic areas

SELECT

latitude,

longitude,

num_drivers

FROM driver_locations_aggregated

WHERE num_drivers > (SELECT avg(num_drivers) FROM driver_locations_aggregated);

In this case, we can observe that querying a materialized view on Clickhouse takes 111ms which is more than 270 times faster than querying a Postgres (which took us ~30 seconds).

These findings reinforce ClickHouse's superiority in handling analytical tasks, offering quicker insights and enabling more responsive decision-making in logistics operations.

CPU Utilisation​

ClickHouse demonstrates superior efficiency in handling queries compared to Postgres, evident in CPU utilization metrics. ClickHouse (materialized) evenly distributes CPU usage across all cores in a multi-core system, maintaining a maximum of 80% utilization (on a 8GB RAM machine). In contrast, Postgres shows uneven distribution, with some cores reaching 100% utilization while others hover around 20%.

Clickhouse​

Postgres​

Conclusion​

Comparing the performance of ClickHouse and PostgreSQL databases shows that ClickHouse is better at handling complex queries efficiently.

By using both ClickHouse and PostgreSQL together, logistics companies can have a complete solution that combines transactional reliability with strong analytical capabilities. Connecting PostgreSQL to ClickHouse in real-time allows for smooth integration of transactional and analytical data, making the most of each database's strengths for better overall performance.

Moreover, ClickHouse's use of materialized views simplifies analytical tasks, making it faster to gain insights and make decisions in real-time analytics situations. This hybrid approach, using PostgreSQL for transactions and ClickHouse with materialized views for real-time analytics, gives logistics companies an edge in the market.

Dozer stands out as a great tool for this setup, making it easy to replicate data from PostgreSQL to ClickHouse in real-time. By bringing together different data sources seamlessly, Dozer lets logistics companies create real-time data views using standard SQL and turn them into accessible APIs with minimal effort.

Dozer acts as a real-time analytics layer, ensuring that insights are obtained quickly and enabling confident decision-making based on data. Its scalability means that companies can focus on innovation without worrying about managing infrastructure, as Dozer can handle increasing data needs effortlessly.

In summary, by adopting this two-database strategy, logistics companies can make the most of their data, improving efficiency, enhancing customer experience, and staying competitive in the fast-paced logistics industry.

Difference Between GPTs and Assistants:​

GPTs (Customizable ChatGPTs)​

Purpose : To enable developers to create personalized AI models.

Key Features:

1. No Coding Requirement: It simplifies the process for non-programmers, allowing them to develop AI models through basic instructions.
2. Customization: Developers can tailor the AI to specific needs or industries.
3. Ease of Use: The process is as simple as starting a conversation, making AI more accessible to a wider range of users.

Assistant APIs​

Purpose: To provide a more agent-like, task-oriented AI experience.

Key Features:

1. Specificity: Designed for defined purposes with clear instructions and additional knowledge.
2. Advanced Capabilities: Includes features like Code Interpreter and Retrieval, allowing the AI to perform more complex tasks.
3. Model and Tool Integration: The ability to call upon various models and tools for executing specific tasks.
4. Simplified Development: The API streamlines the development process, making it easier to create sophisticated AI assistants.

LLM Limitations: RAG (Retrieval Augmented Generation) and Fine Tuning

LLMs are great but are limited in their knowledge base. Two primary challenges encapsulate the issues faced by Large Language Models (LLMs):

1. Stale Training Data: The data used to train LLMs tends to become outdated, posing a hurdle in keeping up with recent trends and events.
2. Extrapolation and Hallucination: In the absence of factual information, LLMs often resort to extrapolation, resulting in the generation of confidently stated yet false statements.

These are the two popular ways of bridging this gap.

1. RAG:

   Integrates a retrieval component with a generative model, dynamically fetching external information to enhance the generation process. It's great for queries needing up-to-date or specific knowledge, but can struggle with data quality, relevance, and increased response latency.

2. Fine-Tuning:

   Involves adjusting a pre-trained model on a specific dataset to tailor its responses. It offers more control over the model's output and consistency in performance, but lacks the ability to incorporate real-time information and can be limited by the scope of the training data.

RAG (Retrieval-Augmented Generation) can be seen as superior to fine-tuning in certain scenarios due to its dynamic nature and ability to incorporate a wide range of up-to-date information. RAG retrieves information in real-time, it can adapt to new topics or recent developments more effectively than a fine-tuned model, which might require retraining to stay current.

Langchain has been a leading solution in integrating knowledge using RAG, gaining wide popularity. However, OpenAI's recent advancements, particularly in assistant APIs, suggest a shift in the landscape. These APIs now enable direct integration of similar functionalities with potentially higher quality, potentially rendering approaches like vector stores and Langchain less critical in certain applications.

The Assistants API allows you to build AI assistants within your own applications. An Assistant has instructions and can leverage models, tools, and knowledge to respond to user queries. The Assistants API currently supports three types of tools: Code Interpreter, Retrieval, and Function calling.

How Dozer Can Help You with RAG?

Dozer supercharges AI applications with lightning-fast API data retrieval, seamlessly fueling retrieval systems and tools for smarter, more informed responses.

• Enhanced Data Access for RAG:

  RAG models rely on external data for generating informed and accurate responses. Dozer's ability to quickly fetch data via APIs means that these models can access a broader range of current and relevant information, improving the quality of their output.

• Real-Time Information Retrieval:

  In scenarios where up-to-date information is crucial, Dozer's rapid API calls enable real-time data retrieval. This is particularly useful for applications that require the latest data, like news updates, stock prices, or weather reports.

• Integration with Diverse Data Sources:

  The ability to tap into various APIs allows for the integration of diverse data sources. This is beneficial in creating more comprehensive and multifaceted AI applications that can pull in specialized information from different fields.

• Streamlining Development with Assistant APIs:

  When building applications with Assistant APIs, the rapid integration of external data through Dozer can streamline the development process. It allows developers to focus on building the core functionality of their applications without worrying about the complexities of data retrieval.

• Customization and Flexibility:

  Dozer's API-driven approach provides developers with the flexibility to customize their retrieval sources. This customization is crucial in building applications that cater to specific domains or user needs.

• Scalability and Efficiency:

  Efficient data retrieval via APIs contributes to the scalability of AI applications. It ensures that as user demand increases, the system can maintain performance without significant increases in latency or resource consumption.

In our next article, we'll dive into a hands-on example illustrating how Dozer's effortless API integration can enhance RAG with Open AI APIs. Please reach out to us with your thoughts or if you want to have a discussion.

The Assistants API empowers you to construct AI assistants seamlessly integrated into your applications. These assistants, equipped with instructions, harness models, tools, and knowledge to adeptly address user queries. Presently, the Assistants API accommodates three essential tools: Code Interpreter, Retrieval, and Function Calling. The integration process for the Assistants API follows a straightforward flow:

1. Assistant Creation: Initiate an Assistant in the API, defining custom instructions and selecting a model. Optionally, enable tools like Code Interpreter, Retrieval, and Function Calling to enhance functionality.

2. Thread Creation: Establish a Thread when a user initiates a conversation.

3. Message Addition: Populate the Thread with Messages as users pose questions or interact.

4. Assistant Execution: Invoke the Assistant on the Thread to elicit responses. This automatically engages the relevant tools, ensuring a dynamic and responsive interaction.

In this instance, we are embarking on the development of an assistant specialized in knowledge retrieval, specifically to answer questions about the rules of chess using a document. Once developed, we plan to test it in the Assistant Playground and potentially publish it to the GPT Store.

Setting Up the Client​

The script kicks off with the straightforward import of the OpenAI package and the client's definition. It is recommended to handle the API key through an environment variable to prevent inadvertent leaks to version control systems.

from openai import OpenAI
import time

# Initialize Client
client = OpenAI(
api_key= "YOUR_API_KEY",
)

Uploading Knowledge Source​

Following the client's initialization, the script proceeds to provide the knowledge source by uploading a file to the client's context.

# Upload file
file = client.files.create(
file=open("LawsOfChess.pdf", "rb"),
purpose='assistants'
) 

Initializing the Chess Assistant​

Once the file is uploaded and the client is set up, the next step is to select the model on which the assistant will run. The assistant is then initialized with the required tools and knowledge base. In this case, the assistant is named "Chess Assistant," and specific instructions are provided.

# Add the file to the assistant
assistant = client.beta.assistants.create(
name="Chess Assistant",
instructions="""You are an assistant to help people with the game of Chess. You know the rules. Use the knowledge found in the uploaded file to best respond to players' queries.""",
model="gpt-4-1106-preview",
tools=[{"type": "retrieval"}],
file_ids=[file.id]
)

Note the use of the "retrieval" tool, indicating the need to retrieve knowledge from external sources. OpenAI also supports tools like function calling and a code interpreter for more complex use cases.

Organizing Conversations with Threads​

The script organizes conversations as "threads," similar to the "Chain Of Thought" concept for agents in Langchain .

# Create a thread

thread = client.beta.threads.create()

Interacting with the Chess Assistant​

With the setup complete, the script proceeds to interact with the assistant by sending a message and waiting for a response.

# Send a message to the assistant
message = client.beta.threads.messages.create(
thread_id=thread.id,
role="user",
content="What is Quickplay finish in Chess?",
file_ids=[file.id]
)

Running the Chess Assistant​

The assistant is then run by associating the thread and assistant.

# Run the assistant
run = client.beta.threads.runs.create(
thread_id=thread.id,
assistant_id=assistant.id,
)

To complete the process, the script continually checks the status of the run until it is 'completed' and then processes the response.

# Wait for completion
while run.status != "completed":
            time.sleep(1)

# Retrieve and print the answer
messages = client.beta.threads.messages.list(
thread_id=thread.id
)

print(f"ANSWER: {messages}")

Exploring Assistant Capabilities in OpenAI Playground​

As we have seen, it is incredibly easy to setup an Assistant and use it for interaction. We can even view our assistant in the OpenAI Playground and test it online.

As we explore the capabilities of OpenAI's Assistant API, it's crucial to understand two key approaches to enhance the performance and relevance of your AI assistant: fine-tuning and retrieval.

Fine-tuning:​

Fine-tuning involves training a pre-existing model on a specific dataset to tailor it for a particular task or domain. This is done to prevent the models to make a certain kind of mistake again and again. While fine-tuning allows for a high level of customization, it requires a considerable amount of domain-specific data and expertise. This process is ideal when your AI assistant needs to address very niche or specialized queries.

In the context of our chess assistant, fine-tuning could be applied to make the model more adept at answering intricate chess strategy questions or providing insights into specific chess openings. However, the trade-off is the resource-intensive nature of fine-tuning, both in terms of time and data.

Retrieval:​

On the other hand, retrieval is a more flexible approach that leverages external knowledge bases or documents to enhance the assistant's responses. OpenAI's Assistant API supports the retrieval tool, allowing the assistant to pull in information from an external source.

In our chess assistant example, retrieval could be used to fetch the latest tournament results, rule updates, or any dynamic information related to chess. This ensures that the assistant stays up-to-date without the need for continuous fine-tuning.

Choosing the Right Approach:​

The choice between fine-tuning and retrieval depends on the specific requirements of your AI application. If your domain is highly specialized and you have access to a substantial amount of domain-specific data, fine-tuning may be the preferred route. On the other hand, if your application requires constant updates or draws upon a diverse range of information, retrieval offers a more dynamic and resource-efficient solution.

In the pursuit of creating powerful GenAI applications, integrating real-time data is paramount. While static knowledge serves as the foundation, real-time data adds the fuel needed to power dynamic and hyper-personalized user experiences.

As showcased in the earlier sections, OpenAI's Assistant API, coupled with tools like retrieval, enables seamless integration of external knowledge sources. However, to truly unlock the potential of your chatbot, incorporating real-time data is essential. This could include the latest chess tournament results, rule updates, or even personalized user information. By integrating personalized user data, the assistant can tailor responses based on the user's preferences, history, or interactions, creating a more personalized and user-centric interaction.

Dozer, a powerful tool in the data integration landscape, allows you to effortlessly integrate real-time data from various sources, adding an additional layer of personalization and specificity to your assistant's responses.

As we continue our exploration, it's essential to highlight the versatility of these chatbots beyond the OpenAI Playground. We'll guide you through the steps to effortlessly integrate your assistant into third-party applications such as Slack, Microsoft Teams, and more.

Stay tuned for the upcoming articles, where we'll delve into the process of extending the reach of your assistant and unlocking its potential in various digital channels.

In the next segment, we'll specifically focus on the step-by-step process of integrating your assistant with popular messaging platforms using Dozer, enabling you to deploy your chatbot where your users are most active.

The era of AI-powered, data-integrated chatbots is here, and the possibilities are limitless. Join us as we continue to simplify, enhance, and redefine the landscape of interactive AI applications.