Self engineered GenAI chat for enterprise

TL:DR PostNL’s AICoE built a solid internal GenAI chat bot on AWS, then the market caught up. Still learnt a lot as a company, now transitioning it to a innovation platform.

Intro:

With the fast-changing pace of Generative AI (GenAI) in business, most companies are scrambling to enable employees access to this new technology. How do large companies roll out technology in a fast and cost-effective manner, across business units that are diverse in both roles and as well as technology usage patterns (From Excel Tigers to Rust Developers and everything in-between).

There are many solid SaaS product offerings in the market that have a large and diverse product offering, slick user interfaces and immediate access to the newest technologies. Although this level of technology is cutting edge and ready for use out of the box, the cost profile of these licences match the level of technology. The value of these licences are justifiable if we could easily demonstrate that the value of these products are greater than this cost for all users. How do we cater for the ‘casual user’, the ‘once a month’ user, the ‘help me with a single task a week’ user?

PostNL has struggled with this exact challenge. We have in no way found the perfect solution but this article will explore some of our thought processes around the implementation of an application that addresses this exact problem, also looking at the business and AWS architectural decisions that made this possible. PostNLChat was born, as the first project roll out for the newly formed AI Center of Excellence (AICoE).

n.b. In March 2025, Microsoft Co-Pilot was released in March a product that seems to be catered to this exact market segment.

Software choice:

There are many open-source Chat UIs available for GenAI chat bots on Github. We looked at a variety of repos including

Libre Chat is an open-source project, that we decided was the best option for our application. It includes a Chat UI and back-end components to allow a simple full stack deployment in a cloud environment (Databases, Search, Application Containers).

Librechat includes many connection mechanisms for different AI Labs (OpenAI, Anthropic) and Cloud providers (AWS, Azure). It offers document uploading and automatic text vectorisation; converting text into numerical formats that models can understand and retrieve relevant content from (RAG). It also includes MeiliSearch (an open-source AI search engine), which enables users to search through previous conversations more efficiently.

It offers a similar user interface compared to other providers, enabling a familiar user experience which was essential for the rollout of this product. Quick adoption and limited training on a solution was one of the key requirements for this product; training is a large indirect cost that is often forgotten.

This is what a user can expect, when logging in for the first time:

Infrastructure:

PostNL has an internal ‘Landing Zone’ (a ready-made AWS environment, in a PostNL flavour) built in house by the AWSCoE that is 100% serverless (no Ec2’s here). This platform is a low cost (comparatively to other options available) and is used widely across the business. Given the environment choice, we had to look in the AWS serverless catalog for an appropriate solution to host this solution. AWS Fargate was chosen, due to its long running nature (compared to Lambda) as well as the simplicity of infrastructure maintenance. Fargate also boasts scalability and low relative compute costs, making it a perfect serverless platform for app development. EFS was chosen for persistent storage and consistency of data in scale up situations. The ability to only focus on the code in the container, not patching and maintenance of the underlying servers, has enabled us to quickly roll out and maintain this product.

The usual Enterprise level of Authorisation and Authentication is enabled with AWS Cognito that integrates within PostNL’s SSO flow, allowing for seamless integration within the PostNL tech ecosystem. Routing of requests to the containers is handled through an Application Load Balancer.

This service was deployed using AWS CDK (Cloud Development Kit) as Infrastructure as Code. With some minor tweaks from our standard internal code patterns, it was deployed quickly and securely within the PostNL ecosystem. The initial Proof of Concept for this product came from of our internal innovation teams (Mad effort on their behalf).

Amazon Bedrock integration:

We are using Amazon Bedrock as our main Large Language Model (LLM) provider, allowing for simple connections to the other AWS infrastructure. This integration is handled within Librechat and uses a simple config to connect different models, enabling us to redeploy the config file when an additional model becomes available. Selected OpenAI models are also available in the app, through Azure AI Studio.

Here is an example of a model config for our default model:

- name: PostNLChat | Everyday V2.2
  label: PostNLChat | Everyday V2.2
  description: This is a faster and compact multimodal model for less complex tasks
  default: true
  preset:
    endpoint: bedrock
    model: anthropic.claude-3-haiku-20240307-v1:0
    modelLabel: PostNLChat | Everyday
    greeting: Hi, ik ben PostNLChat, je virtuele assistent. Waarmee kan ik je vandaag helpen?
    promptPrefix: >
      ADD YOUR SYSTEM PROMPT HERE

Here are our current model offerings:

Claude 3 Haiku. This is the default model and is used for less complex and everyday tasks. As most users never change from the default, it was important to choose a model that was fast, competent and fit for general use (its age is countered by its speed and cost, with only a minor reduction in thinking ability compared to larger, newer models).
Amazon Nova Lite. An alternate everyday model. As a newer model we haven’t found its exact place yet.
OpenAI 4o. This is our more advanced model that is used for more complex tasks. Recommended for tasks that may have surpassed Haiku’s reasoning power.
Claude 3.7. This model was added recently, allowing users to experiment with the newer models. The hybrid reasoning features makes it more focused towards the technical teams, such as the engineering community.
Amazon Nova Pro. An alternative advanced model. As a newer model we haven’t found its exact place yet.

n.b. We are using the cross-region inference models in Amazon Bedrock (except for our default model). Although they have a slightly higher latency, the reliability from having multi region routing was worth it in our opinion. See Cross-region inference tab in Amazon Bedrock console for their ID’s.

Model prompts:

We released the models on PostNLChat with system prompts to apply some constraint to the models, removing some scope to ensure a professional, corporate conversation. We also limited the models to not accept any personally identifiable information (PII) in the systems prompt; this is to provide an additional level of data security and ensure the model understands the restriction on using PII within the platform. We found this to be a reliable and visible way of limiting any ability to upload PII with a not so gentle reminder of this requirement back to the user.

Here is a summary of one of our system prompts:

************Summary of intent of the prompt*********
PostNLChat is instructed to give PostNL colleagues fast, factual help in a professional tone. 
By default it replies in English but automatically switches to Dutch (or any language explicitly requested).
It avoids jokes or comments that might reflect poorly on the company. 
It refuses to process GDPR-sensitive personal data, advises users to handle complex calculations in external tools. 
It asks clarifying questions rather than guessing when unsure.
*********************************************************

Here is an example as to how the system handles Personally Identifiable Information Data requests.

**Taal van PostNL (Language of PostNL):
**
One of the custom prompts that was incorporated in the application was a tool for standardisation of our language, made by one of our communication teams. This feature is called ‘Taal van PostNL’ or ‘Language of PostNL’ for our English readers. It is a custom system prompt to help align communication across the business to a consistent standard. The basics of the prompt follow these rules:

Adhering to the PostNL language guidelines, such as using B1-level language, active writing, short sentences, first-person when referring to PostNL, and focusing on cooperation and solutions.
Providing tips for improvement and rewriting texts to fully comply with the guidelines.
Keeping to the specific language guidelines for PostNL, covering areas like tone of voice, structure, titles, paragraphs, sentences, subheadings, lists, and language simplicity.

This feature has the most impact for non-technical teams across the business, allowing them to use modern tools with immediacy that is typically only enjoyed by tech teams. Work is being done across the business to identify tasks that this technology can assist with in both the technical and non-technical space, as this technology is pushing past traditional tech usage barriers for under serviced non-technical teams. The Taal van PostNL feature shows there is a business need for this technology, as well as a willingness to be up-skilled.

Data Privacy and Compliance:

As anyone in Europe will know GDPR is a substantial part of any data intensive applications development considerations and data privacy requirements. When building this application we kept this in mind, a comprehensive terms of service policy is provided with the app that users must agree to that includes details such as:

User conduct and responsibilities
Risk and Compliance

As the builders of this application we are responsible for:

Data management (Data retention and access controls to databases)
Access controls to the application using SSO and elevated privileges notifications
AWS internal controls for application access (Least privileges IAM permissions)

This help maintain both user and application security, to align the application within our internal security frameworks as well as the external regulatory requirements. Given the rate of change with AI and the speed that this new technology is advancing, great care needs to be taken. We are ensuring that AI solutions are thought about in a responsible way and a security and compliance mindset is an important part of that process. You cannot discard default security principles in the name of innovation or speed, as this will lead to an unsustainable future of applications.

What is the impact of PostNLChat:

Currently we have over 600 active users on the platform. The product was first released in Nov 2024 (sorry for the late blog post), and we saw many immediate sign-ups. With the addition of other workplace tools that have been released recently (Microsoft Co-Pilot, Github Co-Pilot and Cursor), we have noticed a reduction in the specific need of this application. Here at the AICoE we are constantly developing internal solutions and see this application playing a role within the AI capacity at PostNL. We expect that many users who rely on PostNLChat for day-to-day tasks will transition to more integrated solutions like Microsoft Co-Pilot which are already embedded in the tools they use. We cannot keep up with the big players in regard to development speed, but we want to use PostNLChat as a space for innovation. It allows us to experiment with more advanced and experimental AI features, staying current with developments happening in the AI space.

What is the cost:

The main internal push for this product was a cost reduction. The identification of the low volume/sporadic user base for a simpler more cost-effective solution, meant that this product was immediately viable. Across all over environment (Dev, Test, Acc and Prd) this app costs around a total of €800 a month in direct AWS costs. This works out to be around €1.33 per user per month, a significant saving over current premium licensing costs. This was an immediate cost saving identified on day 1, when we reduced licenses with other providers and migrated to the internal solution.

Final thoughts:

GenAI is becoming more and more integrated in our daily business activities, no business can take on the cost of enabling all of the SaaS products available in the market. Taking a more pragmatic approach to enabling the business with new technology, but also not going over the top with internal app development is essential in a rapidly evolving space. You cannot take a year to bring an internal app to development; the industry will swallow it up before the stories are refined. This product didn’t solve every problem, we still use a large catalog of GenAI products across the business. It did enable a wide user base of technical and non-technical people access to the modern technology, in a compliant and measured manner.

Self engineered GenAI chat for enterprise was originally published in PostNL Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.