How will AI change the business landscape, how quickly will it evolve, and what should we look out for in the future? In a special episode of The Godel POD, Matthew Strachan, Client Director at Godel had the pleasure of speaking to Jorge Garcia de Bustos, one of Godel’s Presales Consultants.

What began with a natural interest in the topic and keen to widen his knowledge, Jorge has been researching applications for generative AI within the business which resulted in the development of an aptly named tool called ‘JorgeGPT’. This interview dives into the development of JorgeGPT and catches some of Jorge’s insights into what he predicts for the future of AI.

See below to read the interview or listen to this episode of The Godel POD for the full conversation.

For more conversations on the latest in tech, subscribe to The Godel POD on Podbean and Spotify. Podcast jingle provided by the Hideout YouthZone

Can you provide an overview of the tool you have built?

It’s an AI power tool that is capable of answering a series of questions about the contents of a document corpus, a collection of proprietary documents that our company has. The way it works is it harnesses the GPT model from open AI, which is the chat engine that uh lies underneath ChatGPT. When a user using this this tool formulates a question about the contents of these documents, I am actually letting the GPT model respond to that question. However the knowledge of the GPT model stopped knowledge of the world in 2021, which is when the training stopped also the knowledge of GPT does not include proprietary documentation, so it has been trained with publicly available documents.

The technique that I’m using to allow it to respond to questions used on the basis of proprietary information is something called retrieval augmented generation which is the idea of almost answering the question in two stages. On stage one, I take the question from the user and I do a search across all the documents that we are indexing, finding the most relevant snippets, let’s say 5 or 6 relevant snippets whose meaning is kind of similar to the question. And I’m taking those snippets of text and feeding that to GPT as context in order to answer the question. It’s almost like I’m saying to GPT, you are an assistant that answers questions. The question from the user is this one, and please use this block of text as the context in order to respond to it. Then what GPT does is analyse that context and formulate and rework an answer that is based on the content of those blocks of text that are there.

To give you an example, if we were talking about Infosec policy. For instance, if you wanted to answer the question, can I open an attachment from a sender that I’ve never met? We use something called semantic search to go through all the documents to find the most relevant bits and then GPT answers the question “shall I open this e-mail” based on all the relevant blurbs from all the documents that we have found.

By searching the documents you use semantic search there, so how do you qualify?

The search that a lot of people kind of understand intuitively is keyword-based search. You have documents and what you look for is for snippets of text that contain a certain keyword or a combination of keywords or little variations or keywords. Semantic search is a way of searching using machine learning techniques that instead of looking for keywords, looks for the most accurate matches for the meaning. What we do in semantic search is we take blocks of text and we extract the machine’s encoding of the understanding of the text or the meaning of that text as a vector of numbers. We retrieve a vector for every document in our corpus of documents, and we extract those vectors denoting meaning. Then what we do later on is we try to find the most approximate or the most accurate vector in the precalculated collection to match the meaning of the question asked by our user. This probably goes really, really technical for anyone who reads. You probably don’t want to go very much further than basically saying that it uses retrieval augmented generation, which is a technique to enrich the GPT engine with additional information, and that the way we do it, the way we extract that information is used in basically again machine learning techniques to find the most relevant snippets in the text.

But instead of using keywords, we use AI to look for the blocks of text that have the right meaning, and that’s an important thing because this engine can actually even cross the barriers of language. You could ask a question in English and then look at text written in any other language. And because what the machine looks is for, meaning as opposed to keywords, it can actually find snippets that have close meaning to the questions that you’re asking and use those to enrich the the context for GPT to answer the question.

What was the most challenging task in creating JorgeGPT?

The millions of wrong approaches and bad ideas that I had to basically discard before getting to the right one. I tried using pre-existing models and kind of using a machine learning technique called fine tuning, which is literally taking something that is almost like 95% trained and finalised the last 5% with my documentation but the results took forever, the training took forever and was super resource intensive and the results were terrible. I also had to use models that were not good matches.

The difficult thing was finding the right approach which required making a few initial experiments that were failures, going back to the drawing board and reading a lot, actually investing in a few books and familiarising myself with the state-of-the-art of technology and then redirecting the experiments and start looking at stuff that has it more carefully and trying to choose an approach that would work and then fine-tuning that approach with the help of the information and the education that I had just given myself. The good thing is that once I was on the right path, the progress was much, much faster, but until then it was very, very frustrating.

So when you identify the right approach, did you know instantly that this is the right way to go? This is not going to work?

Yes, because even though I was basically taking baby steps, those baby steps were already much more promising than everything that I had done initially.

How do you foresee JorgeGPT being used at Godel?

In the first instance, my idea is to use it as a mechanism to help people kind of ask questions about large document collections that are very difficult to search into. So an example is, for instance, our sales team would greatly benefit from being able to ask questions about our past engagements and past partnerships, about technologies that we have worked with, business sectors where we have had successful engagements, types of projects that we have developed before in order to prospect kind of for future work.

There is also another aspect of a lot of people in Godel that have relocated into places like Poland or Lithuania where any kind of bureaucracy and kind of the legalisation process for immigrants and everything is complicated and it’s actually a lot of the documentation is written in the in the local language. So being able to ask questions or to have a summarised view of some of the documents would be very useful.

The idea of being able to take documents that are difficult to scan into if you want and being able to answer questions about the content of those in the language or the tone is more kind of convenient for the person asking the question. Ultimately, pointing back or redirecting the person asking the question to the original source document and even the snippet saying “For more information go here”. So that would be almost like the Holy Grail for this.

If we can actually do that using something like a Microsoft Teams bot pretty much everybody in the right kind of group of users or anything like that can ask these questions online and get answers without having to delve deep into the documents. I think that would be like really helpful as a local experiment, but ultimately as something that we can repurpose and re-engineer for our clients to use in the real world.

In theory, do you think it could work on any large document?

The importance there is the way that you index the documentation initially to facilitate those searches because obviously, GPT is very good at rewording things and understanding the meaning of things and almost playing those back to that you see in plain language. What it means is good context there. It’s important to index the documentation well so that in that initial semantic search step, a lot of valuable information is pulled out so that we can enrich the conversation thread in GPT. Ultimately the success of it depends and the quality of documentation that you’re feeding into it.

What is JorgeGPT good at, and what does it struggle with?

So what it is really good at is when you have well-structured, well-formed phrases and sentences and content that actually follows almost like the conventions of language. The machine is very good at actually understanding that flow and finding the right meaning for that. I’ll give you an example, it was great to answer questions about the policies of my kid’s school. I had like 20 documents like PDF and Word documents. With apps or all the terms and conditions of the school and the policies regarding the kind of disciplinaries and everything. And it’s really, really good at finding the right snippets of documentation and in return finding sensible answers to those. It struggles with documentation that is ‘bitty’ like bullet point lists without context that need to be broken into chunks due to the nature of the tool, but it’s difficult to keep an overall context across the whole of them.

If you have a very large document and you need to break it into chunks to get the meaning of those and you need to invent a way to make sure that all the chunks are almost like tagged with the “this chunk of text pertains to this”, “this chunk of 10 pertains to this”. Keeping that context is relatively it’s relatively complicated, so when you have documents that are not written in an understandable language or don’t follow a succession of facts, the engine struggles a little bit more to find the right information.

If we were in a situation where the documentation being fed into the model was a ‘more bitty’ and there wasn’t a theme of context going through it, is there any safeguard in place to ensure the validity of what the models are outputting?

One of the important things is like. I have two answers for those. The first thing is that you should never use ML to substitute a kind of human judgment where the cost of making mistakes is very high. It’s a great tool to give you a first approximation to everything, but you should not take the output of anything that e-mail does and supervise or unfiltered. There has to be an element of human discretion to actually take the content of that and in separate say does this work or not, just like you would know do. Unless you’re an American lawyer and you want to be disbarred, you would not take the content of GPT to create an augmentation to go in front of a judge. What GPT produces is a first good approximation that you can then tweak, improve and use as part of your work. Something similar happens here. If you ask a question, you need to take the answer with a grain of salt and just understand the machine’s best approximation.

The other important thing is that we don’t want the answer. Although the answer in itself is useful, one of our objectives in the further development of GPT is to provide answers that actually point you back to the source documents so that you can verify that. So if our sales team for instance ask the question, “Have we ever worked with this technology or this business sector, or have we ever solved this challenge?” The important thing is not just a yes, we’ve done it for XYZ, but also, we want to point them in the direction of documents so that our sales team can actually go back to those and verify that’s the case.

I’m sure that we can ensure the validity of the model output more than what we can do is put in front some safeguards around the usage and kind of almost like a contract with the user saying do not think that what comes out of this is gospel but don’t trust it with your life and check the output a little bit. If the answers are systematic misfires, we will try to tweak the process of indexing or the way that we keep the contact in documents and everything but in general, you should treat the output relatively suspect and use it as a good starting point rather than the answer that you want to use.

What sort of task do you believe the AI tools we have today are good at solving/automating? Where do you see the big opportunities? 

Traditionally the answer to “What is ML good at?” used to be that if you have an activity that requires a kind of expert knowledge, an expert might take almost a fraction of a second or two seconds to actually make a decision. That used to be almost like the classical rule of thumb of where machine learning is good. But that was back in the day before, generative AI started to appear on the horizon. Now with the advent of all kinds of chatbots and chat engines and image generation and everything, the picture is becoming much richer in terms of the areas where you can have agents working. Something that is going to become kind of commonplace is this idea of answering questions with a specialist knowledge of something that we’re going to be able to trust like ML to do very, very soon if not something that you can trust to do it already.

So this idea of enriching the chat models with information that you retrieve on the spot that is relevant, this idea of the relevant generation that’s already kind of usable and is something that can be put into production without a lot of effort. Machines are inherently much worse than humans. So if your job involves doing things that are almost very run-of-the-mill very, very boilerplate, the reality is that AI is basically coming after that job. The main constraints will be basically the cost of it. The cost of training the machine and running it and everything is going to actually be higher than the cost of basically paying the salaries of those people. There aren’t going to be many areas where you’re going to be really safe quite frankly, the impact is going to be pervasive.

On the other hand, where do you see generative AI’s usefulness being more limited?

We become very wary of robots or avatars that are very similar to humans, we prefer the cartoony and I don’t think that anybody’s ever going to trust a sales agent or something like that, anything relating to a fellow human and anything that involves persuasion. Anything that like, I don’t think that those things are ever going to be in the hands of machines. That does not depend so much on cognitive capability but depends more on what makes humans unique. There’s probably safe everything else is it’s probably just going to be upheaval over the next 10 to 20 years, 10 to 20 twenty years is going to be insane.

In the same way that we don’t recognise the world before the Internet and mobile phones, we won’t recognise the world after these technologies have popularised.

Do you think the skills required for a job or like any human, if you need real human skills, like compassion and communication, do you think those skills are the ones machines can’t replicate?

Every time that you’re doing a job where things are very unique, where days are very different one day from the other. If you’re doing something that is repetitive and every day is exactly the same as the next one, be prepared. If, however, you do a job where every day is a little bit different or very different and the adaptability of machines is never going to get there, or at least not for a long, long time, that’s my hunch.

What do you believe the next major breakthrough for generative AI will be over the next 2-3 years? 

Everybody talks about the kind of the next breakthrough being the idea of a general AI, basically an AI that is capable of doing any kind of intellectual task that a human can do. I’m not sure if any of the approaches that we have right now and any of the kinds of technologies are ready to take us there. When it happens, it will be like an earthquake and everything that we have. But right now, I have no idea when the next big thing is going to appear.

The approaches that we have right now, although incrementally are at introducing kind of amazing changes, I’m not entirely sure that they will scale to the point of having something comparable to the human brain. We have things that do aspects of what we do in a very limited fashion, but that only don’t replicate our ability to understand language, replicate our ability to understand images, but we have nothing that actually binds the two of them without being very kind of purpose-specific.

I guess move on to a slightly different section What should businesses do today to put themselves in a position to capitalise on this new(ish) technology to stay on par with competitors in this space?

I guess with this, there’s been a lot of things that you could do with machine learning in the last few years that were very usable and where the readiness of data was one of the obstacles. I guess that what makes the GPTs in the world more usable is the fact that the state of your data doesn’t need to be ready. They are better at working with things that are in a bit of a rougher state.

One important aspect of how would a business capitalise on the new AI techniques is that first of all, wanting to do it and by wanting to do it, it’s actually embracing a little bit of risk tolerance and attitude.

Yes, there are reputational risks and there are risks of making mistakes and everything, nothing ventured, nothing gained. So if you want to eat the AI omelette, you’re going to need to break some eggs. The idea of setting up teams or collaborating with partners that have the capability and can help you experiment and do joint experiments and understand that experiments, sometimes the results are great, other times kind of are a bit of not so great. But out of everything, you basically get learning.

The first thing is, you cannot expect that this is going to be, “Give me an AI and we’ll just basically bolt on top”. There’s an element there of learning to work with it and embracing the kind of more experimental mindset that comes with this technology.

What role do you think AI will have to play in your average business 10 years from now?

There are a lot of areas, company divisions, company departments, and company roles, that just will not be there because there’ll be no general purpose. There might be a specialist AI to help with a lot of a lot of tasks. So every time you do something mind-numbing time-consuming or repeatable, you might be able to ask the machine to investigate for you and give you ideas of how it’s going to be done. We will use them a lot, and as a result, there’s going to be an increase in our productivity and our ability to get stuff done, but it’s going to require a significant kind of cognitive change in the way that we work.

Personally, it’s the reason why I’m working with it, other than the fact that I find the topic fascinating is also that I know it’s coming. So for myself building an intuition of how these things work is probably the best way of ensuring that over the next 5-10 years, I’ll be personally prepared for what will happen in the next 10 years. For anyone who’s not using AI, it’s almost like anyone who’s not using a laptop or an e-mail to communicate. So you might be able to get away at the beginning with not using them, but once they become generally available tools, anybody who’s not using them will be at a significant disadvantage versus everyone else who is using them.

Huge thank you to all the listeners out there who’ve listened to this instalment of the Godel pods. If you like what you hear and would like to know when we’re releasing even more episodes, please just subscribe to the Godel page on Podbean or Spotify.