This episode focuses on AI and more specifically, generative AI. Many people will have recently seen the trending discussions and thought pieces on #ChatGPT. To find out what it all means for legal professionals, we speak with Jonny Badrock, Senior Director for Data Services at SYKE. We explore the immediate applications of generative AI for legal teams, consider its current limitations and then cast our eyes forward to discuss what it might mean in the medium and long term.
This episode focuses on AI and more specifically, generative AI. Many people will have recently seen the trending discussions and thought pieces on #ChatGPT.
To find out what it all means for legal professionals, I spoke with Jonny Badrock, Senior Director for Data Services at SYKE.
We explore the immediate applications of generative AI for legal teams, consider its current limitations and then cast our eyes forward to discuss what it might mean in the medium and long term.
Further reading
Generative AI
Mark: [00:00:00] Johnny. Can you tell us, this is a bit about who you are?Â
Jonny: Hi, I'm Johnny Badrock and I'm a senior director for Data Services at SYKE. So I'm part of SYKE's leadership team. And one of the things that I help our clients with is implementing AI for legal.
So I spend a lot of my time working with kind of a data scientist and the developers in our team, thinking about kind of practical implications for how we can use AI to help lawyers do their jobs better.Â
Mark: Brilliant. You're the right person to be speaking to because today's episode is all about generative ai.
We actually, we, I look back in the archive. So we did an episode, one of our first episodes on LODcast was about AI back in 2019, which I guess isn't that far ago, but it seems like a long time ago now. And the landscape has changed quite a lot since then, , particularly for ai. And it feels like it's shifted quite a lot just the past couple of weeks with the, the kind of arrival of, of chat G P T, which we're gonna get into by thought.
For starters, why don't we just start with what are we, what is ChatGPT?Â
Jonny: Yeah, definitely. It's, it is a good observation, mark. Cause I think AI has moved on a lot [00:01:00] in the last three years and certainly in legal as well. And it'll continue to do so. I think one of the nice things is that some of the stuff that we were talking about back in 2019, It's now kind of practical implication has been embedded.
It is been used by corporations and law firms globally. So using AI for kind of contract review, contract extraction activities was, was still being talked about a lot in 2018, 2019 and now is kind of implemented, embedded in a lot of processes. So it's really exciting just to show how much kind of things have moved on in the last few years.
I think ChatGPT is is now kind of the next thing around the corner and is, is really interesting. There's been a lot of hype around it over the last few weeks. I've been speaking to a lot of people in legal, in tech, , , and also not, and everybody seems to have heard of it, which is exciting.
And it is really powerful. I think it's also really important to remember, but it's still really early stage and it's still very much concept. So it's really interesting to show the art of what's possible and the direction of ai, but it's certainly nowhere near final product. And essentially what it is, is a really modern language processing model.
So effectively it's a chatbot, a little bit like what people are already used [00:02:00] to. But what makes it really interesting is the quality of the machine learning algorithms that sit under underneath it. And I guess more importantly, what that's enabling is a conversational exchange to happen between human and machine for the first time.
So those response are generated. . But what it's actually really interesting is it allows the user to input their questions in a conversational manner. So historically, we've kind of search engines, and when you're trying to use those chat bots for kind of utility companies, it takes about 30 attempts to try and get the right keyword in to get the answer that you want, right?
What's starting to be more interesting with G P T is actually, it can start to interpret a little bit what you're trying to get to, and it allows you to engage in a more conversational way.Â
Mark: Yeah. And the observation that lots of people know about it is true. So not just, not, not just tech technologists, it's anyone.
A lot of my friends, a lot of lawyers, in-house leaders, non-legal people non-tech people. Have heard about it and I've certainly spent some time playing around with it. Just for, for the benefit of listeners, will, will include a link to the, to basically, I think the [00:03:00] best ways to go and have a look at it.
It, it's up and down a bit at the moment. I think the demand that are up to a million users, there's a lot of people using it. So it's, it's a bit hit and miss sometimes being able to use it, but it will add a link so you can go and have a look at it. But yeah, as you say, it's just basically, it's a, it's a large language model.
It's, it's a super advanced chatbot and it's really impressive in its ability to do most stuff, and it's quite unimpressive in some areas. But we're gonna get into that. We're gonna get into some of the limitations, but before we do that why don't, who created ChatGPT.Â
Jonny: Yeah, so it's created by a, a research institute named Open ai.
Open AI itself was set up around 2015, I think it was. And it's got prominent backers, including Elon Musk, who I'm sure most people listening have heard of. And it's now owned by a range of investors, including Microsoft and the Y Combinator. So it was a lot of money and a lot of kind of power and a lot of interest backing this And I guess we've gotta remember, you know, the reason why it's got so much traction is cuz quite, quite frankly, it's fun.
It's fun to use, it's fun to engage with. And I guess, you know, the next exciting bit will be where [00:04:00] do we start to put practical applications around this kind of technology? But I think with that kind of backing we can be fairly certain that it's not just gonna kind of disappear anytime soon.
Mark: Yeah, absolutely. And, and I think fun is a fun point. I mean, I, I connected, obviously, we we worked together, Johnny, but we connected over this because you did a LinkedIn post about it. And then I did one a few days later. I had just spent the weekend kind of playing around with it and was very, you know, I felt compelled to write about it because it was quite in, quite impressive in terms of its application for.
I guess thought leadership pieces, but also like you could tell it to write a poem about Roger Federer and Santa Claus and it would just do it instantly or, you know, whatever your, you know, topic of a choice might be. I think that's where it really excels. So, so it's been around for about a fortnight.
People, you and I have both spent some time playing around with it. Why don't we go straight to initial impressions. What are you thinking straight off the bat.Â
Jonny: Yeah, it's a good question. I think, as I said before, I think it's fun and I think it's really impressive and I think it's, it's really helpful to kind of demonstrate to the wider world.
[00:05:00] Just how far AI has come in recent years and, and just how much further there is to go, but opening up kind of the ideas and concepts and, and kind of starting to provoke people into kind of potential practical use cases for it. You know, it's really fun for content creation. Like you said, you can write random poetry.
I got to write some LinkedIn post for me about legal tech which were actually pretty good. And probably better than, than a lot of a person that I read most. It, it's also really powerful that you can get it to write code as well. I dunno if you play with that part of it yet, mark. But you can, it'll actually write kind of code for you to, you know, create websites, great applications which is really powerful.
And I think. I think it's a good kind of reminder for people and a good recognition. It puts it in the hands of everyday people so they can see just how powerful and practical it is. But it also serves a good reminder as to just how far AI is coming, kind of more generally. So, you know, there's other use cases going on in the world like image recognition now being used by doctors to, for kind of a diagnosis of cancer.
And it's picking up on kind of things that doctors otherwise. miss ,in kind of CT scans and things, which is really impressive. [00:06:00] Things like fully autonomous driving vehicles as well are now a really real thing. There's lots of cities globally that are now kind of piloting and have you know, tens, hundreds of fully autonomous cars driving around cities, picking people up.
So I think it's, it's, yeah, a lot of that you don't see it kind of sits in the background unless you're really interested in that kinda subject matter area. So it's just a really fun way of kind of bringing it back to. Everyday people who can log on and have a little bit of fun with it. But I think it's still important to note a bit of a minute.
Some of the limitations we'll probably talk about in a bit. It, it is just a bit of fun right now. And we've gotta be really careful not to get too carried away, particularly in kind of its current. Its current guise .Â
Mark: Yeah, absolutely. I mean, my impressions are, are similar to yours. I haven't used the code part of it, but I know it's kind of the basis of GitHub's co-piloting feature, which I know a lot of coders are very impressed with.
And you're right, I think it's a very prominent newsworthy application of ai and I think it's think it's captured so much attention because, because of the fun element, but also. It is, it can genuinely write quite good stuff. That was my, [00:07:00] my impression. Now it, it falls down very quickly in certain areas like maths, which we'll get to.
But, but it is very impressive. Now most of our listeners are lawyers, so let's get into the applications for lawyers and law students. And, and that's your also your area of expertise. Johnny, being a AI expert. One thing before we, before we get into, into your thoughts and the applications, I did see that it did just pass the bar exam that someone fed it, some of those questions, and it, and it got 70%, which, you know, if you were randomly picking, it'd be almost impossible to get that.
So I know there are some very obvious applications for law students out there. But, but maybe we could, we could touch on what it means for you know, qualified lawyers.Â
Jonny: Yeah. So I think and, you know, A little bit. I guess we could talk about the theory a little bit in some of the limitations and, and how it works and I guess why it's able to do some of what you've just discussed.
And, and essentially that's because it's, got a lot of training data and so presumably in that training data there were probably some good examples of bar exam could've been written before, which are good quality ones. So essentially all this [00:08:00] is, is it's a little bit like a search engine.
It just enables that kind of natural conversational piece of a front end to it. So essentially somewhere sat in the corpus of information that it's being trained on. Will there been some good answers to those exam questions? So essentially all it's doing is recalling that for you and pulling it back and possibly piecing a couple of pieces together.
So in theory that's, that's how it's able to do that. It's not necessarily intelligent, it's just good at finding information.
It's just good at finding information and recording that information. And, and I think, you know, the current practical Uses for lawyers are fairly limited because it, what it hasn't been PR provided and trained upon is lots of legal context and content. But I guess this is sharing the future of where it can go.
So AI is already used inside of law for kind of legal research for big kind of providers like Lexus Nexus and Thomson Roy as an another, as a kind of using AI to help improve searches. But it's still relying on that kind of search engine frontend. So I think in time where we'll get to is. It will become kind of a powerful research [00:09:00] tool.
It will become a powerful document drafter and content generator. And in theory, a lot of this is, is kind of possible now, but what we need to do is then connect up these algorithms to really good quality database and data sources so that then it becomes kind of reliable information. There's still, there's still that argument as to, you know, where a lawyer kind of makes her money and makes the difference is by interpreting that information, providing that advice on it.
But it will be a really powerful way of kind of getting that information to those lawyers' fingertips kind of much more quickly and a lot more easier than we can today. Cuz most lawyers I talk to don't even really know how to use search engines properly. So I think that's the kinda enabler that this will be.
So I think another few years time as these algorithms improve and as we connect it up to kind of good quality database. You know, we will start to see it really assisting with kind of those, you know, legal questions and answering kind of legal questions based on that knowledge that Sat sat in its database.
Mark: That's right. Uh uh, Richard Trumans from the artificial lawyer wrote. A good piece last week, and he kind of, he, he, he kind of viewed applications as q [00:10:00] and a, but like you, what you were just talking about it, it summarizes stuff pretty well. I think you could put in, you know, you could put in cases or case law and get it to summarize stuff.
It does text completion like people might be familiar with in other more common cases like mobile, phone, text completion, but it's a bit more advanced than that. Potentially whole document creation. But I think you are right. The, the data set it uses. Mainly Google and Wikipedia and books. I, I don't think it's been had access to proprietary information such, such like that held by LexisNexis and Thomson Reuters.
And because of that doesn't have maintained information. You definitely need to be careful of the accuracy. I think it gives a lot of incorrect just wrong answers. So we need to be, be conscious of that. One really interesting application I saw today was the DoNotPay. This is I guess, some more consumer law applications than, than commercial or in-house legal.
But on the consumer side, the, they got the basically a ChatGPT bot to successfully negotiate a reduction with a [00:11:00] Comcast, a provider in, in in America. So there's some interesting applications around potentially negotiations and even at a point where you may. Bots or AI trained models negotiating with each other on, on behalf of companies who've trained it with their various positions and risk threshold.
Jonny: Yeah, a absolutely, and actually there's it's a little bit kind of tangential to law. And to be honest, I don't think he's getting enough coverage at the minute in the legal sector. But very huge investments being made at the minute around kind of logistics technology where kind of Maersk and various of a big organizations are now investing heavily and as various kind of venture funds.
Setting up procurement negotiation technology that's based on ai. So for those repetitive negotiations using AI to negotiate on behalf of both parties to come to like an acceptable position, like that's really real and is happening today. Not necessarily , from a legal perspective, but certainly on those logistics contracts that's here and now and is developing fairly quickly.
So you can definitely see a world not too far away where. Start to [00:12:00] happen. And it, and you know, it comes back to a lot of the principles then around automation. Automation in legal has been going on for 10, 15, 20 years probably further. And it, it all becomes about the right use cases. So at the minute, what we're automating are those kind of high volume, low value contracts where we're trying to take the lawyer out of the room and introduce governance for other people to enable that. And that's where we'll be with ai. It's still not gonna fully replace humans. It's still not gonna fully replace lawyers, and it's probably not gonna be doing the really kind of sophisticated high value negotiations where there's a lot of other things that come into this around kind of commerciality and human intelligence and human emotion. And, and so all of those tactics, you know, AI is not gonna be able to interpret and, and respond to.Â
Mark: Absolutely. And I think that that's some, some good summaries of some applications. And I think, you know, we, as you mentioned earlier, we'll have to tread carefully for the moment. But it's not too hard to imagine in a couple years' time when you have, you know, specific legally trained AIs to be able to do a lot of what we're talking about [00:13:00] there. A lot of the negotiation. Whole document creation, q and a and things like that.
Let's quickly talk about the limitations, cause I think they're powerful and, and worth talking about. I wanted to start with um, Sam Altman's, the CEO of OpenAI, and he tweeted a couple of days ago. "ChatGPT is incredibly limited, but good enough at some things to create a misleading impression of greatness. It's a mistake to be relying on it for anything important right now. It's a preview of progress. We have lots of work to do on the robustness and truthfulness."
I think that's a pretty good place to start in terms of limitation, . What are your thoughts?.Â
Jonny: Yeah, I completely agree. In terms of the limitations you know, we touched on it before. It's limited to the data that it's being trained on. So that content hasn't been curated. It's not being curated for a specific purpose, it's just a whole bunch of stuff, kind of from the internet information that's available.
That's been pulled together. And there's even kind of a timing point on that as well. So I think the most recent information that's in that database was from around September, 2021. So it's already [00:14:00] 15, 16 months out of date. So there's a point around that in terms of a quality information that's in there.
I think the second point to, to note as well that we've got to remember is at the end of the day, this is just a statistical model and it's all based on probabilities. So where you were talking before about it, you know, quite simply getting things wrong that speaks, it's based on probabilities and it will make mistakes.
I mean the same that humans will the same but subject matter experts do. If you've not seen something before, but you might not know about it, you know, it's the same with lawyers. If there's a new piece of case law, Last week that the lawyer hasn't quite seen yet then their answer is gonna be incorrect because we've not got the latest information to kind of include in their response.
So, you know, there's a, there's a point around kind of timing. There's a point around kind of probability and getting the answer right. And I think the other key thing as well is whilst it's generating human-like responses, you know, it's not sentient and it doesn't actually understand and properly understand or comprehend the meaning of the words.
So again, what it's producing back to you isn't always relevant and perfect because whilst on the algorithms it can kind of interpret. [00:15:00] based on the words that you've got there, what it thinks you're trying to say, it doesn't properly understand the true meaning. And, you know, we see this with humans, you know, different kind of dialects, different people kind of translating from different languages.
You sometimes lose that meaning by not understanding kind of the local, you know, the local colloquialism or whatever it might be. And that just becomes even more prevalent when we're kind of dealing with ai. So, you know, we've gotta remember that whilst it looks like it's kind of been really intelligent and human-like it, it's not quite there.
Mark: Absolutely. And as you say, it's, it's based on probabilities and if my World Cup tips or anything to go by, that's not a guarantee for success. There's, I, I think you may have been alluding to it, but I'm gonna add a link in the show notes to a, a paper called: "On the dangers of stochastic parrots. Can language models be too big?"
There's a really good paper. I mean, it's a little bit dense potentially for some, some people, including myself, but there's one, one part where they describe a language model as a, and I'm just gonna read it verbatim: "It is a system for haphazardly stitching together sequences of linguistic forms. It is a, [00:16:00] it has observed in its vast training data according to probabilistic information about how they combine without any reference to meaning. A stochastic parrot.", and I think that's, I can't put it better than that, but that's basically what you're getting at, right? It's just, it's not, it's not actually meaning, it's just probabilities and, and putting words or tokens together.
Jonny: Yeah, absolutely. I that, that's it better than I did. I'll turn back for next .Â
Mark: Yeah. As I said, I'll add a link to that paper. It's, it's, it's worth reading and, and definitely goes into a bit more detail around the limitations, but also some of the implications. In terms of finance and, and environment, and also some of the dangers of, I guess, biased information being used to, train the model.
That's, I don't think we'll go any more into that. It's a little bit beyond the scope of this podcast. As I said, I'll put a link in the show notes to the paper for those who wanna dive a bit more deeper into the limitations surrounding large language models. Penultimate question for us, Johnny, is if we cast our eyes [00:17:00] forward, what do we think it means for lawyers in the medium to long term?
Jonny: Yeah, I think it's a really good question. And you know, like all new or exciting technologies you know, it's still a concept, but still a lot to be worked out. We still need to think about how we implement this and how it can be used properly. But I think. The really interesting bit for me is, is, is that this has a pathway to enabling lawyers to interact essentially with information systems, but at a human level.
So, you know, as I said before, you know, a lot of lawyers aren't it trained and we don't understand how to interact properly with things like search engines. So removing the need from, to have that knowledge about how to do search properly. Is a really powerful thing to do. So they can just ask straightforward questions in the language, but they'd naturally ask it in and get the answers that the otherwise we'll try and get from some of those legal databases.
I think that in itself is extremely powerful. And we need to, we need to kind of not overlook that it might be slightly more boring than some of the use cases, but we're seeing out there in the real world. But actually it's really real and really powerful for, for kind of [00:18:00] lawyers. And I think it means that you.
In turn, it will enable lawyers to, to do kind of better researching. And I think you know, we'll see a world where this kind of system will then be helping with doing kind of research tasks, generating and creating new documents. And I think you touched on it earlier, but even providing kind of guidance notes or advice as well.
So a lot of lawyers today, like we've gotta remember a lot of a lawyer's job today is working from precedents that have been set before, whether that's kind of in case law or in documents or in different forms. Even our advice notes that lawyers are producing for clients often come from lots of advice notes that have gone before and comp compiling that and compiling the best ones and applying it to the facts.
And I think you could start to see if you think about how this, these systems are working and the stuff that we've talked about today. You could see a lot of parallels there. So you could see the starting help lawyers kind of coll, collate all of that information to produce kind of advice notes going forward.
I think in the consumer world as well, and again you touched upon this earlier, but enabling you know, non-lawyers, people who don't understand legalese, to be able to interact with kind of legal content and [00:19:00] ask questions around what does this mean? What does this mean for me? And getting some kind of form of answer back again is, is really powerful.
And, you know, touched upon the whole kinda access to justice point, which is, which is really important. And that's where organizations like do not pay. They're doing some really cool work at the minute. So I think, I think lots to come. I think lots of other kind of tangential technology and other developments that will happen alongside it, but it's just another example of, of seeing how kind of a world is changing and how this will have impact on law as many other kind of professions and, and areas.
Mark: Brilliant. Well, I can't add anything more sensible than that. So I wanted to end the podcast, Johnny, by saying, are there, is there anything you think we missed? Any other points you'd like to make before we sign off?Â
Jonny: I mean, there's, there's lots we could talk about, but I think you know, I think it's interesting.
It's here. It's exciting, it's gonna open up a lot of opportunity, but we just need to remind people that, you know, it's not replacing lawyers just yet and there's still a lot of work.Â
Mark: Absolutely no one's outta the day job yet. , we'll be watching this area with a lot of interest and if anything more dramatic happens, we'll be doing another podcast, I'm sure Johnny.
[00:20:00] But it just leaves to say, thanks so much for joining us today.Â
Jonny: Thanks, mark. Thanks very much.