YouTube transcript summary
Prompt
How to Setup DeepEval for Fast, Easy, and Powerful LLM Evaluations 0:00 one of the most important and overlooked 0:02 aspects of deploying llms in production 0:05 is evals Now setting up evals can be a 0:08 little bit tricky but it's a completely 0:11 necessary step if you're actually going 0:13 to use llms across your company in your 0:16 product or in production at scale so 0:18 today I'm going to show you a quick 0:20 tutorial of one of the most popular 0:22 evalve Frameworks out there today which 0:25 is deep eval deep eval is an open-source 0:29 evaluation frame workor for large 0:31 language models and it comes out of the 0:33 box with a ton of different evaluation 0:36 methods that you can use or you can 0:38 write your own evaluation metric so in 0:41 this walkthrough we're going to explore 0:42 the fundamental concepts of deep eval 0:45 how to set up your development 0:47 environment to be able to run these and 0:49 we're going to create our first eval 0:51 test let's Dive Right In okay so we're 0:54 here in the terminal and the first thing 0:56 we need to do is create a folder to hold 1:00 all of our work so we can uh create a 1:03 virtual environment to go in this folder 1:05 so the first thing I'm going to do is 1:07 create a folder called DP Val test and 1:11 then I will navigate into that folder 1:15 and so now at the top of the quick start 1:19 introduction of the documentation for DP 1:21 Val there will be a little bit of code 1:24 to create your python virtual 1:26 environment so all I did was uh paste 1:30 this code and now we've got our virtual 1:33 environment and we can see right here 1:34 we've confirmed that we have the virtual 1:37 environment so the next thing we're 1:39 going to do is PIP install DP Val itself 1:43 into the virtual 1:45 environment and while that is running 1:49 the last step before our environment is 1:52 complete is we have the option to log in 1:56 to the Deep 1:58 eval web UI now this is completely 2:01 optional and it's free by uh confident 2:05 AI which is the people who created the 2:06 Deep eval framework but we have this 2:09 option to essentially instead of just 2:12 looking at our evals in the terminal as 2:15 it returns naturally we'll have this 2:17 nice interface that I'll show you later 2:19 where you can actually break down and do 2:21 a little analytics and have a nicer um 2:24 situation so right here everything is 2:27 installed if you want to just skip to 2:28 the next step and not use the web UI you 2:30 can but if you want to use it you'll run 2:33 the command dbal login it'll basically 2:36 ask you to authenticate and then you're 2:39 ready to actually create your first test 2:41 so let's go in and create our first test 2:45 let's actually dive into the code that 2:48 will run our first eal so I already ran 2:51 the command uh touch text example.py 2:55 which will create a file with that name 2:59 um for us to hold our code and so we can 3:02 just paste this in this is directly from 3:04 the quick start documentation and let's 3:06 actually walk through what this is so 3:10 first of all um the metric that we're 3:13 evaluating on is the answer relevancy 3:16 metric and so basically what this does 3:19 is it takes a uh a user input and then 3:24 what we want um to actually you know 3:27 respond with and it checks to see if the 3:31 response is relevant to the user input 3:34 so we have here the test case this would 3:37 be where the user input would go which 3:39 is what if these shoes don't fit and 3:41 what this answer relevancy metric 3:44 evaluates is did the llm 3:47 respond with a relevant answer so what 3:52 if these shoes don't fit says a user of 3:55 our application and the llm says we 3:57 offer a 30-day full refund at no extra 4:00 cost that to me seems like a pretty 4:03 relevant answer and we can actually 4:07 modulate the uh threshold of relevance 4:10 from zero being um you know even if it's 4:14 completely irrelevant it's going to pass 4:16 and then all the way up to one which is 4:18 it needs to be extremely relevant and 4:20 specific so I would anticipate that this 4:22 would be on the higher end of uh 4:25 relevance but to give an example to make 4:27 it more relevant we could say we offer a 4:29 a 30-day full refund on all shoe 4:33 purchases at no extra cost so now we see 4:36 we've in improved the actual you know 4:39 llm response to be more relevant so this 4:42 is the code but before we actually run 4:44 the eval as a test there's one step we 4:47 need to add we need to add our open AI 4:50 key as an environment variable so that 4:54 gp4 can actually run this answer 4:56 relevancy metric so let's do that very 4:59 quickly 5:01 okay I'm flipping to the documentation 5:02 very quickly because this is a part 5:04 where some people might get tripped up 5:06 so we've been working through in this 5:09 video the quick introduction o on the 5:11 Deep eval uh documentation page so we 5:14 set up our python environment we 5:16 installed DB Val we logged into the 5:20 confident AI uh web UI platform if we 5:23 wanted and we've got our code and the 5:26 next step if we're following along 5:28 directly in the documentation is to run 5:30 this command deep eval test run um on 5:34 the file that we just created well it 5:37 says that our test case should have 5:39 passed but if you actually ran this 5:41 Command right now you would actually 5:44 fail because we have not connected uh 5:48 open AI as uh an llm to run the eval 5:53 itself so we need to navigate over to 5:56 platform. 5:58 open.com ap- keys and create a new API 6:02 key right up here so you're going to 6:05 create this and Export it as an 6:07 environment variable in Python so you're 6:11 going to run the export command and um 6:14 you can see right here we're going to do 6:17 export open a aore aior key equals and 6:23 then paste in our um command so you're 6:25 going to see that at the top of the next 6:28 segment but this is a really important 6:29 important uh sticking point because we 6:31 don't want you to get errors whenever 6:33 you run your first eval so make sure you 6:35 create a new private uh API key for open 6:39 Ai and Export it as an environment 6:42 variable so we can actually run our 6:45 evals great so I exported my open a AI 6:49 API key as an environment variable and 6:52 now we're ready to run our eval so we 6:54 just run D eval test run and then the 6:56 name of the file so I'm going to hit 6:58 enter and let's watch as yeah we've got 7:02 a good response and it's running now and 7:05 it's complete we've done our first eval 7:07 it actually told us how many tokens it 7:09 took to um uh actually run the eal it 7:13 told us the score was one basically the 7:15 maximum score and gave a nice little um 7:18 explanation here now it says at the 7:20 bottom run dpal login to view evaluation 7:23 results so if we were logged in it would 7:26 actually give us a URL to open into the 7:29 confident AI web UI so let's go and 7:33 check that out right 7:34 now so let's just log in to The 7:38 Confident AI UI which pairs perfectly 7:42 with the DP valve framework set so go to 7:44 confident D ai.com log in and you see a 7:49 nice dashboard here of stuff I've got 7:52 mine empty right now it's just a dummy 7:53 account but if we scroll down to unit 7:56 test we can actually see that we ran 7:59 some test and if we click into them 8:01 we're able to um go and take a look and 8:05 look at statistics so this is a great 8:08 application um that allows you to 8:11 interact more deeply with the data 8:13 especially at scale and it tells you how 8:15 much the test cost and everything like 8:17 that so if you're doing bulk data test 8:19 we only ran one but if you're doing bulk 8:21 this is really necessary and if we 8:23 actually click on test cases we actually 8:25 can see what type of uh eval we ran what 8:31 was the input what was the output 8:32 whether it passed how long it took to 8:34 run so definitely recommend using the 8:37 Deep eval login feature and using the 8:40 entire web UI they built for you because 8:43 it really allows you to dive deeper into 8:45 the analysis so hopefully you feel 8:48 comfortable uh getting started with DP 8:51 Val as an eval framework and walking 8:53 through the documentation and building 8:56 your own evals using their native evals 8:59 as well well to improve your prompts and 9:01 your production llm applications --- summarize in structured, hierarchical way, clearly highlight each claim
