Shiplight, New Vibium Release,Pydantic AI and Playwright and More TGNS194

Speaker A: Is 60% of your week going to playwright maintenance? And AI wrote you hundreds of tests, but other selectors are used, even real and your build is green. But can you actually ship? Let me know in this episode of the Test Skill News show the week of August 24th. So grab your favorite cup of coffee or tea and let's do this. If you ever watched a week disappear to Playwright Maintenance, start here, because this week's sponsor built their whole product around this exact problem. Let's check it out. So here's a tool that just launched. It's actually free for now, so you definitely should check it out because Shiplight has just launched an agentic native testing platform platform aimed squarely at anyone losing their week to Playwright Maintenance in One of the first things I notice is the testimonials. One is from hey Gen's head of QA, which reports going from 60% of his time on playwright upkeep to zero the following month. And I got a chance to meet the co founders Will and Fang. They come out of Airbnb and Meta and Fang worked on Chrome and V8 from day one, so he really knows his stuff and he told me, and I think I've seen this as well, is that AI coding moved the bottleneck from writing code to actually verifying it. So this work is Shiplight hands your coding agent a real browser, so the agent walks your app and writes tests as plain English YAML and then lives in your repo, and then also writes playwright uh at runtime. So the subtle steps then replay from a located cache. So it saves you money on tokens, so no model calls. So you run at full Playwright speed. And when a locator goes stale mid run, Ship Light resolves the original intent again instead of failing the build. So real regression still fails loudly, but all the noise is handled for you. All right, this sounds cool, but let's check it out for ourselves really quick. Or you could just copy and paste and use your local LLM, but I'm just going to start for free because you get a whole bunch of other things when you do that this way. Right after registering, I create an API key and I'm just going to run the command for cursor. It's all configured. I'm going to run a test and Shiplight create YAML test and what's really cool about this that, uh, Ship Light didn't just add Playwright. It turned my test tool Matcher code into a testable product with the spec locators captured from a real browser walk and two YAML tests that already passed against my local install, all without me doing anything, just letting the AI do the driving so it's easy to get started. I highly recommend you start for free using that link down below. Check it out, let me know your thoughts. So next up is a major announcement from Jason Huggins. He just shipped Vium 2.6.8.21 in the release notes. Call it the largest feature release since version one. And looking at the release notes I think the main thing that pops out is Firefox the engine Firefox flag now drives Firefox end to end auto installed on macOS and Linux and also cached alongside Chrome for testing. Navigation captures all work on Firefox now. Jason credits this new functionality to the webdriver Bidi spec and Firefox shipping official support for it this week. And this is something we covered last week with the newer version of Selenium announcement and the next one that pops out to me is the recording feature. So it has a recording dot start which gains a video track on every single surface captured natively over by die screencasts rather than stitching together from screenshots and some other features as well. They should definitely check out as well using that link down below. So the next one is about an open source tool released by Phil Chen whose release Verify, which is an open source claude code plugin under the MIT license that turns a feature description into review test plans and and running playwright test. So the pipeline is five steps Test plan, writes a plan from your feature description, audits, read your local source for stable selectors and grades how testable the app is. Discover opens the app in a real browser to verify those selectors, implement, writes those playwright specs into your repo and executes, runs it and tells you whether a failure looks like a selector that moved or is an actual bug in your app. But Phil is really honest. He says the test plan does not crawl your app and it doesn't parse a spec, it reads your description, reasons about it, fills in a template. So it really is more about educated guesses rather than what I showed you earlier with Shiplight, which really understands your code base and gives you really real code and real locators. And Phil also mentioned that he designed this to put you in the loop on purpose. So every step writes a plain markdown file and stops before the implementation will run. A human has to resolve every to do in the plan, set open questions to none and mark it approved by name and date. All right, another thing that caught my attention once again is Serenity, they're always upgrading this. This is from Yarn, who just released a new version of their HTML reporter and as always it's open source and free. And Yann mentions it works with Playwright, Test, webdriver, IO, Cucumber, Mocha and Jasmine with no migration required. And the centerpiece is something he calls the confidence score, which calculates automatically from your current and historical test results using pass rate, consistency and completeness. This framing is that instead of debating whether a build looks good, you get an objective signal grounded in your actual run data. You also get a full activity trail with screenshots, HTTP exchanges, video and stack traces. Deals are grouped by root causes so you can see whether you're dealing with one problem or 10. And it flags which tests are inconsistent, degraded or recovering across CI runs with the history to back it up. And it tracks confidence, pass rate and consistency over time with trend reports as well as and along these same lines I found actually someone talk about this release that got a response I think is worth hearing more about about this reporter and it's from Jan and this from a different Yarn writing on um, Pilot Period says the release is a good reason to ask an uncomfortable question. What is it? Well, it's that if your test report only tells you what passed and what failed, what you have is an automation log with a nicer HTML. And what he highlights here is what the test actually knows. So a reporter can't recover intent that was never expressed by your test. So if your test only describe things like navigating, locating, filling, clicking and asserting, that's all any report can hand back. And he points out how Serenity JS can present capabilities, scenarios, task interactions and assertions because those concepts exist while the test was running. And so I like how this really highlights once again how you're writing tests. You should really write them in a way that when they fail, when you get a report, you can actually see what is happening, what the intent was, rather than diving into the guts of your actual test script. All right, this isn't about testing, but I think it's a really cool use of Playwright that your agent can use to use your browser for other type of activities as well. And this is from Pamela Fox at Microsoft who's published a walkthrough on Azure Developer Community Blog that turns Playwright into an agent capability. So Pydantic is best known as a Python library that validates data types using hints. Pydantic AI applies those structured type safe approaches to building AI, uh, agents, while Pydantic AI harness supplies ready made capabilities those AI agents then use. So in this example she gives its Playwright browser capabilities, given the agent tools for things like navigating pages, clicking, typing, reading page content, and running JavaScript and taking screenshots. The example she gives is the example prompts an agent act like a careful manual tester, plan its work, find high value usability and functional issues, and then save it as a markdown report. It also shows guardrails, restrict which domains it can reach, block private addresses, and limit actions and navigation times. These reduce risk, but aren't, uh, a complete security boundary. She also goes over how you can use Open Telemetry, which records the agent's browser actions, page visits and token usage. A coding agent can then review that trace and suggest better prompts and settings. So this gives you a really cool record of what pages are browsed, what commands ran, and how many tokens are burned. Last up is from Umesh, who I've been following for years. I found this on LinkedIn. Once again, he works as a platform engineer, but he started off as a test automation engineer I believe, and he points to an A16Z chart showing agents consuming near five times as many tokens as humans. So with agent usage up roughly 14 times since February. So his concern is what enterprises do next. Take their existing APIs, put an MC layer in front of it and expose that to an agent. But APIs are deterministic and LLMs are probabilistic in a model, dozens of operations and large payloads and you're giving it many more paths to evaluate before it gets to where you really want. So I think it's a really great story how maybe to watch out for different ways that your AI can be consuming tokens that not necessary. All right, so that's it for this episode of the Test Skill New show. I'm, um, Joe. My mission is help you succeed in creating end to end full stack pipeline automation awesomeness and as always, test everything and keep the good. Cheers.