Seeing the Forest for the Trees: Why Accessibility through Automation Requires a Clear Vision and Holistic Process

FINISHED FILE

M-ENABLING SUMMIT
WASHINGTON, DC

OCTOBER 24, 2022
4:15 PM ET

STUDIO B
SEEING THE FOREST FOR THE TREES: WHY
ACCESSIBILITY THROUGH AUTOMATION REQUIRES A CLEAR
VISION AND HOLISTIC PROCESS

Services provided by:
Caption First, Inc.
P.O. Box 3066
Monument, CO 80132
800 825 5234
www.captionfirst.com

***
This text, document, or file is based on live
transcription. Communication Access Realtime Translation
(CART), captioning, and/or live transcription are provided in
order to facilitate communication accessibility and may not be
a totally verbatim record of the proceedings. This text,
Document, or file is not to be distributed or used in any way
That may violate copyright law.
***

MIKE PACIELLO: Okay. We want to welcome everyone here. Welcome, everyone. This is the session on Seeing the Forest for the Trees: Why Accessibility through Automation Requires a Clear Vision and Holistic Process. So, if you came here for chocolate ice cream, you are in the wrong room! My name is Mike Paciello. I am playing a dual role today, both with WebABLE and Pearson Education. We put together, I think, a great panel. One of the reasons why I asked the four gentlemen here with me today, because they represent vertical markets in the accessibility industry, that are critical where automated accessibility is in high demand, but also is a strong play for machine learning and artificial intelligence.
You can hear, I think, through our various members of our team. The title of the session is Seeing the Forest for the Trees. I always thought it was see the forest through the trees. I am still feeling like that is what it is, but I had to look it up a couple times and read it. We are going to talk about why automated accessibility requires more than just AI and machine learning. These gentlemen will be able to I love being a moderator, right?
I don't really have to say anything smart. I just have to read my slides. It is these guys. The focus of the panel session involves a discussion on the accessibility of automated software platforms. This panel will address key accessibility and usability requirements involving automated systems as an accessibility solution, and they will consider recommendations to ensure that existing and I really want to underscore existing, because there is a lot out there that future automated platforms support a holistic and engineering and accessibility lifecycle.
This is a big challenge. Automation is basically about doing things without human interactions in the long run, moving humans out and moving machines in. But we still need to focus on humans. We service our constituency, our people with disabilities. Our clients serve people with disabilities, so what we talk about and how we interact and the things that we are trying to enhance to ensure the usable and accessible I had this discussion with Lionel last night.
I said it is usable and accessible. Accessibility can't stand on its own. It doesn't work for people with disabilities. If that is a new concept for you, that is true. It doesn't. An image with an ALT Text can be accessible, but if the ALT Text says it is a stop sign, but, in fact, it is a green light, there is no context, there is no usability to it, right? A simple analogy.
So, it is usable and accessible. You will see the Section 508 Committee, that was a key theme of that particular Board. So, let's talk about our market and why it is so important. These are statistics that I pulled off of the web.
The global accessibility testing market is valued at some $481 million, and it has a growth rate of 4.5%. That is from a company calls straits research on the testing market. This company is a market research firm. Their job is to go out and figure out, is there a business here to invest in. They have vested interest in making sure people have the right information the population of people that use assisted technology to navigate the web we are talking more than just the web today, is over $350 billion in size, that is a US Census Bureau stat.
By 2023, the number of people with disabilities employee, employee, not users, but employed, will triple due to artificial intelligence and emerging technology. You may not have thought that, especially since we just made a disconnect between machine learning, AI and humans. In fact, this report is saying due to AI in emerging technology, machine learning, for example, and other things, brain interfaces and what not, we will see triple growth in the employment of people with disabilities. That is a very important aspect.
So, I already introduced myself. Let me introduce our panelists. I am actually going to have our panelists from left to right introduce themselves. We will start with Navin Thadani. Please?

NAVIN THADANI: Hi, I am Navin Thadani, the Co Founder and CEO at Evinced, accessibility into development processes is what we do.

ANTHONY FERNANDO: Hi, I am Anthony Fernando, the Accessibility Compliance Director at the Pearson on the accessibility side of the business. I am excited to hear about what others are going to say about, you know, automation to a degree. It is not something I knew 13 years ago as a tester. Now I am so excited about these offerings and I will share some of these aspects, as well.

JOSH MILLER: I am Josh Miller, Co Founder and co CEO of 3Play Media. Our focus is to make it as easy as possible for a media manager, media producer, media publishers to make their audio and video content accessible.
We are using a combination of machine learning, natural language processing techniques and other characteristics, along with humans, to achieve what we talk about as quality at scale. That could mean it is live. It could mean that it is thousands of hours every month. It could mean it needs to be back in two hours.
So, there are a whole number of ways we think about that. All of those come into play all at the same time, so we need to balance all of that and make sure we are able to get people what they need and make sure it is accessible and easy.

CHAD CHELIUS: My name is Chad Chelius, I am the Director of training solutions and Principal at CHAX Training and Consulting. My business partner and I focus on document accessibility and help train companies and help them overcome the challenges of making their documents accessible.

MIKE PACIELLO: Thank you, gentlemen. We will talk about three areas of technology discussion today. Josh just talked about captioning in media. We will talk about publishing and eDocs, which is what Chad is here for and we will talk about publication in web and apps which is what Navin and Anthony are here for. You are not missing anything on the screen. It is just text up on the screen. There is one table that Anthony will speak to based on a rubric that his team developed at Pearson. He will do the description at that level.
We wanted to do a quick poll. I will ask Nicholas if he can come up. It is just a two question poll. It is not your left hand and right hand. That is what I can tell you. Give us a little idea of what you folks are working on. Go ahead, Nicholas.

NICHOLAS GENNARO SCIASCI: Thank you, Mike. So, throughout the audience, you will see there is a piece of paper with a link, as well as a QR code. So, if you could just help your neighbors with scanning the QR code for access to the first question. I will let you know what the options are.
The first question is the audience currently automating anything including user interface texting? You need to respond to 22333. If you would like to answer yes, the code is 57620. If you would like to respond no, it is 57633. I am happy to repeat those codes for anybody that might need them.

We need one person to repeat it. Okay? Everywhere got it in? Anyone need a little extra time? All right. Let's go with the second question.

We have a verification question. Are you saying am I using this on my own personal phone, or does anyone in this room use AI, or UI, excuse me.

I think it is general, anyone using it.

NICHOLAS GENNARO SCIASCI: Second question?

MIKE PACIELLO: Go ahead. I think so. Just getting it set up. Numbers the next question is, are you completely new to automation, or are you here to increase your knowledge about automation systems, performance and usage? Including functional UI testing for websites, and mobile apps? And response yes, you text the letter A. To respond no, the letter B.

I have a clarification question. This is a two part question. So, which would you like for us to respond to?

NICHOLAS GENNARO SCIASCI: The first question that was previously on the screen, in terms of

MIKE PACIELLO: Let's talk about this question. It is one question with a choice, either/or.

In regards automation, yes or no?

MIKE PACIELLO: That is a good point. I can see where the confusion is. Yeah. Yeah. It should be yes, no, it should be A or B or something. So, if you are completely new to automation, answer yes. If you are here to increase your knowledge about automating systems, answer no.

Got it.

MIKE PACIELLO: I will give it a couple minutes. I didn't see the results of the first poll.

NICHOLAS GENNARO SCIASCI: We will pull the results.

MIKE PACIELLO: Are you able to see what we are looking at here? A lot of this is knowledge. Awesome. Awesome. A high degree of automation. Good, awesome. This is to help these guys understand what they need to talk about. All right. Thank you, Nicholas. I appreciate it.

NICHOLAS GENNARO SCIASCI: You are welcome.

MIKE PACIELLO: Thank you, all. So, first question to our audience. I feel like I am in a game show. The first question to our audience. What are the accessibility and usability benefits of automated technologies? I will start with Navin, because he has events, this is a big part of their solution. Go ahead, Navin, please.

NAVIN THADANI: Yes. The way I think about this, is that you can be big, or you can be nimble, but in order to be big and nimble, you have to automate. You have to automate your accessibility testing, otherwise you are just not going to be able to keep up.
But the important thing is that you have to do it right. Because and so far the reality is that the industry has not had tools and technologies in order to do it right. And things can do really awry if you try an automation effort, and you don't have the right tools and technologies.
I know off of very sophisticated software company in fact, quite a few of them in this one particular example the company had 15 to 20 engineers working from 12 to 18 months, depending on how you measure it, on an automation effort. I would say the results were spectacularly bad. In that, out of the tens of thousands of functional tests that they ran, you know how many actually made it through the accessibility testing part? Just two. Okay?
So, you need the right tools, you need the right technology, and you have to do it right, otherwise it has such a negative ramification, that go trying your engineers after that to do it again, and they will not do it. You have lost quite a few years in terms of accessibility program if you try to automate in the wrong way.
It is way in which you automate that is really important. I will talk about three phases of the industry a little later, but I do want to give my colleague on the panel some time to tell us what they think about it.

MIKE PACIELLO: And we will leave time on the back end for the panelist questions. If it is moving fast, I apologize. Anthony?

ANTHONY FERNANDO: For me leading a team with testers, the number one thing for me is to introduce productivity. Often times we get to see a lot of pause that decreases our productivity. I don't want to use that kind of tool. My team doesn't want to use it, but over the years I have seen advancement of this technology, which is great for the industry.
Another thing is even though we use these tools, especially in my team. I have several folks with a disability, so, if your tool is an accessible, I can't even think of purchasing it, period. They need to put more effort into their tool itself, making that tool accessible.
Some tools are way ahead. That is great. If not, you need to think about getting more user feedback, actual feedback from people with disabilities. And as Mike mentioned earlier, you may be compliant about, you know, if it is not usable, again, it is not a go forward for me, right? So, that is another thing to think about as tool providers.
I am not targeting any specific folks here, but it is something that you need to think about. User accessibility is a must, end to end workflow, right?

MIKE PACIELLO: Thank you. Josh?

JOSH MILLER: I will echo what has been said here. Ultimately it comes down to scale. How are we able to repeat the process, create a repeatable process so it is somewhat standardized? The only way to do that is with technology. But I think what we have started to see, there is a lot of hype and kind of excitement around some of the automation. We can kind of lose the usability issue and the accuracy and quality of what we are trying to achieve, so there has to be a balance.
I do believe that strongly, a pragmatic approach does not mean a black and white approach. In thinking it through very, very carefully, where you apply the technology, where you apply the human, is what will ultimately lead to the success. And understanding where the technology can perform really well as opposed to just perform.
I think that is a huge part of what we think about all the time, is speech recognition is front and center in our world. When does it do well? When does it not do well? And understanding that and being honest and open about that.
What we don't want to do is talk to a customer who says I need to save money, so I need to use speech recognition. Well, maybe. Maybe you are right. Maybe your content is really well suited for speech recognition, so you can keep your budget down, but it might not and we as the vendor needs to be really honest about that and not just say yes to whatever the customer says all the time.
Because usability is very, very real, especially as automation gets introduced.

MIKE PACIELLO: Thank you. Chad, talk to us about the challenges as it relates to e DOCX in publishing?

CHAD CHELIUS: Sure. The advantages I see in automated technology is a consistent application of accessibility standards. Right now document accessibility, it requires a lot of knowledge, a lot of skill, and it is a bit subjective, right? You give five documents to five different people and you will get five different results.
It is not to say that it is wrong in any of those cases sorry, I am losing my voice thanks to the Phillies game last night. But, you know, automation, you know, as a quick example, Acrobat’s implementation of autotag is very rudimentary. It says if you have 18-point type, it must be in H3. No, that doesn't work. You know what I mean?
You have to evaluate the document. You need to say, okay, 09% of the text is 12-point. That has to be body. 7% of the text is 18-point, that must be a main heading.
So, automation, I think, has a lot of potential in the game. It just hasn't, at this point, really been well implemented.

MIKE PACIELLO: Good. Thank you, all. Interesting answers. Let's go to the second question. What are the cost in workflow benefits in automated technology. I will reverse order. Chad, I will have you start first, especially on the documentation side of this, publishing.

CHAD CHELIUS: I guess, the cost benefit is that you are not and, again, I am speaking to document accessibility, but you are not throwing people at the problem, right? With automation, I think there is a significant cost savings. I mean, not only there is not only an accuracy benefit, but the cost savings. the amount of time we spend getting documents compliant is extensive in some cases, right? I mean, there have been plenty of 250 page documents that come across my desk. That is not a 10 minute job, right?
So, automation has the potential to be able to evaluate that content, consistently apply accessibility standards, and, you know, properly tag that document in a way that, you know, a human can't effectively.

MIKE PACIELLO: Yes. Thank you.
Josh, I see this as a particular challenge in your industry. Especially, because as you were talking about, voice recognition and trying to deal with it. How do you address this?

JOSH MILLER: It is both a blessing and a curse for us. The better the speech recognition means the less work the human has the correct it, which is our model, so that is a great thing.
Understand, if it gets too good, we are out of business. So, that is something we think a lot about.
It is not just about automation. This isn't a story of machines taking over, and I do want to kind of also say that one thing that wasn't said in the first round, I think is important, technology enables more content and more things to be accessible, and that is a good thing. We want to think about that. that is really, really important, but we want do it well.
We think of how to use automation beyond just speech recognition in terms of applying heuristics to our modeling in how we assign jobs and make sure the right jobs are taken by the right people at the right time. That is technology and automation that is being applied to improve not only what the customer gets back, but even the experience of the person doing the job in the first place.
So, there are a lot of ways to use the technology to improve the human components that are necessary to get to the outcome we want. And that is actually probably where there is more magic happening in our world, which is really, really important, to make sure that we get the file that is due back in two hours done by the person who actually understands the content, and is available. That is something that can be very, very challenging when you are processing many, many files across many different constituents.

MIKE PACIELLO: Anthony, your team here and you as the manager, have to be concerned about cost and workflow. Especially because your team is made up of several individuals with disabilities. So, talk to us a little bit from that perspective.

ANTHONY FERNANDO: Yes. For me it is more like setting up process. We care about the process, and streamline it. As Chad mentioned, different folks may have several subjective viewpoints of an issue. It can be time consuming if you are going to argue about it, you know? Automation itself helps that part of the process.
Also, we preach that we need to start early, shift left, and developers itself love automation, especially if they can find issues early on. So, for them to do that kind of testing early on, and when it comes to testing, we get the difficult ones, the manual effort.
That is fine, right? It saves a lot of time for us. In terms of, you know, cost benefits, that is huge. Because we don't have to deal with the 80% of the problem anymore, because they have already solved that issue in the unit testing and development life cycle. So, that is a great benefit for me.

MIKE PACIELLO: Thank you. Navin, I would imagine within the Evinced platform, workflow is an important attribute of the entire solution. Again, cost is important, but if you could speak to that.

NAVIN THADANI: Yes. Given where we are in the industry today, I actually think automation has nothing to do with cost. It is irrelevant in the grand scheme of things. It has everything to do with feasibility. Can you even make a digital asset accessible on an ongoing basis without automation? I think the answer is no. I mean, think of yourself as an enterprise like metrics that releases software 4,000 times a day.
There is no amount of money that you can throw at the problem with just manual testing. It is just never going to happen. You will never have an accessible platform, because your developers are moving faster, they are moving features around all the time, changing buttons, changing names, so I believe it isn't about cost alone. It is a secondary benefit and we will get to that later, but right now it is about feasibility.
I think we are in sort of three stages of any industry. The first one being manual. We are kind of done with that. Everybody knows how to do that now. We are in automation. I would argue we haven't had the tools yet in order to automate things properly.
And we are trying to do everything we can to help that. But what I am really excited about is what I call automatic automation. That is where I think the future of this industry is. Where, your developers and test engineers don't even have any accessibility tools at all. It is all baked into your development build, or your test build, and as they are browsing it, they are automatically doing that analysis without even knowing it, it is going to a centralized place, you are getting all these results, bugs are being opened.
That is the nirvana, our future, that will get your cost down and also solve the feasibility problem, so that is what we have to look for.

MIKE PACIELLO: So it is realtime accessibility from the development standpoint?

NAVIN THADANI: At the development standpoint. When you are releasing software 4,000 time as day, you are not doing the testing. You are getting the user to do the testing in canary builds. You have the rollout production environment of 2% to 3% of the users. They do the testing and all that for you. And as part of testing, they are also testing for accessibility, automatically. They don't even know it. That is cool. When we get there, I think the industry is really evolved to that point.

MIKE PACIELLO: I am not challenging you in any way, shape or form. But remember, our theme is about a holistic process. Evenly in an automated, agile development environment, you still have QA, right? You still have eyes on the code to check that, right?

NAVIN THADANI: Yes.

MIKE PACIELLO: So, we are not talking about eliminating that but increasing the quality.

I would double down. You talk about a developer sits idle. The last thing you want them to do is sit around waiting for an accessibility report to come out. From a customer angle, in our world, we have a customer that literally sends us 400 videos a day. There is no way someone is going to sit there, upload a video to us, get the captions back, put it into their system. It doesn't make any sense. That is the other side of automation that we think about all the time. The workflow automation not just for how we produce captions, but how does the customer get the captions onto the video is critical.
So, in a form of automation can get overlooked as a table sake but the amount of time and cost that saves the customer that doesn't even get into the cost or price on paper is huge.

MIKE PACIELLO: Good. True. Third question, what is the challenge of automated accessibility and platforms. This should take a little more leeway, especially to the product folks at this level. Anthony, I will ask you to answer this question first, because we also have your side to show the approach and rubric that your team uses in terms of deployment of an automated system, or a platform and testing. So, I will bring that up. I think if I just click on this, is that right, Nicholas, if I click on this it will come up? If it doesn't and the whole system blows up, I apologize. Okay.
There is a rubric, a table, four columns and several rows, look like about a dozen row table. I will let Anthony explain the rubric.

ANTHONY FERNANDO: Yes. So, the first challenge for us is selecting the right tool, because there are a lot in the industry. Which are great ones, but we need to figure out what is fitting for us. You know, it may be a different product for you, not be the same for me, my team.
So, rather than being bias and picking one particular product, I think it is a good way to evaluate them and give them a chance to prove themselves, and for that, we have come up with this rubric, you know, of 15 subsets of criteria. It can be different for you based on your need, and come up with a score.
We get to do a pilot, like at least I think it is good if you can pick three products, a little bit in the market, and evaluate those products, those subsets, criteria, do we have a Wi pad, if it is there for testing it is two. If it is accurate, it is three. And accessibility, my number one concern for break, you know. Yes. Do the deal.
Are there accessibility issues, which is really great. These companies do care about accessibility. Some, they don't. For them I am telling, you need to be ashamed. You need to be fixing them soon.
Accuracy is something we need to take care of. Because sometimes they provide several features, sometimes they will do the scrolling, automation check, you know, the browse extensions. Sometimes they are not in par with their results.
So, we apply several approaches and it wastes our time sometimes. And, you know, automation setup is another thing. Because not all teams use automation.
In the pool you saw, some people don't use it still. For them, what is the game plan for these automation providers? I think, because setting up the automation, they need to figure out, you know, how to enable automation into their product, right? Then go along with the automation. Usually they come up with, okay, it is three lines of code you had to do. Not really. It is more than that, right? It is a journey. It is a relationship.
They need guidance. It is like asking an infant to go to high school. Because we don't know the tool, even though we know about accessibility, right? We need that guidance from the, you know, these vendors. Because there is so much resistance from product teams. They don't want to do it, because they have all sorts of other work. Accessibility is their priority strategy, even though it is my biggest priority. We need to get their work schedule moving.
So, I will probably share this link later on. Maybe in LinkedIn or somewhere. You can go and check it out.
And the last one, product offering. After doing the POC, Proof of Concepts, if you tell your customer, we are going to rescue with ten use accesses with limited scans per year, limited products per year, that is really bad.
Probably I would say, giving them unlimited access is the ideal scenario, but it is not going to work, I know. At least give the flexibility during the first year. Build the relationship so that we get to try out, and then come to an agreement. These are the numbers that we are looking at. These are the budgets we are looking at. Others are not going to make a deal, probably. That is it.

MIKE PACIELLO: It is a fairly simple scoring system. Poor, average and excellent. We will talk about this a little later with Anthony and Pearson. It is fairly simple, scale.
Now we get to talk with the vendors themselves to get a sense of, what are the challenges of automated accessibility systems in their platforms. Navin, if you can speak to that. I am sure it is not just let's roll it out, we have a product in sales and we are making money.

NAVIN THADANI: Yes. Sure. Where we are today again, we finally got the technology, which allows us to easily integrate automation into, let's say a functional test just as an example.
We are at a point right now where we with just three lines of code, you can inject accessibility into any number of selenium tests, or functional tests, whether it is one test or 3,000 tests, it is only three lines of code. That is the kind of thing that will help overcome the challenges we have seen in the past.
When I have gone and talked to several very large enterprises trying to do this earlier, they would go and take apart an existing test they would have to practically rewrite it, inject accessibility codes everywhere that led to a mess, and no developer looked at the results. No one.
So, make it easy to integrate, make it easy to consume the results. And unless we can use that, we have no artificial intelligence.

MIKE PACIELLO: I appreciate that, but that doesn't exactly address the question. The question is what are the challenges your company has in terms of automated accessibility systems and imitating them?

NAVIN THADANI: We work with our customers and we have our own testing, all our own products are tested with our own software, even today. So that is how we go about it. And things scanning automatically, those kinds of things make it easy to integrate. That is what solving these challenges there today.
As everybody has pointed out here, not every product has automation. Or maybe the automation isn't good. It is like building a house of cards at this point.
If your fundamental automation itself, the selenium test the cyber tests, aren't being maintained, you can't use that for accessibility. So you have to use an automatic concept. I think the industry needs to move there as quickly as it can.

MIKE PACIELLO: Thank you. Josh?

JOSH MILLER: There are two things that come to mind for me. One is managing expectations, and the other is applying, or getting in the right fit between the problem and the solution. I think one of the challenges with automation and a lot of the automated tools out there, there are really interesting solutions in search of a problem, and, really, we should be starting with the problem, not the solution. What is the problem we are trying to solve? Does the automation fit that problem? I think that is really important.
Then understanding what are the pros and cons with going with automation and how it fits in?
That is something we deal with all the time. Whether it be live content, recorded content, the different scenarios are different. And everywhere is dealing with a different problem, and we just need to be really thoughtful about how we go about solving those.

MIKE PACIELLO: Hmm. Thank you, Josh. Chad?

CHAD CHELIUS: I agree with Navin that the technology is there, but I have to say in my opinion the obstacle is cost, right? If it were feasible, why aren't companies like Microsoft or Adobe implementing that technology into their product, right? Has anybody used the auto generate ALT Text in Word? How many could actually use what it generated? It is pitiful, right?

MIKE PACIELLO: Wait a minute, Microsoft?

CHAD CHELIUS: Was I allowed to say that? (Chuckles) And I included Adobe in there, too. I don't want to single out any company. But, I mean, the automation that they have implemented has just been really poor.
I mean, Facebook knows when I want to buy crazy socks to wear, you know, at a party for next week. Imagine if we could leverage that technology into, like, automating some of the tasks we have to do on a day to day basis in Documents and others for that matter. So I have to believe cost is a significant barrier, otherwise why wouldn't it have been implemented already.

MIKE PACIELLO: Interesting. Good.
Last question, gentlemen. What solutions, technical or process oriented, do you recommend will ensure existing and future automated AI/ML platforms support a holistic end to end accessibility life cycle. Basically what, solutions do you recommend, without selling your product? (Chuckles) The solution?

I can start. The entire workflow for automation needs to start with designers, designers putting out accessible designs, when it goes to developers, developers writing code in as successful a manner as possible. Then in the automated systems, while the tests are running, they are automatically checking for accessibility. Then when it goes to the QA team, while sitting and going through the user journey and flow, they are in the background checking for accessibility. And when it gets to the production environment, you are scanning it, making sure nothing slipped through the crack and checking the compliance data. In my view that, is the holistic end to end accessibility lifecycle.
The security team is integrating security into the process in the same way there. are multiple points and products that work together to come out with something as secure as possible. That doesn't mean they don't do a soft tool audit at the end of the year. They do it. It is the same thing with accessibility. Integrate it in the development process so you have as accessible software as possible, but every year, periodically, more often than that, awesome, you have to do an accessibility audit. We need to learn from them.

MIKE PACIELLO: Thank you. Good point. Anthony, please?

ANTHONY FERNANDO: For me, automation will help future testers like people who are coming into is industry to get to know things better, because accessibility domain itself, getting more, you know, complex day by day, with 2.2 coming, 3 is coming. How do we get to more long? It is vastly changing. It is more end goal. For me, if we can do something to help their lives, make a quick test and improve the productivity that, is what I am looking for.

MIKE PACIELLO: Okay. Josh?

JOSH MILLER: On the process side, I think a lot of it comes down to building alignment. I don't know what the success criteria is. Some of it can be overlooked, okay, we implemented it, go. But what do we want it to do for us down the road, come back to it and check on that. Having a testing plan, success criteria and alignment around the success criteria, I think is critical. The test part, I think, is really important, too. Six months from now, three months from now, the timeframe, going back and saying are we getting what we expected, testing and checking on that, and of course, the feedback. Without the user feedback, is it usable, achieving what we want? That is ultimately what we are here for, so we need to make sure we have a system in place to get that feedback.

MIKE PACIELLO: Thank you, Josh. Chad?

CHAD CHELIUS: Yeah. So, I always say, like, compliance is a combination of automated and manual checks, so achieve full compliance.
I think AI and ML needs to come up with a way to kind of separate the milk from the cream and grade a document on how well it thinks it did in making that, you know, document or website accessible.
You know, there needs to be a way, you know, for it to evaluate that document, and let us know. So, that we can make an adjustment or make changes based on that, and, I mean, obviously we would love to see all our documents have a very high, you know, grade applied to them.
I think as AI and ML gets better, it is simply achievable, it is certainly possible. But I think it is important that we know, you know, similar to what these other guys were talking about, we need to know at what level we are with this content.

MIKE PACIELLO: Good. Thank you. Okay. Well, we have come to the point where we can ask audience questions. Now, the arrangement looks like it has come up here, but do we have anyone that can handle the microphones? Can you do that? Okay. Can we get this gentleman right over here? Let's see. The fourth row back. He was the first one to have his hand up. Next one, please.

Do you want me to stand up?

MIKE PACIELLO: No, you don't have to stand up. They are not that good!

I am Robert Petris from Virginia Tech. I had a question about so, with these automated tests, and these companies, how are they getting the automation? For instance, if I am using your product, and I am testing something out, and it gives me feedback, or it asks me a question about whether something is an accessibility problem, and I answer, yes, that is an accessibility problem, or no it isn't, is that data being fed back into the platform to make future scans by somebody else more accurate?
And if that is the case, how do we avoid having a situation where one player becomes massively more dominant, because it is kind of recursive? Does that make sense?

MIKE PACIELLO: I have Josh, because I know that you have probably dealt with this and understand it. Josh first, then Navin?

JOSH MILLER: I think we are getting into a closed loop concept system here. Which is when you have the truth data from the human, what do you do with it? In our world, yes, we are using that constantly to know did the machine do it right or wrong so we can improve it in the future. So, yes, there is a modeling concept in our role to improve the speed track acquisition output, whatever the job assignment heuristics may be, we are constantly looking at how to use that data because that is the most valuable data, the truth data, because that is the only way we can get better.

MIKE PACIELLO: The larger share of the market you have, you are going to once you get far apart ahead of everybody else, I don't know, anyway.

JOSH MILLER: To some degree, I think it is an interesting point. Still there is a use case situation that definitely come into play, I would say.

MIKE PACIELLO: Navin?

NAVIN THADANI: There are multiple ways to do it. You don’t have to use customer data. For instance, you can build an algorithm that will use data from the public internet. So, what is the problem in automating accessibility testing?
Essentially what, you are trying to do is to understand if there is a particular dropdown on the screen. You want to know it is a dropdown so you can assert well against it. You can say I know this thing is a dropdown, so let me look at the code to figure out if it has been implemented correctly or not.
So, if I want to train a machine to do that, tick comb the interpreter for as many images of dropdowns that I need and train my machine on that. Now, I can theoretically take a user input and augment that, but I don't need it. That is the thing. The way to look at this, user input is a good thing, and it certainly can help.
But, also, a lot of technology players, us included, but others, as well, are working with very large enterprises that have very sensitive data. So we are not necessarily looking at extracting input from the users immediately. That maybe will come later with additional trust and things like that. But, AI/ML doesn't mean by definition that you are using user data.

MIKE PACIELLO: Thank you. Let's take a question up here please from this gentleman.

Thank you for this. I wanted to discuss how we set up a standard for quality assurance, and how we have an error word rate I am sorry, word error rate, EWR. It is very subjective, because not all words are created equal. Some words matter more than other, and AI may not be able to capture that, so I am curious how you are able to develop a unified standard of expectation to know that this is good enough, and to meet the expectations of the quality automation?

MIKE PACIELLO: Josh?

JOSH MILLER: That is a great question. It is one that has not been solved perfectly, I will say. We like to talk about format edit rate which takes into consideration function weighs and capitalization because that affects meaning. We put out a report every year on benchmark error rates and formatted error rates. That is a start, but you are right, not all errors are equal. And that is one that is very, very difficult to automatically assess, and to assess that at scale. That is where the user testing has to come into play. Fortunately, or unfortunately, but there is not a perfect work around for that.

MIKE PACIELLO: Thank you, Josh. Well, that gives us room for improvement. We have time, that is one thing we all still have. Time. Excellent, excellent question. Thank you. We had a question here, the gentlemen the third in. Thank you.

Hi, Adam (?) from Citigroup. Are you able to generate audit transcription and if not, do you foresee getting there?

JOSH MILLER: We have human writers and we can create and scale the audio, that part of the process. That somewhere the automation come into play. We have designed some pieces of the process to speed up the writing, as well, but it is still a human doing the writing.
As far as identifying the right things to describe in a moving image, I would say we are pretty far away from getting that right for this purpose. There are interesting things that could be done to help with the process, but as far as anything stand alone, I think we are a very long time away from that, unfortunately.
To get that right, there is a lot of risk, right? A good example would be, it is very easy for a machine to identify that a car is an image. It may be completely irrelevant to what is going on. The way a lot of OCR systems out there are designed, is really identifying objects. There is no way for the machine to understand what is relevant and what is not, so that is a big part of what audio description is all about. Not yet. We will see.

MIKE PACIELLO: I think a distributive video is a long way from automating that.

And we can't get static images.

MIKE PACIELLO: Right. Exactly right. Lionel, then we have a gentleman in the back. So, right here. We have three or four more minutes here, folks.

Two questions if I can. One very specific and one general. The specific is Navin said you can go across the public interpreter to find things that look like dropdowns and do sufficient learning from that.
But somebody else spoke about you need a human to find out if you are right or wrong. It goes out there, just like if it went out and looked for cars, it would find a lot of cars and stuff that aren't cars. Do you want to comment on training on the public internet?

NAVIN THADANI: Yes. The public internet is not full of only incorrectly implemented elements. There are some correctly implemented elements, as well. So, semantically, if you can figure that out, you can take those images and say these are the correct dropdowns and use it for that. So there are many techniques that are possible to use that way.
There are also much more advanced learning machine models coming out that will help with that. But the primary presumption is that somebody has done some things right, and we want to learn from that.

And the general question is, you had the word holistic in the title. I thought it was a very powerful word. I wonder if we can ask for a few more comments on what it means to be truly holistic in this approach.

MIKE PACIELLO: Thank you. Chad, do you want to start?

CHAD CHELIUS: That is a tough one. When I think of holistic, I think of a more, like, natural process of how you would implement, you know, AI or ML as we are talking about, how we go about that. I think the idea is that the technology tries to utilize a method to understand the content in as natural a way as possible, right?
Like, when you utilize machines, they tend to be very, like, black and white, right? It would be cool if that AI/ML, could be a little more, you know, natural in its process generate more accurate content.

MIKE PACIELLO: Good. Thank you. Josh?

JOSH MILLER: I actually really like the way Chad put that. I think it is this idea of not being black and white about what the technology can and can't do. It is taking kind of all aspects into account, and thinking, where does the technology excel? Where is a human needed, or some other kind of input needed and being thoughtful about it, is way I would answer that.

MIKE PACIELLO: Anthony?

ANTHONY FERNANDO: For me it is end to end. It is about soft development life cycle. How you start from the start itself and end, and how to improve that life cycle, how do you parse that life cycle to give, as someone mentioned, give better output of the result. Just like security, I think. Navin mentioned, we are really, you know, heading there, but a long way to go. End to end.

MIKE PACIELLO: And Navin?

NAVIN THADANI: I am very much with Anthony, as I said. It should be all the way from design to production, and there is even some other sectors that we probably need to look at, including the content itself. I guess that is what the videos and documents and all that are about. But from an application or digital application, web application or mobile app perspective, we see holistic as design all the way to production.

MIKE PACIELLO: Good. Excellent. Lionel, one more question. We have about two minutes left. Here in the back, this gentleman right over here. Thank you.

Thanks. Gary Moore, National Cancer Institute, National Institutes of Health. Are any of you involved maybe it is directly for 3Play Media, and it follows up on the captioning question and the audio description question. are any of you aware of or studies of brain fatigue or cognitive fatigue. If you are listening to automated captioning and filling in the blanks, or listening to machine read audio description rather than human voice audio description, that the experience is different. The experience degrades, if you are listening, if you are watching audio, you know automated captioning, less than perfect captioning, or machine read audio description for somebody blind or vision impaired? And is there room for research, and what do you think about that?

JOSH MILLER: I think there is absolutely room for research. I think it is a really interesting question. I would probably broaden it, quite honestly, and think about, many modes, and the way people are consuming content, and, you know, like we talked about Zoom fatigue, and the same thing I think applies.
Whether it be reading poor quality captions, I would be curious about all of that. We are not currently involved in a study on that specifically, but I think it is a really interesting question.

MIKE PACIELLO: It seems like it is a good suggestion for NITRE to fund that kind of research, right? Right.
We have run out of time because we want to make sure we get everyone here back to the main hall for the last keynote session, but thank you, everyone. I want to thank our panelists. And, hopefully you get a chance to talk to them after.
Thank you, guys, excellent.

Thank you for having us.

MIKE PACIELLO: My pleasure.
(Session was concluded at 5:14 PM ET)

***
This text, document, or file is based on live
transcription. Communication Access Realtime Translation
(CART), captioning, and/or live transcription are provided in
order to facilitate communication accessibility and may not be a
totally verbatim record of the proceedings. This text,
Document, or file is not to be distributed or used in any way
That may violate copyright law.
***