When Hanna Wallach first started testing machine learning models, the tasks were well-defined and easy to evaluate. Did the model correctly identify the cats in an image? Did it accurately predict the ratings different viewers gave to a movie? Did it transcribe the exact words someone just spoke?
But this work of evaluating a model’s performance has been transformed by the creation of generative AI, such as large language models (LLMs) that interact with people. So Wallach’s focus as a researcher at Microsoft has shifted to measuring AI responses for potential risks that aren’t easy to quantify — “fuzzy human concepts,” she says, such as fairness or psychological safety.
This new approach to measurement, or defining and assessing risks in AI and ensuring solutions are effective, looks at both social and technical elements of how the generative technology interacts with people. That makes it far more complex but also critical for helping to keep AI safe for everyone.
This post is part of Microsoft’s Building AI Responsibly series, which explores top concerns with deploying AI and how the company is addressing them with its responsible AI practices and tools.
“A lot of what my team does is figuring out how these ideas from the social sciences can be used in the context of responsible AI,” Wallach says. “It’s not possible to understand the technical aspects of AI without understanding the social aspects, and vice versa.”
Her team of applied scientists in Microsoft Research analyzes risks that are uncovered by customer feedback, researchers, Microsoft’s product and policy teams, and the company’s AI Red Team — a group of technologists and other experts who poke and prod AI systems to see where things might go wrong.
When potential issues emerge — with unfairness, for example, such as an AI system showing only women in the kitchen or only men as CEOs — Wallach’s team and others around the company step in to understand and define the context and extent of those risks and all the different ways they might show up in various interactions with the system.
Once other teams develop fixes for any risks users might encounter, her group measures the system’s responses again to make sure those adjustments are effective.
She and her colleagues grapple with nebulous concepts, such as what it means for AI to stereotype or demean particular groups of people. Their approach adapts frameworks from linguistics and the social sciences to pin down concrete definitions while respecting any contested meanings — a process known as “systematization.” Once they’ve defined, or systematized, a risk, they start measuring it using annotation techniques, or methods used to label system responses, in simulated and real-world interactions. Then they score those responses to see if the AI system performed acceptably or not.
The team’s work helps with engineering decisions, giving granular information to Microsoft technologists as they develop mitigations. It also supports the company’s internal policy decisions, with the measurements helping leaders decide if and when a system is ready for deployment.
Since generative AI systems deal with text, images and other modalities that represent society and the world around us, Wallach’s team was formed with a unique mix of expertise. Her group includes applied scientists from computer science and linguistics backgrounds who study how different types of risks can manifest. They partner with researchers, domain experts, policy advisors, engineers and others to include as many perspectives and backgrounds as possible.
As AI systems become more prevalent, it’s increasingly important that they represent and treat marginalized groups fairly. So last year, for example, the group worked with Microsoft’s chief accessibility officer’s team to understand fairness-related risks affecting people with disabilities. They started by diving deep into what it means to represent people with disabilities fairly and identifying how AI system responses can reflect ableism. The group also engaged with community leaders to gain insight into the experiences people with disabilities have when interacting with AI.
Turning those findings into a clearly systematized concept helps with developing methods to measure the risks, revise systems as needed and then monitor the technology to ensure a better experience for people with disabilities.
One of the new methodological tools Wallach’s team has helped develop, Azure AI Studio safety evaluations, uses generative AI itself — a breakthrough that can continuously measure and monitor increasingly complex and widespread systems, says Sarah Bird, Microsoft’s chief product officer of responsible AI.
Once the tool is given the right inputs and training in how to label an AI system’s outputs, it roleplays — for example, as someone trying to elicit inappropriate sexual content. It then rates the system’s responses, based on guidelines that reflect the carefully systematized risk. The resulting scores are then aggregated using metrics to assess the extent of the risk. Groups of experts regularly audit the testing to make sure it’s accurate and in alignment with humans’ ratings, Bird says.
“Getting the AI system to behave like the experts, that’s something that takes a lot of work and innovation and is really challenging and fun to develop” as Microsoft invests in the evolving field of evaluation science, she says.
Microsoft customers can use the tool, too, to measure how their chatbots or other AI systems are performing against their specific safety goals.
“Evaluation is the robust thing that helps us understand how an AI system is behaving at scale,” Bird says. “How will we know if our mitigations and solutions are effective unless we measure?
“This is the most important thing in responsible AI right now.”
Source: microsoft site
Balko, Radley (November 3, 2008).
3. Top Area: The upside stalls and the stock tops out.
This implies estimating how effectively ready you are to safely accomplish a process and making a judgement call.
The preliminary goal rate of inflation was an RPIX of 2.5.
The Institut qu茅b茅cois de planification financi猫re (IQPF) and the Financial Planning Standards Council (FPSC), the two organizations that oversee the profession of financial planner have developed the reference document Financial planning in Canada: definitions, standards and skills.
They strengthen the body’s skill to efficiently fight particular diseases by stimulating antibodies, or proteins that combat off infections.
American Hospital Association | AHA Information.
Their nanoaugmentations are highly related to the Gray Dying.
Businesses that are able to make good financial decisions will likely survive tough times.
As in so many different areas of the world, these communities that at the moment are underserved by fossil fuels can profit most from such clean energy advances.
But in either cases it would be always good to give users an automatic wallet address generated from the service itself and not out of the service.
First seen during World War II, is that this car American or German?
However, stick vegetables and lettuce have high water content, which means these veggies don’t freeze as well.
Be proactive. Do not anticipate the payments to start out coming in: Negotiate early.
In Phoenix, many restaurateurs have ushered al fresco dining to cool new heights.
These are hardware and software engineers.
One person I cannot thank sufficient for his musical influence on me is my Junior Highschool bandleader, Arthur Berman.
A moon in Cancer values shut-knit family bonds and a cozy, nurturing home, whereas a moon in Capricorn could prioritize construction and responsibility of their home life.
The orders for this are valid just for the extended trading sessions and aren’t demanded with traditional trading.
Some of the suggestions will be selling an ownership stake in the company through a stock or bond issue.
Donald William Ford, Chairman, TWIL Ltd.
Now, at maturity, you’ll have about thirty lac rupees shining like freshly baked gold in your account.
P.N. Gadgil & Sons have latest designer & traditional collection of gold, silver, diamond jewellery online.
These home windows are easy and decorative, frequently utilising the skills of the provincial plumber/glazier.
An excessive amount of of 1 shade will be overwhelming to the attention, so it is best to follow the 60-30-10 rule.
The Ma’alot massacre in Could 1974 concerned a two-day hostage-taking of 115 people which ended within the deaths of over 25 hostages.
He had two more marriages, the last to Gretchen Kresl Hassler.
Not making the grade or scoring the winning point are worthwhile lessons for youths as well, whereas witnessing a mother or father screaming at a instructor or coach only undermines that grownup’s authority.
They’ll give you a plan of action that puts you on track to perform your financial targets.
Whereas 80 percent of docs will prescribe generic treatment to save lots of patients cash, not practically as many will counsel low-cost treatments.
It is presumed that the allocations mentioned in the ultimate laws seek advice from tax allocations.
The Chinese government’s reaction to the 2009 financial crisis was to direct banks to loan to Chinese state-owned enterprises (SOEs), which then built factories and equipment to stimulate the economy despite the lack of demand for the products created.
Totally different faculties of astrology have primarily linked the Turquoise to 2 planets – Jupiter and Ketu.