编程知识 cdmana.com

Exclusive | the hidden algorithm war that keeps people in poverty is coming

author :Karen Hao

translate : Chen Chao

proofreading :Kay

The length of this paper is 4400 word , Recommended reading 10+ minute

This paper will introduce a series of legal and practical problems caused by the wide application of automatic algorithm in the field of daily credit integral .

More and more lawyers are exposing 、 Guide and deal with the housing problems that hinder the poor 、 Automation systems that use basic services at work .


encounter Nick when ,Miriam Only 21 year . She's a photographer , Just graduated from college , Working as a waiter in a restaurant .Nick I'm older than her 16 year , Is a local entrepreneur who used to work in the financial industry . He's very charming and charming , Take her on a romantic date and buy her anything .Miriam Soon fell into his gentle trap .

It started with a credit card , At that time , This is her only credit card . In limine ,Nick Will overdraft the maximum amount 5000 dollar , And quickly returned the next day .Miriam( For fear of interfering with the divorce process they are now in , She asked me not to use their real names ) Found out that it would improve her credit score , She grew up in a low-income single parent family , So I believe in Nick I know more about it than she does .Nick I'm glad to encourage , And give Miriam Brainwashing , Said she didn't understand finance , therefore Miriam In her own name Nick More credit cards .

But three years ago , Trouble is coming. .Nick Ask her to quit her job , To help him do business , She did it . He asked her to go to graduate school , And don't worry about increasing her existing student debt , She did the same . He promised to take care of everything , She believes in him . But soon after , He stopped clearing her credit card balance , Her credit score began to drop .


even so ,Miriam Or with him . They are married . They have three children . And then one day ,FBI Came to their house and arrested him . In federal court , The judge ruled that he was involved in approaching 250000 Dollar Telecom scam .Miriam Found that he put tens of thousands of dollars in debt in her name .“ The day he went to prison , I only 250 USD cash , A foreclosure house , A car ready to be withdrawn , Three children ,” She said .“ I never had a nanny in a month 、 Beautiful houses and almost anything to be destitute .”

Miriam It's called “ Forced debt ” Survivors of ,“ Forced debt ” A form of abuse usually committed by a close partner or family member . Texas RioGrande Carla, a lawyer for legal aid · sanchez · Adams (Carla Sanchez-Adams) Express , Although economic abuse is a long-standing problem , But digital banking makes it easier to open accounts and withdraw loans in the name of victims . In the age of automatic credit scoring algorithms , The impact could also be more damaging .

Credit scoring has been used to assess consumers' credit rating for decades , But they are more professional with the support of algorithms : It's not just an enhancement in the amount and type of data being analyzed , It's more about whether you can buy a car , Rent an apartment or get a full-time job . Their widespread impact means that if you lose your score , It's almost irrecoverable . To make matters worse , This algorithm is owned by private companies , They don't reveal the way they come to a conclusion . Victims can go into a vicious circle , They either end up with nothing , Or return to the arms of the abuser .


Credit scoring algorithm is not the only one that will affect people's economic status and access to basic services . Now? , The algorithm can determine which children are fostered , Which patients receive medical care , Which families can use stable housing . Those of us who are capable can live our lives without any awareness . But for low-income individuals , The rapid growth and adoption of automated decision-making systems has created a hidden network of interlocking traps .


Fortunately, , More and more civil lawyers are beginning to standardize this issue . Learn from the criminal defense community's resistance to risk assessment algorithm , They started to teach themselves these systems , Building a community , And build defense strategies .“ Basically every civil lawyer starts to deal with this kind of affairs , Because all of our customers are more or less affected by this kind of system ,” Professor of clinical law at the University of Baltimore Michele Gilman Say .“ We need to be sober , And get trained . If we want to be really all-round lawyers , We need to understand that .”


“ Do I need to question the algorithm ?”

Gilman Practicing in Baltimore 20 year . As a civil lawyer and a poor lawyer , All her cases come down to the same thing : For the lack of access to basic needs ( For example, housing , food , education , Work or health care ) People in our country are saying . Sometimes it means confronting government agencies . The rest of the time is with credit reporting agencies , Or the landlord . Now more and more lawsuits about customers' rights involve some algorithms .

“ This is all-round for our customers ,” She said .“ They're stuck in a problem with different algorithms that prohibit them from using basic services . And customers may not be aware of , Because these systems are invisible .”

For low-income individuals , Temporary economic difficulties can create a vicious circle , Only bankruptcy or homelessness can end


She can't remember when she realized that some usability decisions were influenced by algorithms . By then, when the deal first happened , It's not obvious . There is a , She's acting for an elderly disabled client , She was somehow cut off from Medicaid funded family medical aid .“ We can't find out why ,”Gilman Remember .“ She was more sick , If you get worse , You'll get more time , Not less .”

Until they stand in court , The government has just adopted a new algorithm . The witness was a nurse , She can't explain anything .“ Of course she couldn't explain , They're off the shelf ”.

Gilman say .“ She's a nurse , Not a computer scientist . She was unable to answer the cause of the question . How it is weighted ? What results do you want ? So I talked to my students in my law firm , It's like ‘ Oh , I want to question an algorithm ?’”

Yes Arkansas My legal aid lawyer Kevin De Liban Come on , This change is also covert .2014 year , His state has also released a new system for distributing Medicaid funded family medical assistance , Cut off access to a large group of people who could have been funded . at that time , He and his colleagues couldn't identify the root of the problem . They just know that something has changed .“ We found that the evaluation system starts from a 20 The paper questionnaire became 283 The electronic questionnaire of the question ” He said .

Two years later , When the error of the algorithm brings it under the scrutiny of the law again ,De Liban Finally, the root of the problem was found . He realized that the nurse told the patient ,“ ok , The computer pot —— Is not my .”“ It inspired us to ,” He said .“ If we had known we were 2016 The fact that I didn't know until 20 years ago , I may be in 2014 I've done better work support since I was a child ”, He added .

“ A person experiences many systems every day ”

after that ,Gilman Become smarter . Judging from her strength in handling a range of issues on behalf of her clients , She observed the rise and collision of two algorithmic networks . The first category includes credit reporting algorithms , For example, trouble Miriam The algorithm of , These algorithms have an impact on personal goods and services ( Such as car , Housing and employment opportunities ) Acquisition . The second category includes algorithms used by government agencies , These algorithms can affect people's access to things like healthcare , Public interests like unemployment and child care services .

From the perspective of algorithm reporting , Data collection and sharing is easier than before , This also promoted the development of algorithms . Credit reports are not new , But these days they've become much broader . Consumer reports include credit bureaus , Screening Tenant Companies , Or check the validation service , Gathering information from a wider range of resources : Public records , social media , Web browsing , Banking activities ,app Use and more . Algorithms can assign values to people “ value ” branch , This is in the bank , employer , landlord or landlady , Even school background checks play an important role .

On the other hand , The government must adopt algorithms in the process of system modernization . stay 2000 Beginning of the year , For web based app And the adoption of data tools , And with more data-driven self starting systems and AI And keep going . The reasons behind these changes are good . During the outbreak , Many unemployment benefits systems deal with a lot of new requests , It leads to a significant delay . Modernizing these systems will guarantee faster 、 More reliable results to get .

But the software procurement process is not likely to be transparent , So there's a lack of explicability . Public agencies can usually buy self initiated decision-making tools directly from private suppliers . The result is that when the system goes wrong , Individuals will be affected —— And their lawyers —— It's going to get into trouble .“ They don't advertise anywhere ,” Lawyers for legal aid in Philadelphia JuliaSimon-Mishel say .“ It's not usually written in any policy guidance or policy manual . We are at a disadvantage .”

The lack of public scrutiny also makes the system more error prone . The most amazing failure occurred in 2013 Michigan . After a lot of effort to automate the state's unemployment benefits system , The algorithm abnormally marks more than 34000 People cheat .“ It leads to massive welfare losses ,”Simon-Mishel say .“ There are bankruptcies ; Unfortunately , There are suicides, too . What a mess .”

  Gilman is worried about , New crown related debt and eviction will be written into the credit score , It's going to make it hard for people to get jobs forever 、 Housing and loans .


The impact on the algorithm , Low income individuals bear the brunt . They are the most vulnerable to the short-term economic difficulties that are reported by consumers , And those who need and seek public welfare most . In the past few years ,Gilman I've seen more and more customers venture into a vicious circle .“ A person is exposed to a lot of systems every day ” She said .“ Well, I mean , We are all like this . But the results have a greater impact on the poor and the minority .”

She took a recent case in her office as an example . A family member lost his job due to the epidemic , And was denied unemployment benefits because of problems with the automated system . The family is in arrears with the rent , Led their landlord to sue for eviction . However, expulsion is not legal , because CDC The ban , The lawsuit is still in the public record .

Those records will be put into the tenant filtering algorithm , This will make it harder for the family to find a stable home in the future . Their failure to pay rent and utilities also has an impact on their credit score , It's having an impact again .“ If they try to set up a phone service or take out a loan or buy a car or apply for a job , And then there's a ripple effect ,”Gilman Say .

“ Every case becomes an algorithm case ”

In September ,Gilman Already a staff member of data and social research institutions , A report was published to document all the algorithms that poor lawyers might encounter . It's called the poverty algorithm , It's a guide for colleagues in the field . It's divided into specific areas , Like consumer law , Family law , housing 、 public interest , It explains how to deal with the problems of algorithms and other data-driven technologies from an existing legal perspective .

for example , If a client is refused an apartment due to a poor credit score , The report recommends that lawyers first check whether the data entered into the scoring system is accurate . according to 《 Fair Credit Reporting Act 》, Require reporting agencies to ensure the effectiveness of their information , But it's not always the case . Objection to any false claim can help restore the customer's reputation , To get housing . However , The report acknowledged that , The existing law can only do so much .Gilman Express , There are still regulatory gaps to fill .

Gilman I hope this report will sound an alarm . Many of her colleagues are still not aware of the problems that are happening , They can't ask the right questions to reveal the algorithm . Those who are aware of the problem are scattered around the United States , understand , Lead and alone declare war on these systems . She saw an opportunity to connect them and create a community of mutual help .“ We all need training 、 More knowledge —— It's not just the law , And it's in these systems ,” She said .“ In the end, it's like all cases will become an algorithm case .”

Related stories

Log in and out : Inside the virtual expulsion crisis in the United States

Thousands of tenants have been evicted from their homes through mobile phones and video phones .

In the long run , She wants inspiration from the criminal law community . She said , When organized as a community and opposed to deterministic decision-making risk assessment algorithms , Criminal lawyers have been in “ Leading position ”. She wants to see civil lawyers do the same thing : Start a campaign , Bring more public scrutiny and supervision to the hidden algorithm network that customers face . She said :“ In some cases , Maybe it should be turned off , Because there's no way to make it fair .”

as for Miriam, stay Nick After being charged with a crime , She left . She moved to a new state with three children , And connected with non-profit organizations that support survivors of forced debt and domestic violence . Through them , She took a series of courses on how to manage her property . This organization helped her solve most of her forced debt , Learn more about credit score . When she went to buy a car , Her credit score just barely passed the minimum , Her father is a co sponsor . Since then , Her regular car loan and student loan gradually increased her credit score .

Miriam Still need to be vigilant .Nick With her social security code , They're not divorced yet . She had been worried that he might open more accounts , More loans in her name . For a while , She checks her credit report for fraud every day . But these days , She also has something to look forward to . Her father is in his sixties , Want to retire . The two of them are now planning to buy a house .“ I'm very anxious about this . My goal is to save by the end of this year 700 branch ,” She said her credit score ,“ And then I'm ready to buy a house ”

“ I've never lived in a house I own ,” She added. .“ He and I are working together to save money for a permanent home .”

Original title :

The coming war on the hiddenalgorithms that trap people in poverty

Link to the original text :https://www.technologyreview.com/2020/12/04/1013068/algorithms-create-a-poverty-trap-lawyers-fight-back/

edit : Huang Jiyan

proofreading : Wang Yuqing

Introduction to translator

Chen Chao , Master of applied psychology, Peking University . I majored in computer science , After that, he made unremitting efforts on the road of psychology . It is increasingly found that data analysis and programming have become two required survival skills , So make every effort to better contact and understand the relevant knowledge in daily life , But the road is long , I'm still on my way .

Recruitment information of translation team

Job content : Need a careful heart , Translate the selected foreign articles into fluent Chinese . If you're Data Science / statistical / Computer students , Or work overseas , Or have confidence in their foreign language level of friends welcome to join the translation team .

You can get : Regular translation training to improve the translation level of volunteers , Raise awareness of the cutting edge of Data Science , Overseas friends can keep in touch with domestic technology application development ,THU The background of data delivery industry university research brings good development opportunities for volunteers .

Other benefits : Data scientists from famous enterprises , Tsinghua University and overseas students will be your partners in the translation team .

Click the end of the article “ Read the original ” Join the data group ~

Reprint notice

If you want to reprint , Please indicate the author and the source in the prominent position at the beginning of the article ( from : Data pie ID:DatapiTHU), And at the end of the article, put the data school eye-catching QR code . Articles with original logo , Please send 【 Article title - Name and number of the official account to be authorized. ID】 To contact email , Apply for white list authorization and edit as required .

Please send the link back to contact email after publishing ( See below ). Unauthorized reproduction and adaptation by , We will investigate its legal responsibility according to law .

Click on “ Read the original ” Embrace the organization

本文为[osc_ 7e2pw1w9]所创,转载请带上原文链接,感谢

Scroll to Top