You might not have noticed it yet, but in numerous organisations today, an algorithm has taken a seat at the management table and, in many cases, is running the meeting.
Artificial Intelligence has embedded itself into the very architecture of how people are hired, monitored, and fired – with HR departments’ consent. And this applies to everything, from reviewing resumes and planning shifts to rating performance and suggesting firings. Now, the question is whether leaders have the ethical backbone, or willingness, to oversee it wisely or if they will simply let the machine run.
Because the numbers of unfair cases are staggering, we’ve taken the initiative to examine the systemic risks of AI-driven workforce management, the emerging legal landscape fighting back against unchecked algorithms, the fragile promise of Human-in-the-Loop (HITL) oversight, and the principles that serious leaders need to adopt to navigate what is, openly, one of the most consequential leadership challenges of our era.
The Scale of Adoption: A Quiet Revolution
A landmark 2025 OECD (Organisation for Economic Co-Operation and Development) survey of over 6,000 firms across six countries – France, Germany, Italy, Japan, Spain, and the United States – found that algorithmic management tools have been adopted by 90% of US companies and an average of 79% of European firms. More cautious, Japan remained at around 40%.
In the US, the deployment is notably intensive. Big businesses use both monitoring and evaluation software, with a shocking 55% eavesdropping on workers’ communications (compared with just 6% in Europe and 8% in Japan). What is more, three-quarters of American companies report using ten or more algorithmic managerial resources simultaneously. These systems allocate work, track attendance and output, deliver productivity scores, and shape decisions around pay and promotion.
The tools go by many names: workforce analytics platforms, productivity trackers, and AI-powered performance management systems. In its bluntest form, critics call it “bossware“. Such a “digital supervisor”, installed on employee devices, logs keystrokes, mouse movements, and website visits and can even take random screenshots or record audio. In simple terms, workers’ efficiency is measured by how much they move their cursor, or the quantity of keys they strike on the keyboard – not their results.
It stems from the pandemic-era shift to remote work, which was deeply marked by managerial distrust as millions of employees left the office. Post-pandemic hybrid model arrangements sped up the adoption of “bossware”. What started as a response to an emergency management crisis has become routine infrastructure. And therein lies the danger.
The Risks of Algorithmic Management: Bias Baked into the System
The most well-documented peril of AI in HR is one that the technology was, ironically, supposed to cure: discrimination. For instance, when Amazon built an automated candidate-ranking tool, it discovered in 2018 that the system consistently downgraded resumes from women because it had been trained on a decade of historical, male-dominated hiring data.
The bias wasn’t programmed in intentionally. It was learnt. That distinction matters enormously, and it makes the problem harder, not easier, to solve.
University of Washington researchers tested three state-of-the-art LLMs (Large Language Models) against over 550 real-world resumes. They discovered that AI screening tools favoured white-associated names 85% of the time versus black-associated names (9%). And female-associated names were preferred on a mere 11% of occasions.
It seems not to be marginal rounding errors but a systematic hierarchy of exclusion operating at an industrial scale.
A Workday Case
Derek Mobley, a Black man over 40, claims he applied to more than 100 jobs through companies using Workday’s AI-based screening tools over several years. He was rejected every single time, often within minutes or hours of applying. In May 2025, a federal judge allowed a lawsuit to proceed as a nationwide class action under the Age Discrimination in Employment Act, noting that the case centred on whether Workday’s AI “has a disparate impact on applicants over forty”. The ruling established that AI tool vendors, not just the employers, can be held directly liable for discriminatory outcomes.
The Amazon Algorithm: Management Without Conscience
The most infamous real-world case of algorithmic firing remains Amazon’s warehouse management platform.
Documents obtained by The Verge in 2019 – since widely corroborated – revealed that Amazon’s system tracked every individual worker’s productivity and automatically generated warnings or terminations without input from supervisors.
Workers described being “treated like robots” because they were supervised entirely by bossware, with the system tracking “time off task”, including bathroom breaks, and could start a dismissal procedure if accumulated time exceeded thresholds, devoid of any contextual human review.
At a single Baltimore facility, roughly 300 full-time workers (around 10% of the total workforce) were laid off for efficiency reasons between August 2017 and September 2018. Amazon disputed that firings were fully automated, stating managers could override the process. But the documents showed that the override was the exception, not the norm.
This Amazon case stood as a harbinger. As automation deepens, the pattern is spreading beyond warehouses into white-collar work, healthcare, retail, and logistics.
The Psychosocial Toll
The mental health consequences of algorithmic management are now drawing serious scientific attention. In late 2025, a review published in the Scandinavian Journal of Work, Environment & Health identified AI-based management as an emerging occupational safety and health challenge, documenting links to reduced job satisfaction, rising workloads, higher stress levels, lower trust, and increased job insecurity.
A further investigation of Finnish food delivery couriers using the Job Demand-Control-Support framework found that bossware had “direct and indirect intertwined negative psychological influence”, increasing work demands, decreasing workers’ autonomy over their own labour, and limiting workplace social support.
We must agree that these are not abstract concerns. They map directly onto established risk factors for burnout, anxiety, and depression.
In the OECD survey, managers themselves registered alarm, with 27% reporting apprehension regarding inadequate protection of workers’ physical and mental health. And a similar proportion flagged that they found it difficult to understand the logic of algorithmic tools they were supposed to oversee. When the people “in charge” of these tools are confused and worried about their effects, something has gone structurally wrong.
Workplaces built around dashboards, metrics, and risk logic are staffed by humans who know that being watched is not the same as being managed.
The Responsibility Vacuum
An added dangerous feature of algorithmic management is the accountability gap it creates. When a human manager makes a bad call, there is at least a chain of culpability: they can be questioned, challenged, or overruled. When an algorithm fires someone, the employer often claims the tool “recommended” the decision; the vendor says it only provided software; and the worker is left with no one to appeal to and no way to understand what happened.
Citing the OECD survey once more, 28% of managers reported unclear accountability for bossware decisions. Uber drivers challenging automated pay-setting and deactivation faced this fog directly. To illustrate, when they requested access to their own performance data under GDPR (General Data Protection Regulation), Uber resisted, maintaining secret driver profiles with categories like “negative attitude” and “inappropriate behaviour” without disclosing how AI calculated these scores or what consequences they triggered.
In 2023, an Amsterdam court ordered Uber and Ola Cabs to provide full transparency to workers on automated decision-making relating to work allocation and fares. Thrilled, drivers saw this as a landmark victory for gig worker digital rights.
The Legal Reckoning: The EU AI Act – The First Serious Regulatory Framework
Members of the European Union have moved more decisively than any other jurisdiction to confront the risks of AI in HR. Meet the EU AI Act, which came into effect in August 2024 and begins full enforcement in August 2026, explicitly classifying AI systems used in employment decisions as high-risk. This includes recruitment and selection tools, performance management platforms, task allocation, monitoring of worker behaviour, and applications that can lead to promotion, demotion, or dismissal.
Under the Act, providers and operators of shoddy artificial intelligence in human resources must meet demanding requirements, such as robust transparency and explainability, human oversight mechanisms, documentation of training data and methodology, regular bias auditing, and meaningful access for workers to understand decisions that affect them.
If we look at Article 26(7), it specifically requires vendors to inform staff representatives and employees that AI is being used in circumstances that impact them.
The GDPR already provides a foundational right under Article 22, stating that individuals have the right not to be subject to decisions based solely on automated processing that result in legal or similarly significant effects, such as being fired or denied employment. As a complement, controllers must offer mechanisms for human intervention and the ability to challenge machine-driven results.
Legal Precedents Beyond Europe
Courts outside Europe are also pushing back. In a historical ruling in April 2026, China’s Hangzhou Intermediate People’s Court ruled that a technology company had illegally terminated an employee by replacing his role with an AI and firing him when he refused a demotion. The court held that using AI was not a valid legal basis for making it “impossible to continue the employment contract”, ordering compensation for the worker.
“Companies cannot unilaterally lay off employees or cut salaries due to technological progress,” the court ruled.
In California, for instance, employment lawyers are increasingly advising workers that algorithmic decisions in hiring and termination can constitute illegal discrimination even when bias wasn’t intentional under the disparate impact doctrine.
The Workday class action, potentially covering millions of job seekers, has sent a clear signal: the era of blameless algorithmic discrimination is reaching the end of the road.
On its side, the UK is considering its own reckoning. An Artificial Intelligence (Regulation and Employment Rights) bill has been under parliamentary discussion, which could introduce specific protections for workers subject to automated decision-making. Though the current Employment Rights Act 1996, Equality Act 2010, and Data Protection Act 2018 already impose obligations on employers regardless of whether decisions are made by humans or algorithms, the courts are actively developing the application of existing law to this new terrain.
Human-in-the-Loop: Promise and Peril
The phrase “Human-in-the-Loop” (HITL) has become the standard reassurance in corporate AI governance conversations. Keep a human involved, the logic goes, and you ensure accountability. Well, not quite. In specific cases, HITL frameworks may provide a false sense of ethical security while failing to deliver it.
What Is HITL Supposed to Do?
In its best conception, HITL means that AI handles the pattern-recognition and data-processing heavy lifting – screening applications, flagging performance issues, identifying scheduling conflicts – while humans hold responsibility for decisions with meaningful impact on people’s working lives. IBM describes the goal clearly: “to allow AI systems to achieve the efficiency of automation without sacrificing the precision, nuance, and ethical reasoning of human oversight.”
The Society for Human Resource Management (SHRM) approach involves experts scrutinising AI outputs, building guardrails into governance processes, and ensuring that domain knowledge is never allowed to atrophy just because a machine is doing more of the work.
A study published in the Current Trends in Information Technology journal in 2026, drawing on artificial intelligence governance reports from the Big 4 consulting firms, found that HITL AI does meaningfully enhance fairness and compliance in HR decisions but that it faces significant challenges around scalability, workforce acceptance, and regulatory complexity.
The Rubber Stamp Problem
Here’s where the promise frays. Research from the EU, published in early 2025, determined that when humans review AI recommendations, they accept biased guidance just as often as unbiased information.
The study involved HR and banking professionals from Italy and Germany making hiring and lending decisions influenced by AI inputs, and the results painted a bleak picture: even “fair” AI didn’t eliminate human prejudice, and swayed AI was largely followed without question.
The mechanism is well understood: automation bias. When an AI system presents a shortlist, a score, or a recommendation, humans instinctively treat it as more objective and authoritative than their own assessment.
For instance, a hiring manager who reviews a pre-selected list of candidates thinks she is exercising independent judgment, but she never sees the hundreds of qualified applicants the algorithm filtered out before the list reached her desk. At the end of the day, she hasn’t overridden the bias – she’s inherited and laundered it through human approval.
What we often see is that “human-in-the-loop” simply means adding a final signature step to a process that is otherwise 100% automated.
What Genuine HITL Looks Like
Meaningful human oversight requires more than procedural presence. Research from MIT Sloan and SHRM points to a complementary model in which humans add ethics, context, and creativity, while AI adds scale and pattern recognition. An effective HITL framework must:
- Focus human review on high-risk, low-confidence cases rather than applying perfunctory sign-offs to all decisions
- Set clear accountability and decision rights so that an identifiable person is responsible for each consequential outcome, not a diffuse system
- Equip reviewers with full context, including what the AI considered, what it excluded, and how confident it was, not just what it concluded
- Monitor override rates: if human analysts are almost never overriding AI recommendations, that is a warning signal of automation bias, not a reassuring sign of AI accuracy
- Protect cognitive engagement: the risk of “metacognitive laziness”, where over-reliance on AI erodes critical thinking, is real and documented
- Ensure workers can challenge decisions: the ability to ask questions and receive explanations is not a courtesy but a legal right under GDPR and the EU AI Act and an ethical imperative beyond them
The Nordic Model
One of the most instructive counter-narratives comes from the Baltic region, where the interaction between AI adoption and workplace culture has produced strikingly different outcomes.
A 2026 Deloitte survey of 170 senior Nordic executives found that only 45% expect AI to cause major job automation within the next decade, compared to 65% globally. Nordic organisations are betting on augmentation, using AI to enhance human capabilities as opposed to replacing them.
This isn’t cultural sentimentality. It reflects structural realities such as strong labour protections, high rates of union membership, and a deeply embedded tradition of social dialogue between employers and employees.
Companies in Denmark, Finland, Norway, and Sweden negotiate significant workplace changes instead of imposing them – a process that slows transformation, no doubt, but reduces resistance and improves long-term outcomes.
The 2025 OECD report specifically noted that evidence from Nordic countries shows the negative effects of algorithmic management can be mitigated where workers have substantial influence over company decisions and meaningful engagement during implementation.
As the Trade Union Advisory Committee (TUAC) concluded, collective bargaining and worker participation are merely desirable in the age of automated HR, but they are essential safeguards against its abuse.
Algorithmic HR is not a future menace. It’s a present reality reshaping workplaces at speed, often faster than the ethical frameworks needed to govern it. The risks are real, verifiable, and, in some jurisdictions, now legally actionable:
- Algorithmic bias that entrenches discrimination
- Surveillance tools that erode trust and fuel burnout
- Automated firing systems that strip workers of due process
- Human-in-the-loop mythology that often amounts to rubber-stamping machine decisions without genuine oversight
Ethical leadership, if there’s still any left, requires the courage to question vendors, to demand transparency, to give employees a real voice, to override an algorithm when context demands it, and to resist the comfortable illusion that “the AI decided” absolves anyone of moral responsibility.
For better or worse, the technology will continue its relentless march forward. The law is beginning to catch up. What remains, and what cannot be automated, is the choice to deal with human dignity at the centre.
